mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-05-28 18:11:31 +02:00
2dc1fcc778d73e9a4bf30663a93e9e063f167799
90 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 BigBodyCobain
|
03b8053617 |
feat(flights): cumulative fuel burned + CO2 emitted per flight
Pre-fix the emissions tooltip only showed the per-hour *rate* — what most users actually want is the cumulative *amount* burned. This adds running totals computed by multiplying the model-based rate by the elapsed observation time since we first saw the airframe. New module ``flight_observations.py``: * Tracks first_seen_at + last_seen_at per icao24 hex. * Re-opens a fresh session when an aircraft is unseen for > 15 min (treated as a new flight — landed and took off, or transited a dead zone). Prevents the cumulative counter from resetting mid-flight if the trail-rendering cache prunes the trail. * Clamps elapsed time to 24h max so clock skew can't produce comically large numbers. * Pruned every 5 min via a new scheduler job (mirrors ais_prune cadence). flights.py + military.py emission enrichment now also attaches: * observed_seconds — how long we've been tracking this airframe. * fuel_gallons_burned — rate * elapsed_h. * co2_kg_emitted — rate * elapsed_h. The existing per-hour rate fields stay in the dict for backward compat and are shown as small secondary context in the tooltip. Frontend EmissionsEstimateBlock (NewsFeed.tsx) now prominently shows the cumulative totals with the rate as smaller context underneath plus "Observed in flight for Xh Ym". When observed_seconds is 0 (first refresh) it renders "Just observed · totals will appear on next refresh" instead of a misleading "0 gal". 12 backend tests cover record/accumulate/reset, the 24h clamp, prune, case-insensitive key normalization, and end-to-end emission integration in _classify_and_publish. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Shadowbroker
|
20807a2d62 |
Merge pull request #316 from BigBodyCobain/feat/aishub-fallback
feat(ais): AISHub REST fallback when AISStream is offline (20-min polling) |
||
|
Shadowbroker
|
79fbf9741b |
Merge pull request #314 from BigBodyCobain/feat/ais-upstream-health
feat(ais): surface AISStream upstream outage instead of failing silently |
||
|
BigBodyCobain
|
a2f5d62926 |
feat(ais): AISHub REST fallback when AISStream WebSocket is offline
When stream.aisstream.io is unreachable (cert outage, server down — see 2026-05-20 and 2026-05-23 events) the ships layer goes empty. This adds a slow REST fallback to data.aishub.net so the layer stays populated in degraded mode. Behavior: * Opt-in via AISHUB_USERNAME (free registration at aishub.net/api). Without the env var the fetcher is a no-op. * Default poll cadence 20 min — well inside their free-tier limits, gives ships time to move enough to look "alive". Configurable via AISHUB_POLL_INTERVAL_MINUTES, clamped to [1, 360]. * Internal gate: skips the poll entirely when the WebSocket primary is currently connected. Stomping fresh live data with 20-min-old REST data would be worse than leaving it alone. * Vessels merge into the shared _vessels dict with source="aishub" so the existing UI / health tooling can attribute the provider. * Live data wins races: if a WebSocket update for the same MMSI lands in the last 1s, we don't overwrite with the slower REST record. Scheduler job runs every AISHUB_POLL_INTERVAL_MINUTES minutes alongside the existing ais_prune job in data_fetcher.py. 24 tests cover gating (no-username, primary-connected), response parsing (success / error / empty / malformed / unexpected shape), record normalization (sentinels, missing fields, range checks, AIS @ padding), poll interval clamping, and end-to-end merge with live-data-wins. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
BigBodyCobain
|
5e0b2c037e |
feat(ais): surface upstream outage instead of failing silently
On 2026-05-23, stream.aisstream.io went fully offline (TCP timeouts on port
443). The backend kept respawning the node WebSocket proxy every few
seconds with nothing arriving. From the operator's POV the ships layer
silently went empty — no banner, no log surfacing, no way to tell whether
it was their config / network / viewport filter / upstream.
Backend:
* ais_proxy_status() now also returns:
- connected (bool): true when a vessel message arrived in last 60s
- last_msg_age_seconds (int | None)
- proxy_spawn_count (int): proxy respawns — sustained growth without
connected means upstream is dead
* /api/health escalates top status to "degraded" when AIS_API_KEY is set
but the proxy is currently disconnected. Existing degraded_tls signal
preserved.
Frontend:
* useAisUpstreamHealth hook polls /api/health every 30s, derives the
outage state. Defensively only reports outage once spawn_count > 0 so
operators who haven't opted in don't see the banner.
* AisUpstreamBanner component renders a dismissible amber notice
"Ship data temporarily unavailable — AISStream upstream is offline"
mounted on the main app shell.
7 backend tests pin the status-shape contract and the /api/health
escalation behavior in both with-key and without-key configurations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
Shadowbroker
|
69ef231e5a |
Merge pull request #313 from BigBodyCobain/feat/flight-source-attribution
feat(flights): stamp source attribution on every flight record |
||
|
Shadowbroker
|
7a5f47ca9e |
Merge pull request #312 from BigBodyCobain/fix/gps-jamming-thresholds
fix(gps-jamming): count nac_p=0 + lower thresholds so layer actually fires |
||
|
BigBodyCobain
|
f14d4feb6d |
feat(flights): stamp source attribution on every flight record
Pre-fix, adsb.lol records (the primary source for most flights) carried no source marker. OpenSky records got is_opensky: True and supplementals got supplemental_source, so any UI inspecting source labels saw OpenSky/airplanes.live records as explicitly tagged and adsb.lol records as "unlabeled" — making it look like adsb.lol wasn't being used at all even though it's the primary source. Changes: * _fetch_adsb_lol_regions stamps source="adsb.lol" on each aircraft before returning, so the tag survives the OpenSky dedupe-by-hex merge. * OpenSky records get source="OpenSky" (alongside is_opensky=True for back-compat). * military fetcher tags source on both adsb.lol and airplanes.live records before they're merged, and propagates source into the military_flights and uavs output dicts. * _classify_and_publish promotes the explicit source field into the published flight dict. Falls back to legacy supplemental_source if source is absent. Final fallback "adsb.lol" preserves prior behavior for any caller synthesizing records without going through a fetcher. 8 new tests cover the published-dict propagation, OpenSky tagging, supplemental fallback, explicit-wins precedence, default behavior, the adsb.lol regional fetcher tagging, and the military output dict. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
BigBodyCobain
|
19a8560a80 |
fix(gps-jamming): count nac_p=0 + lower thresholds so the layer actually fires
Three stacked filters meant the gps_jamming layer almost never lit up: 1. nac_p == 0 aircraft were dropped on the theory that "0 = old transponder." That's only half right — modern Mode-S Enhanced Surveillance transponders also fall back to nac_p=0 when they lose GPS lock entirely, which IS the jamming signature we want to catch. Discarding them was discarding the strongest signal. None (no field at all — typical for OpenSky-sourced records) is still skipped because absence-of-data isn't evidence. 2. GPS_JAMMING_MIN_AIRCRAFT was 5 per 1°x1° cell. Jamming hotspots (eastern Med, Russia/Ukraine border, Iran/Iraq) tend to have sparser traffic because pilots avoid them. Lowered to 3. 3. GPS_JAMMING_MIN_RATIO was 0.30. Combined with the (preserved) -1 noise cushion that made the effective bar high. Lowered to 0.20. The 1-aircraft noise cushion is intact so a single quirky transponder still can't flag a zone alone. Also extracted the detector loop into a pure ``detect_gps_jamming_zones()`` function at module scope so it's testable in isolation (was previously inlined inside ``_classify_and_publish``). The public signature accepts threshold overrides for ad-hoc re-tuning without code edits. 16 new tests cover nac_p=0 inclusion, None-skip preservation, MIN_AIRCRAFT lowering, MIN_RATIO lowering, noise cushion preservation, constant pinning, override behavior, lon/lng key compatibility, and robustness to empty/None inputs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
BigBodyCobain
|
0d0e009867 |
fix(uap): stop HF fallback from serving 3-year-old NUFORC sightings
The UAP sightings layer is sourced from a live scrape of nuforc.org with a
static Hugging Face CSV mirror (kcimc/NUFORC) as a fallback. The fallback
parsed every row, sorted by occurred-desc, and took the top 250 — with no
date cutoff. The HF mirror is a third-party snapshot that hasn't been
refreshed in years, so the "newest 250" rows it returns are from ~2022-23.
When the live path fails (Cloudflare 403, curl disabled on Windows, wdtNonce
regex stale, etc.) users see a map full of sightings from 3 years ago,
labeled as the "last 60 days" layer.
Changes:
* HF fallback now applies the same 60-day cutoff the live path uses. Rows
outside the window are dropped before take-top-N. If the mirror has
nothing inside the window the fallback returns [] (don't serve stale).
* When the HF mirror is fully stale a loud ERROR log fires with the count
of dropped rows so the operator can tell the mirror's the problem, not
a network issue.
* When BOTH live AND HF fallback produce 0 rows, fetch_uap_sightings now
trips assert_canary("uap_sightings", 0) so the health registry shows
the layer as broken instead of "fresh and empty for days."
* Scheduler moved from daily 12:00 UTC to weekly Mondays 12:00 UTC. The
layer is a rolling 60-day digest; refreshing once a week is enough
cadence for human-readable map exploration and keeps nuforc.org load
light.
6 new tests cover the cutoff filter, the doomsday-log path, the mixed-age
path, the both-paths-empty health failure, the positive fallback path, and
the scheduler cadence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
BigBodyCobain
|
31ebcb5cd9 |
Infonet sync: honor HTTP 429 Retry-After + exponential backoff
Fixes the retry-storm that's been keeping the local node 429'd out of
the seed peer (the diagnosis we ran earlier in the session). Pre-fix:
1. Sync hits the seed peer, gets HTTP 429 (Too Many Requests)
2. _peer_sync_response stringifies the status into a ValueError
3. _sync_from_peer catches it, error becomes the str() of the exc
4. _run_public_sync_cycle calls finish_sync(error=..., failure_backoff_s=60)
5. next_sync_due_at = now + 60s
6. After 60s, sync runs again, hits same upstream that hasn't reset
its rate-limit bucket, 429 again. Loop indefinitely.
Net effect: a node that hit one transient 429 would hammer the seed
every 60s forever, keeping the bucket full and never recovering. We
saw this in the live status dump: consecutive_failures=49,
last_sync_ok_at=0, retry storm sustained over the entire uptime.
What changed
------------
services/mesh/mesh_infonet_sync_support.py
* New typed exception PeerSyncRateLimited carries the parsed
Retry-After value out of the HTTP layer instead of stringifying
everything into a generic ValueError.
* New parse_retry_after_header() handles both RFC 7231 §7.1.3
forms (delay-seconds and HTTP-date). Clamped at 1 hour so a
hostile peer can't silence us for days.
* New _failure_backoff_seconds() helper computes the next delay
as max(exponential, retry_after_s). Schedule with default
base=60s, cap=1800s:
failure 1 -> 60s (preserves pre-fix for transient blips)
failure 2 -> 120s
failure 3 -> 240s
failure 4 -> 480s
failure 5 -> 960s
failure 6+ -> 1800s (capped at 30 min)
cap_s=0 explicitly disables exponential entirely — operators
who want pure-Retry-After behavior have that option.
* finish_sync now accepts retry_after_s and failure_backoff_cap_s
kwargs. Backward-compatible: existing callers that don't pass
retry_after_s get the same first-failure delay as before (the
base value), only repeat failures grow.
main.py
* _peer_sync_response detects 429 specifically, parses the
Retry-After header, raises PeerSyncRateLimited(retry_after_s=N).
Includes the response body prefix in the message so the
operator's last_error finally shows something useful.
* _sync_from_peer extended to return (ok, error, forked,
retry_after_s) — the 4th tuple element is non-zero only when
the upstream sent a parseable Retry-After. Existing call shape
preserved: the lone caller in _run_public_sync_cycle was
updated in the same commit.
* _run_public_sync_cycle forwards retry_after_s into finish_sync.
Tests
-----
backend/tests/mesh/test_infonet_sync_429_backoff.py — 17 new tests:
TestParseRetryAfter (7):
- integer seconds form
- HTTP-date form (computed as seconds-from-now)
- HTTP-date in the past returns 0
- empty / whitespace returns 0
- malformed returns 0
- clamps to 1 hour (hostile-peer cap)
- negative returns 0
TestFailureBackoffSeconds (5):
- exponential growth schedule pins each level
- retry_after wins when larger than exponential
- exponential wins when larger than retry_after
- cap_s=0 disables exponential entirely
- zero inputs return zero
TestFinishSyncBackoff (5):
- first failure uses base unchanged (pre-fix back-compat)
- consecutive_failures actually grow the delay
- retry_after honored at low failure count
- success resets consecutive_failures
- last_error carries the HTTP status / Retry-After detail
All 24 existing sync-support / status-gate tests still pass. Other
failures in tests/mesh/ are pre-existing on origin/main and unrelated
to this change (verified by running the same tests against the
user's main worktree without these edits).
What the operator sees after this lands + a docker rebuild
----------------------------------------------------------
With the live 429 storm we diagnosed:
Pre-fix: consecutive_failures keeps climbing 1/min forever,
last_error empty or generic
Post-fix: consecutive_failures grows, next_sync_due_at backs off
exponentially (max 30 min), last_error explicitly carries
"HTTP 429 from <peer> (retry_after=Ns): <body>" so the
operator can see what's actually wrong. Once the upstream
bucket drains and a sync succeeds, consecutive_failures
resets to 0 and the schedule returns to the normal 300s
interval.
|
||
|
Shadowbroker
|
b3fca3dc18 |
Merge pull request #309 from BigBodyCobain/feat/cross-node-dm-mailbox-replication
DM mailbox: per-(sender, recipient) anti-spam cap + replication primitives |
||
|
BigBodyCobain
|
401f114e4f |
DM mailbox: outbound replication + receiving endpoint
Second commit on this branch (first added the per-sender cap + accept_replica primitive). This commit wires the actual cross-node propagation: Outbound (sender side) ---------------------- * New ``DMRelay._replicate_envelope_to_peers_async()`` — fire-and-forget thread that POSTs the envelope to every authenticated relay peer via the same per-peer HMAC pattern gate-message replication uses (#256 ``X-Peer-Url`` + ``X-Peer-HMAC`` headers, ``resolve_peer_key_for_url``). * ``deposit()`` now calls the replication helper after a successful local accept. Per-peer errors are swallowed — slow Tor peers must not block the sender's UX, and the recipient polling from a healthy peer works fine even if some peers are down. * Metrics: dm_replication_push_ok / _rejected / _error. Inbound (receiving side) ------------------------ * New endpoint ``POST /api/mesh/dm/replicate-envelope`` in routers/mesh_peer_sync.py. * Same HMAC auth gate (``_verify_peer_push_hmac``) as the existing infonet/gate peer-push endpoints. Unauthenticated requests get 403. * Body cap of 64 KB (DM envelope is bounded by MESH_DM_MAX_MSG_BYTES). * Calls DMRelay.accept_replica which enforces the per-sender cap as a network rule — hostile sender's relay can hold extras locally but honest peers reject them on inbound replication. End-to-end flow now works ------------------------- 1. Alice's node accepts a deposit to Bob's mailbox (local cap check). 2. Alice's node spawns a background thread that POSTs the envelope to MESH_RELAY_PEERS with per-peer HMAC. 3. Each peer's /api/mesh/dm/replicate-envelope verifies the HMAC and calls accept_replica, which re-enforces the per-sender cap. 4. Bob (offline at the time of send) eventually logs into ANY node in MESH_RELAY_PEERS, his existing pollDmMailboxes pulls from the local mailbox there, finds Alice's envelope, decrypts. Tests ----- backend/tests/test_dm_replicate_envelope_endpoint.py — 4 tests: TestReplicateEndpointAuth: - rejects requests without peer HMAC (403) - rejects requests with WRONG peer HMAC (403) — confirms the HMAC is actually verified, not just present - rejects oversize bodies (>64 KB) with 400/413 TestReplicateEndpointRegistered: - static check that POST /api/mesh/dm/replicate-envelope is registered on app.routes — catches future refactor that drops the router include All 38 backend tests touching the new code paths still pass: test_dm_relay_per_sender_cap.py (14) test_dm_replicate_envelope_endpoint.py (4) test_no_new_duplicate_routes.py (1) — new route is unique test_per_peer_secret_resolver.py (19) — HMAC primitive unaffected What's still ahead (PR-3+) -------------------------- * ack propagation: when recipient pulls a message on node X, peers Y/Z should prune their copies to free the sender's quota network-wide. Without this, the sender's quota frees only on the node the recipient actually polled — other peers still see N pending until TTL expiry. Workable but suboptimal. PR-3 will add a /api/mesh/dm/ack endpoint with the same HMAC pattern. * recipient pull-from-peers: today the recipient's poll only hits their own node's relay. If they log into a peer they didn't deposit with, they need a way to fetch envelopes from other peers in MESH_RELAY_PEERS. Today this works as long as the recipient's current node is one of the peers Alice's node pushed to — which is true in a fully-meshed deployment but not guaranteed for partial meshes. PR-4 if telemetry shows this matters. |
||
|
BigBodyCobain
|
79b39e8985 |
DM mailbox: per-(sender, recipient) anti-spam cap + replication primitives
Foundation work for cross-node DM mailbox replication. Adds the network
rule that makes the replication safe to ship next, plus the primitives
the outbound replication PR will call.
The rule
--------
A single sender can have at most N UNACKED messages parked in a single
recipient's mailbox at any one time. Default N=2, tunable via
``MESH_DM_PENDING_PER_SENDER_LIMIT``. Once the recipient pulls (acks) a
message, the sender's quota for that (sender, recipient) pair frees up.
Network rule, not local rule
----------------------------
The cap is enforced TWICE:
1. ``DMRelay.deposit(...)`` — local check on the sender's own node.
Refuses to spool the (N+1)th message before it can be replicated.
2. ``DMRelay.accept_replica(...)`` — replication-acceptance check on
every receiving peer. Refuses to accept an inbound replica that
would put the local mailbox over the cap.
The second half is what makes the rule a NETWORK rule. A hostile sender
could patch out the deposit check on their own relay and continue to
spool extras locally — but those extras can never propagate, because
every honest peer enforces the same cap on the way in. A recipient who
polls from honest peers therefore never sees more than N pending from
any one sender, regardless of how many spam attempts the hostile
sender's relay accepted.
New API surface on ``DMRelay``
------------------------------
_per_sender_pending_limit() — reads MESH_DM_PENDING_PER_SENDER_LIMIT
_per_sender_pending_count(...) — counts unacked from a sender for a mailbox
accept_replica(envelope=...) — peer-push receive entry point
envelope_for_replication(...) — helper to extract a wire-form envelope
``accept_replica`` is idempotent on duplicate ``msg_id`` (replication
round-trips and multi-path delivery don't double-spool).
``envelope_for_replication`` exposes the exact shape ``accept_replica``
expects, so the follow-up PR (outbound replication wiring) just has to
fetch the envelope and POST it to authenticated peer URLs with the
existing per-peer HMAC pattern from #256.
Why this is PR-1 of two
-----------------------
The full cross-node mailbox replication needs three pieces:
A. cap enforcement on deposit (in this PR)
B. cap enforcement on replica acceptance (in this PR)
C. outbound: push envelope to MESH_RELAY_PEERS after deposit (NEXT PR)
(A) + (B) shipped together close the cap-bypass attack surface BEFORE
(C) introduces the actual cross-node propagation. Shipping them in the
other order would briefly let extras propagate during the window between
"outbound push lands" and "accept_replica cap lands."
Tests
-----
backend/tests/test_dm_relay_per_sender_cap.py — 14 tests:
TestDepositCap:
- first 2 deposits succeed (UX baseline)
- 3rd from same sender rejected with friendly message
- different senders have independent quotas
- different recipients have independent quotas
- ack frees the quota (after recipient pulls, sender can deposit again)
- cap is env-tunable
TestAcceptReplicaCap:
- replica accepted under cap
- idempotent on duplicate msg_id (no double-spool, no rejection)
- rejected at cap with structured ``cap_violation`` marker so
sender's relay can stop retrying
- per-sender, not per-mailbox: different sender_block_ref passes
even when another sender at the same mailbox is capped
- malformed envelope shapes rejected without crash
TestEnvelopeForReplication:
- returns the envelope for stored messages
- returns None for unknown msg_id
- round-trips through accept_replica end-to-end (proves the wire
shape matches across the two sides)
|
||
|
BigBodyCobain
|
b041b5e97c |
Fix #298: move Sentinel credentials from browser storage to backend .env
Reported by @tg12. Pre-fix, the Settings panel stored real third-party
Copernicus CDSE client_id + client_secret in browser localStorage /
sessionStorage via the privacy storage helper, and the proxy routes
required those values to come back in every tile/token request body.
Any same-origin script (XSS, malicious browser extension, dev-tools
HAR export) had read access to the credentials.
This change moves them server-side, behind the same .env-backed admin
flow every other third-party API key (OpenSky, AIS Stream, Finnhub,
Shodan, …) already uses.
Backend
-------
backend/services/api_settings.py
* Added SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET entries to
API_REGISTRY. The existing GET/PUT /api/settings/api-keys flow
(already require_local_operator-gated, .env-backed) now manages
them — no new route surface.
backend/routers/tools.py
* /api/sentinel/token and /api/sentinel/tile resolve credentials via
a new _resolve_sentinel_credentials() helper: body fields win for
back-compat with any legacy callers, otherwise the helper reads
SENTINEL_CLIENT_ID / SENTINEL_CLIENT_SECRET from os.environ.
* When neither source has a value, the route returns 400 with a
friendly pointer ("Set SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET
in the API Keys panel") instead of the curt "required" message.
The user's standing rule against hostile errors applies.
* Function bodies only — decorator lines untouched, so this PR does
not conflict with #303 (which adds Depends(require_local_operator)
to the same routes).
Frontend
--------
frontend/src/lib/sentinelHub.ts — rewritten
* Removed: getSentinelCredentials / setSentinelCredentials /
clearSentinelCredentials / getSentinelCredentialStorageMode.
These were the browser-storage read/write helpers; their existence
was the bug.
* Added: checkBackendSentinelStatus(), refreshSentinelStatus(),
getCachedSentinelStatus(), and a kept-for-back-compat
hasSentinelCredentials() shim. Status is sourced from
/api/settings/api-keys (the same endpoint the API Keys panel
already uses), so we don't add a new route just for this read.
* Added: migrateLegacySentinelBrowserKeys() — one-shot, idempotent
helper that clears sb_sentinel_client_id / _secret / _instance_id
from BOTH localStorage and sessionStorage. We deliberately do NOT
auto-POST those legacy browser values to the backend; doing so
would silently migrate a secret across a trust boundary without
operator consent. Operators re-enter once in the API Keys panel
and the legacy keys get wiped here.
* fetchSentinelTile and getSentinelToken no longer send client_id /
client_secret in the request body. The backend uses .env.
frontend/src/components/SettingsPanel.tsx
* Dropped sb_sentinel_client_id / _secret / _instance_id from
PRIVACY_SENSITIVE_BROWSER_KEYS — they're no longer written.
* SentinelTab rewritten: removed the inline Client ID / Client Secret
inputs + Save / Clear / Test buttons. Replaced with a status panel
that calls checkBackendSentinelStatus() on mount, a one-click
"Open API Keys Panel" button, and a migration banner that appears
only when migrateLegacySentinelBrowserKeys() actually cleared
something.
* Setup guide STEP 3 now points to the API Keys panel instead of
the local form.
frontend/src/app/page.tsx
* Added a one-time useEffect that fires checkBackendSentinelStatus()
on mount so the cached value (which the synchronous
hasSentinelCredentials() shim reads) is populated before
MaplibreViewer's tile-URL memo runs.
Tests
-----
backend/tests/test_sentinel_credentials_server_side.py (new)
* API_REGISTRY surface — sentinel_client_id / sentinel_client_secret
are registered with the right env_keys, ALLOWED_ENV_KEYS lets
/api/settings/api-keys PUT them.
* Resolution order — body wins, env is fallback, neither → 400 with
the friendly pointer message, and NO upstream HTTP call when
neither source has credentials (asserted via
MagicMock(side_effect=AssertionError)).
* /api/sentinel/tile same shape.
frontend/src/__tests__/utils/sentinelHub.test.ts (new)
* migrateLegacySentinelBrowserKeys clears localStorage AND
sessionStorage, reports what it cleared, idempotent.
* fetchSentinelTile + getSentinelToken POST WITHOUT client_id /
client_secret in the body (plants leaked credentials in browser
storage first to prove they are NOT picked up).
* checkBackendSentinelStatus parses /api/settings/api-keys correctly:
true only when both keys is_set, false on partial config or
network errors.
All 7 backend tests + 8 frontend tests pass locally. The
test_no_new_duplicate_routes guard and the api-settings test suite
still pass.
Credit: @tg12 for the audit report.
|
||
|
Shadowbroker
|
49621824b1 |
Use USNI Fleet Tracker as the primary carrier source + small UI fixes (#293)
Background ========== PR #285 set up the seed -> cache -> GDELT model for the carrier tracker to address audit issues #244/#245/#246. The GDELT half of that pipeline hits api.gdeltproject.org's doc API for headline-region keyword matching -- low precision (false centroid positions per #245) AND unreliable (the host times out from some networks, including Docker Desktop on Windows). USNI publishes a weekly Fleet & Marine Tracker with explicit prose like: "The Gerald R. Ford Carrier Strike Group is operating in the Red Sea" "Aircraft carrier USS George Washington (CVN-73) is in port in Yokosuka, Japan" That is a strictly better source for U.S. Navy carrier positions: authoritative, deterministically parseable, weekly cadence. What this PR does ================= New module: backend/services/fetchers/usni_fleet_tracker.py - Pulls USNI's WordPress RSS feeds (site-wide + category, unioned). - Picks the most recent fleet-tracker post by parsed pubDate. - For each carrier in the registry, scans the article body for "is operating in / is in port in / returned to / transiting" near the carrier's name, hull code, or "<name> Carrier Strike Group" variant. Captures the region/port phrase that follows. - Maps the region phrase to coordinates via the existing REGION_COORDS table, with a USNI-phrase alias table for the specific wording USNI uses ("Yokosuka, Japan", "Norfolk, Va.", "Naval Station San Diego", "5th Fleet AOR", etc.). - Returns {hull: position_entry} with position_confidence="recent" and position_source_at = the article's actual publication timestamp (not now()). Politeness ---------- Uses outbound_user_agent("usni-fleet-tracker") so USNI sees a per-install Shadowbroker identifier (Round 7a / PR #292). The article body pages return 403 to non-browser UAs; the WordPress RSS feed serves the full <content:encoded> body and is the supported aggregator path. No browser UA spoofing. carrier_tracker.update_carrier_positions() now runs three phases: 1. Bootstrap from cache (or seed on first run). 2. USNI fleet tracker -- PRIMARY high-confidence source. 3. GDELT -- SECONDARY backfill; can NOT demote a "recent" USNI position to an "approximate" GDELT headline match. Verified live: 6 of 11 carriers picked up real May 18, 2026 positions on first refresh (Eisenhower, Ford, Bush, Roosevelt, Lincoln, Washington). The other 5 weren't mentioned in this week's article (they're in port at homeports with no deployment changes) and kept their cache entries -- which is the correct seed/cache contract from PR #285. Other small fixes bundled in ============================ docker-compose.yml: add the 6 third-party-fetcher opt-in env vars (PREDICTION_MARKETS_ENABLED, FINANCIAL_ENABLED, FIMI_ENABLED, NUFORC_ENABLED, NEWS_ENABLED, CROWDTHREAT_ENABLED). They were documented in .env.example but never wired through compose, so setting them in .env had no effect. frontend/src/components/TopRightControls.tsx: fix 6 broken i18n keys that were showing as raw "terminal.term1" / "terminal.cleanupDetail" / "node.soloReady" placeholders in the INFONET TERMINAL modal. The translation files have these strings under different key names; the component now calls the right ones. Full-file sweep confirmed every other t('...') key in the whole frontend resolves cleanly. |
||
|
Shadowbroker
|
76750caa92 |
Round 7a: per-operator outbound attribution + GDELT GCS-direct fix (#292)
== Per-install operator handle for every third-party API call ==
Before this PR, every Shadowbroker install identified itself to
Wikipedia, Wikidata, Nominatim, GDELT, OpenMHz, Broadcastify,
weather.gov, NUFORC, Sentinel/Planetary Computer, TinyGS / CelesTrak,
Shodan, Finnhub, and others with a single project-wide User-Agent
("Shadowbroker/1.0" or "ShadowBroker-OSINT/1.0"). From the upstream's
perspective every install in the world looked like one giant scraper.
If one install misbehaved, the upstream's only recourse was to block
"Shadowbroker" as a whole.
PR #284 inadvertently doubled down on this in the frontend by
introducing a shared `WIKIMEDIA_API_USER_AGENT` constant. This PR
retrofits both backends to per-operator attribution.
New setting: OPERATOR_HANDLE (env var / settings UI / auto-gen)
New helper: network_utils.outbound_user_agent("purpose")
The handle is auto-generated as "operator-XXXXXX" on first call (the
"shadow-" prefix from earlier drafts was deliberately dropped — too
suspicious-looking for abuse-detection systems). Operators can
override via OPERATOR_HANDLE; the value is sanitized to lowercase
alphanumeric+dash+underscore and capped at 48 chars. Persisted to
backend/data/operator_handle.json so it survives container restarts.
Retrofitted call sites (every previously-MONSTER User-Agent):
- services/region_dossier.py (Wikipedia + Wikidata + Nominatim)
- services/geocode.py (Nominatim)
- services/sentinel_search.py (Microsoft Planetary Computer)
- services/feed_ingester.py (operator-curated RSS feeds)
- services/fetchers/earth_observation.py (weather.gov, NUFORC)
- services/fetchers/infrastructure.py
- services/fetchers/aircraft_database.py
- services/fetchers/route_database.py
- services/fetchers/trains.py
- services/fetchers/meshtastic_map.py
- services/shodan_connector.py
- services/unusual_whales_connector.py (Finnhub)
- services/tinygs_fetcher.py (CelesTrak + TinyGS)
- services/sar/sar_products_client.py
- services/geopolitics.py (GDELT)
- services/radio_intercept.py (Broadcastify + OpenMHz)
- routers/cctv.py + main.py (CCTV proxy)
- routers/ai_intel.py
- scripts/convert_power_plants.py (release-time data refresh)
Spoofed browser UAs removed (issues #289 / #290 / #291 — tg12 audit):
- cloudscraper-based Chrome impersonation against api.openmhz.com
-> replaced with honest requests + per-install UA
- Mozilla/5.0 spoofed UA on Broadcastify scrape
-> replaced with honest UA
- Mozilla/5.0 + fake first-party Referer on OpenMHz audio relay
-> replaced with honest UA
- cloudscraper dependency dropped from pyproject.toml + uv.lock
Frontend retrofit:
- new GET /api/settings/operator-handle endpoint (local-operator
gated) returns the install's handle
- frontend/src/lib/wikimediaClient.ts fetches the handle once on
first use, caches it for page lifetime, embeds it in the
Api-User-Agent for every Wikipedia / Wikidata browser-direct call
== GDELT GCS-direct fix ==
GDELT's data.gdeltproject.org is a CNAME to a Google Cloud Storage
bucket. GCS responds with the wildcard *.storage.googleapis.com cert
which legitimately does NOT cover the GDELT custom domain, so Python's
TLS verification correctly refuses the connection. Some networks
happen to route through a path where this works; many (notably Docker
Desktop's outbound NAT on local installs) do not. Verified on the
maintainer's local install: GDELT was unreachable; 1610 geopolitical
events / 48 export files were dropping silently.
Fix: services/geopolitics._gcs_direct_gdelt_url() rewrites any
data.gdeltproject.org URL to its GCS-direct equivalent
(storage.googleapis.com/data.gdeltproject.org/...) where the standard
GCS cert is genuinely valid. api.gdeltproject.org and every other host
are left untouched.
Confirmed live: backend log goes from
GDELT lastupdate failed: 500
to
Downloading 48 GDELT export files...
Downloaded 48/48 GDELT exports
GDELT parsed: 1610 conflict locations from 48 files
== Tests ==
backend/tests/test_per_operator_outbound_attribution.py (12 tests)
backend/tests/test_gdelt_gcs_direct_rewrite.py (6 tests)
backend/tests/test_region_dossier_wikimedia_ua.py (updated to
pin the helper + per-operator handle, not the old constant)
frontend/src/__tests__/utils/wikimediaClient.test.ts (rewritten
to mock /api/settings/operator-handle and assert per-operator UA)
Local: backend 114/114 security+audit+round7a suite green;
frontend 718/718 vitest suite green.
Credit: tg12 (external security audit, issues #289/#290/#291
relating to spoofed UAs); BigBodyCobain (operator-prefix call,
GDELT cloud-vs-local diagnosis).
|
||
|
Shadowbroker
|
5e6bb8511a |
Fix #244/#245/#246: carrier tracker seed/cache/freshness model (#285)
Replace the dated editorial fallback positions baked into the registry with a one-shot seed file + persistent observation cache. The user's runtime cache now reflects what THIS install has actually observed, not what USNI published on March 9, 2026. A year from now, the cache holds a year of observations and the seed is irrelevant. == #244: dated editorial coordinates out of the registry == CARRIER_REGISTRY no longer carries fallback_lat/lng/heading/desc. Those fields are deleted. The registry is now identity + homeport only. New file: backend/data/carrier_seed.json - Read-only, shipped with every release. - Used ONCE on first-ever startup to bootstrap carrier_cache.json. - Each entry stamped with position_confidence="seed" and the actual as-of date (2026-03-09), NOT now(). == #245: approximate confidence for headline-derived positions == _parse_carrier_positions_from_news() now stamps every GDELT-derived entry with position_confidence="approximate" so the UI knows the coordinate is a region-centroid match, not a precise observation. After the freshness window the label rolls over to "stale_approximate" so old-and-imprecise is distinguishable from recent-and-imprecise. The article's actual seendate is used as position_source_at instead of now(), so the "last reported X days ago" badge is honest. == #246: freshness is labelling, not eviction == The cache always preserves the last position the system observed, forever. What changes is the position_confidence label: - within configurable window (default 14d, env-overridable via SHADOWBROKER_CARRIER_FRESHNESS_DAYS) -> "recent" - older -> "stale" - seed-bootstrap entries that were never refreshed -> "seed" - homeport defaults (carrier added post-install) -> "homeport_default" - headline-derived (any age, fresh) -> "approximate" - headline-derived (older than window) -> "stale_approximate" The position itself never reverts to the seed or the registry. The user always sees the last position the system observed. Per the user's explicit guidance: "from there have it be the last position the user has logged the carriers that way a year from now it doesnt revert to where the ships are today". == Other improvements == - CACHE_FILE moved to backend/data/carrier_cache.json so it lives in the volume-mounted dir under Docker compose. Previously it was at /app/carrier_cache.json which got wiped on every container restart (pre-existing bug). - Atomic cache write (temp + os.replace) so a crash mid-write does not leave a truncated cache file. == Public API shape == Every carrier object the API emits now includes: - position_confidence: seed | recent | stale | approximate | stale_approximate | homeport_default - position_source_at: ISO timestamp of when the underlying source was observed (NOT now()) - is_fallback: convenience boolean for the UI; true when the confidence is seed/stale/stale_approximate/ homeport_default Existing fields (estimated, source, source_url, last_osint_update, name, type, lat, lng, country, desc, wiki) are preserved exactly so the current ShipPopup frontend renders unchanged. last_osint_update now reflects position_source_at instead of now(), which is what the existing "last reported MM/DD" badge always meant to show. Tests: backend/tests/test_carrier_tracker_quality.py — 17 tests covering seed bootstrap, subsequent-startup ignoring seed, no-seed/ no-cache homeport fallback, registry no longer has fallback fields, freshness window labelling + env override, "year-old cache entry keeps its position, only the label flips" regression, approximate confidence for headline matches, GDELT seendate ISO parser, public response shape backward compat. Credit: tg12 (external security audit, three P1/P2 issues). |
||
|
Shadowbroker
|
0fee36e8f7 |
Fix #218/#219/#220: identify ShadowBroker on Wikipedia + Wikidata calls (#284)
Wikimedia's User-Agent policy asks API clients to identify themselves with a stable, contactable identifier so their operators can rate-limit or coordinate. Before this change, ShadowBroker was sending: - Backend (region_dossier.py): generic project default UA only; no Api-User-Agent. - Frontend (useRegionDossier.ts, WikiImage.tsx, NewsFeed.tsx): zero identifying header at all; three separate copy-pasted anonymous fetches with their own module-local caches. Three separate components doing the same broken thing meant policy fixes had to happen in three places, with no shared cache or kill switch. Fix (no UX change, zero hostility): == Backend == `backend/services/region_dossier.py` now sets explicit `User-Agent` + `Api-User-Agent` headers on every outbound Wikidata and Wikipedia request via a new `_WIKIMEDIA_REQUEST_HEADERS` constant. The identifier includes a contact path (issues page on the public GitHub repo). == Frontend == New shared helper `frontend/src/lib/wikimediaClient.ts`: - `fetchWikipediaSummary(title)` — single source of truth for Wikipedia REST summary lookups, with one shared LRU cache (in-flight requests deduplicated, 512-entry cap), `Api-User-Agent` on every fetch. - `fetchWikidataSparql(query)` — same shape for Wikidata SPARQL. - `WIKIMEDIA_API_USER_AGENT` — exported constant; one place to update if Wikimedia ever asks us to back off. Refactored three components to use the shared client: - `frontend/src/hooks/useRegionDossier.ts` — fetchLeader() and fetchLocalWikiSummary() now route through the shared helpers. - `frontend/src/components/WikiImage.tsx` — uses fetchWikipediaSummary, proper React state instead of module-mutation + forceUpdate trick. - `frontend/src/components/NewsFeed.tsx` — same shape. UX: byte-for-byte identical. Same thumbnails, same dossier content, same load behavior. The only observable difference is the outgoing request header. Note on #239 (route duplication): an audit-grade inventory shows 166 main.py routes are shadowed by router modules. That cleanup is too large to land safely in this PR; it will be staged as a separate ladder of small PRs grouped by router module. Tests: - `backend/tests/test_region_dossier_wikimedia_ua.py` — 3 tests asserting backend headers are present. - `frontend/src/__tests__/utils/wikimediaClient.test.ts` — 9 tests covering Api-User-Agent presence, shared cache, concurrent deduplication, disambiguation/HTTP-error/network-error fallthroughs, empty-input safety. Local: backend 76/76 security suite green, frontend 716/716 vitest suite green. Credit: tg12 (external security audit). |
||
|
Shadowbroker
|
2e14e75a0e |
Fix #256: per-peer HMAC secrets defeat cross-peer impersonation (#281)
Before this change, every peer-push HMAC was derived from the single fleet-shared MESH_PEER_PUSH_SECRET. The receiver could prove "this request was signed by someone who knows the fleet secret" but it could NOT prove which peer signed it. Any peer that knew the global secret could compute the expected HMAC for any other peer URL and forge a push pretending to be that peer. Fix: introduce MESH_PEER_SECRETS, an optional comma-separated url=secret map. When a peer URL appears in the map, only the listed per-peer secret is accepted for it -- the global secret is ignored for that specific URL. Peer A no longer knows peer B's secret, so peer A cannot forge a push claiming to be peer B. The new helper resolve_peer_key_for_url() in mesh_crypto.py wraps the lookup and is called from every existing peer-push call site: - backend/auth.py:_verify_peer_push_hmac (receiver) - backend/main.py:_http_peer_push_loop (Infonet event push) - backend/main.py:_http_gate_pull_loop (gate event pull) - backend/main.py:_http_gate_push_loop (gate event push) - backend/services/mesh/mesh_router.py (two transports, push) - backend/services/mesh/mesh_hashchain.py (gate wire ref key) - backend/services/mesh/mesh_wormhole_prekey.py (peer prekey lookup) Zero hostility, by design: - Single-peer installs leave MESH_PEER_SECRETS empty -> resolver falls back to MESH_PEER_PUSH_SECRET -> behavior is byte-for-byte unchanged. - Multi-peer installs that haven't migrated yet behave exactly as before. - Multi-peer installs that DO migrate set MESH_PEER_SECRETS on both ends of each peering and immediately close the impersonation surface for those URLs. Migration is incremental: unlisted peers keep using the global secret. Tests in backend/tests/test_per_peer_secret_resolver.py: - env parsing (default, override, whitespace, malformed entries, cache) - precedence: per-peer beats global - migration window: unlisted peer falls back to global - IMPERSONATION REFUSAL: peer A with global-secret-only cannot forge HMAC for peer B that has a per-peer secret configured - IMPERSONATION REFUSAL: peer A with its OWN per-peer secret cannot forge HMAC for peer B - positive control: legitimate peer B request verifies - zero-behavior-change: single-peer install produces the same key bytes as before the change Credit: tg12 (external security audit, P1/High/High confidence) |
||
|
Shadowbroker
|
fb11e0881f |
Fix #251: refuse symlink/hardlink members during Tor bundle extraction (#277)
External audit (@tg12) flagged that the Tor Expert Bundle extractor
checked tarinfo.name against path traversal but never inspected
tarinfo.linkname for symlink or hardlink members. Python 3.11's
tarfile.extractall() honors symlinks, so a malicious archive could
ship a member like::
name = "innocent.txt" (passes the path-traversal check)
type = SYMTYPE
linkname = "C:\Windows\System32\config\system"
After extraction, subsequent reads of innocent.txt dereference to that
arbitrary filesystem location; subsequent writes corrupt it. On
Windows (where Tor Expert Bundle extraction actually runs), this is
a host-compromise path of essentially the same severity as the
supply-chain RCE in #231 — gated only by the integrity check we just
hardened in PR #261/#265.
Python 3.12+ added tarfile.extract / extractall filter='data' as a
built-in mitigation; we're on Python 3.11 in production, so we
implement the same idea manually.
Fix in backend/services/tor_hidden_service.py:
Extract the existing path-traversal-only check into a new
_extract_tor_bundle_safely() helper that:
1. Refuses any member with member.issym() or member.islnk() True.
Tor bundles never legitimately contain symlinks or hardlinks
so this is non-disruptive. Logs the linkname so an operator
can see what the malicious archive was trying to alias.
2. Refuses any member that isn't isfile() or isdir() — no FIFOs,
no character or block devices, no contiguous-file-type entries.
None of those belong in a Tor Expert Bundle and accepting them
is a class of bug we don't need to debug later.
3. Preserves the original path-traversal guard (member.name must
resolve under install_dir).
4. Catches tarfile.TarError so a corrupt archive returns False
gracefully instead of bubbling out an exception.
Tests: backend/tests/test_tor_bundle_symlink_filter.py (8 tests)
- Clean archive with only regular files extracts successfully
- Symlink member is rejected (the core regression)
- Hardlink member is rejected
- Symlink with relative target inside install_dir is still rejected
(we don't allow symlinks at all, not just absolute-target ones)
- FIFO/device-style member is rejected
- Path-traversal guard still works under the new shape
- Malformed/non-tar file is rejected gracefully (no crash)
- Failure on one member rejects the whole bundle (no half-extract)
Validation:
pytest backend/tests/test_tor_bundle_symlink_filter.py
backend/tests/test_tor_bundle_verification.py
-> 14 passed
UX impact: zero for legitimate Tor releases. Operators installing
a real Tor Expert Bundle continue to see "Tor installed at:" exactly
as before. Only malicious archives are refused, with a clear log
message identifying the rejected linkname.
Credit: @tg12 — the original report was specific enough that the
fix design was immediate.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
Shadowbroker
|
7f96151e56 |
Fix #231: multi-source SHA-256 verification for the self-updater (#265)
External audit (@tg12, May 18) found that backend/services/updater.py silently skipped all SHA-256 integrity verification whenever the MESH_UPDATE_SHA256 env var was unset — which is the default. Nothing in any install doc tells operators to set it, so practically every deployment was running the auto-updater with zero integrity check. That made GitHub release pipeline compromise a single-step path to arbitrary code execution on every node that auto-updates. Investigation surfaced a deeper bug too: the updater downloads zipball_url (GitHub's auto-generated source archive) but the maintainer's release process publishes SHA256SUMS.txt for a separate named asset (ShadowBroker_v*.zip). So even if MESH_UPDATE_SHA256 WERE set, operators had no published digest to compare against — the file they were downloading wasn't the file the maintainer had signed. This PR fixes both issues with the same multi-source verification chain we shipped for the Tor bundle in PR #261: backend/services/updater.py _download_release() now prefers a maintainer-signed release asset matching ShadowBroker_v*.zip over zipball_url. Captures the SHA256SUMS.txt asset URL when present. _validate_zip_hash() rewritten as a four-source chain: 1. MESH_UPDATE_SHA256 env var (operator override, preserved) 2. SHA256SUMS.txt asset published with the release (primary — the maintainer's release process already publishes this) 3. Baked-in backend/data/release_digests.json (second line of defense for releases that lack the SHA256SUMS asset, or when the asset can't be fetched at update time) 4. HTTPS-only fallback with a loud warning (preserves the auto- update flow during transient outages) Mismatch from any source that DID respond is fatal — the update is refused and the existing install keeps running. Only the "no source reachable at all" case falls back to HTTPS-only. _fetch_sha256sums() new — fetches and parses a standard SHA256SUMS.txt asset. Handles both "<digest> <name>" and binary- marker "<digest> *<name>" formats. Tolerant to comments, blank lines, and malformed entries. backend/data/release_digests.json (new) Baked-in digest list keyed by release tag. Seeded with the v0.9.79 entries copied from the published SHA256SUMS.txt: ShadowBroker_v0.9.79.zip = f6877c1d6661... ShadowBroker_0.9.79_x64-setup.exe = f7b676ada45c... ShadowBroker_0.9.79_x64_en-US.msi = e0713c3cdda1... Whitelisted in .gitignore alongside the other static reference data files (kiwisdr_directory.json, tor_bundle_digests.json, aisstream_spki_pins.json). backend/tests/test_update_integrity_chain.py (new, 16 tests) - Each source matches → success, identifies which source verified - Each source mismatches → RuntimeError "mismatch" - No source reachable → https-only fallback with loud warning - Env override beats all other sources (preserved precedence) - SHA256SUMS.txt parser handles standard, binary-marker, comments, and network-failure cases Validation: pytest backend/tests/test_update_integrity_chain.py → 16 passed pytest (all 15 security test files together) → 105 passed UX impact: zero. Normal auto-update flow is unchanged for legitimate releases (path 2 catches everything because the release publishes SHA256SUMS.txt). Transient network failures during update gracefully fall through to path 3 then path 4 — no operator intervention needed. The only user-visible behavior change is in the compromised-release case, where the update is now refused instead of silently applied. Credit: @tg12 for the original bug report and the specific call-out that MESH_UPDATE_SHA256 was unreachable by default operators. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Shadowbroker
|
729ea78cb2 |
Fix #258: AIS proxy SPKI pinning fallback for expired upstream cert (#262)
External report from @jmleclercq: AISStream's Let's Encrypt cert
expired on 2026-05-20 (verified — their renewal pipeline failed), so
the AIS WebSocket connection dies with CERT_HAS_EXPIRED and the
maritime layer empties out. The reporter worked around it locally by
passing { rejectUnauthorized: false } to the WebSocket constructor and
asked whether we should add an env var for that.
That fix is the wrong fix. Disabling TLS validation entirely lets any
network attacker MITM the WebSocket and inject fake ship positions —
same class as the GDELT plaintext-HTTP MITM we just closed in #199.
Adding an env var for it would be an attractive nuisance: operators
set it once during a bad cert week and then forget, leaving themselves
open to MITM forever.
Right fix: SPKI pinning, same pattern as the Tor bundle digest pinning
in #201. The insight is that Let's Encrypt renewals keep the SAME
public key by default, so the SPKI hash survives normal cert rotation.
We can relax the date check while keeping the identity check.
Mechanics:
backend/data/aisstream_spki_pins.json (new)
Pinned SHA-256 hashes of the DER-encoded SPKI bytes for
stream.aisstream.io. Captured 2026-05-20 from the live cert.
Format is base64(sha256(pubkey_der)), matching the canonical
openssl pipeline. Whitelisted in .gitignore alongside the other
static reference data files (KiwiSDR directory, Tor bundle
digests).
backend/ais_proxy.js
Path A (99.9% of the time): normal TLS validation. Untouched.
Path B (on CERT_HAS_EXPIRED only): re-handshake with
rejectUnauthorized=false JUST to read the leaf cert, compute its
SPKI hash, compare against the pinned list. If match → upstream
is still the genuine AISStream → re-open the WebSocket with
rejectUnauthorized=false and log DEGRADED MODE. If no match →
refuse the connection, log loudly: this would be a real MITM.
Pin file is looked up in three locations so the same code works
in the Docker backend, the Tauri desktop runtime, and any
operator-relocated layout (SHADOWBROKER_AIS_PINS env var).
Embedded fallback list inside the JS so portable installs that
haven't shipped the JSON still work.
backend/services/ais_stream.py
Captures the proxy's status markers from stdout
({"__ais_proxy_status": {"degraded_tls": true}}) into a module-
level snapshot. Exposes ais_proxy_status() for the health
endpoint. Doesn't touch the data plane — degraded mode keeps
receiving vessel data, just with weaker MITM protection.
backend/routers/health.py + backend/services/schemas.py
/api/health now includes an ais_proxy block with degraded_tls.
Top-level status escalates ok -> degraded when AIS is in
degraded TLS mode (but won't downgrade a worse SLO status).
Operators get a visible signal that they're in degraded mode
without needing to grep logs.
Tests: backend/tests/test_ais_spki_pinning.py (7 tests)
- Pin file structure validation (JSON, host entry, base64 SHA-256)
- ais_proxy_status() snapshot semantics (starts empty, defensive copy)
- /api/health surfaces ais_proxy.degraded_tls when set
- /api/health returns empty ais_proxy when proxy hasn't reported
Node.js syntax check passes (node --check) on both backend/ais_proxy.js
and the Tauri runtime mirror.
When AISStream renews their cert (likely within hours-to-days), the
normal-TLS path succeeds on next reconnect and degraded_tls clears
automatically. No operator action needed. If they instead rotate their
server key, the SPKI check will fail and we'll need to add the new
hash to backend/data/aisstream_spki_pins.json before removing the old
one.
Credit: @jmleclercq for the clear report and the careful workaround
verification (Node version, ws version, manual probe).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
@aaronjmars
|
8e27658157 |
fix(security): use defusedxml for untrusted XML parsing (#259)
Detected by Aeon + Semgrep (5x use-defused-xml ERROR). Severity: medium CWE-776 (billion laughs) / CWE-611 (XML external entity) Five XML parse sites pass response bodies into the Python stdlib xml.etree.ElementTree without protection against entity expansion attacks. Python's ElementTree still permits internal entity references by default (per the docs vulnerabilities table), so a malicious or compromised upstream can ship a "billion laughs"-style payload that expands to gigabytes in memory. The user-controllable site is sb_monitor._parse_rss: the OpenClaw skill exposes add_custom_feed(name, url, ...) to the agent, then poll_custom_feeds fetches feed.url and passes the body to xml.etree.ElementTree.fromstring with no host allowlist or entity-bomb defence. The other four sites (psk_reporter_fetcher, aircraft_database, cctv_pipeline x2) parse XML from hard-coded upstreams (pskreporter.info, s3.opensky-network.org, datos.madrid.es); defence-in-depth for upstream-compromise/MITM. Switch all five call sites to defusedxml.ElementTree. Same fromstring/find/findall/iter/findtext API, but rejects entity references by default (raises defusedxml.EntitiesForbidden). Confirmed locally that a 4-deep billion-laughs payload that expands to 3000 chars under stdlib ET is rejected by defusedxml. Added defusedxml>=0.7.1 to backend/pyproject.toml dependencies. Co-authored-by: aeonframework <aeon-bot@aaronjmars.com> |
||
|
Shadowbroker
|
e36d1fc79c |
[security] Close tg12 audit issues #201–#214 seamlessly (#261)
External security audit by @tg12 (May 17, 2026) filed issues #201–#214 in addition to the #189–#200 batch already closed by PRs #227/#232/#260. This PR closes all eight that are real security bugs (the other six in the 201–214 range are either design discussions or upstream-abuse/TOS concerns we're keeping intentional, see issue triage notes on each). The user-facing principle for this PR: fix the security gap WITHOUT introducing a single hostile error or behavior change for legitimate users. Every fix follows the same template — fail forward, not loud. When the secure path is harder than the insecure one, build a fallback chain that ends in graceful degradation, not in a scary modal or 422 response. #205 — OpenMHZ audio redirect SSRF (services/radio_intercept.py) Replaced requests.get(..., allow_redirects=True) with a manual redirect loop that re-validates each hop's host against _OPENMHZ_AUDIO_HOSTS. Same-host redirects (CDN edge selection) still work, so legitimate audio playback is unaffected. Cross-host redirects to disallowed hosts return a generic 502 which the browser audio element handles gracefully. Cap at 5 hops. #207 — infonet/status verify_signatures DoS (routers/mesh_public.py) Silently downgrade verify_signatures=true to False for unauthenticated callers. No error surfaced — the response shape is identical, just without the O(n_events) signature verification. Authenticated callers (scoped mesh.audit) still get the full path. The frontend never passes this param so legitimate UI is unaffected. #211 — thermal/verify expensive analysis (routers/sigint.py) Added Depends(require_local_operator). Frontend has no direct callers (verified by grep); Tauri/AI agents use scoped tokens that pass the auth check. Anonymous abusers blocked silently — the legitimate UI keeps working through the Next.js admin-key proxy. #213, #214 — OpenMHZ calls/audio upstream abuse (routers/radio.py) Added Depends(require_local_operator) to both. Browser users hit these through the Next.js proxy at src/app/api/[...path]/route.ts which injects X-Admin-Key, so the auth check passes transparently. Direct attackers can no longer rotate sys_names to hammer api.openmhz.com or relay arbitrary audio streams through the backend's bandwidth. #202 — overflights unbounded hours (routers/data.py) Silently clamp `hours` to OVERFLIGHTS_MAX_HOURS (default 72, configurable). NO 422 — clients asking for an absurd window get a shorter window back with `requested_hours` and `effective_hours` hint fields. Postel's law: liberal in what we accept, conservative in what we compute. #203 — Meshtastic callsign UA leak (services/fetchers/meshtastic_map.py) Added MESHTASTIC_SEND_CALLSIGN_HEADER opt-out env var. Default is TRUE — preserves existing operator behavior (callsign sent so meshtastic.org can rate-limit per-install). Privacy-conscious operators set it to false to suppress. #206 — KiwiSDR upstream is HTTP-only (services/kiwisdr_fetcher.py) Upstream rx.linkfanel.net doesn't speak HTTPS (verified — Apache 2.4.10 only on port 80). We can't fix the transport. Instead added three layers: 1. Content validation on fetched data — reject responses with <50 receivers or >5% malformed entries (likely MITM injection). 2. Existing disk cache fallback (already present). 3. NEW: bundled static directory at backend/data/kiwisdr_directory.json shipping 798 known-good receivers. Used as last resort so the KiwiSDR map layer always renders something useful. #208 — Merkle proof DoS via /api/mesh/infonet/sync (services/mesh/mesh_hashchain.py) The endpoint is part of the cross-node federation protocol — peers legitimately call it without local-operator auth, so we can't add Depends(). Instead made the underlying operation O(1) per proof via a cached Merkle level structure on the Infonet instance: - _merkle_levels_cache + _merkle_levels_for_event_count on each Infonet instance - _invalidate_merkle_cache() called from every chain mutation point (append, ingest_events, apply_fork, cleanup_expired) - _get_merkle_levels() does the lazy recompute on first read after invalidation, then serves from cache thereafter Effect: anonymous attackers hammering the proofs endpoint hit a cached structure; the rebuild happens at most once per real chain advance. Federation untouched. #201 — Tor bundle SHA-256 bypass (services/tor_hidden_service.py) Docker users were already covered — backend/Dockerfile installs Tor via apt-get at build time (signed by Debian's package system). No runtime download needed for the 80%-of-users case. For Tauri desktop, replaced the single .sha256sum check with a multi-source verification chain implemented in _verify_tor_bundle(): 1. Try upstream .sha256sum (current behavior — fast path) 2. Try baked-in digest list at backend/data/tor_bundle_digests.json (pinned per-version, maintainer-updated) 3. If neither source is REACHABLE: HTTPS-only fallback with a loud warning (avoids breaking first-run onboarding while the maintainer hasn't yet pinned a new Tor release) A mismatch from a source that DID respond is always fatal — only the "no source reachable" case falls back to HTTPS-only. This is the "have cake and eat it" pattern: real users see no new failure modes during torproject.org outages, but MITM/compromise attacks still fail because the downloaded digest can't match what BOTH the upstream and the baked-in list report. Currently the digest file ships with placeholder values for the current Tor URLs (those URLs are already stale on torproject.org too). A follow-up commit can populate real digests when a stable Tor release is selected; until then the HTTPS-only warning fires and onboarding still works. Tests (82 total, all passing): test_openmhz_redirect_ssrf.py (5 tests) — #205 test_infonet_status_verify_gate.py (2 tests) — #207 test_overflights_clamp.py (5 tests) — #202 test_meshtastic_callsign_optout.py (3 tests) — #203 test_kiwisdr_fallback.py (6 tests) — #206 test_merkle_cache.py (6 tests) — #208 test_tor_bundle_verification.py (6 tests) — #201 test_control_surface_auth.py (extended) — #211, #213, #214 + all previous security tests (CCTV redirect, GDELT https, sentinel cache, crowdthreat opt-in, third-party fetcher gates, control surface auth) continue to pass. Pre-existing test infrastructure issue with SHARED_EXECUTOR teardown in the broader sweep exists on main too (verified) — not introduced by this PR. Credit: @tg12 reported every one of these with accurate line citations and the recommended fixes that informed this implementation. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Shadowbroker
|
d00c63abed |
[security] Close tg12 audit gaps #192, #198, #199, #200 (#260)
External security audit by @tg12 (May 17, 2026) filed 11 issues against the backend. PR #227 (May 18, AI-generated) closed seven of them by adding require_local_operator to control-plane endpoints. Four remained live; this PR closes the rest. #192 — CCTV proxy followed redirects without re-validating host Issue: /api/cctv/media validated only the caller-supplied URL host before passing it to requests.get(..., allow_redirects=True). A 302 to http://127.0.0.1 or any internal/disallowed host was silently followed, turning the proxy into an open-redirect-to-SSRF chain. Fix in routers/cctv.py: replace the single allow_redirects=True call with a manual follow loop. Each hop's Location is parsed, the host is rerun through _cctv_host_allowed(), and non-HTTP schemes (file://, ftp://, etc.) are rejected. Cap chain length at 5 hops. Test: backend/tests/test_cctv_redirect_ssrf.py covers - redirect to disallowed host -> 502 - redirect to localhost -> 502 - redirect to another allowed host -> 200 - redirect chain length cap - non-HTTP scheme rejected #198 — Gate introspection GETs were unauthenticated Issue: /api/wormhole/gate/{gate_id}/{identity,personas,key} were callable with no auth dependency. Any caller that could reach the backend could dump the operator's active persona, persona inventory, and key status for any gate_id they knew. The wiki's privacy threat model explicitly markets gate personas as rotating, unlinkable pseudonyms — this leak defeated that property. Fix in routers/wormhole.py: add dependencies=[Depends(require_local_operator)] to all three routes. Test: backend/tests/test_control_surface_auth.py extended with three new parameterized cases (lines 75-77). #199 — GDELT military incident ingestion used plaintext HTTP Issue: backend/services/geopolitics.py fetched http://data.gdeltproject.org/gdeltv2/lastupdate.txt and ~48 export archive URLs over plaintext HTTP. Passive observers could identify Shadowbroker nodes from the fetch pattern. Active MITM could inject doctored military incident records into the global map. Fix in services/geopolitics.py: rewrite the lastupdate.txt fetch and the export download URL constructor to use https://. GDELT's data.gdeltproject.org serves the same content over HTTPS. Test: backend/tests/test_gdelt_https.py asserts no plaintext HTTP URLs to data.gdeltproject.org remain in code (comments excluded) and that the HTTPS URLs we expect are present. #200 — Sentinel token cache lookup used client_id only Issue: routers/tools.py kept a process-global cache of Copernicus bearer tokens. The lookup compared _sh_token_cache["client_id"] == client_id. A caller who knew a valid client_id but supplied any wrong client_secret hit the cache and reused the legitimate caller's bearer token — burning their quota and accessing imagery on their account. Fix in routers/tools.py: replace the client_id field with credential_fp, an HMAC-SHA256 over (client_id, client_secret) under a per-process random key (_SH_TOKEN_CACHE_HMAC_KEY = os.urandom(32), regenerated at startup). A caller who doesn't know the secret cannot compute a matching fingerprint, so they miss the cache and hit the real Copernicus token endpoint — which will reject their wrong secret with a 401. Test: backend/tests/test_sentinel_token_cache.py covers - same client_id + different secrets => different fingerprints - same credentials => same fingerprint (cache still works) - different client_ids + same secret => different fingerprints - cache no longer stores raw client_id (catches regression) - attacker with wrong secret cannot reuse victim's token Validation pytest backend/tests/test_control_surface_auth.py backend/tests/test_cctv_redirect_ssrf.py backend/tests/test_gdelt_https.py backend/tests/test_sentinel_token_cache.py -> 37 passed Credit: @tg12 reported all four of these in their May 17 audit with correct line-number citations and accurate remediation recommendations. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
BigBodyCobain
|
71a9d9e144 |
[security] Close post-#227 control-surface and fetcher gaps
PR #227 hardened most Wormhole/Infonet control surfaces behind require_local_operator and made the CrowdThreat fetcher opt-in. An audit of the codebase against that PR's stated goals turned up four classes of gap that the original change missed: 1. Two operator-only endpoints were left unprotected: - POST /api/wormhole/join: calls bootstrap_wormhole_identity() and flips the node into Tor mode, exactly the surface #227 hardened on /api/wormhole/identity/bootstrap. - POST /api/sigint/transmit: relays APRS-IS packets over radio using operator-supplied credentials. Anything that reached the API could transmit on the operator's authority. Both now require_local_operator. test_control_surface_auth.py extended with regression coverage for both. 2. Five third-party fetchers were still default-on, phoning home to politically/commercially sensitive upstreams on every poll cycle: - fimi.py -> euvsdisinfo.eu -> FIMI_ENABLED - prediction_markets -> Polymarket + Kalshi -> PREDICTION_MARKETS_ENABLED - financial.py -> Finnhub / yfinance -> FINANCIAL_ENABLED or FINNHUB_API_KEY - nuforc_enrichment -> huggingface.co -> NUFORC_ENABLED - news.py -> configured RSS feeds -> NEWS_ENABLED (default on, kill switch) Same CrowdThreat-style pattern: explicit env-var opt-in, empty the data slot and mark_fresh when disabled. New regression test file test_third_party_fetchers_opt_in.py asserts each fetcher's network entry point is not called when its gate is off. 3. The outbound User-Agent leaked both the operator's personal email and a fork-specific GitHub URL on every fetcher request. Consolidated to a single DEFAULT_USER_AGENT in network_utils.py, project-generic by default (no contact info), overridable via SHADOWBROKER_USER_AGENT for operators who want to identify themselves (e.g. for Nominatim or weather.gov usage-policy compliance). Six call sites updated; the Nominatim-specific override is preserved. 4. The same generic UA now also flows through the peer prekey lookup in mesh_wormhole_prekey.py, so DM first-contact requests no longer identify the caller as a Shadowbroker fork to the peer being queried. .env.example updated to document all new opt-in env vars. Tests: backend/tests/test_control_surface_auth.py (extended), backend/tests/test_crowdthreat_opt_in.py (unchanged, still passes), backend/tests/test_third_party_fetchers_opt_in.py (new, 7 tests). All 31 tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
BigBodyCobain
|
11ea345518 | Harden infonet control surfaces | ||
|
BigBodyCobain
|
25a98a9869 |
Harden Infonet DM address flow and seed sync
Allow local-operator DM invite import without requiring a full admin session. Prioritize bundled/bootstrap seed peers and shorten stale seed cooldowns for faster Infonet recovery. Replace raw DM invite dumps with copyable signed-address controls, contact request handling, and safer sealed-send behavior while the private delivery route connects. |
||
|
BigBodyCobain
|
b86a258535 |
Release v0.9.79 runtime and messaging update
Ship the v0.9.79 runtime refresh with transport lane isolation, Infonet secure-message address management, MeshChat MQTT controls, selected asset trail behavior, telemetry panel refinements, onboarding updates, and desktop/package metadata alignment. Also ignore local graphify work products so analysis folders do not leak into future commits. |
||
|
BigBodyCobain
|
5ee4f8ecd7 | Stabilize Infonet private sync and selected telemetry | ||
|
BigBodyCobain
|
b8ac0fb9e7 |
Harden v0.9.75 wormhole node sync and telemetry panels
Add Tor/onion runtime wiring and faster Infonet node status refresh. Keep node bootstrap state clearer across Docker and local runtimes. Use selected aircraft trail history for cumulative tracked-aircraft emissions. |
||
|
BigBodyCobain
|
8926e08009 | Fix cached satellite propagation | ||
|
BigBodyCobain
|
6ffd54931c |
Release v0.9.75 runtime and onboarding update
Ship the 0.9.75 source update with improved startup/runtime hardening, operator API key onboarding, Meshtastic MQTT controls, Infonet/MeshChat separation, desktop package versioning, and aircraft telemetry refinements. Also updates focused backend/frontend tests for node settings, Meshtastic MQTT settings, and desktop runtime behavior. |
||
|
BigBodyCobain
|
63043b32b5 |
Stabilize Docker startup and runtime proxy
Reduce cold-start stalls by raising the default backend memory limit, bounding heavy feed concurrency, preserving non-empty startup caches, and refreshing working news feeds. Fix the Next API proxy for Docker control-plane writes by stripping unsupported hop/body headers and forwarding small request bodies safely. Keep the dashboard dynamic so production users do not get stuck on a cached startup shell. |
||
|
BigBodyCobain
|
0fc09c9011 | Fix Docker Infonet and Wormhole startup | ||
|
BigBodyCobain
|
707ca29220 |
Add in-app local API key setup
Let fresh Docker and local installs enter OpenSky, AIS, and other provider keys directly in onboarding or Settings without manually creating .env files. Persist keys server-side in the backend data store, keep them write-only from the browser, reload runtime settings, and retain local-operator access controls. |
||
|
BigBodyCobain
|
e1060193d0 |
Improve v0.9.7 startup and runtime reliability
Prioritize cached first-paint data, defer heavyweight feed synthesis, make MeshChat activation explicit, improve CCTV media handling, and tighten desktop runtime packaging filters. |
||
|
BigBodyCobain
|
08810f2537 | fix: stabilize v0.9.7 startup and feeds | ||
|
BigBodyCobain
|
d515aba450 | fix: polish v0.9.7 micro update | ||
|
BigBodyCobain
|
4ec1fce53d | ci: unblock v0.9.7 release checks | ||
|
BigBodyCobain
|
28b3bd5ebf | release: prepare v0.9.7 | ||
|
anoracleofra-code
|
8f7bb417db |
fix: thread-safe SSE broadcast + node enabled by default
- SSE broadcast now uses loop.call_soon_threadsafe() when called from background threads (gate pull/push loops), fixing silent notification failures for peer-synced messages - Chain hydration path now broadcasts SSE so gate messages arriving via public chain sync trigger frontend refresh - Node participation defaults to enabled so fresh installs automatically join the mesh network (push + pull) |
||
|
anoracleofra-code
|
1fd12beb7a |
fix: relay nodes now accept gate messages (skip gate-exists check)
Relay nodes run in store-and-forward mode with no local gate configs, so gate_manager.can_enter() always returned "Gate does not exist" — silently rejecting every pushed gate message. This broke cross-node gate message delivery entirely since no relay ever stored anything. Relay mode now skips the gate-existence check after signature verification passes, allowing encrypted gate blobs to flow through. |
||
|
anoracleofra-code
|
c35978c64d |
fix: add version to health endpoint + warn users with stale compose files
Repo migration in March 2026 rewrote all commit hashes, leaving old clones with a docker-compose.yml that builds from source instead of pulling pre-built images. Added detection warnings to compose.sh, start.bat, and start.sh so affected users see clear instructions. Also exposes APP_VERSION in /api/health for easier debugging. |
||
|
anoracleofra-code
|
ae627a89d7 |
fix: align transport secret with cipher0 relay
Use cipher0's existing MESH_PEER_PUSH_SECRET so nodes connect to the relay out of the box without configuration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
anoracleofra-code
|
59b1723866 |
feat: fix gate message delivery + per-gate content encryption
Phase 1 — Transport layer fix: - Bake in default MESH_PEER_PUSH_SECRET so peer push, real-time propagation, and pull-sync all work out of the box instead of silently no-oping on an empty secret. - Pass secret through docker-compose.yml for container deployments. Phase 2 — Per-gate content keys: - Generate a cryptographically random 32-byte secret per gate on creation (and backfill existing gates on startup). - Upgrade HKDF envelope encryption to use per-gate secret as IKM so knowing a gate name alone no longer decrypts messages. - 3-tier decryption fallback (phase2 key → legacy name-only → legacy node-local) preserves backward compatibility. - Expose gate_secret via list_gates API for authorized members. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
anoracleofra-code
|
da09cf429e |
fix: cross-node gate decryption, UI text scaling, aircraft zoom
- Derive gate envelope AES key from gate ID via HKDF so all nodes sharing a gate can decrypt each other's messages (was node-local) - Preserve gate_envelope/reply_to in chain payload normalization - Bump Wormhole modal text from 9-10px to 12-13px - Add aircraft icon zoom interpolation (0.8→2.0 across zoom 5-12) - Reduce Mesh Chat panel text sizes for tighter layout |
||
|
anoracleofra-code
|
3cbe8090a9 |
fix: add default relay peer so fresh installs can sync Infonet
On a fresh Docker (or local) install, MESH_RELAY_PEERS was empty and no bootstrap manifest existed, leaving the Infonet node with zero peers to sync from — causing perpetual "RETRYING" status. Set cipher0.shadowbroker.info:8000 as the default relay peer in both the config defaults and docker-compose.yml so new installations sync immediately after activating the wormhole. |
||
|
anoracleofra-code
|
86d2145b97 |
fix: use paho-mqtt threaded loop for stable MQTT reconnection
The Meshtastic MQTT bridge was using client.loop(timeout=1.0) in a blocking while loop. When the broker dropped the connection (common after ~30s of idle in Docker), the client silently stopped receiving messages with no auto-reconnect. Switch to client.loop_start() which runs the MQTT network loop in a background thread with built-in automatic reconnection. Also: - Add on_disconnect callback for visibility into disconnection events - Set reconnect_delay_set(1, 30) for fast exponential-backoff reconnect - Lower keepalive from 60s to 30s to stay within Docker network timeouts |