mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-05-28 18:11:31 +02:00
8dfa6a719986faceec598880476ba2ebbd4fc256
28 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Shadowbroker
|
8dfa6a7199 |
release: v0.9.8 — Cumulative Fuel/CO2, AIS Resilience, Data-Layer Repair (#321)
Bumps every hardcoded 0.9.79 → 0.9.8 across backend, frontend,
desktop-shell, helm, lockfiles, test fixtures. Refreshes the in-app
ChangelogModal HEADLINE_FEATURES, NEW_FEATURES, and BUG_FIXES with the
v0.9.8 highlights.
Release artifacts built locally and hashed into release_digests.json:
ShadowBroker_v0.9.8.zip 6.06 MB
d506f6b8462ccb12096f0cd9462233be58928094240416b65fb3127bdd1f3820
ShadowBroker_0.9.8_x64_en-US.msi 122.4 MB
d4be4cb68c3e6409fff54c225acdcdd08e27d5d6d2b31616d78d2a4f6812991d
ShadowBroker_0.9.8_x64-setup.exe 76.5 MB
1115d1f5cf37edd03ea2c21d821c7626e1bf3319c990402aaa0293bca46fea67
Sizes match the v0.9.79 reference shape (5.76 MB / 117 MB / 72.9 MB)
within expected drift for new code. The .zip is a `git archive` of the
v0.9.8 source tree (matching v0.9.79's approach).
Audit confirms no .env, .key, .venv-dir, or cache files leaked into the
backend-runtime bundle. Python 3.11.9 + 199 site-packages + privacy_core
all staged correctly.
Headline changes since v0.9.79:
* Cumulative fuel/CO2 per flight (#317) — running totals since first
observation, not just per-hour rate.
* AIS maritime resilience (#314, #316) — outage banner + AISHub REST
fallback when AISStream WebSocket primary is offline.
* Data-layer repair (#311, #312) — UAP fallback respects the 60-day
cutoff; GPS jamming threshold tuning + nac_p=0 inclusion so the layer
actually fires.
* Per-flight source attribution (#313) — source field on every record.
* Cross-node DM mailbox replication (#309).
* Infonet sync HTTP 429 honored (#310).
Test fixtures updated:
* test_per_operator_outbound_attribution.py — added v0.9.8 UA strings
to the banned-aggregate-literals list (alongside v0.9.79).
* updateRuntime.test.ts — bumped asset filename fixtures to v0.9.8.
release_digests.json keeps the v0.9.79 block alongside v0.9.8 so
operators still on 0.9.79 validate cleanly during the rollout.
The accent narrowing fix in ChangelogModal (one feature uses 'purple',
two use 'cyan' so the renderer's `accent === 'purple'` comparison
still type-checks) is included.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
BigBodyCobain
|
5e0b2c037e |
feat(ais): surface upstream outage instead of failing silently
On 2026-05-23, stream.aisstream.io went fully offline (TCP timeouts on port
443). The backend kept respawning the node WebSocket proxy every few
seconds with nothing arriving. From the operator's POV the ships layer
silently went empty — no banner, no log surfacing, no way to tell whether
it was their config / network / viewport filter / upstream.
Backend:
* ais_proxy_status() now also returns:
- connected (bool): true when a vessel message arrived in last 60s
- last_msg_age_seconds (int | None)
- proxy_spawn_count (int): proxy respawns — sustained growth without
connected means upstream is dead
* /api/health escalates top status to "degraded" when AIS_API_KEY is set
but the proxy is currently disconnected. Existing degraded_tls signal
preserved.
Frontend:
* useAisUpstreamHealth hook polls /api/health every 30s, derives the
outage state. Defensively only reports outage once spawn_count > 0 so
operators who haven't opted in don't see the banner.
* AisUpstreamBanner component renders a dismissible amber notice
"Ship data temporarily unavailable — AISStream upstream is offline"
mounted on the main app shell.
7 backend tests pin the status-shape contract and the /api/health
escalation behavior in both with-key and without-key configurations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
Shadowbroker
|
b3fca3dc18 |
Merge pull request #309 from BigBodyCobain/feat/cross-node-dm-mailbox-replication
DM mailbox: per-(sender, recipient) anti-spam cap + replication primitives |
||
|
BigBodyCobain
|
401f114e4f |
DM mailbox: outbound replication + receiving endpoint
Second commit on this branch (first added the per-sender cap + accept_replica primitive). This commit wires the actual cross-node propagation: Outbound (sender side) ---------------------- * New ``DMRelay._replicate_envelope_to_peers_async()`` — fire-and-forget thread that POSTs the envelope to every authenticated relay peer via the same per-peer HMAC pattern gate-message replication uses (#256 ``X-Peer-Url`` + ``X-Peer-HMAC`` headers, ``resolve_peer_key_for_url``). * ``deposit()`` now calls the replication helper after a successful local accept. Per-peer errors are swallowed — slow Tor peers must not block the sender's UX, and the recipient polling from a healthy peer works fine even if some peers are down. * Metrics: dm_replication_push_ok / _rejected / _error. Inbound (receiving side) ------------------------ * New endpoint ``POST /api/mesh/dm/replicate-envelope`` in routers/mesh_peer_sync.py. * Same HMAC auth gate (``_verify_peer_push_hmac``) as the existing infonet/gate peer-push endpoints. Unauthenticated requests get 403. * Body cap of 64 KB (DM envelope is bounded by MESH_DM_MAX_MSG_BYTES). * Calls DMRelay.accept_replica which enforces the per-sender cap as a network rule — hostile sender's relay can hold extras locally but honest peers reject them on inbound replication. End-to-end flow now works ------------------------- 1. Alice's node accepts a deposit to Bob's mailbox (local cap check). 2. Alice's node spawns a background thread that POSTs the envelope to MESH_RELAY_PEERS with per-peer HMAC. 3. Each peer's /api/mesh/dm/replicate-envelope verifies the HMAC and calls accept_replica, which re-enforces the per-sender cap. 4. Bob (offline at the time of send) eventually logs into ANY node in MESH_RELAY_PEERS, his existing pollDmMailboxes pulls from the local mailbox there, finds Alice's envelope, decrypts. Tests ----- backend/tests/test_dm_replicate_envelope_endpoint.py — 4 tests: TestReplicateEndpointAuth: - rejects requests without peer HMAC (403) - rejects requests with WRONG peer HMAC (403) — confirms the HMAC is actually verified, not just present - rejects oversize bodies (>64 KB) with 400/413 TestReplicateEndpointRegistered: - static check that POST /api/mesh/dm/replicate-envelope is registered on app.routes — catches future refactor that drops the router include All 38 backend tests touching the new code paths still pass: test_dm_relay_per_sender_cap.py (14) test_dm_replicate_envelope_endpoint.py (4) test_no_new_duplicate_routes.py (1) — new route is unique test_per_peer_secret_resolver.py (19) — HMAC primitive unaffected What's still ahead (PR-3+) -------------------------- * ack propagation: when recipient pulls a message on node X, peers Y/Z should prune their copies to free the sender's quota network-wide. Without this, the sender's quota frees only on the node the recipient actually polled — other peers still see N pending until TTL expiry. Workable but suboptimal. PR-3 will add a /api/mesh/dm/ack endpoint with the same HMAC pattern. * recipient pull-from-peers: today the recipient's poll only hits their own node's relay. If they log into a peer they didn't deposit with, they need a way to fetch envelopes from other peers in MESH_RELAY_PEERS. Today this works as long as the recipient's current node is one of the peers Alice's node pushed to — which is true in a fully-meshed deployment but not guaranteed for partial meshes. PR-4 if telemetry shows this matters. |
||
|
Shadowbroker
|
ba39d3b9aa |
Merge pull request #307 from BigBodyCobain/fix/302-openclaw-hmac-reveal-hardening
Fix #302: split OpenClaw HMAC reveal into dedicated POST with no-store headers |
||
|
BigBodyCobain
|
f91ddcf38b |
Fix #302: split OpenClaw HMAC reveal into dedicated POST with no-store
Reported by @tg12. Pre-fix, two problems lived on the GET endpoint:
1. `GET /api/ai/connect-info?reveal=true` returned the full HMAC
secret in the response body on every Connect modal open. Even
gated to require_local_operator, that put the secret into
browser history, dev-tools network panels, browser disk caches,
HAR exports, and screen captures.
2. The same GET endpoint auto-bootstrapped (generated + persisted)
the secret on a mere read. Side effects on a GET are a footgun:
browser prefetchers, mirror tools, and casual curl-from-history
would all silently mint+persist a fresh secret.
Backend (backend/routers/ai_intel.py)
-------------------------------------
GET /api/ai/connect-info — always returns the MASKED
fingerprint (first6 + bullets
+ last4). No `?reveal` param.
NO auto-bootstrap. When the
secret is missing, returns
`hmac_secret_set: false` and
tells the caller to POST to
/bootstrap.
POST /api/ai/connect-info/bootstrap — NEW. Mints+persists the secret
if missing. Idempotent. Never
returns the full secret in the
response body.
POST /api/ai/connect-info/reveal — NEW. Returns the full secret
with Cache-Control: no-store,
no-cache, must-revalidate +
Pragma: no-cache + Expires: 0.
POST so the body never lands
in URL history. 404 (with a
pointer to /bootstrap) when
the secret isn't set.
POST /api/ai/connect-info/regenerate — keeps existing one-time-reveal
behavior (regen IS a deliberate
destructive action triggered
by the operator). Same
no-store/no-cache headers added
so even the regen response
doesn't get cached.
Frontend (AIIntelPanel.tsx, OnboardingModal.tsx)
------------------------------------------------
* On mount: GET (masked only). If hmac_secret_set: false, fire a
transparent POST /bootstrap and refresh the masked fingerprint.
Operator sees no behavior change from pre-#302.
* Reveal (eye icon): lazy POST /reveal — secret only travels when
the operator explicitly clicks the button.
* Copy: lazy POST /reveal too — copying without a prior reveal
works exactly like before, just routed through the new endpoint.
* Regenerate: POST returns the new secret (same as before, but the
response now has no-store headers).
* The displayed snippet uses the masked fingerprint until the
operator clicks Reveal or Copy.
Tests (backend/tests/test_openclaw_connect_info_reveal.py — 13 tests)
---------------------------------------------------------------------
* GET returns masked + the full secret never appears in r.text
* GET does NOT auto-bootstrap when missing
* GET silently ignores any ?reveal=true query (back-compat noise)
* POST /bootstrap mints when missing, idempotent when set
* POST /bootstrap never returns the full secret
* POST /reveal returns the full secret with Cache-Control: no-store,
no-cache + Pragma: no-cache + Expires: 0
* POST /reveal 404s with a pointer to /bootstrap when no secret
* POST /regenerate returns the new secret with the same headers
* Anonymous remote callers get 403 on ALL FOUR endpoints (parametric
regression against the same allowlist used elsewhere).
Adjacent suites still green: test_openclaw_route_security,
test_no_new_duplicate_routes, test_control_surface_auth. 67/67 pass
locally.
Credit: @tg12 for the audit report.
|
||
|
BigBodyCobain
|
32b8421a1c |
Merge origin/main into fix/298: resolve tools.py conflict
PR #303 landed on main and added Depends(require_local_operator) to the @router.post decorators for /api/sentinel/token and /api/sentinel/tile. PR #298 (this branch) edited the same decorator lines AND function bodies to add the env-credential fallback resolver. Resolution keeps BOTH: * The require_local_operator dependency from #303 (the auth gate) * The _resolve_sentinel_credentials helper from #298 * The env-fallback path inside the function bodies Both layers are independent — the gate blocks anonymous callers, the env fallback lets legitimate (gated) callers omit credentials from the body. Verified: 46 tests pass against the merged code, including both test_sentinel_credentials_server_side.py (#298 fallback) and test_sentinel_routes_auth_gate.py (#303 gate). |
||
|
BigBodyCobain
|
b041b5e97c |
Fix #298: move Sentinel credentials from browser storage to backend .env
Reported by @tg12. Pre-fix, the Settings panel stored real third-party
Copernicus CDSE client_id + client_secret in browser localStorage /
sessionStorage via the privacy storage helper, and the proxy routes
required those values to come back in every tile/token request body.
Any same-origin script (XSS, malicious browser extension, dev-tools
HAR export) had read access to the credentials.
This change moves them server-side, behind the same .env-backed admin
flow every other third-party API key (OpenSky, AIS Stream, Finnhub,
Shodan, …) already uses.
Backend
-------
backend/services/api_settings.py
* Added SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET entries to
API_REGISTRY. The existing GET/PUT /api/settings/api-keys flow
(already require_local_operator-gated, .env-backed) now manages
them — no new route surface.
backend/routers/tools.py
* /api/sentinel/token and /api/sentinel/tile resolve credentials via
a new _resolve_sentinel_credentials() helper: body fields win for
back-compat with any legacy callers, otherwise the helper reads
SENTINEL_CLIENT_ID / SENTINEL_CLIENT_SECRET from os.environ.
* When neither source has a value, the route returns 400 with a
friendly pointer ("Set SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET
in the API Keys panel") instead of the curt "required" message.
The user's standing rule against hostile errors applies.
* Function bodies only — decorator lines untouched, so this PR does
not conflict with #303 (which adds Depends(require_local_operator)
to the same routes).
Frontend
--------
frontend/src/lib/sentinelHub.ts — rewritten
* Removed: getSentinelCredentials / setSentinelCredentials /
clearSentinelCredentials / getSentinelCredentialStorageMode.
These were the browser-storage read/write helpers; their existence
was the bug.
* Added: checkBackendSentinelStatus(), refreshSentinelStatus(),
getCachedSentinelStatus(), and a kept-for-back-compat
hasSentinelCredentials() shim. Status is sourced from
/api/settings/api-keys (the same endpoint the API Keys panel
already uses), so we don't add a new route just for this read.
* Added: migrateLegacySentinelBrowserKeys() — one-shot, idempotent
helper that clears sb_sentinel_client_id / _secret / _instance_id
from BOTH localStorage and sessionStorage. We deliberately do NOT
auto-POST those legacy browser values to the backend; doing so
would silently migrate a secret across a trust boundary without
operator consent. Operators re-enter once in the API Keys panel
and the legacy keys get wiped here.
* fetchSentinelTile and getSentinelToken no longer send client_id /
client_secret in the request body. The backend uses .env.
frontend/src/components/SettingsPanel.tsx
* Dropped sb_sentinel_client_id / _secret / _instance_id from
PRIVACY_SENSITIVE_BROWSER_KEYS — they're no longer written.
* SentinelTab rewritten: removed the inline Client ID / Client Secret
inputs + Save / Clear / Test buttons. Replaced with a status panel
that calls checkBackendSentinelStatus() on mount, a one-click
"Open API Keys Panel" button, and a migration banner that appears
only when migrateLegacySentinelBrowserKeys() actually cleared
something.
* Setup guide STEP 3 now points to the API Keys panel instead of
the local form.
frontend/src/app/page.tsx
* Added a one-time useEffect that fires checkBackendSentinelStatus()
on mount so the cached value (which the synchronous
hasSentinelCredentials() shim reads) is populated before
MaplibreViewer's tile-URL memo runs.
Tests
-----
backend/tests/test_sentinel_credentials_server_side.py (new)
* API_REGISTRY surface — sentinel_client_id / sentinel_client_secret
are registered with the right env_keys, ALLOWED_ENV_KEYS lets
/api/settings/api-keys PUT them.
* Resolution order — body wins, env is fallback, neither → 400 with
the friendly pointer message, and NO upstream HTTP call when
neither source has credentials (asserted via
MagicMock(side_effect=AssertionError)).
* /api/sentinel/tile same shape.
frontend/src/__tests__/utils/sentinelHub.test.ts (new)
* migrateLegacySentinelBrowserKeys clears localStorage AND
sessionStorage, reports what it cleared, idempotent.
* fetchSentinelTile + getSentinelToken POST WITHOUT client_id /
client_secret in the body (plants leaked credentials in browser
storage first to prove they are NOT picked up).
* checkBackendSentinelStatus parses /api/settings/api-keys correctly:
true only when both keys is_set, false on partial config or
network errors.
All 7 backend tests + 8 frontend tests pass locally. The
test_no_new_duplicate_routes guard and the api-settings test suite
still pass.
Credit: @tg12 for the audit report.
|
||
|
BigBodyCobain
|
c54ea7fd9f |
Fix #299/#300/#301: gate Sentinel proxy routes with require_local_operator
Reported by @tg12 in three audit issues opened the same day: #299 — POST /api/sentinel/token is an unauthenticated Copernicus OAuth relay for caller-supplied client_id/secret. #300 — POST /api/sentinel/tile is an unauthenticated quota/bandwidth relay for Sentinel Hub Process API tile fetches. #301 — GET /api/sentinel2/search is an unauthenticated Planetary Computer STAC + Esri imagery search relay. All three lived in backend/routers/tools.py decorated only with @limiter.limit(...) — no Depends(require_local_operator). That made the backend a free anonymous relay for any caller's Sentinel / Planetary Computer queries, in the same shape we already closed for #240/#241 (oracle resolve) and #211/#213/#214 (thermal verify, OpenMHZ calls + audio relay). Fix: add dependencies=[Depends(require_local_operator)] to each route. Loopback / Docker-bridge / admin-key callers (the operator dashboard) are unaffected — they still resolve through the same allowlist used by every other operator-only helper in this file. Anonymous remote callers now receive 403 BEFORE any outbound HTTP call to Copernicus or Planetary Computer happens. Tests ----- test_sentinel_routes_auth_gate.py — 8 new tests: * anonymous-remote → 403 on all three routes * NO upstream HTTP call when the gate fires (asserted via MagicMock(side_effect=AssertionError) on requests.post and services.sentinel_search.search_sentinel2_scene). This is the property that makes the gate real — without it, a 403 returned after the upstream call still burns quota. * 127.0.0.1 loopback caller reaches the handler (no false-positive where the gate accidentally blocks the local operator too). * Uses raw ASGITransport(client=(peer_ip, ...)) rather than FastAPI's TestClient because TestClient reports client.host as "testclient" which is not on the loopback allowlist. test_control_surface_auth.py — extended the existing parameterised regression with the three new routes. That regression is the global "no remote control surface ships without auth" guard for the whole codebase; adding these to it means a future refactor that drops the dependency from any of them will fail CI alongside the existing ~30 gated routes. The egress-on-403 property and the parameterised regression together give two independent proofs that the gate fires before the upstream network call, even if FastAPI's internal dependant tree shape changes across versions (an earlier draft of this PR included a static walker of the route table; it was removed because behavioural evidence is strictly stronger and version-independent). |
||
|
Shadowbroker
|
19fb7f0b1e |
Fix #288: viewport-scoped live-data for heavy layers only (#294)
Reported by @tg12 in the external security/correctness audit.
Before this change, /api/live-data/{fast,slow} accepted s/w/n/e query
params but their Query() descriptions explicitly said "(ignored)". The
endpoints shipped the full in-memory world dataset on every poll:
/api/live-data/fast → 16.88 MB
/api/live-data/slow → 10.12 MB
── 27 MB per poll cycle, regardless of zoom
For a node with N operators each polling at the steady 15s/120s cadence,
this is hundreds of MB/minute of outbound traffic that never gets used —
the GPU just culls everything outside the viewport client-side. On a
Tor-bridged or LTE-backed node, that bandwidth bill is the actual cost.
This change makes the existing s/w/n/e params honored — when all four
bounds are supplied, the backend bbox-filters a curated set of heavy,
density-driven, time-sensitive collections to that viewport (with the
existing 20% padding from _bbox_filter):
/fast: commercial_flights, military_flights, private_flights,
private_jets, tracked_flights, ships, cctv, uavs, liveuamap,
gps_jamming, sigint, trains
/slow: gdelt, firms_fires, kiwisdr, scanners, psk_reporter
Static reference layers (satellites, datacenters, military_bases,
power_plants, satnogs, weather, news, stocks, etc.) deliberately STAY
world-scale so panning never reveals an "empty world" of infrastructure.
That preserves the no-hostile-UX feel of the existing dashboard.
Behavior contract:
* Without bbox params (or with a partial bbox), the response is
byte-for-byte identical to the pre-#288 implementation. No
behavior change for any existing caller that hasn't opted in.
* World-scale bbox (lng_span >= 300 or lat_span >= 120) short-circuits
filtering and shares the global ETag — zoomed-out operators all
hit the same 304 cache exactly like before.
* ETag now mixes a 1°-quantized bbox suffix when filtering engages,
so two viewports never poison each other's 304 cache. Sub-degree
pans land in the same ETag bucket (i.e. don't bust the cache on
every mouse drag).
Polling cadence, rate-limit windows, and the 304 short-circuit are all
unchanged. Only the SIZE of the responses changes, and only when the
caller opts in via bounds.
Frontend wiring: useViewportBounds reuses the same coarsened/
expanded bounds it already computes for the AIS /api/viewport POST and
pushes them into a new module-level liveDataViewport store.
useDataPolling reads from that store via appendLiveDataBoundsParams
when building each live-data URL.
Tests cover: no-bbox → world data; bbox → heavy layers filtered;
bbox → reference layers untouched; world-scale bbox → no filter;
partial bbox → treated as no bbox; ETag changes with bbox; sub-degree
pan → same ETag; 304 path works; antimeridian-crossing bbox handled.
Co-authored-by: BigBodyCobain <moatbc@gmail.com>
|
||
|
Shadowbroker
|
76750caa92 |
Round 7a: per-operator outbound attribution + GDELT GCS-direct fix (#292)
== Per-install operator handle for every third-party API call ==
Before this PR, every Shadowbroker install identified itself to
Wikipedia, Wikidata, Nominatim, GDELT, OpenMHz, Broadcastify,
weather.gov, NUFORC, Sentinel/Planetary Computer, TinyGS / CelesTrak,
Shodan, Finnhub, and others with a single project-wide User-Agent
("Shadowbroker/1.0" or "ShadowBroker-OSINT/1.0"). From the upstream's
perspective every install in the world looked like one giant scraper.
If one install misbehaved, the upstream's only recourse was to block
"Shadowbroker" as a whole.
PR #284 inadvertently doubled down on this in the frontend by
introducing a shared `WIKIMEDIA_API_USER_AGENT` constant. This PR
retrofits both backends to per-operator attribution.
New setting: OPERATOR_HANDLE (env var / settings UI / auto-gen)
New helper: network_utils.outbound_user_agent("purpose")
The handle is auto-generated as "operator-XXXXXX" on first call (the
"shadow-" prefix from earlier drafts was deliberately dropped — too
suspicious-looking for abuse-detection systems). Operators can
override via OPERATOR_HANDLE; the value is sanitized to lowercase
alphanumeric+dash+underscore and capped at 48 chars. Persisted to
backend/data/operator_handle.json so it survives container restarts.
Retrofitted call sites (every previously-MONSTER User-Agent):
- services/region_dossier.py (Wikipedia + Wikidata + Nominatim)
- services/geocode.py (Nominatim)
- services/sentinel_search.py (Microsoft Planetary Computer)
- services/feed_ingester.py (operator-curated RSS feeds)
- services/fetchers/earth_observation.py (weather.gov, NUFORC)
- services/fetchers/infrastructure.py
- services/fetchers/aircraft_database.py
- services/fetchers/route_database.py
- services/fetchers/trains.py
- services/fetchers/meshtastic_map.py
- services/shodan_connector.py
- services/unusual_whales_connector.py (Finnhub)
- services/tinygs_fetcher.py (CelesTrak + TinyGS)
- services/sar/sar_products_client.py
- services/geopolitics.py (GDELT)
- services/radio_intercept.py (Broadcastify + OpenMHz)
- routers/cctv.py + main.py (CCTV proxy)
- routers/ai_intel.py
- scripts/convert_power_plants.py (release-time data refresh)
Spoofed browser UAs removed (issues #289 / #290 / #291 — tg12 audit):
- cloudscraper-based Chrome impersonation against api.openmhz.com
-> replaced with honest requests + per-install UA
- Mozilla/5.0 spoofed UA on Broadcastify scrape
-> replaced with honest UA
- Mozilla/5.0 + fake first-party Referer on OpenMHz audio relay
-> replaced with honest UA
- cloudscraper dependency dropped from pyproject.toml + uv.lock
Frontend retrofit:
- new GET /api/settings/operator-handle endpoint (local-operator
gated) returns the install's handle
- frontend/src/lib/wikimediaClient.ts fetches the handle once on
first use, caches it for page lifetime, embeds it in the
Api-User-Agent for every Wikipedia / Wikidata browser-direct call
== GDELT GCS-direct fix ==
GDELT's data.gdeltproject.org is a CNAME to a Google Cloud Storage
bucket. GCS responds with the wildcard *.storage.googleapis.com cert
which legitimately does NOT cover the GDELT custom domain, so Python's
TLS verification correctly refuses the connection. Some networks
happen to route through a path where this works; many (notably Docker
Desktop's outbound NAT on local installs) do not. Verified on the
maintainer's local install: GDELT was unreachable; 1610 geopolitical
events / 48 export files were dropping silently.
Fix: services/geopolitics._gcs_direct_gdelt_url() rewrites any
data.gdeltproject.org URL to its GCS-direct equivalent
(storage.googleapis.com/data.gdeltproject.org/...) where the standard
GCS cert is genuinely valid. api.gdeltproject.org and every other host
are left untouched.
Confirmed live: backend log goes from
GDELT lastupdate failed: 500
to
Downloading 48 GDELT export files...
Downloaded 48/48 GDELT exports
GDELT parsed: 1610 conflict locations from 48 files
== Tests ==
backend/tests/test_per_operator_outbound_attribution.py (12 tests)
backend/tests/test_gdelt_gcs_direct_rewrite.py (6 tests)
backend/tests/test_region_dossier_wikimedia_ua.py (updated to
pin the helper + per-operator handle, not the old constant)
frontend/src/__tests__/utils/wikimediaClient.test.ts (rewritten
to mock /api/settings/operator-handle and assert per-operator UA)
Local: backend 114/114 security+audit+round7a suite green;
frontend 718/718 vitest suite green.
Credit: tg12 (external security audit, issues #289/#290/#291
relating to spoofed UAs); BigBodyCobain (operator-prefix call,
GDELT cloud-vs-local diagnosis).
|
||
|
Shadowbroker
|
e125467721 |
Fix #243/#252/#253: stop leaking settings posture to anonymous callers (#283)
Three settings endpoints were disclosing operational posture or operator-curated configuration to any network caller. This change either tightens the redacted-public view (#243) or adds a local-operator auth gate (#252, #253) per the audit recommendations. Zero hostility to legitimate users: in all three cases, the Tauri shell (loopback), the Docker bridge frontend container (#250 + #278), and any caller with an admin key continue to see the full data. Only anonymous LAN/internet callers see the reduced surface. == #243 (Wormhole transport posture, anonymous-mode, profile, node mode) Tightened the public-redaction allowlists in BOTH the main.py and routers/wormhole.py copies: - _WORMHOLE_PUBLIC_SETTINGS_FIELDS: {enabled, transport, anonymous_mode} -> {enabled} - _WORMHOLE_PUBLIC_PROFILE_FIELDS: {profile, wormhole_enabled} -> {wormhole_enabled} `GET /api/settings/node` (both the routers/admin.py and main.py copies) now returns an empty stub for unauthenticated callers and the full node_mode + node_enabled fields only for authenticated callers via _scoped_view_authenticated(request, "node"). == #252 (news feed inventory disclosure) `GET /api/settings/news-feeds` now requires Depends(require_local_operator) in both the canonical routers/admin.py handler and the duplicate main.py handler. Anonymous callers can no longer enumerate operator-curated feed names and URLs. == #253 (Time Machine archival-capture posture disclosure) `GET /api/settings/timemachine` now requires Depends(require_local_operator). Anonymous callers can no longer fingerprint whether a deployment is retaining replayable historical surveillance data. Tests: backend/tests/test_round5_settings_info_disclosure.py (10 tests) - Wormhole settings: anonymous sees only `enabled`; authenticated sees full state. - Privacy profile: anonymous sees only `wormhole_enabled`; authenticated sees `profile` + `transport` + `anonymous_mode`. - Node settings: anonymous sees `{}`; authenticated sees node_mode + node_enabled + persisted state. - news-feeds: anonymous gets 403 (and get_feeds() is NOT called); authenticated gets full inventory. - timemachine: anonymous gets 403; authenticated sees enabled + storage_warning. Local: 73/73 security suite (round 5 + earlier rounds) green. Credit: tg12 (external security audit, P1 + 2x Medium). |
||
|
Shadowbroker
|
084e563412 |
Fix #240/#241: require admin auth on oracle resolve endpoints (#280)
Both POST /api/mesh/oracle/resolve and POST /api/mesh/oracle/resolve-stakes were previously gated only by a rate limit (5/min) and tagged with `mesh_write_exempt(MeshWriteExemption.ADMIN_CONTROL)`. The exemption decorator is metadata only — it tells the mesh signed-write middleware not to require a signature envelope, it does NOT enforce caller authorization. Any network caller could: - /resolve: settle any prediction market to any outcome (corrupts every downstream profile/win-loss count derived from that ledger). - /resolve-stakes: trigger stake settlement for all expired contests at a time of their choosing (race against operator intent). Fix: add `dependencies=[Depends(require_admin)]` to both routes. The existing `mesh_write_exempt` tag stays in place because it accurately describes the route's relationship to the signed-write envelope system; adding `require_admin` is what closes the actual auth hole. Tests in backend/tests/test_oracle_resolve_auth_gate.py: - anonymous caller -> 403, ledger mutator NOT called - wrong admin key -> 403, ledger mutator NOT called - valid admin key -> 200, ledger mutator called - admin key unconfigured + no debug/insecure-admin -> 403 Credit: tg12 (external security audit) |
||
|
Shadowbroker
|
729ea78cb2 |
Fix #258: AIS proxy SPKI pinning fallback for expired upstream cert (#262)
External report from @jmleclercq: AISStream's Let's Encrypt cert
expired on 2026-05-20 (verified — their renewal pipeline failed), so
the AIS WebSocket connection dies with CERT_HAS_EXPIRED and the
maritime layer empties out. The reporter worked around it locally by
passing { rejectUnauthorized: false } to the WebSocket constructor and
asked whether we should add an env var for that.
That fix is the wrong fix. Disabling TLS validation entirely lets any
network attacker MITM the WebSocket and inject fake ship positions —
same class as the GDELT plaintext-HTTP MITM we just closed in #199.
Adding an env var for it would be an attractive nuisance: operators
set it once during a bad cert week and then forget, leaving themselves
open to MITM forever.
Right fix: SPKI pinning, same pattern as the Tor bundle digest pinning
in #201. The insight is that Let's Encrypt renewals keep the SAME
public key by default, so the SPKI hash survives normal cert rotation.
We can relax the date check while keeping the identity check.
Mechanics:
backend/data/aisstream_spki_pins.json (new)
Pinned SHA-256 hashes of the DER-encoded SPKI bytes for
stream.aisstream.io. Captured 2026-05-20 from the live cert.
Format is base64(sha256(pubkey_der)), matching the canonical
openssl pipeline. Whitelisted in .gitignore alongside the other
static reference data files (KiwiSDR directory, Tor bundle
digests).
backend/ais_proxy.js
Path A (99.9% of the time): normal TLS validation. Untouched.
Path B (on CERT_HAS_EXPIRED only): re-handshake with
rejectUnauthorized=false JUST to read the leaf cert, compute its
SPKI hash, compare against the pinned list. If match → upstream
is still the genuine AISStream → re-open the WebSocket with
rejectUnauthorized=false and log DEGRADED MODE. If no match →
refuse the connection, log loudly: this would be a real MITM.
Pin file is looked up in three locations so the same code works
in the Docker backend, the Tauri desktop runtime, and any
operator-relocated layout (SHADOWBROKER_AIS_PINS env var).
Embedded fallback list inside the JS so portable installs that
haven't shipped the JSON still work.
backend/services/ais_stream.py
Captures the proxy's status markers from stdout
({"__ais_proxy_status": {"degraded_tls": true}}) into a module-
level snapshot. Exposes ais_proxy_status() for the health
endpoint. Doesn't touch the data plane — degraded mode keeps
receiving vessel data, just with weaker MITM protection.
backend/routers/health.py + backend/services/schemas.py
/api/health now includes an ais_proxy block with degraded_tls.
Top-level status escalates ok -> degraded when AIS is in
degraded TLS mode (but won't downgrade a worse SLO status).
Operators get a visible signal that they're in degraded mode
without needing to grep logs.
Tests: backend/tests/test_ais_spki_pinning.py (7 tests)
- Pin file structure validation (JSON, host entry, base64 SHA-256)
- ais_proxy_status() snapshot semantics (starts empty, defensive copy)
- /api/health surfaces ais_proxy.degraded_tls when set
- /api/health returns empty ais_proxy when proxy hasn't reported
Node.js syntax check passes (node --check) on both backend/ais_proxy.js
and the Tauri runtime mirror.
When AISStream renews their cert (likely within hours-to-days), the
normal-TLS path succeeds on next reconnect and degraded_tls clears
automatically. No operator action needed. If they instead rotate their
server key, the SPKI check will fail and we'll need to add the new
hash to backend/data/aisstream_spki_pins.json before removing the old
one.
Credit: @jmleclercq for the clear report and the careful workaround
verification (Node version, ws version, manual probe).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
Shadowbroker
|
e36d1fc79c |
[security] Close tg12 audit issues #201–#214 seamlessly (#261)
External security audit by @tg12 (May 17, 2026) filed issues #201–#214 in addition to the #189–#200 batch already closed by PRs #227/#232/#260. This PR closes all eight that are real security bugs (the other six in the 201–214 range are either design discussions or upstream-abuse/TOS concerns we're keeping intentional, see issue triage notes on each). The user-facing principle for this PR: fix the security gap WITHOUT introducing a single hostile error or behavior change for legitimate users. Every fix follows the same template — fail forward, not loud. When the secure path is harder than the insecure one, build a fallback chain that ends in graceful degradation, not in a scary modal or 422 response. #205 — OpenMHZ audio redirect SSRF (services/radio_intercept.py) Replaced requests.get(..., allow_redirects=True) with a manual redirect loop that re-validates each hop's host against _OPENMHZ_AUDIO_HOSTS. Same-host redirects (CDN edge selection) still work, so legitimate audio playback is unaffected. Cross-host redirects to disallowed hosts return a generic 502 which the browser audio element handles gracefully. Cap at 5 hops. #207 — infonet/status verify_signatures DoS (routers/mesh_public.py) Silently downgrade verify_signatures=true to False for unauthenticated callers. No error surfaced — the response shape is identical, just without the O(n_events) signature verification. Authenticated callers (scoped mesh.audit) still get the full path. The frontend never passes this param so legitimate UI is unaffected. #211 — thermal/verify expensive analysis (routers/sigint.py) Added Depends(require_local_operator). Frontend has no direct callers (verified by grep); Tauri/AI agents use scoped tokens that pass the auth check. Anonymous abusers blocked silently — the legitimate UI keeps working through the Next.js admin-key proxy. #213, #214 — OpenMHZ calls/audio upstream abuse (routers/radio.py) Added Depends(require_local_operator) to both. Browser users hit these through the Next.js proxy at src/app/api/[...path]/route.ts which injects X-Admin-Key, so the auth check passes transparently. Direct attackers can no longer rotate sys_names to hammer api.openmhz.com or relay arbitrary audio streams through the backend's bandwidth. #202 — overflights unbounded hours (routers/data.py) Silently clamp `hours` to OVERFLIGHTS_MAX_HOURS (default 72, configurable). NO 422 — clients asking for an absurd window get a shorter window back with `requested_hours` and `effective_hours` hint fields. Postel's law: liberal in what we accept, conservative in what we compute. #203 — Meshtastic callsign UA leak (services/fetchers/meshtastic_map.py) Added MESHTASTIC_SEND_CALLSIGN_HEADER opt-out env var. Default is TRUE — preserves existing operator behavior (callsign sent so meshtastic.org can rate-limit per-install). Privacy-conscious operators set it to false to suppress. #206 — KiwiSDR upstream is HTTP-only (services/kiwisdr_fetcher.py) Upstream rx.linkfanel.net doesn't speak HTTPS (verified — Apache 2.4.10 only on port 80). We can't fix the transport. Instead added three layers: 1. Content validation on fetched data — reject responses with <50 receivers or >5% malformed entries (likely MITM injection). 2. Existing disk cache fallback (already present). 3. NEW: bundled static directory at backend/data/kiwisdr_directory.json shipping 798 known-good receivers. Used as last resort so the KiwiSDR map layer always renders something useful. #208 — Merkle proof DoS via /api/mesh/infonet/sync (services/mesh/mesh_hashchain.py) The endpoint is part of the cross-node federation protocol — peers legitimately call it without local-operator auth, so we can't add Depends(). Instead made the underlying operation O(1) per proof via a cached Merkle level structure on the Infonet instance: - _merkle_levels_cache + _merkle_levels_for_event_count on each Infonet instance - _invalidate_merkle_cache() called from every chain mutation point (append, ingest_events, apply_fork, cleanup_expired) - _get_merkle_levels() does the lazy recompute on first read after invalidation, then serves from cache thereafter Effect: anonymous attackers hammering the proofs endpoint hit a cached structure; the rebuild happens at most once per real chain advance. Federation untouched. #201 — Tor bundle SHA-256 bypass (services/tor_hidden_service.py) Docker users were already covered — backend/Dockerfile installs Tor via apt-get at build time (signed by Debian's package system). No runtime download needed for the 80%-of-users case. For Tauri desktop, replaced the single .sha256sum check with a multi-source verification chain implemented in _verify_tor_bundle(): 1. Try upstream .sha256sum (current behavior — fast path) 2. Try baked-in digest list at backend/data/tor_bundle_digests.json (pinned per-version, maintainer-updated) 3. If neither source is REACHABLE: HTTPS-only fallback with a loud warning (avoids breaking first-run onboarding while the maintainer hasn't yet pinned a new Tor release) A mismatch from a source that DID respond is always fatal — only the "no source reachable" case falls back to HTTPS-only. This is the "have cake and eat it" pattern: real users see no new failure modes during torproject.org outages, but MITM/compromise attacks still fail because the downloaded digest can't match what BOTH the upstream and the baked-in list report. Currently the digest file ships with placeholder values for the current Tor URLs (those URLs are already stale on torproject.org too). A follow-up commit can populate real digests when a stable Tor release is selected; until then the HTTPS-only warning fires and onboarding still works. Tests (82 total, all passing): test_openmhz_redirect_ssrf.py (5 tests) — #205 test_infonet_status_verify_gate.py (2 tests) — #207 test_overflights_clamp.py (5 tests) — #202 test_meshtastic_callsign_optout.py (3 tests) — #203 test_kiwisdr_fallback.py (6 tests) — #206 test_merkle_cache.py (6 tests) — #208 test_tor_bundle_verification.py (6 tests) — #201 test_control_surface_auth.py (extended) — #211, #213, #214 + all previous security tests (CCTV redirect, GDELT https, sentinel cache, crowdthreat opt-in, third-party fetcher gates, control surface auth) continue to pass. Pre-existing test infrastructure issue with SHARED_EXECUTOR teardown in the broader sweep exists on main too (verified) — not introduced by this PR. Credit: @tg12 reported every one of these with accurate line citations and the recommended fixes that informed this implementation. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Shadowbroker
|
d00c63abed |
[security] Close tg12 audit gaps #192, #198, #199, #200 (#260)
External security audit by @tg12 (May 17, 2026) filed 11 issues against the backend. PR #227 (May 18, AI-generated) closed seven of them by adding require_local_operator to control-plane endpoints. Four remained live; this PR closes the rest. #192 — CCTV proxy followed redirects without re-validating host Issue: /api/cctv/media validated only the caller-supplied URL host before passing it to requests.get(..., allow_redirects=True). A 302 to http://127.0.0.1 or any internal/disallowed host was silently followed, turning the proxy into an open-redirect-to-SSRF chain. Fix in routers/cctv.py: replace the single allow_redirects=True call with a manual follow loop. Each hop's Location is parsed, the host is rerun through _cctv_host_allowed(), and non-HTTP schemes (file://, ftp://, etc.) are rejected. Cap chain length at 5 hops. Test: backend/tests/test_cctv_redirect_ssrf.py covers - redirect to disallowed host -> 502 - redirect to localhost -> 502 - redirect to another allowed host -> 200 - redirect chain length cap - non-HTTP scheme rejected #198 — Gate introspection GETs were unauthenticated Issue: /api/wormhole/gate/{gate_id}/{identity,personas,key} were callable with no auth dependency. Any caller that could reach the backend could dump the operator's active persona, persona inventory, and key status for any gate_id they knew. The wiki's privacy threat model explicitly markets gate personas as rotating, unlinkable pseudonyms — this leak defeated that property. Fix in routers/wormhole.py: add dependencies=[Depends(require_local_operator)] to all three routes. Test: backend/tests/test_control_surface_auth.py extended with three new parameterized cases (lines 75-77). #199 — GDELT military incident ingestion used plaintext HTTP Issue: backend/services/geopolitics.py fetched http://data.gdeltproject.org/gdeltv2/lastupdate.txt and ~48 export archive URLs over plaintext HTTP. Passive observers could identify Shadowbroker nodes from the fetch pattern. Active MITM could inject doctored military incident records into the global map. Fix in services/geopolitics.py: rewrite the lastupdate.txt fetch and the export download URL constructor to use https://. GDELT's data.gdeltproject.org serves the same content over HTTPS. Test: backend/tests/test_gdelt_https.py asserts no plaintext HTTP URLs to data.gdeltproject.org remain in code (comments excluded) and that the HTTPS URLs we expect are present. #200 — Sentinel token cache lookup used client_id only Issue: routers/tools.py kept a process-global cache of Copernicus bearer tokens. The lookup compared _sh_token_cache["client_id"] == client_id. A caller who knew a valid client_id but supplied any wrong client_secret hit the cache and reused the legitimate caller's bearer token — burning their quota and accessing imagery on their account. Fix in routers/tools.py: replace the client_id field with credential_fp, an HMAC-SHA256 over (client_id, client_secret) under a per-process random key (_SH_TOKEN_CACHE_HMAC_KEY = os.urandom(32), regenerated at startup). A caller who doesn't know the secret cannot compute a matching fingerprint, so they miss the cache and hit the real Copernicus token endpoint — which will reject their wrong secret with a 401. Test: backend/tests/test_sentinel_token_cache.py covers - same client_id + different secrets => different fingerprints - same credentials => same fingerprint (cache still works) - different client_ids + same secret => different fingerprints - cache no longer stores raw client_id (catches regression) - attacker with wrong secret cannot reuse victim's token Validation pytest backend/tests/test_control_surface_auth.py backend/tests/test_cctv_redirect_ssrf.py backend/tests/test_gdelt_https.py backend/tests/test_sentinel_token_cache.py -> 37 passed Credit: @tg12 reported all four of these in their May 17 audit with correct line-number citations and accurate remediation recommendations. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
BigBodyCobain
|
71a9d9e144 |
[security] Close post-#227 control-surface and fetcher gaps
PR #227 hardened most Wormhole/Infonet control surfaces behind require_local_operator and made the CrowdThreat fetcher opt-in. An audit of the codebase against that PR's stated goals turned up four classes of gap that the original change missed: 1. Two operator-only endpoints were left unprotected: - POST /api/wormhole/join: calls bootstrap_wormhole_identity() and flips the node into Tor mode, exactly the surface #227 hardened on /api/wormhole/identity/bootstrap. - POST /api/sigint/transmit: relays APRS-IS packets over radio using operator-supplied credentials. Anything that reached the API could transmit on the operator's authority. Both now require_local_operator. test_control_surface_auth.py extended with regression coverage for both. 2. Five third-party fetchers were still default-on, phoning home to politically/commercially sensitive upstreams on every poll cycle: - fimi.py -> euvsdisinfo.eu -> FIMI_ENABLED - prediction_markets -> Polymarket + Kalshi -> PREDICTION_MARKETS_ENABLED - financial.py -> Finnhub / yfinance -> FINANCIAL_ENABLED or FINNHUB_API_KEY - nuforc_enrichment -> huggingface.co -> NUFORC_ENABLED - news.py -> configured RSS feeds -> NEWS_ENABLED (default on, kill switch) Same CrowdThreat-style pattern: explicit env-var opt-in, empty the data slot and mark_fresh when disabled. New regression test file test_third_party_fetchers_opt_in.py asserts each fetcher's network entry point is not called when its gate is off. 3. The outbound User-Agent leaked both the operator's personal email and a fork-specific GitHub URL on every fetcher request. Consolidated to a single DEFAULT_USER_AGENT in network_utils.py, project-generic by default (no contact info), overridable via SHADOWBROKER_USER_AGENT for operators who want to identify themselves (e.g. for Nominatim or weather.gov usage-policy compliance). Six call sites updated; the Nominatim-specific override is preserved. 4. The same generic UA now also flows through the peer prekey lookup in mesh_wormhole_prekey.py, so DM first-contact requests no longer identify the caller as a Shadowbroker fork to the peer being queried. .env.example updated to document all new opt-in env vars. Tests: backend/tests/test_control_surface_auth.py (extended), backend/tests/test_crowdthreat_opt_in.py (unchanged, still passes), backend/tests/test_third_party_fetchers_opt_in.py (new, 7 tests). All 31 tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
BigBodyCobain
|
11ea345518 | Harden infonet control surfaces | ||
|
BigBodyCobain
|
25a98a9869 |
Harden Infonet DM address flow and seed sync
Allow local-operator DM invite import without requiring a full admin session. Prioritize bundled/bootstrap seed peers and shorten stale seed cooldowns for faster Infonet recovery. Replace raw DM invite dumps with copyable signed-address controls, contact request handling, and safer sealed-send behavior while the private delivery route connects. |
||
|
BigBodyCobain
|
b86a258535 |
Release v0.9.79 runtime and messaging update
Ship the v0.9.79 runtime refresh with transport lane isolation, Infonet secure-message address management, MeshChat MQTT controls, selected asset trail behavior, telemetry panel refinements, onboarding updates, and desktop/package metadata alignment. Also ignore local graphify work products so analysis folders do not leak into future commits. |
||
|
BigBodyCobain
|
5ee4f8ecd7 | Stabilize Infonet private sync and selected telemetry | ||
|
BigBodyCobain
|
b8ac0fb9e7 |
Harden v0.9.75 wormhole node sync and telemetry panels
Add Tor/onion runtime wiring and faster Infonet node status refresh. Keep node bootstrap state clearer across Docker and local runtimes. Use selected aircraft trail history for cumulative tracked-aircraft emissions. |
||
|
BigBodyCobain
|
6ffd54931c |
Release v0.9.75 runtime and onboarding update
Ship the 0.9.75 source update with improved startup/runtime hardening, operator API key onboarding, Meshtastic MQTT controls, Infonet/MeshChat separation, desktop package versioning, and aircraft telemetry refinements. Also updates focused backend/frontend tests for node settings, Meshtastic MQTT settings, and desktop runtime behavior. |
||
|
BigBodyCobain
|
707ca29220 |
Add in-app local API key setup
Let fresh Docker and local installs enter OpenSky, AIS, and other provider keys directly in onboarding or Settings without manually creating .env files. Persist keys server-side in the backend data store, keep them write-only from the browser, reload runtime settings, and retain local-operator access controls. |
||
|
BigBodyCobain
|
e1060193d0 |
Improve v0.9.7 startup and runtime reliability
Prioritize cached first-paint data, defer heavyweight feed synthesis, make MeshChat activation explicit, improve CCTV media handling, and tighten desktop runtime packaging filters. |
||
|
BigBodyCobain
|
08810f2537 | fix: stabilize v0.9.7 startup and feeds | ||
|
BigBodyCobain
|
4ec1fce53d | ci: unblock v0.9.7 release checks | ||
|
BigBodyCobain
|
28b3bd5ebf | release: prepare v0.9.7 |