Add Telegram OSINT with hourly incremental t.me scraping, metro geocoding
separate from news centroids, threat-intercept popup UI with inline media,
and HTML markers above alert boxes so pins stay clickable. Expose GFW_API_TOKEN
in onboarding and Settings Maritime; harden GFW/CCTV/geo fetchers. Port Osiris-
derived recon, SCM, entity graph, malware/cyber feeds, sanctions, and submarine
cable layers with tests and documentation.
Co-authored-by: Cursor <cursoragent@cursor.com>
Operators enable Polymarket/Kalshi correlation from Global Threat Intercept with a consent dialog; polls use a jittered schedule separate from the slow tier. Right-click Sentinel imagery returns up to three signed scenes again.
Co-authored-by: Cursor <cursoragent@cursor.com>
- User-Agent is per-install handle only (no Shadowbroker product token)
- LiveUAMap: Windows UI consent when enabling Global Incidents; env override
- Meshtastic callsign upstream header off by default (opt-in true)
- Expanded docs/OUTBOUND_DATA.md and README link for CCTV, basemap, Broadcastify
Co-authored-by: Cursor <cursoragent@cursor.com>
On 2026-05-23, stream.aisstream.io went fully offline (TCP timeouts on port
443). The backend kept respawning the node WebSocket proxy every few
seconds with nothing arriving. From the operator's POV the ships layer
silently went empty — no banner, no log surfacing, no way to tell whether
it was their config / network / viewport filter / upstream.
Backend:
* ais_proxy_status() now also returns:
- connected (bool): true when a vessel message arrived in last 60s
- last_msg_age_seconds (int | None)
- proxy_spawn_count (int): proxy respawns — sustained growth without
connected means upstream is dead
* /api/health escalates top status to "degraded" when AIS_API_KEY is set
but the proxy is currently disconnected. Existing degraded_tls signal
preserved.
Frontend:
* useAisUpstreamHealth hook polls /api/health every 30s, derives the
outage state. Defensively only reports outage once spawn_count > 0 so
operators who haven't opted in don't see the banner.
* AisUpstreamBanner component renders a dismissible amber notice
"Ship data temporarily unavailable — AISStream upstream is offline"
mounted on the main app shell.
7 backend tests pin the status-shape contract and the /api/health
escalation behavior in both with-key and without-key configurations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reported by @tg12 in the external security/correctness audit.
Before this change, /api/live-data/{fast,slow} accepted s/w/n/e query
params but their Query() descriptions explicitly said "(ignored)". The
endpoints shipped the full in-memory world dataset on every poll:
/api/live-data/fast → 16.88 MB
/api/live-data/slow → 10.12 MB
── 27 MB per poll cycle, regardless of zoom
For a node with N operators each polling at the steady 15s/120s cadence,
this is hundreds of MB/minute of outbound traffic that never gets used —
the GPU just culls everything outside the viewport client-side. On a
Tor-bridged or LTE-backed node, that bandwidth bill is the actual cost.
This change makes the existing s/w/n/e params honored — when all four
bounds are supplied, the backend bbox-filters a curated set of heavy,
density-driven, time-sensitive collections to that viewport (with the
existing 20% padding from _bbox_filter):
/fast: commercial_flights, military_flights, private_flights,
private_jets, tracked_flights, ships, cctv, uavs, liveuamap,
gps_jamming, sigint, trains
/slow: gdelt, firms_fires, kiwisdr, scanners, psk_reporter
Static reference layers (satellites, datacenters, military_bases,
power_plants, satnogs, weather, news, stocks, etc.) deliberately STAY
world-scale so panning never reveals an "empty world" of infrastructure.
That preserves the no-hostile-UX feel of the existing dashboard.
Behavior contract:
* Without bbox params (or with a partial bbox), the response is
byte-for-byte identical to the pre-#288 implementation. No
behavior change for any existing caller that hasn't opted in.
* World-scale bbox (lng_span >= 300 or lat_span >= 120) short-circuits
filtering and shares the global ETag — zoomed-out operators all
hit the same 304 cache exactly like before.
* ETag now mixes a 1°-quantized bbox suffix when filtering engages,
so two viewports never poison each other's 304 cache. Sub-degree
pans land in the same ETag bucket (i.e. don't bust the cache on
every mouse drag).
Polling cadence, rate-limit windows, and the 304 short-circuit are all
unchanged. Only the SIZE of the responses changes, and only when the
caller opts in via bounds.
Frontend wiring: useViewportBounds reuses the same coarsened/
expanded bounds it already computes for the AIS /api/viewport POST and
pushes them into a new module-level liveDataViewport store.
useDataPolling reads from that store via appendLiveDataBoundsParams
when building each live-data URL.
Tests cover: no-bbox → world data; bbox → heavy layers filtered;
bbox → reference layers untouched; world-scale bbox → no filter;
partial bbox → treated as no bbox; ETag changes with bbox; sub-degree
pan → same ETag; 304 path works; antimeridian-crossing bbox handled.
Co-authored-by: BigBodyCobain <moatbc@gmail.com>
Wikimedia's User-Agent policy asks API clients to identify themselves
with a stable, contactable identifier so their operators can rate-limit
or coordinate. Before this change, ShadowBroker was sending:
- Backend (region_dossier.py): generic project default UA only; no
Api-User-Agent.
- Frontend (useRegionDossier.ts, WikiImage.tsx, NewsFeed.tsx): zero
identifying header at all; three separate copy-pasted anonymous
fetches with their own module-local caches.
Three separate components doing the same broken thing meant policy
fixes had to happen in three places, with no shared cache or kill
switch.
Fix (no UX change, zero hostility):
== Backend ==
`backend/services/region_dossier.py` now sets explicit `User-Agent` +
`Api-User-Agent` headers on every outbound Wikidata and Wikipedia
request via a new `_WIKIMEDIA_REQUEST_HEADERS` constant. The identifier
includes a contact path (issues page on the public GitHub repo).
== Frontend ==
New shared helper `frontend/src/lib/wikimediaClient.ts`:
- `fetchWikipediaSummary(title)` — single source of truth for Wikipedia
REST summary lookups, with one shared LRU cache (in-flight requests
deduplicated, 512-entry cap), `Api-User-Agent` on every fetch.
- `fetchWikidataSparql(query)` — same shape for Wikidata SPARQL.
- `WIKIMEDIA_API_USER_AGENT` — exported constant; one place to update
if Wikimedia ever asks us to back off.
Refactored three components to use the shared client:
- `frontend/src/hooks/useRegionDossier.ts` — fetchLeader() and
fetchLocalWikiSummary() now route through the shared helpers.
- `frontend/src/components/WikiImage.tsx` — uses fetchWikipediaSummary,
proper React state instead of module-mutation + forceUpdate trick.
- `frontend/src/components/NewsFeed.tsx` — same shape.
UX: byte-for-byte identical. Same thumbnails, same dossier content,
same load behavior. The only observable difference is the outgoing
request header.
Note on #239 (route duplication): an audit-grade inventory shows 166
main.py routes are shadowed by router modules. That cleanup is too
large to land safely in this PR; it will be staged as a separate
ladder of small PRs grouped by router module.
Tests:
- `backend/tests/test_region_dossier_wikimedia_ua.py` — 3 tests
asserting backend headers are present.
- `frontend/src/__tests__/utils/wikimediaClient.test.ts` — 9 tests
covering Api-User-Agent presence, shared cache, concurrent
deduplication, disambiguation/HTTP-error/network-error fallthroughs,
empty-input safety.
Local: backend 76/76 security suite green, frontend 716/716 vitest
suite green.
Credit: tg12 (external security audit).
Each alert toast had a 5-second auto-dismiss timer that fired even
while the user was reading the card. This adds pause-on-hover: the
dismiss timer stops while the mouse is over a toast and restarts (full
lifetime) on mouse leave. The progress bar animation pauses with it,
so the visual matches the actual remaining time.
All other behavior is preserved: same cyber/mono styling, same spring
slide-in, same risk-color border + glow, same warning icon, same
LVL X/10 readout, same title/source layout, same click-to-fly + dismiss
on body click, same × dismiss button.
Implementation notes:
- Extract a ToastCard sub-component so each card can own its own
paused state (useState can't be array-indexed in the parent).
- Move the auto-dismiss timer out of useAlertToasts.ts and into
ToastCard. The hook previously scheduled the dismiss itself, which
meant the UI couldn't pause it — only the component knows whether
the user is interacting.
- Add tests covering: title/source/severity render, auto-dismiss
fires at 5s, hover pauses indefinitely, mouse-leave restarts the
full lifetime, × dismisses without flying, body-click flies +
dismisses.
This implements the genuine UX improvement that PR #234 was reaching
for, without #234's broken syntax, missing-field bug, duplicate
timer logic, or design regression.
Refs: #234
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
- Add Server-Sent Events endpoint at GET /api/mesh/gate/stream that
broadcasts ALL gate events to connected frontends (privacy: no
per-gate subscriptions, clients filter locally)
- Hook SSE broadcast into all gate event entry points: local append,
peer push receiver, and pull loop
- Reduce push/pull intervals from 30s to 10s for faster relay sync
- Add useGateSSE hook for frontend EventSource integration
- GateView + MeshChat use SSE for instant refresh, polling demoted
to 30s fallback
Latency: same-node instant, cross-node ~10s avg (was ~34s)
Gate messages now propagate via the Infonet hashchain as encrypted blobs — every node syncs them
through normal chain sync while only Gate members with MLS keys can decrypt. Added mesh reputation
system, peer push workers, voluntary Wormhole opt-in for node participation, fork recovery,
killwormhole scripts, obfuscated terminology, and hardened the self-updater to protect encryption
keys and chain state during updates.
New features: Shodan search, train tracking, Sentinel Hub imagery, 8 new intelligence layers,
CCTV expansion to 11,000+ cameras across 6 countries, Mesh Terminal CLI, prediction markets,
desktop-shell scaffold, and comprehensive mesh test suite (215 frontend + backend tests passing).
Community contributors: @wa1id, @AlborzNazari, @adust09, @Xpirix, @imqdcr, @csysp, @suranyami,
@chr0n1x, @johan-martensson, @singularfailure, @smithbh, @OrfeoTerkuci, @deuza, @tm-const,
@Elhard1, @ttulttul
BigBodyCobain