CalvinBackup/Shadowbroker
1
0
Fork 0
mirror of https://github.com/BigBodyCobain/Shadowbroker.git synced 2026-06-09 07:43:59 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: Telegram OSINT map layer, Osiris intel ports, and maritime settings

Browse Source 
Add Telegram OSINT with hourly incremental t.me scraping, metro geocoding
separate from news centroids, threat-intercept popup UI with inline media,
and HTML markers above alert boxes so pins stay clickable. Expose GFW_API_TOKEN
in onboarding and Settings Maritime; harden GFW/CCTV/geo fetchers. Port Osiris-
derived recon, SCM, entity graph, malware/cyber feeds, sanctions, and submarine
cable layers with tests and documentation.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
BigBodyCobain
2026-06-08 21:04:08 -06:00
parent b64b9e0962
commit af9b3d08cc
76 changed files with 5769 additions and 218 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+17
View File
@@ -10,6 +10,23 @@ OPENSKY_CLIENT_ID=
OPENSKY_CLIENT_SECRET=
AIS_API_KEY=
# Global Fishing Watch — fishing vessel activity events (Fishing Activity map layer).
# Free API token from https://globalfishingwatch.org/our-apis/tokens
# Without this the fishing_activity layer stays empty.
# GFW_API_TOKEN=
# Optional tuning — GFW can return 40k+ global events; defaults cap fetch for map paint.
# GFW_EVENTS_PAGE_SIZE=500
# GFW_EVENTS_MAX_PAGES=10
# GFW_EVENTS_LOOKBACK_DAYS=7
# GFW_EVENTS_TIMEOUT_S=90
# Windy Webcams global CCTV layer — free key from https://api.windy.com/webcams/docs
# WINDY_API_KEY=
# Telegram OSINT map layer — scrapes public t.me/s channel previews (no bot token).
# TELEGRAM_OSINT_ENABLED=true
# TELEGRAM_OSINT_CHANNELS=osintdefender,insiderpaper,aljazeeraenglish,nexta_live,war_monitor
# Admin key to protect sensitive endpoints (settings, updates).
# If blank, loopback/localhost requests still work for local single-host dev.
# Remote/non-loopback admin access requires ADMIN_KEY, or ALLOW_INSECURE_ADMIN=true in debug-only setups.
README.md
+85 -13
View File
@@ -19,7 +19,7 @@
**ShadowBroker** is a decentralized intelligence platform that aggregates real-time, multi-domain OSINT telemetry from 60+ live intelligence feeds into a single dark-ops map interface. Aircraft, ships, satellites, conflict zones, CCTV networks, GPS jamming, internet-connected devices, police scanners, mesh radio nodes, and breaking geopolitical events — all updating in real time on one screen as well as an obfuscated communications protocol and information exchange infrastructure.
Built with **Next.js**, **MapLibre GL**, **FastAPI**, and **Python**. 35+ toggleable data layers, including SAR ground-change detection. Multiple visual modes (DEFAULT / SATELLITE / FLIR / NVG / CRT). Right-click any point on Earth for a country dossier, head-of-state lookup, and the latest Sentinel-2 satellite photo. ShadowBroker has no accounts, product telemetry, or analytics; the dashboard talks to your self-hosted backend, while optional live OSINT panels may contact their configured public data providers when you use them.
Built with **Next.js**, **MapLibre GL**, **FastAPI**, and **Python**. 40+ toggleable data layers, including SAR ground-change detection, a **server-side recon toolkit** (DNS, WHOIS, sanctions, BGP, IP sweep, and more), supply-chain risk overlays, and malware/C2 threat feeds. Multiple visual modes (DEFAULT / SATELLITE / FLIR / NVG / CRT). Right-click any point on Earth for a country dossier, head-of-state lookup, entity-graph expansion, and the latest Sentinel-2 satellite photo. ShadowBroker has no accounts, product telemetry, or analytics; the dashboard talks to your self-hosted backend. Sensitive recon and Shodan queries never hit third-party APIs from the browser — they are proxied through the backend with SSRF guards and local-operator auth.
Designed for analysts, researchers, radio operators, and anyone who wants to see what the world looks like when every public signal is on the same map.
@@ -30,16 +30,18 @@ A surprising amount of global telemetry is already public — aircraft ADS-B bro
The project does not introduce new surveillance capabilities — it aggregates and visualizes existing public datasets. It is fully open-source so anyone can audit exactly what data is accessed and how. ShadowBroker does not include product telemetry, analytics, or accounts. Operator-supplied keys stay in your local deployment, but live OSINT features necessarily make outbound requests to the public data providers you enable or query.
### Shodan Connector
### Shodan & Recon (security-first)
ShadowBroker includes an optional Shodan connector for operator-supplied API access. Shodan results are fetched with your own `SHODAN_API_KEY`, rendered as a local investigative overlay (not merged into core feeds), and remain subject to Shodan’s terms of service.
ShadowBroker includes an optional **Shodan connector** for operator-supplied API access (`SHODAN_API_KEY`) and a **Recon Toolkit** panel for keyless OSINT lookups. Both run **server-side only**: the browser calls your self-hosted `/api/osint/*` and `/api/tools/shodan/*` routes; outbound requests are made by the backend after SSRF validation. Recon requires **local-operator** access (same trust model as layer toggles and admin routes). Shodan results render as a separate map overlay and remain subject to Shodan’s terms of service.
> **Not included:** embedded live-news YouTube grids or a built-in Gemini AI analyst panel — use the **OpenClaw / agent channel** for AI-assisted analysis instead.
---
## Interesting Use Cases
* **Track Air Force One**, the private jets of billionaires and dictators, and every military tanker, ISR, and fighter broadcasting ADS-B. Air Force One and all of the accompanying Presidential/Vice Presidential planes are highlighted and monitored from the moment they leave the ground.
* **Connect an AI agent as a co-analyst** through ShadowBroker's HMAC-signed agentic command channel — supports OpenClaw and any other agent that speaks the protocol (Claude, GPT, LangChain, custom). The agent gets full read/write access to all 35+ data layers, pin placement, map control, SAR ground-change, mesh networking, and alert delivery. It sees everything the operator sees and can take actions on the map in real time.
* **Connect an AI agent as a co-analyst** through ShadowBroker's HMAC-signed agentic command channel — supports OpenClaw and any other agent that speaks the protocol (Claude, GPT, LangChain, custom). The agent gets full read/write access to all 40+ data layers, pin placement, map control, SAR ground-change, mesh networking, and alert delivery. It sees everything the operator sees and can take actions on the map in real time.
* **Communicate on the InfoNet testnet** — The first decentralized intelligence mesh built into an OSINT tool. Obfuscated messaging with gate personas, Dead Drop peer-to-peer exchange, and a built-in terminal CLI. No accounts, no signup. Privacy is not guaranteed yet — this is an experimental testnet — but the protocol is live and being hardened.
* **Right-click anywhere on Earth** for a country dossier (head of state, population, languages), Wikipedia summary, and the latest Sentinel-2 satellite photo at 10m resolution
* **Click a KiwiSDR node** and tune into live shortwave radio directly in the dashboard. Click a police scanner feed and eavesdrop in one click.
@@ -55,6 +57,11 @@ ShadowBroker includes an optional Shodan connector for operator-supplied API acc
* **Track trains** across the US (Amtrak) and Europe (DigiTraffic) in real time
* **Estimate where US aircraft carriers are** using automated GDELT news scraping — no other open tool does this
* **Search internet-connected devices worldwide** via Shodan — cameras, SCADA systems, databases — plotted as a live overlay on the map
* **Run a full recon toolkit** from the left sidebar — IP geolocation, DNS, RDAP/WHOIS, certificate transparency, BGP/ASN, OFAC sanctions search, CVE lookup, Tor/OTX threat checks, and subnet sweeps (InternetDB proxied server-side)
* **Expand an entity graph** when you select an aircraft, vessel, company, or IP — Wikidata + OFAC + live store cross-links rendered in the Entity Graph panel
* **Monitor supply-chain risk** — Tier 1/2 semiconductor and battery fabs scored against nearby earthquakes, wildfires, and conflict events (SCM panel)
* **Toggle malware C2 hotspots** — abuse.ch Feodo Tracker + URLhaus feeds mapped by country (opt-in layer)
* **Overlay global submarine cables** — static TeleGeography-derived cable routes (opt-in layer)
---
@@ -239,11 +246,26 @@ The first decentralized intelligence communication and governance layer built di
> **Experimental Testnet — No Privacy Guarantee:** InfoNet messages are obfuscated but NOT end-to-end encrypted. The Mesh network (Meshtastic/APRS) is NOT private — radio transmissions are inherently public. The privacy primitive contracts are scaffolded but not yet wired. Do not send anything sensitive on any channel. Treat all channels as open and public for now.
### 🔍 Shodan Device Search (NEW in v0.9.6)
### 🔍 Recon Toolkit & Shodan (Osiris-derived, security-first)
* **Internet Device Search** — Query Shodan directly from ShadowBroker. Search by keyword, CVE, port, or service — results plotted as a live overlay on the map
Adapted from the [OSIRIS](https://github.com/simplifaisoul/osiris) recon stack (MIT) with ShadowBroker’s proxy model. Attribution: `backend/third_party/osiris/NOTICE.md`.
**Recon Toolkit** (left sidebar — local operator only):
* **IP / DNS / WHOIS** — ip-api.com geolocation, Google DNS-over-HTTPS, RDAP registrant data with optional HTTP security header scoring
* **Certificates & BGP** — crt.sh subdomain discovery, bgpview.io ASN/prefix lookups
* **Threat intel** — AlienVault OTX pulses, Tor exit-node checks, optional per-IP/domain reputation
* **Sanctions** — OpenSanctions `us_ofac_sdn` index (CC-BY); cross-checks on WHOIS entities and IP ISP/org strings
* **CVE / MAC / GitHub / leaks** — MITRE CVE API, MAC vendor lookup, GitHub profile recon, public breach checks
* **IP sweep** — `/api/osint/sweep/scan` geolocates a target /24–/32 and proxies Shodan InternetDB host discovery server-side (browser never contacts InternetDB directly)
* **SSRF guard** — Private, loopback, link-local, and metadata hostnames are blocked before any user-supplied fetch
**Entity graph** — Select any map entity to open the Entity Graph panel (`GET /api/entity/expand`). Resolves aircraft, vessels, companies, persons, IPs, and countries into a node/link graph (Wikidata SPARQL + OFAC + in-memory flight/ship store).
**Shodan overlay** (unchanged):
* **Internet Device Search** — Query Shodan with your own API key; results plotted as a live overlay
* **Configurable Markers** — Shape, color, and size customization for Shodan results
* **Operator-Supplied API** — Uses your own `SHODAN_API_KEY`; results rendered as a local investigative overlay
### 🛩️ Aviation Tracking
@@ -331,11 +353,12 @@ The first decentralized intelligence communication and governance layer built di
### 📷 Surveillance
* **CCTV Mesh** — 11,000+ live traffic cameras from 13 sources across 6 countries:
* **CCTV Mesh** — 22,000+ live traffic cameras from 21 ingestors across 10 countries (US, UK, Canada, Australia, Austria, Spain, Singapore, Netherlands when NDW feed is up, plus OSM):
* 🇬🇧 Transport for London JamCams
* 🇺🇸 NYC DOT, Austin TX (TxDOT)
* 🇺🇸 California (12 Caltrans districts), Washington State (WSDOT), Georgia DOT, Illinois DOT, Michigan DOT
* 🇪🇸 Spain DGT National (20 cities), Madrid City (357 cameras via KML)
* 🇦🇹 Austria ASFINAG motorway webcams
* 🇸🇬 Singapore LTA
* 🌍 Windy Webcams
* **Feed Rendering** — Automatic detection & rendering of video, MJPEG, HLS, embed, satellite tile, and image feeds
@@ -356,6 +379,11 @@ The first decentralized intelligence communication and governance layer built di
* **Data Center Mapping** — 2,000+ global data centers plotted from a curated dataset. Clustered purple markers with server-rack icons. Click for operator, location, and automatic internet outage cross-referencing by country.
* **Military Bases** — Global military installation and missile facility database (NEW)
* **Power Plants** — 35,000+ global power plants from the WRI database (NEW)
* **Submarine Cables** — Global undersea cable routes from static TeleGeography-derived GeoJSON (`frontend/public/data/submarine-cables.json`). Opt-in line overlay.
* **Malware C2 Layer** — Botnet C2 servers (Feodo Tracker) and recent malware URLs (URLhaus) from abuse.ch, refreshed on the slow tier when the layer is enabled.
* **SCM Supplier Risk** — Tier 1/2 fabs and battery plants (TSMC, Samsung, CATL, etc.) cross-referenced against earthquakes, FIRMS fires, and GDELT conflict proximity. Alerts in the SCM panel; optional map layer.
* **Cyber Threats Feed** — Recent CISA Known Exploited Vulnerabilities (KEV) entries exposed via `/api/cyber-threats` and the layer toggle.
* **Country Risk Index** — Static geopolitical risk scores with USGS earthquake enrichment via `/api/country-risk`.
### 🌐 Additional Layers & Tools
@@ -381,7 +409,7 @@ v0.9.7 turns ShadowBroker from a dashboard a human watches into an intelligence
**Capabilities:**
* **Full Telemetry Access** — The agent queries all 35+ data layers: flights, ships, satellites, SIGINT, conflict events, earthquakes, fires, wastewater, prediction markets, and more. Fast and slow tier endpoints return enriched data with geographic coordinates, timestamps, and source attribution.
* **Full Telemetry Access** — The agent queries all 40+ data layers: flights, ships, satellites, SIGINT, conflict events, earthquakes, fires, wastewater, malware/C2, SCM overlays, prediction markets, and more. Fast and slow tier endpoints return enriched data with geographic coordinates, timestamps, and source attribution.
* **AI Intel Pins** — Place color-coded investigation markers directly on the operator's map. 14 pin categories (threat, anomaly, military, maritime, aviation, SIGINT, infrastructure, etc.) with confidence scores, TTL expiry, source URLs, and batch placement up to 100 pins at once.
* **Map Control** — Fly the operator's map view to any coordinate, trigger satellite imagery lookups, and open region dossiers. The agent can direct the operator's attention to specific locations in real time.
* **SAR Ground-Change** — Query SAR anomaly feeds, inspect pin details, manage AOIs, and fly the map to watch areas. The agent can monitor for ground deformation, flood extent, or damage and promote anomalies to pins.
@@ -543,9 +571,19 @@ ShadowBroker v0.9.7 is composed of three vertically-stacked planes — the **Ope
| [GDELT Project](https://www.gdeltproject.org) | Global conflict events | ~6h | No |
| [DeepState Map](https://deepstatemap.live) | Ukraine frontline | ~30min | No |
| [Shodan](https://www.shodan.io) | Internet-connected device search | On-demand | **Yes** |
| [OpenSanctions](https://www.opensanctions.org) | OFAC SDN sanctions index (recon + entity graph) | 24h cache | No |
| [abuse.ch Feodo + URLhaus](https://abuse.ch) | Malware C2 / distribution URLs | ~5min (opt-in layer) | No |
| [CISA KEV](https://www.cisa.gov/known-exploited-vulnerabilities-catalog) | Known exploited CVEs | ~5min (opt-in layer) | No |
| [ip-api.com](https://ip-api.com) | IP geolocation (recon, entity graph) | On-demand | No |
| [Google Public DNS](https://dns.google) | DNS-over-HTTPS lookups (recon) | On-demand | No |
| [RDAP.org](https://rdap.org) | Domain registration data (recon) | On-demand | No |
| [crt.sh](https://crt.sh) | Certificate transparency (recon) | On-demand | No |
| [bgpview.io](https://bgpview.io) | BGP/ASN routing (recon) | On-demand | No |
| TeleGeography (static) | Submarine cable routes | Static | No |
| [ASFINAG](https://www.asfinag.at) | Austria motorway webcams | ~10min | No |
| [Amtrak](https://www.amtrak.com) | US train positions | ~60s | No |
| [DigiTraffic](https://www.digitraffic.fi) | European rail positions | ~60s | No |
| [Global Fishing Watch](https://globalfishingwatch.org) | Fishing vessel activity events | ~10min | No |
| [Global Fishing Watch](https://globalfishingwatch.org) | Fishing vessel activity events | ~1hr | **Yes** (`GFW_API_TOKEN`) |
| Transport for London, NYC DOT, TxDOT | CCTV cameras (UK, US) | ~10min | No |
| Caltrans, WSDOT, GDOT, IDOT, MDOT | CCTV cameras (5 US states) | ~10min | No |
| Spain DGT, Madrid City | CCTV cameras (Spain) | ~10min | No |
@@ -821,7 +859,7 @@ AIS-catcher decodes VHF radio signals on 161.975 MHz and 162.025 MHz and POSTs d
## 🎛️ Data Layers
All 37 layers are independently toggleable from the left panel:
All 41 layers are independently toggleable from the left panel:
| Layer | Default | Description |
|---|---|---|
@@ -863,6 +901,20 @@ All 37 layers are independently toggleable from the left panel:
| VIIRS Nightlights | ❌ OFF | Night-time light change detection |
| Power Plants | ❌ OFF | 35,000+ global power plants |
| Shodan Overlay | ❌ OFF | Internet device search results |
| Road Freight Trends | ❌ OFF | Sentinel-2 truck-motion trends on major highways (Analyze Here) |
| Submarine Cables | ❌ OFF | Global undersea cable routes (static GeoJSON) |
| Malware C2 | ❌ OFF | abuse.ch Feodo + URLhaus threat points |
| SCM Suppliers | ❌ OFF | Tier 1/2 supply-chain risk markers + panel alerts |
| Cyber Threats | ❌ OFF | Recent CISA KEV entries (stats in slow-tier payload) |
| SAR | ✅ ON | Synthetic aperture radar catalog + anomaly alerts |
**Recon & entity tools** (not map layers — left sidebar / selection):
| Tool | Access | Description |
|---|---|---|
| Recon Toolkit | Local operator | DNS, WHOIS, sanctions, BGP, CVE, sweep, etc. via `/api/osint/*` |
| SCM Risk panel | Local operator | Live supplier threat rollup via `/api/scm-suppliers` |
| Entity Graph | Local operator | Graph expansion on selected entities via `/api/entity/expand` |
---
@@ -895,7 +947,16 @@ Shadowbroker/
│ │ ├── data_fetcher.py # Core scheduler — orchestrates all data sources
│ │ ├── ais_stream.py # AIS WebSocket client (25K+ vessels)
│ │ ├── carrier_tracker.py # OSINT carrier position estimator (GDELT news scraping)
│ │ ├── cctv_pipeline.py # 13-source CCTV camera ingestion pipeline
│ │ ├── cctv_pipeline.py # 14-source CCTV camera ingestion pipeline
│ │ ├── ssrf_guard.py # SSRF validation for operator recon fetches
│ │ ├── sanctions/ofac.py # OpenSanctions OFAC SDN index
│ │ ├── osint/lookups.py # Server-side recon lookups (Osiris port)
│ │ ├── osint_intel/resolve.py # Entity graph resolver (Wikidata + OFAC)
│ │ ├── scm/suppliers.py # Supply-chain risk overlay
│ │ ├── intel_feeds/ # Country risk index helpers
│ │ ├── fetchers/malware.py # abuse.ch Feodo + URLhaus
│ │ ├── fetchers/cyber_status.py # CISA KEV feed
│ │ ├── third_party/osiris/ # MIT attribution for Osiris-derived code
│ │ ├── geopolitics.py # GDELT + Ukraine frontline + air alerts
│ │ ├── region_dossier.py # Right-click country/city intelligence
│ │ ├── radio_intercept.py # Police scanner feeds + OpenMHZ
@@ -933,7 +994,14 @@ Shadowbroker/
│ │ ├── mesh_reputation.py # Node reputation scoring
│ │ ├── mesh_oracle.py # Oracle consensus protocol
│ │ └── mesh_secure_storage.py # Secure credential storage
│ ├── routers/
│ │ ├── osint.py # /api/osint/* recon routes (local operator)
│ │ ├── entity_graph.py # /api/entity/expand
│ │ ├── scm.py # /api/scm-suppliers
│ │ └── intel_feeds.py # /api/malware, /api/cyber-threats, /api/country-risk
├── frontend/
│ ├── public/data/
│ │ └── submarine-cables.json # Static undersea cable GeoJSON
│ ├── src/
│ │ ├── app/
│ │ │ └── page.tsx # Main dashboard — state, polling, layout
@@ -942,7 +1010,11 @@ Shadowbroker/
│ │ ├── MeshChat.tsx # InfoNet / Mesh / Dead Drop chat panel
│ │ ├── MeshTerminal.tsx # Draggable CLI terminal
│ │ ├── NewsFeed.tsx # SIGINT feed + entity detail panels
│ │ ├── WorldviewLeftPanel.tsx # Data layer toggles (35+ layers)
│ │ ├── WorldviewLeftPanel.tsx # Data layer toggles (40+ layers)
│ │ ├── ShodanPanel.tsx # Shodan device search overlay
│ │ ├── ReconPanel.tsx # Server-side OSINT recon toolkit
│ │ ├── ScmPanel.tsx # Supply-chain risk command panel
│ │ ├── EntityGraphPanel.tsx # Entity graph on map selection
│ │ ├── WorldviewRightPanel.tsx # Search + filter sidebar
│ │ ├── AdvancedFilterModal.tsx # Airport/country/owner filtering
│ │ ├── MapLegend.tsx # Dynamic legend with all icons
backend/.env.example
+13
View File
@@ -100,6 +100,19 @@ AIS_API_KEY= # https://aisstream.io/ — free tier WebSocket key
# configured news feeds (kill switch for the news layer).
# NEWS_ENABLED=true
# Global Fishing Watch — fishing vessel activity events (Fishing Activity map layer).
# Free API token from https://globalfishingwatch.org/our-apis/tokens
# Without this the fishing_activity layer stays empty.
# GFW_API_TOKEN=
# Optional tuning — GFW can return 40k+ global events; defaults cap fetch for map paint.
# GFW_EVENTS_PAGE_SIZE=500
# GFW_EVENTS_MAX_PAGES=10
# GFW_EVENTS_LOOKBACK_DAYS=7
# GFW_EVENTS_TIMEOUT_S=90
# Windy Webcams global CCTV layer — free key from https://api.windy.com/webcams/docs
# WINDY_API_KEY=
# LTA Singapore traffic cameras — leave blank to skip this data source.
# LTA_ACCOUNT_KEY=
backend/main.py
+96 -2
View File
@@ -366,6 +366,10 @@ ai_intel_router = _load_optional_router("routers.ai_intel")
sar_router = _load_optional_router("routers.sar")
infonet_router = _load_optional_router("routers.infonet")
road_corridors_router = _load_optional_router("routers.road_corridors")
osint_router = _load_optional_router("routers.osint")
scm_router = _load_optional_router("routers.scm")
entity_graph_router = _load_optional_router("routers.entity_graph")
intel_feeds_router = _load_optional_router("routers.intel_feeds")
# ---------------------------------------------------------------------------
@@ -3643,6 +3647,10 @@ app.include_router(ai_intel_router)
app.include_router(sar_router)
app.include_router(infonet_router)
app.include_router(road_corridors_router)
app.include_router(osint_router)
app.include_router(scm_router)
app.include_router(entity_graph_router)
app.include_router(intel_feeds_router)
from services.data_fetcher import update_all_data
@@ -3774,6 +3782,8 @@ async def update_layers(update: LayerUpdate, request: Request):
old_mesh = is_any_active("sigint_meshtastic")
old_aprs = is_any_active("sigint_aprs")
old_viirs = is_any_active("viirs_nightlights")
old_datacenters = is_any_active("datacenters")
old_fishing = is_any_active("fishing_activity")
# Update only known keys
changed = False
@@ -3792,6 +3802,8 @@ async def update_layers(update: LayerUpdate, request: Request):
new_mesh = is_any_active("sigint_meshtastic")
new_aprs = is_any_active("sigint_aprs")
new_viirs = is_any_active("viirs_nightlights")
new_datacenters = is_any_active("datacenters")
new_fishing = is_any_active("fishing_activity")
# Start/stop AIS stream on transition
if old_ships and not new_ships:
@@ -3847,6 +3859,18 @@ async def update_layers(update: LayerUpdate, request: Request):
_queue_viirs_change_refresh()
logger.info("VIIRS change refresh queued (layer enabled)")
if not old_datacenters and new_datacenters:
from services.fetchers.infrastructure import fetch_datacenters
fetch_datacenters()
logger.info("Datacenters loaded (layer enabled)")
if not old_fishing and new_fishing:
from services.fetchers.geo import fetch_fishing_activity
fetch_fishing_activity()
logger.info("Fishing activity refresh queued (layer enabled)")
return {"status": "ok"}
@@ -7834,6 +7858,8 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
"www.tripcheck.com",
"infocar.dgt.es", # Spain DGT
"informo.madrid.es", # Madrid
"webcams2.asfinag.at", # Austria ASFINAG motorway cameras
"odo.asfinag.at", # ASFINAG catalog API host
"www.windy.com",
"imgproxy.windy.com", # Windy preview image CDN
"www.lakecountypassage.com", # Illinois Lake County PASSAGE snapshots
@@ -7842,6 +7868,14 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
"www.nps.gov", # WSDOT-linked Mount Rainier camera
"home.lewiscounty.com", # WSDOT partner public camera
"www.seattle.gov", # Seattle traffic camera media linked from WSDOT
"511on.ca", # Ontario 511 cameras
"511.alberta.ca", # Alberta 511 cameras
"fl511.com", # Florida 511 cameras
"www.fl511.com",
"webcams.transport.nsw.gov.au", # NSW Live Traffic camera snapshots
"www.livetraffic.com",
"livetraffic.com",
"opendata.ndw.nu", # Netherlands RWS legacy open-data host
}
@@ -7937,7 +7971,7 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
cache_seconds=15,
headers={
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "http://navigator-c2c.dot.ga.gov/",
"Referer": "https://navigator-c2c.dot.ga.gov/",
},
)
if host == "511ga.org":
@@ -7957,7 +7991,7 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
cache_seconds=10,
headers={
"Accept": "application/vnd.apple.mpegurl,application/x-mpegURL,video/*,*/*;q=0.8",
"Referer": "http://navigator-c2c.dot.ga.gov/",
"Referer": "https://navigator-c2c.dot.ga.gov/",
},
)
if host in {"gettingaroundillinois.com", "cctv.travelmidwest.com"}:
@@ -8039,6 +8073,16 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
"Referer": "https://informo.madrid.es/",
},
)
if host in {"webcams2.asfinag.at", "odo.asfinag.at"}:
return _CCTVProxyProfile(
name="asfinag-austria",
timeout=(5.0, 15.0),
cache_seconds=60,
headers={
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://www.asfinag.at/",
},
)
if host in {"www.windy.com", "imgproxy.windy.com"}:
return _CCTVProxyProfile(
name="windy-webcams",
@@ -8049,6 +8093,56 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
"Referer": "https://www.windy.com/",
},
)
if host == "511on.ca":
return _CCTVProxyProfile(
name="ontario-511",
timeout=(5.0, 15.0),
cache_seconds=30,
headers={
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://511on.ca/",
},
)
if host == "511.alberta.ca":
return _CCTVProxyProfile(
name="alberta-511",
timeout=(5.0, 15.0),
cache_seconds=30,
headers={
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://511.alberta.ca/",
},
)
if host in {"fl511.com", "www.fl511.com"}:
return _CCTVProxyProfile(
name="florida-511",
timeout=(5.0, 15.0),
cache_seconds=30,
headers={
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://fl511.com/",
},
)
if host == "webcams.transport.nsw.gov.au":
return _CCTVProxyProfile(
name="nsw-live-traffic",
timeout=(5.0, 12.0),
cache_seconds=60,
headers={
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://www.livetraffic.com/",
},
)
if host in {"opendata.ndw.nu", "www.ndw.nu"}:
return _CCTVProxyProfile(
name="ndw-netherlands",
timeout=(5.0, 12.0),
cache_seconds=120,
headers={
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://www.ndw.nu/",
},
)
if host in {
"webcam.forkswa.com",
"webcam.sunmountainlodge.com",
backend/routers/cctv.py
+36 -2
View File
@@ -47,6 +47,8 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
"www.tripcheck.com",
"infocar.dgt.es",
"informo.madrid.es",
"webcams2.asfinag.at",
"odo.asfinag.at",
"www.windy.com",
"imgproxy.windy.com",
"www.lakecountypassage.com",
@@ -55,6 +57,14 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
"www.nps.gov",
"home.lewiscounty.com",
"www.seattle.gov",
"511on.ca",
"511.alberta.ca",
"fl511.com",
"www.fl511.com",
"webcams.transport.nsw.gov.au",
"www.livetraffic.com",
"livetraffic.com",
"opendata.ndw.nu",
}
@@ -120,7 +130,7 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
read_timeout = 18.0 if "/snapshots/" in path else 12.0
return _CCTVProxyProfile(name="gdot-snapshot", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, read_timeout), cache_seconds=15,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "http://navigator-c2c.dot.ga.gov/"})
"Referer": "https://navigator-c2c.dot.ga.gov/"})
if host == "511ga.org":
return _CCTVProxyProfile(name="gdot-511ga-image", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=15,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
@@ -128,7 +138,7 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
if host.startswith("vss") and host.endswith("dot.ga.gov"):
return _CCTVProxyProfile(name="gdot-hls", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 20.0), cache_seconds=10,
headers={"Accept": "application/vnd.apple.mpegurl,application/x-mpegURL,video/*,*/*;q=0.8",
"Referer": "http://navigator-c2c.dot.ga.gov/"})
"Referer": "https://navigator-c2c.dot.ga.gov/"})
if host in {"gettingaroundillinois.com", "cctv.travelmidwest.com"}:
return _CCTVProxyProfile(name="illinois-dot", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=30,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8"})
@@ -156,10 +166,34 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
return _CCTVProxyProfile(name="madrid-city", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=30,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://informo.madrid.es/"})
if host in {"webcams2.asfinag.at", "odo.asfinag.at"}:
return _CCTVProxyProfile(name="asfinag-austria", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 15.0), cache_seconds=60,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://www.asfinag.at/"})
if host in {"www.windy.com", "imgproxy.windy.com"}:
return _CCTVProxyProfile(name="windy-webcams", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=60,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://www.windy.com/"})
if host == "511on.ca":
return _CCTVProxyProfile(name="ontario-511", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 15.0), cache_seconds=30,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://511on.ca/"})
if host == "511.alberta.ca":
return _CCTVProxyProfile(name="alberta-511", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 15.0), cache_seconds=30,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://511.alberta.ca/"})
if host in {"fl511.com", "www.fl511.com"}:
return _CCTVProxyProfile(name="florida-511", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 15.0), cache_seconds=30,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://fl511.com/"})
if host == "webcams.transport.nsw.gov.au":
return _CCTVProxyProfile(name="nsw-live-traffic", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=60,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://www.livetraffic.com/"})
if host in {"opendata.ndw.nu", "www.ndw.nu"}:
return _CCTVProxyProfile(name="ndw-netherlands", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=120,
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
"Referer": "https://www.ndw.nu/"})
return _CCTVProxyProfile(name="generic-cctv", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 8.0), cache_seconds=30,
headers={"Accept": "*/*"})
backend/routers/data.py
+35
View File
@@ -502,6 +502,8 @@ async def update_layers(update: LayerUpdate, request: Request):
old_mesh = is_any_active("sigint_meshtastic")
old_aprs = is_any_active("sigint_aprs")
old_viirs = is_any_active("viirs_nightlights")
old_datacenters = is_any_active("datacenters")
old_fishing = is_any_active("fishing_activity")
changed = False
for key, value in update.layers.items():
if key in active_layers:
@@ -514,6 +516,8 @@ async def update_layers(update: LayerUpdate, request: Request):
new_mesh = is_any_active("sigint_meshtastic")
new_aprs = is_any_active("sigint_aprs")
new_viirs = is_any_active("viirs_nightlights")
new_datacenters = is_any_active("datacenters")
new_fishing = is_any_active("fishing_activity")
if old_ships and not new_ships:
from services.ais_stream import stop_ais_stream
stop_ais_stream()
@@ -557,6 +561,16 @@ async def update_layers(update: LayerUpdate, request: Request):
if not old_viirs and new_viirs:
_queue_viirs_change_refresh()
logger.info("VIIRS change refresh queued (layer enabled)")
if not old_datacenters and new_datacenters:
from services.fetchers.infrastructure import fetch_datacenters
fetch_datacenters()
logger.info("Datacenters loaded (layer enabled)")
if not old_fishing and new_fishing:
from services.fetchers.geo import fetch_fishing_activity
fetch_fishing_activity()
logger.info("Fishing activity refresh queued (layer enabled)")
return {"status": "ok"}
@@ -759,6 +773,7 @@ async def live_data_slow(
"scanners", "weather_alerts", "ukraine_alerts", "air_quality", "volcanoes",
"fishing_activity", "psk_reporter", "correlations", "uap_sightings", "wastewater",
"crowdthreat", "threat_level", "trending_markets", "road_corridor_trends",
"malware_threats", "cyber_threats", "scm_suppliers", "telegram_osint",
)
freshness = get_source_timestamps_snapshot()
payload = {
@@ -804,6 +819,26 @@ async def live_data_slow(
)
if active_layers.get("road_corridor_trends", False)
else {"updated_at": None, "corridors": []},
"malware_threats": (
d.get("malware_threats") or {"threats": [], "total": 0}
)
if active_layers.get("malware_c2", False)
else {"threats": [], "total": 0},
"cyber_threats": (
d.get("cyber_threats") or {"threats": [], "stats": {}}
)
if active_layers.get("cyber_threats", False)
else {"threats": [], "stats": {}},
"scm_suppliers": (
d.get("scm_suppliers") or {"suppliers": [], "total": 0, "critical_count": 0}
)
if active_layers.get("scm_suppliers", False)
else {"suppliers": [], "total": 0, "critical_count": 0},
"telegram_osint": (
d.get("telegram_osint") or {"posts": [], "total": 0, "geolocated": 0}
)
if active_layers.get("telegram_osint", True)
else {"posts": [], "total": 0, "geolocated": 0},
"freshness": freshness,
}
# Issue #288: bbox filter heavy/dense layers only when all four bounds
backend/routers/entity_graph.py
+30
View File
@@ -0,0 +1,30 @@
"""Entity graph expansion (intel layer)."""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from auth import require_local_operator
from limiter import limiter
from services.osint_intel.resolve import resolve_entity
router = APIRouter()
@router.get("/api/entity/expand")
@limiter.limit("30/minute")
async def entity_expand(
request: Request,
_: None = Depends(require_local_operator),
type: str = Query(..., min_length=3, max_length=32),
id: str = Query(..., min_length=2, max_length=200),
registration: str | None = Query(default=None, max_length=32),
model: str | None = Query(default=None, max_length=64),
icao24: str | None = Query(default=None, max_length=16),
) -> dict:
props = {"label": id, "registration": registration, "model": model, "icao24": icao24}
try:
return resolve_entity(type, id, props)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except Exception as exc:
raise HTTPException(status_code=502, detail="Intelligence layer unavailable") from exc
backend/routers/intel_feeds.py
+122
View File
@@ -0,0 +1,122 @@
"""Malware, cyber threats, and country risk feeds."""
from __future__ import annotations
import logging
from urllib.parse import urlparse
import requests
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from starlette.background import BackgroundTask
from limiter import limiter
from services.fetchers._store import get_latest_data_subset_refs
from services.fetchers.telegram_osint import telegram_media_host_allowed
from services.intel_feeds.country_risk import build_country_risk_payload
from services.network_utils import outbound_user_agent
logger = logging.getLogger(__name__)
router = APIRouter()
@router.get("/api/malware")
@limiter.limit("60/minute")
async def malware_feed(request: Request) -> dict:
snap = get_latest_data_subset_refs("malware_threats")
payload = snap.get("malware_threats")
if isinstance(payload, dict) and payload.get("threats") is not None:
return payload
return {"threats": [], "total": 0, "timestamp": None, "source": "abuse.ch"}
@router.get("/api/cyber-threats")
@limiter.limit("60/minute")
async def cyber_threats(request: Request) -> dict:
snap = get_latest_data_subset_refs("cyber_threats")
return snap.get("cyber_threats") or {"threats": [], "stats": {}}
@router.get("/api/country-risk")
@limiter.limit("30/minute")
async def country_risk(request: Request) -> dict:
return build_country_risk_payload()
@router.get("/api/telegram-feed")
@limiter.limit("30/minute")
async def telegram_feed(request: Request) -> dict:
snap = get_latest_data_subset_refs("telegram_osint")
payload = snap.get("telegram_osint")
if isinstance(payload, dict) and payload.get("posts") is not None:
return payload
return {"posts": [], "total": 0, "geolocated": 0, "timestamp": None}
def _infer_telegram_media_type(target_url: str, content_type: str) -> str:
clean_type = str(content_type or "").split(";", 1)[0].strip().lower()
if clean_type and clean_type not in {"application/octet-stream", "binary/octet-stream"}:
return content_type
path = str(urlparse(target_url).path or "").lower()
if path.endswith((".jpg", ".jpeg")):
return "image/jpeg"
if path.endswith(".png"):
return "image/png"
if path.endswith(".webp"):
return "image/webp"
if path.endswith(".gif"):
return "image/gif"
if path.endswith(".mp4"):
return "video/mp4"
if path.endswith(".webm"):
return "video/webm"
return content_type or "application/octet-stream"
@router.get("/api/telegram/media")
@limiter.limit("60/minute")
async def telegram_media_proxy(request: Request, url: str = Query(...)) -> StreamingResponse:
"""Stream Telegram CDN media for in-app playback (host allowlist only)."""
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
raise HTTPException(status_code=400, detail="Invalid scheme")
if not telegram_media_host_allowed(parsed.hostname):
raise HTTPException(status_code=403, detail="Host not allowed")
headers = {
"User-Agent": (
f"Mozilla/5.0 (compatible; {outbound_user_agent('telegram-media')}) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
),
"Accept": "*/*",
}
if range_header := request.headers.get("range"):
headers["Range"] = range_header
try:
resp = requests.get(url, stream=True, timeout=(3, 45), headers=headers)
except requests.RequestException as exc:
logger.warning("Telegram media upstream failure %s: %s", url, exc)
raise HTTPException(status_code=502, detail="Upstream fetch failed") from exc
if resp.status_code >= 400:
resp.close()
raise HTTPException(status_code=int(resp.status_code), detail=f"Upstream returned {resp.status_code}")
media_type = _infer_telegram_media_type(url, resp.headers.get("Content-Type", "application/octet-stream"))
response_headers = {
"Cache-Control": "private, max-age=300",
"Accept-Ranges": resp.headers.get("Accept-Ranges", "bytes"),
}
if content_length := resp.headers.get("Content-Length"):
response_headers["Content-Length"] = content_length
if content_range := resp.headers.get("Content-Range"):
response_headers["Content-Range"] = content_range
return StreamingResponse(
resp.iter_content(chunk_size=65536),
status_code=resp.status_code,
media_type=media_type,
headers=response_headers,
background=BackgroundTask(resp.close),
)
backend/routers/osint.py
+151
View File
@@ -0,0 +1,151 @@
"""Operator OSINT recon routes (server-side proxies, SSRF guarded)."""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from auth import require_local_operator
from limiter import limiter
from services.osint import lookups
router = APIRouter(dependencies=[Depends(require_local_operator)])
_ALLOWED_SCHEMAS = {
"Person",
"Organization",
"Company",
"Vessel",
"Airplane",
"LegalEntity",
}
class SweepScanRequest(BaseModel):
ip: str = Field(min_length=7, max_length=45)
cidr: int = Field(default=24, ge=24, le=32)
def _bad_request(exc: ValueError) -> HTTPException:
return HTTPException(status_code=400, detail=str(exc))
@router.get("/api/osint/ip")
@limiter.limit("20/minute")
async def osint_ip(request: Request, ip: str = Query(..., min_length=7, max_length=45)) -> dict:
try:
return lookups.lookup_ip(ip)
except ValueError as exc:
raise _bad_request(exc) from exc
@router.get("/api/osint/dns")
@limiter.limit("20/minute")
async def osint_dns(request: Request, domain: str = Query(..., min_length=4, max_length=253)) -> dict:
try:
return lookups.lookup_dns(domain)
except ValueError as exc:
raise _bad_request(exc) from exc
@router.get("/api/osint/whois")
@limiter.limit("20/minute")
async def osint_whois(request: Request, domain: str = Query(..., min_length=4, max_length=253)) -> dict:
try:
return lookups.lookup_whois(domain)
except ValueError as exc:
raise _bad_request(exc) from exc
@router.get("/api/osint/certs")
@limiter.limit("20/minute")
async def osint_certs(request: Request, domain: str = Query(..., min_length=4, max_length=253)) -> dict:
try:
return lookups.lookup_certs(domain)
except ValueError as exc:
raise _bad_request(exc) from exc
@router.get("/api/osint/threats")
@limiter.limit("20/minute")
async def osint_threats(request: Request, query: str | None = Query(default=None, max_length=253)) -> dict:
return lookups.lookup_threats(query)
@router.get("/api/osint/bgp")
@limiter.limit("20/minute")
async def osint_bgp(request: Request, query: str = Query(..., min_length=2, max_length=64)) -> dict:
try:
return lookups.lookup_bgp(query)
except ValueError as exc:
raise _bad_request(exc) from exc
@router.get("/api/osint/sanctions")
@limiter.limit("20/minute")
async def osint_sanctions(
request: Request,
query: str = Query(..., min_length=4, max_length=200),
schema: str | None = Query(default=None),
limit: int = Query(default=25, ge=1, le=100),
) -> dict:
if schema and schema not in _ALLOWED_SCHEMAS:
raise HTTPException(status_code=400, detail=f"Invalid schema. Allowed: {', '.join(sorted(_ALLOWED_SCHEMAS))}")
return lookups.lookup_sanctions(query, schema=schema, limit=limit)
@router.get("/api/osint/cve")
@limiter.limit("30/minute")
async def osint_cve(request: Request, cve: str = Query(..., min_length=10, max_length=32)) -> dict:
try:
return lookups.lookup_cve(cve)
except ValueError as exc:
raise HTTPException(status_code=404 if "not found" in str(exc).lower() else 400, detail=str(exc)) from exc
@router.get("/api/osint/mac")
@limiter.limit("20/minute")
async def osint_mac(request: Request, mac: str = Query(..., min_length=5, max_length=32)) -> dict:
return lookups.lookup_mac(mac)
@router.get("/api/osint/github")
@limiter.limit("20/minute")
async def osint_github(request: Request, username: str = Query(..., min_length=1, max_length=64)) -> dict:
try:
return lookups.lookup_github(username)
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
@router.get("/api/osint/leaks")
@limiter.limit("10/minute")
async def osint_leaks(request: Request, email: str = Query(..., min_length=5, max_length=254)) -> dict:
try:
return lookups.lookup_leaks(email)
except ValueError as exc:
raise _bad_request(exc) from exc
@router.get("/api/osint/sweep")
@limiter.limit("5/minute")
async def osint_sweep_init(
request: Request,
ip: str = Query(..., min_length=7, max_length=45),
cidr: int = Query(default=24, ge=24, le=32),
) -> dict:
try:
return lookups.sweep_init(ip, cidr)
except ValueError as exc:
raise _bad_request(exc) from exc
@router.post("/api/osint/sweep/scan")
@limiter.limit("3/minute")
async def osint_sweep_scan(request: Request, payload: SweepScanRequest) -> dict:
try:
subnet = lookups.subnet_start_for(payload.ip, payload.cidr)
scan = lookups.sweep_scan(subnet, payload.cidr)
init = lookups.sweep_init(payload.ip, payload.cidr)
return {**init, **scan, "subnet": f"{subnet}/{payload.cidr}"}
except ValueError as exc:
raise _bad_request(exc) from exc
backend/routers/scm.py
+16
View File
@@ -0,0 +1,16 @@
"""Supply-chain risk overlay."""
from __future__ import annotations
from fastapi import APIRouter, Depends, Request
from auth import require_local_operator
from limiter import limiter
from services.scm.suppliers import build_scm_payload
router = APIRouter()
@router.get("/api/scm-suppliers")
@limiter.limit("30/minute")
async def scm_suppliers(request: Request, _: None = Depends(require_local_operator)) -> dict:
return build_scm_payload()
backend/services/api_settings.py
+9
View File
@@ -51,6 +51,15 @@ API_REGISTRY = [
"url": "https://aisstream.io/",
"required": True,
},
{
"id": "gfw_api_token",
"env_key": "GFW_API_TOKEN",
"name": "Global Fishing Watch",
"description": "Bearer token for Global Fishing Watch fishing-vessel activity events (Fishing Activity map layer). Free registration at globalfishingwatch.org.",
"category": "Maritime",
"url": "https://globalfishingwatch.org/our-apis/",
"required": False,
},
{
"id": "adsb_lol",
"env_key": None,
backend/services/cctv_pipeline.py
+416 -109
View File
@@ -17,6 +17,9 @@ _KNOWN_CCTV_MEDIA_HOST_ALIASES = {
# Trusted upstream occasionally publishes a typo for this Georgia camera
# host. Normalize it at ingest so the proxy and client stay consistent.
"navigatos-c2c.dot.ga.gov": "navigator-c2c.dot.ga.gov",
# TravelIQ staging hosts occasionally appear in 511 catalog metadata.
"on.stage.traveliq.co": "511on.ca",
"ab.stage.traveliq.co": "511.alberta.ca",
}
_POINT_WKT_RE = re.compile(
@@ -40,6 +43,17 @@ def _normalize_cctv_media_url(raw_url: str) -> str:
return urlunparse(parsed._replace(netloc=netloc))
def _ensure_https_url(raw_url: str) -> str:
"""Upgrade http:// media/catalog URLs to https:// at ingest time."""
candidate = _normalize_cctv_media_url(str(raw_url or "").strip())
if not candidate:
return ""
parsed = urlparse(candidate)
if parsed.scheme.lower() == "http":
return urlunparse(parsed._replace(scheme="https"))
return candidate
def _looks_like_direct_cctv_media_url(url: str) -> bool:
candidate = str(url or "").strip().lower()
if not candidate.startswith(("http://", "https://")):
@@ -93,6 +107,165 @@ def _parse_wkt_point(raw_point: str) -> tuple[float | None, float | None]:
return lat, lon
def _fetch_traveliq_v2_cameras(
*,
api_url: str,
base_url: str,
id_prefix: str,
source_agency: str,
) -> List[Dict[str, Any]]:
"""Parse TravelIQ-style GET /api/v2/get/cameras feeds (Ontario, Alberta)."""
resp = fetch_with_curl(
api_url,
timeout=30,
headers={"Accept": "application/json"},
)
if not resp or resp.status_code != 200:
logger.error(
"%s CCTV fetch failed: HTTP %s",
source_agency,
resp.status_code if resp else "no response",
)
return []
data = resp.json()
if not isinstance(data, list):
return []
cameras: List[Dict[str, Any]] = []
for cam in data:
if not isinstance(cam, dict):
continue
try:
lat = float(cam.get("Latitude"))
lon = float(cam.get("Longitude"))
except (TypeError, ValueError):
continue
site_id = cam.get("Id")
location = str(cam.get("Location") or cam.get("Roadway") or "Camera")[:120]
views = cam.get("Views") or []
if not views:
continue
for view in views:
if not isinstance(view, dict):
continue
status = str(view.get("Status") or "enabled").strip().lower()
if status and status not in {"enabled", "active"}:
continue
media_url = _ensure_https_url(
urljoin(base_url, str(view.get("Url") or "").strip())
)
if not media_url:
continue
view_id = view.get("Id") or site_id
if site_id is None or view_id is None:
continue
label = str(view.get("Description") or location or "Camera")[:120]
cameras.append(
{
"id": f"{id_prefix}-{site_id}-{view_id}",
"source_agency": source_agency,
"lat": lat,
"lon": lon,
"direction_facing": label,
"media_url": media_url,
"media_type": "image",
"refresh_rate_seconds": 60,
}
)
return cameras
def _fetch_511_datatables_cameras(
*,
list_url: str,
base_url: str,
id_prefix: str,
source_agency: str,
referer: str,
page_size: int = 500,
) -> List[Dict[str, Any]]:
"""Parse 511 DataTables POST /List/GetData/Cameras feeds (Georgia, Florida)."""
cameras: List[Dict[str, Any]] = []
start = 0
draw = 1
while True:
resp = fetch_with_curl(
list_url,
method="POST",
json_data={"draw": draw, "start": start, "length": page_size},
timeout=30,
headers={
"Accept": "application/json",
"Referer": referer,
"Origin": base_url.rstrip("/"),
},
)
if not resp or resp.status_code != 200:
logger.error(
"%s CCTV fetch failed: HTTP %s",
source_agency,
resp.status_code if resp else "no response",
)
break
data = resp.json()
rows = data.get("data") or []
if not rows:
break
for row in rows:
if not isinstance(row, dict):
continue
site_id = row.get("id") or row.get("DT_RowId")
location = row.get("location") or row.get("roadway") or source_agency
lat_lng = row.get("latLng") or {}
geography = lat_lng.get("geography") if isinstance(lat_lng, dict) else {}
lat, lon = _parse_wkt_point(
geography.get("wellKnownText") if isinstance(geography, dict) else ""
)
images = row.get("images") or []
image = next(
(
candidate
for candidate in images
if str(candidate.get("imageUrl") or "").strip()
and not bool(candidate.get("blocked"))
),
None,
)
if not (site_id and image and lat is not None and lon is not None):
continue
media_url = _ensure_https_url(
urljoin(base_url, str(image.get("imageUrl") or "").strip())
)
if not media_url:
continue
cameras.append(
{
"id": f"{id_prefix}-{site_id}",
"source_agency": source_agency,
"lat": lat,
"lon": lon,
"direction_facing": str(location)[:120],
"media_url": media_url,
"media_type": "image",
"refresh_rate_seconds": 60,
}
)
start += len(rows)
draw += 1
total = int(data.get("recordsTotal") or 0)
if total and start >= total:
break
if not total and len(rows) < page_size:
break
return cameras
def init_db():
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(str(DB_PATH))
@@ -169,7 +342,7 @@ class BaseCCTVIngestor(ABC):
cam.get("lat"),
cam.get("lon"),
cam.get("direction_facing", "Unknown"),
cam.get("media_url"),
_ensure_https_url(cam.get("media_url", "")),
cam.get("media_type", _detect_media_type(cam.get("media_url", ""))),
cam.get("refresh_rate_seconds", 60),
),
@@ -454,77 +627,14 @@ class WSDOTIngestor(BaseCCTVIngestor):
class GeorgiaDOTIngestor(BaseCCTVIngestor):
"""Georgia cameras via the public 511GA list feed."""
URL = "https://511ga.org/List/GetData/Cameras"
BASE_URL = "https://511ga.org"
PAGE_SIZE = 500
def fetch_data(self) -> List[Dict[str, Any]]:
cameras = []
start = 0
draw = 1
while True:
resp = fetch_with_curl(
self.URL,
method="POST",
json_data={"draw": draw, "start": start, "length": self.PAGE_SIZE},
timeout=30,
headers={
"Accept": "application/json",
"Referer": "https://511ga.org/cctv",
"Origin": "https://511ga.org",
},
)
if not resp or resp.status_code != 200:
logger.error(
"Georgia CCTV fetch failed: HTTP %s",
resp.status_code if resp else "no response",
)
break
data = resp.json()
rows = data.get("data") or []
if not rows:
break
for row in rows:
site_id = row.get("id") or row.get("DT_RowId")
location = row.get("location") or row.get("roadway") or "GA Camera"
lat_lng = row.get("latLng") or {}
geography = lat_lng.get("geography") if isinstance(lat_lng, dict) else {}
lat, lon = _parse_wkt_point(geography.get("wellKnownText") if isinstance(geography, dict) else "")
images = row.get("images") or []
image = next(
(
candidate
for candidate in images
if str(candidate.get("imageUrl") or "").strip()
and not bool(candidate.get("blocked"))
),
None,
)
if not (site_id and image and lat is not None and lon is not None):
continue
media_url = _normalize_cctv_media_url(
urljoin(self.BASE_URL, str(image.get("imageUrl") or "").strip())
)
cameras.append(
{
"id": f"GDOT-{site_id}",
"source_agency": "Georgia DOT",
"lat": lat,
"lon": lon,
"direction_facing": str(location)[:120],
"media_url": media_url,
"media_type": "image",
"refresh_rate_seconds": 60,
}
)
start += len(rows)
draw += 1
total = int(data.get("recordsTotal") or 0)
if total and start >= total:
break
if not total and len(rows) < self.PAGE_SIZE:
break
return cameras
return _fetch_511_datatables_cameras(
list_url="https://511ga.org/List/GetData/Cameras",
base_url="https://511ga.org",
id_prefix="GDOT",
source_agency="Georgia DOT",
referer="https://511ga.org/cctv",
)
class IllinoisDOTIngestor(BaseCCTVIngestor):
@@ -1009,30 +1119,66 @@ def _extract_img_src(html_fragment: str):
return None
class AsfinagIngestor(BaseCCTVIngestor):
"""Austria ASFINAG motorway webcams (Osiris port)."""
API_URL = "https://odo.asfinag.at/odo/rest/sec/resource/001/json/webcams?language=atDE"
HEADERS = {
"User-Agent": "Shadowbroker-CCTV/1.0",
"Accept": "application/json",
"Referer": "https://www.asfinag.at/",
"Authorization": "Basic bWFwX3dpZGdldDp0ZWdkaXc=",
}
def fetch_data(self) -> List[Dict[str, Any]]:
try:
response = fetch_with_curl(self.API_URL, timeout=15, headers=self.HEADERS)
response.raise_for_status()
payload = response.json()
except Exception as exc:
logger.error("AsfinagIngestor: fetch failed: %s", exc)
return []
if not isinstance(payload, list):
return []
cameras: List[Dict[str, Any]] = []
for cam in payload:
cam_id = cam.get("wcs_id")
lat = cam.get("wgs84_lat")
lon = cam.get("wgs84_lon")
image_url = cam.get("url_campic")
if not cam_id or lat is None or lon is None or not image_url:
continue
if str(cam_id).startswith("Utinform"):
continue
label = cam.get("position_txt") or cam.get("direction_txt") or "ASFINAG Webcam"
secure_url = _ensure_https_url(image_url)
if not secure_url:
continue
cameras.append(
{
"id": f"ASFINAG-{cam_id}",
"source_agency": "ASFINAG Austria",
"lat": float(lat),
"lon": float(lon),
"direction_facing": label,
"media_url": secure_url,
"media_type": "image",
"refresh_rate_seconds": 300,
}
)
logger.info("AsfinagIngestor: parsed %s cameras", len(cameras))
return cameras
class MadridCityIngestor(BaseCCTVIngestor):
"""Madrid City Hall traffic cameras from datos.madrid.es KML feed."""
KML_URL_HTTPS = "https://datos.madrid.es/egob/catalogo/202088-0-trafico-camaras.kml"
KML_URL_HTTP = "http://datos.madrid.es/egob/catalogo/202088-0-trafico-camaras.kml"
KML_URL = "https://datos.madrid.es/egob/catalogo/202088-0-trafico-camaras.kml"
def _fetch_kml(self):
"""Prefer HTTPS; fall back to legacy HTTP if the catalog is HTTP-only (#363)."""
last_error: Exception | None = None
for url in (self.KML_URL_HTTPS, self.KML_URL_HTTP):
try:
response = fetch_with_curl(url, timeout=20)
response.raise_for_status()
if url == self.KML_URL_HTTP:
logger.warning(
"MadridCityIngestor: HTTPS KML unavailable, using HTTP catalog feed"
)
return response
except Exception as e:
last_error = e
logger.debug("MadridCityIngestor: KML fetch failed for %s: %s", url, e)
if last_error is not None:
raise last_error
raise RuntimeError("Madrid KML fetch failed")
response = fetch_with_curl(self.KML_URL, timeout=20)
response.raise_for_status()
return response
def fetch_data(self) -> List[Dict[str, Any]]:
import defusedxml.ElementTree as ET
@@ -1074,6 +1220,9 @@ class MadridCityIngestor(BaseCCTVIngestor):
if desc_el is not None and desc_el.text:
image_url = _extract_img_src(desc_el.text)
if not image_url:
continue
image_url = _ensure_https_url(image_url)
if not image_url:
continue
@@ -1095,6 +1244,153 @@ class MadridCityIngestor(BaseCCTVIngestor):
return cameras
class Ontario511Ingestor(BaseCCTVIngestor):
"""Ontario highway cameras via 511on.ca TravelIQ API."""
def fetch_data(self) -> List[Dict[str, Any]]:
return _fetch_traveliq_v2_cameras(
api_url="https://511on.ca/api/v2/get/cameras",
base_url="https://511on.ca",
id_prefix="ON511",
source_agency="511 Ontario",
)
class Alberta511Ingestor(BaseCCTVIngestor):
"""Alberta highway cameras via 511 Alberta TravelIQ API."""
def fetch_data(self) -> List[Dict[str, Any]]:
return _fetch_traveliq_v2_cameras(
api_url="https://511.alberta.ca/api/v2/get/cameras",
base_url="https://511.alberta.ca",
id_prefix="AB511",
source_agency="511 Alberta",
)
class Florida511Ingestor(BaseCCTVIngestor):
"""Florida cameras via FL511 DataTables feed (~4,800 sites)."""
def fetch_data(self) -> List[Dict[str, Any]]:
return _fetch_511_datatables_cameras(
list_url="https://fl511.com/List/GetData/Cameras",
base_url="https://fl511.com",
id_prefix="FL511",
source_agency="Florida 511",
referer="https://fl511.com/",
)
class AustraliaLiveTrafficIngestor(BaseCCTVIngestor):
"""NSW / Australia live traffic cameras via Transport for NSW JSON feed."""
URL = "https://www.livetraffic.com/datajson/all-feeds-web.json"
def fetch_data(self) -> List[Dict[str, Any]]:
resp = fetch_with_curl(self.URL, timeout=35, headers={"Accept": "application/json"})
if not resp or resp.status_code != 200:
logger.error(
"Australia Live Traffic CCTV fetch failed: HTTP %s",
resp.status_code if resp else "no response",
)
return []
data = resp.json()
if not isinstance(data, list):
return []
cameras: List[Dict[str, Any]] = []
for item in data:
if not isinstance(item, dict) or item.get("eventType") != "liveCams":
continue
geometry = item.get("geometry") if isinstance(item.get("geometry"), dict) else {}
coords = geometry.get("coordinates") if isinstance(geometry.get("coordinates"), list) else []
if len(coords) < 2:
continue
try:
lon = float(coords[0])
lat = float(coords[1])
except (TypeError, ValueError):
continue
props = item.get("properties") if isinstance(item.get("properties"), dict) else {}
media_url = _ensure_https_url(str(props.get("href") or "").strip())
if not media_url:
continue
cam_id = str(item.get("path") or props.get("id") or len(cameras)).strip("/")
label = str(props.get("title") or props.get("headline") or "Australia Camera")[:120]
cameras.append(
{
"id": f"AUS-{cam_id}",
"source_agency": "NSW Live Traffic",
"lat": lat,
"lon": lon,
"direction_facing": label,
"media_url": media_url,
"media_type": "image",
"refresh_rate_seconds": 120,
}
)
logger.info("AustraliaLiveTrafficIngestor: parsed %s cameras", len(cameras))
return cameras
class NetherlandsRWSIngestor(BaseCCTVIngestor):
"""Netherlands Rijkswaterstaat cameras from legacy NDW open-data JSON.
The opendata.ndw.nu/cameras.json feed Osiris used is often offline; when
unavailable this ingestor returns an empty set and logs a warning.
"""
URL = "https://opendata.ndw.nu/cameras.json"
MAX_CAMERAS = 1200
def fetch_data(self) -> List[Dict[str, Any]]:
resp = fetch_with_curl(self.URL, timeout=25, headers={"Accept": "application/json"})
if not resp or resp.status_code != 200:
logger.warning(
"Netherlands RWS cameras.json unavailable (HTTP %s) — "
"NDW retired this open-data endpoint; no cameras ingested",
resp.status_code if resp else "no response",
)
return []
data = resp.json()
if not isinstance(data, list):
return []
cameras: List[Dict[str, Any]] = []
for i, cam in enumerate(data[: self.MAX_CAMERAS]):
if not isinstance(cam, dict):
continue
lat = cam.get("lat") if cam.get("lat") is not None else cam.get("latitude")
lon = cam.get("lng") if cam.get("lng") is not None else cam.get("longitude")
media_url = _ensure_https_url(
str(cam.get("imageUrl") or cam.get("feed_url") or cam.get("url") or "").strip()
)
if lat is None or lon is None or not media_url:
continue
try:
lat_f, lon_f = float(lat), float(lon)
except (TypeError, ValueError):
continue
cameras.append(
{
"id": f"NLRWS-{cam.get('id') or i}",
"source_agency": "Rijkswaterstaat",
"lat": lat_f,
"lon": lon_f,
"direction_facing": str(cam.get("name") or "Netherlands Camera")[:120],
"media_url": media_url,
"media_type": "image",
"refresh_rate_seconds": 120,
}
)
logger.info("NetherlandsRWSIngestor: parsed %s cameras", len(cameras))
return cameras
def _detect_media_type(url: str) -> str:
"""Detect the media type from a camera URL for proper frontend rendering."""
if not url:
@@ -1113,29 +1409,40 @@ def _detect_media_type(url: str) -> str:
return "image"
def scheduled_cctv_ingestors() -> List[tuple["BaseCCTVIngestor", str]]:
"""Canonical list of CCTV ingestors for startup, scheduler, and DB seeding."""
return [
(TFLJamCamIngestor(), "cctv_tfl"),
(LTASingaporeIngestor(), "cctv_lta"),
(AustinTXIngestor(), "cctv_atx"),
(NYCDOTIngestor(), "cctv_nyc"),
(CaltransIngestor(), "cctv_caltrans"),
(ColoradoDOTIngestor(), "cctv_codot"),
(WSDOTIngestor(), "cctv_wsdot"),
(GeorgiaDOTIngestor(), "cctv_gdot"),
(IllinoisDOTIngestor(), "cctv_idot"),
(MichiganDOTIngestor(), "cctv_mdot"),
(WindyWebcamsIngestor(), "cctv_windy"),
(DGTNationalIngestor(), "cctv_dgt"),
(MadridCityIngestor(), "cctv_madrid"),
(OSMTrafficCameraIngestor(), "cctv_osm"),
(AsfinagIngestor(), "cctv_asfinag"),
(OSMALPRCameraIngestor(), "cctv_osm_alpr"),
(Ontario511Ingestor(), "cctv_on511"),
(Alberta511Ingestor(), "cctv_ab511"),
(Florida511Ingestor(), "cctv_fl511"),
(AustraliaLiveTrafficIngestor(), "cctv_australia"),
(NetherlandsRWSIngestor(), "cctv_nl_rws"),
]
def run_all_ingestors():
"""Run all CCTV ingestors synchronously. Used for first-run DB seeding."""
ingestors = [
TFLJamCamIngestor(),
LTASingaporeIngestor(),
AustinTXIngestor(),
NYCDOTIngestor(),
CaltransIngestor(),
ColoradoDOTIngestor(),
WSDOTIngestor(),
GeorgiaDOTIngestor(),
IllinoisDOTIngestor(),
MichiganDOTIngestor(),
WindyWebcamsIngestor(),
OSMTrafficCameraIngestor(),
DGTNationalIngestor(),
MadridCityIngestor(),
]
for ing in ingestors:
for ingestor, _name in scheduled_cctv_ingestors():
try:
ing.ingest()
ingestor.ingest()
except Exception as e:
logger.warning(f"Ingestor {ing.__class__.__name__} failed during seed: {e}")
logger.warning(f"Ingestor {ingestor.__class__.__name__} failed during seed: {e}")
def get_all_cameras() -> List[Dict[str, Any]]:
backend/services/data_fetcher.py
+35 -48
View File
@@ -101,6 +101,10 @@ from services.fetchers.crowdthreat import fetch_crowdthreat # noqa: F401
from services.fetchers.wastewater import fetch_wastewater # noqa: F401
from services.fetchers.sar_catalog import fetch_sar_catalog # noqa: F401
from services.fetchers.sar_products import fetch_sar_products # noqa: F401
from services.fetchers.malware import fetch_malware_threats # noqa: F401
from services.fetchers.telegram_osint import fetch_telegram_osint # noqa: F401
from services.fetchers.cyber_status import fetch_cyber_threats # noqa: F401
from services.scm.suppliers import fetch_scm_suppliers # noqa: F401
from services.ais_stream import prune_stale_vessels # noqa: F401
logger = logging.getLogger(__name__)
@@ -480,6 +484,9 @@ def update_slow_data():
fetch_fishing_activity,
fetch_power_plants,
fetch_ukraine_air_raid_alerts,
fetch_malware_threats,
fetch_cyber_threats,
fetch_scm_suppliers,
]
_run_tasks("slow-tier", slow_funcs)
# Run correlation engine after all data is fresh
@@ -523,6 +530,15 @@ def _load_cctv_cache_for_startup() -> None:
logger.warning("Startup CCTV cache load failed (non-fatal): %s", e)
def _load_static_infrastructure_for_startup() -> None:
"""Disk-backed reference layers — instant, no network."""
for func in (fetch_datacenters, fetch_military_bases, fetch_power_plants):
try:
func()
except Exception as e:
logger.warning("Startup static infrastructure load failed for %s: %s", func.__name__, e)
def _run_delayed_startup_heavy_refresh() -> None:
if _STARTUP_HEAVY_REFRESH_DELAY_S > 0:
logger.info(
@@ -535,6 +551,7 @@ def _run_delayed_startup_heavy_refresh() -> None:
"startup-heavy",
[
update_slow_data,
fetch_telegram_osint,
fetch_volcanoes,
fetch_viirs_change_nodes,
fetch_unusual_whales,
@@ -573,6 +590,7 @@ def update_all_data(*, startup_mode: bool = False):
logger.info("Full data update starting (parallel)...")
# Preload Meshtastic map cache immediately (instant, from disk)
seed_startup_caches()
_load_static_infrastructure_for_startup()
with _data_lock:
meshtastic_seeded = bool(latest_data.get("meshtastic_map_nodes"))
if startup_mode:
@@ -649,22 +667,9 @@ def update_all_data(*, startup_mode: bool = False):
# (the scheduled job also runs every 10 min for ongoing refresh).
if startup_mode:
try:
from services.cctv_pipeline import (
TFLJamCamIngestor, LTASingaporeIngestor, AustinTXIngestor,
NYCDOTIngestor, CaltransIngestor, ColoradoDOTIngestor,
WSDOTIngestor, GeorgiaDOTIngestor, IllinoisDOTIngestor,
MichiganDOTIngestor, WindyWebcamsIngestor, DGTNationalIngestor,
MadridCityIngestor, OSMTrafficCameraIngestor, get_all_cameras,
)
from services.cctv_pipeline import OSMALPRCameraIngestor
_startup_ingestors = [
TFLJamCamIngestor(), LTASingaporeIngestor(), AustinTXIngestor(),
NYCDOTIngestor(), CaltransIngestor(), ColoradoDOTIngestor(),
WSDOTIngestor(), GeorgiaDOTIngestor(), IllinoisDOTIngestor(),
MichiganDOTIngestor(), WindyWebcamsIngestor(), DGTNationalIngestor(),
MadridCityIngestor(), OSMTrafficCameraIngestor(),
OSMALPRCameraIngestor(),
]
from services.cctv_pipeline import get_all_cameras, scheduled_cctv_ingestors
_startup_ingestors = [ing for ing, _name in scheduled_cctv_ingestors()]
logger.info("Running CCTV ingest at startup (%d ingestors)...", len(_startup_ingestors))
ingest_futures = {
_SHARED_EXECUTOR.submit(ing.ingest): ing.__class__.__name__
@@ -800,6 +805,18 @@ def start_scheduler():
misfire_grace_time=120,
)
# Telegram OSINT — hourly t.me/s channel scrape (kept off the 5-minute slow tier).
_telegram_interval_m = max(15, int(os.environ.get("TELEGRAM_OSINT_INTERVAL_MINUTES", "60")))
_scheduler.add_job(
lambda: _run_task_with_health(fetch_telegram_osint, "fetch_telegram_osint"),
"interval",
minutes=_telegram_interval_m,
next_run_time=datetime.utcnow() + timedelta(seconds=45),
id="telegram_osint",
max_instances=1,
misfire_grace_time=600,
)
# Prediction markets — own jittered cadence (Polymarket/Kalshi clearnet egress).
# Kept off the fixed 5-minute slow tier so poll timing is less fingerprintable.
from services.fetchers.prediction_markets import fetch_prediction_markets
@@ -938,39 +955,9 @@ def start_scheduler():
# CCTV pipeline refresh — runs all ingestors, then refreshes in-memory data.
# Delay the first run slightly so startup serves cached/DB-backed data first.
from services.cctv_pipeline import (
TFLJamCamIngestor,
LTASingaporeIngestor,
AustinTXIngestor,
NYCDOTIngestor,
CaltransIngestor,
ColoradoDOTIngestor,
WSDOTIngestor,
GeorgiaDOTIngestor,
IllinoisDOTIngestor,
MichiganDOTIngestor,
WindyWebcamsIngestor,
DGTNationalIngestor,
MadridCityIngestor,
OSMTrafficCameraIngestor,
)
from services.cctv_pipeline import scheduled_cctv_ingestors
_cctv_ingestors = [
(TFLJamCamIngestor(), "cctv_tfl"),
(LTASingaporeIngestor(), "cctv_lta"),
(AustinTXIngestor(), "cctv_atx"),
(NYCDOTIngestor(), "cctv_nyc"),
(CaltransIngestor(), "cctv_caltrans"),
(ColoradoDOTIngestor(), "cctv_codot"),
(WSDOTIngestor(), "cctv_wsdot"),
(GeorgiaDOTIngestor(), "cctv_gdot"),
(IllinoisDOTIngestor(), "cctv_idot"),
(MichiganDOTIngestor(), "cctv_mdot"),
(WindyWebcamsIngestor(), "cctv_windy"),
(DGTNationalIngestor(), "cctv_dgt"),
(MadridCityIngestor(), "cctv_madrid"),
(OSMTrafficCameraIngestor(), "cctv_osm"),
]
_cctv_ingestors = scheduled_cctv_ingestors()
def _run_cctv_ingest_cycle():
from services.fetchers._store import is_any_active
backend/services/env_check.py
+1
View File
@@ -46,6 +46,7 @@ _CRITICAL_WARN = {
_OPTIONAL = {
"AIS_API_KEY": "AIS vessel streaming (ships layer will be empty without it)",
"GFW_API_TOKEN": "Global Fishing Watch fishing-vessel activity (fishing_activity layer)",
"LTA_ACCOUNT_KEY": "Singapore LTA traffic cameras (CCTV layer)",
"PUBLIC_API_KEY": "Optional client auth for public endpoints (recommended for exposed deployments)",
}
backend/services/fetchers/_store.py
+13
View File
@@ -70,6 +70,10 @@ class DashboardData(TypedDict, total=False):
sar_anomalies: List[Dict[str, Any]]
sar_aoi_coverage: List[Dict[str, Any]]
road_corridor_trends: Dict[str, Any]
malware_threats: Dict[str, Any]
cyber_threats: Dict[str, Any]
scm_suppliers: Dict[str, Any]
telegram_osint: Dict[str, Any]
# In-memory store
@@ -121,6 +125,10 @@ latest_data: DashboardData = {
"sar_anomalies": [],
"sar_aoi_coverage": [],
"road_corridor_trends": {"updated_at": None, "corridors": []},
"malware_threats": {"threats": [], "total": 0, "timestamp": None},
"cyber_threats": {"threats": [], "stats": {}},
"scm_suppliers": {"suppliers": [], "total": 0, "critical_count": 0},
"telegram_osint": {"posts": [], "total": 0, "geolocated": 0, "timestamp": None},
}
# Per-source freshness timestamps
@@ -331,6 +339,11 @@ active_layers: dict[str, bool] = {
"crowdthreat": False,
"sar": True,
"road_corridor_trends": False,
"malware_c2": False,
"submarine_cables": False,
"scm_suppliers": False,
"cyber_threats": False,
"telegram_osint": True,
}
backend/services/fetchers/cyber_status.py
+62
View File
@@ -0,0 +1,62 @@
"""CISA KEV + cyber threat stats (Osiris port)."""
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import Any
from services.fetchers._store import _data_lock, _mark_fresh, is_any_active, latest_data
from services.network_utils import fetch_with_curl
logger = logging.getLogger(__name__)
def fetch_cyber_threats() -> dict[str, Any]:
if not is_any_active("cyber_threats"):
return latest_data.get("cyber_threats") or {"threats": [], "stats": {}}
results: dict[str, Any] = {"threats": [], "stats": {}, "timestamp": datetime.now(timezone.utc).isoformat()}
try:
resp = fetch_with_curl(
"https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json",
timeout=15,
)
if resp.status_code == 200:
data = resp.json()
vulns = data.get("vulnerabilities") or []
results["stats"]["cisa_total"] = len(vulns)
now = datetime.now(timezone.utc)
recent = []
for v in vulns:
try:
added = datetime.fromisoformat(v.get("dateAdded", "").replace("Z", "+00:00"))
days = (now - added).total_seconds() / 86400
except Exception:
continue
if days <= 30:
recent.append(v)
recent = recent[:10]
results["threats"] = [
{
"id": v.get("cveID"),
"name": v.get("vulnerabilityName"),
"vendor": v.get("vendorProject"),
"product": v.get("product"),
"severity": "CRITICAL",
"date": v.get("dateAdded"),
"due": v.get("dueDate"),
"source": "CISA KEV",
}
for v in recent
]
except Exception as exc:
logger.warning("CISA KEV fetch failed: %s", exc)
count = len(results["threats"])
results["stats"]["active_cves"] = count
results["stats"]["threat_level"] = "CRITICAL" if count >= 8 else "HIGH" if count >= 4 else "ELEVATED"
with _data_lock:
latest_data["cyber_threats"] = results
_mark_fresh("cyber_threats")
return results
backend/services/fetchers/geo.py
+34 -3
View File
@@ -278,6 +278,16 @@ _FISHING_FETCH_INTERVAL_S = 3600 # once per hour — GFW data has ~5 day lag
_last_fishing_fetch_ts: float = 0.0
def _gfw_int_env(name: str, default: int, *, minimum: int = 1, maximum: int | None = None) -> int:
try:
value = int(os.environ.get(name, str(default)) or default)
except (TypeError, ValueError):
value = default
if maximum is not None:
value = min(maximum, value)
return max(minimum, value)
@with_retry(max_retries=1, base_delay=5)
def fetch_fishing_activity():
"""Fetch recent fishing events from Global Fishing Watch (~5 day lag)."""
@@ -300,10 +310,16 @@ def fetch_fishing_activity():
try:
import datetime as _dt
# GFW publishes with ~5 day lag; windows shorter than ~7 days often return 0 events.
lookback_days = _gfw_int_env("GFW_EVENTS_LOOKBACK_DAYS", 7, minimum=1, maximum=14)
max_pages = _gfw_int_env("GFW_EVENTS_MAX_PAGES", 10, minimum=1, maximum=100)
timeout_s = _gfw_int_env("GFW_EVENTS_TIMEOUT_S", 90, minimum=30, maximum=180)
_end = _dt.date.today().isoformat()
_start = (_dt.date.today() - _dt.timedelta(days=7)).isoformat()
page_size = max(1, int(os.environ.get("GFW_EVENTS_PAGE_SIZE", "500") or "500"))
_start = (_dt.date.today() - _dt.timedelta(days=lookback_days)).isoformat()
page_size = _gfw_int_env("GFW_EVENTS_PAGE_SIZE", 500, minimum=1, maximum=1000)
offset = 0
pages_fetched = 0
total_available: int | None = None
seen_offsets: set[int] = set()
seen_ids: set[str] = set()
headers = {"Authorization": f"Bearer {token}"}
@@ -324,7 +340,7 @@ def fetch_fishing_activity():
}
)
url = f"https://gateway.api.globalfishingwatch.org/v3/events?{query}"
response = fetch_with_curl(url, timeout=30, headers=headers)
response = fetch_with_curl(url, timeout=timeout_s, headers=headers)
if response.status_code != 200:
logger.warning(
"Fishing activity fetch failed at offset=%s: HTTP %s",
@@ -334,10 +350,16 @@ def fetch_fishing_activity():
break
payload = response.json() or {}
if total_available is None:
try:
total_available = int(payload.get("total")) if payload.get("total") is not None else None
except (TypeError, ValueError):
total_available = None
entries = payload.get("entries", [])
if not entries:
break
pages_fetched += 1
added_this_page = 0
for e in entries:
pos = e.get("position", {})
@@ -372,6 +394,15 @@ def fetch_fishing_activity():
if len(entries) < page_size:
break
if pages_fetched >= max_pages:
logger.info(
"Fishing activity: capped at %s pages (%s events fetched; GFW total=%s)",
max_pages,
len(events),
total_available if total_available is not None else "unknown",
)
break
next_offset = payload.get("nextOffset")
if next_offset is None:
next_offset = (payload.get("pagination") or {}).get("nextOffset")
backend/services/fetchers/infrastructure.py
+4 -4
View File
@@ -235,11 +235,11 @@ _DC_GEOCODED_PATH = Path(__file__).parent.parent.parent / "data" / "datacenters_
def fetch_datacenters():
"""Load geocoded data centers (5K+ street-level precise locations)."""
from services.fetchers._store import is_any_active
"""Load geocoded data centers (5K+ street-level precise locations).
if not is_any_active("datacenters"):
return
Always loads from disk; /api/live-data/slow gates the payload on the
datacenters layer toggle so enabling the layer can render immediately.
"""
dcs = []
try:
if not _DC_GEOCODED_PATH.exists():
backend/services/fetchers/malware.py
+107
View File
@@ -0,0 +1,107 @@
"""Malware C2 / URLhaus feed (abuse.ch, Osiris port)."""
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import Any
from services.fetchers._store import _data_lock, _mark_fresh, is_any_active, latest_data
from services.network_utils import fetch_with_curl
logger = logging.getLogger(__name__)
COUNTRY_CENTROIDS: dict[str, tuple[float, float]] = {
"AF": (65, 33), "AL": (20, 41), "DZ": (3, 28), "AR": (-64, -34), "AU": (134, -25),
"AT": (14, 47.5), "BE": (4, 50.8), "BR": (-51, -10), "CA": (-96, 62), "CN": (105, 35),
"DE": (10, 51), "FR": (2, 46), "GB": (-2, 54), "IN": (79, 22), "IR": (53, 32),
"IT": (12.5, 42.8), "JP": (138, 36), "KR": (128, 36), "MX": (-102, 23.5), "NL": (5.5, 52.5),
"PL": (19.5, 52), "RU": (100, 60), "SG": (103.8, 1.35), "TW": (121, 23.7), "UA": (32, 49),
"US": (-97, 38), "VN": (106, 16),
}
def fetch_malware_threats() -> list[dict[str, Any]]:
if not is_any_active("malware_c2"):
return latest_data.get("malware_threats") or []
threats: list[dict[str, Any]] = []
threat_id = 0
try:
resp = fetch_with_curl(
"https://feodotracker.abuse.ch/downloads/ipblocklist.json",
timeout=10,
headers={"User-Agent": "Shadowbroker/1.0", "Accept": "application/json"},
)
if resp.status_code == 200:
entries = resp.json()
if not isinstance(entries, list):
entries = []
for entry in entries[:200]:
cc = entry.get("country")
if not cc or cc not in COUNTRY_CENTROIDS:
continue
lng, lat = COUNTRY_CENTROIDS[cc]
j_lng = ((threat_id * 173.7) % 200 - 100) / 100 * 4
j_lat = ((threat_id * 293.1) % 200 - 100) / 100 * 4
threats.append(
{
"id": f"feodo-{threat_id}",
"lat": lat + j_lat,
"lng": lng + j_lng,
"ip": entry.get("ip_address") or "unknown",
"port": entry.get("dst_port") or 0,
"malware": entry.get("malware") or "unknown",
"status": entry.get("status") or "active",
"first_seen": entry.get("first_seen"),
"last_online": entry.get("last_online"),
"country": cc,
"threat_type": "botnet_c2",
}
)
threat_id += 1
except Exception as exc:
logger.warning("Feodo fetch failed: %s", exc)
try:
resp = fetch_with_curl(
"https://urlhaus-api.abuse.ch/v1/urls/recent/limit/100/",
timeout=8,
)
if resp.status_code == 200:
urls = (resp.json() or {}).get("urls") or []
for u in urls:
cc = u.get("country")
if not cc or cc not in COUNTRY_CENTROIDS:
cc = next(iter(COUNTRY_CENTROIDS))
lng, lat = COUNTRY_CENTROIDS[cc]
j_lng = ((threat_id * 137.3) % 200 - 100) / 100 * 5
j_lat = ((threat_id * 211.7) % 200 - 100) / 100 * 5
threats.append(
{
"id": f"urlhaus-{threat_id}",
"lat": lat + j_lat,
"lng": lng + j_lng,
"ip": u.get("host") or "unknown",
"port": 0,
"malware": ", ".join(u.get("tags") or []) or u.get("threat") or "malware",
"status": u.get("url_status") or "online",
"first_seen": u.get("dateadded"),
"country": cc,
"threat_type": "malware_url",
}
)
threat_id += 1
except Exception as exc:
logger.debug("URLhaus supplement failed: %s", exc)
payload = {
"threats": threats,
"total": len(threats),
"timestamp": datetime.now(timezone.utc).isoformat(),
"source": "abuse.ch Feodo Tracker + URLhaus",
}
with _data_lock:
latest_data["malware_threats"] = payload
_mark_fresh("malware_threats")
return threats
backend/services/fetchers/news.py
+14 -9
View File
@@ -158,21 +158,26 @@ _KEYWORD_COORDS = {
_SORTED_KEYWORDS = sorted(_KEYWORD_COORDS.items(), key=lambda x: len(x[0]), reverse=True)
def resolve_coords_match(text: str) -> tuple[tuple[float, float], str] | None:
"""Return ((lat, lng), matched_keyword) for the most specific keyword hit."""
padded_text = f" {text} "
for kw, coords in _SORTED_KEYWORDS:
if kw.startswith(" ") or kw.endswith(" "):
if kw in padded_text:
return coords, kw
elif re.search(r"\b" + re.escape(kw) + r"\b", text):
return coords, kw
return None
def _resolve_coords(text: str) -> tuple[float, float] | None:
"""Return (lat, lng) for the most specific keyword match, or None.
Longer keywords are tried first. Space-padded keywords (" us ", " uk ")
use substring matching on padded text; all others use word-boundary regex.
"""
padded_text = f" {text} "
for kw, coords in _SORTED_KEYWORDS:
if kw.startswith(" ") or kw.endswith(" "):
if kw in padded_text:
return coords
else:
if re.search(r'\b' + re.escape(kw) + r'\b', text):
return coords
return None
match = resolve_coords_match(text)
return match[0] if match else None
@with_retry(max_retries=1, base_delay=2)
backend/services/fetchers/telegram_osint.py
+381
View File
@@ -0,0 +1,381 @@
"""Telegram OSINT — public channel web previews (t.me/s) with keyword geoparsing."""
from __future__ import annotations
import hashlib
import logging
import os
import re
from datetime import datetime, timezone
from typing import Any
from services.fetchers._store import _data_lock, _mark_fresh, is_any_active, latest_data
from services.fetchers.news import resolve_coords_match
from services.network_utils import fetch_with_curl, outbound_user_agent
logger = logging.getLogger(__name__)
_DEFAULT_CHANNELS = (
"osintdefender",
"insiderpaper",
"aljazeeraenglish",
"nexta_live",
"war_monitor",
"OSINTtechnical",
"Liveuamap",
)
_MESSAGE_BLOCK_RE = re.compile(
r'<div class="tgme_widget_message_wrap js-widget_message_wrap"[\s\S]*?</div>\s*</div>\s*</div>',
re.IGNORECASE,
)
_TEXT_RE = re.compile(
r'<div class="tgme_widget_message_text[^>]*>([\s\S]*?)</div>',
re.IGNORECASE,
)
_DATE_RE = re.compile(
r'<a class="tgme_widget_message_date" href="(https://t\.me/[^"]+)".*?<time datetime="([^"]+)"',
re.IGNORECASE,
)
_HAS_VIDEO_RE = re.compile(
r'tgme_widget_message_video|js-message_video|<video\s',
re.IGNORECASE,
)
_HAS_PHOTO_RE = re.compile(r'tgme_widget_message_photo_wrap', re.IGNORECASE)
_VIDEO_SRC_RE = re.compile(r'<video[^>]+src="([^"]+)"', re.IGNORECASE)
_BG_IMAGE_RE = re.compile(r"background-image:url\('([^']+)'\)", re.IGNORECASE)
_TELEGRAM_MEDIA_HOST_SUFFIXES = (".telesco.pe", ".telegram-cdn.org")
# Cyrillic / Arabic aliases for war-reporting channels (merged after English resolver).
_EXTRA_PLACE_KEYWORDS: dict[str, tuple[float, float]] = {
"киев": (50.450, 30.523),
"київ": (50.450, 30.523),
"харьков": (49.993, 36.231),
"харків": (49.993, 36.231),
"одесса": (46.482, 30.724),
"одеса": (46.482, 30.724),
"донецк": (48.015, 37.803),
"донецьк": (48.015, 37.803),
"луганск": (48.574, 39.307),
"луганськ": (48.574, 39.307),
"москва": (55.755, 37.617),
"крым": (45.000, 34.000),
"крим": (45.000, 34.000),
"бахмут": (48.595, 38.000),
"запорожье": (47.838, 35.139),
"запоріжжя": (47.838, 35.139),
"غزة": (31.416, 34.333),
"دمشق": (33.513, 36.276),
"بيروت": (33.893, 35.501),
"tel aviv": (32.085, 34.781),
"תל אביב": (32.085, 34.781),
}
# Country-level news geocodes sit on national centroids that stack with threat alerts.
# Telegram uses major metro anchors so pins land on a different map cell than news.
_TELEGRAM_ANCHOR_OVERRIDES: dict[str, tuple[float, float]] = {
"israel": (32.085, 34.781), # Tel Aviv (news uses central Israel ~Jerusalem corridor)
"middle east": (32.085, 34.781),
"china": (39.904, 116.407), # Beijing (news uses country centroid)
"united states": (40.712, -74.006), # New York (news uses Washington DC)
"usa": (40.712, -74.006),
"us": (40.712, -74.006),
"america": (40.712, -74.006),
"uk": (51.507, -0.127), # London
"iran": (35.689, 51.389), # Tehran
"russia": (55.755, 37.617), # Moscow
"ukraine": (50.450, 30.523), # Kyiv
"france": (48.856, 2.352), # Paris
"germany": (52.520, 13.405), # Berlin
"lebanon": (34.433, 35.844), # Tripoli (news uses Beirut corridor)
}
_RISK_KEYWORDS = (
"war",
"missile",
"strike",
"attack",
"crisis",
"tension",
"military",
"conflict",
"defense",
"clash",
"nuclear",
"invasion",
"bomb",
"drone",
"weapon",
"sanctions",
"ceasefire",
"escalation",
"killed",
"destroyed",
"operation",
"casualty",
"frontline",
"threat",
"explosion",
"shelling",
)
def telegram_osint_enabled() -> bool:
return str(os.environ.get("TELEGRAM_OSINT_ENABLED", "true")).strip().lower() not in {
"0",
"false",
"no",
"off",
"",
}
def _configured_channels() -> list[str]:
raw = str(os.environ.get("TELEGRAM_OSINT_CHANNELS", "")).strip()
if raw:
return [part.strip().lstrip("@") for part in raw.split(",") if part.strip()]
return list(_DEFAULT_CHANNELS)
def telegram_media_host_allowed(hostname: str | None) -> bool:
host = str(hostname or "").strip().lower()
if not host:
return False
return any(host.endswith(suffix) for suffix in _TELEGRAM_MEDIA_HOST_SUFFIXES)
def _extract_media(block: str, link: str) -> dict[str, Any]:
has_video = bool(_HAS_VIDEO_RE.search(block))
has_photo = bool(_HAS_PHOTO_RE.search(block))
media_type: str | None = None
media_url: str | None = None
if has_video:
media_type = "video"
video_match = _VIDEO_SRC_RE.search(block)
if video_match:
media_url = video_match.group(1).strip()
elif has_photo:
media_type = "photo"
photo_match = _BG_IMAGE_RE.search(block)
if photo_match:
media_url = photo_match.group(1).strip()
embed_url: str | None = None
if media_type and link:
embed_url = f"{link}?embed=1"
return {
"media_type": media_type,
"media_url": media_url,
"embed_url": embed_url,
}
def _strip_html(text: str) -> str:
cleaned = re.sub(r"<br\s*/?>", "\n", text, flags=re.IGNORECASE)
cleaned = re.sub(r"<[^>]+>", "", cleaned)
return (
cleaned.replace("&quot;", '"')
.replace("&amp;", "&")
.replace("&lt;", "<")
.replace("&gt;", ">")
.strip()
)
def _score_risk(text: str) -> int:
lower = text.lower()
score = 1
for kw in _RISK_KEYWORDS:
if kw in lower:
score += 2
return min(10, score)
def _refresh_post_coords(post: dict[str, Any]) -> dict[str, Any]:
"""Re-apply geoparsing so stored posts pick up anchor updates."""
text = "\n".join(
str(part).strip()
for part in (post.get("title"), post.get("description"))
if part and str(part).strip()
)
if not text:
return post
coords = _resolve_telegram_coords(text)
if not coords:
return post
updated = dict(post)
updated["coords"] = [coords[0], coords[1]]
return updated
def _resolve_telegram_coords(text: str) -> tuple[float, float] | None:
lower = text.lower()
match = resolve_coords_match(lower)
if match:
_coords, keyword = match
anchor = _TELEGRAM_ANCHOR_OVERRIDES.get(keyword.strip().lower())
if anchor:
return anchor
return _coords
for keyword, coords in sorted(_EXTRA_PLACE_KEYWORDS.items(), key=lambda x: len(x[0]), reverse=True):
if keyword in lower:
return coords
return None
def _post_link(post: dict[str, Any]) -> str:
return str(post.get("link") or "").strip()
def _extract_new_channel_posts(
html: str,
channel: str,
known_links: set[str],
*,
bootstrap_limit: int = 12,
) -> list[dict[str, Any]]:
"""Return unseen posts from a channel page; stop once we hit a stored link."""
parsed = parse_telegram_channel_html(html, channel)
if not parsed:
return []
if not known_links:
return parsed[-bootstrap_limit:]
fresh: list[dict[str, Any]] = []
for post in reversed(parsed):
link = _post_link(post)
if not link:
continue
if link in known_links:
break
fresh.append(post)
fresh.reverse()
return fresh
def _merge_telegram_posts(
existing: list[dict[str, Any]],
incoming: list[dict[str, Any]],
*,
max_posts: int = 120,
) -> tuple[list[dict[str, Any]], int]:
known_links = {_post_link(post) for post in existing if _post_link(post)}
added = 0
for post in incoming:
link = _post_link(post)
if not link or link in known_links:
continue
known_links.add(link)
existing.append(post)
added += 1
existing.sort(key=lambda p: str(p.get("published") or ""), reverse=True)
return existing[:max_posts], added
def parse_telegram_channel_html(html: str, channel: str) -> list[dict[str, Any]]:
"""Parse public t.me/s channel preview HTML into post dicts."""
posts: list[dict[str, Any]] = []
for block in _MESSAGE_BLOCK_RE.findall(html or ""):
text_match = _TEXT_RE.search(block)
if not text_match:
continue
text = _strip_html(text_match.group(1))
if len(text) < 10:
continue
date_match = _DATE_RE.search(block)
link = date_match.group(1) if date_match else f"https://t.me/{channel}"
published = date_match.group(2) if date_match else datetime.now(timezone.utc).isoformat()
title = text.split("\n", 1)[0][:160]
risk_score = _score_risk(text)
coords = _resolve_telegram_coords(text)
post_id = hashlib.sha1(f"{link}|{published}".encode("utf-8")).hexdigest()[:16]
media = _extract_media(block, link)
posts.append(
{
"id": post_id,
"title": title,
"description": text[:1200],
"link": link,
"published": published,
"source": f"t.me/{channel}",
"channel": channel,
"risk_score": risk_score,
"coords": [coords[0], coords[1]] if coords else None,
**media,
}
)
return posts
def fetch_telegram_osint() -> dict[str, Any]:
if not is_any_active("telegram_osint"):
return latest_data.get("telegram_osint") or {"posts": [], "total": 0, "timestamp": None}
if not telegram_osint_enabled():
with _data_lock:
latest_data["telegram_osint"] = {"posts": [], "total": 0, "timestamp": None, "disabled": True}
_mark_fresh("telegram_osint")
return latest_data["telegram_osint"]
headers = {
"User-Agent": (
f"Mozilla/5.0 (compatible; {outbound_user_agent('telegram-osint')}) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
),
"Accept": "text/html,application/xhtml+xml",
}
with _data_lock:
prior = latest_data.get("telegram_osint") or {}
existing_posts = list(prior.get("posts") or [])
known_links = {_post_link(post) for post in existing_posts if _post_link(post)}
incoming: list[dict[str, Any]] = []
for channel in _configured_channels():
url = f"https://t.me/s/{channel}"
try:
resp = fetch_with_curl(url, timeout=15, headers=headers)
if not resp or resp.status_code != 200:
logger.warning(
"Telegram channel %s fetch failed: HTTP %s",
channel,
resp.status_code if resp else "no response",
)
continue
channel_new = _extract_new_channel_posts(resp.text, channel, known_links)
for post in channel_new:
link = _post_link(post)
if not link or link in known_links:
continue
known_links.add(link)
incoming.append(post)
except Exception as exc:
logger.warning("Telegram channel %s parse failed: %s", channel, exc)
merged_posts, added = _merge_telegram_posts(existing_posts, incoming)
merged_posts = [_refresh_post_coords(post) for post in merged_posts]
geolocated = sum(1 for p in merged_posts if p.get("coords"))
payload = {
"posts": merged_posts,
"total": len(merged_posts),
"geolocated": geolocated,
"timestamp": datetime.now(timezone.utc).isoformat(),
"channels": _configured_channels(),
"last_fetch_new": added,
}
with _data_lock:
latest_data["telegram_osint"] = payload
_mark_fresh("telegram_osint")
logger.info(
"Telegram OSINT: +%s new, %s retained (%s geolocated)",
added,
len(merged_posts),
geolocated,
)
return payload
backend/services/intel_feeds/__init__.py
View File
backend/services/intel_feeds/country_risk.py
+94
View File
@@ -0,0 +1,94 @@
"""Country risk index (static scores + USGS quake enrichment)."""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any
from zoneinfo import ZoneInfo
from services.network_utils import fetch_with_curl
RISK_FACTORS: dict[str, dict[str, Any]] = {
"UA": {"base": 85, "tags": ["active_conflict", "infrastructure_damage"]},
"RU": {"base": 72, "tags": ["sanctions", "military_mobilization"]},
"IL": {"base": 78, "tags": ["active_conflict", "regional_instability"]},
"PS": {"base": 90, "tags": ["active_conflict", "humanitarian_crisis"]},
"SY": {"base": 82, "tags": ["post_conflict", "infrastructure_damage"]},
"YE": {"base": 88, "tags": ["active_conflict", "humanitarian_crisis"]},
"MM": {"base": 76, "tags": ["civil_unrest", "military_junta"]},
"SD": {"base": 84, "tags": ["active_conflict", "humanitarian_crisis"]},
"AF": {"base": 80, "tags": ["post_conflict", "governance_collapse"]},
"KP": {"base": 70, "tags": ["nuclear_risk", "isolation"]},
"IR": {"base": 68, "tags": ["sanctions", "nuclear_program", "regional_proxy"]},
"CN": {"base": 35, "tags": ["strategic_competition", "taiwan_tensions"]},
"TW": {"base": 45, "tags": ["invasion_risk", "semiconductor_dependency"]},
"VE": {"base": 60, "tags": ["economic_collapse", "political_instability"]},
"HT": {"base": 85, "tags": ["gang_violence", "governance_collapse"]},
"LB": {"base": 65, "tags": ["economic_crisis", "political_deadlock"]},
"PK": {"base": 55, "tags": ["terrorism", "political_instability"]},
"SO": {"base": 82, "tags": ["terrorism", "state_fragility"]},
"LY": {"base": 72, "tags": ["divided_government", "militia_control"]},
"ET": {"base": 62, "tags": ["ethnic_tensions", "regional_conflicts"]},
}
EXCHANGES = [
{"name": "NYSE", "tz": "America/New_York", "open": 9.5, "close": 16, "country": "US"},
{"name": "NASDAQ", "tz": "America/New_York", "open": 9.5, "close": 16, "country": "US"},
{"name": "LSE", "tz": "Europe/London", "open": 8, "close": 16.5, "country": "GB"},
{"name": "TSE", "tz": "Asia/Tokyo", "open": 9, "close": 15, "country": "JP"},
{"name": "SSE", "tz": "Asia/Shanghai", "open": 9.5, "close": 15, "country": "CN"},
{"name": "HKEX", "tz": "Asia/Hong_Kong", "open": 9.5, "close": 16, "country": "HK"},
{"name": "FRA", "tz": "Europe/Berlin", "open": 8, "close": 20, "country": "DE"},
{"name": "TSX", "tz": "America/Toronto", "open": 9.5, "close": 16, "country": "CA"},
{"name": "MOEX", "tz": "Europe/Moscow", "open": 10, "close": 18.5, "country": "RU"},
]
def _exchange_open(ex: dict[str, Any]) -> bool:
try:
now = datetime.now(ZoneInfo(ex["tz"]))
if now.weekday() >= 5:
return False
decimal = now.hour + now.minute / 60
return ex["open"] <= decimal < ex["close"]
except Exception:
return False
def build_country_risk_payload() -> dict[str, Any]:
quake_risks: dict[str, float] = {}
try:
resp = fetch_with_curl(
"https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_day.geojson",
timeout=5,
)
if resp.status_code == 200:
for f in resp.json().get("features") or []:
place = (f.get("properties") or {}).get("place") or ""
mag = (f.get("properties") or {}).get("mag") or 0
for code in RISK_FACTORS:
if code.lower() in place.lower():
quake_risks[code] = quake_risks.get(code, 0) + mag
except Exception:
pass
countries = []
for code, data in RISK_FACTORS.items():
base = data["base"]
score = min(100, base + quake_risks.get(code, 0))
countries.append(
{
"code": code,
"risk_score": score,
"risk_level": "CRITICAL" if base >= 80 else "HIGH" if base >= 60 else "ELEVATED" if base >= 40 else "LOW",
"tags": data["tags"],
}
)
countries.sort(key=lambda c: c["risk_score"], reverse=True)
exchanges = [{"name": e["name"], "country": e["country"], "open": _exchange_open(e)} for e in EXCHANGES]
return {
"countries": countries,
"exchanges": exchanges,
"open_exchanges": sum(1 for e in exchanges if e["open"]),
"total_exchanges": len(exchanges),
"timestamp": datetime.now(timezone.utc).isoformat(),
}
backend/services/osint/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Operator-initiated OSINT lookups (server-side proxies)."""
backend/services/osint/lookups.py
+492
View File
@@ -0,0 +1,492 @@
"""Server-side OSINT lookups (Osiris port, HTTPS outbound only)."""
from __future__ import annotations
import ipaddress
import json
import logging
import re
import socket
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from typing import Any
from urllib.parse import quote
from services.network_utils import fetch_with_curl
from services.sanctions.ofac import match_exact, search_sanctions
from services.ssrf_guard import safe_get, validate_domain, validate_host
logger = logging.getLogger(__name__)
_IPV4_RE = re.compile(r"^(\d{1,3}\.){3}\d{1,3}$")
_IPV6_RE = re.compile(r"^[0-9a-fA-F:]+$")
_CVE_RE = re.compile(r"^CVE-\d{4}-\d{4,}$", re.I)
_ASN_RE = re.compile(r"^(AS)?\d+$", re.I)
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat()
def _json_get(url: str, *, timeout: float = 8.0, headers: dict[str, str] | None = None) -> Any:
resp = fetch_with_curl(url, timeout=timeout, headers=headers or {"Accept": "application/json"})
if resp.status_code != 200:
return None
try:
return resp.json()
except Exception:
return None
def _sanctions_hits(*values: str) -> list[dict[str, Any]] | None:
hits: list[dict[str, Any]] = []
seen: set[str] = set()
for value in values:
if not value or value in seen:
continue
seen.add(value)
entries = match_exact(value)
if entries:
hits.append({"matched_value": value, "entries": entries})
return hits or None
def lookup_ip(ip: str) -> dict[str, Any]:
if not _IPV4_RE.match(ip) and not _IPV6_RE.match(ip):
raise ValueError("Invalid IP format")
check = validate_host(ip.strip("[]"))
if not check.get("ok"):
raise ValueError(check.get("reason", "blocked IP"))
results: dict[str, Any] = {"ip": ip, "timestamp": _now_iso()}
fields = (
"status,message,continent,country,countryCode,region,regionName,city,zip,"
"lat,lon,timezone,isp,org,as,asname,mobile,proxy,hosting,query"
)
geo = _json_get(f"https://ip-api.com/json/{quote(ip)}?fields={fields}", timeout=5)
if isinstance(geo, dict) and geo.get("status") == "success":
results["geo"] = {
"country": geo.get("country"),
"country_code": geo.get("countryCode"),
"region": geo.get("regionName"),
"city": geo.get("city"),
"lat": geo.get("lat"),
"lon": geo.get("lon"),
"timezone": geo.get("timezone"),
"isp": geo.get("isp"),
"org": geo.get("org"),
"as_number": geo.get("as"),
"as_name": geo.get("asname"),
"is_mobile": geo.get("mobile"),
"is_proxy": geo.get("proxy"),
"is_hosting": geo.get("hosting"),
}
results["reputation"] = {
"is_proxy": bool(geo.get("proxy")),
"is_hosting": bool(geo.get("hosting")),
"is_mobile": bool(geo.get("mobile")),
"risk_level": "HIGH" if geo.get("proxy") else "MEDIUM" if geo.get("hosting") else "LOW",
}
sm = _sanctions_hits(geo.get("org") or "", geo.get("isp") or "", geo.get("asname") or "")
if sm:
results["sanctions_match"] = {"source": "OFAC SDN", "hits": sm}
return results
def lookup_dns(domain: str) -> dict[str, Any]:
if not validate_domain(domain):
raise ValueError("Invalid domain format")
results: dict[str, Any] = {"domain": domain, "records": {}, "timestamp": _now_iso()}
for rtype in ("A", "AAAA", "MX", "NS", "TXT", "CNAME", "SOA"):
data = _json_get(
f"https://dns.google/resolve?name={quote(domain)}&type={rtype}",
timeout=5,
)
answers = []
if isinstance(data, dict):
for ans in data.get("Answer") or []:
answers.append(
{
"name": ans.get("name"),
"type": ans.get("type"),
"ttl": ans.get("TTL"),
"data": ans.get("data"),
}
)
results["records"][rtype] = answers
a_records = results["records"].get("A") or []
mx_records = results["records"].get("MX") or []
ns_records = results["records"].get("NS") or []
results["summary"] = {
"ip_addresses": [r["data"] for r in a_records if r.get("data")],
"mail_servers": [r["data"] for r in mx_records if r.get("data")],
"nameservers": [r["data"] for r in ns_records if r.get("data")],
"total_records": sum(len(v) for v in results["records"].values()),
}
return results
def lookup_whois(domain: str) -> dict[str, Any]:
if not validate_domain(domain):
raise ValueError("Invalid domain format")
results: dict[str, Any] = {"domain": domain, "timestamp": _now_iso()}
rdap = _json_get(f"https://rdap.org/domain/{quote(domain)}", timeout=8)
if isinstance(rdap, dict):
entities = []
for ent in rdap.get("entities") or []:
vcard = ent.get("vcardArray")
name = org = None
if isinstance(vcard, list) and len(vcard) > 1:
for row in vcard[1]:
if row[0] == "fn":
name = row[3]
if row[0] == "org":
org = row[3]
if name or org:
entities.append({"handle": ent.get("handle"), "roles": ent.get("roles"), "name": name, "org": org})
events = [
{"action": e.get("eventAction"), "date": e.get("eventDate")}
for e in (rdap.get("events") or [])
]
results["rdap"] = {
"handle": rdap.get("handle"),
"name": rdap.get("ldhName"),
"status": rdap.get("status"),
"events": events,
"nameservers": [ns.get("ldhName") for ns in (rdap.get("nameservers") or [])],
"entities": entities,
}
results["registration"] = next((e["date"] for e in events if e["action"] == "registration"), None)
results["expiration"] = next((e["date"] for e in events if e["action"] == "expiration"), None)
results["last_changed"] = next((e["date"] for e in events if e["action"] == "last changed"), None)
sm = _sanctions_hits(*(e.get("name") or "" for e in entities), *(e.get("org") or "" for e in entities))
if sm:
results["sanctions_match"] = {"source": "OFAC SDN", "hits": sm}
try:
res = safe_get(f"https://{domain}", timeout=5, headers={"User-Agent": "Shadowbroker-OSINT/1.0"})
headers = {}
for h in (
"server",
"x-powered-by",
"x-frame-options",
"strict-transport-security",
"content-security-policy",
"x-content-type-options",
"x-xss-protection",
"referrer-policy",
"permissions-policy",
):
val = res.headers.get(h)
if val:
headers[h] = val
score = sum(
1
for k in (
"strict-transport-security",
"content-security-policy",
"x-frame-options",
"x-content-type-options",
"referrer-policy",
)
if k in headers
) + (2 if "strict-transport-security" in headers else 0) + (2 if "content-security-policy" in headers else 0)
results["http"] = {"status": res.status_code, "headers": headers, "final_url": res.url}
results["security_score"] = {
"score": score,
"max": 7,
"grade": "A" if score >= 5 else "B" if score >= 3 else "C" if score >= 1 else "F",
}
except Exception as exc:
logger.debug("WHOIS header probe failed for %s: %s", domain, exc)
return results
def lookup_certs(domain: str) -> dict[str, Any]:
if not validate_domain(domain):
raise ValueError("Invalid domain format")
resp = fetch_with_curl(
f"https://crt.sh/?q=%25.{quote(domain)}&output=json",
timeout=10,
headers={"User-Agent": "Shadowbroker-OSINT/1.0"},
)
if resp.status_code != 200:
return {"domain": domain, "certificates": [], "error": "crt.sh unavailable"}
try:
certs = resp.json()
except Exception:
certs = []
seen: set[str] = set()
subdomains: set[str] = set()
unique: list[dict[str, Any]] = []
for cert in (certs or [])[:200]:
key = f"{cert.get('common_name')}-{cert.get('serial_number')}"
if key in seen:
continue
seen.add(key)
for name in (cert.get("name_value") or "").split("\n"):
clean = name.strip().replace("*.", "")
if clean.endswith(domain):
subdomains.add(clean)
unique.append(
{
"id": cert.get("id"),
"issuer": cert.get("issuer_name"),
"common_name": cert.get("common_name"),
"not_before": cert.get("not_before"),
"not_after": cert.get("not_after"),
}
)
return {
"domain": domain,
"certificates": unique[:50],
"subdomains": sorted(subdomains)[:100],
"total_found": len(certs or []),
"timestamp": _now_iso(),
}
def lookup_threats(query: str | None = None) -> dict[str, Any]:
results: dict[str, Any] = {"timestamp": _now_iso()}
pulses = _json_get("https://otx.alienvault.com/api/v1/pulses/activity?limit=10", timeout=8)
if isinstance(pulses, dict):
results["pulses"] = [
{
"name": p.get("name"),
"description": (p.get("description") or "")[:200],
"created": p.get("created"),
"tags": (p.get("tags") or [])[:5],
"adversary": p.get("adversary"),
"indicators_count": p.get("indicator_count"),
}
for p in (pulses.get("results") or [])[:10]
]
if query:
if _IPV4_RE.match(query):
try:
tor_resp = fetch_with_curl("https://check.torproject.org/torbulkexitlist", timeout=5)
results["tor_exit_node"] = query in (tor_resp.text or "").splitlines() if tor_resp.status_code == 200 else None
except Exception:
results["tor_exit_node"] = None
otx = _json_get(f"https://otx.alienvault.com/api/v1/indicators/IPv4/{quote(query)}/general", timeout=5)
if isinstance(otx, dict):
results["otx"] = {
"reputation": otx.get("reputation"),
"pulse_count": (otx.get("pulse_info") or {}).get("count", 0),
"country": otx.get("country_name"),
"asn": otx.get("asn"),
}
elif validate_domain(query):
otx = _json_get(f"https://otx.alienvault.com/api/v1/indicators/domain/{quote(query)}/general", timeout=5)
if isinstance(otx, dict):
results["otx"] = {"pulse_count": (otx.get("pulse_info") or {}).get("count", 0)}
pulse_count = (results.get("otx") or {}).get("pulse_count", 0)
results["threat_level"] = "HIGH" if pulse_count > 5 else "MEDIUM" if pulse_count > 0 else "LOW"
return results
def lookup_bgp(query: str) -> dict[str, Any]:
results: dict[str, Any] = {"query": query, "timestamp": _now_iso()}
if _IPV4_RE.match(query):
data = _json_get(f"https://api.bgpview.io/ip/{quote(query)}", timeout=8)
if isinstance(data, dict) and data.get("status") == "ok":
results["ip"] = data.get("data")
results["type"] = "ip"
return results
if _ASN_RE.match(query):
asn_num = re.sub(r"^AS", "", query, flags=re.I)
asn = _json_get(f"https://api.bgpview.io/asn/{asn_num}", timeout=8)
prefixes = _json_get(f"https://api.bgpview.io/asn/{asn_num}/prefixes", timeout=8)
peers = _json_get(f"https://api.bgpview.io/asn/{asn_num}/peers", timeout=8)
if isinstance(asn, dict) and asn.get("status") == "ok":
results["asn"] = asn.get("data")
if isinstance(prefixes, dict) and prefixes.get("status") == "ok":
pdata = prefixes.get("data") or {}
results["prefixes"] = {
"ipv4": (pdata.get("ipv4_prefixes") or [])[:20],
"ipv6": (pdata.get("ipv6_prefixes") or [])[:10],
"total_v4": len(pdata.get("ipv4_prefixes") or []),
"total_v6": len(pdata.get("ipv6_prefixes") or []),
}
if isinstance(peers, dict) and peers.get("status") == "ok":
pdata = peers.get("data") or {}
results["peers"] = {
"upstream": (pdata.get("ipv4_peers") or [])[:10],
"total": len(pdata.get("ipv4_peers") or []),
}
results["type"] = "asn"
return results
raise ValueError("Unrecognized query format. Use IP address or AS number.")
def lookup_sanctions(query: str, *, schema: str | None = None, limit: int = 25) -> dict[str, Any]:
matches = search_sanctions(query, schema=schema, limit=limit)
return {
"query": query,
"schema": schema,
"total": len(matches),
"matches": matches,
"source": "OpenSanctions / US OFAC SDN",
"timestamp": _now_iso(),
}
def lookup_cve(cve: str) -> dict[str, Any]:
if not _CVE_RE.match(cve):
raise ValueError("Invalid CVE format")
cve_id = cve.upper()
data = _json_get(f"https://cveawg.mitre.org/api/cve/{quote(cve_id)}", timeout=8)
if isinstance(data, dict) and data.get("cveMetadata"):
meta = data["cveMetadata"]
desc = ""
for block in (data.get("containers") or {}).get("cna", {}).get("descriptions") or []:
if block.get("lang") == "en":
desc = block.get("value") or desc
return {"id": meta.get("cveId", cve_id), "description": desc or "No description.", "timestamp": _now_iso()}
fallback = _json_get(f"https://cve.circl.lu/api/cve/{quote(cve_id)}", timeout=8)
if isinstance(fallback, dict):
return {
"id": fallback.get("id", cve_id),
"description": fallback.get("summary") or "No description.",
"cvss": fallback.get("cvss"),
"references": (fallback.get("references") or [])[:5],
"timestamp": _now_iso(),
}
raise ValueError("CVE not found")
def lookup_mac(mac: str) -> dict[str, Any]:
clean = mac.strip().upper()
clean = re.sub(r"[^A-F0-9:-]", "", clean)
data = _json_get(f"https://api.macvendors.com/{quote(clean)}", timeout=8)
if isinstance(data, dict):
return {"mac": clean, "vendor": data.get("company") or data.get("organization") or "Not Found"}
if isinstance(data, str) and data:
return {"mac": clean, "vendor": data}
return {"mac": clean, "vendor": "Not Found"}
def lookup_github(username: str) -> dict[str, Any]:
user = _json_get(f"https://api.github.com/users/{quote(username)}", timeout=8)
if not isinstance(user, dict) or user.get("message") == "Not Found":
raise ValueError("GitHub user not found")
repos = _json_get(f"https://api.github.com/users/{quote(username)}/repos?per_page=10&sort=updated", timeout=8)
return {
"username": username,
"profile": {
"name": user.get("name"),
"bio": user.get("bio"),
"company": user.get("company"),
"location": user.get("location"),
"public_repos": user.get("public_repos"),
"followers": user.get("followers"),
"created_at": user.get("created_at"),
"html_url": user.get("html_url"),
},
"repos": [
{"name": r.get("name"), "language": r.get("language"), "stars": r.get("stargazers_count")}
for r in (repos or [])[:10]
if isinstance(r, dict)
],
"timestamp": _now_iso(),
}
def lookup_leaks(email: str) -> dict[str, Any]:
if "@" not in email or len(email) < 5:
raise ValueError("Invalid email")
# HIBP requires API key for v3; use public breach directory style via leak-lookup (rate limited)
data = _json_get(f"https://leakcheck.io/api/public?check={quote(email)}", timeout=8)
if isinstance(data, dict):
return {
"email": email,
"found": bool(data.get("found")),
"sources": data.get("sources") or [],
"timestamp": _now_iso(),
}
return {"email": email, "found": False, "sources": [], "timestamp": _now_iso()}
def sweep_init(ip: str, cidr: int = 24) -> dict[str, Any]:
try:
addr = ipaddress.IPv4Address(ip)
except ValueError as exc:
raise ValueError("Invalid IPv4 address format") from exc
if addr.is_private or addr.is_loopback or addr.is_link_local or addr.is_reserved:
raise ValueError("Private and reserved IP ranges are not allowed")
if cidr < 24 or cidr > 32:
raise ValueError("CIDR must be between 24 and 32")
fields = "status,message,country,countryCode,region,regionName,city,lat,lon,isp,org,as,proxy,hosting"
geo = _json_get(f"https://ip-api.com/json/{quote(ip)}?fields={fields}", timeout=5)
if not isinstance(geo, dict) or geo.get("status") != "success":
raise ValueError(f"Geolocation failed: {(geo or {}).get('message', 'unknown')}")
return {
"center": {
"lat": geo.get("lat"),
"lng": geo.get("lon"),
"city": geo.get("city"),
"region": geo.get("regionName"),
"country": geo.get("country"),
"countryCode": geo.get("countryCode"),
"isp": geo.get("isp"),
"asn": geo.get("as") or "",
"org": geo.get("org") or "",
},
"target_ip": ip,
"cidr": cidr,
}
def _internetdb_lookup(ip: str) -> dict[str, Any] | None:
try:
resp = fetch_with_curl(
f"https://internetdb.shodan.io/{quote(ip)}",
timeout=4,
headers={"Accept": "application/json"},
)
if resp.status_code == 404:
return None
if resp.status_code != 200:
return None
return resp.json()
except Exception:
return None
def sweep_scan(subnet_start: str, cidr: int, *, max_workers: int = 12) -> dict[str, Any]:
"""Scan a /24-/32 via Shodan InternetDB (server-side proxy)."""
base = int(ipaddress.IPv4Address(subnet_start))
host_count = 2 ** (32 - cidr)
if host_count > 256:
raise ValueError("Subnet too large")
ips = [str(ipaddress.IPv4Address(base + i)) for i in range(host_count)]
devices: list[dict[str, Any]] = []
t0 = time.time()
with ThreadPoolExecutor(max_workers=max_workers) as pool:
futures = {pool.submit(_internetdb_lookup, ip): ip for ip in ips}
for fut in as_completed(futures):
ip = futures[fut]
data = fut.result()
if not data:
continue
devices.append(
{
"ip": data.get("ip") or ip,
"ports": data.get("ports") or [],
"hostnames": data.get("hostnames") or [],
"cpes": data.get("cpes") or [],
"vulns": data.get("vulns") or [],
"tags": data.get("tags") or [],
}
)
return {
"devices": devices,
"summary": {"total_hosts": host_count, "total_responsive": len(devices)},
"sweep_time_ms": int((time.time() - t0) * 1000),
}
def subnet_start_for(ip: str, cidr: int) -> str:
net = ipaddress.IPv4Network(f"{ip}/{cidr}", strict=False)
return str(net.network_address)
backend/services/osint_intel/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Entity graph resolution (Osiris intel layer port)."""
backend/services/osint_intel/resolve.py
+268
View File
@@ -0,0 +1,268 @@
"""Entity graph resolver (Python port of Osiris intel/server.js)."""
from __future__ import annotations
import logging
import re
import threading
import time
from typing import Any
from urllib.parse import quote
from services.network_utils import fetch_with_curl
from services.sanctions.ofac import match_exact, search_sanctions
logger = logging.getLogger(__name__)
ALLOWED_TYPES = frozenset({"aircraft", "vessel", "company", "person", "ip", "country"})
_WD_CACHE: dict[str, tuple[float, dict[str, Any]]] = {}
_WD_LOCK = threading.Lock()
_WD_TTL = 24 * 60 * 60
_WD_UA = "Shadowbroker-Intel/1.0 (ontology engine)"
def _dedup(nodes: list[dict], links: list[dict]) -> dict[str, Any]:
node_map: dict[str, dict] = {}
for n in nodes:
node_map[n["id"]] = n
seen_links: set[str] = set()
out_links: list[dict] = []
for link in links:
key = f"{link['source']}→{link['target']}→{link['label']}"
if key in seen_links:
continue
seen_links.add(key)
out_links.append(link)
return {"nodes": list(node_map.values()), "links": out_links}
def _wd_cache_get(key: str) -> dict[str, Any] | None:
with _WD_LOCK:
entry = _WD_CACHE.get(key)
if not entry:
return None
ts, data = entry
if time.time() - ts > _WD_TTL:
_WD_CACHE.pop(key, None)
return None
return data
def _wd_cache_set(key: str, data: dict[str, Any]) -> None:
with _WD_LOCK:
if len(_WD_CACHE) > 5000:
oldest = next(iter(_WD_CACHE))
_WD_CACHE.pop(oldest, None)
_WD_CACHE[key] = (time.time(), data)
def _add_sanctions(id_label: str, root_id: str, nodes: list, links: list) -> None:
for hit in search_sanctions(id_label, limit=3):
sid = f"sanction:{hit['id']}"
nodes.append(
{
"id": sid,
"label": hit["name"],
"type": "sanction",
"properties": {"programs": hit.get("programs"), "source": "OFAC SDN"},
}
)
links.append({"source": root_id, "target": sid, "label": "SANCTIONS MATCH"})
def _sparql(query: str) -> list[dict[str, Any]]:
url = f"https://query.wikidata.org/sparql?query={quote(query)}&format=json"
resp = fetch_with_curl(url, timeout=10, headers={"User-Agent": _WD_UA, "Accept": "application/sparql-results+json"})
if resp.status_code != 200:
return []
try:
data = resp.json()
except Exception:
return []
return data.get("results", {}).get("bindings", [])
def _wd_search(label: str) -> str | None:
url = (
"https://www.wikidata.org/w/api.php?action=wbsearchentities"
f"&search={quote(label)}&language=en&limit=1&format=json"
)
resp = fetch_with_curl(url, timeout=5, headers={"User-Agent": _WD_UA})
if resp.status_code != 200:
return None
try:
hits = resp.json().get("search") or []
except Exception:
return None
return hits[0]["id"] if hits else None
def _resolve_ip(id_value: str) -> dict[str, Any]:
cache_key = f"ip:{id_value}"
cached = _wd_cache_get(cache_key)
if cached:
return cached
root_id = f"ip:{id_value}"
nodes: list[dict] = [{"id": root_id, "label": id_value, "type": "ip", "properties": {}}]
links: list[dict] = []
geo = fetch_with_curl(
f"https://ip-api.com/json/{quote(id_value)}"
"?fields=status,country,countryCode,city,lat,lon,isp,org,as,asname,proxy,hosting,mobile",
timeout=8,
)
if geo.status_code == 200:
try:
data = geo.json()
except Exception:
data = {}
if data.get("status") == "success":
nodes[0]["properties"] = {
"proxy": bool(data.get("proxy")),
"hosting": bool(data.get("hosting")),
"mobile": bool(data.get("mobile")),
"source": "ip-api.com",
}
if data.get("isp"):
iid = f"company:{data['isp']}"
nodes.append({"id": iid, "label": data["isp"], "type": "company", "properties": {"role": "ISP"}})
links.append({"source": root_id, "target": iid, "label": "HOSTED_BY"})
if data.get("country"):
cid = f"country:{data['country']}"
nodes.append(
{
"id": cid,
"label": data["country"],
"type": "country",
"properties": {"code": data.get("countryCode")},
}
)
links.append({"source": root_id, "target": cid, "label": "LOCATED_IN"})
for val in (data.get("isp"), data.get("org"), data.get("asname")):
if val:
for entry in match_exact(val):
sid = f"sanction:{entry['id']}"
nodes.append({"id": sid, "label": entry["name"], "type": "sanction", "properties": {}})
links.append({"source": root_id, "target": sid, "label": "SANCTIONS MATCH"})
whois = fetch_with_curl(
f"https://stat.ripe.net/data/whois/data.json?resource={quote(id_value)}",
timeout=8,
)
if whois.status_code == 200:
try:
records = whois.json().get("data", {}).get("records") or []
except Exception:
records = []
for record in records:
for field in record:
if field.get("key") in ("netname", "NetName"):
nid = f"company:{field['value']}"
nodes.append({"id": nid, "label": field["value"], "type": "company", "properties": {"role": "Network"}})
links.append({"source": root_id, "target": nid, "label": "HOSTED_BY"})
result = _dedup(nodes, links)
_wd_cache_set(cache_key, result)
return result
def _resolve_company(id_value: str) -> dict[str, Any]:
cache_key = f"company:{id_value}"
cached = _wd_cache_get(cache_key)
if cached:
return cached
root_id = f"company:{id_value}"
nodes = [{"id": root_id, "label": id_value, "type": "company", "properties": {}}]
links: list[dict] = []
safe = re.sub(r'[^a-zA-Z0-9 \-._]', '', id_value).strip()
qid = _wd_search(safe)
filt = f"VALUES ?item {{ wd:{qid} }}" if qid else f'?item rdfs:label "{safe}"@en . ?item wdt:P31/wdt:P279* wd:Q4830453 .'
rows = _sparql(
f"""
SELECT ?countryLabel ?parentLabel ?ceoLabel WHERE {{
{filt}
OPTIONAL {{ ?item wdt:P17 ?country . }}
OPTIONAL {{ ?item wdt:P749 ?parent . }}
OPTIONAL {{ ?item wdt:P169 ?ceo . }}
SERVICE wikibase:label {{ bd:serviceParam wikibase:language "en" . }}
}} LIMIT 10
"""
)
for row in rows:
if row.get("countryLabel", {}).get("value"):
cid = f"country:{row['countryLabel']['value']}"
nodes.append({"id": cid, "label": row["countryLabel"]["value"], "type": "country", "properties": {}})
links.append({"source": root_id, "target": cid, "label": "HEADQUARTERED"})
if row.get("parentLabel", {}).get("value"):
pid = f"company:{row['parentLabel']['value']}"
nodes.append({"id": pid, "label": row["parentLabel"]["value"], "type": "company", "properties": {}})
links.append({"source": root_id, "target": pid, "label": "PARENT ORG"})
if row.get("ceoLabel", {}).get("value"):
pid = f"person:{row['ceoLabel']['value']}"
nodes.append({"id": pid, "label": row["ceoLabel"]["value"], "type": "person", "properties": {"role": "CEO"}})
links.append({"source": root_id, "target": pid, "label": "CEO"})
_add_sanctions(id_value, root_id, nodes, links)
result = _dedup(nodes, links)
_wd_cache_set(cache_key, result)
return result
def _resolve_from_store(entity_type: str, id_value: str, props: dict[str, Any]) -> dict[str, Any]:
from services.fetchers._store import get_latest_data_subset_refs
root_id = f"{entity_type}:{id_value}"
nodes = [{"id": root_id, "label": props.get("label") or id_value, "type": entity_type, "properties": props}]
links: list[dict] = []
data = get_latest_data_subset_refs("flights", "ships", "military_flights", "tracked_flights")
if entity_type == "aircraft":
icao = (props.get("icao24") or id_value).lower()
for bucket in ("military_flights", "tracked_flights", "flights"):
for f in data.get(bucket) or []:
if str(f.get("icao24", "")).lower() == icao:
if f.get("country"):
cid = f"country:{f['country']}"
nodes.append({"id": cid, "label": f["country"], "type": "country", "properties": {}})
links.append({"source": root_id, "target": cid, "label": "REGISTERED_IN"})
if f.get("registration"):
nodes[0]["properties"]["registration"] = f["registration"]
break
elif entity_type == "vessel":
mmsi = str(props.get("mmsi") or id_value)
for ship in data.get("ships") or []:
if str(ship.get("mmsi")) == mmsi:
if ship.get("country"):
cid = f"country:{ship['country']}"
nodes.append({"id": cid, "label": ship["country"], "type": "country", "properties": {}})
links.append({"source": root_id, "target": cid, "label": "FLAG"})
break
_add_sanctions(id_value, root_id, nodes, links)
return _dedup(nodes, links)
def resolve_entity(entity_type: str, id_value: str, properties: dict[str, Any] | None = None) -> dict[str, Any]:
etype = (entity_type or "").lower().strip()
eid = (id_value or "").strip()
if etype not in ALLOWED_TYPES:
raise ValueError(f"Invalid type. Allowed: {', '.join(sorted(ALLOWED_TYPES))}")
if len(eid) < 2 or len(eid) > 200:
raise ValueError("Invalid id (2-200 chars)")
props = properties or {}
if etype == "ip":
return _resolve_ip(eid)
if etype in ("company", "person", "country"):
if etype == "company":
return _resolve_company(eid)
if etype == "person":
root_id = f"person:{eid}"
nodes = [{"id": root_id, "label": eid, "type": "person", "properties": {}}]
links: list[dict] = []
_add_sanctions(eid, root_id, nodes, links)
return _dedup(nodes, links)
root_id = f"country:{eid}"
nodes = [{"id": root_id, "label": eid, "type": "country", "properties": {}}]
links = []
_add_sanctions(eid, root_id, nodes, links)
return _dedup(nodes, links)
return _resolve_from_store(etype, eid, props)
backend/services/sanctions/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Sanctions screening (OpenSanctions OFAC SDN)."""
backend/services/sanctions/ofac.py
+154
View File
@@ -0,0 +1,154 @@
"""OFAC SDN index via OpenSanctions (adapted from Osiris sanctions.ts)."""
from __future__ import annotations
import csv
import io
import logging
import re
import threading
import time
from dataclasses import dataclass, field
from typing import Any
from services.network_utils import fetch_with_curl
logger = logging.getLogger(__name__)
SDN_CSV_URL = "https://data.opensanctions.org/datasets/latest/us_ofac_sdn/targets.simple.csv"
TTL_S = 24 * 60 * 60
_lock = threading.Lock()
_cache: dict[str, Any] | None = None
_cache_at: float = 0.0
_inflight: threading.Event | None = None
@dataclass
class SanctionEntry:
id: str
schema: str
name: str
aliases: list[str] = field(default_factory=list)
countries: list[str] = field(default_factory=list)
programs: list[str] = field(default_factory=list)
sanctions: str = ""
first_seen: str | None = None
last_seen: str | None = None
def to_dict(self) -> dict[str, Any]:
return {
"id": self.id,
"schema": self.schema,
"name": self.name,
"aliases": self.aliases,
"countries": self.countries,
"programs": self.programs,
"sanctions": self.sanctions,
"first_seen": self.first_seen,
"last_seen": self.last_seen,
}
def norm_name(s: str) -> str:
s = re.sub(r"[^\w\s]+", " ", s.lower(), flags=re.UNICODE)
return re.sub(r"\s+", " ", s).strip()
def _split_semi(val: str) -> list[str]:
return [x.strip() for x in (val or "").split(";") if x.strip()]
def _load_list() -> dict[str, Any]:
global _cache, _cache_at
with _lock:
if _cache and (time.time() - _cache_at) < TTL_S:
return _cache
try:
resp = fetch_with_curl(SDN_CSV_URL, timeout=45, headers={"Accept": "text/csv"})
if resp.status_code != 200:
raise RuntimeError(f"OpenSanctions HTTP {resp.status_code}")
text = resp.text
reader = csv.DictReader(io.StringIO(text))
entries: list[SanctionEntry] = []
by_norm: dict[str, list[SanctionEntry]] = {}
for row in reader:
name = (row.get("name") or "").strip()
if not name:
continue
entry = SanctionEntry(
id=row.get("id") or "",
schema=row.get("schema") or "LegalEntity",
name=name,
aliases=_split_semi(row.get("aliases") or ""),
countries=_split_semi(row.get("countries") or ""),
programs=_split_semi(row.get("program_ids") or ""),
sanctions=row.get("sanctions") or "",
first_seen=row.get("first_seen") or None,
last_seen=row.get("last_seen") or None,
)
entries.append(entry)
for key in {norm_name(name), *(norm_name(a) for a in entry.aliases)}:
if not key:
continue
by_norm.setdefault(key, []).append(entry)
loaded = {"entries": entries, "by_norm": by_norm, "fetched_at": time.time()}
with _lock:
_cache = loaded
_cache_at = time.time()
logger.info("OFAC SDN index loaded: %s entries", len(entries))
return loaded
except Exception as exc:
logger.error("OFAC SDN load failed: %s", exc)
with _lock:
if _cache:
return _cache
raise
def match_exact(query: str) -> list[dict[str, Any]]:
if not query or len(query) < 3:
return []
data = _load_list()
hits = data["by_norm"].get(norm_name(query), [])
return [e.to_dict() for e in hits]
def search_sanctions(query: str, *, schema: str | None = None, limit: int = 50) -> list[dict[str, Any]]:
if not query or len(query) < 4:
return []
data = _load_list()
q = norm_name(query)
exact_name: list[SanctionEntry] = []
exact_alias: list[SanctionEntry] = []
sub_name: list[SanctionEntry] = []
sub_alias: list[SanctionEntry] = []
seen: set[str] = set()
def push(bucket: list[SanctionEntry], entry: SanctionEntry) -> None:
if entry.id in seen:
return
if schema and entry.schema != schema:
return
seen.add(entry.id)
bucket.append(entry)
for entry in data["entries"]:
name_norm = norm_name(entry.name)
if name_norm == q:
push(exact_name, entry)
elif any(norm_name(a) == q for a in entry.aliases):
push(exact_alias, entry)
elif q in name_norm:
push(sub_name, entry)
elif any(q in norm_name(a) for a in entry.aliases):
push(sub_alias, entry)
if len(seen) >= limit * 4:
break
ordered = exact_name + exact_alias + sub_name + sub_alias
return [e.to_dict() for e in ordered[:limit]]
def index_size() -> int:
return len(_load_list()["entries"])
backend/services/scm/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Supply-chain risk overlay."""
backend/services/scm/suppliers.py
+154
View File
@@ -0,0 +1,154 @@
"""SCM supplier risk overlay (Osiris port, uses in-memory dashboard data)."""
from __future__ import annotations
import math
from datetime import datetime, timezone
from typing import Any
from services.fetchers._store import _data_lock, _mark_fresh, get_latest_data_subset_refs, is_any_active, latest_data
from services.network_utils import fetch_with_curl
SUPPLIERS: list[dict[str, Any]] = [
{"id": "sup-tsmc-hsinchu", "name": "TSMC Fab 12 (Tier 1)", "city": "Hsinchu", "country": "Taiwan", "lat": 24.774, "lng": 120.992, "category": "Semiconductor"},
{"id": "sup-tsmc-tainan", "name": "TSMC Fab 14 (Tier 1)", "city": "Tainan", "country": "Taiwan", "lat": 23.111, "lng": 120.273, "category": "Semiconductor"},
{"id": "sup-sec-giheung", "name": "Samsung Electronics (Tier 1)", "city": "Giheung", "country": "South Korea", "lat": 37.221, "lng": 127.098, "category": "Semiconductor"},
{"id": "sup-sk-icheon", "name": "SK Hynix (Tier 1)", "city": "Icheon", "country": "South Korea", "lat": 37.256, "lng": 127.483, "category": "Semiconductor"},
{"id": "sup-sony-kumamoto", "name": "Sony Semiconductor (Tier 2)", "city": "Kikuyo", "country": "Japan", "lat": 32.883, "lng": 130.825, "category": "Electronics"},
{"id": "sup-mlcc-murata", "name": "Murata MLCC (Tier 2)", "city": "Izumo", "country": "Japan", "lat": 35.361, "lng": 132.756, "category": "Electronics"},
{"id": "sup-bosch-stuttgart", "name": "Bosch Auto Parts (Tier 1)", "city": "Stuttgart", "country": "Germany", "lat": 48.815, "lng": 9.176, "category": "Automotive"},
{"id": "sup-zf-bavaria", "name": "ZF Friedrichshafen (Tier 1)", "city": "Friedrichshafen", "country": "Germany", "lat": 47.662, "lng": 9.489, "category": "Automotive"},
{"id": "sup-valeo-paris", "name": "Valeo R&D (Tier 2)", "city": "Paris", "country": "France", "lat": 48.878, "lng": 2.308, "category": "Automotive"},
{"id": "sup-magna-celaya", "name": "Magna Assembly (Tier 2)", "city": "Celaya", "country": "Mexico", "lat": 20.525, "lng": -100.814, "category": "Automotive"},
{"id": "sup-denso-monterrey", "name": "Denso Corp (Tier 1)", "city": "Monterrey", "country": "Mexico", "lat": 25.772, "lng": -100.174, "category": "Automotive"},
{"id": "sup-catl-ningde", "name": "CATL Battery HQ (Tier 1)", "city": "Ningde", "country": "China", "lat": 26.666, "lng": 119.544, "category": "Battery"},
{"id": "sup-byd-shenzhen", "name": "BYD Gigafactory (Tier 1)", "city": "Shenzhen", "country": "China", "lat": 22.684, "lng": 114.341, "category": "Battery"},
{"id": "sup-panasonic-nevada", "name": "Panasonic Giga (Tier 1)", "city": "Sparks", "country": "US", "lat": 39.539, "lng": -119.439, "category": "Battery"},
]
def _distance_km(lat1: float, lng1: float, lat2: float, lng2: float) -> float:
dx = (lng1 - lng2) * math.cos(math.radians((lat1 + lat2) / 2))
dy = lat1 - lat2
return math.sqrt(dx * dx + dy * dy) * 111.32
def _seismic_risk_level(distance_km: float, magnitude: float) -> str | None:
"""Meaningful fab impact only — ignore routine micro-quakes (e.g. Taiwan M3.x)."""
if magnitude < 4.5:
return None
if magnitude >= 6.0 and distance_km <= 200:
return "CRITICAL"
if magnitude >= 5.5 and distance_km <= 75:
return "CRITICAL"
if magnitude >= 5.0 and distance_km <= 100:
return "HIGH"
if magnitude >= 4.5 and distance_km <= 40:
return "HIGH"
return None
def _apply_seismic_threats(suppliers: list[dict[str, Any]], earthquakes: list[dict[str, Any]]) -> None:
for sup in suppliers:
best: tuple[str, float] | None = None
for eq in earthquakes:
lat = eq.get("lat")
lng = eq.get("lng") or eq.get("lon")
mag = float(eq.get("mag") or eq.get("magnitude") or 0)
if lat is None or lng is None or mag < 4.5:
continue
dist = _distance_km(sup["lat"], sup["lng"], float(lat), float(lng))
level = _seismic_risk_level(dist, mag)
if not level:
continue
severity = {"HIGH": 1, "CRITICAL": 2}
if best is None:
best = (level, mag)
else:
cur = severity[level]
prev = severity[best[0]]
if cur > prev or (cur == prev and mag > best[1]):
best = (level, mag)
if best:
level, mag = best
if sup["risk_level"] == "NORMAL" or (
level == "CRITICAL" and sup["risk_level"] != "CRITICAL"
):
sup["risk_level"] = level
elif level == "CRITICAL" and sup["risk_level"] == "HIGH":
sup["risk_level"] = "CRITICAL"
sup["active_threats"].append(f"SEISMIC PROXIMITY (M{mag:.1f})")
def build_scm_payload() -> dict[str, Any]:
suppliers = [{**s, "risk_level": "NORMAL", "active_threats": []} for s in SUPPLIERS]
refs = get_latest_data_subset_refs("earthquakes", "firms_fires", "gdelt")
earthquakes = refs.get("earthquakes") or []
_apply_seismic_threats(suppliers, earthquakes)
fires = refs.get("firms_fires") or []
for sup in suppliers:
count = 0
for fire in fires:
lat = fire.get("lat") or fire.get("latitude")
lng = fire.get("lng") or fire.get("lon") or fire.get("longitude")
if lat is None or lng is None:
continue
if _distance_km(sup["lat"], sup["lng"], float(lat), float(lng)) < 50:
count += 1
if count:
if sup["risk_level"] == "NORMAL":
sup["risk_level"] = "HIGH"
sup["active_threats"].append(f"WILDFIRE PROXIMITY ({count} hotspots)")
conflicts = refs.get("gdelt") or []
for sup in suppliers:
for event in conflicts:
lat = event.get("lat")
lng = event.get("lng") or event.get("lon")
if lat is None or lng is None:
continue
if _distance_km(sup["lat"], sup["lng"], float(lat), float(lng)) < 100:
sup["risk_level"] = "CRITICAL"
sup["active_threats"].append("ARMED CONFLICT / RIOT")
break
# USGS fallback if earthquakes empty
if not earthquakes:
try:
resp = fetch_with_curl(
"https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_day.geojson",
timeout=5,
)
if resp.status_code == 200:
features = resp.json().get("features") or []
usgs_quakes = [
{
"lat": f.get("geometry", {}).get("coordinates", [None, None])[1],
"lng": f.get("geometry", {}).get("coordinates", [None, None])[0],
"mag": f.get("properties", {}).get("mag") or 0,
}
for f in features
if len(f.get("geometry", {}).get("coordinates") or []) >= 2
]
_apply_seismic_threats(suppliers, usgs_quakes)
except Exception:
pass
critical = sum(1 for s in suppliers if s["risk_level"] == "CRITICAL")
return {
"suppliers": suppliers,
"total": len(suppliers),
"critical_count": critical,
"timestamp": datetime.now(timezone.utc).isoformat(),
}
def fetch_scm_suppliers() -> dict[str, Any]:
if not is_any_active("scm_suppliers"):
return latest_data.get("scm_suppliers") or {}
payload = build_scm_payload()
with _data_lock:
latest_data["scm_suppliers"] = payload
_mark_fresh("scm_suppliers")
return payload
backend/services/ssrf_guard.py
+141
View File
@@ -0,0 +1,141 @@
"""SSRF guard for operator-initiated recon (ported from Osiris ssrf-guard.ts)."""
from __future__ import annotations
import ipaddress
import re
import socket
from typing import Any
from urllib.parse import urljoin, urlparse
import requests
_IPV4_BLOCKS = [
ipaddress.ip_network("0.0.0.0/8"),
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("100.64.0.0/10"),
ipaddress.ip_network("127.0.0.0/8"),
ipaddress.ip_network("169.254.0.0/16"),
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.0.0.0/24"),
ipaddress.ip_network("192.0.2.0/24"),
ipaddress.ip_network("192.168.0.0/16"),
ipaddress.ip_network("198.18.0.0/15"),
ipaddress.ip_network("198.51.100.0/24"),
ipaddress.ip_network("203.0.113.0/24"),
ipaddress.ip_network("224.0.0.0/4"),
ipaddress.ip_network("240.0.0.0/4"),
]
_NAME_BLOCKLIST = (
re.compile(r"^localhost$", re.I),
re.compile(r"\.localhost$", re.I),
re.compile(r"^host\.docker\.internal$", re.I),
re.compile(r"\.local$", re.I),
re.compile(r"\.internal$", re.I),
re.compile(r"^metadata\.google\.internal$", re.I),
)
_HOSTNAME_RE = re.compile(
r"^[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?)*$"
)
def _ipv4_blocked(ip: str) -> bool:
try:
addr = ipaddress.ip_address(ip)
except ValueError:
return True
if not isinstance(addr, ipaddress.IPv4Address):
return False
return any(addr in net for net in _IPV4_BLOCKS)
def _ip_blocked(ip: str) -> bool:
try:
addr = ipaddress.ip_address(ip)
except ValueError:
return True
if isinstance(addr, ipaddress.IPv4Address):
return _ipv4_blocked(ip)
return (
addr.is_loopback
or addr.is_private
or addr.is_link_local
or addr.is_multicast
or addr.is_reserved
or addr.is_unspecified
)
def validate_host(host: str) -> dict[str, Any]:
trimmed = (host or "").strip()
if not trimmed:
return {"ok": False, "reason": "empty host"}
bracketed = trimmed.strip("[]")
lower = trimmed.lower()
if any(p.search(lower) for p in _NAME_BLOCKLIST):
return {"ok": False, "reason": "hostname matches reserved name pattern"}
try:
ipaddress.ip_address(bracketed)
is_literal = True
except ValueError:
is_literal = False
if is_literal:
if _ip_blocked(bracketed):
return {"ok": False, "reason": "IP in reserved range"}
return {"ok": True, "resolved": [bracketed]}
if not _HOSTNAME_RE.match(trimmed):
return {"ok": False, "reason": "invalid hostname syntax"}
try:
infos = socket.getaddrinfo(trimmed, None, proto=socket.IPPROTO_TCP)
except OSError as exc:
return {"ok": False, "reason": f"DNS lookup failed: {exc}"}
if not infos:
return {"ok": False, "reason": "hostname has no A/AAAA records"}
resolved: list[str] = []
for info in infos:
addr = info[4][0]
if _ip_blocked(addr):
return {"ok": False, "reason": f"hostname resolves to reserved IP {addr}"}
resolved.append(addr)
return {"ok": True, "resolved": resolved}
def safe_get(
url: str,
*,
timeout: float = 8.0,
headers: dict[str, str] | None = None,
max_redirects: int = 3,
) -> requests.Response:
current = url
for _ in range(max_redirects + 1):
parsed = urlparse(current)
if parsed.scheme not in ("http", "https"):
raise ValueError(f"blocked protocol {parsed.scheme}")
check = validate_host(parsed.hostname or "")
if not check.get("ok"):
raise ValueError(f"blocked target — {check.get('reason')}")
res = requests.get(
current,
timeout=timeout,
headers=headers or {},
allow_redirects=False,
)
if 300 <= res.status_code < 400:
loc = res.headers.get("location")
if not loc:
return res
current = urljoin(current, loc)
continue
return res
raise ValueError("too many redirects")
def validate_domain(domain: str) -> bool:
return bool(re.match(r"^[a-zA-Z0-9][a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$", domain or ""))
backend/tests/test_cctv_pipeline.py
+59
View File
@@ -77,3 +77,62 @@ def test_ingest_updates_existing_rows_in_persistent_data_dir(tmp_path, monkeypat
assert len(cameras) == 1
assert cameras[0]["media_url"] == "https://example.com/live.m3u8"
assert cameras[0]["media_type"] == "hls"
def test_scheduled_cctv_ingestors_include_asfinag_and_alpr():
names = {ing.__class__.__name__ for ing, _ in cctv_pipeline.scheduled_cctv_ingestors()}
assert "AsfinagIngestor" in names
assert "OSMALPRCameraIngestor" in names
assert "OSMTrafficCameraIngestor" in names
assert "Ontario511Ingestor" in names
assert "Alberta511Ingestor" in names
assert "Florida511Ingestor" in names
assert "AustraliaLiveTrafficIngestor" in names
assert "NetherlandsRWSIngestor" in names
assert len(names) == 21
def test_fetch_traveliq_v2_cameras_parses_views(monkeypatch):
class FakeResp:
status_code = 200
@staticmethod
def json():
return [
{
"Id": 9,
"Latitude": 45.0,
"Longitude": -75.0,
"Location": "Test Highway",
"Views": [
{
"Id": 42,
"Url": "/map/Cctv/42",
"Status": "Enabled",
"Description": "Northbound",
}
],
}
]
monkeypatch.setattr(cctv_pipeline, "fetch_with_curl", lambda *a, **k: FakeResp())
cameras = cctv_pipeline._fetch_traveliq_v2_cameras(
api_url="https://511on.ca/api/v2/get/cameras",
base_url="https://511on.ca",
id_prefix="ON511",
source_agency="511 Ontario",
)
assert len(cameras) == 1
assert cameras[0]["id"] == "ON511-9-42"
assert cameras[0]["media_url"] == "https://511on.ca/map/Cctv/42"
def test_ensure_https_upgrades_http_media_urls():
assert (
cctv_pipeline._ensure_https_url("http://example.com/camera.jpg")
== "https://example.com/camera.jpg"
)
assert (
cctv_pipeline._ensure_https_url("https://secure.example.com/live.m3u8")
== "https://secure.example.com/live.m3u8"
)
backend/tests/test_datacenters_fetch.py
+10
View File
@@ -0,0 +1,10 @@
"""Datacenters load from static JSON regardless of layer toggle."""
from services.fetchers import _store
from services.fetchers.infrastructure import fetch_datacenters
def test_fetch_datacenters_populates_store_when_layer_disabled(monkeypatch):
monkeypatch.setitem(_store.active_layers, "datacenters", False)
_store.latest_data["datacenters"] = []
fetch_datacenters()
assert len(_store.latest_data.get("datacenters") or []) > 0
backend/tests/test_geo_fetchers.py
+49
View File
@@ -113,3 +113,52 @@ def test_fetch_fishing_activity_dedupes_to_latest_event_per_vessel(monkeypatch):
assert latest_data["fishing_activity"][0]["vessel_ssvid"] == "ssvid-1"
finally:
latest_data["fishing_activity"] = original
def test_fetch_fishing_activity_respects_max_pages(monkeypatch):
from services.fetchers import geo
from services.fetchers._store import latest_data
original = list(latest_data.get("fishing_activity") or [])
requests: list[str] = []
def fake_fetch(url, timeout=30, headers=None):
requests.append(url)
offset = 0
if "offset=500" in url:
offset = 500
payload = {
"total": 5000,
"entries": [
{
"id": f"evt-{offset + i}",
"position": {"lat": 10.0 + i, "lon": 20.0 + i},
"event": {"duration": 3600},
"vessel": {
"id": f"v-{offset + i}",
"ssvid": f"ssvid-{offset + i}",
"name": f"Vessel-{offset + i}",
"flag": "US",
},
}
for i in range(500)
],
"nextOffset": offset + 500,
}
return SimpleNamespace(status_code=200, json=lambda p=payload: p)
monkeypatch.setenv("GFW_API_TOKEN", "test-token")
monkeypatch.setenv("GFW_EVENTS_PAGE_SIZE", "500")
monkeypatch.setenv("GFW_EVENTS_MAX_PAGES", "2")
monkeypatch.setattr("services.fetchers._store.is_any_active", lambda *args: True)
monkeypatch.setattr(geo, "fetch_with_curl", fake_fetch)
monkeypatch.setattr(geo, "_mark_fresh", lambda *args, **kwargs: None)
monkeypatch.setattr(geo, "_last_fishing_fetch_ts", 0.0)
try:
geo.fetch_fishing_activity()
assert len(latest_data["fishing_activity"]) == 1000
assert len(requests) == 2
assert all("offset=0" in url or "offset=500" in url for url in requests)
finally:
latest_data["fishing_activity"] = original
backend/tests/test_osiris_port.py
+43
View File
@@ -0,0 +1,43 @@
"""Tests for Osiris-ported security and sanctions modules."""
from __future__ import annotations
import pytest
from services.ssrf_guard import validate_host, validate_domain
from services.sanctions.ofac import norm_name, search_sanctions
def test_ssrf_blocks_localhost():
result = validate_host("localhost")
assert result["ok"] is False
def test_ssrf_blocks_private_ip():
result = validate_host("192.168.1.1")
assert result["ok"] is False
def test_ssrf_blocks_metadata_endpoint():
result = validate_host("metadata.google.internal")
assert result["ok"] is False
def test_validate_domain_rejects_garbage():
assert validate_domain("not a domain") is False
assert validate_domain("example.com") is True
def test_norm_name_strips_punctuation():
assert norm_name("ACME, Inc.") == norm_name("acme inc")
def test_search_sanctions_requires_min_length():
assert search_sanctions("ab") == []
@pytest.mark.parametrize("query", ["127.0.0.1", "10.0.0.1"])
def test_sweep_init_rejects_private(query: str):
from services.osint.lookups import sweep_init
with pytest.raises(ValueError, match="Private|reserved|Invalid"):
sweep_init(query, 24)
backend/tests/test_scm_suppliers.py
+13
View File
@@ -0,0 +1,13 @@
from services.scm.suppliers import _seismic_risk_level
def test_micro_quakes_ignored():
assert _seismic_risk_level(10.0, 3.9) is None
assert _seismic_risk_level(10.0, 4.4) is None
def test_meaningful_quake_thresholds():
assert _seismic_risk_level(30.0, 4.6) == "HIGH"
assert _seismic_risk_level(80.0, 5.2) == "HIGH"
assert _seismic_risk_level(50.0, 5.6) == "CRITICAL"
assert _seismic_risk_level(150.0, 6.1) == "CRITICAL"
backend/tests/test_telegram_osint.py
+103
View File
@@ -0,0 +1,103 @@
"""Telegram OSINT HTML parsing and geoparsing."""
from services.fetchers import telegram_osint
SAMPLE_HTML = """
<div class="tgme_widget_message_wrap js-widget_message_wrap">
<div class="tgme_widget_message_text">Missile strike reported near Kyiv overnight.</div>
<a class="tgme_widget_message_date" href="https://t.me/osintdefender/12345">
<time datetime="2026-06-02T12:00:00+00:00"></time>
</a>
</div>
</div>
</div>
"""
SAMPLE_VIDEO_HTML = """
<div class="tgme_widget_message_wrap js-widget_message_wrap">
<div class="tgme_widget_message_text">Drone footage from Kharkiv.</div>
<video src="https://cdn4.telesco.pe/file/sample.mp4?token=abc" class="tgme_widget_message_video js-message_video"></video>
<a class="tgme_widget_message_date" href="https://t.me/osintdefender/99999">
<time datetime="2026-06-02T13:00:00+00:00"></time>
</a>
</div>
</div>
</div>
"""
def test_parse_telegram_channel_html_extracts_geolocated_post():
posts = telegram_osint.parse_telegram_channel_html(SAMPLE_HTML, "osintdefender")
assert len(posts) == 1
post = posts[0]
assert "Kyiv" in post["title"]
assert post["coords"] == [50.45, 30.523]
assert post["risk_score"] >= 3
assert post["link"].startswith("https://t.me/")
def test_resolve_telegram_coords_handles_cyrillic():
coords = telegram_osint._resolve_telegram_coords("Обстріл біля Харкова")
assert coords == (49.993, 36.231)
def test_resolve_telegram_coords_uses_metro_anchors_for_country_tags():
assert telegram_osint._resolve_telegram_coords("#Israel #Iran") == (32.085, 34.781)
assert telegram_osint._resolve_telegram_coords("China announces policy") == (39.904, 116.407)
assert telegram_osint._resolve_telegram_coords("#USA response") == (40.712, -74.006)
def test_resolve_telegram_coords_keeps_specific_cities_over_country_anchor():
assert telegram_osint._resolve_telegram_coords("Strike near Gaza") == (31.416, 34.333)
assert telegram_osint._resolve_telegram_coords("Missile strike reported near Kyiv overnight") == (
50.45,
30.523,
)
def test_parse_telegram_channel_html_extracts_video_media():
posts = telegram_osint.parse_telegram_channel_html(SAMPLE_VIDEO_HTML, "osintdefender")
assert len(posts) == 1
post = posts[0]
assert post["media_type"] == "video"
assert post["media_url"].startswith("https://cdn4.telesco.pe/")
assert post["embed_url"] == "https://t.me/osintdefender/99999?embed=1"
def test_telegram_media_host_allowed():
assert telegram_osint.telegram_media_host_allowed("cdn4.telesco.pe")
assert telegram_osint.telegram_media_host_allowed("cdn4.telegram-cdn.org")
assert not telegram_osint.telegram_media_host_allowed("evil.example.com")
def test_extract_new_channel_posts_stops_at_known_links():
known = {"https://t.me/osintdefender/12345"}
fresh = telegram_osint._extract_new_channel_posts(SAMPLE_HTML, "osintdefender", known)
assert fresh == []
def test_merge_telegram_posts_keeps_existing_and_adds_only_new():
existing = [
{
"id": "old",
"link": "https://t.me/osintdefender/111",
"published": "2026-06-01T12:00:00+00:00",
}
]
incoming = [
{
"id": "dup",
"link": "https://t.me/osintdefender/111",
"published": "2026-06-02T12:00:00+00:00",
},
{
"id": "new",
"link": "https://t.me/osintdefender/222",
"published": "2026-06-03T12:00:00+00:00",
},
]
merged, added = telegram_osint._merge_telegram_posts(existing, incoming)
assert added == 1
assert len(merged) == 2
assert merged[0]["link"] == "https://t.me/osintdefender/222"
backend/third_party/osiris/NOTICE.md
Vendored
+14
View File
@@ -0,0 +1,14 @@
# Osiris-derived components — third-party notice
Portions of the recon toolkit, sanctions index, SCM overlay, entity graph,
malware feeds, and related UI were adapted from:
- **OSIRIS** — MIT License — Copyright (c) 2026 simplifaisoul
https://github.com/simplifaisoul/osiris
Additional data attribution:
- **OpenSanctions** `us_ofac_sdn` dataset — CC-BY 4.0
https://www.opensanctions.org/
- **TeleGeography** submarine cable map data (static GeoJSON)
- **abuse.ch** Feodo Tracker / URLhaus (malware feeds)
docker-compose.yml
+9
View File
@@ -17,6 +17,12 @@ services:
- OPENSKY_CLIENT_ID=${OPENSKY_CLIENT_ID:-}
- OPENSKY_CLIENT_SECRET=${OPENSKY_CLIENT_SECRET:-}
- LTA_ACCOUNT_KEY=${LTA_ACCOUNT_KEY:-}
- GFW_API_TOKEN=${GFW_API_TOKEN:-}
- GFW_EVENTS_PAGE_SIZE=${GFW_EVENTS_PAGE_SIZE:-500}
- GFW_EVENTS_MAX_PAGES=${GFW_EVENTS_MAX_PAGES:-10}
- GFW_EVENTS_LOOKBACK_DAYS=${GFW_EVENTS_LOOKBACK_DAYS:-7}
- GFW_EVENTS_TIMEOUT_S=${GFW_EVENTS_TIMEOUT_S:-90}
- WINDY_API_KEY=${WINDY_API_KEY:-}
- ADMIN_KEY=${ADMIN_KEY:-}
- FINNHUB_API_KEY=${FINNHUB_API_KEY:-}
# Override allowed CORS origins (comma-separated). Auto-detects LAN IPs if empty.
@@ -77,6 +83,9 @@ services:
- FIMI_ENABLED=${FIMI_ENABLED:-false}
- NUFORC_ENABLED=${NUFORC_ENABLED:-false}
- NEWS_ENABLED=${NEWS_ENABLED:-true}
- TELEGRAM_OSINT_ENABLED=${TELEGRAM_OSINT_ENABLED:-true}
- TELEGRAM_OSINT_CHANNELS=${TELEGRAM_OSINT_CHANNELS:-}
- TELEGRAM_OSINT_INTERVAL_MINUTES=${TELEGRAM_OSINT_INTERVAL_MINUTES:-60}
volumes:
- backend_data:/app/data
restart: unless-stopped
frontend/public/data/submarine-cables.json
+1
View File
File diff suppressed because one or more lines are too long
frontend/src/__tests__/hooks/useDataPollingViewport.test.ts
+26
View File
@@ -0,0 +1,26 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { VIEWPORT_COMMITTED_EVENT } from '@/components/map/hooks/useViewportBounds';
import { setLiveDataBounds } from '@/lib/liveDataViewport';
describe('viewport fast refetch wiring', () => {
beforeEach(() => {
vi.useFakeTimers();
setLiveDataBounds({ south: 10, west: 20, north: 12, east: 22 });
});
afterEach(() => {
setLiveDataBounds(null);
vi.useRealTimers();
vi.restoreAllMocks();
});
it('VIEWPORT_COMMITTED_EVENT is a stable custom event name', () => {
expect(VIEWPORT_COMMITTED_EVENT).toBe('shadowbroker:viewport-committed');
const handler = vi.fn();
window.addEventListener(VIEWPORT_COMMITTED_EVENT, handler);
window.dispatchEvent(new CustomEvent(VIEWPORT_COMMITTED_EVENT));
expect(handler).toHaveBeenCalledTimes(1);
window.removeEventListener(VIEWPORT_COMMITTED_EVENT, handler);
});
});
frontend/src/__tests__/lib/submarineCables.test.ts
+61
View File
@@ -0,0 +1,61 @@
import { sanitizeSubmarineCables } from '@/lib/submarineCables';
describe('sanitizeSubmarineCables', () => {
it('removes synthetic corridor overlays', () => {
const out = sanitizeSubmarineCables({
type: 'FeatureCollection',
features: [
{
type: 'Feature',
properties: { name: 'SEA-ME-WE Corridor' },
geometry: {
type: 'LineString',
coordinates: [
[-5, 51],
[73, 17],
],
},
},
{
type: 'Feature',
properties: { name: 'FEA' },
geometry: {
type: 'LineString',
coordinates: [
[32, 30],
[33, 29],
],
},
},
],
});
expect(out.features).toHaveLength(1);
expect(out.features[0].properties?.name).toBe('FEA');
});
it('splits trans-ocean jumps into separate segments', () => {
const out = sanitizeSubmarineCables({
type: 'FeatureCollection',
features: [
{
type: 'Feature',
properties: { name: 'Test Pacific' },
geometry: {
type: 'LineString',
coordinates: [
[-120, 35],
[-125, 36],
[100, 13],
[101, 12],
],
},
},
],
});
const geom = out.features[0].geometry;
expect(geom?.type).toBe('MultiLineString');
if (geom?.type === 'MultiLineString') {
expect(geom.coordinates).toHaveLength(2);
}
});
});
frontend/src/__tests__/map/telegramGeoJSON.test.ts
+94
View File
@@ -0,0 +1,94 @@
import { describe, expect, it } from 'vitest';
import {
applyTelegramAlertAvoidance,
buildTelegramOsintGeoJSON,
telegramClusterKey,
telegramClusterNearNewsAlert,
telegramMapPinCoords,
TELEGRAM_ALERT_AVOID_METERS,
} from '@/components/map/geoJSONBuilders';
describe('telegramMapPinCoords', () => {
it('stays on the geocoded city when no threat alert overlaps', () => {
const [lat, lng] = telegramMapPinCoords(31.046, 34.851, false);
expect(lat).toBe(31.046);
expect(lng).toBe(34.851);
});
it('nudges ~5 mi northeast only when avoiding an alert', () => {
const [lat, lng] = telegramMapPinCoords(31.046, 34.851, true);
expect(lat).toBeGreaterThan(31.046);
expect(lng).toBeGreaterThan(34.851);
const toRad = (deg: number) => (deg * Math.PI) / 180;
const dLat = toRad(lat - 31.046);
const meters = 6371000 * dLat;
expect(meters).toBeGreaterThan(4_000);
expect(meters).toBeLessThan(TELEGRAM_ALERT_AVOID_METERS + 2_000);
});
});
describe('telegramClusterNearNewsAlert', () => {
it('detects news on the same city grid', () => {
const news = [{ coords: [31.046, 34.851] as [number, number] }];
expect(telegramClusterNearNewsAlert(31.049, 34.849, news)).toBe(true);
expect(telegramClusterNearNewsAlert(50.45, 30.52, news)).toBe(false);
});
});
describe('telegramClusterKey', () => {
it('groups nearby coordinates to the same city bucket', () => {
expect(telegramClusterKey(50.451, 30.521)).toBe(telegramClusterKey(50.449, 30.519));
});
});
describe('buildTelegramOsintGeoJSON', () => {
it('places the dot on the geocoded city by default', () => {
const geo = buildTelegramOsintGeoJSON({
posts: [
{
id: 'tg-1',
title: 'Strike near Kyiv',
coords: [50.45, 30.52],
},
],
});
const feature = geo?.features[0];
expect(feature).toBeTruthy();
const [lng, lat] = feature!.geometry!.coordinates as [number, number];
expect(lat).toBeCloseTo(50.45, 2);
expect(lng).toBeCloseTo(30.52, 2);
});
it('merges posts in the same city into one pin', () => {
const geo = buildTelegramOsintGeoJSON({
posts: [
{ id: 'a', title: 'Post A', coords: [50.45, 30.52] },
{ id: 'b', title: 'Post B', coords: [50.451, 30.521] },
{ id: 'c', title: 'Post C', coords: [48.0, 37.8] },
],
});
expect(geo?.features).toHaveLength(2);
const kyiv = geo?.features.find((f) => f.properties?.post_count === 2);
expect(kyiv).toBeTruthy();
expect(kyiv?.properties?.id).toBe(telegramClusterKey(50.45, 30.52));
});
});
describe('applyTelegramAlertAvoidance', () => {
it('offsets only clusters that share a grid cell with a news alert', () => {
const geo = buildTelegramOsintGeoJSON({
posts: [
{ id: 'il', title: 'Israel post', coords: [31.046, 34.851] },
{ id: 'ua', title: 'Kyiv post', coords: [50.45, 30.52] },
],
});
const placed = applyTelegramAlertAvoidance(geo, [{ coords: [31.046, 34.851] }]);
const israel = placed?.features.find((f) => f.properties?.id === telegramClusterKey(31.046, 34.851));
const kyiv = placed?.features.find((f) => f.properties?.id === telegramClusterKey(50.45, 30.52));
const [ilLng, ilLat] = israel!.geometry!.coordinates as [number, number];
const [uaLng, uaLat] = kyiv!.geometry!.coordinates as [number, number];
expect(ilLat).toBeGreaterThan(31.046);
expect(uaLat).toBeCloseTo(50.45, 2);
expect(uaLng).toBeCloseTo(30.52, 2);
});
});
frontend/src/__tests__/utils/viewportPrivacy.test.ts
+12
View File
@@ -5,6 +5,10 @@ import {
coarsenViewBounds,
expandBoundsToRadius,
} from '@/lib/viewportPrivacy';
import {
liveDataBoundsKey,
setLiveDataBounds,
} from '@/lib/liveDataViewport';
describe('viewport privacy helper', () => {
it('coarsens narrow bounds outward without clipping the original view', () => {
@@ -45,6 +49,14 @@ describe('viewport privacy helper', () => {
expect(b).toBe(a);
});
it('liveDataBoundsKey matches quantized fetch params and clears for world view', () => {
setLiveDataBounds({ south: 33.6, west: -84.5, north: 33.8, east: -84.2 });
expect(liveDataBoundsKey()).toBe('33,-85,34,-84');
setLiveDataBounds(null);
expect(liveDataBoundsKey()).toBeNull();
});
it('expands bounds to a fixed preload radius around the current view center', () => {
const original = {
south: 39.55,
frontend/src/app/page.tsx
+29 -1
View File
@@ -21,6 +21,10 @@ import InfonetTerminal from '@/components/InfonetTerminal';
import { leaveWormhole, fetchWormholeState } from '@/mesh/wormholeClient';
import { teardownWormholeOnClose } from '@/lib/wormholeTeardown';
import ShodanPanel from '@/components/ShodanPanel';
import ReconPanel from '@/components/ReconPanel';
import ScmPanel from '@/components/ScmPanel';
import EntityGraphPanel from '@/components/EntityGraphPanel';
import { isEntityGraphEligible } from '@/lib/entityGraph';
import AIIntelPanel from '@/components/AIIntelPanel';
import GlobalTicker from '@/components/GlobalTicker';
import ErrorBoundary from '@/components/ErrorBoundary';
@@ -71,6 +75,10 @@ export default function Dashboard() {
useDataPolling();
const { mouseCoords, locationLabel, handleMouseCoords } = useReverseGeocode();
const [selectedEntity, setSelectedEntity] = useState<SelectedEntity | null>(null);
const [showEntityGraph, setShowEntityGraph] = useState(false);
useEffect(() => {
setShowEntityGraph(false);
}, [selectedEntity]);
const [trackedSdr, setTrackedSdr] = useState<KiwiSDR | null>(null);
const [trackedScanner, setTrackedScanner] = useState<Scanner | null>(null);
const { regionDossier, regionDossierLoading, handleMapRightClick } = useRegionDossier(
@@ -186,6 +194,11 @@ export default function Dashboard() {
sentinel_hub: false,
viirs_nightlights: false,
road_corridor_trends: false,
malware_c2: false,
submarine_cables: false,
scm_suppliers: false,
cyber_threats: false,
telegram_osint: true,
// Hazards — no fire, rest ON
earthquakes: true,
firms: false,
@@ -636,7 +649,15 @@ export default function Dashboard() {
</div>
)}
{/* 4. AI INTEL (Below Shodan) */}
{/* 4. RECON + SCM */}
{secondaryBootReady && (
<div className="contents" style={{ direction: 'ltr' }}>
<ReconPanel />
<ScmPanel layerEnabled={activeLayers.scm_suppliers} />
</div>
)}
{/* 5. AI INTEL */}
{secondaryBootReady && (
<div className="contents" style={{ direction: 'ltr' }}>
<AIIntelPanel
@@ -748,6 +769,9 @@ export default function Dashboard() {
selectedEntity={selectedEntity}
regionDossier={regionDossier}
regionDossierLoading={regionDossierLoading}
onExpandEntityGraph={() => {
if (isEntityGraphEligible(selectedEntity)) setShowEntityGraph(true);
}}
onArticleClick={(idx, lat, lng, title) => {
if (lat !== undefined && lng !== undefined) {
setFlyToLocation({ lat, lng, ts: Date.now() });
@@ -989,6 +1013,10 @@ export default function Dashboard() {
onSettingsClick={() => setSettingsOpen(true)}
/>
{showEntityGraph && selectedEntity && isEntityGraphEligible(selectedEntity) && (
<EntityGraphPanel entity={selectedEntity} onClose={() => setShowEntityGraph(false)} />
)}
{/* INFONET TERMINAL */}
<InfonetTerminal
isOpen={infonetOpen}
frontend/src/components/EntityGraphPanel.tsx
+165
View File
@@ -0,0 +1,165 @@
'use client';
import React, { useCallback, useEffect, useState } from 'react';
import { Loader2, Minus, Network, Plus, X } from 'lucide-react';
import { API_BASE } from '@/lib/api';
import { isEntityGraphEligible, mapEntityToGraphType } from '@/lib/entityGraph';
import type { SelectedEntity } from '@/types/dashboard';
interface GraphNode {
id: string;
label: string;
type: string;
properties?: Record<string, unknown>;
}
interface GraphLink {
source: string;
target: string;
label: string;
}
interface Props {
entity: SelectedEntity | null;
onClose: () => void;
}
const TYPE_COLORS: Record<string, string> = {
aircraft: 'text-cyan-300',
vessel: 'text-cyan-400',
company: 'text-amber-300',
person: 'text-violet-300',
country: 'text-emerald-300',
sanction: 'text-red-300',
ip: 'text-orange-300',
event: 'text-yellow-300',
};
export default function EntityGraphPanel({ entity, onClose }: Props) {
const [isMinimized, setIsMinimized] = useState(false);
const [nodes, setNodes] = useState<GraphNode[]>([]);
const [links, setLinks] = useState<GraphLink[]>([]);
const [loading, setLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const loadGraph = useCallback(async () => {
if (!entity || !isEntityGraphEligible(entity)) return;
const type = mapEntityToGraphType(entity.type);
if (!type) return;
const id = String(entity.name || entity.extra?.callsign || entity.extra?.registration || entity.id);
const params = new URLSearchParams({ type, id });
if (entity.extra?.registration) params.set('registration', String(entity.extra.registration));
if (entity.extra?.icao24) params.set('icao24', String(entity.extra.icao24));
if (entity.extra?.model) params.set('model', String(entity.extra.model));
setLoading(true);
setError(null);
try {
const res = await fetch(`${API_BASE}/api/entity/expand?${params}`);
const data = await res.json();
if (!res.ok) throw new Error(data.detail || data.error || 'Expand failed');
setNodes(data.nodes || []);
setLinks(data.links || []);
} catch (err) {
setError(err instanceof Error ? err.message : 'Graph unavailable');
setNodes([]);
setLinks([]);
} finally {
setLoading(false);
}
}, [entity]);
useEffect(() => {
if (entity) loadGraph();
else {
setNodes([]);
setLinks([]);
}
}, [entity, loadGraph]);
if (!entity || !isEntityGraphEligible(entity)) return null;
return (
<div className="fixed bottom-4 right-4 z-[250] w-80 max-h-[50vh] pointer-events-auto flex flex-col border border-cyan-700/40 bg-black/85 backdrop-blur-sm shadow-[0_0_24px_rgba(34,211,238,0.12)]">
<div
className="flex items-center justify-between border-b border-cyan-700/30 bg-cyan-950/25 px-3 py-2.5 cursor-pointer hover:bg-cyan-950/40 transition-colors"
onClick={() => setIsMinimized((prev) => !prev)}
>
<div className="flex items-center gap-2 min-w-0">
<Network size={16} className="text-cyan-400 shrink-0" />
<span className="text-[12px] font-mono font-bold tracking-widest text-cyan-400 truncate">
ENTITY GRAPH
</span>
</div>
<div className="flex items-center gap-2 shrink-0">
<button
type="button"
onClick={(e) => {
e.stopPropagation();
onClose();
}}
className="text-cyan-600 hover:text-cyan-300 transition-colors"
title="Close"
>
<X size={14} />
</button>
{isMinimized ? (
<Plus size={16} className="text-cyan-400" />
) : (
<Minus size={16} className="text-cyan-400" />
)}
</div>
</div>
{!isMinimized && (
<div className="px-3 py-2 overflow-y-auto styled-scrollbar flex-1 space-y-2">
<div className="text-[10px] font-mono tracking-wider text-cyan-600 truncate">
{entity.type.toUpperCase()} · {entity.name || entity.id}
</div>
{loading && (
<div className="flex items-center gap-2 text-[11px] font-mono text-cyan-500 tracking-wider">
<Loader2 size={12} className="animate-spin" />
RESOLVING…
</div>
)}
{error && (
<div className="border border-red-500/30 bg-red-950/20 px-2 py-1.5 text-[11px] font-mono text-red-400">
{error}
</div>
)}
{!loading && !error && (
<>
<div className="space-y-1">
{nodes.map((n) => (
<div
key={n.id}
className="border border-cyan-900/40 bg-black/50 px-2 py-1.5"
>
<div className={`text-[9px] font-mono tracking-[0.2em] uppercase opacity-70 ${TYPE_COLORS[n.type] || 'text-cyan-500'}`}>
{n.type}
</div>
<div className="text-[11px] font-mono text-cyan-200 leading-snug">{n.label}</div>
</div>
))}
</div>
{links.length > 0 && (
<div className="border-t border-cyan-900/40 pt-2">
<div className="text-[10px] font-mono tracking-[0.2em] text-cyan-600 mb-1">RELATIONSHIPS</div>
{links.slice(0, 24).map((l, i) => (
<div key={`${l.source}-${l.target}-${i}`} className="text-[10px] font-mono text-cyan-500/90 truncate leading-relaxed">
{l.label}: {l.source.split(':').pop()} → {l.target.split(':').pop()}
</div>
))}
</div>
)}
</>
)}
</div>
)}
</div>
);
}
frontend/src/components/MapLegend.tsx
+1
View File
@@ -183,6 +183,7 @@ const LEGEND: LegendCategory[] = [
color: 'text-red-400 border-red-500/30',
items: [
{ svg: triangle('#ffaa00'), label: 'GDELT / LiveUA event (yellow)' },
{ svg: dot('#ef4444'), label: 'Telegram OSINT post (red, geolocated)' },
{ svg: triangle('#ff0000'), label: 'Violent / Kinetic event (red)' },
{
svg: `<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="#ffff00" stroke="#ff0000" stroke-width="2"><path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3Z" /><path d="M12 9v4" /><path d="M12 17h.01" /></svg>`,
frontend/src/components/MaplibreViewer.tsx
+214 -1
View File
@@ -158,16 +158,25 @@ import {
UavLabels,
EarthquakeLabels,
ThreatMarkers,
TelegramOsintMarkers,
} from '@/components/map/MapMarkers';
import type { DashboardData, Flight, KiwiSDR, MaplibreViewerProps, Scanner, Ship, SigintSignal } from '@/types/dashboard';
import { useDataKeys } from '@/hooks/useDataStore';
import { useInterpolation } from '@/components/map/hooks/useInterpolation';
import { useClusterLabels } from '@/components/map/hooks/useClusterLabels';
import { spreadAlertItems } from '@/utils/alertSpread';
import {
applyTelegramAlertAvoidance,
telegramClusterKey,
telegramClusterNearNewsAlert,
telegramMapPinCoords,
} from '@/components/map/geoJSONBuilders';
import { useViewportBounds } from '@/components/map/hooks/useViewportBounds';
import { getLiveDataBounds } from '@/lib/liveDataViewport';
import { MeasurementLayers } from '@/components/map/layers/MeasurementLayers';
import { buildCctvProxyUrl } from '@/lib/cctvProxy';
import { sanitizeSubmarineCables } from '@/lib/submarineCables';
import { CctvFullscreenModal } from '@/components/MaplibreViewer/CctvFullscreenModal';
import { SatellitePopup } from '@/components/MaplibreViewer/popups/SatellitePopup';
import { ShipPopup } from '@/components/MaplibreViewer/popups/ShipPopup';
@@ -176,6 +185,7 @@ import { CorrelationPopup } from '@/components/MaplibreViewer/popups/Correlation
import { WastewaterPopup } from '@/components/MaplibreViewer/popups/WastewaterPopup';
import { MilitaryBasePopup } from '@/components/MaplibreViewer/popups/MilitaryBasePopup';
import { RegionDossierPanel } from '@/components/MaplibreViewer/popups/RegionDossierPanel';
import { TelegramOsintPopup } from '@/components/MaplibreViewer/popups/TelegramOsintPopup';
import {
buildSentinelTileUrl,
hasSentinelCredentials,
@@ -294,6 +304,8 @@ const MAP_EXTRA_DATA_KEYS = [
'commercial_flights',
'correlations',
'crowdthreat',
'malware_threats',
'telegram_osint',
'datacenters',
'firms_fires',
'fishing_activity',
@@ -1156,6 +1168,30 @@ const MaplibreViewer = ({
const staticUapSightings = activeLayers.uap_sightings ? data?.uap_sightings : undefined;
const staticWastewater = activeLayers.wastewater ? data?.wastewater : undefined;
const staticCrowdthreat = activeLayers.crowdthreat ? data?.crowdthreat : undefined;
const staticMalwareThreats = activeLayers.malware_c2 ? data?.malware_threats?.threats : undefined;
const staticTelegramOsintPosts = activeLayers.telegram_osint
? data?.telegram_osint?.posts
: undefined;
const [submarineCablesGeoJSON, setSubmarineCablesGeoJSON] = useState<GeoJSON.FeatureCollection | null>(null);
useEffect(() => {
if (!activeLayers.submarine_cables) {
setSubmarineCablesGeoJSON(null);
return;
}
let cancelled = false;
fetch('/data/submarine-cables.json')
.then((r) => r.json())
.then((geo) => {
if (!cancelled) setSubmarineCablesGeoJSON(sanitizeSubmarineCables(geo));
})
.catch(() => {
if (!cancelled) setSubmarineCablesGeoJSON(null);
});
return () => {
cancelled = true;
};
}, [activeLayers.submarine_cables]);
const dynamicMapLayers = useDynamicMapLayersWorker(
{
@@ -1186,6 +1222,7 @@ const MaplibreViewer = ({
],
{
bounds: mapBounds,
serverBboxScoped: getLiveDataBounds() !== null,
dtSeconds: dtSeconds.current,
trackedIcaos: Array.from(trackedIcaoSet),
activeLayers: {
@@ -1247,6 +1284,8 @@ const MaplibreViewer = ({
uapSightings: staticUapSightings,
wastewater: staticWastewater,
crowdthreat: staticCrowdthreat,
malwareThreats: staticMalwareThreats,
telegramOsintPosts: staticTelegramOsintPosts,
},
[
staticCctv,
@@ -1270,6 +1309,9 @@ const MaplibreViewer = ({
staticUapSightings,
staticWastewater,
staticCrowdthreat,
staticMalwareThreats,
staticTelegramOsintPosts,
mapZoom,
],
{
bounds: mapBounds,
@@ -1293,6 +1335,8 @@ const MaplibreViewer = ({
uap_sightings: activeLayers.uap_sightings,
wastewater: activeLayers.wastewater,
crowdthreat: activeLayers.crowdthreat,
malware_c2: activeLayers.malware_c2,
telegram_osint: activeLayers.telegram_osint,
},
},
[
@@ -1316,6 +1360,8 @@ const MaplibreViewer = ({
activeLayers.uap_sightings,
activeLayers.wastewater,
activeLayers.crowdthreat,
activeLayers.malware_c2,
activeLayers.telegram_osint,
],
);
@@ -1351,8 +1397,15 @@ const MaplibreViewer = ({
uapSightingsGeoJSON,
wastewaterGeoJSON,
crowdthreatGeoJSON,
malwareGeoJSON,
telegramOsintGeoJSON,
} = staticMapLayers;
const telegramOsintGeoJSONPlaced = useMemo(
() => applyTelegramAlertAvoidance(telegramOsintGeoJSON, data?.news),
[telegramOsintGeoJSON, data?.news],
);
// Extract cluster label positions via shared hook
const shipClusters = useClusterLabels(mapRef, 'ships-clusters-layer', shipsGeoJSON);
const eqClusters = useClusterLabels(mapRef, 'eq-clusters-layer', earthquakesGeoJSON);
@@ -1659,6 +1712,9 @@ const MaplibreViewer = ({
wastewaterGeoJSON && 'wastewater-dot',
wastewaterGeoJSON && 'wastewater-layer',
crowdthreatGeoJSON && 'crowdthreat-layer',
malwareGeoJSON && 'malware-clusters',
malwareGeoJSON && 'malware-layer',
submarineCablesGeoJSON && 'submarine-cables-layer',
sarAnomaliesGeoJSON && 'sar-anomalies-layer',
sarAoisGeoJSON && 'sar-aois-fill',
aiIntelGeoJSON && 'ai-intel-clusters',
@@ -1731,6 +1787,9 @@ const MaplibreViewer = ({
useImperativeSource(mapForHook, 'uap-sightings-source', uapSightingsGeoJSON, 100);
useImperativeSource(mapForHook, 'wastewater-source', wastewaterGeoJSON, 100);
useImperativeSource(mapForHook, 'crowdthreat-source', crowdthreatGeoJSON, 100);
useImperativeSource(mapForHook, 'malware-source', malwareGeoJSON, 100);
useImperativeSource(mapForHook, 'telegram-osint-source', telegramOsintGeoJSONPlaced, 100);
useImperativeSource(mapForHook, 'submarine-cables-source', submarineCablesGeoJSON, 600);
useImperativeSource(mapForHook, 'ships', shipsGeoJSON, 75);
useImperativeSource(mapForHook, 'meshtastic-source', meshtasticGeoJSON, 60);
useImperativeSource(mapForHook, 'aprs-source', aprsGeoJSON, 60);
@@ -1761,7 +1820,7 @@ const MaplibreViewer = ({
return (
<div
className={`relative h-full w-full z-0 isolate ${selectedEntity && ['region_dossier', 'gdelt', 'liveuamap', 'news'].includes(selectedEntity.type) ? 'map-focus-active' : ''}`}
className={`relative h-full w-full z-0 isolate ${selectedEntity && ['region_dossier', 'gdelt', 'liveuamap', 'news', 'telegram_osint'].includes(selectedEntity.type) ? 'map-focus-active' : ''}`}
style={pinPlacementMode || sarAoiDropMode ? { cursor: 'crosshair' } : undefined}
>
<Map
@@ -3688,6 +3747,71 @@ const MaplibreViewer = ({
/>
</Source>
{/* Telegram OSINT — one pin per geocoded city; scroll posts in popup */}
<Source id="telegram-osint-source" type="geojson" data={EMPTY_FC}>
<Layer
id="telegram-osint-layer"
type="circle"
minzoom={4}
paint={{
'circle-radius': [
'interpolate',
['linear'],
['zoom'],
4,
['case', ['>', ['get', 'post_count'], 1], 14, 11],
8,
['case', ['>', ['get', 'post_count'], 1], 20, 16],
12,
['case', ['>', ['get', 'post_count'], 1], 26, 22],
],
'circle-color': '#ef4444',
'circle-stroke-width': 0,
'circle-stroke-color': '#fca5a5',
'circle-opacity': 0,
}}
/>
</Source>
{/* Malware C2 — abuse.ch Feodo + URLhaus */}
<Source id="malware-source" type="geojson" data={EMPTY_FC} cluster={true} clusterMaxZoom={6} clusterRadius={35}>
<Layer
id="malware-clusters"
type="circle"
filter={['has', 'point_count']}
paint={{
'circle-radius': ['step', ['get', 'point_count'], 12, 8, 16, 30, 22],
'circle-color': 'rgba(255, 61, 61, 0.75)',
'circle-stroke-width': 2,
'circle-stroke-color': '#ff3d3d',
}}
/>
<Layer
id="malware-layer"
type="circle"
filter={['!', ['has', 'point_count']]}
paint={{
'circle-radius': 5,
'circle-color': '#ff1744',
'circle-stroke-width': 1,
'circle-stroke-color': '#ff8a80',
}}
/>
</Source>
{/* Submarine cables — static TeleGeography GeoJSON */}
<Source id="submarine-cables-source" type="geojson" data={EMPTY_FC}>
<Layer
id="submarine-cables-layer"
type="line"
paint={{
'line-color': '#eab308',
'line-width': ['interpolate', ['linear'], ['zoom'], 2, 0.5, 6, 1.2, 10, 2],
'line-opacity': 0.75,
}}
/>
</Source>
{/* Ships — rendered below flights (water surface level) */}
<Source
id="ships"
@@ -4290,6 +4414,13 @@ const MaplibreViewer = ({
/>
)}
{activeLayers.telegram_osint && !isMapInteracting && telegramOsintGeoJSONPlaced?.features?.length ? (
<TelegramOsintMarkers
features={telegramOsintGeoJSONPlaced.features}
onEntityClick={onEntityClick}
/>
) : null}
{/* Satellite positions — mission-type icons */}
{/* satellites: data pushed imperatively */}
<Source id="satellites" type="geojson" data={EMPTY_FC}>
@@ -5428,6 +5559,66 @@ const MaplibreViewer = ({
);
})()}
{/* Earthquake popup */}
{selectedEntity?.type === 'earthquake' &&
(() => {
const extra = (selectedEntity.extra || {}) as Record<string, unknown>;
const idx = Number(selectedEntity.id);
const eq = Number.isFinite(idx)
? data?.earthquakes?.[idx]
: data?.earthquakes?.find((e) => e.id === String(selectedEntity.id));
const lat = typeof eq?.lat === 'number' ? eq.lat : Number(extra.lat);
const lng = typeof eq?.lng === 'number' ? eq.lng : Number(extra.lng);
if (!Number.isFinite(lat) || !Number.isFinite(lng)) return null;
const mag = eq?.mag ?? Number(extra.mag);
const place = eq?.place || String(extra.place || selectedEntity.name || 'Unknown location');
const accent = mag >= 6 ? '#ef4444' : mag >= 4.5 ? '#f97316' : '#eab308';
return (
<Popup
longitude={lng}
latitude={lat}
closeButton={false}
closeOnClick={false}
onClose={() => onEntityClick?.(null)}
className="threat-popup"
maxWidth="280px"
>
<div className="map-popup bg-[#1a1035] min-w-[200px]" style={{ borderColor: `${accent}66` }}>
<div className="map-popup-title pb-1" style={{ color: accent, borderBottom: `1px solid ${accent}33` }}>
M{Number.isFinite(mag) ? mag.toFixed(1) : '?'} — EARTHQUAKE
</div>
<div className="map-popup-row">
Location: <span className="text-white">{place}</span>
</div>
<div className="map-popup-row">
Coords:{' '}
<span className="text-white font-mono">
{lat.toFixed(3)}, {lng.toFixed(3)}
</span>
</div>
{oracleIntel?.found && (
<div className="mt-2 pt-2 border-t border-yellow-500/20">
<div className="text-[10px] font-mono text-yellow-500/80 tracking-wider mb-1">REGION INTEL</div>
<div className="text-[10px] font-mono text-white/70">
ORACLE: {oracleIntel.tier}
{oracleIntel.avg_sentiment != null && (
<span className="text-gray-400">
{' '}
· SENT {oracleIntel.avg_sentiment > 0 ? '+' : ''}
{oracleIntel.avg_sentiment.toFixed(2)}
</span>
)}
</div>
</div>
)}
<div className="mt-1.5 text-[9px] tracking-wider" style={{ color: `${accent}99` }}>
SEISMIC — USGS
</div>
</div>
</Popup>
);
})()}
{/* Volcano popup */}
{selectedEntity?.type === 'volcano' &&
(() => {
@@ -5521,6 +5712,28 @@ const MaplibreViewer = ({
return <FishingDestinationRoute vesselLat={event.lat} vesselLng={event.lng} destination={dest} />;
})()}
{(() => {
if (selectedEntity?.type !== 'telegram_osint' || !data?.telegram_osint?.posts) return null;
const allPosts = data.telegram_osint.posts;
const clusterPosts = allPosts.filter((p) => {
if (!p.coords || p.coords.length < 2) return false;
const key = telegramClusterKey(p.coords[0], p.coords[1]);
return key === selectedEntity.id || p.id === selectedEntity.id;
});
const anchor = clusterPosts[0]?.coords;
if (!anchor || anchor.length < 2) return null;
const avoidAlert = telegramClusterNearNewsAlert(anchor[0], anchor[1], data?.news);
const [pinLat, pinLng] = telegramMapPinCoords(anchor[0], anchor[1], avoidAlert);
return (
<TelegramOsintPopup
posts={clusterPosts}
lat={pinLat}
lng={pinLng}
onClose={() => onEntityClick?.(null)}
/>
);
})()}
{(() => {
if (selectedEntity?.type !== 'gdelt' || !data?.gdelt) return null;
const item = data.gdelt.find(
frontend/src/components/MaplibreViewer/popups/TelegramOsintPopup.tsx
+255
View File
@@ -0,0 +1,255 @@
'use client';
import React, { useMemo } from 'react';
import { Popup } from 'react-map-gl/maplibre';
import { Radio } from 'lucide-react';
import { useTranslation } from '@/i18n';
import { TELEGRAM_MARKER_OFFSET } from '@/components/map/geoJSONBuilders';
import { buildTelegramMediaProxyUrl } from '@/lib/telegramProxy';
import type { TelegramOsintPost } from '@/types/dashboard';
export interface TelegramOsintPopupProps {
posts: TelegramOsintPost[];
lat: number;
lng: number;
onClose: () => void;
}
function formatTime(pubDate?: string) {
if (!pubDate) return '';
try {
return new Date(pubDate).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
} catch {
return '';
}
}
function riskTheme(rs: number) {
if (rs >= 9) {
return {
hex: '#ef4444',
threatColor: 'text-red-400',
borderColor: 'border-red-700',
bgHeaderColor: 'bg-red-950/50',
bgClass: 'bg-red-950/20 border-red-500/30',
titleClass: 'text-cyan-300 font-bold',
badgeClass: 'bg-red-500/10 text-red-400 border-red-500/30',
};
}
if (rs >= 7) {
return {
hex: '#f97316',
threatColor: 'text-orange-400',
borderColor: 'border-orange-700',
bgHeaderColor: 'bg-orange-950/50',
bgClass: 'bg-orange-950/20 border-orange-500/30',
titleClass: 'text-cyan-300 font-bold',
badgeClass: 'bg-orange-500/10 text-orange-400 border-orange-500/30',
};
}
if (rs >= 4) {
return {
hex: '#eab308',
threatColor: 'text-yellow-400',
borderColor: 'border-yellow-800',
bgHeaderColor: 'bg-yellow-950/50',
bgClass: 'bg-yellow-950/20 border-yellow-500/30',
titleClass: 'text-cyan-300 font-bold',
badgeClass: 'bg-yellow-500/10 text-yellow-500 border-yellow-500/30',
};
}
return {
hex: '#22c55e',
threatColor: 'text-green-400',
borderColor: 'border-green-800',
bgHeaderColor: 'bg-green-950/50',
bgClass: 'bg-green-950/20 border-green-500/30',
titleClass: 'text-cyan-300 font-medium',
badgeClass: 'bg-green-500/10 text-green-400 border-green-500/30',
};
}
function postHeadline(post: TelegramOsintPost): string {
return String(post.title || post.description || 'Telegram intercept').trim();
}
function postDetail(post: TelegramOsintPost): string | null {
const title = String(post.title || '').trim();
const description = String(post.description || '').trim();
if (!description || description === title || description.startsWith(title)) return null;
const extra = description.startsWith(title) ? description.slice(title.length).trim() : description;
return extra || null;
}
function TelegramPostMedia({ post }: { post: TelegramOsintPost }) {
const { t } = useTranslation();
const proxyUrl = post.media_url ? buildTelegramMediaProxyUrl(post.media_url) : null;
let media: React.ReactNode = null;
if (post.media_type === 'video' && proxyUrl) {
media = (
<video
src={proxyUrl}
controls
playsInline
preload="metadata"
className="w-full max-h-52 bg-black"
/>
);
} else if (post.media_type === 'photo' && proxyUrl) {
media = (
// eslint-disable-next-line @next/next/no-img-element
<img src={proxyUrl} alt="" className="w-full max-h-52 object-contain bg-black" />
);
} else if (post.embed_url) {
media = (
<iframe
src={post.embed_url}
title={t('telegram.embedTitle')}
className="w-full"
height={240}
style={{ border: 'none' }}
loading="lazy"
referrerPolicy="no-referrer"
/>
);
}
if (!media) return null;
return (
<div className="mt-2 rounded-sm border border-cyan-900/40 overflow-hidden bg-black/70">
{media}
</div>
);
}
function TelegramPostCard({ post }: { post: TelegramOsintPost }) {
const { t } = useTranslation();
const rs = post.risk_score ?? 1;
const theme = riskTheme(rs);
const headline = postHeadline(post);
const detail = postDetail(post);
const isHigh = rs >= 8;
return (
<article
className={`p-2 rounded-sm border-l-[2px] border-r border-t border-b ${theme.bgClass} flex flex-col gap-1`}
>
<div className="flex items-center justify-between text-[12px] text-[var(--text-secondary)] uppercase tracking-widest">
<span className="font-bold flex items-center gap-1 text-white">
{isHigh && <span className="text-red-400 mr-1">BREAKING</span>}
&gt;_ {post.source || 'TELEGRAM'}
</span>
<span>[{formatTime(post.published)}]</span>
</div>
<h3 className={`text-[12px] leading-tight ${theme.titleClass}`}>{headline}</h3>
{detail ? (
<p className="text-[11px] text-[var(--text-muted)] leading-relaxed whitespace-pre-wrap">{detail}</p>
) : null}
<TelegramPostMedia post={post} />
<div className="flex items-center gap-1.5 mt-1 flex-wrap">
<span className={`text-[11px] font-bold font-mono px-1.5 py-0.5 rounded-sm border ${theme.badgeClass}`}>
{isHigh ? 'BREAKING' : `LVL: ${rs}/10`}
</span>
{post.link ? (
<a
href={post.link}
target="_blank"
rel="noopener noreferrer"
className="text-[11px] font-mono text-cyan-500 hover:text-cyan-300 transition-colors"
>
{t('telegram.openOriginal')}
</a>
) : null}
</div>
</article>
);
}
export function TelegramOsintPopup({ posts, lat, lng, onClose }: TelegramOsintPopupProps) {
const { t } = useTranslation();
const sortedPosts = useMemo(
() =>
[...posts].sort(
(a, b) =>
(b.risk_score ?? 0) - (a.risk_score ?? 0) ||
String(b.published || '').localeCompare(String(a.published || '')),
),
[posts],
);
const maxRisk = sortedPosts[0]?.risk_score ?? 1;
const header = riskTheme(maxRisk);
return (
<Popup
longitude={lng}
latitude={lat}
closeButton={false}
closeOnClick={false}
onClose={onClose}
anchor="bottom"
offset={TELEGRAM_MARKER_OFFSET}
className="threat-popup"
maxWidth="560px"
>
<div
className={`bg-[#080c12] border ${header.borderColor} rounded-lg flex flex-col font-mono overflow-hidden w-[min(520px,92vw)]`}
style={{
boxShadow: `0 0 60px ${header.hex}33, 0 0 160px ${header.hex}11, inset 0 1px 0 rgba(255,255,255,0.05)`,
}}
>
<div
className={`px-4 py-3 border-b ${header.borderColor}/60 ${header.bgHeaderColor} flex justify-between items-center shrink-0`}
>
<div className="flex items-center gap-2">
<Radio size={16} className={header.threatColor} />
<span className={`text-[13px] tracking-[0.25em] font-bold ${header.threatColor}`}>
TELEGRAM INTERCEPT
</span>
{maxRisk >= 8 && (
<span className="text-[9px] bg-red-500 text-white px-2 py-0.5 rounded-sm font-bold animate-pulse">
LIVE
</span>
)}
</div>
<div className="flex items-center gap-3">
<span className={`text-[12px] ${header.threatColor} font-bold`}>
ALERT LVL: {maxRisk}/10
</span>
<button
type="button"
onClick={onClose}
className="text-[var(--text-secondary)] hover:text-white text-lg leading-none px-1 hover:bg-white/10 rounded transition-colors"
aria-label="Close"
>
✕
</button>
</div>
</div>
<div className="px-3 py-2 border-b border-cyan-900/40 bg-black/40 shrink-0">
<div className="text-[11px] text-[var(--text-muted)] uppercase tracking-widest mb-1">
{t('telegram.postsAtLocation').replace('{count}', String(sortedPosts.length))}
</div>
<div className="p-2 bg-black/60 border border-amber-700/40 rounded-sm text-[11px] text-amber-100/90 leading-relaxed relative overflow-hidden">
<div className="absolute top-0 left-0 w-[2px] h-full bg-amber-500/80" />
<span className="font-bold text-amber-300">&gt;_ SYS.NOTICE: </span>
{t('telegram.disclaimer')}
</div>
</div>
<div className="overflow-y-auto styled-scrollbar flex flex-col gap-2 p-3 max-h-[min(420px,55vh)]">
{sortedPosts.map((post) => (
<TelegramPostCard key={post.id} post={post} />
))}
</div>
</div>
</Popup>
);
}
frontend/src/components/NewsFeed.tsx
+19 -1
View File
@@ -321,7 +321,7 @@ function EmissionsEstimateBlock({ flight }: { flight: any }) {
);
}
function NewsFeedInner({ selectedEntity, regionDossier, regionDossierLoading, onArticleClick }: { selectedEntity?: SelectedEntity | null, regionDossier?: RegionDossier | null, regionDossierLoading?: boolean, onArticleClick?: (idx: number, lat?: number, lng?: number, title?: string) => void }) {
function NewsFeedInner({ selectedEntity, regionDossier, regionDossierLoading, onArticleClick, onExpandEntityGraph }: { selectedEntity?: SelectedEntity | null, regionDossier?: RegionDossier | null, regionDossierLoading?: boolean, onArticleClick?: (idx: number, lat?: number, lng?: number, title?: string) => void, onExpandEntityGraph?: () => void }) {
const data = useDataKeys([
'news', 'fimi', 'commercial_flights', 'private_flights', 'private_jets',
'military_flights', 'tracked_flights', 'ships', 'gdelt', 'liveuamap',
@@ -1097,6 +1097,15 @@ function NewsFeedInner({ selectedEntity, regionDossier, regionDossierLoading, on
</a>
</div>
)}
{onExpandEntityGraph && (
<button
type="button"
onClick={onExpandEntityGraph}
className="w-full py-1.5 text-[10px] font-mono tracking-wider border border-cyan-700/40 text-cyan-400 hover:bg-cyan-950/30 transition-colors"
>
INTEL GRAPH →
</button>
)}
</div>
</motion.div>
)
@@ -1206,6 +1215,15 @@ function NewsFeedInner({ selectedEntity, regionDossier, regionDossierLoading, on
/>
</div>
)}
{onExpandEntityGraph && (
<button
type="button"
onClick={onExpandEntityGraph}
className="w-full py-1.5 text-[10px] font-mono tracking-wider border border-cyan-700/40 text-cyan-400 hover:bg-cyan-950/30 transition-colors"
>
INTEL GRAPH →
</button>
)}
</div>
</motion.div>
)
frontend/src/components/OnboardingModal.tsx
+34 -6
View File
@@ -38,6 +38,20 @@ const API_GUIDES = [
url: 'https://aisstream.io/authenticate',
color: 'blue',
},
{
name: 'Global Fishing Watch',
icon: <Ship size={14} className="text-teal-400" />,
required: false,
description:
'Fishing-vessel activity events for the Fishing Activity map layer. Optional but recommended for maritime OSINT.',
steps: [
'Create a free account at globalfishingwatch.org',
'Open Our APIs and create an API token',
'Paste the token into Quick Local Setup above or Settings → API Keys → Maritime',
],
url: 'https://globalfishingwatch.org/our-apis/',
color: 'teal',
},
];
const FREE_SOURCES = [
@@ -65,6 +79,7 @@ const OnboardingModal = React.memo(function OnboardingModal({
OPENSKY_CLIENT_ID: '',
OPENSKY_CLIENT_SECRET: '',
AIS_API_KEY: '',
GFW_API_TOKEN: '',
});
const [setupSaving, setSetupSaving] = useState(false);
const [setupMsg, setSetupMsg] = useState<{ type: 'ok' | 'err'; text: string } | null>(null);
@@ -110,7 +125,12 @@ const OnboardingModal = React.memo(function OnboardingModal({
if (!res.ok || data?.ok === false) {
throw new Error(data?.detail || 'Could not save API keys.');
}
setSetupKeys({ OPENSKY_CLIENT_ID: '', OPENSKY_CLIENT_SECRET: '', AIS_API_KEY: '' });
setSetupKeys({
OPENSKY_CLIENT_ID: '',
OPENSKY_CLIENT_SECRET: '',
AIS_API_KEY: '',
GFW_API_TOKEN: '',
});
setSetupMsg({ type: 'ok', text: 'Keys saved locally. Restart or refresh feeds to use them.' });
} catch (error) {
setSetupMsg({
@@ -557,8 +577,9 @@ const OnboardingModal = React.memo(function OnboardingModal({
</p>
<p className="text-sm text-[var(--text-secondary)] font-mono leading-relaxed">
OpenSky Network and AIS Stream are the free keys that make ShadowBroker
useful immediately: live aircraft and vessel tracking. Paste them below or
use Settings later; secrets stay on the local backend.
useful immediately: live aircraft and vessel tracking. Global Fishing Watch
unlocks the fishing-activity layer. Paste them below or use Settings later;
secrets stay on the local backend.
</p>
</div>
</div>
@@ -578,6 +599,7 @@ const OnboardingModal = React.memo(function OnboardingModal({
['OPENSKY_CLIENT_ID', 'OpenSky Client ID'],
['OPENSKY_CLIENT_SECRET', 'OpenSky Client Secret'],
['AIS_API_KEY', 'AIS Stream API Key'],
['GFW_API_TOKEN', 'Global Fishing Watch API Token (optional)'],
].map(([key, label]) => (
<input
key={key}
@@ -618,9 +640,15 @@ const OnboardingModal = React.memo(function OnboardingModal({
<div className="flex items-center gap-2">
{api.icon}
<span className="text-xs font-mono text-white font-bold">{api.name}</span>
<span className="text-[12px] font-mono px-1.5 py-0.5 border border-yellow-500/30 text-yellow-400 bg-yellow-950/20">
REQUIRED
</span>
{api.required ? (
<span className="text-[12px] font-mono px-1.5 py-0.5 border border-yellow-500/30 text-yellow-400 bg-yellow-950/20">
REQUIRED
</span>
) : (
<span className="text-[12px] font-mono px-1.5 py-0.5 border border-teal-500/30 text-teal-300 bg-teal-950/20">
OPTIONAL
</span>
)}
</div>
<a
href={api.url}
frontend/src/components/ReconPanel.tsx
+176
View File
@@ -0,0 +1,176 @@
'use client';
import React, { useCallback, useState } from 'react';
import { Loader2, Minus, Plus, Radar, RefreshCw, Search, Shield } from 'lucide-react';
import { API_BASE } from '@/lib/api';
import { useTranslation } from '@/i18n';
import ReconResults from '@/components/ReconResults';
type TabId =
| 'ip'
| 'dns'
| 'whois'
| 'certs'
| 'threats'
| 'bgp'
| 'sanctions'
| 'cve'
| 'mac'
| 'github'
| 'leaks'
| 'sweep';
const TABS: Array<{
id: TabId;
label: string;
param: string;
path: string;
optional?: boolean;
}> = [
{ id: 'ip', label: 'IP LOOKUP', param: 'ip', path: 'ip' },
{ id: 'dns', label: 'DNS', param: 'domain', path: 'dns' },
{ id: 'whois', label: 'WHOIS / RDAP', param: 'domain', path: 'whois' },
{ id: 'certs', label: 'CERTS', param: 'domain', path: 'certs' },
{ id: 'threats', label: 'THREATS', param: 'query', path: 'threats', optional: true },
{ id: 'bgp', label: 'BGP / ASN', param: 'query', path: 'bgp' },
{ id: 'sanctions', label: 'OFAC SDN', param: 'query', path: 'sanctions' },
{ id: 'cve', label: 'CVE', param: 'cve', path: 'cve' },
{ id: 'mac', label: 'MAC', param: 'mac', path: 'mac' },
{ id: 'github', label: 'GITHUB', param: 'username', path: 'github' },
{ id: 'leaks', label: 'LEAKS', param: 'email', path: 'leaks' },
{ id: 'sweep', label: 'IP SWEEP', param: 'ip', path: 'sweep' },
];
export default function ReconPanel() {
const { t } = useTranslation();
const [isMinimized, setIsMinimized] = useState(true);
const [activeTab, setActiveTab] = useState<TabId>('ip');
const [query, setQuery] = useState('');
const [loading, setLoading] = useState(false);
const [error, setError] = useState('');
const [results, setResults] = useState<unknown>(null);
const active = TABS.find((tab) => tab.id === activeTab);
const runLookup = useCallback(async () => {
if (!active || loading) return;
if (!active.optional && !query.trim()) return;
setLoading(true);
setError('');
setResults(null);
try {
if (activeTab === 'sweep') {
const res = await fetch(`${API_BASE}/api/osint/sweep/scan`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ ip: query.trim(), cidr: 24 }),
});
const data = await res.json();
if (!res.ok) throw new Error(data.detail || data.error || `HTTP ${res.status}`);
setResults(data);
} else {
const params = new URLSearchParams();
if (query.trim()) params.set(active.param, query.trim());
const res = await fetch(`${API_BASE}/api/osint/${active.path}?${params}`);
const data = await res.json();
if (!res.ok) throw new Error(data.detail || data.error || `HTTP ${res.status}`);
setResults(data);
}
} catch (err) {
setError(err instanceof Error ? err.message : 'Lookup failed');
} finally {
setLoading(false);
}
}, [active, activeTab, query, loading]);
return (
<div className="pointer-events-auto flex-shrink-0 border border-cyan-700/40 bg-black/75 backdrop-blur-sm shadow-[0_0_18px_rgba(34,211,238,0.10)]">
<div
className="flex items-center justify-between border-b border-cyan-700/30 bg-cyan-950/20 px-3 py-2.5 cursor-pointer hover:bg-cyan-950/40 transition-colors"
onClick={() => setIsMinimized((prev) => !prev)}
>
<div className="flex items-center gap-2">
<Radar size={16} className="text-cyan-400" />
<span className="text-[12px] font-mono font-bold tracking-widest text-cyan-400">
{t('recon.title').toUpperCase()}
</span>
</div>
<div className="flex items-center gap-2">
{isMinimized ? (
<Plus size={16} className="text-cyan-400" />
) : (
<Minus size={16} className="text-cyan-400" />
)}
</div>
</div>
{!isMinimized && (
<div className="px-3 py-2 space-y-2">
<div className="flex items-center gap-1.5 text-[11px] font-mono">
<select
value={activeTab}
onChange={(e) => {
setActiveTab(e.target.value as TabId);
setResults(null);
setError('');
}}
className="flex-1 border border-cyan-900/50 bg-black/70 px-2 py-1 text-[11px] font-mono text-cyan-300 tracking-[0.12em] outline-none transition-colors focus:border-cyan-500/60"
>
{TABS.map((tab) => (
<option key={tab.id} value={tab.id}>
{tab.label}
</option>
))}
</select>
<button
type="button"
onClick={() => {
setQuery('');
setResults(null);
setError('');
}}
title="Clear"
className="text-cyan-600 transition-colors hover:text-cyan-400 p-0.5"
>
<RefreshCw size={11} />
</button>
</div>
<div className="flex items-center gap-1">
<input
value={query}
onChange={(e) => setQuery(e.target.value)}
onKeyDown={(e) => e.key === 'Enter' && runLookup()}
placeholder={active?.param || 'query'}
className="flex-1 border border-cyan-900/50 bg-black/70 px-2 py-1 text-[11px] font-mono text-cyan-300 outline-none transition-colors focus:border-cyan-500/60 placeholder:text-cyan-800"
/>
<button
type="button"
onClick={runLookup}
disabled={loading}
className="border border-cyan-600/40 px-2 py-1 text-[10px] font-mono tracking-wider text-cyan-400 transition-colors hover:border-cyan-500/70 disabled:opacity-40 flex items-center gap-1"
>
{loading ? <Loader2 size={12} className="animate-spin" /> : <Search size={12} />}
RUN
</button>
</div>
<div className="flex items-center gap-1.5 text-[10px] font-mono text-cyan-600 tracking-wider">
<Shield size={10} />
<span>{t('recon.proxyNote')}</span>
</div>
{error && (
<div className="border border-red-500/30 bg-red-950/20 px-2 py-1.5 text-[11px] font-mono text-red-400">
{error}
</div>
)}
{results != null && <ReconResults tabId={activeTab} results={results} />}
</div>
)}
</div>
);
}
frontend/src/components/ReconResults.tsx
+416
View File
@@ -0,0 +1,416 @@
'use client';
import React from 'react';
import { ExternalLink } from 'lucide-react';
function asRecord(value: unknown): Record<string, unknown> | null {
return value && typeof value === 'object' && !Array.isArray(value)
? (value as Record<string, unknown>)
: null;
}
function asArray(value: unknown): unknown[] {
return Array.isArray(value) ? value : [];
}
function fmt(value: unknown): string {
if (value == null || value === '') return '—';
if (typeof value === 'boolean') return value ? 'YES' : 'NO';
if (typeof value === 'number') return String(value);
if (typeof value === 'string') return value;
return JSON.stringify(value);
}
function DossierShell({ title, children }: { title?: string; children: React.ReactNode }) {
return (
<div className="max-h-52 overflow-y-auto styled-scrollbar border border-cyan-900/40 bg-black/60">
{title && (
<div className="border-b border-cyan-900/40 bg-cyan-950/25 px-2 py-1.5 text-[10px] font-mono font-bold tracking-[0.2em] text-cyan-400">
{title}
</div>
)}
<div className="px-2 py-1.5 space-y-0">{children}</div>
</div>
);
}
function DossierRow({
label,
value,
href,
highlight,
}: {
label: string;
value: React.ReactNode;
href?: string;
highlight?: 'red' | 'amber' | 'green' | 'cyan';
}) {
const tone =
highlight === 'red'
? 'text-red-300'
: highlight === 'amber'
? 'text-amber-300'
: highlight === 'green'
? 'text-green-300'
: 'text-cyan-200';
const content =
href && typeof value === 'string' ? (
<a
href={href}
target="_blank"
rel="noopener noreferrer"
className={`${tone} hover:underline inline-flex items-center gap-1 justify-end`}
>
{value}
<ExternalLink size={9} className="shrink-0 opacity-70" />
</a>
) : (
<span className={`${tone} text-right break-all leading-snug`}>{value}</span>
);
return (
<div className="flex justify-between items-start gap-3 border-b border-cyan-900/25 py-1 last:border-0">
<span className="text-[10px] font-mono text-cyan-600 tracking-wider shrink-0 pt-0.5">
{label}
</span>
<span className="text-[11px] font-mono min-w-0">{content}</span>
</div>
);
}
function SectionLabel({ children }: { children: React.ReactNode }) {
return (
<div className="text-[9px] font-mono tracking-[0.22em] text-cyan-500/80 pt-2 pb-0.5">
{children}
</div>
);
}
function GitHubDossier({ data }: { data: Record<string, unknown> }) {
const profile = asRecord(data.profile) || {};
const repos = asArray(data.repos) as Array<Record<string, unknown>>;
const displayName = fmt(profile.name) !== '—' ? String(profile.name) : fmt(data.username);
return (
<DossierShell title="GITHUB DOSSIER">
<DossierRow label="HANDLE" value={`@${fmt(data.username)}`} highlight="cyan" />
<DossierRow label="NAME" value={displayName} />
{profile.bio ? <DossierRow label="BIO" value={fmt(profile.bio)} /> : null}
{profile.location ? <DossierRow label="LOCATION" value={fmt(profile.location)} /> : null}
{profile.company ? <DossierRow label="COMPANY" value={fmt(profile.company)} /> : null}
<DossierRow label="FOLLOWERS" value={fmt(profile.followers)} />
<DossierRow label="PUBLIC REPOS" value={fmt(profile.public_repos)} />
{profile.created_at ? (
<DossierRow
label="MEMBER SINCE"
value={new Date(String(profile.created_at)).toLocaleDateString()}
/>
) : null}
{profile.html_url ? (
<DossierRow label="PROFILE" value="Open on GitHub" href={String(profile.html_url)} />
) : null}
{repos.length > 0 && (
<>
<SectionLabel>RECENT REPOSITORIES</SectionLabel>
{repos.slice(0, 6).map((repo) => (
<DossierRow
key={String(repo.name)}
label={String(repo.language || 'REPO')}
value={`${repo.name}${repo.stars != null ? ` · ★${repo.stars}` : ''}`}
/>
))}
</>
)}
</DossierShell>
);
}
function IpDossier({ data }: { data: Record<string, unknown> }) {
const geo = asRecord(data.geo) || {};
const rep = asRecord(data.reputation) || {};
const sanctions = asRecord(data.sanctions_match);
const risk = String(rep.risk_level || 'UNKNOWN');
return (
<DossierShell title="IP DOSSIER">
<DossierRow label="TARGET" value={fmt(data.ip)} highlight="cyan" />
<DossierRow
label="LOCATION"
value={[geo.city, geo.region, geo.country].filter(Boolean).join(', ') || '—'}
/>
<DossierRow label="ISP" value={fmt(geo.isp)} />
<DossierRow label="ORG" value={fmt(geo.org)} />
<DossierRow label="ASN" value={fmt(geo.as_number)} />
<DossierRow
label="RISK"
value={risk}
highlight={risk === 'HIGH' ? 'red' : risk === 'MEDIUM' ? 'amber' : 'green'}
/>
<DossierRow label="PROXY" value={fmt(rep.is_proxy)} />
<DossierRow label="HOSTING" value={fmt(rep.is_hosting)} />
{sanctions ? (
<DossierRow
label="SANCTIONS"
value={`${asArray(sanctions.hits).length} OFAC hit(s)`}
highlight="red"
/>
) : null}
</DossierShell>
);
}
function DnsDossier({ data }: { data: Record<string, unknown> }) {
const summary = asRecord(data.summary) || {};
return (
<DossierShell title="DNS DOSSIER">
<DossierRow label="DOMAIN" value={fmt(data.domain)} highlight="cyan" />
<DossierRow label="A RECORDS" value={asArray(summary.ip_addresses).join(', ') || '—'} />
<DossierRow label="MAIL (MX)" value={asArray(summary.mail_servers).join(', ') || '—'} />
<DossierRow label="NAMESERVERS" value={asArray(summary.nameservers).join(', ') || '—'} />
<DossierRow label="TOTAL RECORDS" value={fmt(summary.total_records)} />
</DossierShell>
);
}
function WhoisDossier({ data }: { data: Record<string, unknown> }) {
const rdap = asRecord(data.rdap) || {};
const http = asRecord(data.http) || {};
const score = asRecord(data.security_score) || {};
const entity = asRecord(asArray(rdap.entities)[0]);
return (
<DossierShell title="WHOIS / RDAP DOSSIER">
<DossierRow label="DOMAIN" value={fmt(data.domain)} highlight="cyan" />
<DossierRow label="REGISTRAR" value={fmt(entity?.org || entity?.name)} />
<DossierRow label="REGISTERED" value={fmt(data.registration)} />
<DossierRow label="EXPIRES" value={fmt(data.expiration)} />
<DossierRow label="LAST CHANGED" value={fmt(data.last_changed)} />
<DossierRow label="HTTP STATUS" value={fmt(http.status)} />
<DossierRow
label="SECURITY"
value={score.grade ? `${score.grade} (${score.score}/${score.max})` : '—'}
highlight={score.grade === 'A' ? 'green' : score.grade === 'F' ? 'red' : 'amber'}
/>
<DossierRow label="NAMESERVERS" value={asArray(rdap.nameservers).slice(0, 4).join(', ') || '—'} />
</DossierShell>
);
}
function CertsDossier({ data }: { data: Record<string, unknown> }) {
const subs = asArray(data.subdomains) as string[];
const certs = asArray(data.certificates) as Array<Record<string, unknown>>;
return (
<DossierShell title="CERTIFICATE DOSSIER">
<DossierRow label="DOMAIN" value={fmt(data.domain)} highlight="cyan" />
<DossierRow label="CERTS FOUND" value={fmt(data.total_found)} />
<DossierRow label="SUBDOMAINS" value={subs.length ? `${subs.length} discovered` : '—'} />
{subs.slice(0, 5).map((sub) => (
<DossierRow key={sub} label="HOST" value={sub} />
))}
{certs[0] ? (
<DossierRow label="LATEST CN" value={fmt(certs[0].common_name)} />
) : null}
</DossierShell>
);
}
function SanctionsDossier({ data }: { data: Record<string, unknown> }) {
const matches = asArray(data.matches) as Array<Record<string, unknown>>;
return (
<DossierShell title="SANCTIONS DOSSIER">
<DossierRow label="QUERY" value={fmt(data.query)} highlight="cyan" />
<DossierRow label="MATCHES" value={fmt(data.total)} highlight={matches.length ? 'red' : 'green'} />
<DossierRow label="SOURCE" value={fmt(data.source)} />
{matches.slice(0, 8).map((hit, i) => (
<DossierRow
key={`${hit.id || i}`}
label={String(hit.schema || 'ENTITY').toUpperCase()}
value={fmt(hit.caption || hit.name || hit.id)}
highlight="red"
/>
))}
</DossierShell>
);
}
function CveDossier({ data }: { data: Record<string, unknown> }) {
return (
<DossierShell title="CVE DOSSIER">
<DossierRow label="CVE" value={fmt(data.id)} highlight="cyan" />
{'cvss' in data ? <DossierRow label="CVSS" value={fmt(data.cvss)} /> : null}
<div className="pt-1 text-[11px] font-mono text-cyan-200/90 leading-relaxed">
{fmt(data.description)}
</div>
</DossierShell>
);
}
function LeaksDossier({ data }: { data: Record<string, unknown> }) {
const sources = asArray(data.sources);
const found = Boolean(data.found);
return (
<DossierShell title="BREACH DOSSIER">
<DossierRow label="EMAIL" value={fmt(data.email)} highlight="cyan" />
<DossierRow
label="EXPOSED"
value={found ? 'YES' : 'NO'}
highlight={found ? 'red' : 'green'}
/>
{sources.length > 0 ? (
<DossierRow label="SOURCES" value={sources.map((s) => fmt(s)).join(', ')} highlight="red" />
) : null}
</DossierShell>
);
}
function MacDossier({ data }: { data: Record<string, unknown> }) {
return (
<DossierShell title="MAC DOSSIER">
<DossierRow label="MAC" value={fmt(data.mac)} highlight="cyan" />
<DossierRow label="VENDOR" value={fmt(data.vendor)} />
</DossierShell>
);
}
function ThreatsDossier({ data }: { data: Record<string, unknown> }) {
const otx = asRecord(data.otx) || {};
const pulses = asArray(data.pulses) as Array<Record<string, unknown>>;
const level = String(data.threat_level || 'LOW');
return (
<DossierShell title="THREAT INTEL DOSSIER">
<DossierRow
label="THREAT LEVEL"
value={level}
highlight={level === 'HIGH' ? 'red' : level === 'MEDIUM' ? 'amber' : 'green'}
/>
{'pulse_count' in otx ? <DossierRow label="OTX PULSES" value={fmt(otx.pulse_count)} /> : null}
{'tor_exit_node' in data ? (
<DossierRow label="TOR EXIT" value={fmt(data.tor_exit_node)} highlight="amber" />
) : null}
{pulses.slice(0, 4).map((pulse, i) => (
<div key={i} className="border-b border-cyan-900/25 py-1 last:border-0">
<div className="text-[10px] font-mono text-cyan-300 leading-snug">{fmt(pulse.name)}</div>
{pulse.adversary ? (
<div className="text-[9px] font-mono text-cyan-600 mt-0.5">{fmt(pulse.adversary)}</div>
) : null}
</div>
))}
</DossierShell>
);
}
function BgpDossier({ data }: { data: Record<string, unknown> }) {
const asn = asRecord(data.asn) || {};
const ip = asRecord(data.ip) || {};
const prefixes = asRecord(data.prefixes) || {};
return (
<DossierShell title="BGP DOSSIER">
<DossierRow label="QUERY" value={fmt(data.query)} highlight="cyan" />
{data.type === 'asn' ? (
<>
<DossierRow label="ASN" value={fmt(asn.asn)} />
<DossierRow label="NAME" value={fmt(asn.name)} />
<DossierRow label="COUNTRY" value={fmt(asn.country_code)} />
<DossierRow label="PREFIXES V4" value={fmt(prefixes.total_v4)} />
</>
) : (
<>
<DossierRow label="PREFIX" value={fmt(ip.prefix)} />
<DossierRow label="ASN" value={fmt(ip.asn)} />
<DossierRow label="NAME" value={fmt(ip.name)} />
</>
)}
</DossierShell>
);
}
function SweepDossier({ data }: { data: Record<string, unknown> }) {
const summary = asRecord(data.summary) || {};
const devices = asArray(data.devices) as Array<Record<string, unknown>>;
return (
<DossierShell title="SWEEP DOSSIER">
<DossierRow label="SCANNED" value={fmt(summary.total_hosts)} />
<DossierRow
label="RESPONSIVE"
value={fmt(summary.total_responsive)}
highlight={Number(summary.total_responsive) > 0 ? 'amber' : 'green'}
/>
<DossierRow label="DURATION" value={data.sweep_time_ms != null ? `${data.sweep_time_ms} ms` : '—'} />
{devices.slice(0, 8).map((device) => (
<DossierRow
key={String(device.ip)}
label={fmt(device.ip)}
value={[
asArray(device.ports).length ? `ports ${asArray(device.ports).join(',')}` : '',
asArray(device.vulns).length ? `${asArray(device.vulns).length} vuln(s)` : '',
]
.filter(Boolean)
.join(' · ') || 'responsive'}
highlight={asArray(device.vulns).length ? 'red' : undefined}
/>
))}
</DossierShell>
);
}
function GenericDossier({ data }: { data: Record<string, unknown> }) {
const rows = Object.entries(data).filter(([key]) => key !== 'timestamp');
return (
<DossierShell title="RECON DOSSIER">
{rows.slice(0, 12).map(([key, value]) => (
<DossierRow
key={key}
label={key.replace(/_/g, ' ').toUpperCase()}
value={
typeof value === 'object' && value !== null
? Array.isArray(value)
? `${value.length} item(s)`
: 'See details'
: fmt(value)
}
/>
))}
</DossierShell>
);
}
export default function ReconResults({
tabId,
results,
}: {
tabId: string;
results: unknown;
}) {
const data = asRecord(results);
if (!data) return null;
switch (tabId) {
case 'github':
return <GitHubDossier data={data} />;
case 'ip':
return <IpDossier data={data} />;
case 'dns':
return <DnsDossier data={data} />;
case 'whois':
return <WhoisDossier data={data} />;
case 'certs':
return <CertsDossier data={data} />;
case 'sanctions':
return <SanctionsDossier data={data} />;
case 'cve':
return <CveDossier data={data} />;
case 'leaks':
return <LeaksDossier data={data} />;
case 'mac':
return <MacDossier data={data} />;
case 'threats':
return <ThreatsDossier data={data} />;
case 'bgp':
return <BgpDossier data={data} />;
case 'sweep':
return <SweepDossier data={data} />;
default:
return <GenericDossier data={data} />;
}
}
frontend/src/components/ScmPanel.tsx
+137
View File
@@ -0,0 +1,137 @@
'use client';
import React, { useCallback, useEffect, useState } from 'react';
import { AlertTriangle, Minus, Plus, RefreshCw, Target } from 'lucide-react';
import { API_BASE } from '@/lib/api';
import { useTranslation } from '@/i18n';
interface Supplier {
id: string;
name: string;
city: string;
country: string;
category: string;
risk_level: string;
active_threats: string[];
}
interface ScmPayload {
suppliers: Supplier[];
critical_count: number;
total: number;
timestamp?: string;
}
interface Props {
/** Only evaluate threats when the map layer is enabled. */
layerEnabled?: boolean;
}
export default function ScmPanel({ layerEnabled = false }: Props) {
const { t } = useTranslation();
const [isMinimized, setIsMinimized] = useState(true);
const [data, setData] = useState<ScmPayload | null>(null);
const [loading, setLoading] = useState(false);
const refresh = useCallback(async () => {
if (!layerEnabled) {
setData(null);
return;
}
setLoading(true);
try {
const res = await fetch(`${API_BASE}/api/scm-suppliers`);
if (res.ok) setData(await res.json());
} catch {
/* non-fatal */
} finally {
setLoading(false);
}
}, [layerEnabled]);
useEffect(() => {
refresh();
if (!layerEnabled) return undefined;
const id = setInterval(refresh, 5 * 60_000);
return () => clearInterval(id);
}, [refresh, layerEnabled]);
const critical = (data?.suppliers || []).filter(
(s) => s.risk_level === 'CRITICAL' || s.risk_level === 'HIGH',
);
return (
<div className="pointer-events-auto flex-shrink-0 border border-cyan-700/40 bg-black/75 backdrop-blur-sm shadow-[0_0_18px_rgba(34,211,238,0.10)]">
<div
className="flex items-center justify-between border-b border-cyan-700/30 bg-cyan-950/20 px-3 py-2.5 cursor-pointer hover:bg-cyan-950/40 transition-colors"
onClick={() => setIsMinimized((prev) => !prev)}
>
<div className="flex items-center gap-2">
<Target size={16} className="text-cyan-400" />
<span className="text-[12px] font-mono font-bold tracking-widest text-cyan-400">
{t('scm.title').toUpperCase()}
</span>
{layerEnabled && critical.length > 0 && (
<span className="text-[11px] font-mono px-1.5 py-0.5 bg-red-900/30 border border-red-700/40 text-red-300 tracking-wider">
{critical.length} ALERT{critical.length === 1 ? '' : 'S'}
</span>
)}
</div>
<div className="flex items-center gap-2">
<button
type="button"
onClick={(e) => {
e.stopPropagation();
refresh();
}}
title="Refresh SCM overlay"
className="text-cyan-600 transition-colors hover:text-cyan-400 p-0.5"
>
<RefreshCw size={11} className={loading ? 'animate-spin' : ''} />
</button>
{isMinimized ? (
<Plus size={16} className="text-cyan-400" />
) : (
<Minus size={16} className="text-cyan-400" />
)}
</div>
</div>
{!isMinimized && (
<div className="px-3 py-2 max-h-44 overflow-y-auto styled-scrollbar space-y-1.5">
{!layerEnabled ? (
<div className="text-[11px] font-mono tracking-wider text-cyan-600/70 py-1">
{t('scm.layerOff')}
</div>
) : critical.length === 0 ? (
<div className="text-[11px] font-mono tracking-wider text-cyan-500/80 py-1">
{t('scm.allClear')}
</div>
) : (
critical.map((s) => (
<div key={s.id} className="border border-red-700/30 bg-red-950/15 px-2 py-1.5">
<div className="flex items-start justify-between gap-2">
<span className="text-[11px] font-mono font-bold tracking-wide text-red-300 leading-tight">
{s.name}
</span>
<span className="text-[10px] font-mono tracking-widest text-red-400 shrink-0">
{s.risk_level}
</span>
</div>
<div className="text-[10px] font-mono text-cyan-600/80 mt-0.5">
{s.city}, {s.country}
</div>
{s.active_threats.map((threat) => (
<div key={threat} className="flex items-center gap-1.5 text-[10px] font-mono text-amber-400/90 mt-1 tracking-wide">
<AlertTriangle size={10} className="shrink-0" />
{threat}
</div>
))}
</div>
))
)}
</div>
)}
</div>
);
}
frontend/src/components/WorldviewLeftPanel.tsx
+44 -1
View File
@@ -110,6 +110,11 @@ const FRESHNESS_MAP: Record<string, string> = {
ai_intel: '',
crowdthreat: 'crowdthreat',
road_corridor_trends: 'road_corridor_trends',
malware_c2: 'malware_threats',
submarine_cables: '',
scm_suppliers: 'scm_suppliers',
cyber_threats: 'cyber_threats',
telegram_osint: 'telegram_osint',
};
// POTUS fleet ICAO hex codes for client-side filtering
@@ -1187,6 +1192,34 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
count: data?.trains?.length || 0,
icon: TrainFront,
},
{
id: 'submarine_cables',
name: t('layers.submarineCables'),
source: 'TeleGeography (static)',
count: null,
icon: Globe,
},
{
id: 'malware_c2',
name: t('layers.malwareC2'),
source: 'abuse.ch',
count: data?.malware_threats?.total || 0,
icon: Shield,
},
{
id: 'scm_suppliers',
name: t('layers.scmSuppliers'),
source: 'Tier 1/2 overlay',
count: data?.scm_suppliers?.critical_count || 0,
icon: Truck,
},
{
id: 'cyber_threats',
name: t('layers.cyberThreats'),
source: 'CISA KEV',
count: data?.cyber_threats?.threats?.length || 0,
icon: AlertTriangle,
},
],
},
{
@@ -1275,6 +1308,13 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
count: data?.gdelt?.length || 0,
icon: Activity,
},
{
id: 'telegram_osint',
name: t('layers.telegramOsint'),
source: 't.me public channels',
count: data?.telegram_osint?.geolocated || 0,
icon: Radio,
},
{
id: 'crowdthreat',
name: t('layers.crowdThreat'),
@@ -1317,7 +1357,10 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
const [expandedSections, setExpandedSections] = useState<Record<string, boolean>>(() => {
const initial: Record<string, boolean> = {};
sections.forEach((s) => {
initial[s.label] = false;
// Keep high-traffic intel overlays visible on first paint (GDELT, Telegram, etc.)
initial[s.label] = s.layers.some((l) =>
['global_incidents', 'telegram_osint', 'ukraine_frontline'].includes(l.id),
);
});
return initial;
});
frontend/src/components/map/MapMarkers.tsx
+58
View File
@@ -2,6 +2,7 @@ import React from 'react';
import { Marker } from 'react-map-gl/maplibre';
import type { Earthquake, SelectedEntity, Ship, TrackedFlight, UAV } from '@/types/dashboard';
import type { SpreadAlertItem } from '@/utils/alertSpread';
import { TELEGRAM_MARKER_OFFSET } from '@/components/map/geoJSONBuilders';
// Shared monospace label style base
const LABEL_BASE: React.CSSProperties = {
@@ -473,3 +474,60 @@ export function ThreatMarkers({
</>
);
}
// -- Telegram OSINT pins (HTML, above threat alert boxes) --
interface TelegramOsintMarkersProps {
features: GeoJSON.Feature[];
onEntityClick?: (entity: SelectedEntity | null) => void;
}
export function TelegramOsintMarkers({ features, onEntityClick }: TelegramOsintMarkersProps) {
if (!features.length) return null;
return (
<>
{features.map((feature) => {
if (feature.geometry?.type !== 'Point') return null;
const [lng, lat] = feature.geometry.coordinates as [number, number];
const props = feature.properties || {};
const id = String(props.id || '');
if (!id) return null;
const postCount = Number(props.post_count || 1);
const size = postCount > 1 ? Math.min(30, 16 + Math.log2(postCount) * 5) : 16;
return (
<Marker
key={`telegram-osint-${id}`}
longitude={lng}
latitude={lat}
anchor="center"
offset={TELEGRAM_MARKER_OFFSET}
style={{ zIndex: 95 }}
onClick={(e) => {
e.originalEvent.stopPropagation();
onEntityClick?.({
id,
type: 'telegram_osint',
name: String(props.name || 'Telegram OSINT'),
});
}}
>
<div
title={`Telegram OSINT${postCount > 1 ? ` (${postCount} posts)` : ''}`}
style={{
width: size,
height: size,
borderRadius: '50%',
background: '#ef4444',
border: '2.5px solid #fca5a5',
boxShadow: '0 0 14px rgba(239, 68, 68, 0.75)',
cursor: 'pointer',
}}
/>
</Marker>
);
})}
</>
);
}
frontend/src/components/map/dynamicMapLayers.worker.ts
+34 -7
View File
@@ -26,6 +26,8 @@ export type DynamicMapLayersDataPayload = DynamicMapLayersPayload;
export type DynamicMapLayersBuildPayload = {
bounds: BoundsTuple;
/** When true, /api/live-data/fast already bbox-filtered this payload — skip client cull. */
serverBboxScoped?: boolean;
dtSeconds: number;
trackedIcaos: string[];
activeLayers: {
@@ -173,6 +175,16 @@ function inView(lat: number, lng: number, bounds: BoundsTuple): boolean {
return lng >= bounds[0] && lng <= bounds[2] && lat >= bounds[1] && lat <= bounds[3];
}
function passesViewFilter(
lat: number,
lng: number,
bounds: BoundsTuple,
serverBboxScoped: boolean,
): boolean {
if (serverBboxScoped) return true;
return inView(lat, lng, bounds);
}
function cleanLabel(value: unknown): string {
if (typeof value !== 'string' && typeof value !== 'number') return '';
return String(value).trim();
@@ -239,6 +251,7 @@ function buildFlightLayerGeoJSONWorker(
bounds: BoundsTuple,
dtSeconds: number,
trackedIcaos: Set<string>,
serverBboxScoped: boolean,
): FC {
if (!flights?.length) return null;
const { colorMap, groundedMap, typeLabel, idPrefix, milSpecialMap, useTrackHeading } = config;
@@ -248,7 +261,7 @@ function buildFlightLayerGeoJSONWorker(
const f = flights[i];
if (f.lat == null || f.lng == null) continue;
const [iLng, iLat] = interpFlightPosition(f, dtSeconds);
if (!inView(iLat, iLng, bounds)) continue;
if (!passesViewFilter(iLat, iLng, bounds, serverBboxScoped)) continue;
if (f.icao24 && trackedIcaos.has(f.icao24.toLowerCase())) continue;
const acType = classifyAircraft(f.model, f.aircraft_category);
@@ -288,6 +301,7 @@ function buildTrackedFlightsGeoJSONWorker(
flights: Flight[] | undefined,
bounds: BoundsTuple,
dtSeconds: number,
serverBboxScoped: boolean,
): FC {
if (!flights?.length) return null;
const features: GeoJSON.Feature[] = [];
@@ -296,7 +310,7 @@ function buildTrackedFlightsGeoJSONWorker(
const f = flights[i];
if (f.lat == null || f.lng == null) continue;
const [lng, lat] = interpFlightPosition(f, dtSeconds);
if (!inView(lat, lng, bounds)) continue;
if (!passesViewFilter(lat, lng, bounds, serverBboxScoped)) continue;
const alertColor = ('alert_color' in f ? f.alert_color : '') || 'white';
const acType = classifyAircraft(f.model, f.aircraft_category);
@@ -334,6 +348,7 @@ function buildShipsGeoJSONWorker(
activeLayers: DynamicMapLayersBuildPayload['activeLayers'],
bounds: BoundsTuple,
dtSeconds: number,
serverBboxScoped: boolean,
): FC {
if (
!ships?.length ||
@@ -353,7 +368,7 @@ function buildShipsGeoJSONWorker(
const s = ships[i];
if (s.lat == null || s.lng == null) continue;
const [iLng, iLat] = interpShipPosition(s, dtSeconds);
if (!inView(iLat, iLng, bounds)) continue;
if (!passesViewFilter(iLat, iLng, bounds, serverBboxScoped)) continue;
if (s.type === 'carrier') continue;
const isTrackedYacht = Boolean(s.yacht_alert);
@@ -394,6 +409,7 @@ function buildSigintGeoJSONWorker(
signals: SigintSignal[] | undefined,
source: 'meshtastic' | 'aprs',
bounds: BoundsTuple,
serverBboxScoped: boolean,
): FC {
if (!signals?.length) return null;
const wanted =
@@ -405,7 +421,7 @@ function buildSigintGeoJSONWorker(
for (let i = 0; i < signals.length; i += 1) {
const sig = signals[i];
if (!wanted(sig) || sig.lat == null || sig.lng == null) continue;
if (!inView(sig.lat, sig.lng, bounds)) continue;
if (!passesViewFilter(sig.lat, sig.lng, bounds, serverBboxScoped)) continue;
features.push({
type: 'Feature',
properties: {
@@ -537,6 +553,7 @@ function applyFilters(activeFilters: Record<string, string[]> | undefined) {
function buildDynamicLayers(payload: DynamicMapLayersBuildPayload): DynamicMapLayersResult {
const trackedIcaos = new Set(payload.trackedIcaos);
const filtered = applyFilters(payload.activeFilters);
const serverBboxScoped = Boolean(payload.serverBboxScoped);
return {
commercialFlightsGeoJSON: payload.activeLayers.flights
? buildFlightLayerGeoJSONWorker(
@@ -545,6 +562,7 @@ function buildDynamicLayers(payload: DynamicMapLayersBuildPayload): DynamicMapLa
payload.bounds,
payload.dtSeconds,
trackedIcaos,
serverBboxScoped,
)
: null,
privateFlightsGeoJSON: payload.activeLayers.private
@@ -554,6 +572,7 @@ function buildDynamicLayers(payload: DynamicMapLayersBuildPayload): DynamicMapLa
payload.bounds,
payload.dtSeconds,
trackedIcaos,
serverBboxScoped,
)
: null,
privateJetsGeoJSON: payload.activeLayers.jets
@@ -563,6 +582,7 @@ function buildDynamicLayers(payload: DynamicMapLayersBuildPayload): DynamicMapLa
payload.bounds,
payload.dtSeconds,
trackedIcaos,
serverBboxScoped,
)
: null,
militaryFlightsGeoJSON: payload.activeLayers.military
@@ -572,22 +592,29 @@ function buildDynamicLayers(payload: DynamicMapLayersBuildPayload): DynamicMapLa
payload.bounds,
payload.dtSeconds,
trackedIcaos,
serverBboxScoped,
)
: null,
trackedFlightsGeoJSON: payload.activeLayers.tracked
? buildTrackedFlightsGeoJSONWorker(filtered.tracked, payload.bounds, payload.dtSeconds)
? buildTrackedFlightsGeoJSONWorker(
filtered.tracked,
payload.bounds,
payload.dtSeconds,
serverBboxScoped,
)
: null,
shipsGeoJSON: buildShipsGeoJSONWorker(
filtered.ships,
payload.activeLayers,
payload.bounds,
payload.dtSeconds,
serverBboxScoped,
),
meshtasticGeoJSON: payload.activeLayers.sigint_meshtastic
? buildSigintGeoJSONWorker(dynamicData.sigint, 'meshtastic', payload.bounds)
? buildSigintGeoJSONWorker(dynamicData.sigint, 'meshtastic', payload.bounds, serverBboxScoped)
: null,
aprsGeoJSON: payload.activeLayers.sigint_aprs
? buildSigintGeoJSONWorker(dynamicData.sigint, 'aprs', payload.bounds)
? buildSigintGeoJSONWorker(dynamicData.sigint, 'aprs', payload.bounds, serverBboxScoped)
: null,
};
}
frontend/src/components/map/geoJSONBuilders.ts
+182 -1
View File
@@ -165,8 +165,12 @@ export function buildEarthquakesGeoJSON(earthquakes?: Earthquake[]): FC {
properties: {
id: i,
type: 'earthquake',
name: `[M${eq.mag}]\n${eq.place || 'Unknown Location'}`,
name: `[M${eq.mag}] ${eq.place || 'Unknown Location'}`,
title: eq.title,
lat: eq.lat,
lng: eq.lng,
mag: eq.mag,
place: eq.place,
},
geometry: { type: 'Point' as const, coordinates: [eq.lng, eq.lat] },
};
@@ -1566,6 +1570,183 @@ export function buildCrowdThreatGeoJSON(threats?: CrowdThreatItem[], inView?: In
};
}
// ─── Telegram OSINT ───────────────────────────────────────────────────────
/** Group geoparsed posts by city-level coordinates (~1 km grid). */
export function telegramClusterKey(lat: number, lng: number): string {
return `${lat.toFixed(2)}_${lng.toFixed(2)}`;
}
/** Small fixed shift (~5 mi NE) only when a threat alert shares the same city grid. */
export const TELEGRAM_ALERT_AVOID_METERS = 8_000;
export const TELEGRAM_ALERT_AVOID_BEARING = 45;
/** HTML marker nudge — threat alerts are DOM overlays that cover map canvas dots. */
export const TELEGRAM_MARKER_OFFSET: [number, number] = [28, -24];
export function telegramClusterNearNewsAlert(
lat: number,
lng: number,
news?: Array<{ coords?: [number, number] | null }> | null,
): boolean {
if (!news?.length) return false;
const key = telegramClusterKey(lat, lng);
return news.some((item) => {
const coords = item.coords;
if (!coords || coords.length < 2) return false;
return telegramClusterKey(coords[0], coords[1]) === key;
});
}
export function telegramMapPinCoords(
lat: number,
lng: number,
avoidAlert: boolean,
): [number, number] {
if (!avoidAlert) return [lat, lng];
return projectPoint(lat, lng, TELEGRAM_ALERT_AVOID_BEARING, TELEGRAM_ALERT_AVOID_METERS);
}
export function applyTelegramAlertAvoidance(
geo: FC,
news?: Array<{ coords?: [number, number] | null }> | null,
): FC {
if (!geo?.features?.length) return geo;
return {
...geo,
features: geo.features.map((feature) => {
const geometry = feature.geometry;
if (!geometry || geometry.type !== 'Point') return feature;
const point = geometry.coordinates;
if (!point || point.length < 2) return feature;
const lng = point[0];
const lat = point[1];
const avoid = telegramClusterNearNewsAlert(lat, lng, news);
if (!avoid) return feature;
const [pinLat, pinLng] = telegramMapPinCoords(lat, lng, true);
return {
...feature,
geometry: {
type: 'Point' as const,
coordinates: [pinLng, pinLat],
},
};
}),
};
}
export function buildTelegramOsintGeoJSON(
payload?: {
posts?: Array<{
id: string;
title?: string;
description?: string;
link?: string;
source?: string;
channel?: string;
risk_score?: number;
coords?: [number, number] | null;
}>;
},
inView?: InViewFilter,
): FC {
const posts = payload?.posts;
if (!posts?.length) return null;
const clusters = new Map<
string,
{
lat: number;
lng: number;
posts: NonNullable<typeof posts>;
maxRisk: number;
}
>();
for (const post of posts) {
const coords = post.coords;
if (!coords || coords.length < 2) continue;
const lat = coords[0];
const lng = coords[1];
if (inView && !inView(lat, lng)) continue;
const key = telegramClusterKey(lat, lng);
const bucket = clusters.get(key);
if (bucket) {
bucket.posts.push(post);
bucket.maxRisk = Math.max(bucket.maxRisk, post.risk_score ?? 1);
} else {
clusters.set(key, {
lat,
lng,
posts: [post],
maxRisk: post.risk_score ?? 1,
});
}
}
if (!clusters.size) return null;
return {
type: 'FeatureCollection' as const,
features: Array.from(clusters.entries()).map(([key, cluster]) => {
const lead = cluster.posts[0];
const count = cluster.posts.length;
return {
type: 'Feature' as const,
properties: {
id: key,
type: 'telegram_osint',
name:
count > 1
? `Telegram OSINT (${count} posts)`
: lead.title || 'Telegram OSINT',
description: lead.description || '',
link: lead.link || '',
source: lead.source || '',
channel: lead.channel || '',
risk_score: cluster.maxRisk,
post_count: count,
},
geometry: {
type: 'Point' as const,
coordinates: [cluster.lng, cluster.lat],
},
};
}),
};
}
// ─── Malware C2 / URLhaus ─────────────────────────────────────────────────
export function buildMalwareGeoJSON(
payload?: { threats?: Array<{ id: string; lat: number; lng: number; ip: string; malware: string; threat_type?: string; country?: string }> },
inView?: InViewFilter,
): FC {
const threats = payload?.threats;
if (!threats?.length) return null;
return {
type: 'FeatureCollection' as const,
features: threats
.map((t) => {
if (t.lat == null || t.lng == null) return null;
if (inView && !inView(t.lat, t.lng)) return null;
return {
type: 'Feature' as const,
properties: {
id: t.id,
type: 'malware',
name: t.malware,
ip: t.ip,
threat_type: t.threat_type || 'malware',
country: t.country || '',
},
geometry: { type: 'Point' as const, coordinates: [t.lng, t.lat] },
};
})
.filter(Boolean) as GeoJSON.Feature[],
};
}
// ─── Wastewater colors by alert level ────────────────────────────────────
const WW_COLORS = {
alert: '#ff3333', // red — elevated pathogen detected
frontend/src/components/map/hooks/useStaticMapLayersWorker.ts
+2
View File
@@ -48,6 +48,8 @@ const EMPTY_RESULT: StaticMapLayersResult = {
uapSightingsGeoJSON: null,
wastewaterGeoJSON: null,
crowdthreatGeoJSON: null,
malwareGeoJSON: null,
telegramOsintGeoJSON: null,
};
let worker: Worker | null = null;
frontend/src/components/map/staticMapLayers.worker.ts
+24
View File
@@ -21,6 +21,8 @@ import {
buildUapSightingsGeoJSON,
buildWastewaterGeoJSON,
buildCrowdThreatGeoJSON,
buildMalwareGeoJSON,
buildTelegramOsintGeoJSON,
} from '@/components/map/geoJSONBuilders';
import type {
AirQualityStation,
@@ -44,6 +46,7 @@ import type {
VIIRSChangeNode,
Volcano,
CrowdThreatItem,
MalwareThreat,
} from '@/types/dashboard';
type BoundsTuple = [number, number, number, number];
@@ -71,6 +74,17 @@ export type StaticMapLayersDataPayload = {
uapSightings?: UAPSighting[];
wastewater?: WastewaterPlant[];
crowdthreat?: CrowdThreatItem[];
malwareThreats?: MalwareThreat[];
telegramOsintPosts?: Array<{
id: string;
title?: string;
description?: string;
link?: string;
source?: string;
channel?: string;
risk_score?: number;
coords?: [number, number] | null;
}>;
};
export type StaticMapLayersBuildPayload = {
@@ -95,6 +109,8 @@ export type StaticMapLayersBuildPayload = {
uap_sightings: boolean;
wastewater: boolean;
crowdthreat: boolean;
malware_c2: boolean;
telegram_osint: boolean;
};
};
@@ -119,6 +135,8 @@ export type StaticMapLayersResult = {
uapSightingsGeoJSON: FC;
wastewaterGeoJSON: FC;
crowdthreatGeoJSON: FC;
malwareGeoJSON: FC;
telegramOsintGeoJSON: FC;
};
type SyncRequest = {
@@ -191,6 +209,12 @@ function buildStaticLayers(payload: StaticMapLayersBuildPayload): StaticMapLayer
uapSightingsGeoJSON: payload.activeLayers.uap_sightings ? buildUapSightingsGeoJSON(staticData.uapSightings) : null,
wastewaterGeoJSON: payload.activeLayers.wastewater ? buildWastewaterGeoJSON(staticData.wastewater) : null,
crowdthreatGeoJSON: payload.activeLayers.crowdthreat ? buildCrowdThreatGeoJSON(staticData.crowdthreat, inView) : null,
malwareGeoJSON: payload.activeLayers.malware_c2
? buildMalwareGeoJSON({ threats: staticData.malwareThreats }, inView)
: null,
telegramOsintGeoJSON: payload.activeLayers.telegram_osint
? buildTelegramOsintGeoJSON({ posts: staticData.telegramOsintPosts })
: null,
};
}
frontend/src/hooks/useDataPolling.ts
+62 -6
View File
@@ -1,7 +1,8 @@
import { useEffect, useRef } from "react";
import { API_BASE } from "@/lib/api";
import { mergeData, setBackendStatus as setStoreBackendStatus } from "./useDataStore";
import { appendLiveDataBoundsParams } from "@/lib/liveDataViewport";
import { appendLiveDataBoundsParams, liveDataBoundsKey } from "@/lib/liveDataViewport";
import { VIEWPORT_COMMITTED_EVENT } from "@/components/map/hooks/useViewportBounds";
export type BackendStatus = 'connecting' | 'connected' | 'disconnected';
@@ -83,6 +84,10 @@ function hasMeaningfulFastData(json: FastDataProbe): boolean {
*/
export const LAYER_TOGGLE_EVENT = 'sb:layer-toggle';
/** Debounce rapid pans; min gap keeps viewport refetches under the 120/min rate limit. */
const VIEWPORT_FAST_REFETCH_DEBOUNCE_MS = 400;
const VIEWPORT_FAST_REFETCH_MIN_INTERVAL_MS = 2500;
/**
* Polls the backend for fast and slow data tiers.
*
@@ -94,6 +99,9 @@ export const LAYER_TOGGLE_EVENT = 'sb:layer-toggle';
* infrastructure. World-zoomed views skip bbox params entirely and hit
* the shared ETag cache exactly like the pre-#288 behaviour.
*
* Viewport commits trigger a debounced fast-tier refetch so regional pans
* refill aircraft/ships without waiting for the 15s poll cadence.
*
* The AIS stream viewport POST (/api/viewport) is still handled separately
* by useViewportBounds to limit upstream AIS ingestion.
*/
@@ -110,8 +118,12 @@ export function useDataPolling() {
let fetchedStartupFastPayload = false;
let fastTimerId: ReturnType<typeof setTimeout> | null = null;
let slowTimerId: ReturnType<typeof setTimeout> | null = null;
let viewportDebounceTimer: ReturnType<typeof setTimeout> | null = null;
const fastAbortRef = { current: null as AbortController | null };
const slowAbortRef = { current: null as AbortController | null };
const fastFetchGenRef = { current: 0 };
let lastViewportFetchKey: string | null = null;
let lastViewportFetchAt = 0;
const fetchCriticalBootstrap = async () => {
try {
@@ -138,6 +150,13 @@ export function useDataPolling() {
}
};
const abortInFlightFastFetch = () => {
if (fastAbortRef.current) {
fastAbortRef.current.abort();
fastAbortRef.current = null;
}
};
const fetchFastData = async () => {
if (fastTimerId) {
clearTimeout(fastTimerId);
@@ -145,9 +164,12 @@ export function useDataPolling() {
}
// Skip fetch when Time Machine snapshot mode is active
if (_pollingPaused) { scheduleNext('fast'); return; }
if (fastAbortRef.current) return;
abortInFlightFastFetch();
const controller = new AbortController();
fastAbortRef.current = controller;
const fetchGen = ++fastFetchGenRef.current;
try {
const useStartupPayload = !fetchedStartupFastPayload && !fastEtag.current;
const headers: Record<string, string> = {};
@@ -159,9 +181,10 @@ export function useDataPolling() {
headers,
signal: controller.signal,
});
if (fetchGen !== fastFetchGenRef.current) return;
if (res.status === 304) {
setStoreBackendStatus('connected');
scheduleNext('fast');
scheduleNext('fast', fetchGen);
return;
}
if (res.ok) {
@@ -171,6 +194,7 @@ export function useDataPolling() {
fastEtag.current = useStartupPayload ? null : res.headers.get('etag') || null;
if (useStartupPayload) fetchedStartupFastPayload = true;
const json = await res.json();
if (fetchGen !== fastFetchGenRef.current) return;
mergeData(json);
if (hasMeaningfulFastData(json)) hasData = true;
}
@@ -189,7 +213,7 @@ export function useDataPolling() {
fastAbortRef.current = null;
}
}
scheduleNext('fast');
scheduleNext('fast', fetchGen);
};
const fetchSlowData = async () => {
@@ -231,8 +255,9 @@ export function useDataPolling() {
};
// Adaptive polling: retry every 3s during startup, back off to normal cadence once data arrives
const scheduleNext = (tier: 'fast' | 'slow') => {
const scheduleNext = (tier: 'fast' | 'slow', fetchGen?: number) => {
if (tier === 'fast') {
if (fetchGen !== undefined && fetchGen !== fastFetchGenRef.current) return;
const delay = hasData ? 15000 : 3000; // 3s startup retry → 15s steady state
const needsFullFastPayload = fetchedStartupFastPayload && !fastEtag.current;
fastTimerId = setTimeout(fetchFastData, needsFullFastPayload ? 750 : delay);
@@ -242,6 +267,34 @@ export function useDataPolling() {
}
};
const queueViewportFastRefetch = () => {
if (_pollingPaused) return;
const key = liveDataBoundsKey();
if (!key) {
lastViewportFetchKey = null;
return;
}
if (key === lastViewportFetchKey) return;
if (viewportDebounceTimer) clearTimeout(viewportDebounceTimer);
viewportDebounceTimer = setTimeout(() => {
viewportDebounceTimer = null;
if (_pollingPaused) return;
const currentKey = liveDataBoundsKey();
if (!currentKey || currentKey === lastViewportFetchKey) return;
const now = Date.now();
if (now - lastViewportFetchAt < VIEWPORT_FAST_REFETCH_MIN_INTERVAL_MS) return;
lastViewportFetchKey = currentKey;
lastViewportFetchAt = now;
fastEtag.current = null;
void fetchFastData();
}, VIEWPORT_FAST_REFETCH_DEBOUNCE_MS);
};
// When a layer toggle fires, immediately refetch slow data so the user
// doesn't wait up to 120s for power plants / GDELT / etc. to appear.
const onLayerToggle = () => {
@@ -251,6 +304,7 @@ export function useDataPolling() {
fetchSlowData();
};
window.addEventListener(LAYER_TOGGLE_EVENT, onLayerToggle);
window.addEventListener(VIEWPORT_COMMITTED_EVENT, queueViewportFastRefetch);
void (async () => {
await fetchCriticalBootstrap();
@@ -261,9 +315,11 @@ export function useDataPolling() {
return () => {
window.removeEventListener(LAYER_TOGGLE_EVENT, onLayerToggle);
window.removeEventListener(VIEWPORT_COMMITTED_EVENT, queueViewportFastRefetch);
if (fastTimerId) clearTimeout(fastTimerId);
if (slowTimerId) clearTimeout(slowTimerId);
if (fastAbortRef.current) fastAbortRef.current.abort();
if (viewportDebounceTimer) clearTimeout(viewportDebounceTimer);
abortInFlightFastFetch();
if (slowAbortRef.current) slowAbortRef.current.abort();
};
}, []);
frontend/src/i18n/translations/en.json
+22 -1
View File
@@ -203,7 +203,12 @@
"aiIntel": "AI Intel",
"sar": "SAR",
"roadCorridorTrends": "Road Freight Trends",
"roadCorridorSource": "Copernicus S-2 · trends not live"
"roadCorridorSource": "Copernicus S-2 · trends not live",
"submarineCables": "Submarine Cables",
"malwareC2": "Malware C2",
"scmSuppliers": "SCM Suppliers",
"cyberThreats": "Cyber Threats",
"telegramOsint": "Telegram OSINT"
},
"roadCorridor": {
"analyzeHere": "ANALYZE HERE",
@@ -214,6 +219,15 @@
"panMapFirst": "Pan the map to choose an area",
"analyzeFailed": "Analysis failed"
},
"recon": {
"title": "Recon Toolkit",
"proxyNote": "Server-side proxy · local operator only"
},
"scm": {
"title": "Supply Chain",
"allClear": "No elevated risk at monitored Tier 1/2 nodes.",
"layerOff": "Off — enable Supply Chain in Data Layers to monitor fabs."
},
"shodan": {
"title": "Shodan Connector",
"searchPlaceholder": "Search devices...",
@@ -253,5 +267,12 @@
"vegetation": "Vegetation Disturbance",
"damage": "Damage Assessment",
"coherence": "Coherence Change"
},
"telegram": {
"disclaimer": "WARNING: Content below is loaded from public Telegram channels. Shadowbroker does not host, verify, or endorse it. What you view is entirely at your own risk and has nothing to do with Shadowbroker.",
"loadMedia": "VIEW MEDIA (TELEGRAM)",
"openOriginal": "OPEN ON TELEGRAM →",
"embedTitle": "Telegram post embed",
"postsAtLocation": "{count} posts at this location — scroll for more"
}
}
frontend/src/i18n/translations/fr.json
+22 -1
View File
@@ -203,7 +203,12 @@
"aiIntel": "Infos IA",
"sar": "SAR",
"roadCorridorTrends": "Tendances fret routier",
"roadCorridorSource": "Copernicus S-2 · tendances (pas en direct)"
"roadCorridorSource": "Copernicus S-2 · tendances (pas en direct)",
"submarineCables": "Câbles sous-marins",
"malwareC2": "Malware C2",
"scmSuppliers": "Fournisseurs SCM",
"cyberThreats": "Cybermenaces",
"telegramOsint": "OSINT Telegram"
},
"roadCorridor": {
"analyzeHere": "ANALYSER ICI",
@@ -214,6 +219,15 @@
"panMapFirst": "Déplacez la carte pour choisir une zone",
"analyzeFailed": "Échec de l'analyse"
},
"recon": {
"title": "Boîte à outils recon",
"proxyNote": "Proxy côté serveur · opérateur local uniquement"
},
"scm": {
"title": "Chaîne d'approvisionnement",
"allClear": "Aucun risque élevé sur les nœuds Tier 1/2 surveillés.",
"layerOff": "Désactivé — activez la couche dans Données pour surveiller les fabs."
},
"shodan": {
"title": "Connecteur Shodan",
"searchPlaceholder": "Rechercher des appareils...",
@@ -253,5 +267,12 @@
"vegetation": "Perturbation végétale",
"damage": "Évaluation des dégâts",
"coherence": "Changement de cohérence"
},
"telegram": {
"disclaimer": "AVERTISSEMENT : le contenu ci-dessous provient de canaux Telegram publics. Shadowbroker ne l'héberge pas, ne le vérifie pas et ne le cautionne pas. Vous le consultez à vos risques et cela n'a aucun lien avec Shadowbroker.",
"loadMedia": "AFFICHER LE MÉDIA (TELEGRAM)",
"openOriginal": "OUVRIR SUR TELEGRAM →",
"embedTitle": "Intégration Telegram",
"postsAtLocation": "{count} posts à cet endroit — faites défiler"
}
}
frontend/src/i18n/translations/zh-CN.json
+22 -1
View File
@@ -203,7 +203,12 @@
"aiIntel": "AI 情报",
"sar": "SAR",
"roadCorridorTrends": "公路货运趋势",
"roadCorridorSource": "Copernicus S-2 · 趋势（非实时）"
"roadCorridorSource": "Copernicus S-2 · 趋势（非实时）",
"submarineCables": "海底电缆",
"malwareC2": "恶意软件 C2",
"scmSuppliers": "供应链供应商",
"cyberThreats": "网络威胁",
"telegramOsint": "Telegram OSINT"
},
"roadCorridor": {
"analyzeHere": "分析此处",
@@ -214,6 +219,15 @@
"panMapFirst": "请先平移地图以选择区域",
"analyzeFailed": "分析失败"
},
"recon": {
"title": "侦察工具包",
"proxyNote": "服务端代理 · 仅本地操作员"
},
"scm": {
"title": "供应链",
"allClear": "受监控的 Tier 1/2 节点无升高风险。",
"layerOff": "已关闭 — 在数据图层中启用供应链以监控晶圆厂。"
},
"shodan": {
"title": "Shodan 连接器",
"searchPlaceholder": "搜索设备...",
@@ -253,5 +267,12 @@
"vegetation": "植被干扰",
"damage": "损毁评估",
"coherence": "相干变化"
},
"telegram": {
"disclaimer": "警告：以下内容来自公开 Telegram 频道。Shadowbroker 不托管、不核实、不背书该内容。您自行承担一切风险，与 Shadowbroker 无关。",
"loadMedia": "查看媒体（Telegram）",
"openOriginal": "在 Telegram 打开 →",
"embedTitle": "Telegram 帖子嵌入",
"postsAtLocation": "此位置 {count} 条帖子 — 向下滚动查看更多"
}
}
frontend/src/lib/entityGraph.ts
+22
View File
@@ -0,0 +1,22 @@
import type { SelectedEntity } from '@/types/dashboard';
const GRAPH_TYPES = new Set(['aircraft', 'vessel', 'company', 'person', 'ip', 'country']);
const SELECTION_TO_GRAPH: Record<string, string> = {
flight: 'aircraft',
private_flight: 'aircraft',
military_flight: 'aircraft',
private_jet: 'aircraft',
tracked_flight: 'aircraft',
ship: 'vessel',
};
export function mapEntityToGraphType(type: string): string | null {
const mapped = SELECTION_TO_GRAPH[type] || type;
return GRAPH_TYPES.has(mapped) ? mapped : null;
}
export function isEntityGraphEligible(entity: SelectedEntity | null | undefined): boolean {
if (!entity) return false;
return mapEntityToGraphType(entity.type) !== null;
}
frontend/src/lib/liveDataViewport.ts
+13
View File
@@ -67,6 +67,19 @@ export function getLiveDataBounds(): LiveDataBounds | null {
return _current;
}
/** Stable cache key for the active bbox-scoped fetch window (1° quantization,
* matching appendLiveDataBoundsParams / backend ETag). Returns null when
* world-scale fetching is active. */
export function liveDataBoundsKey(): string | null {
const b = _current;
if (!b) return null;
const s = Math.floor(b.south);
const w = Math.floor(b.west);
const n = Math.ceil(b.north);
const e = Math.ceil(b.east);
return `${s},${w},${n},${e}`;
}
/** Append `s/w/n/e` query params to a URL when bounds are set, otherwise
* return the URL unchanged. Centralised so all live-data callers stay in
* sync about quantization and the world-scale skip rule. */
frontend/src/lib/submarineCables.ts
+97
View File
@@ -0,0 +1,97 @@
/** Synthetic TeleGeography corridor overlays — not real cable routes. */
const SYNTHETIC_CABLE_NAMES = new Set([
'SEA-ME-WE Corridor',
'Trans-Atlantic North',
'Trans-Atlantic South',
'WACS / SAT-3 Corridor',
'EASSy / SEACOM',
'East Asia Corridor',
'Asia-Australia',
'Trans-Pacific',
'South Atlantic',
]);
type LngLat = [number, number];
function lonJumpDegrees(a: LngLat, b: LngLat): number {
const d = Math.abs(b[0] - a[0]);
return Math.min(d, 360 - d);
}
function iterParts(geometry: GeoJSON.Geometry): LngLat[][] {
if (geometry.type === 'LineString') {
return [geometry.coordinates as LngLat[]];
}
if (geometry.type === 'MultiLineString') {
return geometry.coordinates as LngLat[][];
}
return [];
}
/** Split a path when consecutive vertices jump across continents / dateline. */
function splitAtJumps(coords: LngLat[], maxJumpDeg = 90): LngLat[][] {
if (coords.length < 2) return coords.length ? [coords] : [];
const segments: LngLat[][] = [[coords[0]]];
for (let i = 1; i < coords.length; i += 1) {
const prev = segments[segments.length - 1][segments[segments.length - 1].length - 1];
const next = coords[i];
if (lonJumpDegrees(prev, next) > maxJumpDeg) {
segments.push([next]);
} else {
segments[segments.length - 1].push(next);
}
}
return segments.filter((seg) => seg.length >= 2);
}
function partsToGeometry(parts: LngLat[][]): GeoJSON.LineString | GeoJSON.MultiLineString | null {
if (!parts.length) return null;
if (parts.length === 1) {
return { type: 'LineString', coordinates: parts[0] };
}
return { type: 'MultiLineString', coordinates: parts };
}
/**
* Drop synthetic corridor junk and split lines that cut across the dateline.
* Land-crossing segments are stripped at build time (see scripts/sanitize_submarine_cables.py).
*/
export function sanitizeSubmarineCables(
collection: GeoJSON.FeatureCollection,
): GeoJSON.FeatureCollection {
const byName = new Map<string, GeoJSON.Feature>();
for (const feature of collection.features) {
const name = String(feature.properties?.name || '').trim();
if (!name || SYNTHETIC_CABLE_NAMES.has(name)) continue;
if (!feature.geometry || feature.geometry.type === 'GeometryCollection') continue;
const splitParts: LngLat[][] = [];
for (const part of iterParts(feature.geometry)) {
splitParts.push(...splitAtJumps(part));
}
const geometry = partsToGeometry(splitParts);
if (!geometry) continue;
const cleaned: GeoJSON.Feature = {
type: 'Feature',
properties: feature.properties ?? {},
geometry,
};
const existing = byName.get(name);
if (!existing) {
byName.set(name, cleaned);
continue;
}
const existingPts = iterParts(existing.geometry!).flat().length;
const newPts = splitParts.flat().length;
if (newPts > existingPts) byName.set(name, cleaned);
}
return {
type: 'FeatureCollection',
features: Array.from(byName.values()),
};
}
frontend/src/lib/telegramProxy.ts
+6
View File
@@ -0,0 +1,6 @@
/** Proxy Telegram CDN media through the backend (host allowlist + range requests). */
export function buildTelegramMediaProxyUrl(rawUrl: string): string {
return rawUrl.startsWith('http')
? `/api/telegram/media?url=${encodeURIComponent(rawUrl)}`
: rawUrl;
}
frontend/src/middleware.ts
+2 -1
View File
@@ -27,7 +27,8 @@ function buildCsp(nonce: string, strictScripts = false): string {
"object-src 'none'",
"worker-src 'self' blob:",
"child-src 'self' blob:",
"frame-src 'self' https://video.ibm.com https://ustream.tv https://www.ustream.tv",
"frame-src 'self' https://video.ibm.com https://ustream.tv https://www.ustream.tv https://t.me",
"media-src 'self' blob:",
"frame-ancestors 'none'",
"base-uri 'self'",
"form-action 'self'",
frontend/src/types/dashboard.ts
+76
View File
@@ -934,6 +934,77 @@ export interface DashboardData {
error?: string | null;
}>;
};
malware_threats?: {
threats?: MalwareThreat[];
total?: number;
timestamp?: string | null;
source?: string;
};
cyber_threats?: {
threats?: Array<{
id: string;
name: string;
vendor?: string;
product?: string;
severity?: string;
date?: string;
source?: string;
}>;
stats?: Record<string, unknown>;
};
scm_suppliers?: {
suppliers?: ScmSupplier[];
critical_count?: number;
total?: number;
timestamp?: string;
};
telegram_osint?: {
posts?: TelegramOsintPost[];
total?: number;
geolocated?: number;
timestamp?: string | null;
channels?: string[];
};
}
export interface TelegramOsintPost {
id: string;
title?: string;
description?: string;
link?: string;
published?: string;
source?: string;
channel?: string;
risk_score?: number;
coords?: [number, number] | null;
media_type?: 'video' | 'photo' | null;
media_url?: string | null;
embed_url?: string | null;
}
export interface MalwareThreat {
id: string;
lat: number;
lng: number;
ip: string;
port?: number;
malware: string;
status?: string;
country?: string;
threat_type?: string;
}
export interface ScmSupplier {
id: string;
name: string;
city: string;
country: string;
category: string;
lat: number;
lng: number;
risk_level: string;
active_threats: string[];
}
// ─── SAR ─────────────────────────────────────────────────────────────────────
@@ -1044,6 +1115,11 @@ export interface ActiveLayers {
crowdthreat: boolean;
sar: boolean;
road_corridor_trends: boolean;
malware_c2: boolean;
submarine_cables: boolean;
scm_suppliers: boolean;
cyber_threats: boolean;
telegram_osint: boolean;
}
export interface SelectedEntity {
frontend/src/utils/alertSpread.test.ts
+1
View File
@@ -61,3 +61,4 @@ describe('spreadAlertItems', () => {
expect(hasNonZeroOffset).toBe(true);
});
});
frontend/src/utils/alertSpread.ts
+1
View File
@@ -184,3 +184,4 @@ export function spreadAlertItems(
showLine: Math.abs(item.offsetX) > 5 || Math.abs(item.offsetY) > 5,
})) as SpreadAlertItem[];
}
scripts/data/ne_110m_land.geojson
+1
View File
File diff suppressed because one or more lines are too long
scripts/data/ne_50m_land.geojson
+1
View File
File diff suppressed because one or more lines are too long
scripts/sanitize_submarine_cables.py
+153
View File
@@ -0,0 +1,153 @@
#!/usr/bin/env python3
"""Clean submarine cable GeoJSON: drop synthetic corridors and land-crossing segments."""
from __future__ import annotations
import json
from pathlib import Path
from shapely.geometry import LineString, shape
from shapely.ops import unary_union
SYNTHETIC = {
"SEA-ME-WE Corridor",
"Trans-Atlantic North",
"Trans-Atlantic South",
"WACS / SAT-3 Corridor",
"EASSy / SEACOM",
"East Asia Corridor",
"Asia-Australia",
"Trans-Pacific",
"South Atlantic",
}
# Drop segments where this much of the path lies on land (110m/50m Natural Earth).
LAND_OVERLAP_MAX = 0.12
def lon_jump(a: list[float], b: list[float]) -> float:
d = abs(b[0] - a[0])
return min(d, 360 - d)
def iter_parts(geom: dict) -> list[list[list[float]]]:
t = geom["type"]
c = geom["coordinates"]
if t == "LineString":
return [c]
if t == "MultiLineString":
return c
return []
def split_at_jumps(coords: list[list[float]], max_jump: float = 90) -> list[list[list[float]]]:
if len(coords) < 2:
return [coords] if coords else []
segments: list[list[list[float]]] = [[coords[0]]]
for point in coords[1:]:
prev = segments[-1][-1]
if lon_jump(prev, point) > max_jump:
segments.append([point])
else:
segments[-1].append(point)
return [seg for seg in segments if len(seg) >= 2]
def segment_land_overlap(a: list[float], b: list[float], land) -> float:
line = LineString([a, b])
if line.length == 0:
return 0.0
return float(line.intersection(land).length / line.length)
def filter_land_segments(coords: list[list[float]], land) -> list[list[list[float]]]:
if len(coords) < 2:
return []
parts: list[list[list[float]]] = []
current = [coords[0]]
for a, b in zip(coords, coords[1:]):
if segment_land_overlap(a, b, land) <= LAND_OVERLAP_MAX:
if current[-1] != a:
if len(current) >= 2:
parts.append(current)
current = [a]
current.append(b)
else:
if len(current) >= 2:
parts.append(current)
current = [b]
if len(current) >= 2:
parts.append(current)
return parts
def parts_to_geometry(parts: list[list[list[float]]]) -> dict | None:
if not parts:
return None
if len(parts) == 1:
return {"type": "LineString", "coordinates": parts[0]}
return {"type": "MultiLineString", "coordinates": parts}
def load_land(root: Path):
data_dir = root / "scripts" / "data"
data_dir.mkdir(parents=True, exist_ok=True)
land_path = data_dir / "ne_50m_land.geojson"
if not land_path.exists():
land_path = data_dir / "ne_110m_land.geojson"
if not land_path.exists():
import urllib.request
land_path = data_dir / "ne_110m_land.geojson"
url = (
"https://raw.githubusercontent.com/nvkelso/natural-earth-vector/"
"master/geojson/ne_110m_land.geojson"
)
urllib.request.urlretrieve(url, land_path)
data = json.loads(land_path.read_text(encoding="utf-8"))
return unary_union([shape(feat["geometry"]) for feat in data["features"]])
def sanitize(data: dict, land) -> dict:
by_name: dict[str, dict] = {}
for feature in data.get("features", []):
name = str((feature.get("properties") or {}).get("name") or "").strip()
if not name or name in SYNTHETIC:
continue
geom = feature.get("geometry")
if not geom:
continue
split_parts: list[list[list[float]]] = []
for part in iter_parts(geom):
for jump_part in split_at_jumps(part):
split_parts.extend(filter_land_segments(jump_part, land))
geometry = parts_to_geometry(split_parts)
if not geometry:
continue
cleaned = {
"type": "Feature",
"properties": feature.get("properties") or {},
"geometry": geometry,
}
existing = by_name.get(name)
if not existing:
by_name[name] = cleaned
continue
existing_pts = sum(len(p) for p in iter_parts(existing["geometry"]))
new_pts = sum(len(p) for p in split_parts)
if new_pts > existing_pts:
by_name[name] = cleaned
return {"type": "FeatureCollection", "features": list(by_name.values())}
def main() -> None:
root = Path(__file__).resolve().parents[1]
src = root / "frontend" / "public" / "data" / "submarine-cables.json"
raw = json.loads(src.read_text(encoding="utf-8"))
land = load_land(root)
cleaned = sanitize(raw, land)
src.write_text(json.dumps(cleaned, separators=(",", ":")), encoding="utf-8")
print(f"Wrote {len(cleaned['features'])} features to {src}")
if __name__ == "__main__":
main()