feat(gt): experimental Derived OSINT analytics with lean-node safeguards

Cherry-picked from @Bobpick PR #391 (GT + OpenClaw slice): Bayesian strategic-risk engine, map overlay, OpenClaw commands, and telegram_rhetoric watchdog. Off by default (GT_ANALYTICS_ENABLED=false, gt_risk layer false). 1 vCPU nodes get cgroup detection, UI warning on layer toggle, and lean profile that skips scheduled ingest/Louvain unless GT_ANALYTICS_ACK_LOW_CPU=true. Backtest HUD removed from dashboard (OpenClaw/API regression only).

Co-authored-by: Robert Pickett <bobpickettsr@yahoo.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

.env.example

@@ -62,9 +62,6 @@ ADMIN_KEY=
 # Free MAP_KEY from https://firms.modaps.eosdis.nasa.gov/
 # FIRMS_MAP_KEY=
 
-# Airframes.io ACARS/VDL datalink (plane dossier messages). Dashboard → API Key.
-# AIRFRAMES_API_KEY=
-
 # Ukraine air raid alerts — free token from https://alerts.in.ua/
 # ALERTS_IN_UA_TOKEN=
 

DATA-ATTRIBUTION.md

@@ -34,7 +34,6 @@ These sources have their own terms; consult each link before redistributing.
 | Source | URL | License / Terms | Notes |
 |---|---|---|---|
 | OpenSky Network | https://opensky-network.org | OpenSky API terms | Commercial and private aircraft tracking |
-| Airframes.io | https://airframes.io | Airframes API terms | Optional ACARS/VDL datalink messages in aircraft dossiers |
 | CelesTrak | https://celestrak.org | Public domain / no restrictions | Satellite TLE data |
 | USGS Earthquake Hazards | https://earthquake.usgs.gov | Public domain (US Federal) | Seismic events |
 | NASA FIRMS | https://firms.modaps.eosdis.nasa.gov | NASA Open Data | Fire/thermal anomalies (VIIRS) |

README.md

@@ -19,42 +19,26 @@
 
 **ShadowBroker** is a decentralized intelligence platform that aggregates real-time, multi-domain OSINT telemetry from 60+ live intelligence feeds into a single dark-ops map interface. Aircraft, ships, satellites, conflict zones, CCTV networks, GPS jamming, internet-connected devices, police scanners, mesh radio nodes, and breaking geopolitical events — all updating in real time on one screen as well as an obfuscated communications protocol and information exchange infrastructure.
 
-<details>
-<summary>🛰️ Project Description</summary>
-
 Built with **Next.js**, **MapLibre GL**, **FastAPI**, and **Python**. 40+ toggleable data layers, including SAR ground-change detection, **Telegram OSINT** (public channel previews geoparsed onto the map), a **server-side recon toolkit** (DNS, WHOIS, sanctions, BGP, IP sweep, and more), supply-chain risk overlays, and malware/C2 + CISA KEV cyber threat feeds. Multiple visual modes (DEFAULT / SATELLITE / FLIR / NVG / CRT). Right-click any point on Earth for a country dossier, head-of-state lookup, entity-graph expansion, and the latest Sentinel-2 satellite photo. ShadowBroker has no accounts, product telemetry, or analytics; the dashboard talks to your self-hosted backend. Sensitive recon and Shodan queries never hit third-party APIs from the browser — they are proxied through the backend with SSRF guards and local-operator auth. The **OpenClaw / agent command channel** exposes the same recon backends plus full telemetry search — no separate API integration required.
 
 Designed for analysts, researchers, radio operators, and anyone who wants to see what the world looks like when every public signal is on the same map.
-</details>
 
----
 
-<details>
-<summary>🌍 Why This Exists</summary>
+## Why This Exists
 
 A surprising amount of global telemetry is already public — aircraft ADS-B broadcasts, maritime AIS signals, satellite orbital data, earthquake sensors, mesh radio networks, police scanner feeds, environmental monitoring stations, internet infrastructure telemetry, and more. This data is scattered across dozens of tools and APIs. ShadowBroker combines all of it into a single interface.
 
 The project does not introduce new surveillance capabilities — it aggregates and visualizes existing public datasets. It is fully open-source so anyone can audit exactly what data is accessed and how. ShadowBroker does not include product telemetry, analytics, or accounts. Operator-supplied keys stay in your local deployment, but live OSINT features necessarily make outbound requests to the public data providers you enable or query.
 
-</details>
-
----
-
-
-<details>
-<summary>📡 Shodan & Recon (security-first)</summary>
+### Shodan & Recon (security-first)
 
 ShadowBroker includes an optional **Shodan connector** for operator-supplied API access (`SHODAN_API_KEY`) and a **Recon Toolkit** panel for keyless OSINT lookups. Both run **server-side only**: the browser calls your self-hosted `/api/osint/*` and `/api/tools/shodan/*` routes; outbound requests are made by the backend after SSRF validation. Recon requires **local-operator** access (same trust model as layer toggles and admin routes). Shodan results render as a separate map overlay and remain subject to Shodan’s terms of service.
 
 > **Not included:** embedded live-news YouTube grids or a built-in Gemini AI analyst panel — use the **OpenClaw / agent channel** for AI-assisted analysis instead.
 
-</details>
-
 ---
 
-
-<details>
-<summary>🗺️ Interesting Use Cases</summary>
+## Interesting Use Cases
 
 * **Track Air Force One**, the private jets of billionaires and dictators, and every military tanker, ISR, and fighter broadcasting ADS-B. Air Force One and all of the accompanying Presidential/Vice Presidential planes are highlighted and monitored from the moment they leave the ground. 
 * **Connect an AI agent as a co-analyst** through ShadowBroker's HMAC-signed agentic command channel — supports OpenClaw and any other agent that speaks the protocol (Claude, GPT, LangChain, custom). The agent gets full read/write access to all 40+ data layers, compact cross-layer search (`search_telemetry`, `search_news`), the full recon toolkit (`osint_lookup` for IP/DNS/WHOIS/sanctions/CVE/etc.), entity-graph expansion, pin placement, map control, SAR ground-change, mesh networking, and alert delivery. It sees everything the operator sees and can take actions on the map in real time.
@@ -80,12 +64,10 @@ ShadowBroker includes an optional **Shodan connector** for operator-supplied API
 * **Monitor Telegram OSINT channels** — public `t.me/s` war/conflict feeds (OSINTdefender, NEXTA, etc.) scraped hourly, risk-scored, geoparsed to metro anchors, and plotted as clickable map pins with inline media
 * **Overlay global submarine cables** — static TeleGeography-derived cable routes (opt-in layer)
 
-</details>
 
 ---
 
-
-⚡ Quick Start (Docker)</summary>
+## ⚡ Quick Start (Docker)
 
 ### From GitHub (default — uses GHCR images)
 
@@ -117,12 +99,9 @@ Open `http://localhost:3000` to view the dashboard! *(Requires [Docker Desktop](
 
 > **Podman users:** Podman works, but `podman compose` is a wrapper and still needs a Compose provider installed. On Windows/WSL, if you see `looking up compose provider failed`, install `podman-compose` and run `podman-compose pull` followed by `podman-compose up -d` from inside the cloned `Shadowbroker` folder. On Linux/macOS/WSL shells you can also use `./compose.sh --engine podman pull` and `./compose.sh --engine podman up -d`.
 
-
-
 ---
 
-<details>
-<summary>🔄 How to Update</summary>
+##  🔄 **How to Update**
 
 ShadowBroker uses pre-built Docker images — no local building required. Updating takes seconds:
 
@@ -180,13 +159,9 @@ docker compose up -d
 * **Prune old images:** `docker image prune -f`
 * **Check logs:** `docker compose logs -f backend`
 
-
-</details>
-
 ---
 
-<details>
-<summary>☸️ Kubernetes / Helm (Advanced)</summary>
+### **☸️ Kubernetes / Helm (Advanced)**
 
 For high-availability deployments or home-lab clusters, ShadowBroker supports deployment via **Helm**. This chart is based on the `bjw-s-labs` template and provides a robust, modular setup for both the backend and frontend.
 
@@ -214,13 +189,9 @@ helm install shadowbroker ./helm/chart --create-namespace --namespace shadowbrok
 
 *Special thanks to [@chr0n1x](https://github.com/chr0n1x) for contributing the initial Kubernetes architecture.*
 
-</details>
-
 ---
 
-<details>
-<summary>🌐 Experimental Testnet — No Privacy Guarantee</summary>
-
+## Experimental Testnet — No Privacy Guarantee
 
 ShadowBroker v0.9.7 ships **InfoNet** (decentralized intelligence mesh + Sovereign Shell governance economy), an **agentic AI command channel** (supports OpenClaw and any HMAC-signing agent), **Time Machine snapshot playback**, and **SAR satellite ground-change detection**. This is an **experimental testnet** — not a private messenger and not a production governance system.
 
@@ -241,12 +212,10 @@ ShadowBroker v0.9.7 ships **InfoNet** (decentralized intelligence mesh + Soverei
 > sentence above is mapped there to the code path that enforces it (or
 > doesn't).**
 
-</details>
-
 ---
 
-<details>
-<summary>✨ Features</summary>
+
+## ✨ Features
 
 ### 🧅 InfoNet — Decentralized Intelligence Mesh + Sovereign Shell (expanded in v0.9.7)
 
@@ -485,12 +454,9 @@ Settings → API Keys is now a read-only registry. Key values never reach the br
 
 OpenSky API credentials are now a **critical-warn** environment requirement: the startup environment check flags missing OpenSky OAuth2 credentials with a strong warning, and the changelog modal links directly to the free registration page. Without them, the flights layer falls back to ADS-B-only coverage with significant gaps in Africa, Asia, and Latin America.
 
-</details>
-
 ---
 
-<details>
-<summary>🏗️ Architecture</summary>
+## 🏗️ Architecture
 
 ShadowBroker v0.9.7 is composed of three vertically-stacked planes — the **Operator UI**, the **Backend Service Plane**, and the **Decentralized Layer (InfoNet)** — plus two cross-cutting bridges (the **Time Machine** and the **Agentic AI Channel**, which is the protocol that OpenClaw and any other compatible agent connects through) and a **Privacy Core** Rust crate that backstops both the legacy mesh and the future shielded coin / DEX work.
 
@@ -599,12 +565,9 @@ ShadowBroker v0.9.7 is composed of three vertically-stacked planes — the **Ope
      Desktop:           Tauri shell  →  packaged backend-runtime  +  Next.js frontend
 ```
 
-</details>
-
 ---
 
-<details>
-<summary>📊 Data Sources & APIs</summary>
+## 📊 Data Sources & APIs
 
 | Source | Data | Update Frequency | API Key Required |
 |---|---|---|---|
@@ -663,12 +626,9 @@ ShadowBroker v0.9.7 is composed of three vertically-stacked planes — the **Ope
 
 **Outbound privacy & audit (#348–#366):** Each self-hosted install uses its own backend IP and per-install User-Agent handle. See [docs/OUTBOUND_DATA.md](docs/OUTBOUND_DATA.md) for what contacts third parties, opt-in/env controls, and accepted tradeoffs (CCTV Referer, basemap CDN, LiveUAMap, etc.).
 
-</details>
-
 ---
 
-<details>
-<summary>🚀 Getting Started</summary>
+## 🚀 Getting Started
 
 ### 🐳 Docker Setup (Recommended for Self-Hosting)
 
@@ -739,12 +699,10 @@ If you are in a bash-compatible shell, the included wrapper can auto-detect Dock
 ./compose.sh --engine podman pull
 ./compose.sh --engine podman up -d
 ```
-</details>
 
 ---
 
-<details>
-<summary>🐋 Standalone Deploy (Portainer, Uncloud, NAS, etc.)</summary>
+### 🐋 Standalone Deploy (Portainer, Uncloud, NAS, etc.)
 
 No need to clone the repo. Use the pre-built images from GitHub Container Registry. GitLab registry images may be used as a mirror if you publish them there.
 
@@ -796,12 +754,9 @@ volumes:
 >
 > `BACKEND_URL` is a plain runtime environment variable (not a build-time `NEXT_PUBLIC_*`), so you can change it in Portainer, Uncloud, or any compose editor without rebuilding the image. Set it to the address where your backend is reachable from inside the Docker network (e.g. `http://backend:8000`, `http://192.168.1.50:8000`).
 
-</details>
-
 ---
 
-<details>
-<summary>📦 Quick Start (No Code Required)</summary>
+### 📦 Quick Start (No Code Required)
 
 If you just want to run the dashboard without dealing with terminal commands:
 
@@ -819,12 +774,9 @@ Local launcher notes:
 - For DM root witness, transparency, and operator monitoring rollout, start with `docs/mesh/wormhole-dm-root-operations-runbook.md`.
 - For sample DM root ops bridge assets, also see `scripts/mesh/poll-dm-root-health-alerts.mjs`, `scripts/mesh/export-dm-root-health-prometheus.mjs`, `scripts/mesh/publish-external-root-witness-package.mjs`, `scripts/mesh/smoke-external-root-witness-flow.mjs`, `scripts/mesh/smoke-root-transparency-publication-flow.mjs`, `scripts/mesh/smoke-dm-root-deployment-flow.mjs`, `scripts/mesh/sync-dm-root-external-assurance.mjs`, and `docs/mesh/examples/`.
 
-</details>
-
 ---
 
-<details>
-<summary>💻 Developer Setup</summary>
+### 💻 Developer Setup
 
 If you want to modify the code or run from source:
 
@@ -912,13 +864,9 @@ AIS-catcher decodes VHF radio signals on 161.975 MHz and 162.025 MHz and POSTs d
 
 **Note:** AIS range depends on your antenna — typically 20-40 nautical miles with a basic setup, 60+ nm with a marine VHF antenna at elevation.
 
-</details>
-
-
 ---
 
-<details>
-<summary>🎛️ Data Layers</summary>
+## 🎛️ Data Layers
 
 All 41 layers are independently toggleable from the left panel:
 
@@ -981,12 +929,9 @@ All 41 layers are independently toggleable from the left panel:
 
 † `osint_sweep` (active InternetDB scan) requires `OPENCLAW_ACCESS_TIER=full`.
 
-</details>
-
 ---
 
-<details>
-<summary>🔧 Performance</summary>
+## 🔧 Performance
 
 The platform is optimized for handling massive real-time datasets:
 
@@ -1000,13 +945,9 @@ The platform is optimized for handling massive real-time datasets:
 * **React.memo** — Heavy components wrapped to prevent unnecessary re-renders
 * **Coordinate Precision** — Lat/lng rounded to 5 decimals (~1m) to reduce JSON size
 
-</details>
-
 ---
 
-<details>
-<summary>📁 Project Structure</summary>
-
+## 📁 Project Structure
 
 ```
 Shadowbroker/
@@ -1104,12 +1045,9 @@ Shadowbroker/
 │   └── package.json
 ```
 
-</details>
-
 ---
 
-<details>
-<summary>🔑 Environment Variables</summary>
+## 🔑 Environment Variables
 
 ### Backend (`backend/.env`)
 
@@ -1164,13 +1102,9 @@ Then confirm authenticated `GET /api/wormhole/status` or `GET /api/settings/worm
 
 **How it works:** The frontend proxies all `/api/*` requests through the Next.js server to `BACKEND_URL` using Docker's internal networking. Browsers only talk to port 3000; the backend host port is only for local diagnostics. For local dev without Docker, `BACKEND_URL` defaults to `http://localhost:8000`.
 
-</details>
-
 ---
 
-
-<details>
-<summary>🤝 Contributors</summary>
+## 🤝 Contributors
 
 ShadowBroker is built in the open. These people shipped real code:
 
@@ -1186,8 +1120,6 @@ ShadowBroker is built in the open. These people shipped real code:
 | [@suranyami](https://github.com/suranyami) | Parallel multi-arch Docker builds (11min → 3min) + runtime BACKEND_URL fix | #35, #44 |
 | [@chr0n1x](https://github.com/chr0n1x) | Kubernetes / Helm chart architecture for HA deployments | — |
 
-</details>
-
 ---
 
 ## ⚠️ Disclaimer

backend/.env.example

@@ -120,9 +120,6 @@ AIS_API_KEY=              # https://aisstream.io/ — free tier WebSocket key
 # Free MAP_KEY from https://firms.modaps.eosdis.nasa.gov/map/#d:24hrs;@0.0,0.0,3.0z
 # FIRMS_MAP_KEY=
 
-# Airframes.io ACARS/VDL datalink (plane dossier messages). Dashboard → API Key.
-# AIRFRAMES_API_KEY=
-
 # Ukraine frontline mirror (GitHub). Default follows cyterat/deepstate-map-data@main.
 # Pin an immutable commit SHA so ingest cannot silently change if main is force-pushed (#362).
 # Example (verify on GitHub before use): main @ b479954e94696bc5622c7818fd20a64a699f4fe8

backend/main.py

@@ -3927,9 +3927,7 @@ class LayerUpdate(BaseModel):
 async def update_layers(update: LayerUpdate, request: Request):
     """Receive frontend layer toggle state. Starts/stops streams accordingly."""
     from services.fetchers._store import active_layers, bump_active_layers_version, is_any_active
-    from services.layer_enable_refresh import refresh_newly_enabled_layers, snapshot_active_layers
 
-    layers_before = snapshot_active_layers()
     # Snapshot old stream states before applying changes
     old_ships = is_any_active(
         "ships_military", "ships_cargo", "ships_civilian", "ships_passenger", "ships_tracked_yachts"
@@ -3937,6 +3935,8 @@ async def update_layers(update: LayerUpdate, request: Request):
     old_mesh = is_any_active("sigint_meshtastic")
     old_aprs = is_any_active("sigint_aprs")
     old_viirs = is_any_active("viirs_nightlights")
+    old_datacenters = is_any_active("datacenters")
+    old_fishing = is_any_active("fishing_activity")
 
     # Update only known keys
     changed = False
@@ -3955,6 +3955,8 @@ async def update_layers(update: LayerUpdate, request: Request):
     new_mesh = is_any_active("sigint_meshtastic")
     new_aprs = is_any_active("sigint_aprs")
     new_viirs = is_any_active("viirs_nightlights")
+    new_datacenters = is_any_active("datacenters")
+    new_fishing = is_any_active("fishing_activity")
 
     # Start/stop AIS stream on transition
     if old_ships and not new_ships:
@@ -4010,7 +4012,17 @@ async def update_layers(update: LayerUpdate, request: Request):
         _queue_viirs_change_refresh()
         logger.info("VIIRS change refresh queued (layer enabled)")
 
-    refresh_newly_enabled_layers(layers_before)
+    if not old_datacenters and new_datacenters:
+        from services.fetchers.infrastructure import fetch_datacenters
+
+        fetch_datacenters()
+        logger.info("Datacenters loaded (layer enabled)")
+
+    if not old_fishing and new_fishing:
+        from services.fetchers.geo import fetch_fishing_activity
+
+        fetch_fishing_activity()
+        logger.info("Fishing activity refresh queued (layer enabled)")
 
     return {"status": "ok"}
 
@@ -8140,7 +8152,6 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
     "tripcheck.com",  # Oregon DOT / TripCheck
     "www.tripcheck.com",
     "infocar.dgt.es",  # Spain DGT
-    "etraffic.dgt.es",  # Spain DGT (etrafficWEB cameras host, 2026)
     "informo.madrid.es",  # Madrid
     "webcams2.asfinag.at",  # Austria ASFINAG motorway cameras
     "odo.asfinag.at",  # ASFINAG catalog API host
@@ -8337,14 +8348,14 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
             cache_seconds=30,
             headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8"},
         )
-    if host in {"infocar.dgt.es", "etraffic.dgt.es"}:
+    if host == "infocar.dgt.es":
         return _CCTVProxyProfile(
             name="dgt-spain",
             timeout=(5.0, 8.0),
             cache_seconds=60,
             headers={
                 "Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
-                "Referer": "https://etraffic.dgt.es/",
+                "Referer": "https://infocar.dgt.es/",
             },
         )
     if host == "informo.madrid.es":

backend/package-lock.json

@@ -4,15 +4,14 @@
   "requires": true,
   "packages": {
     "": {
-      "name": "backend",
       "dependencies": {
         "ws": "^8.19.0"
       }
     },
     "node_modules/ws": {
-      "version": "8.21.0",
-      "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz",
-      "integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==",
+      "version": "8.19.0",
+      "resolved": "https://registry.npmjs.org/ws/-/ws-8.19.0.tgz",
+      "integrity": "sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg==",
       "license": "MIT",
       "engines": {
         "node": ">=10.0.0"

backend/package.json

@@ -1,5 +1,5 @@
 {
   "dependencies": {
-    "ws": "^8.21.0"
+    "ws": "^8.19.0"
   }
 }

backend/pyproject.toml

@@ -15,7 +15,7 @@ dependencies = [
     "cachetools==5.5.2",
     "cryptography>=46.0.7",
     "defusedxml>=0.7.1",
-    "fastapi==0.138.0",
+    "fastapi==0.136.3",
     "feedparser==6.0.10",
     "httpx==0.28.1",
     "playwright==1.59.0",

backend/routers/agent_shell.py

@@ -29,7 +29,6 @@ from services.agent_shell_settings import (
     get_agent_shell_settings,
     set_agent_shell_working_directory,
 )
-from services.agent_shell_ws_token import consume_agent_shell_ws_token, mint_agent_shell_ws_token
 
 logger = logging.getLogger(__name__)
 router = APIRouter(tags=["agent-shell"])
@@ -44,15 +43,32 @@ def _set_winsize(fd: int, rows: int, cols: int) -> None:
     fcntl.ioctl(fd, termios.TIOCSWINSZ, winsize)
 
 
-async def _authorize_agent_shell_ws(
-    ws: WebSocket,
-    admin_key_query: str = "",
-    ws_token_query: str = "",
-) -> None:
+def _published_local_dashboard_ws(ws: WebSocket) -> bool:
+    """Browser → published Docker port appears as a bridge IP, not loopback.
+
+    For the operator shell only, also accept when the upgrade request clearly
+    targets the local dashboard (Host/Origin on localhost).
+    """
+    host_header = str(ws.headers.get("host") or "").strip().lower()
+    host_name = host_header.split(":", 1)[0]
+    if host_name in {"127.0.0.1", "localhost", "::1"}:
+        return True
+
+    origin = str(ws.headers.get("origin") or "").strip().lower()
+    if origin.startswith("http://127.0.0.1:") or origin.startswith("http://localhost:"):
+        return True
+    if origin.startswith("https://127.0.0.1:") or origin.startswith("https://localhost:"):
+        return True
+    return False
+
+
+async def _authorize_agent_shell_ws(ws: WebSocket, admin_key_query: str = "") -> None:
     host = (ws.client.host or "").lower() if ws.client else ""
-    if _is_trusted_local_runtime_host(host) or (_debug_mode_enabled() and host == "test"):
-        return
-    if consume_agent_shell_ws_token(ws_token_query):
+    if (
+        _is_trusted_local_runtime_host(host)
+        or _published_local_dashboard_ws(ws)
+        or (_debug_mode_enabled() and host == "test")
+    ):
         return
     admin_key = _current_admin_key()
     presented = str(admin_key_query or ws.headers.get("x-admin-key", "") or "").strip()
@@ -126,12 +142,6 @@ async def read_agent_shell_settings() -> dict[str, Any]:
     return get_agent_shell_settings()
 
 
-@router.post("/api/agent-shell/ws-token", dependencies=[Depends(require_local_operator)])
-async def mint_agent_shell_ws_token_route() -> dict[str, Any]:
-    token, expires_in = mint_agent_shell_ws_token()
-    return {"token": token, "expires_in": expires_in}
-
-
 @router.put("/api/agent-shell/settings", dependencies=[Depends(require_local_operator)])
 async def write_agent_shell_settings(body: AgentShellSettingsUpdate) -> dict[str, Any]:
     try:
@@ -150,11 +160,10 @@ async def agent_shell_websocket(
     cols: int = Query(default=80),
     rows: int = Query(default=24),
     admin_key: str = Query(default=""),
-    ws_token: str = Query(default=""),
 ) -> None:
     await ws.accept()
     try:
-        await _authorize_agent_shell_ws(ws, admin_key, ws_token)
+        await _authorize_agent_shell_ws(ws, admin_key)
     except WebSocketDisconnect:
         return
 

backend/routers/cctv.py

@@ -46,7 +46,6 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
     "tripcheck.com",
     "www.tripcheck.com",
     "infocar.dgt.es",
-    "etraffic.dgt.es",
     "informo.madrid.es",
     "webcams2.asfinag.at",
     "odo.asfinag.at",
@@ -159,10 +158,10 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
     if host in {"tripcheck.com", "www.tripcheck.com"}:
         return _CCTVProxyProfile(name="odot-tripcheck", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=30,
             headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8"})
-    if host in {"infocar.dgt.es", "etraffic.dgt.es"}:
+    if host == "infocar.dgt.es":
         return _CCTVProxyProfile(name="dgt-spain", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 8.0), cache_seconds=60,
             headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
-                     "Referer": "https://etraffic.dgt.es/"})
+                     "Referer": "https://infocar.dgt.es/"})
     if host == "informo.madrid.es":
         return _CCTVProxyProfile(name="madrid-city", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=30,
             headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",

backend/routers/data.py

@@ -396,52 +396,6 @@ async def get_selected_ship_trail(mmsi: int, request: Request):  # noqa: ARG001
     return {"id": mmsi, "trail": get_vessel_trail(mmsi)}
 
 
-@router.get("/api/aviation/datalink/status")
-@limiter.limit("60/minute")
-async def aviation_datalink_status(request: Request):  # noqa: ARG001
-    from services.fetchers.airframes import get_datalink_status
-
-    return get_datalink_status()
-
-
-@router.get("/api/aviation/datalink/messages")
-@limiter.limit("240/minute")
-async def aviation_datalink_messages(
-    request: Request,  # noqa: ARG001
-    icao24: str = Query("", description="ICAO24 hex for the aircraft"),
-    registration: str = Query("", description="Tail / registration number"),
-    callsign: str = Query("", description="Optional callsign filter"),
-    live: bool = Query(
-        False,
-        description="When true, fetch from Airframes if cache has no messages (slower)",
-    ),
-):
-    from services.fetchers.airframes import lookup_datalink_messages
-
-    return lookup_datalink_messages(
-        icao24=icao24,
-        registration=registration,
-        callsign=callsign,
-        allow_live=live,
-    )
-
-
-@router.get("/api/sigint/meshtastic/status")
-@limiter.limit("120/minute")
-async def meshtastic_map_status(request: Request):  # noqa: ARG001
-    from services.fetchers.meshtastic_map import get_meshtastic_map_status
-
-    return get_meshtastic_map_status()
-
-
-@router.post("/api/sigint/meshtastic/scan", dependencies=[Depends(require_local_operator)])
-@limiter.limit("3/hour")
-async def meshtastic_planet_scan(request: Request):  # noqa: ARG001
-    from services.fetchers.meshtastic_map import start_meshtastic_planet_scan
-
-    return start_meshtastic_planet_scan()
-
-
 @router.post("/api/viewport")
 @limiter.limit("60/minute")
 async def update_viewport(vp: ViewportUpdate, request: Request):  # noqa: ARG001
@@ -544,13 +498,12 @@ def _run_prediction_markets_disable() -> None:
 async def update_layers(update: LayerUpdate, request: Request):
     """Receive frontend layer toggle state. Starts/stops streams accordingly."""
     from services.fetchers._store import active_layers, bump_active_layers_version, is_any_active
-    from services.layer_enable_refresh import refresh_newly_enabled_layers, snapshot_active_layers
-
-    layers_before = snapshot_active_layers()
     old_ships = is_any_active("ships_military", "ships_cargo", "ships_civilian", "ships_passenger", "ships_tracked_yachts")
     old_mesh = is_any_active("sigint_meshtastic")
     old_aprs = is_any_active("sigint_aprs")
     old_viirs = is_any_active("viirs_nightlights")
+    old_datacenters = is_any_active("datacenters")
+    old_fishing = is_any_active("fishing_activity")
     changed = False
     for key, value in update.layers.items():
         if key in active_layers:
@@ -563,6 +516,8 @@ async def update_layers(update: LayerUpdate, request: Request):
     new_mesh = is_any_active("sigint_meshtastic")
     new_aprs = is_any_active("sigint_aprs")
     new_viirs = is_any_active("viirs_nightlights")
+    new_datacenters = is_any_active("datacenters")
+    new_fishing = is_any_active("fishing_activity")
     if old_ships and not new_ships:
         from services.ais_stream import stop_ais_stream
         stop_ais_stream()
@@ -606,7 +561,16 @@ async def update_layers(update: LayerUpdate, request: Request):
     if not old_viirs and new_viirs:
         _queue_viirs_change_refresh()
         logger.info("VIIRS change refresh queued (layer enabled)")
-    refresh_newly_enabled_layers(layers_before)
+    if not old_datacenters and new_datacenters:
+        from services.fetchers.infrastructure import fetch_datacenters
+
+        fetch_datacenters()
+        logger.info("Datacenters loaded (layer enabled)")
+    if not old_fishing and new_fishing:
+        from services.fetchers.geo import fetch_fishing_activity
+
+        fetch_fishing_activity()
+        logger.info("Fishing activity refresh queued (layer enabled)")
     return {"status": "ok"}
 
 

backend/routers/health.py

@@ -4,50 +4,10 @@ from fastapi.responses import JSONResponse
 from pydantic import BaseModel
 from limiter import limiter
 from auth import require_admin
+from services.data_fetcher import get_latest_data
 from services.schemas import HealthResponse
 import os
 
-# Health/SLO probes only need counts + freshness — not a full dashboard deepcopy.
-_HEALTH_DATA_KEYS: tuple[str, ...] = (
-    "last_updated",
-    "commercial_flights",
-    "military_flights",
-    "private_jets",
-    "ships",
-    "satellites",
-    "earthquakes",
-    "cctv",
-    "news",
-    "uavs",
-    "firms_fires",
-    "liveuamap",
-    "gdelt",
-    "uap_sightings",
-    "wastewater",
-    "fimi",
-    "space_weather",
-    "weather_alerts",
-    "volcanoes",
-    "prediction_markets",
-)
-
-
-def _health_data_snapshot() -> dict:
-    from services.fetchers._store import get_latest_data_subset_refs
-    from services.slo import SLO_REGISTRY
-
-    keys = tuple(dict.fromkeys((*_HEALTH_DATA_KEYS, *SLO_REGISTRY.keys())))
-    return get_latest_data_subset_refs(*keys)
-
-
-def _health_row_count(value) -> int:
-    if value is None:
-        return 0
-    try:
-        return len(value)
-    except TypeError:
-        return 0
-
 APP_VERSION = os.environ.get("_HEALTH_APP_VERSION", "0.9.82")
 
 router = APIRouter()
@@ -81,7 +41,7 @@ async def health_check(request: Request):
     from services.fetchers._store import get_source_timestamps_snapshot
     from services.slo import compute_all_statuses, summarise_statuses
 
-    d = _health_data_snapshot()
+    d = get_latest_data()
     last = d.get("last_updated")
     timestamps = get_source_timestamps_snapshot()
     slo_statuses = compute_all_statuses(d, timestamps)
@@ -142,18 +102,18 @@ async def health_check(request: Request):
         "version": _get_app_version(),
         "last_updated": last,
         "sources": {
-            "flights": _health_row_count(d.get("commercial_flights")),
-            "military": _health_row_count(d.get("military_flights")),
-            "ships": _health_row_count(d.get("ships")),
-            "satellites": _health_row_count(d.get("satellites")),
-            "earthquakes": _health_row_count(d.get("earthquakes")),
-            "cctv": _health_row_count(d.get("cctv")),
-            "news": _health_row_count(d.get("news")),
-            "uavs": _health_row_count(d.get("uavs")),
-            "firms_fires": _health_row_count(d.get("firms_fires")),
-            "liveuamap": _health_row_count(d.get("liveuamap")),
-            "gdelt": _health_row_count(d.get("gdelt")),
-            "uap_sightings": _health_row_count(d.get("uap_sightings")),
+            "flights": len(d.get("commercial_flights", [])),
+            "military": len(d.get("military_flights", [])),
+            "ships": len(d.get("ships", [])),
+            "satellites": len(d.get("satellites", [])),
+            "earthquakes": len(d.get("earthquakes", [])),
+            "cctv": len(d.get("cctv", [])),
+            "news": len(d.get("news", [])),
+            "uavs": len(d.get("uavs", [])),
+            "firms_fires": len(d.get("firms_fires", [])),
+            "liveuamap": len(d.get("liveuamap", [])),
+            "gdelt": len(d.get("gdelt", [])),
+            "uap_sightings": len(d.get("uap_sightings", [])),
         },
         "freshness": timestamps,
         "uptime_seconds": round(_time_mod.time() - _get_start_time()),
@@ -167,7 +127,4 @@ async def health_check(request: Request):
 @router.get("/api/debug-latest", dependencies=[Depends(require_admin)])
 @limiter.limit("30/minute")
 async def debug_latest_data(request: Request):
-    from services.fetchers._store import latest_data, _data_lock
-
-    with _data_lock:
-        return list(latest_data.keys())
+    return list(get_latest_data().keys())

backend/services/agent_shell_ws_token.py

@@ -1,52 +0,0 @@
-"""Short-lived, single-use WebSocket bootstrap tokens for the agent shell."""
-
-from __future__ import annotations
-
-import secrets
-import time
-from threading import Lock
-
-_TOKEN_TTL_SECONDS = 60.0
-_MAX_ACTIVE_TOKENS = 256
-
-_store: dict[str, float] = {}
-_lock = Lock()
-
-
-def _purge_expired(*, force: bool = False) -> None:
-    now = time.time()
-    with _lock:
-        expired = [token for token, expires in _store.items() if expires <= now]
-        for token in expired:
-            _store.pop(token, None)
-        if force and len(_store) > _MAX_ACTIVE_TOKENS:
-            for token in list(_store.keys())[: len(_store) - _MAX_ACTIVE_TOKENS]:
-                _store.pop(token, None)
-
-
-def mint_agent_shell_ws_token() -> tuple[str, int]:
-    """Return (token, expires_in_seconds)."""
-    _purge_expired()
-    token = secrets.token_urlsafe(32)
-    expires_at = time.time() + _TOKEN_TTL_SECONDS
-    with _lock:
-        if len(_store) >= _MAX_ACTIVE_TOKENS:
-            _purge_expired(force=True)
-        _store[token] = expires_at
-    return token, int(_TOKEN_TTL_SECONDS)
-
-
-def consume_agent_shell_ws_token(token: str) -> bool:
-    """Validate and burn a one-time token. Returns True when accepted."""
-    cleaned = str(token or "").strip()
-    if not cleaned:
-        return False
-    now = time.time()
-    with _lock:
-        expires_at = _store.pop(cleaned, None)
-    return expires_at is not None and expires_at > now
-
-
-def reset_agent_shell_ws_tokens_for_tests() -> None:
-    with _lock:
-        _store.clear()

backend/services/api_settings.py

@@ -6,7 +6,6 @@ Keys are stored in the backend .env file and loaded via python-dotenv.
 import os
 import re
 import tempfile
-import threading
 from pathlib import Path
 
 # Path to the backend .env file
@@ -79,24 +78,6 @@ API_REGISTRY = [
         "url": "https://earthquake.usgs.gov/",
         "required": False,
     },
-    {
-        "id": "firms_map_key",
-        "env_key": "FIRMS_MAP_KEY",
-        "name": "NASA FIRMS — MAP Key (optional)",
-        "description": "Optional NASA Earthdata MAP key for country-scoped VIIRS fire enrichment. Global VIIRS hotspots work without a key; set this only if you want per-country FIRMS detail. Free from NASA Earthdata.",
-        "category": "Geophysical",
-        "url": "https://firms.modaps.eosdis.nasa.gov/api/area/",
-        "required": False,
-    },
-    {
-        "id": "airframes_api_key",
-        "env_key": "AIRFRAMES_API_KEY",
-        "name": "Airframes.io — API Key",
-        "description": "ACARS/VDL datalink for plane dossiers. ShadowBroker bulk-ingests the global Airframes firehose (up to 100 messages per API call, one call every 2s, refill every 15 minutes) and indexes by tail/ICAO. Opening a dossier with no cache queues a single-plane lookup. Get a key at app.airframes.io → Dashboard → API Key.",
-        "category": "Aviation",
-        "url": "https://app.airframes.io/user/dashboard",
-        "required": False,
-    },
     {
         "id": "celestrak",
         "env_key": None,
@@ -376,17 +357,6 @@ def save_api_keys(updates: dict[str, str]) -> dict:
             flights.opensky_client.expires_at = 0
         except Exception:
             pass
-    if "AIRFRAMES_API_KEY" in clean:
-        try:
-            from services.fetchers.airframes import sync_airframes_messages
-
-            threading.Thread(
-                target=lambda: sync_airframes_messages(force=True),
-                daemon=True,
-                name="airframes-initial-sync",
-            ).start()
-        except Exception:
-            pass
 
     try:
         from services.config import get_settings

backend/services/cctv_pipeline.py

@@ -1031,8 +1031,7 @@ out body;
 # ---------------------------------------------------------------------------
 # DGT Spain — National Road Cameras
 # ---------------------------------------------------------------------------
-# Image URL pattern confirmed working: etraffic.dgt.es/camarasEtraffic/{id}.jpg
-# (DGT migrated 2026: old infocar.dgt.es/etraffic/data/camaras path now 302->etrafficWEB)
+# Image URL pattern confirmed working: infocar.dgt.es/etraffic/data/camaras/{id}.jpg
 # Source: DGT (Dirección General de Tráfico) — public open data (Ley 37/2007).
 # Author credit: Alborz Nazari (github.com/AlborzNazari) — PR #91
 
@@ -1066,10 +1065,10 @@ class DGTNationalIngestor(BaseCCTVIngestor):
         cameras = []
         probe_headers = {
             "Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
-            "Referer": "https://etraffic.dgt.es/",
+            "Referer": "https://infocar.dgt.es/",
         }
         for cam_id, lat, lon, description in self.KNOWN_CAMERAS:
-            media_url = f"https://etraffic.dgt.es/camarasEtraffic/{cam_id}.jpg"
+            media_url = f"https://infocar.dgt.es/etraffic/data/camaras/{cam_id}.jpg"
             if not _media_url_reachable(media_url, timeout=6, headers=probe_headers):
                 continue
             cameras.append({

backend/services/data_fetcher.py

@@ -77,7 +77,6 @@ from services.fetchers.infrastructure import (  # noqa: F401
     fetch_psk_reporter,
 )
 from services.fetchers.road_corridor_sat import fetch_road_corridor_trends  # noqa: F401
-from services.fetchers.airframes import sync_airframes_messages  # noqa: F401
 from services.fetchers.geo import (  # noqa: F401
     fetch_ships,
     fetch_airports,
@@ -480,27 +479,24 @@ def update_slow_data():
         fetch_military_bases,
         fetch_scanners,
         fetch_psk_reporter,
-        # weather_alerts + ukraine_alerts: owned by dedicated scheduler jobs
-        # (5 min and 2 min) — keep off slow tier to avoid duplicate upstream work.
+        fetch_weather_alerts,
         fetch_air_quality,
         fetch_fishing_activity,
         fetch_power_plants,
+        fetch_ukraine_air_raid_alerts,
         fetch_malware_threats,
         fetch_cyber_threats,
         fetch_scm_suppliers,
     ]
     _run_tasks("slow-tier", slow_funcs)
-    # Run correlation engine after all data is fresh (skip when overlay is off).
+    # Run correlation engine after all data is fresh
     try:
-        from services.fetchers._store import is_any_active
         from services.correlation_engine import compute_correlations
-
-        if is_any_active("correlations"):
-            with _data_lock:
-                snapshot = dict(latest_data)
-            correlations = compute_correlations(snapshot)
-            with _data_lock:
-                latest_data["correlations"] = correlations
+        with _data_lock:
+            snapshot = dict(latest_data)
+        correlations = compute_correlations(snapshot)
+        with _data_lock:
+            latest_data["correlations"] = correlations
     except Exception as e:
         logger.error("Correlation engine failed: %s", e)
     try:
@@ -1264,17 +1260,6 @@ def start_scheduler():
         next_run_time=datetime.utcnow() + timedelta(minutes=5),  # first snapshot 5m after startup
     )
 
-    _airframes_interval_m = max(5, int(os.environ.get("AIRFRAMES_SYNC_INTERVAL_MINUTES", "15")))
-    _scheduler.add_job(
-        lambda: _run_task_with_health(sync_airframes_messages, "sync_airframes_messages"),
-        "interval",
-        minutes=_airframes_interval_m,
-        id="airframes_datalink",
-        max_instances=1,
-        misfire_grace_time=120,
-        next_run_time=datetime.utcnow() + timedelta(seconds=90),
-    )
-
     _scheduler.start()
     logger.info("Scheduler started.")
 

backend/services/fetchers/_store.py

@@ -334,15 +334,15 @@ active_layers: dict[str, bool] = {
     "ships_passenger": True,
     "ships_tracked_yachts": True,
     "earthquakes": True,
-    "cctv": False,
+    "cctv": True,
     "ukraine_frontline": True,
     "global_incidents": True,
     "gps_jamming": True,
     "kiwisdr": True,
     "scanners": True,
-    "firms": False,
+    "firms": True,
     "internet_outages": True,
-    "datacenters": False,
+    "datacenters": True,
     "military_bases": True,
     "sigint_meshtastic": True,
     "sigint_aprs": True,
@@ -353,9 +353,9 @@ active_layers: dict[str, bool] = {
     "satnogs": True,
     "tinygs": True,
     "ukraine_alerts": True,
-    "power_plants": False,
+    "power_plants": True,
     "viirs_nightlights": False,
-    "psk_reporter": False,
+    "psk_reporter": True,
     "correlations": True,
     "contradictions": True,
     "uap_sightings": True,

backend/services/fetchers/acars_summarize.py

@@ -1,570 +0,0 @@
-"""Heuristics to summarize ACARS/VDL payloads across airlines for dossier display."""
-
-from __future__ import annotations
-
-import re
-from typing import Any
-
-# --- shared patterns ---
-
-_ICAO_AIRPORT = re.compile(r"\b([A-Z]{4})\b")
-_TAIL = re.compile(r"\b(?:N[0-9A-Z]{3,6}|G-[A-Z]{4,5}|[A-Z]-[A-Z]{4,5}|[A-Z]{2}-[A-Z]{3,4})\b")
-# Major carriers — explicit list avoids matching FL280, GS450, etc.
-_FLIGHT = re.compile(
-    r"\b(?:"
-    r"WN|SWA|UA|UAL|AA|AAL|DL|DAL|AS|ASA|B6|JBU|NK|NKS|F9|FFT|G4|HA|HAL|SY|MX|"
-    r"FDX|UPS|GTI|ABX|ATN|RCH|CNV|EVAC|SAM|REACH|"
-    r"BA|BAW|AF|AFR|LH|DLH|KL|KLM|QF|QFA|EK|UAE|QR|QTR|TK|THY|AC|ACA|WS|WJA|"
-    r"FR|RYR|U2|EZY|VS|VIR|NH|ANA|JL|JAL|CX|CPA|SQ|SIA|NZ|ANZ|"
-    r"UA|CO|NW|US|HP|TW|VX|AS|QX|OO|YX|MQ|OH|9E|"
-    r"JT|JSA|VA|VOZ|NZ|QF|EK|ET|MS|SU|LO|SK|AY|IB|UX|TP|TAP"
-    r")\d{1,5}\b",
-    re.I,
-)
-# IATA flight numbers on FI lines and standalone (e.g. UO614, CX889).
-_FI_FLIGHT = re.compile(r"\b([A-Z]{2,3}\d{1,4})\b")
-_NON_FLIGHT_TOKENS = frozenset(
-    {"FL", "FT", "GS", "KT", "RW", "NM", "TD", "TO", "ON", "IN", "OF", "AT", "DA", "AA", "AD"}
-)
-_FI_BLOCK = re.compile(
-    r"FI\s+([A-Z0-9]{2,5}\d{1,5})"
-    r"(?:/AN\s+([A-Z0-9\-]+))?"
-    r"(?:/DA\s+([A-Z]{4}))?"
-    r"(?:/(?:AA|AD|DS)\s+([A-Z]{4}))?",
-    re.I,
-)
-_AC_TYPE = re.compile(
-    r"\b(?:B\d{3,4}(?:-\d{3}|MAX|ER|LR|F)?|A\d{3,4}(?:-\d{3}|NEO|LR)?|"
-    r"E\d{3}|MD-\d{2}|DC-\d{2}|B77[0-9LWR]?|B78[79]|A35[09]|A33[0-9]|CRJ\d{2,3}|E\d{3})\b",
-    re.I,
-)
-
-# --- message family patterns ---
-
-_TRACK_HEADER = re.compile(
-    r"^\+\+86501,([^,]+),([^,]+),(\d{6}),([^,]+),([A-Z]{4}),([A-Z]{4})",
-    re.I,
-)
-_POS_HEADER = re.compile(r"^POS(N?\d{4,5}[NS]?\d{4,5}[EW]?)", re.I)
-_POS_COORDS = re.compile(r"^N?(\d{4,5})([NS])(\d{4,6})([EW])", re.I)
-_WAYPOINT = re.compile(
-    r"^(?:N)?(\d{1,2}\d{2}\.\d),W(\d{1,3}\d{2}\.\d),(\d{6}),(\d+),",
-    re.I,
-)
-_PERF_HEADER = re.compile(
-    r"^[\w]+,(\d+),([^,]+),(\d{6}),([^,]+),([A-Z]{4}),([A-Z]{4})",
-    re.I,
-)
-_PHASE_SNAPSHOT = re.compile(
-    r"^(\d{2}\.\d{2}\.\d{2}),(CL|CR|DE|TO|LD|ER|GND),[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,"
-    r"N(\d+)\.(\d+),W(\d+)\.(\d+)",
-    re.I,
-)
-_TRAJECTORY_HEADER = re.compile(r"^76401\s*$", re.I)
-_TRAJECTORY_ROUTE = re.compile(r"^02E24([A-Z]{4})([A-Z]{4})\s*$", re.I)
-_COMPRESSED_WP = re.compile(r"^N(\d{5})W(\d{5})", re.I)
-_FPN = re.compile(r"^FPN/?", re.I)
-_OOOI_TIMES = re.compile(r"\b(OUT|OFF|ON|IN)\s*(\d{4,6})\b", re.I)
-_OOOI_STATUS = re.compile(r"\b(OUT|OFF|ON|IN)\s*,\s*(LO|CL|ON|OF|CLOS)\b", re.I)
-_ETA = re.compile(r"\bETA\s+(\d{3,4}Z?)\b", re.I)
-_DEP_ARR = re.compile(r"^(DEP|ARR|DLA|ALR)\b", re.I)
-_WX = re.compile(r"^(?:WXR?\d*|WX\s|MET\b|/WX\b)", re.I)
-_REQ = re.compile(r"^(?:REQ|REQUEST)\b", re.I)
-_LDR = re.compile(r"^LDR\d+", re.I)
-_PIREP = re.compile(r"^#(?:CFB|DFB)", re.I)
-_ATN = re.compile(r"^USADCXA\.AT1\.", re.I)
-_CPDLC = re.compile(r"^(?:DM-|UM-|AT1\.|ATC\s)", re.I)
-_ENG = re.compile(r"^(?:ENG\d|/ENG|OILTEMP|EGT\b)", re.I)
-_DOOR = re.compile(r"^(?:DOOR|CABIN|SMOKE)\b", re.I)
-_VDL_FRAME = re.compile(r"^[0-9A-F]{6,8}[A-Z]?\s*$", re.I)
-_FRAGMENT = re.compile(r"^[,0\s]+(?:,\d{5,8},\d{5,8},\d{5,8})*$", re.I)
-_GARBLED_VDL = re.compile(r"[)Z][A-Z0-9,\-:]{20,}")
-_MOSTLY_OPAQUE = re.compile(r"^[0-9A-Fa-f\s.\-+/,]{40,}$")
-_FREE_TEXT_POS = re.compile(
-    r"^POS\s+N?(\d{1,2}\.\d+)\s+([NS])\s+W?(\d{1,3}\.\d+)\s+([EW])\s+FL(\d{3})",
-    re.I,
-)
-_CLIMB_REQ = re.compile(r"\b(?:CLIMB|DESCEND|REQUEST)\s+(?:FL)?(\d{2,3})\b", re.I)
-
-_LABEL_HINTS: dict[str, str] = {
-    "00": "out (gate)",
-    "01": "off (takeoff)",
-    "02": "on (landing)",
-    "03": "in (gate)",
-    "10": "position",
-    "15": "waypoint",
-    "20": "position",
-    "40": "ops / clearance",
-    "44": "OOOI + position",
-    "80": "weather",
-    "81": "wind",
-    "B1": "engine 1",
-    "B2": "engine 2",
-    "B3": "engine 3",
-    "B4": "engine 4",
-    "M1": "maintenance",
-    "M2": "maintenance",
-    "M3": "maintenance",
-    "M4": "maintenance",
-    "Q0": "position / OOOI",
-    "H1": "terminal",
-    "D0": "ATC clearance",
-    "S1": "system status",
-    "SA": "system status",
-    "SB": "system status",
-    "4T": "met report",
-    "5Z": "free text",
-}
-
-
-def _result(
-    summary: str,
-    *,
-    kind: str,
-    readable: bool = True,
-    hidden: bool = False,
-) -> dict[str, Any]:
-    return {
-        "summary": summary,
-        "kind": kind,
-        "readable": readable,
-        "hidden": hidden,
-    }
-
-
-def _phase_name(code: str) -> str:
-    return {
-        "CL": "climb",
-        "CR": "cruise",
-        "DE": "descent",
-        "ER": "en route",
-        "TO": "takeoff",
-        "LD": "landed",
-        "ON": "on ground",
-        "OF": "off block",
-        "GND": "on ground",
-        "LO": "level",
-    }.get(code.upper(), code.upper() or "unknown")
-
-
-def _fmt_coords(lat_deg: str, lat_frac: str, lon_deg: str, lon_frac: str) -> str:
-    return f"{int(lat_deg)}°{lat_frac}'N {int(lon_deg)}°{lon_frac}'W"
-
-
-def _parse_pos_coords(token: str) -> str | None:
-    token = token.upper().lstrip("POS")
-    match = _POS_COORDS.match(token)
-    if not match:
-        return None
-    lat, lat_dir, lon, lon_dir = match.groups()
-    lat_v = f"{int(lat[:2])}°{lat[2:]}.{lat[4:] if len(lat) > 4 else '0'}'{lat_dir}"
-    lon_v = f"{int(lon[:3])}°{lon[3:]}.{lon[5:] if len(lon) > 5 else '0'}'{lon_dir}"
-    return f"{lat_v} {lon_v}"
-
-
-def _extract_route(raw: str) -> str:
-    fi = _FI_BLOCK.search(raw)
-    if fi:
-        flight, _tail, dep, dest = fi.groups()
-        parts = [flight.upper()]
-        if dep and dest:
-            parts.append(f"{dep}→{dest}")
-        elif dep:
-            parts.append(f"from {dep}")
-        elif dest:
-            parts.append(f"to {dest}")
-        return " · ".join(parts)
-
-    airports = _ICAO_AIRPORT.findall(raw)
-    # Filter duplicates while preserving order
-    seen: set[str] = set()
-    ordered: list[str] = []
-    for apt in airports:
-        if apt in seen:
-            continue
-        seen.add(apt)
-        ordered.append(apt)
-    if len(ordered) >= 2:
-        return f"{ordered[0]}→{ordered[-1]}"
-    if ordered:
-        return ordered[0]
-    return ""
-
-
-def _extract_flight(raw: str) -> str:
-    fi = _FI_BLOCK.search(raw)
-    if fi and fi.group(1):
-        return fi.group(1).upper()
-    for match in _FLIGHT.finditer(raw):
-        return match.group(0).upper()
-    for match in _FI_FLIGHT.finditer(raw):
-        token = match.group(1).upper()
-        prefix = re.match(r"^([A-Z]+)", token)
-        if prefix and prefix.group(1) not in _NON_FLIGHT_TOKENS:
-            return token
-    return ""
-
-
-def _extract_phase_snapshot(raw: str) -> str | None:
-    for line in raw.splitlines():
-        match = _PHASE_SNAPSHOT.match(line.strip())
-        if not match:
-            continue
-        time_s, phase, lat_d, lat_f, lon_d, lon_f = match.groups()
-        coords = _fmt_coords(lat_d, lat_f, lon_d, lon_f)
-        return f"{_phase_name(phase)} · {coords} · {time_s}Z"
-    return None
-
-
-def _has_aircraft_context(raw: str) -> bool:
-    head = raw[:160].upper()
-    if _AC_TYPE.search(head):
-        return True
-    if _FLIGHT.search(head):
-        return True
-    if _FI_BLOCK.search(head):
-        return True
-    return False
-
-
-def _is_fragment(raw: str) -> bool:
-    first = raw.splitlines()[0].strip()
-    if _FRAGMENT.match(first):
-        return True
-    if re.match(r"^[,0]{1,12}$", first):
-        return True
-    if first.startswith("000000") or first.startswith(",000000"):
-        return True
-    if re.match(r"^\d{2}\.\d{2}\.\d{2},", first) and not _has_aircraft_context(raw):
-        return True
-    return False
-
-
-def _summarize_oooi(raw: str, label: str) -> dict[str, Any] | None:
-    times = _OOOI_TIMES.findall(raw)
-    statuses = _OOOI_STATUS.findall(raw)
-    if not times and not statuses and label not in {"00", "01", "02", "03", "44", "Q0"}:
-        return None
-
-    events: list[str] = []
-    for event, value in times:
-        events.append(f"{event.upper()} {value}")
-    for event, status in statuses:
-        events.append(f"{event.upper()} ({_phase_name(status)})")
-
-    if label in {"00", "01", "02", "03"} and not events:
-        events.append(_LABEL_HINTS[label])
-
-    if not events and "ON  ,LO" not in raw and "OFF,OFF" not in raw:
-        return None
-
-    route = _extract_route(raw)
-    flight = _extract_flight(raw)
-    prefix = "OOOI"
-    if label in _LABEL_HINTS:
-        prefix = f"OOOI ({_LABEL_HINTS[label]})"
-    bits = [prefix]
-    if flight:
-        bits.append(flight)
-    if route:
-        bits.append(route)
-    if events:
-        bits.append(", ".join(events[:4]))
-    return _result(" · ".join(bits), kind="oooi")
-
-
-def _summarize_position(raw: str, first_line: str) -> dict[str, Any] | None:
-    upper = first_line.upper()
-    pos_token = re.search(r"POSN?\d", raw, re.I)
-    if not (upper.startswith("POS") or _POS_HEADER.match(first_line) or pos_token):
-        return None
-
-    coord_line = first_line
-    if pos_token and not upper.startswith("POS"):
-        coord_line = raw[pos_token.start() :].split(",")[0]
-
-    coords = _parse_pos_coords(coord_line)
-    free = _FREE_TEXT_POS.match(raw)
-    fl = ""
-    if free:
-        lat, lat_dir, lon, lon_dir, fl = free.groups()
-        coords = f"{lat}°{lat_dir} {lon}°{lon_dir}"
-        fl = f"FL{fl}"
-
-    route = _extract_route(raw)
-    flight = _extract_flight(raw)
-    parts = ["Position report"]
-    if flight:
-        parts.append(flight)
-    if route:
-        parts.append(route)
-    if coords:
-        parts.append(coords)
-    if fl:
-        parts.append(fl)
-    elif re.search(r"\bFL?\d{3}\b", raw):
-        fl_match = re.search(r"\bFL?(\d{2,3})\b", raw)
-        if fl_match:
-            parts.append(f"FL{fl_match.group(1)}")
-    return _result(" · ".join(parts), kind="position")
-
-
-def _summarize_performance(raw: str, first_line: str) -> dict[str, Any] | None:
-    match = _PERF_HEADER.match(first_line)
-    if not match or not _AC_TYPE.search(first_line):
-        return None
-
-    _serial, ac_type, _date, flight, dep, dest = match.groups()
-    phase_bits = _extract_phase_snapshot(raw) or ""
-    extra = f" · {phase_bits}" if phase_bits else ""
-
-    if "FHP" in raw or "SIN," in raw or "SOU," in raw:
-        title, kind = "Engine health (FHP)", "engine_health"
-    elif "OATTO" in raw or "LPACKCL" in raw or "RPACKCL" in raw:
-        title, kind = "Pack temperature", "pack_temp"
-    elif "FLAPS" in raw.upper():
-        title, kind = "Climb performance", "climb_perf"
-    elif "FRE," in raw or "FEX," in raw:
-        title, kind = "Fuel/performance snapshot", "fuel_perf"
-    else:
-        title, kind = "Flight performance", "performance"
-
-    return _result(
-        f"{title} · {flight} · {ac_type} · {dep}→{dest}{extra}",
-        kind=kind,
-    )
-
-
-def _summarize_by_label(label: str, raw: str, first_line: str) -> dict[str, Any] | None:
-    label_u = label.upper()
-    hint = _LABEL_HINTS.get(label_u, "")
-
-    if label_u in {"B1", "B2", "B3", "B4"} or _ENG.match(first_line):
-        eng = label_u if label_u.startswith("B") else "Engine"
-        return _result(f"{eng} data report", kind="engine", readable=bool(hint))
-
-    if label_u.startswith("M") and label_u[1:2].isdigit():
-        return _result(f"Maintenance ({hint or 'system report'})", kind="maintenance")
-
-    if label_u in {"80", "81", "4T"} or _WX.match(first_line):
-        apt = _ICAO_AIRPORT.search(raw)
-        apt_s = f" · {apt.group(1)}" if apt else ""
-        return _result(f"Weather report{apt_s}", kind="weather")
-
-    if label_u == "D0" or _REQ.match(first_line) or _CLIMB_REQ.search(raw):
-        climb = _CLIMB_REQ.search(raw)
-        if climb:
-            return _result(f"Altitude request · FL{climb.group(1)}", kind="request")
-        return _result("ATC / ops request", kind="request")
-
-    if label_u in {"40", "5Z"} and len(raw) < 200:
-        text = raw.replace("\n", " · ")[:140]
-        return _result(f"Ops message · {text}", kind="ops")
-
-    return None
-
-
-def summarize_datalink_message(
-    *,
-    label: str = "",
-    text: str = "",
-    source_type: str = "",
-) -> dict[str, Any]:
-    """Return {summary, kind, readable, hidden} for a cached datalink message."""
-    raw = (text or "").strip()
-    if not raw:
-        return _result("", kind="empty", readable=False, hidden=True)
-
-    first_line = raw.splitlines()[0].strip()
-    upper = first_line.upper()
-    label_u = label.upper()
-
-    if _is_fragment(raw):
-        return _result(
-            "Split telemetry fragment (part of a longer VDL message)",
-            kind="fragment",
-            readable=False,
-            hidden=True,
-        )
-
-    if _ATN.match(first_line) or _CPDLC.match(first_line):
-        tail = _TAIL.search(raw)
-        return _result(
-            "Datalink protocol / CPDLC header" + (f" · {tail.group(0)}" if tail else ""),
-            kind="protocol",
-            readable=False,
-            hidden=True,
-        )
-
-    if label_u == "37" or (_VDL_FRAME.match(first_line) and len(raw) < 160):
-        if _GARBLED_VDL.search(raw) or len(raw) < 160:
-            return _result("VDL binary frame (undecoded)", kind="vdl_binary", readable=False, hidden=True)
-
-    # --- structured families (order matters) ---
-
-    if _DEP_ARR.match(first_line):
-        kind_word = first_line.split()[0].upper()
-        route = _extract_route(raw)
-        flight = _extract_flight(raw)
-        title = {"DEP": "Departure", "ARR": "Arrival", "DLA": "Delay", "ALR": "Alert"}.get(
-            kind_word, kind_word
-        )
-        bits = [title]
-        if flight:
-            bits.append(flight)
-        if route:
-            bits.append(route)
-        return _result(" · ".join(bits), kind=kind_word.lower())
-
-    oooi = _summarize_oooi(raw, label_u)
-    if oooi:
-        return oooi
-
-    match = _TRACK_HEADER.match(first_line)
-    if match:
-        tail, ac_type, _date, flight, dep, dest = match.groups()
-        lines = [line.strip() for line in raw.splitlines() if line.strip()]
-        waypoint_lines = [line for line in lines if _WAYPOINT.match(line.lstrip("N"))]
-        phase = ""
-        if waypoint_lines:
-            parts = waypoint_lines[-1].rstrip(",").split(",")
-            if len(parts) >= 8:
-                phase = _phase_name(parts[7])
-        wp_count = len(waypoint_lines) or max(0, len(lines) - 2)
-        summary = (
-            f"Track report · {flight} · {tail} ({ac_type}) · {dep}→{dest}"
-            + (f" · {wp_count} waypoint(s)" + (f" · {phase}" if phase else ""))
-        )
-        return _result(summary, kind="track")
-
-    pos = _summarize_position(raw, first_line)
-    if pos:
-        return pos
-
-    if _FPN.match(first_line):
-        route = _extract_route(raw)
-        flight = _extract_flight(raw)
-        bits = ["Flight plan"]
-        if flight:
-            bits.append(flight)
-        if route:
-            bits.append(route)
-        return _result(" · ".join(bits), kind="flight_plan")
-
-    if _PIREP.match(first_line):
-        return _result("Pilot report (PIREP)", kind="pirep")
-
-    lines = [line.strip() for line in raw.splitlines() if line.strip()]
-    if _TRAJECTORY_HEADER.match(first_line) or (len(lines) >= 2 and _TRAJECTORY_ROUTE.match(lines[1])):
-        route = ""
-        route_match = next((m for line in lines if (m := _TRAJECTORY_ROUTE.match(line))), None)
-        if route_match:
-            route = f" · {route_match.group(1)}→{route_match.group(2)}"
-        wp_count = sum(1 for line in lines if _COMPRESSED_WP.match(line))
-        return _result(
-            f"Trajectory / ADS report{route}" + (f" · {wp_count} point(s)" if wp_count else ""),
-            kind="trajectory",
-        )
-
-    perf = _summarize_performance(raw, first_line)
-    if perf:
-        return perf
-
-    if _LDR.match(first_line):
-        route = _extract_route(raw)
-        return _result(f"Load report · {route}" if route else "Load report", kind="load")
-
-    if _WX.match(first_line):
-        route = _extract_route(raw)
-        return _result(f"Weather request · {route or 'en route'}", kind="weather")
-
-    if _DOOR.match(first_line):
-        return _result("Cabin / door advisory", kind="cabin")
-
-    if _WAYPOINT.match(first_line.lstrip("N")):
-        parts = first_line.lstrip("N").rstrip(",").split(",")
-        if len(parts) >= 4:
-            lat, lon, _t, alt = parts[0], parts[1], parts[2], parts[3]
-            phase = _phase_name(parts[7]) if len(parts) >= 8 else ""
-            summary = f"Waypoint · {lat},{lon} · alt {alt} ft" + (f" · {phase}" if phase else "")
-            return _result(summary, kind="waypoint")
-
-    label_summary = _summarize_by_label(label_u, raw, first_line)
-    if label_summary:
-        return label_summary
-
-    flight = _extract_flight(raw)
-    route = _extract_route(raw)
-    if flight and route:
-        return _result(f"Datalink · {flight} · {route}", kind="flight")
-
-    eta = _ETA.search(raw)
-    if eta and flight:
-        return _result(f"ETA update · {flight} · {eta.group(1)}", kind="eta")
-
-    if len(raw) < 100 and not _MOSTLY_OPAQUE.match(raw) and not _GARBLED_VDL.search(raw):
-        clean = raw.replace("\n", " · ")
-        if label_u in _LABEL_HINTS:
-            return _result(f"{_LABEL_HINTS[label_u].title()} · {clean}", kind="short")
-        return _result(clean, kind="short")
-
-    digit_ratio = sum(ch.isdigit() for ch in raw) / max(len(raw), 1)
-    if digit_ratio > 0.55 or _MOSTLY_OPAQUE.match(raw.replace(" ", "")) or _GARBLED_VDL.search(raw):
-        return _result(
-            "Binary / proprietary telemetry (undecoded)",
-            kind="vdl_binary",
-            readable=False,
-            hidden=True,
-        )
-
-    if label_u in _LABEL_HINTS:
-        return _result(
-            f"{_LABEL_HINTS[label_u].title()} message",
-            kind=label_u.lower(),
-            readable=False,
-            hidden=False,
-        )
-
-    return _result(
-        first_line[:100] + ("…" if len(first_line) > 100 else ""),
-        kind="raw",
-        readable=False,
-        hidden=False,
-    )
-
-
-def prepare_datalink_display(messages: list[dict[str, Any]]) -> dict[str, Any]:
-    """Attach summaries and filter noise for dossier display."""
-    enriched: list[dict[str, Any]] = []
-    hidden_count = 0
-    seen_summaries: set[str] = set()
-
-    for message in messages:
-        meta = summarize_datalink_message(
-            label=str(message.get("label") or ""),
-            text=str(message.get("text") or ""),
-            source_type=str(message.get("source_type") or ""),
-        )
-        item = {**message, **meta}
-        if item.get("hidden"):
-            hidden_count += 1
-            continue
-
-        # Drop back-to-back duplicate summaries (common with multi-part VDL)
-        sig = f"{item.get('kind')}|{item.get('summary')}"
-        if sig in seen_summaries and item.get("kind") not in {"short", "ops", "request"}:
-            hidden_count += 1
-            continue
-        seen_summaries.add(sig)
-
-        enriched.append(item)
-
-    return {
-        "messages": enriched,
-        "hidden_count": hidden_count,
-        "total_count": len(messages),
-    }
-
-
-def attach_summaries(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
-    return prepare_datalink_display(messages)["messages"]

backend/services/fetchers/airframes.py

@@ -1,611 +0,0 @@
-"""Airframes.io ACARS/VDL datalink ingest — staggered queue cache for plane dossiers."""
-
-from __future__ import annotations
-
-import json
-import logging
-import os
-import re
-import threading
-import time
-from collections import deque
-from datetime import datetime, timedelta, timezone
-from pathlib import Path
-from typing import Any
-
-import requests
-
-logger = logging.getLogger("services.airframes")
-
-API_BASE = os.environ.get("AIRFRAMES_API_BASE", "https://api.airframes.io/v1").rstrip("/")
-SYNC_INTERVAL_MINUTES = max(5, int(os.environ.get("AIRFRAMES_SYNC_INTERVAL_MINUTES", "15")))
-MAX_BULK_PAGES_PER_CYCLE = max(1, int(os.environ.get("AIRFRAMES_MAX_PAGES_PER_SYNC", "28")))
-MESSAGES_PER_AIRCRAFT = max(5, int(os.environ.get("AIRFRAMES_MESSAGES_PER_AIRCRAFT", "40")))
-RETENTION_HOURS = max(6, int(os.environ.get("AIRFRAMES_RETENTION_HOURS", "48")))
-# 2s between calls => 30/min, safely under Airframes 60/min cap.
-REQUEST_PAUSE_S = float(os.environ.get("AIRFRAMES_REQUEST_PAUSE_S", "2.0"))
-PRIORITY_LOOKBACK_HOURS = max(6, int(os.environ.get("AIRFRAMES_PRIORITY_LOOKBACK_HOURS", "48")))
-FETCH_TIMEOUT_S = max(5, int(os.environ.get("AIRFRAMES_FETCH_TIMEOUT_S", "20")))
-
-_DATA_DIR = Path(os.environ.get("SB_DATA_DIR", str(Path(__file__).resolve().parents[2] / "data")))
-if not _DATA_DIR.is_absolute():
-    _DATA_DIR = Path(__file__).resolve().parents[2] / _DATA_DIR
-_CACHE_PATH = _DATA_DIR / "airframes_datalink_cache.json"
-
-_lock = threading.Lock()
-_queue_lock = threading.Lock()
-_worker_guard = threading.Lock()
-_queue: deque[dict[str, Any]] = deque()
-_queued_aircraft_keys: set[str] = set()
-_bulk_cursor: dict[str, Any] = {"since_iso": "", "before_id": None, "pages": 0}
-_worker_started = False
-_cache_loaded = False
-_save_timer: threading.Timer | None = None
-_save_timer_lock = threading.Lock()
-_api_key_known_configured: bool | None = None
-_cache: dict[str, Any] = {
-    "last_sync_at": None,
-    "last_success_at": None,
-    "last_error": None,
-    "pages_fetched": 0,
-    "messages_ingested": 0,
-    "bulk_pages_this_cycle": 0,
-    "ticks_processed": 0,
-    "by_icao": {},
-    "by_tail": {},
-    "by_callsign": {},
-}
-
-
-def _utc_now() -> datetime:
-    return datetime.now(timezone.utc)
-
-
-def _iso(dt: datetime) -> str:
-    return dt.astimezone(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
-
-
-def _parse_ts(value: str | None) -> datetime | None:
-    if not value:
-        return None
-    try:
-        cleaned = value.replace("Z", "+00:00")
-        return datetime.fromisoformat(cleaned).astimezone(timezone.utc)
-    except ValueError:
-        return None
-
-
-def api_key_configured() -> bool:
-    global _api_key_known_configured
-    if os.environ.get("AIRFRAMES_API_KEY", "").strip():
-        _api_key_known_configured = True
-        return True
-    if _api_key_known_configured is False:
-        return False
-    from services.api_settings import load_persisted_api_keys_into_environ
-
-    load_persisted_api_keys_into_environ()
-    _api_key_known_configured = bool(os.environ.get("AIRFRAMES_API_KEY", "").strip())
-    return _api_key_known_configured
-
-
-def _norm_hex(value: str | None) -> str:
-    return (value or "").strip().lower()
-
-
-def _norm_tail(value: str | None) -> str:
-    return re.sub(r"[^A-Z0-9]", "", (value or "").strip().upper())
-
-
-def _norm_callsign(value: str | None) -> str:
-    return re.sub(r"\s+", "", (value or "").strip().upper())
-
-
-def _aircraft_queue_key(entry: dict[str, str]) -> str:
-    return f"{entry.get('icao24', '')}|{entry.get('registration', '')}|{entry.get('callsign', '')}"
-
-
-def _tail_lookup_keys(value: str | None) -> list[str]:
-    tail = _norm_tail(value)
-    if not tail:
-        return []
-    keys = [tail]
-    raw = (value or "").strip().upper()
-    if raw and raw not in keys:
-        keys.append(raw)
-    return keys
-
-
-def _load_cache_if_cold() -> None:
-    global _cache, _cache_loaded
-    if _cache_loaded:
-        return
-    loaded: dict[str, Any] | None = None
-    if _CACHE_PATH.exists():
-        try:
-            with _CACHE_PATH.open(encoding="utf-8") as handle:
-                parsed = json.load(handle)
-            if isinstance(parsed, dict):
-                loaded = parsed
-        except (OSError, json.JSONDecodeError, ValueError) as exc:
-            logger.warning("Failed to load Airframes cache: %s", exc)
-    with _lock:
-        if _cache_loaded:
-            return
-        if loaded:
-            _cache.update(loaded)
-            _cache.setdefault("by_callsign", {})
-        _cache_loaded = True
-
-
-def _persist_cache_now() -> None:
-    with _lock:
-        snapshot = json.dumps(_cache, indent=2, ensure_ascii=False) + "\n"
-    _DATA_DIR.mkdir(parents=True, exist_ok=True)
-    tmp = _CACHE_PATH.with_suffix(".tmp")
-    tmp.write_text(snapshot, encoding="utf-8")
-    tmp.replace(_CACHE_PATH)
-
-
-def _schedule_cache_persist() -> None:
-    global _save_timer
-
-    def _flush() -> None:
-        global _save_timer
-        try:
-            _persist_cache_now()
-        except OSError as exc:
-            logger.warning("Failed to save Airframes cache: %s", exc)
-        finally:
-            with _save_timer_lock:
-                _save_timer = None
-
-    with _save_timer_lock:
-        if _save_timer is not None:
-            _save_timer.cancel()
-        _save_timer = threading.Timer(0.75, _flush)
-        _save_timer.daemon = True
-        _save_timer.start()
-
-
-def _save_cache() -> None:
-    _schedule_cache_persist()
-
-
-def _compact_message(raw: dict[str, Any]) -> dict[str, Any] | None:
-    text = (raw.get("text") or raw.get("data") or "").strip()
-    if not text:
-        return None
-    msg_id = raw.get("id")
-    if msg_id is None:
-        return None
-    return {
-        "id": int(msg_id),
-        "timestamp": raw.get("timestamp") or raw.get("createdAt") or "",
-        "label": str(raw.get("label") or "").strip(),
-        "text": text[:500],
-        "source_type": str(raw.get("sourceType") or raw.get("source") or "").strip(),
-        "tail": _norm_tail(raw.get("tail")),
-        "flight_number": _norm_callsign(raw.get("flightNumber")),
-        "from_hex": _norm_hex(raw.get("fromHex")),
-        "to_hex": _norm_hex(raw.get("toHex")),
-    }
-
-
-def _bucket_key(store: dict[str, list], key: str, message: dict[str, Any]) -> None:
-    if not key:
-        return
-    bucket = store.setdefault(key, [])
-    if any(existing.get("id") == message["id"] for existing in bucket):
-        return
-    bucket.append(message)
-    bucket.sort(key=lambda item: item.get("timestamp") or "", reverse=True)
-    del bucket[MESSAGES_PER_AIRCRAFT:]
-
-
-def _index_message(compact: dict[str, Any]) -> None:
-    for hex_code in (compact.get("from_hex"), compact.get("to_hex")):
-        if hex_code:
-            _bucket_key(_cache["by_icao"], hex_code, compact)
-    for tail_key in _tail_lookup_keys(compact.get("tail")):
-        _bucket_key(_cache["by_tail"], tail_key, compact)
-    callsign = compact.get("flight_number")
-    if callsign:
-        _bucket_key(_cache["by_callsign"], callsign, compact)
-
-
-def _prune_store(store: dict[str, list]) -> None:
-    cutoff = _utc_now() - timedelta(hours=RETENTION_HOURS)
-    for key in list(store.keys()):
-        kept = []
-        for message in store.get(key, []):
-            ts = _parse_ts(message.get("timestamp"))
-            if ts is None or ts >= cutoff:
-                kept.append(message)
-        if kept:
-            store[key] = kept[:MESSAGES_PER_AIRCRAFT]
-        else:
-            del store[key]
-
-
-def _ingest_message(message: dict[str, Any]) -> bool:
-    compact = _compact_message(message)
-    if not compact:
-        return False
-    _index_message(compact)
-    return True
-
-
-def _ingest_messages_batch(raw_messages: list[dict[str, Any]]) -> int:
-    if not raw_messages:
-        return 0
-    ingested = 0
-    with _lock:
-        _cache.setdefault("by_callsign", {})
-        for raw in raw_messages:
-            if _ingest_message(raw):
-                ingested += 1
-        if ingested:
-            _cache["messages_ingested"] = int(_cache.get("messages_ingested", 0)) + ingested
-            _cache["last_success_at"] = _iso(_utc_now())
-            _save_cache()
-    return ingested
-
-
-def _fetch_messages(*, api_key: str, params: dict[str, Any]) -> list[dict[str, Any]]:
-    response = requests.get(
-        f"{API_BASE}/messages",
-        headers={"Authorization": f"Bearer {api_key}"},
-        params=params,
-        timeout=FETCH_TIMEOUT_S,
-    )
-    if response.status_code == 404:
-        logger.debug("Airframes messages 404 for params=%s", params)
-        return []
-    if response.status_code == 429:
-        retry_after = int(response.headers.get("Retry-After", "60"))
-        raise RuntimeError(f"rate_limited:{retry_after}")
-    response.raise_for_status()
-    payload = response.json()
-    if not isinstance(payload, list):
-        return []
-    return [item for item in payload if isinstance(item, dict)]
-
-
-def _refill_queue(*, since_iso: str, force: bool = False) -> int:
-    """Queue bulk global ingest only — each bulk call returns up to 100 messages
-    across many aircraft. Per-plane calls happen only on dossier cache miss."""
-    global _bulk_cursor, _queued_aircraft_keys
-
-    with _queue_lock:
-        if force:
-            _queue.clear()
-            _queued_aircraft_keys = set()
-            _bulk_cursor = {"since_iso": since_iso, "before_id": None, "pages": 0}
-
-        added = 0
-        has_bulk = any(item.get("type") == "bulk" for item in _queue)
-        if not has_bulk:
-            _bulk_cursor["since_iso"] = since_iso
-            _queue.append({"type": "bulk", "since_iso": since_iso, "before_id": None})
-            added += 1
-
-    with _lock:
-        _cache["bulk_pages_this_cycle"] = 0
-        _save_cache()
-
-    return added
-
-
-def _prioritize_aircraft_scan(entry: dict[str, str]) -> bool:
-    """Jump this aircraft to the front of the queue — next API tick (~2s)."""
-    key = _aircraft_queue_key(entry)
-    if key.replace("|", "").strip() == "":
-        return False
-
-    item = {"type": "aircraft", **entry}
-    with _queue_lock:
-        kept: deque[dict[str, Any]] = deque()
-        for queued in _queue:
-            if queued.get("type") == "aircraft" and _aircraft_queue_key(queued) == key:
-                continue
-            kept.append(queued)
-        _queue.clear()
-        _queue.extend(kept)
-        _queued_aircraft_keys.discard(key)
-        _queued_aircraft_keys.add(key)
-        _queue.appendleft(item)
-    return True
-
-
-def _enqueue_bulk_page(*, since_iso: str, before_id: int | None = None) -> None:
-    with _queue_lock:
-        _queue.append({"type": "bulk", "since_iso": since_iso, "before_id": before_id})
-
-
-def _process_aircraft_item(api_key: str, entry: dict[str, str]) -> int:
-    since_iso = _iso(_utc_now() - timedelta(hours=PRIORITY_LOOKBACK_HOURS))
-    params: dict[str, Any] = {
-        "since": since_iso,
-        "limit": 100,
-        "exclude_errors": "1",
-    }
-    if entry.get("icao24"):
-        params["icao"] = entry["icao24"]
-    elif entry.get("registration"):
-        params["text"] = entry["registration"]
-    elif entry.get("callsign"):
-        params["text"] = entry["callsign"]
-    else:
-        return 0
-
-    try:
-        batch = _fetch_messages(api_key=api_key, params=params)
-    except Exception as exc:
-        logger.debug("Airframes aircraft fetch failed for %s: %s", entry, exc)
-        with _lock:
-            _cache["last_error"] = str(exc)[:240]
-            _save_cache()
-        return 0
-
-    return _ingest_messages_batch(batch)
-
-
-def _process_bulk_item(api_key: str, item: dict[str, Any]) -> int:
-    global _bulk_cursor
-
-    params: dict[str, Any] = {
-        "since": item["since_iso"],
-        "limit": 100,
-        "exclude_errors": "1",
-    }
-    before_id = item.get("before_id")
-    if before_id is not None:
-        params["before_id"] = before_id
-
-    try:
-        batch = _fetch_messages(api_key=api_key, params=params)
-    except Exception as exc:
-        logger.debug("Airframes bulk fetch failed: %s", exc)
-        with _lock:
-            _cache["last_error"] = str(exc)[:240]
-            _save_cache()
-        return 0
-
-    ingested = _ingest_messages_batch(batch)
-
-    with _lock:
-        _bulk_cursor["pages"] = int(_bulk_cursor.get("pages", 0)) + 1
-        _cache["pages_fetched"] = int(_cache.get("pages_fetched", 0)) + 1
-        _save_cache()
-
-    if (
-        batch
-        and len(batch) >= 100
-        and _bulk_cursor.get("pages", 0) < MAX_BULK_PAGES_PER_CYCLE
-    ):
-        ids = [int(row["id"]) for row in batch if row.get("id") is not None]
-        if ids:
-            next_before = min(ids)
-            if before_id is None or next_before < before_id:
-                _enqueue_bulk_page(since_iso=item["since_iso"], before_id=next_before)
-
-    return ingested
-
-
-def _process_one_staggered_tick() -> int:
-    """Process exactly one queued Airframes API call. Used by the background worker."""
-    if not api_key_configured():
-        return 0
-
-    api_key = os.environ.get("AIRFRAMES_API_KEY", "").strip()
-    with _queue_lock:
-        if not _queue:
-            return 0
-        item = _queue.popleft()
-
-    if item.get("type") == "aircraft":
-        key = _aircraft_queue_key(item)
-        with _queue_lock:
-            _queued_aircraft_keys.discard(key)
-        ingested = _process_aircraft_item(api_key, item)
-    elif item.get("type") == "bulk":
-        ingested = _process_bulk_item(api_key, item)
-    else:
-        ingested = 0
-
-    with _lock:
-        _cache["ticks_processed"] = int(_cache.get("ticks_processed", 0)) + 1
-        if int(_cache.get("ticks_processed", 0)) % 25 == 0:
-            for store_key in ("by_icao", "by_tail", "by_callsign"):
-                _prune_store(_cache[store_key])
-            _save_cache()
-
-    return ingested
-
-
-def _stagger_worker_loop() -> None:
-    while True:
-        time.sleep(REQUEST_PAUSE_S)
-        try:
-            _process_one_staggered_tick()
-        except Exception as exc:
-            logger.error("Airframes stagger worker tick failed: %s", exc)
-
-
-def _ensure_stagger_worker() -> None:
-    global _worker_started
-    if _worker_started:
-        return
-    with _worker_guard:
-        if _worker_started:
-            return
-        _worker_started = True
-        threading.Thread(
-            target=_stagger_worker_loop,
-            daemon=True,
-            name="airframes-stagger",
-        ).start()
-        logger.info(
-            "Airframes stagger worker started (bulk ingest: 1 call / %.1fs, up to %s msgs/call, refill every %sm)",
-            REQUEST_PAUSE_S,
-            100,
-            SYNC_INTERVAL_MINUTES,
-        )
-
-
-def sync_airframes_messages(*, force: bool = False) -> dict[str, Any]:
-    """Queue staggered Airframes fetches — one API call every REQUEST_PAUSE_S."""
-    if not api_key_configured():
-        return {"ok": False, "skipped": True, "reason": "AIRFRAMES_API_KEY not configured"}
-
-    started = _utc_now()
-    _load_cache_if_cold()
-
-    with _lock:
-        _cache.setdefault("by_callsign", {})
-        last_sync_at = _parse_ts(_cache.get("last_sync_at"))
-        if (
-            not force
-            and last_sync_at is not None
-            and started - last_sync_at < timedelta(minutes=SYNC_INTERVAL_MINUTES - 1)
-        ):
-            return {"ok": True, "skipped": True, "reason": "sync_interval_not_elapsed"}
-
-        if _cache.get("last_success_at"):
-            since_dt = _parse_ts(_cache.get("last_success_at")) or (
-                started - timedelta(minutes=SYNC_INTERVAL_MINUTES)
-            )
-            since_dt -= timedelta(minutes=2)
-        else:
-            since_dt = started - timedelta(hours=PRIORITY_LOOKBACK_HOURS)
-        since_iso = _iso(since_dt)
-        _cache["last_sync_at"] = _iso(started)
-        _cache["last_error"] = None
-        _save_cache()
-
-    queued = _refill_queue(since_iso=since_iso, force=force)
-    _ensure_stagger_worker()
-
-    with _queue_lock:
-        queue_depth = len(_queue)
-
-    logger.info(
-        "Airframes cycle queued: added=%s depth=%s interval=%.1fs",
-        queued,
-        queue_depth,
-        REQUEST_PAUSE_S,
-    )
-    return {
-        "ok": True,
-        "queued": queued,
-        "queue_depth": queue_depth,
-        "request_interval_s": REQUEST_PAUSE_S,
-        "sync_interval_minutes": SYNC_INTERVAL_MINUTES,
-    }
-
-
-def _lookup_from_cache(
-    *,
-    hex_key: str,
-    tail_keys: list[str],
-    callsign_key: str,
-) -> tuple[list[dict[str, Any]], str | None]:
-    _load_cache_if_cold()
-    with _lock:
-        _cache.setdefault("by_callsign", {})
-        merged: dict[int, dict[str, Any]] = {}
-        if hex_key:
-            for message in _cache.get("by_icao", {}).get(hex_key, []):
-                merged[message["id"]] = message
-        for tail_key in tail_keys:
-            for message in _cache.get("by_tail", {}).get(tail_key, []):
-                merged[message["id"]] = message
-        if callsign_key:
-            for message in _cache.get("by_callsign", {}).get(callsign_key, []):
-                merged[message["id"]] = message
-        last_success_at = _cache.get("last_success_at")
-
-    messages = sorted(merged.values(), key=lambda item: item.get("timestamp") or "", reverse=True)
-    return messages[:MESSAGES_PER_AIRCRAFT], last_success_at
-
-
-def get_datalink_status() -> dict[str, Any]:
-    configured = api_key_configured()
-    _load_cache_if_cold()
-    with _queue_lock:
-        queue_depth = len(_queue)
-    with _lock:
-        return {
-            "configured": configured,
-            "sync_interval_minutes": SYNC_INTERVAL_MINUTES,
-            "request_interval_s": REQUEST_PAUSE_S,
-            "last_sync_at": _cache.get("last_sync_at"),
-            "last_success_at": _cache.get("last_success_at"),
-            "last_error": _cache.get("last_error"),
-            "pages_fetched": _cache.get("pages_fetched", 0),
-            "messages_ingested": _cache.get("messages_ingested", 0),
-            "bulk_pages_this_cycle": int(_bulk_cursor.get("pages", 0)),
-            "bulk_pages_per_cycle": MAX_BULK_PAGES_PER_CYCLE,
-            "messages_per_bulk_call": 100,
-            "queue_depth": queue_depth,
-            "ticks_processed": _cache.get("ticks_processed", 0),
-            "icao_keys": len(_cache.get("by_icao", {})),
-            "tail_keys": len(_cache.get("by_tail", {})),
-            "callsign_keys": len(_cache.get("by_callsign", {})),
-        }
-
-
-def lookup_datalink_messages(
-    *,
-    icao24: str = "",
-    registration: str = "",
-    callsign: str = "",
-    allow_live: bool = False,
-) -> dict[str, Any]:
-    configured = bool(os.environ.get("AIRFRAMES_API_KEY", "").strip()) or api_key_configured()
-    if not configured:
-        return {
-            "configured": False,
-            "messages": [],
-            "hint": "Add AIRFRAMES_API_KEY in Settings → API Keys to enable ACARS datalink.",
-        }
-
-    hex_key = _norm_hex(icao24)
-    tail_keys = _tail_lookup_keys(registration)
-    callsign_key = _norm_callsign(callsign)
-
-    messages, last_success_at = _lookup_from_cache(
-        hex_key=hex_key,
-        tail_keys=tail_keys,
-        callsign_key=callsign_key,
-    )
-
-    queued_refresh = False
-    if hex_key or tail_keys or callsign_key:
-        queued_refresh = _prioritize_aircraft_scan(
-            {
-                "icao24": hex_key,
-                "registration": _norm_tail(registration),
-                "callsign": callsign_key,
-            }
-        )
-        if queued_refresh:
-            _ensure_stagger_worker()
-
-    from services.fetchers.acars_summarize import prepare_datalink_display
-
-    display = prepare_datalink_display(messages)
-
-    return {
-        "configured": True,
-        "messages": display["messages"],
-        "hidden_count": display["hidden_count"],
-        "total_count": display["total_count"],
-        "last_success_at": last_success_at,
-        "queued_refresh": queued_refresh,
-        "priority_scan": queued_refresh,
-    }
-
-
-_load_cache_if_cold()

backend/services/fetchers/meshtastic_map.py

@@ -12,11 +12,9 @@ Polling interval deliberately kept low (4h) to be respectful to the service.
 
 import json
 import logging
-import threading
 import time
 from datetime import datetime, timezone, timedelta
 from pathlib import Path
-from typing import Any
 
 import requests
 
@@ -32,6 +30,9 @@ _MAX_AGE_HOURS = 24  # discard nodes not seen within this window
 # one-person hobby service, so we prefer stale data over hammering it.
 _CACHE_TRUST_HOURS = 20
 
+# Track when we last fetched so the frontend can show staleness
+_last_fetch_ts: float = 0.0
+
 
 def _parse_node(node: dict) -> dict | None:
     """Convert an API node into a slim signal-like dict."""
@@ -131,43 +132,7 @@ def _save_cache(nodes: list[dict], fetch_ts: float):
         logger.warning(f"Failed to save meshtastic cache: {e}")
 
 
-# Track when we last fetched so the frontend can show staleness
-_last_fetch_ts: float = 0.0
-_scan_lock = threading.Lock()
-_scan_in_progress = False
-
-
-def get_meshtastic_map_status() -> dict[str, Any]:
-    from services.fetchers._store import get_latest_data_subset_refs
-
-    snap = get_latest_data_subset_refs("meshtastic_map_nodes", "meshtastic_map_fetched_at")
-    nodes = snap.get("meshtastic_map_nodes") or []
-    fetched_at = snap.get("meshtastic_map_fetched_at")
-    return {
-        "node_count": len(nodes) if isinstance(nodes, list) else 0,
-        "fetched_at": fetched_at,
-        "scan_in_progress": _scan_in_progress,
-    }
-
-
-def start_meshtastic_planet_scan() -> dict[str, Any]:
-    if not _scan_lock.acquire(blocking=False):
-        return {"ok": False, "status": "scan already in progress"}
-
-    def _run() -> None:
-        global _scan_in_progress
-        try:
-            _scan_in_progress = True
-            fetch_meshtastic_nodes(force=True)
-        finally:
-            _scan_in_progress = False
-            _scan_lock.release()
-
-    threading.Thread(target=_run, daemon=True, name="meshtastic-planet-scan").start()
-    return {"ok": True, "status": "scanning"}
-
-
-def fetch_meshtastic_nodes(*, force: bool = False):
+def fetch_meshtastic_nodes():
     """Fetch global Meshtastic node positions from Liam Cottle's map API.
 
     Stores processed nodes in latest_data["meshtastic_map_nodes"].
@@ -175,40 +140,39 @@ def fetch_meshtastic_nodes(*, force: bool = False):
     """
     from services.fetchers._store import is_any_active
 
-    if not force and not is_any_active("sigint_meshtastic"):
+    if not is_any_active("sigint_meshtastic"):
         return
     global _last_fetch_ts
 
     # Trust a recent cache on disk — avoids hammering the upstream HTTP API
     # when every install polls on roughly the same cadence.
-    if not force:
-        try:
-            if _CACHE_FILE.exists():
-                mtime = _CACHE_FILE.stat().st_mtime
-                if time.time() - mtime < _CACHE_TRUST_HOURS * 3600:
-                    # If memory is empty (cold start), hydrate from cache and skip fetch.
-                    with _data_lock:
-                        has_memory = bool(latest_data.get("meshtastic_map_nodes"))
-                    if not has_memory:
-                        cached = _load_cache()
-                        if cached:
-                            with _data_lock:
-                                latest_data["meshtastic_map_nodes"] = cached
-                                latest_data["meshtastic_map_fetched_at"] = mtime
-                            _mark_fresh("meshtastic_map")
-                            logger.info(
-                                "Meshtastic map: cache fresh (<%.0fh), skipping network fetch",
-                                _CACHE_TRUST_HOURS,
-                            )
-                            return
-                    else:
+    try:
+        if _CACHE_FILE.exists():
+            mtime = _CACHE_FILE.stat().st_mtime
+            if time.time() - mtime < _CACHE_TRUST_HOURS * 3600:
+                # If memory is empty (cold start), hydrate from cache and skip fetch.
+                with _data_lock:
+                    has_memory = bool(latest_data.get("meshtastic_map_nodes"))
+                if not has_memory:
+                    cached = _load_cache()
+                    if cached:
+                        with _data_lock:
+                            latest_data["meshtastic_map_nodes"] = cached
+                            latest_data["meshtastic_map_fetched_at"] = mtime
+                        _mark_fresh("meshtastic_map")
                         logger.info(
                             "Meshtastic map: cache fresh (<%.0fh), skipping network fetch",
                             _CACHE_TRUST_HOURS,
                         )
                         return
-        except Exception as e:
-            logger.debug(f"Meshtastic cache freshness check failed: {e}")
+                else:
+                    logger.info(
+                        "Meshtastic map: cache fresh (<%.0fh), skipping network fetch",
+                        _CACHE_TRUST_HOURS,
+                    )
+                    return
+    except Exception as e:
+        logger.debug(f"Meshtastic cache freshness check failed: {e}")
 
     # Build a polite User-Agent. Historically this included the operator
     # callsign so meshtastic.org could rate-limit per-install; that's still

backend/services/fetchers/plane_alert.py

@@ -49,11 +49,11 @@ _CATEGORY_COLOR: dict[str, str] = {
     "Head of State": "#ff1493",
     "Royal Aircraft": "#ff1493",
     "Don't you know who I am?": "#ff1493",
+    "As Seen on TV": "#ff1493",
     "Bizjets": "#ff1493",
     "Vanity Plate": "#ff1493",
     "Football": "#ff1493",
-    # ORANGE — corporate / novelty / Joe Cool / As Seen on TV
-    "As Seen on TV": "orange",
+    # ORANGE — Joe Cool
     "Joe Cool": "orange",
     # WHITE — Climate Crisis
     "Climate Crisis": "white",
@@ -338,10 +338,6 @@ def enrich_with_tracked_names(flight: dict) -> dict:
             flight["alert_color"] = "blue"
         elif is_med:
             flight["alert_color"] = "#32cd32"
-        elif match.get("category") == "Oligarch":
-            flight["alert_color"] = "red"
-        elif match.get("category") in {"Royal", "Celebrity", "People"}:
-            flight["alert_color"] = "#ff1493"
         elif "alert_color" not in flight:
             flight["alert_color"] = "pink"
 

backend/services/layer_enable_refresh.py

@@ -1,123 +0,0 @@
-"""Immediate data refresh when the operator enables a map layer.
-
-Disk/local fetches run inline (milliseconds). Network-heavy fetches run on the
-slow executor so POST /api/layers never blocks the single uvicorn worker for
-tens of seconds (which freezes bootstrap + live-data and makes the map go black).
-"""
-from __future__ import annotations
-
-import logging
-
-logger = logging.getLogger(__name__)
-
-# Inline — local DB / static files only.
-_INSTANT_LAYER_KEYS: frozenset[str] = frozenset(
-    {"cctv", "power_plants", "datacenters"}
-)
-# Background — network-bound; may take seconds.
-_SLOW_LAYER_KEYS: frozenset[str] = frozenset(
-    {"firms", "psk_reporter", "fishing_activity"}
-)
-
-
-def snapshot_active_layers() -> dict[str, bool]:
-    from services.fetchers._store import active_layers
-
-    return dict(active_layers)
-
-
-def _was_off_now_on(before: dict[str, bool], key: str) -> bool:
-    from services.fetchers._store import active_layers
-
-    return not bool(before.get(key, False)) and bool(active_layers.get(key, False))
-
-
-def _instant_fetch(key: str) -> None:
-    if key == "cctv":
-        from services.fetchers.infrastructure import fetch_cctv
-
-        fetch_cctv()
-        logger.info("CCTV loaded (layer enabled)")
-        return
-    if key == "power_plants":
-        from services.fetchers.infrastructure import fetch_power_plants
-
-        fetch_power_plants()
-        logger.info("Power plants loaded (layer enabled)")
-        return
-    if key == "datacenters":
-        from services.fetchers.infrastructure import fetch_datacenters
-
-        fetch_datacenters()
-        logger.info("Datacenters loaded (layer enabled)")
-        return
-    raise KeyError(key)
-
-
-def _slow_fetch(key: str) -> None:
-    if key == "firms":
-        from services.fetchers.earth_observation import (
-            fetch_firms_country_fires,
-            fetch_firms_fires,
-        )
-
-        fetch_firms_fires()
-        fetch_firms_country_fires()
-        logger.info("FIRMS fires loaded (layer enabled)")
-        return
-    if key == "psk_reporter":
-        from services.fetchers.infrastructure import fetch_psk_reporter
-
-        fetch_psk_reporter()
-        logger.info("PSK Reporter loaded (layer enabled)")
-        return
-    if key == "fishing_activity":
-        from services.fetchers.geo import fetch_fishing_activity
-
-        fetch_fishing_activity()
-        logger.info("Fishing activity loaded (layer enabled)")
-        return
-    raise KeyError(key)
-
-
-def _run_slow_enable_fetches(keys: tuple[str, ...]) -> None:
-    from services.fetchers._store import bump_data_version
-
-    for key in keys:
-        try:
-            _slow_fetch(key)
-        except Exception:
-            logger.exception("Layer enable fetch failed for %s", key)
-    bump_data_version()
-
-
-def refresh_newly_enabled_layers(before: dict[str, bool]) -> None:
-    """Fetch any layers that transitioned off → on."""
-    from services.fetchers._store import bump_data_version
-
-    instant_keys: list[str] = []
-    slow_keys: list[str] = []
-
-    for key in _INSTANT_LAYER_KEYS | _SLOW_LAYER_KEYS:
-        if _was_off_now_on(before, key):
-            if key in _INSTANT_LAYER_KEYS:
-                instant_keys.append(key)
-            else:
-                slow_keys.append(key)
-
-    if not instant_keys and not slow_keys:
-        return
-
-    for key in instant_keys:
-        try:
-            _instant_fetch(key)
-        except Exception:
-            logger.exception("Layer enable fetch failed for %s", key)
-
-    if instant_keys:
-        bump_data_version()
-
-    if slow_keys:
-        from services.data_fetcher import _SLOW_EXECUTOR
-
-        _SLOW_EXECUTOR.submit(_run_slow_enable_fetches, tuple(slow_keys))

backend/services/sar/sar_config.py

@@ -167,31 +167,18 @@ def products_fetch_enabled() -> bool:
     return _flag("MESH_SAR_PRODUCTS_FETCH_ACKNOWLEDGE", default=False)
 
 
-def runtime_store_exists() -> bool:
-    """True when ``data/sar_runtime.json`` exists on disk."""
-    return _RUNTIME_FILE.is_file()
-
-
 def products_fetch_status() -> dict[str, Any]:
     """Structured status used by the router for the 'how to enable' UX."""
     raw = _str("MESH_SAR_PRODUCTS_FETCH", default="block").strip().lower()
     fetch_set = raw in {"allow", "enable", "enabled", "true", "on", "1"}
     ack_set = _flag("MESH_SAR_PRODUCTS_FETCH_ACKNOWLEDGE", default=False)
-    token_set = bool(earthdata_token())
-    user_set = bool(earthdata_user())
-    opt_in = fetch_set and ack_set
-    # ``enabled`` historically meant opt-in flags only; ``fully_configured``
-    # is what the fetcher actually needs (flags + Earthdata token).
-    fully_configured = opt_in and token_set
+    enabled = fetch_set and ack_set
     return {
-        "enabled": opt_in,
-        "fully_configured": fully_configured,
+        "enabled": enabled,
         "fetch_flag_set": fetch_set,
         "acknowledge_flag_set": ack_set,
-        "earthdata_token_set": token_set,
-        "earthdata_user_set": user_set,
-        "runtime_store_exists": runtime_store_exists(),
-        "runtime_store_path": str(_RUNTIME_FILE),
+        "earthdata_token_set": bool(earthdata_token()),
+        "earthdata_user_set": bool(earthdata_user()),
         "missing": _missing_for_products(fetch_set, ack_set),
         "help": {
             "summary": (

backend/tests/test_acars_summarize.py

@@ -1,182 +0,0 @@
-from services.fetchers.acars_summarize import prepare_datalink_display, summarize_datalink_message
-
-# --- Southwest (existing) ---
-
-
-def test_summarize_track_report():
-    text = """++86501,N8997Q,B7378MAX,260620,WN3743,KMSP,KMDW,0496,SMX34-2502-F320
-6
-N4432.0,W09305.6,201041,15193,-08.3,310,044,CL,00000,0,"""
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "track"
-    assert "WN3743" in meta["summary"]
-    assert "KMSP→KMDW" in meta["summary"]
-
-
-def test_summarize_sw_performance_cruise():
-    text = """72740,7852,B737-700,260624,WN0120,KABQ,KDEN,1986,SW2501
-18.45.14,CR,1575,28981,280.0,.729,-32.3,-06.5,N3601.3,W10655.7,131240
-   0.48,FHP,AIR
-SIN,-1.42  0.30  0.29"""
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "engine_health"
-    assert "WN0120" in meta["summary"]
-    assert "cruise" in meta["summary"]
-
-
-def test_summarize_sw_climb_performance():
-    text = """05201,7852,B737-700,260624,WN0120,KABQ,KDEN,1986,SW2501
-18.38.08,CL,1149,15631,257.0,.520,000.0,014.5,N3515.4,W10649.4,132800
-001.40,001,4100,FLAPS-UP"""
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "climb_perf"
-    assert "climb" in meta["summary"]
-
-
-def test_summarize_trajectory():
-    text = """76401
-02E24KABQKDEN
-N35112W10679318361096P014343008G000022::I0:9W
-N35195W10681118371370P006337009G000022::Q0OXW"""
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "trajectory"
-    assert "KABQ→KDEN" in meta["summary"]
-
-
-def test_fragment_hidden():
-    text = "0000000,00000000,00000000\n18.38.23,16395,250.2,.510,01.07,01.04,00,00000000"
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "fragment"
-    assert meta["hidden"] is True
-
-
-def test_vdl_binary_hidden():
-    text = "014F63N\n)AJQZ)LC0Z0IP-M7O,ZHN3-M,73ZO,UU-ZOS1Z7PPZMSN1ZN"
-    meta = summarize_datalink_message(label="37", text=text, source_type="vdl")
-    assert meta["hidden"] is True
-
-
-# --- United / Delta / American ---
-
-
-def test_united_free_text_position():
-    text = "POS N40.123 W074.456 FL350 GS450 1425Z"
-    meta = summarize_datalink_message(label="Q0", text=text, source_type="vdl")
-    assert meta["kind"] == "position"
-    assert "FL350" in meta["summary"]
-
-
-def test_delta_oooi_out():
-    text = "OUT 1425 12JAN KATL"
-    meta = summarize_datalink_message(label="00", text=text, source_type="acars")
-    assert meta["kind"] == "oooi"
-    assert "OUT 1425" in meta["summary"]
-
-
-def test_american_fi_block():
-    text = "FI AA100/AN N100AA/DA KDFW/AA KLAX OUT 1832 OFF 1845"
-    meta = summarize_datalink_message(label="44", text=text, source_type="acars")
-    assert "AA100" in meta["summary"]
-    assert "KDFW→KLAX" in meta["summary"]
-
-
-def test_united_performance_a320():
-    text = """88401,4521,A320-200,260624,UA1234,KORD,KDEN,1200,UA2501
-19.10.22,CR,2200,35000,450.0,.820,-45.0,-02.0,N3950.1,W10440.2,125000"""
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "performance"
-    assert "UA1234" in meta["summary"]
-    assert "KORD→KDEN" in meta["summary"]
-
-
-# --- International ---
-
-
-def test_british_airways_engine():
-    text = "ENG1 N1 92.5 N2 95.1 EGT 512 FF 2850"
-    meta = summarize_datalink_message(label="B1", text=text, source_type="satcom")
-    assert meta["kind"] == "engine"
-
-
-def test_qantas_fi_position():
-    text = "FI QF9/AN VH-OQA/DA YSSY/AD EGLL POSN32249E045047,,082806,380,DEBNI"
-    meta = summarize_datalink_message(label="H1", text=text, source_type="acars")
-    assert meta["kind"] == "position"
-    assert "QF9" in meta["summary"]
-    assert "YSSY→EGLL" in meta["summary"]
-
-
-def test_lufthansa_weather():
-    text = "WX 250/045 SAT -42 TB MOD EDDF"
-    meta = summarize_datalink_message(label="80", text=text, source_type="acars")
-    assert meta["kind"] == "weather"
-    assert "EDDF" in meta["summary"]
-
-
-def test_air_france_request():
-    text = "REQUEST FL370 DUE TURB"
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "request"
-    assert "FL370" in meta["summary"]
-
-
-# --- Cargo / military ---
-
-
-def test_fedex_flight():
-    text = "++86501,N123FE,B763,260624,FDX1544,KMEM,KORD,0498,SMX34"
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "track"
-    assert "FDX1544" in meta["summary"]
-
-
-def test_military_rch():
-    text = "POSN3840.5 W07720.1 FL280 RCH123 KADW KDMA"
-    meta = summarize_datalink_message(label="Q0", text=text, source_type="acars")
-    assert "RCH123" in meta["summary"]
-
-
-# --- Ops / misc ---
-
-
-def test_flight_plan():
-    text = "FPN/RI:DA:KJFK:AA:EGLL..MERIT:D:MERIT"
-    meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
-    assert meta["kind"] == "flight_plan"
-    assert "KJFK→EGLL" in meta["summary"]
-
-
-def test_departure_report():
-    text = "DEP FI DL456/DA KATL/AD KLAX OUT 1205"
-    meta = summarize_datalink_message(label="40", text=text, source_type="acars")
-    assert meta["kind"] == "dep"
-    assert "DL456" in meta["summary"]
-
-
-def test_pirep():
-    text = "#CFB/PIREP MOD TURB FL280 N3845 W09030"
-    meta = summarize_datalink_message(label="H1", text=text, source_type="acars")
-    assert meta["kind"] == "pirep"
-
-
-def test_prepare_filters_hidden_and_dedupes():
-    messages = [
-        {"id": 1, "label": "H1", "text": "POSN35259W106517,KABQ,KDEN", "source_type": "vdl"},
-        {"id": 2, "label": "H1", "text": "0000000,00000000,00000000", "source_type": "vdl"},
-        {"id": 3, "label": "37", "text": "014F63N\n)AJQZ)LC0Z", "source_type": "vdl"},
-        {
-            "id": 4,
-            "label": "H1",
-            "text": "72740,7852,B737-700,260624,WN0120,KABQ,KDEN\n18.45.14,CR,1575,28981",
-            "source_type": "vdl",
-        },
-        {
-            "id": 5,
-            "label": "H1",
-            "text": "72740,7852,B737-700,260624,WN0120,KABQ,KDEN\n18.45.14,CR,1575,28981",
-            "source_type": "vdl",
-        },
-    ]
-    display = prepare_datalink_display(messages)
-    assert display["hidden_count"] == 3
-    assert len(display["messages"]) == 2

backend/tests/test_agent_shell_ws_auth.py

@@ -1,129 +0,0 @@
-"""Agent shell WebSocket auth regression tests (issue #407)."""
-
-from __future__ import annotations
-
-import sys
-import types
-from unittest.mock import AsyncMock, MagicMock, patch
-
-import pytest
-from fastapi import FastAPI
-from fastapi.testclient import TestClient
-from starlette.websockets import WebSocketDisconnect
-
-if sys.platform == "win32":
-    fcntl_stub = types.ModuleType("fcntl")
-    fcntl_stub.ioctl = lambda *args, **kwargs: None
-    sys.modules.setdefault("fcntl", fcntl_stub)
-    termios_stub = types.ModuleType("termios")
-    termios_stub.TIOCSWINSZ = 0
-    termios_stub.TCSAFLUSH = 0
-    sys.modules.setdefault("termios", termios_stub)
-    pty_stub = types.ModuleType("pty")
-    pty_stub.openpty = lambda: (0, 0)
-    sys.modules["pty"] = pty_stub
-
-from routers import agent_shell  # noqa: E402
-from services.agent_shell_ws_token import (  # noqa: E402
-    consume_agent_shell_ws_token,
-    mint_agent_shell_ws_token,
-    reset_agent_shell_ws_tokens_for_tests,
-)
-
-
-@pytest.fixture()
-def shell_client():
-    app = FastAPI()
-    app.include_router(agent_shell.router)
-    with TestClient(app) as client:
-        yield client
-
-
-@pytest.fixture(autouse=True)
-def _reset_ws_tokens():
-    reset_agent_shell_ws_tokens_for_tests()
-    yield
-    reset_agent_shell_ws_tokens_for_tests()
-
-
-class TestAgentShellWsTokenStore:
-    def test_mint_and_consume_once(self):
-        token, expires_in = mint_agent_shell_ws_token()
-        assert expires_in > 0
-        assert consume_agent_shell_ws_token(token) is True
-        assert consume_agent_shell_ws_token(token) is False
-
-
-class TestAgentShellWsTokenRoute:
-    def test_loopback_can_mint_token(self, shell_client):
-        transport = shell_client._transport
-        transport.client = ("127.0.0.1", 12345)
-        response = shell_client.post("/api/agent-shell/ws-token")
-        assert response.status_code == 200
-        body = response.json()
-        assert body["token"]
-        assert body["expires_in"] > 0
-
-    def test_remote_caller_cannot_mint_token(self, shell_client):
-        shell_client._transport.client = ("1.2.3.4", 12345)
-        with patch("auth._current_admin_key", return_value="test-admin-key-32chars-xxxxxxxxxx"):
-            response = shell_client.post("/api/agent-shell/ws-token")
-        assert response.status_code == 403
-
-
-class TestAgentShellWsAuthorization:
-    def test_remote_peer_with_spoofed_host_is_denied(self, shell_client):
-        shell_client._transport.client = ("1.2.3.4", 12345)
-        with pytest.raises((WebSocketDisconnect, Exception)):
-            with shell_client.websocket_connect(
-                "/api/agent-shell/ws",
-                headers={"host": "localhost:8000"},
-            ) as ws:
-                ws.receive_text()
-
-    def test_remote_peer_with_spoofed_origin_is_denied(self, shell_client):
-        shell_client._transport.client = ("1.2.3.4", 12345)
-        with pytest.raises((WebSocketDisconnect, Exception)):
-            with shell_client.websocket_connect(
-                "/api/agent-shell/ws",
-                headers={"origin": "http://localhost:3000"},
-            ) as ws:
-                ws.receive_text()
-
-    def test_remote_peer_with_valid_ws_token_is_accepted(self, shell_client):
-        shell_client._transport.client = ("127.0.0.1", 12345)
-        token = shell_client.post("/api/agent-shell/ws-token").json()["token"]
-        shell_client._transport.client = ("1.2.3.4", 12345)
-        with patch("sys.platform", "win32"):
-            with shell_client.websocket_connect(f"/api/agent-shell/ws?ws_token={token}") as ws:
-                payload = ws.receive_json()
-        assert payload["type"] == "error"
-        assert "Windows" in payload["message"]
-
-    def test_ws_token_is_single_use(self, shell_client):
-        shell_client._transport.client = ("127.0.0.1", 12345)
-        token = shell_client.post("/api/agent-shell/ws-token").json()["token"]
-        shell_client._transport.client = ("1.2.3.4", 12345)
-        with patch("sys.platform", "win32"):
-            with shell_client.websocket_connect(f"/api/agent-shell/ws?ws_token={token}") as ws:
-                ws.receive_json()
-        with pytest.raises((WebSocketDisconnect, Exception)):
-            with shell_client.websocket_connect(f"/api/agent-shell/ws?ws_token={token}") as ws:
-                ws.receive_text()
-
-    def test_loopback_peer_does_not_need_ws_token(self, shell_client):
-        shell_client._transport.client = ("127.0.0.1", 12345)
-        with patch("sys.platform", "win32"):
-            with shell_client.websocket_connect("/api/agent-shell/ws") as ws:
-                payload = ws.receive_json()
-        assert payload["type"] == "error"
-        assert "Windows" in payload["message"]
-
-    @pytest.mark.asyncio
-    async def test_authorize_rejects_spoofed_headers_without_token(self):
-        ws = MagicMock()
-        ws.client = MagicMock(host="1.2.3.4")
-        ws.headers = {"host": "localhost:8000", "origin": "http://localhost:3000"}
-        ws.close = AsyncMock()
-        with pytest.raises(WebSocketDisconnect):
-            await agent_shell._authorize_agent_shell_ws(ws)

backend/tests/test_airframes_datalink.py

@@ -1,177 +0,0 @@
-import json
-from unittest.mock import patch
-
-import pytest
-
-
-@pytest.fixture
-def airframes_env(tmp_path, monkeypatch):
-    from services.fetchers import airframes
-
-    cache_path = tmp_path / "airframes_datalink_cache.json"
-    monkeypatch.setattr(airframes, "_CACHE_PATH", cache_path)
-    monkeypatch.setattr(airframes, "_DATA_DIR", tmp_path)
-    airframes._cache = {
-        "last_sync_at": None,
-        "last_success_at": None,
-        "last_error": None,
-        "pages_fetched": 0,
-        "messages_ingested": 0,
-        "priority_aircraft_synced": 0,
-        "bulk_pages_this_cycle": 0,
-        "ticks_processed": 0,
-        "by_icao": {},
-        "by_tail": {},
-        "by_callsign": {},
-    }
-    airframes._queue.clear()
-    airframes._queued_aircraft_keys.clear()
-    airframes._bulk_cursor = {"since_iso": "", "before_id": None, "pages": 0}
-    airframes._cache_loaded = True
-    airframes._api_key_known_configured = True
-    monkeypatch.setenv("AIRFRAMES_API_KEY", "test-key")
-    return airframes
-
-
-def test_sync_skips_without_api_key(airframes_env, monkeypatch):
-    monkeypatch.delenv("AIRFRAMES_API_KEY", raising=False)
-    airframes_env._api_key_known_configured = None
-    result = airframes_env.sync_airframes_messages(force=True)
-    assert result["ok"] is False
-    assert result["skipped"] is True
-
-
-@patch("services.fetchers.airframes.requests.get")
-def test_sync_ingests_messages(mock_get, airframes_env):
-    from datetime import datetime, timezone
-
-    recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
-    mock_get.return_value.status_code = 200
-    mock_get.return_value.headers = {}
-    mock_get.return_value.json.return_value = [
-        {
-            "id": 101,
-            "timestamp": recent,
-            "label": "H1",
-            "text": "ETA 1432",
-            "sourceType": "acars",
-            "fromHex": "A022B9",
-            "tail": "9H-TJZ",
-            "flightNumber": "CXI3SY",
-        }
-    ]
-
-    result = airframes_env.sync_airframes_messages(force=True)
-    assert result["ok"] is True
-    assert result["queued"] >= 1
-
-    ingested = airframes_env._process_one_staggered_tick()
-    assert ingested == 1
-
-    lookup = airframes_env.lookup_datalink_messages(
-        icao24="a022b9",
-        registration="9H-TJZ",
-        callsign="CXI3SY",
-        allow_live=False,
-    )
-    assert lookup["configured"] is True
-    assert len(lookup["messages"]) == 1
-    assert lookup["messages"][0]["text"] == "ETA 1432"
-
-
-def test_lookup_queues_priority_scan_on_every_open(airframes_env):
-    lookup = airframes_env.lookup_datalink_messages(icao24="abc123", allow_live=False)
-    assert lookup["configured"] is True
-    assert lookup["messages"] == []
-    assert lookup["queued_refresh"] is True
-    assert lookup["priority_scan"] is True
-    with airframes_env._queue_lock:
-        assert airframes_env._queue[0]["type"] == "aircraft"
-        assert airframes_env._queue[0]["icao24"] == "abc123"
-
-
-def test_priority_scan_jumps_ahead_of_bulk(airframes_env):
-    airframes_env._refill_queue(since_iso="2026-01-01T00:00:00Z", force=True)
-    with airframes_env._queue_lock:
-        assert airframes_env._queue[0]["type"] == "bulk"
-
-    airframes_env.lookup_datalink_messages(icao24="deadbeef", allow_live=False)
-
-    with airframes_env._queue_lock:
-        assert airframes_env._queue[0]["type"] == "aircraft"
-        assert airframes_env._queue[0]["icao24"] == "deadbeef"
-
-
-def test_lookup_still_queues_when_cache_hit(airframes_env):
-    from datetime import datetime, timezone
-
-    recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
-    airframes_env._ingest_message(
-        {
-            "id": 404,
-            "timestamp": recent,
-            "text": "CACHED MSG",
-            "fromHex": "a022b9",
-        }
-    )
-    lookup = airframes_env.lookup_datalink_messages(icao24="a022b9", allow_live=False)
-    assert len(lookup["messages"]) == 1
-    assert lookup["priority_scan"] is True
-    with airframes_env._queue_lock:
-        assert airframes_env._queue[0]["icao24"] == "a022b9"
-
-
-def test_lookup_unconfigured_shows_hint(airframes_env, monkeypatch):
-    monkeypatch.delenv("AIRFRAMES_API_KEY", raising=False)
-    airframes_env._api_key_known_configured = None
-    lookup = airframes_env.lookup_datalink_messages(icao24="abc123")
-    assert lookup["configured"] is False
-    assert lookup["messages"] == []
-    assert "AIRFRAMES_API_KEY" in lookup["hint"]
-
-
-def test_lookup_indexes_to_hex_and_callsign(airframes_env):
-    from datetime import datetime, timezone
-
-    recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
-    airframes_env._ingest_message(
-        {
-            "id": 202,
-            "timestamp": recent,
-            "text": "DESCENT TO FL100",
-            "fromHex": "ABCDEF",
-            "toHex": "a022b9",
-            "flightNumber": "RCH123",
-        }
-    )
-
-    by_icao = airframes_env.lookup_datalink_messages(icao24="a022b9", allow_live=False)
-    assert len(by_icao["messages"]) == 1
-
-    by_callsign = airframes_env.lookup_datalink_messages(callsign="RCH123", allow_live=False)
-    assert len(by_callsign["messages"]) == 1
-
-
-def test_tail_lookup_normalizes_dashes(airframes_env):
-    from datetime import datetime, timezone
-
-    recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
-    airframes_env._ingest_message(
-        {
-            "id": 303,
-            "timestamp": recent,
-            "text": "ON GROUND",
-            "tail": "9H-TJZ",
-        }
-    )
-
-    lookup = airframes_env.lookup_datalink_messages(registration="9HTJZ", allow_live=False)
-    assert len(lookup["messages"]) == 1
-
-
-def test_api_registry_includes_airframes_key():
-    from services.api_settings import API_REGISTRY, ALLOWED_ENV_KEYS
-
-    entry = next(item for item in API_REGISTRY if item["id"] == "airframes_api_key")
-    assert entry["env_key"] == "AIRFRAMES_API_KEY"
-    assert "AIRFRAMES_API_KEY" in ALLOWED_ENV_KEYS

backend/tests/test_layer_enable_integration.py

@@ -1,26 +0,0 @@
-"""Integration: layer enable triggers immediate data availability."""
-from __future__ import annotations
-
-import time
-
-from services.fetchers._store import active_layers, latest_data, _data_lock
-
-
-def test_firms_enable_populates_slow_payload(client):
-    with _data_lock:
-        active_layers["firms"] = False
-        latest_data["firms_fires"] = []
-
-    r = client.post("/api/layers", json={"layers": {"firms": True}})
-    assert r.status_code == 200
-
-    fires: list = []
-    for _ in range(45):
-        slow = client.get("/api/live-data/slow")
-        assert slow.status_code == 200
-        fires = slow.json().get("firms_fires") or []
-        if fires:
-            break
-        time.sleep(2)
-
-    assert len(fires) > 0, "firms layer should populate after async on-enable fetch"

backend/tests/test_layer_enable_refresh.py

@@ -1,56 +0,0 @@
-"""Tests for on-enable layer refresh (Phase 2 UX guardrail)."""
-from __future__ import annotations
-
-from unittest.mock import patch
-
-from services.fetchers._store import active_layers, bump_active_layers_version
-from services.layer_enable_refresh import refresh_newly_enabled_layers, snapshot_active_layers
-
-
-def test_refresh_firms_on_enable_only():
-    before = snapshot_active_layers()
-    active_layers["firms"] = True
-    bump_active_layers_version()
-
-    with (
-        patch("services.fetchers.earth_observation.fetch_firms_fires") as firms,
-        patch("services.fetchers.earth_observation.fetch_firms_country_fires") as country,
-        patch("services.layer_enable_refresh._run_slow_enable_fetches") as run_slow,
-        patch("services.fetchers._store.bump_data_version") as bump,
-    ):
-        refresh_newly_enabled_layers({**before, "firms": False})
-        firms.assert_not_called()
-        country.assert_not_called()
-        run_slow.assert_called_once()
-        assert run_slow.call_args[0][0] == ("firms",)
-        bump.assert_not_called()
-
-    active_layers["firms"] = before.get("firms", False)
-
-
-def test_refresh_skips_when_layer_stays_off():
-    before = {**snapshot_active_layers(), "cctv": False}
-    active_layers["cctv"] = False
-
-    with patch("services.fetchers.infrastructure.fetch_cctv") as fetch_cctv:
-        refresh_newly_enabled_layers(before)
-
-    fetch_cctv.assert_not_called()
-
-
-def test_refresh_cctv_runs_inline():
-    before = {**snapshot_active_layers(), "cctv": False}
-    active_layers["cctv"] = True
-
-    with (
-        patch("services.fetchers.infrastructure.fetch_cctv") as fetch_cctv,
-        patch("services.fetchers._store.bump_data_version") as bump,
-        patch("services.data_fetcher._SLOW_EXECUTOR") as slow_exec,
-    ):
-        refresh_newly_enabled_layers(before)
-
-    fetch_cctv.assert_called_once()
-    bump.assert_called_once()
-    slow_exec.submit.assert_not_called()
-
-    active_layers["cctv"] = before.get("cctv", False)

backend/tests/test_perf_safe_optimizations.py

@@ -1,59 +0,0 @@
-"""Regression tests for UX-safe performance optimizations."""
-from __future__ import annotations
-
-import inspect
-
-
-def test_slow_tier_skips_duplicate_time_critical_fetchers():
-    """Weather + Ukraine alerts have dedicated scheduler jobs — not slow tier."""
-    from services import data_fetcher
-
-    source = inspect.getsource(data_fetcher.update_slow_data)
-    slow_block = source.split("_run_tasks(\"slow-tier\"", 1)[0]
-    assert "fetch_weather_alerts" not in slow_block
-    assert "fetch_ukraine_air_raid_alerts" not in slow_block
-
-
-def test_slow_tier_gates_correlation_engine_on_active_layer():
-    from services import data_fetcher
-
-    source = inspect.getsource(data_fetcher.update_slow_data)
-    assert 'is_any_active("correlations")' in source
-
-
-def test_health_uses_subset_refs_not_full_deepcopy():
-    from routers import health as health_router
-
-    source = inspect.getsource(health_router.health_check)
-    assert "_health_data_snapshot()" in source
-    assert "get_latest_data()" not in source
-
-    snap_source = inspect.getsource(health_router._health_data_snapshot)
-    assert "get_latest_data_subset_refs" in snap_source
-    assert "deepcopy" not in snap_source
-
-
-def test_active_layers_defaults_match_dashboard_first_paint():
-    """Backend must not prefetch layers the dashboard starts with disabled."""
-    from services.fetchers import _store
-
-    off_by_default = {
-        "cctv": False,
-        "firms": False,
-        "datacenters": False,
-        "power_plants": False,
-        "psk_reporter": False,
-        "viirs_nightlights": False,
-        "crowdthreat": False,
-        "gt_risk": False,
-    }
-    for key, expected in off_by_default.items():
-        assert _store.active_layers.get(key) is expected, key
-
-
-def test_layer_enable_refresh_covers_cold_toggle_layers():
-    from services import layer_enable_refresh
-
-    source = inspect.getsource(layer_enable_refresh.refresh_newly_enabled_layers)
-    for key in ("cctv", "firms", "power_plants", "psk_reporter", "datacenters"):
-        assert key in layer_enable_refresh._INSTANT_LAYER_KEYS | layer_enable_refresh._SLOW_LAYER_KEYS

backend/tests/test_sigint_cctv_accuracy.py

@@ -314,7 +314,7 @@ def test_cctv_proxy_profiles_are_source_specific():
     tfl = main._cctv_proxy_profile_for_url("https://s3-eu-west-1.amazonaws.com/jamcams.tfl.gov.uk/00001.mp4")
     austin = main._cctv_proxy_profile_for_url("https://cctv.austinmobility.io/image/316.jpg")
     georgia = main._cctv_proxy_profile_for_url("https://511ga.org/map/Cctv/22378")
-    spain = main._cctv_proxy_profile_for_url("https://etraffic.dgt.es/camarasEtraffic/1050.jpg")
+    spain = main._cctv_proxy_profile_for_url("https://infocar.dgt.es/etraffic/data/camaras/1050.jpg")
 
     assert tfl.name == "tfl-jamcam"
     assert tfl.headers["Accept"].startswith("video/mp4")
@@ -324,7 +324,7 @@ def test_cctv_proxy_profiles_are_source_specific():
     assert georgia.timeout == (5.0, 12.0)
     assert georgia.headers["Referer"] == "https://511ga.org/cctv"
     assert spain.name == "dgt-spain"
-    assert spain.headers["Referer"] == "https://etraffic.dgt.es/"
+    assert spain.headers["Referer"] == "https://infocar.dgt.es/"
 
 
 def test_cctv_proxy_preserves_upstream_http_status(monkeypatch):

docker-compose.participant.yml

@@ -17,9 +17,6 @@ services:
       WORMHOLE_STARTUP_DEADLINE_S: "90"
       GT_ANALYTICS_ENABLED: "false"
       GT_ANALYTICS_PROFILE: "lean"
-      # Lean 1-vCPU nodes: fewer fetch worker threads reduces scheduler contention.
-      SHADOWBROKER_FETCH_WORKERS: "4"
-      SHADOWBROKER_HEAVY_FETCH_WORKERS: "1"
     deploy:
       resources:
         limits:

docker-compose.yml

@@ -25,7 +25,6 @@ services:
       - WINDY_API_KEY=${WINDY_API_KEY:-}
       - ADMIN_KEY=${ADMIN_KEY:-}
       - FINNHUB_API_KEY=${FINNHUB_API_KEY:-}
-      - AIRFRAMES_API_KEY=${AIRFRAMES_API_KEY:-}
       # Override allowed CORS origins (comma-separated). Auto-detects LAN IPs if empty.
       - CORS_ORIGINS=${CORS_ORIGINS:-}
       # Private Infonet bootstrap seeds. Seeds are discovery hints, not fixed roots.

frontend/package-lock.json

@@ -130,13 +130,13 @@
       "license": "MIT"
     },
     "node_modules/@babel/code-frame": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.7.tgz",
-      "integrity": "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw==",
+      "version": "7.29.0",
+      "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.0.tgz",
+      "integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/helper-validator-identifier": "^7.29.7",
+        "@babel/helper-validator-identifier": "^7.28.5",
         "js-tokens": "^4.0.0",
         "picocolors": "^1.1.1"
       },
@@ -145,9 +145,9 @@
       }
     },
     "node_modules/@babel/compat-data": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/compat-data/-/compat-data-7.29.7.tgz",
-      "integrity": "sha512-locTkQyKvwIEgBzVrn8693ebc97F2U8ZHjbXwDXJ5Fn2TCpNwTlKcaKLkdHop5c/icOFE7qt7Q9JC5hnKNa6Gg==",
+      "version": "7.29.0",
+      "resolved": "https://registry.npmjs.org/@babel/compat-data/-/compat-data-7.29.0.tgz",
+      "integrity": "sha512-T1NCJqT/j9+cn8fvkt7jtwbLBfLC/1y1c7NtCeXFRgzGTsafi68MRv8yzkYSapBnFA6L3U2VSc02ciDzoAJhJg==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -155,21 +155,21 @@
       }
     },
     "node_modules/@babel/core": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.29.7.tgz",
-      "integrity": "sha512-RgHBCvtjbOK2gXSNBNIkNoEc9qoVEtau3hj8gEqKQuL3HZAibKarWFEI3Lfm6EYKkLalOh8eSrj9b+ch9H/VBA==",
+      "version": "7.29.0",
+      "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.29.0.tgz",
+      "integrity": "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/code-frame": "^7.29.7",
-        "@babel/generator": "^7.29.7",
-        "@babel/helper-compilation-targets": "^7.29.7",
-        "@babel/helper-module-transforms": "^7.29.7",
-        "@babel/helpers": "^7.29.7",
-        "@babel/parser": "^7.29.7",
-        "@babel/template": "^7.29.7",
-        "@babel/traverse": "^7.29.7",
-        "@babel/types": "^7.29.7",
+        "@babel/code-frame": "^7.29.0",
+        "@babel/generator": "^7.29.0",
+        "@babel/helper-compilation-targets": "^7.28.6",
+        "@babel/helper-module-transforms": "^7.28.6",
+        "@babel/helpers": "^7.28.6",
+        "@babel/parser": "^7.29.0",
+        "@babel/template": "^7.28.6",
+        "@babel/traverse": "^7.29.0",
+        "@babel/types": "^7.29.0",
         "@jridgewell/remapping": "^2.3.5",
         "convert-source-map": "^2.0.0",
         "debug": "^4.1.0",
@@ -186,14 +186,14 @@
       }
     },
     "node_modules/@babel/generator": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.7.tgz",
-      "integrity": "sha512-DkXD5OJQaAQIdZ1bt3UZdEnHAn9Imd3IVBdX03UFe+ony9Ojw5pzr9YVKGDY1jt+Gcn/FnGkNf8r+Vj5NOJWtQ==",
+      "version": "7.29.1",
+      "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.1.tgz",
+      "integrity": "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/parser": "^7.29.7",
-        "@babel/types": "^7.29.7",
+        "@babel/parser": "^7.29.0",
+        "@babel/types": "^7.29.0",
         "@jridgewell/gen-mapping": "^0.3.12",
         "@jridgewell/trace-mapping": "^0.3.28",
         "jsesc": "^3.0.2"
@@ -203,14 +203,14 @@
       }
     },
     "node_modules/@babel/helper-compilation-targets": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helper-compilation-targets/-/helper-compilation-targets-7.29.7.tgz",
-      "integrity": "sha512-wem6WaBj4NaVYVdNhLPPVacES6ZJ+KBBfSkTMD3YZxbP3rm3Di85tJU5ljaUNhaOynt+Aj0xruhYuzQBt8n71g==",
+      "version": "7.28.6",
+      "resolved": "https://registry.npmjs.org/@babel/helper-compilation-targets/-/helper-compilation-targets-7.28.6.tgz",
+      "integrity": "sha512-JYtls3hqi15fcx5GaSNL7SCTJ2MNmjrkHXg4FSpOA/grxK8KwyZ5bubHsCq8FXCkua6xhuaaBit+3b7+VZRfcA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/compat-data": "^7.29.7",
-        "@babel/helper-validator-option": "^7.29.7",
+        "@babel/compat-data": "^7.28.6",
+        "@babel/helper-validator-option": "^7.27.1",
         "browserslist": "^4.24.0",
         "lru-cache": "^5.1.1",
         "semver": "^6.3.1"
@@ -220,9 +220,9 @@
       }
     },
     "node_modules/@babel/helper-globals": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helper-globals/-/helper-globals-7.29.7.tgz",
-      "integrity": "sha512-3nQVUAtvkKH9zahfWgw96Jc/uFOmjACE1kQz82E2lqWmHBgjzbNlsC22nuQTfahmWeQtTq5nQ/4Nnd2A1wj4zA==",
+      "version": "7.28.0",
+      "resolved": "https://registry.npmjs.org/@babel/helper-globals/-/helper-globals-7.28.0.tgz",
+      "integrity": "sha512-+W6cISkXFa1jXsDEdYA8HeevQT/FULhxzR99pxphltZcVaugps53THCeiWA8SguxxpSp3gKPiuYfSWopkLQ4hw==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -230,29 +230,29 @@
       }
     },
     "node_modules/@babel/helper-module-imports": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helper-module-imports/-/helper-module-imports-7.29.7.tgz",
-      "integrity": "sha512-ejHwrQQYcm9xnTivShn2IDOlIzInN34AXskvq9QicvCtEzq1Vzclu/tKF8Jq1Cg8JG2GL6/EmjgsCT7lXepE3g==",
+      "version": "7.28.6",
+      "resolved": "https://registry.npmjs.org/@babel/helper-module-imports/-/helper-module-imports-7.28.6.tgz",
+      "integrity": "sha512-l5XkZK7r7wa9LucGw9LwZyyCUscb4x37JWTPz7swwFE/0FMQAGpiWUZn8u9DzkSBWEcK25jmvubfpw2dnAMdbw==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/traverse": "^7.29.7",
-        "@babel/types": "^7.29.7"
+        "@babel/traverse": "^7.28.6",
+        "@babel/types": "^7.28.6"
       },
       "engines": {
         "node": ">=6.9.0"
       }
     },
     "node_modules/@babel/helper-module-transforms": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.29.7.tgz",
-      "integrity": "sha512-UPUVSyXbOh627KiCIGQSgwWzGeBKLkaJ9PJEdrngIwMSzxLR4jS4+f1f1jb7VzBbg8nFLaYotvVPFCTqdrmTAg==",
+      "version": "7.28.6",
+      "resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.28.6.tgz",
+      "integrity": "sha512-67oXFAYr2cDLDVGLXTEABjdBJZ6drElUSI7WKp70NrpyISso3plG9SAGEF6y7zbha/wOzUByWWTJvEDVNIUGcA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/helper-module-imports": "^7.29.7",
-        "@babel/helper-validator-identifier": "^7.29.7",
-        "@babel/traverse": "^7.29.7"
+        "@babel/helper-module-imports": "^7.28.6",
+        "@babel/helper-validator-identifier": "^7.28.5",
+        "@babel/traverse": "^7.28.6"
       },
       "engines": {
         "node": ">=6.9.0"
@@ -262,9 +262,9 @@
       }
     },
     "node_modules/@babel/helper-string-parser": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz",
-      "integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==",
+      "version": "7.27.1",
+      "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz",
+      "integrity": "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -272,9 +272,9 @@
       }
     },
     "node_modules/@babel/helper-validator-identifier": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz",
-      "integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==",
+      "version": "7.28.5",
+      "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.28.5.tgz",
+      "integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -282,9 +282,9 @@
       }
     },
     "node_modules/@babel/helper-validator-option": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helper-validator-option/-/helper-validator-option-7.29.7.tgz",
-      "integrity": "sha512-N9ZErrD+yW5geCDtBqnOoxmR8+tNKiGuxKlDpuJxfsqpa2dFcexaziGAE/qoHLiDDreVNMupxGmSoNlyvsA3gw==",
+      "version": "7.27.1",
+      "resolved": "https://registry.npmjs.org/@babel/helper-validator-option/-/helper-validator-option-7.27.1.tgz",
+      "integrity": "sha512-YvjJow9FxbhFFKDSuFnVCe2WxXk1zWc22fFePVNEaWJEu8IrZVlda6N0uHwzZrUM1il7NC9Mlp4MaJYbYd9JSg==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -292,27 +292,27 @@
       }
     },
     "node_modules/@babel/helpers": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.29.7.tgz",
-      "integrity": "sha512-1k2lAGRMfHTcwuNYcCNUmaUffmQv8KWMfh2iJUUeRlwlwH4FdNG7mfPI10NPfLHJFThE4Tyr4mv7kTNZOiPuBg==",
+      "version": "7.28.6",
+      "resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.28.6.tgz",
+      "integrity": "sha512-xOBvwq86HHdB7WUDTfKfT/Vuxh7gElQ+Sfti2Cy6yIWNW05P8iUslOVcZ4/sKbE+/jQaukQAdz/gf3724kYdqw==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/template": "^7.29.7",
-        "@babel/types": "^7.29.7"
+        "@babel/template": "^7.28.6",
+        "@babel/types": "^7.28.6"
       },
       "engines": {
         "node": ">=6.9.0"
       }
     },
     "node_modules/@babel/parser": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz",
-      "integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==",
+      "version": "7.29.0",
+      "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.0.tgz",
+      "integrity": "sha512-IyDgFV5GeDUVX4YdF/3CPULtVGSXXMLh1xVIgdCgxApktqnQV0r7/8Nqthg+8YLGaAtdyIlo2qIdZrbCv4+7ww==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/types": "^7.29.7"
+        "@babel/types": "^7.29.0"
       },
       "bin": {
         "parser": "bin/babel-parser.js"
@@ -332,33 +332,33 @@
       }
     },
     "node_modules/@babel/template": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.29.7.tgz",
-      "integrity": "sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg==",
+      "version": "7.28.6",
+      "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.28.6.tgz",
+      "integrity": "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/code-frame": "^7.29.7",
-        "@babel/parser": "^7.29.7",
-        "@babel/types": "^7.29.7"
+        "@babel/code-frame": "^7.28.6",
+        "@babel/parser": "^7.28.6",
+        "@babel/types": "^7.28.6"
       },
       "engines": {
         "node": ">=6.9.0"
       }
     },
     "node_modules/@babel/traverse": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.29.7.tgz",
-      "integrity": "sha512-EhlfNQtZ+NK22w5BM61ciuiq1m58ed33Wr1Xan//ZRTy6hgjnwyCffRYwzsGXdASJSUJ1guZILsErh1eQcl+zw==",
+      "version": "7.29.0",
+      "resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.29.0.tgz",
+      "integrity": "sha512-4HPiQr0X7+waHfyXPZpWPfWL/J7dcN1mx9gL6WdQVMbPnF3+ZhSMs8tCxN7oHddJE9fhNE7+lxdnlyemKfJRuA==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/code-frame": "^7.29.7",
-        "@babel/generator": "^7.29.7",
-        "@babel/helper-globals": "^7.29.7",
-        "@babel/parser": "^7.29.7",
-        "@babel/template": "^7.29.7",
-        "@babel/types": "^7.29.7",
+        "@babel/code-frame": "^7.29.0",
+        "@babel/generator": "^7.29.0",
+        "@babel/helper-globals": "^7.28.0",
+        "@babel/parser": "^7.29.0",
+        "@babel/template": "^7.28.6",
+        "@babel/types": "^7.29.0",
         "debug": "^4.3.1"
       },
       "engines": {
@@ -366,14 +366,14 @@
       }
     },
     "node_modules/@babel/types": {
-      "version": "7.29.7",
-      "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz",
-      "integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==",
+      "version": "7.29.0",
+      "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.0.tgz",
+      "integrity": "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
-        "@babel/helper-string-parser": "^7.29.7",
-        "@babel/helper-validator-identifier": "^7.29.7"
+        "@babel/helper-string-parser": "^7.27.1",
+        "@babel/helper-validator-identifier": "^7.28.5"
       },
       "engines": {
         "node": ">=6.9.0"
@@ -3627,9 +3627,9 @@
       "license": "MIT"
     },
     "node_modules/baseline-browser-mapping": {
-      "version": "2.10.38",
-      "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.38.tgz",
-      "integrity": "sha512-31/02mVB4yuQU6adKk5SlY6m+mxDwUq5KZkyYgnLrrKl7TEm1+3PyDtDBz2kOv/wxZz41GHsvV1A/u6RmiyBvw==",
+      "version": "2.10.0",
+      "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.0.tgz",
+      "integrity": "sha512-lIyg0szRfYbiy67j9KN8IyeD7q7hcmqnJ1ddWmNt19ItGpNN64mnllmxUNFIOdOm6by97jlL6wfpTTJrmnjWAA==",
       "license": "Apache-2.0",
       "bin": {
         "baseline-browser-mapping": "dist/cli.cjs"
@@ -3673,9 +3673,9 @@
       }
     },
     "node_modules/browserslist": {
-      "version": "4.28.2",
-      "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.2.tgz",
-      "integrity": "sha512-48xSriZYYg+8qXna9kwqjIVzuQxi+KYWp2+5nCYnYKPTr0LvD89Jqk2Or5ogxz0NUMfIjhh2lIUX/LyX9B4oIg==",
+      "version": "4.28.1",
+      "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.1.tgz",
+      "integrity": "sha512-ZC5Bd0LgJXgwGqUknZY/vkUQ04r8NXnJZ3yYi4vDmSiZmC/pdSN0NbNRPxZpbtO4uAfDUAFffO8IZoM3Gj8IkA==",
       "dev": true,
       "funding": [
         {
@@ -3693,11 +3693,11 @@
       ],
       "license": "MIT",
       "dependencies": {
-        "baseline-browser-mapping": "^2.10.12",
-        "caniuse-lite": "^1.0.30001782",
-        "electron-to-chromium": "^1.5.328",
-        "node-releases": "^2.0.36",
-        "update-browserslist-db": "^1.2.3"
+        "baseline-browser-mapping": "^2.9.0",
+        "caniuse-lite": "^1.0.30001759",
+        "electron-to-chromium": "^1.5.263",
+        "node-releases": "^2.0.27",
+        "update-browserslist-db": "^1.2.0"
       },
       "bin": {
         "browserslist": "cli.js"
@@ -3795,9 +3795,9 @@
       }
     },
     "node_modules/caniuse-lite": {
-      "version": "1.0.30001799",
-      "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001799.tgz",
-      "integrity": "sha512-hG1bReV+OUU+MOqK4t/ZWI0tZOyz3rqS9XuhOUz1cIcbwBKjOyJEJuw9ER5JuNyqxNk8u/JUVbGibBOL1yrjFw==",
+      "version": "1.0.30001774",
+      "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001774.tgz",
+      "integrity": "sha512-DDdwPGz99nmIEv216hKSgLD+D4ikHQHjBC/seF98N9CPqRX4M5mSxT9eTV6oyisnJcuzxtZy4n17yKKQYmYQOA==",
       "funding": [
         {
           "type": "opencollective",
@@ -3888,15 +3888,15 @@
       "license": "MIT"
     },
     "node_modules/concurrently": {
-      "version": "9.2.3",
-      "resolved": "https://registry.npmjs.org/concurrently/-/concurrently-9.2.3.tgz",
-      "integrity": "sha512-ihjs0E2SxvDgq/MK418hX6YycQgKhsqxpbZuZbHo0yKfqDWdymWMjWYIpCIzqDDLLKClHlXev8whW/8WXmJ0BA==",
+      "version": "9.2.1",
+      "resolved": "https://registry.npmjs.org/concurrently/-/concurrently-9.2.1.tgz",
+      "integrity": "sha512-fsfrO0MxV64Znoy8/l1vVIjjHa29SZyyqPgQBwhiDcaW8wJc2W3XWVOGx4M3oJBnv/zdUZIIp1gDeS98GzP8Ng==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
         "chalk": "4.1.2",
         "rxjs": "7.8.2",
-        "shell-quote": "1.8.4",
+        "shell-quote": "1.8.3",
         "supports-color": "8.1.1",
         "tree-kill": "1.2.2",
         "yargs": "17.7.2"
@@ -4226,9 +4226,9 @@
       "license": "ISC"
     },
     "node_modules/electron-to-chromium": {
-      "version": "1.5.375",
-      "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.375.tgz",
-      "integrity": "sha512-ZWP5eB4BVPW/ZYo9252hQZHZ5XavtsTgpbhcmMmRwymavC5AsLWQWBPaKMeNd2LW0KGby5HPXvj7+sr4ta5j/Q==",
+      "version": "1.5.302",
+      "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.302.tgz",
+      "integrity": "sha512-sM6HAN2LyK82IyPBpznDRqlTQAtuSaO+ShzFiWTvoMJLHyZ+Y39r8VMfHzwbU8MVBzQ4Wdn85+wlZl2TLGIlwg==",
       "dev": true,
       "license": "ISC"
     },
@@ -6198,20 +6198,10 @@
       "license": "MIT"
     },
     "node_modules/js-yaml": {
-      "version": "4.2.0",
-      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.2.0.tgz",
-      "integrity": "sha512-ePWsvanv0DWuDRsW8dnt+R4jQ31SCRCQ7hhNcPXZPsoBZiemuZNYGf7adZdqX2D86j6rvKp3RpCxVTSb8WQlOw==",
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.1.tgz",
+      "integrity": "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==",
       "dev": true,
-      "funding": [
-        {
-          "type": "github",
-          "url": "https://github.com/sponsors/puzrin"
-        },
-        {
-          "type": "github",
-          "url": "https://github.com/sponsors/nodeca"
-        }
-      ],
       "license": "MIT",
       "dependencies": {
         "argparse": "^2.0.1"
@@ -7083,14 +7073,11 @@
       }
     },
     "node_modules/node-releases": {
-      "version": "2.0.48",
-      "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.48.tgz",
-      "integrity": "sha512-1uz8041X6LoI6ZSdZacM9lVY28vuzDlSKitnpbSNK0RfKoIJkX29NBPVEFXhnuSuEOA9Ww0xnPJ+ILWbGAv8DA==",
+      "version": "2.0.27",
+      "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.27.tgz",
+      "integrity": "sha512-nmh3lCkYZ3grZvqcCH+fjmQ7X+H0OeZgP40OierEaAptX4XofMh5kwNbWh7lBduUzCcV/8kZ+NDLCwm2iorIlA==",
       "dev": true,
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      }
+      "license": "MIT"
     },
     "node_modules/object-assign": {
       "version": "4.1.1",
@@ -8200,9 +8187,9 @@
       }
     },
     "node_modules/shell-quote": {
-      "version": "1.8.4",
-      "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.4.tgz",
-      "integrity": "sha512-VsC6n6vz1ihYYyZZwX7YZSF5l5x36ca17OC+a69h94YqB7X6XLwf+5MOgynYir2SLFUbl8gIYvBo8K8RoNQ6bQ==",
+      "version": "1.8.3",
+      "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.3.tgz",
+      "integrity": "sha512-ObmnIF4hXNg1BqhnHmgbDETF8dLPCggZWBjkQfhZpbszZnYur5DUljTcCHii5LC3J5E0yeO/1LIMyH+UvHQgyw==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -9037,9 +9024,9 @@
       }
     },
     "node_modules/undici": {
-      "version": "7.28.0",
-      "resolved": "https://registry.npmjs.org/undici/-/undici-7.28.0.tgz",
-      "integrity": "sha512-cRZYrTDwWznlnRiPjggAGxZXanty6M8RV1ff8Wm4LWXBp7/IG8v5DnOm74DtUBp9OONpK75YlPnIjQqX0dBDtA==",
+      "version": "7.24.2",
+      "resolved": "https://registry.npmjs.org/undici/-/undici-7.24.2.tgz",
+      "integrity": "sha512-P9J1HWYV/ajFr8uCqk5QixwiRKmB1wOamgS0e+o2Z4A44Ej2+thFVRLG/eA7qprx88XXhnV5Bl8LHXTURpzB3Q==",
       "dev": true,
       "license": "MIT",
       "engines": {

frontend/src/components/DatalinkMessagesBlock.tsx

@@ -1,230 +0,0 @@
-'use client';
-
-import { useCallback, useEffect, useRef, useState } from 'react';
-import { API_BASE } from '@/lib/api';
-
-type DatalinkMessage = {
-  id: number;
-  timestamp?: string;
-  label?: string;
-  text?: string;
-  source_type?: string;
-  summary?: string;
-  kind?: string;
-  readable?: boolean;
-};
-
-const PRIORITY_POLL_MS = 3_000;
-const PRIORITY_POLL_MAX_MS = 45_000;
-
-function DatalinkMessageRow({ message }: { message: DatalinkMessage }) {
-  const [showRaw, setShowRaw] = useState(false);
-  const summary = message.summary?.trim();
-  const raw = message.text?.trim() || '';
-  const hasSummary = Boolean(summary);
-  const showRawBlock = showRaw || (!hasSummary && raw);
-
-  return (
-    <div className="text-[10px] font-mono leading-snug border border-[var(--border-primary)]/60 bg-black/20 px-2 py-1.5">
-      <div className="flex items-center gap-2 text-[var(--text-muted)] mb-0.5">
-        <span>{formatDatalinkTime(message.timestamp)}</span>
-        {message.label ? <span className="text-orange-400/90">{message.label}</span> : null}
-        {message.source_type ? <span className="truncate">{message.source_type}</span> : null}
-      </div>
-      {hasSummary ? (
-        <div className="text-[var(--text-primary)] break-words">{summary}</div>
-      ) : null}
-      {hasSummary && raw ? (
-        <button
-          type="button"
-          onClick={() => setShowRaw((value) => !value)}
-          className="mt-0.5 text-[var(--text-muted)] hover:text-cyan-400/90 underline underline-offset-2"
-        >
-          {showRaw ? 'hide raw' : 'show raw'}
-        </button>
-      ) : null}
-      {showRawBlock && raw ? (
-        <div className="mt-0.5 text-[var(--text-muted)] whitespace-pre-wrap break-words text-[9px] leading-relaxed max-h-24 overflow-y-auto">
-          {raw}
-        </div>
-      ) : null}
-    </div>
-  );
-}
-
-function formatDatalinkTime(value?: string): string {
-  if (!value) return '--:--';
-  try {
-    const date = new Date(value);
-    if (Number.isNaN(date.getTime())) return value.slice(11, 16) || value;
-    return date.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit', hour12: false });
-  } catch {
-    return '--:--';
-  }
-}
-
-export default function DatalinkMessagesBlock({
-  icao24,
-  registration,
-  callsign,
-}: {
-  icao24?: string;
-  registration?: string;
-  callsign?: string;
-}) {
-  const [loading, setLoading] = useState(true);
-  const [configured, setConfigured] = useState<boolean | null>(null);
-  const [messages, setMessages] = useState<DatalinkMessage[]>([]);
-  const [hint, setHint] = useState<string | null>(null);
-  const [loadError, setLoadError] = useState<string | null>(null);
-  const [priorityScanning, setPriorityScanning] = useState(false);
-  const [hiddenCount, setHiddenCount] = useState(0);
-  const [showHidden, setShowHidden] = useState(false);
-  const pollUntilRef = useRef(0);
-
-  const buildParams = useCallback(() => {
-    const params = new URLSearchParams();
-    if (icao24) params.set('icao24', icao24);
-    if (registration) params.set('registration', registration);
-    if (callsign) params.set('callsign', callsign);
-    return params;
-  }, [icao24, registration, callsign]);
-
-  const fetchDatalink = useCallback(
-    async (opts?: { showLoading?: boolean }) => {
-      const params = buildParams();
-      if ([...params.keys()].length === 0) {
-        setLoading(false);
-        return;
-      }
-
-      if (opts?.showLoading) {
-        setLoading(true);
-        setLoadError(null);
-      }
-
-      try {
-        const res = await fetch(`${API_BASE}/api/aviation/datalink/messages?${params.toString()}`, {
-          cache: 'no-store',
-        });
-        if (!res.ok) throw new Error(`datalink ${res.status}`);
-        const json = await res.json();
-        setConfigured(Boolean(json.configured));
-        setMessages(Array.isArray(json.messages) ? json.messages : []);
-        setHiddenCount(typeof json.hidden_count === 'number' ? json.hidden_count : 0);
-        setHint(typeof json.hint === 'string' ? json.hint : null);
-        setLoadError(null);
-        if (json.priority_scan || json.queued_refresh) {
-          setPriorityScanning(true);
-          pollUntilRef.current = Date.now() + PRIORITY_POLL_MAX_MS;
-        }
-        if (Array.isArray(json.messages) && json.messages.length > 0) {
-          setPriorityScanning(false);
-        }
-      } catch {
-        if (opts?.showLoading) {
-          setConfigured(null);
-          setMessages([]);
-          setLoadError('Could not reach ACARS cache. Try again in a moment.');
-        }
-      } finally {
-        if (opts?.showLoading) setLoading(false);
-      }
-    },
-    [buildParams],
-  );
-
-  useEffect(() => {
-    pollUntilRef.current = 0;
-    setPriorityScanning(false);
-    void fetchDatalink({ showLoading: true });
-  }, [fetchDatalink]);
-
-  useEffect(() => {
-    if (!priorityScanning) return;
-
-    const intervalId = window.setInterval(() => {
-      if (Date.now() > pollUntilRef.current) {
-        setPriorityScanning(false);
-        return;
-      }
-      void fetchDatalink();
-    }, PRIORITY_POLL_MS);
-
-    return () => window.clearInterval(intervalId);
-  }, [priorityScanning, fetchDatalink]);
-
-  if (loading) {
-    return (
-      <div className="border-b border-[var(--border-primary)] pb-2">
-        <span className="text-[var(--text-muted)] text-[10px] block mb-1">DATALINK (AIRFRAMES)</span>
-        <span className="text-[10px] font-mono text-[var(--text-muted)]">Loading ACARS cache…</span>
-      </div>
-    );
-  }
-
-  if (loadError) {
-    return (
-      <div className="border-b border-[var(--border-primary)] pb-2">
-        <span className="text-[var(--text-muted)] text-[10px] block mb-1">DATALINK (AIRFRAMES)</span>
-        <p className="text-[10px] font-mono text-amber-400/90 leading-relaxed">{loadError}</p>
-      </div>
-    );
-  }
-
-  if (configured === false) {
-    return (
-      <div className="border-b border-[var(--border-primary)] pb-2">
-        <span className="text-[var(--text-muted)] text-[10px] block mb-1">DATALINK (AIRFRAMES)</span>
-        <p className="text-[10px] font-mono text-amber-400/90 leading-relaxed">
-          {hint || 'Add your Airframes API key in Settings → API Keys to enable ACARS datalink on plane dossiers.'}
-        </p>
-      </div>
-    );
-  }
-
-  if (!messages.length) {
-    return (
-      <div className="border-b border-[var(--border-primary)] pb-2">
-        <span className="text-[var(--text-muted)] text-[10px] block mb-1">DATALINK (AIRFRAMES)</span>
-        <p className="text-[10px] font-mono text-[var(--text-muted)]">
-          {priorityScanning
-            ? 'Priority scan queued for this aircraft (~2s)…'
-            : 'No recent ACARS/VDL messages for this aircraft.'}
-        </p>
-      </div>
-    );
-  }
-
-  return (
-    <div className="border-b border-[var(--border-primary)] pb-2">
-      <span className="text-[var(--text-muted)] text-[10px] block mb-1">DATALINK (AIRFRAMES)</span>
-      {priorityScanning ? (
-        <p className="text-[10px] font-mono text-cyan-500/70 mb-1">Refreshing this aircraft…</p>
-      ) : null}
-      <div className="max-h-36 overflow-y-auto space-y-1.5 pr-1">
-        {messages.map((message) => (
-          <DatalinkMessageRow key={message.id} message={message} />
-        ))}
-      </div>
-      {hiddenCount > 0 ? (
-        <p className="text-[9px] font-mono text-[var(--text-muted)] mt-1">
-          {hiddenCount} binary/fragment message{hiddenCount === 1 ? '' : 's'} hidden.{' '}
-          <button
-            type="button"
-            onClick={() => setShowHidden((value) => !value)}
-            className="underline underline-offset-2 hover:text-cyan-400/90"
-          >
-            {showHidden ? 'Hide note' : 'Why?'}
-          </button>
-          {showHidden ? (
-            <span className="block mt-0.5 text-[var(--text-muted)]/80">
-              VDL splits long telemetry into many frames. Southwest also uses proprietary formats
-              that cannot be decoded without airline keys.
-            </span>
-          ) : null}
-        </p>
-      ) : null}
-    </div>
-  );
-}

frontend/src/components/MapLegend.tsx

@@ -113,7 +113,7 @@ const LEGEND: LegendCategory[] = [
       { svg: heli('#32CD32'), label: 'Medical / Fire / Rescue (lime)' },
       { svg: airliner('yellow'), label: 'Military / Intelligence (yellow)' },
       { svg: airliner('#222'), label: 'PIA — Privacy / Stealth (black)' },
-      { svg: airliner('#FF8C00'), label: 'As Seen on TV / Joe Cool (orange)' },
+      { svg: airliner('#FF8C00'), label: 'Private Flights / Joe Cool (orange)' },
       { svg: airliner('white'), label: 'Climate Crisis (white)' },
       { svg: airliner('#9B59B6'), label: 'Private Jets / Historic / Other (purple)' },
     ],

frontend/src/components/MeshChat/AgentShellPanel.tsx

@@ -12,7 +12,7 @@ import { FitAddon } from '@xterm/addon-fit';
 
 import '@xterm/xterm/css/xterm.css';
 
-import { mintAgentShellWsToken, resolveAgentShellWsUrl } from '@/lib/agentShellWs';
+import { resolveAgentShellWsUrl } from '@/lib/agentShellWs';
 
 
 
@@ -302,12 +302,11 @@ export default function AgentShellPanel({ active, expanded, onExpandedChange }:
 
 
 
-    void (async () => {
-      const wsToken = await mintAgentShellWsToken();
-      const ws = new WebSocket(resolveAgentShellWsUrl(storedCwd, wsToken ?? undefined));
-      ws.binaryType = 'arraybuffer';
+    const ws = new WebSocket(resolveAgentShellWsUrl(storedCwd));
 
-      wsRef.current = ws;
+    ws.binaryType = 'arraybuffer';
+
+    wsRef.current = ws;
 
 
 
@@ -424,7 +423,7 @@ export default function AgentShellPanel({ active, expanded, onExpandedChange }:
       }
 
     });
-    })();
+
   }, []);
 
 

frontend/src/components/NewsFeed.tsx

@@ -11,7 +11,6 @@ import { fetchWikipediaSummary } from '@/lib/wikimediaClient';
 import type { SelectedEntity, RegionDossier, FimiData } from "@/types/dashboard";
 import { useDataKeys } from '@/hooks/useDataStore';
 import { API_BASE } from '@/lib/api';
-import DatalinkMessagesBlock from '@/components/DatalinkMessagesBlock';
 import { lookupShodanHost } from '@/lib/shodanClient';
 import type { ShodanHost } from '@/types/shodan';
 
@@ -910,11 +909,6 @@ function NewsFeedInner({ selectedEntity, regionDossier, regionDossierLoading, gt
                                 <span className={`text-xs font-bold ${flight.squawk === '7700' ? 'text-red-400 animate-pulse' : flight.squawk === '7600' ? 'text-yellow-400' : 'text-[var(--text-primary)]'}`}>{flight.squawk}{flight.squawk === '7700' ? ' ⚠ EMERGENCY' : flight.squawk === '7600' ? ' COMMS LOST' : ''}</span>
                             </div>
                         )}
-                        <DatalinkMessagesBlock
-                            icao24={flight.icao24}
-                            registration={flight.registration}
-                            callsign={flight.callsign}
-                        />
                         <EmissionsEstimateBlock flight={flightForEmissions} />
                         {flight.alert_link && (
                             <div className="flex justify-between items-center border-b border-[var(--border-primary)] pb-2">
@@ -1168,11 +1162,6 @@ function NewsFeedInner({ selectedEntity, regionDossier, regionDossierLoading, gt
                                 <span className={`text-xs font-bold ${flight.squawk === '7700' ? 'text-red-400 animate-pulse' : flight.squawk === '7600' ? 'text-yellow-400' : 'text-[var(--text-primary)]'}`}>{flight.squawk}{flight.squawk === '7700' ? ' ⚠ EMERGENCY' : flight.squawk === '7600' ? ' COMMS LOST' : ''}</span>
                             </div>
                         )}
-                        <DatalinkMessagesBlock
-                            icao24={flight.icao24}
-                            registration={flight.registration}
-                            callsign={flight.callsign}
-                        />
                         <div className="flex justify-between items-center border-b border-[var(--border-primary)] pb-2">
                             <span className="text-[var(--text-muted)] text-[10px]">ROUTE</span>
                             <span className="text-cyan-400 text-xs font-bold">{flight.origin_name !== "UNKNOWN" ? `[${flight.origin_name}] → [${flight.dest_name}]` : "UNKNOWN"}</span>

frontend/src/components/SettingsPanel.tsx

@@ -1184,7 +1184,7 @@ const SettingsPanel = React.memo(function SettingsPanel({
             animate={{ opacity: 1, x: 0 }}
             exit={{ opacity: 0, x: -300 }}
             transition={{ type: 'spring', damping: 25, stiffness: 300 }}
-            className="fixed left-0 top-0 bottom-0 w-[480px] bg-[var(--bg-secondary)]/95 backdrop-blur-sm border-r border-cyan-900/50 z-[9999] flex flex-col overflow-hidden shadow-[4px_0_40px_rgba(0,0,0,0.3)]"
+            className="fixed left-0 top-0 bottom-0 w-[480px] bg-[var(--bg-secondary)]/95 backdrop-blur-sm border-r border-cyan-900/50 z-[9999] flex flex-col shadow-[4px_0_40px_rgba(0,0,0,0.3)]"
           >
             {/* Header */}
             <div className="flex items-center justify-between p-6 border-b border-[var(--border-primary)]/80">
@@ -1362,51 +1362,51 @@ const SettingsPanel = React.memo(function SettingsPanel({
               </div>
             )}
 
-            <div className="grid grid-cols-5 min-w-0 shrink-0 border-b border-[var(--border-primary)]/60">
+            <div className="flex border-b border-[var(--border-primary)]/60">
               <button
                 onClick={() => setActiveTab('api-keys')}
-                className={`min-w-0 px-1.5 py-2.5 text-[11px] font-mono tracking-wide font-bold transition-colors flex items-center justify-center gap-1 ${activeTab === 'api-keys' ? 'text-cyan-400 border-b-2 border-cyan-500 bg-cyan-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
+                className={`flex-1 px-4 py-2.5 text-sm font-mono tracking-widest font-bold transition-colors flex items-center justify-center gap-1.5 ${activeTab === 'api-keys' ? 'text-cyan-400 border-b-2 border-cyan-500 bg-cyan-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
               >
-                <Key size={10} className="shrink-0" />
-                <span className="truncate">{t('settings.general').toUpperCase()}</span>
+                <Key size={10} />
+                {t('settings.general').toUpperCase()}
               </button>
               <button
                 onClick={() => setActiveTab('news-feeds')}
-                className={`min-w-0 px-1.5 py-2.5 text-[11px] font-mono tracking-wide font-bold transition-colors flex items-center justify-center gap-1 ${activeTab === 'news-feeds' ? 'text-orange-400 border-b-2 border-orange-500 bg-orange-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
+                className={`flex-1 px-4 py-2.5 text-sm font-mono tracking-widest font-bold transition-colors flex items-center justify-center gap-1.5 ${activeTab === 'news-feeds' ? 'text-orange-400 border-b-2 border-orange-500 bg-orange-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
               >
-                <Rss size={10} className="shrink-0" />
-                <span className="truncate">{t('settings.feeds').toUpperCase()}</span>
+                <Rss size={10} />
+                {t('settings.feeds').toUpperCase()}
                 {feedsDirty && (
-                  <span className="w-1.5 h-1.5 shrink-0 rounded-full bg-orange-400 animate-pulse" />
+                  <span className="w-1.5 h-1.5 rounded-full bg-orange-400 animate-pulse" />
                 )}
               </button>
               <button
                 onClick={() => setActiveTab('sentinel')}
-                className={`min-w-0 px-1.5 py-2.5 text-[11px] font-mono tracking-wide font-bold transition-colors flex items-center justify-center gap-1 ${activeTab === 'sentinel' ? 'text-purple-400 border-b-2 border-purple-500 bg-purple-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
+                className={`flex-1 px-4 py-2.5 text-sm font-mono tracking-widest font-bold transition-colors flex items-center justify-center gap-1.5 ${activeTab === 'sentinel' ? 'text-purple-400 border-b-2 border-purple-500 bg-purple-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
               >
-                <Satellite size={10} className="shrink-0" />
-                <span className="truncate">SENTINEL</span>
+                <Satellite size={10} />
+                SENTINEL
               </button>
               <button
                 onClick={() => setActiveTab('sar')}
-                className={`min-w-0 px-1.5 py-2.5 text-[11px] font-mono tracking-wide font-bold transition-colors flex items-center justify-center gap-1 ${activeTab === 'sar' ? 'text-amber-400 border-b-2 border-amber-500 bg-amber-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
+                className={`flex-1 px-4 py-2.5 text-sm font-mono tracking-widest font-bold transition-colors flex items-center justify-center gap-1.5 ${activeTab === 'sar' ? 'text-amber-400 border-b-2 border-amber-500 bg-amber-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
               >
-                <Radar size={10} className="shrink-0" />
-                <span className="truncate">{t('settings.sar').toUpperCase()}</span>
+                <Radar size={10} />
+                {t('settings.sar').toUpperCase()}
               </button>
               <button
                 onClick={() => setActiveTab('protocol')}
-                className={`min-w-0 px-1.5 py-2.5 text-[11px] font-mono tracking-wide font-bold transition-colors flex items-center justify-center gap-1 ${activeTab === 'protocol' ? 'text-green-400 border-b-2 border-green-500 bg-green-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
+                className={`flex-1 px-4 py-2.5 text-sm font-mono tracking-widest font-bold transition-colors flex items-center justify-center gap-1.5 ${activeTab === 'protocol' ? 'text-green-400 border-b-2 border-green-500 bg-green-950/10' : 'text-[var(--text-muted)] hover:text-[var(--text-secondary)]'}`}
               >
-                <Shield size={10} className="shrink-0" />
-                <span className="truncate">{t('settings.infonet').toUpperCase()}</span>
+                <Shield size={10} />
+                {t('settings.infonet').toUpperCase()}
               </button>
             </div>
 
             {/* ==================== API KEYS TAB ==================== */}
             {/* ==================== MESH PROTOCOL TAB ==================== */}
             {activeTab === 'protocol' && (
-              <div className="flex-1 min-h-0 flex flex-col overflow-y-auto styled-scrollbar">
+              <div className="flex-1 flex flex-col overflow-y-auto styled-scrollbar">
                 <div className="mx-4 mt-4 p-3 border border-cyan-900/30 bg-cyan-950/12">
                   <div className="flex items-center justify-between gap-3">
                     <div>
@@ -2945,9 +2945,6 @@ function SarSettingsTab() {
   const [loading, setLoading] = useState(true);
   const [actionMsg, setActionMsg] = useState<{ type: 'ok' | 'err'; text: string } | null>(null);
   const [disabling, setDisabling] = useState(false);
-  const [submitting, setSubmitting] = useState(false);
-  const [earthdataUser, setEarthdataUser] = useState('');
-  const [earthdataToken, setEarthdataToken] = useState('');
 
   const fetchStatus = useCallback(async () => {
     try {
@@ -2963,62 +2960,9 @@ function SarSettingsTab() {
   useEffect(() => { fetchStatus(); }, [fetchStatus]);
 
   const products = (status?.products ?? {}) as Record<string, unknown>;
-  const tokenSet = Boolean(products.earthdata_token_set);
-  const fullyConfigured = Boolean(products.fully_configured);
-  const optInEnabled = Boolean(products.enabled);
-  const runtimeStoreExists = Boolean(products.runtime_store_exists);
+  const modeBEnabled = !!products.enabled;
   const catalogEnabled = !!(status?.catalog as Record<string, unknown>)?.enabled;
   const openclawEnabled = !!status?.openclaw_enabled;
-  const missing = Array.isArray(products.missing)
-    ? (products.missing as string[])
-    : [];
-
-  const modeBStatusLabel = (() => {
-    if (fullyConfigured) return 'Active — credentials configured';
-    if (optInEnabled && !tokenSet) return 'Opt-in on — Earthdata token missing';
-    if (tokenSet && !optInEnabled) return 'Token present — enable Mode B below';
-    return 'Not configured';
-  })();
-
-  const handleEnable = async () => {
-    if (earthdataToken.trim().length < 8) {
-      setActionMsg({ type: 'err', text: 'Earthdata token looks too short. Paste the full token string.' });
-      return;
-    }
-    setSubmitting(true);
-    setActionMsg(null);
-    try {
-      const res = await fetch(`${API_BASE}/api/sar/mode-b/enable`, {
-        method: 'POST',
-        headers: { 'Content-Type': 'application/json' },
-        credentials: 'include',
-        body: JSON.stringify({
-          earthdata_user: earthdataUser.trim(),
-          earthdata_token: earthdataToken.trim(),
-        }),
-      });
-      if (!res.ok) {
-        const body = await res.json().catch(() => ({}));
-        let msg = `HTTP ${res.status}`;
-        const d = body?.detail;
-        if (typeof d === 'string') msg = d;
-        else if (Array.isArray(d) && d.length > 0) {
-          msg = d.map((item: { msg?: string; loc?: string[] }) => item?.msg || 'invalid').join('; ');
-        }
-        throw new Error(msg);
-      }
-      setEarthdataToken('');
-      setActionMsg({ type: 'ok', text: 'Mode B enabled. Credentials saved on this node.' });
-      await fetchStatus();
-    } catch (e) {
-      setActionMsg({
-        type: 'err',
-        text: e instanceof Error ? e.message : 'Failed to save SAR credentials',
-      });
-    } finally {
-      setSubmitting(false);
-    }
-  };
 
   const handleDisable = async () => {
     setDisabling(true);
@@ -3032,8 +2976,6 @@ function SarSettingsTab() {
         const body = await res.json().catch(() => ({}));
         throw new Error(typeof body?.detail === 'string' ? body.detail : `HTTP ${res.status}`);
       }
-      setEarthdataUser('');
-      setEarthdataToken('');
       setActionMsg({ type: 'ok', text: 'Mode B disabled. Credentials wiped.' });
       await fetchStatus();
     } catch (e) {
@@ -3073,14 +3015,10 @@ function SarSettingsTab() {
                 </span>
               </div>
               <div className="flex items-center gap-2">
-                <span
-                  className={`w-2 h-2 rounded-full ${
-                    fullyConfigured ? 'bg-green-400' : optInEnabled || tokenSet ? 'bg-yellow-400' : 'bg-gray-500'
-                  }`}
-                />
+                <span className={`w-2 h-2 rounded-full ${modeBEnabled ? 'bg-green-400' : 'bg-yellow-400'}`} />
                 <span className="text-[11px]">
                   <span className="text-amber-300 font-bold">Mode B</span> (Anomalies):{' '}
-                  {modeBStatusLabel}
+                  {modeBEnabled ? 'Active — credentials stored' : 'Not configured'}
                 </span>
               </div>
               <div className="flex items-center gap-2">
@@ -3091,100 +3029,63 @@ function SarSettingsTab() {
                 </span>
               </div>
             </div>
-            {missing.length > 0 && !fullyConfigured && (
-              <p className="text-[10px] text-amber-200/60 pt-1">
-                Still needed: {missing.join(' · ')}
-              </p>
-            )}
           </div>
         </div>
       </div>
 
-      {/* Mode B credential entry — always visible */}
-      <div className="mx-4 mt-3 p-3 border border-amber-900/20 bg-amber-950/5 space-y-3">
-        <p className="text-[11px] font-mono text-amber-300 font-bold tracking-wide">
-          MODE B — EARTHDATA CREDENTIALS
-        </p>
-        <p className="text-[11px] font-mono text-[var(--text-muted)] leading-relaxed">
-          Free NASA Earthdata Login required for OPERA ground-change products. Paste your user
-          token below — stored only on this node
-          {runtimeStoreExists ? ' in the runtime credentials file' : ' (created on first save)'}.
-        </p>
-
-        <ol className="list-decimal list-inside space-y-1 text-[11px] font-mono text-[var(--text-secondary)]">
-          <li>
-            Register at{' '}
-            <a
-              href="https://urs.earthdata.nasa.gov/users/new"
-              target="_blank"
-              rel="noopener noreferrer"
-              className="text-amber-400 underline hover:text-amber-300"
-            >
-              urs.earthdata.nasa.gov
-            </a>
-          </li>
-          <li>
-            Generate a user token from your{' '}
-            <a
-              href="https://urs.earthdata.nasa.gov/profile"
-              target="_blank"
-              rel="noopener noreferrer"
-              className="text-amber-400 underline hover:text-amber-300"
-            >
-              Earthdata profile
-            </a>
-          </li>
-        </ol>
-
-        <div className="space-y-2">
-          <label htmlFor="sar-settings-earthdata-user" className="block text-[11px] text-amber-200/80">
-            Earthdata username (optional)
-          </label>
-          <input
-            id="sar-settings-earthdata-user"
-            type="text"
-            value={earthdataUser}
-            onChange={(e) => setEarthdataUser(e.target.value)}
-            placeholder="yourname"
-            autoComplete="off"
-            className="w-full rounded border border-amber-400/30 bg-zinc-900 px-3 py-2 text-xs text-amber-100 placeholder:text-amber-100/30 focus:border-amber-400/70 focus:outline-none cursor-text"
-          />
-
-          <label htmlFor="sar-settings-earthdata-token" className="block text-[11px] text-amber-200/80 mt-2">
-            Earthdata user token (required)
-          </label>
-          <input
-            id="sar-settings-earthdata-token"
-            type="password"
-            value={earthdataToken}
-            onChange={(e) => setEarthdataToken(e.target.value)}
-            placeholder={tokenSet ? '••••••••  (enter new token to rotate)' : 'eyJ0eXAiOiJKV1QiLCJhbGciOi...'}
-            autoComplete="off"
-            className="w-full rounded border border-amber-400/30 bg-zinc-900 px-3 py-2 text-xs text-amber-100 placeholder:text-amber-100/30 focus:border-amber-400/70 focus:outline-none font-mono cursor-text"
-          />
-        </div>
-
-        <div className="flex flex-wrap gap-2">
-          <button
-            type="button"
-            onClick={() => void handleEnable()}
-            disabled={submitting || earthdataToken.trim().length < 8}
-            className="px-3 py-1.5 text-[10px] font-mono font-bold tracking-wide border border-amber-400/50 text-amber-200 hover:bg-amber-500/10 transition disabled:opacity-50"
-          >
-            {submitting ? 'SAVING...' : fullyConfigured ? 'UPDATE CREDENTIALS' : 'ENABLE MODE B'}
-          </button>
-          {(fullyConfigured || optInEnabled || tokenSet) && (
+      {/* Mode B Controls */}
+      {modeBEnabled && (
+        <div className="mx-4 mt-3 p-3 border border-amber-900/20 bg-amber-950/5 space-y-3">
+          <p className="text-[11px] font-mono text-amber-300 font-bold tracking-wide">
+            MODE B CREDENTIALS
+          </p>
+          <p className="text-[11px] font-mono text-[var(--text-muted)]">
+            Earthdata credentials are stored server-side in{' '}
+            <span className="text-amber-400/80">backend/data/sar_runtime.json</span>.
+            Disabling Mode B wipes them from disk.
+          </p>
+          <div className="flex gap-2">
             <button
               type="button"
-              onClick={() => void handleDisable()}
+              onClick={handleDisable}
               disabled={disabling}
               className="px-3 py-1.5 text-[10px] font-mono font-bold tracking-wide border border-red-500/40 text-red-400 hover:bg-red-500/10 transition disabled:opacity-50"
             >
               {disabling ? 'DISABLING...' : 'REVOKE & DISABLE MODE B'}
             </button>
-          )}
+          </div>
         </div>
-      </div>
+      )}
+
+      {/* Setup Guide (when Mode B not active) */}
+      {!modeBEnabled && (
+        <div className="mx-4 mt-3 p-3 border border-amber-900/20 bg-amber-950/5 space-y-3">
+          <p className="text-[11px] font-mono text-amber-300 font-bold tracking-wide">
+            ENABLE MODE B
+          </p>
+          <p className="text-[11px] font-mono text-[var(--text-muted)]">
+            Mode B requires a free NASA Earthdata account. To set up:
+          </p>
+          <ol className="list-decimal list-inside space-y-1 text-[11px] font-mono text-[var(--text-secondary)]">
+            <li>
+              Register at{' '}
+              <a
+                href="https://urs.earthdata.nasa.gov/users/new"
+                target="_blank"
+                rel="noopener noreferrer"
+                className="text-amber-400 underline hover:text-amber-300"
+              >
+                urs.earthdata.nasa.gov
+              </a>
+            </li>
+            <li>Generate a user token from your Earthdata profile page</li>
+            <li>
+              Toggle the <span className="text-white">SAR Ground-Change</span> layer ON
+              in the left panel — the first-run wizard will prompt for your token
+            </li>
+          </ol>
+        </div>
+      )}
 
       {/* Action feedback */}
       {actionMsg && (

frontend/src/components/WorldviewLeftPanel.tsx

@@ -44,7 +44,6 @@ import {
   Radar,
   MapPin,
   Truck,
-  RefreshCw,
 } from 'lucide-react';
 import RoadCorridorLayerControls from '@/components/RoadCorridorLayerControls';
 import { API_BASE } from '@/lib/api';
@@ -144,61 +143,8 @@ import type {
   KiwiSDR,
   Scanner,
   TrackedFlight,
-  DashboardData,
 } from '@/types/dashboard';
-import { useDataKeys } from '@/hooks/useDataStore';
-
-/** Keys the layer panel reads — avoids re-rendering on unrelated fast-poll keys. */
-const WORLDVIEW_PANEL_DATA_KEYS = [
-  'ships',
-  'sigint_totals',
-  'sigint',
-  'cctv_total',
-  'cctv',
-  'satnogs_total',
-  'satnogs_stations',
-  'tinygs_total',
-  'tinygs_satellites',
-  'tracked_flights',
-  'commercial_flights',
-  'private_flights',
-  'private_jets',
-  'military_flights',
-  'gps_jamming',
-  'fishing_activity',
-  'satellite_source',
-  'satellite_analysis',
-  'satellites',
-  'road_corridor_trends',
-  'earthquakes',
-  'firms_fires',
-  'ukraine_alerts',
-  'weather_alerts',
-  'volcanoes',
-  'air_quality',
-  'sar_anomalies',
-  'sar_scenes',
-  'uap_sightings',
-  'wastewater',
-  'datacenters',
-  'internet_outages',
-  'power_plants',
-  'military_bases',
-  'trains',
-  'malware_threats',
-  'scm_suppliers',
-  'cyber_threats',
-  'kiwisdr',
-  'psk_reporter',
-  'scanners',
-  'frontlines',
-  'gdelt',
-  'telegram_osint',
-  'crowdthreat',
-  'correlations',
-  'gt_risk',
-  'freshness',
-] as const satisfies readonly (keyof DashboardData)[];
+import { useDataSnapshot } from '@/hooks/useDataStore';
 
 // ---------------------------------------------------------------------------
 // ScannerTracker — in-app audio player for tracked police scanner systems
@@ -753,7 +699,7 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
   onOpenSarAoiEditor?: () => void;
   viewBoundsRef?: React.RefObject<{ south: number; west: number; north: number; east: number } | null>;
 }) {
-  const data = useDataKeys(WORLDVIEW_PANEL_DATA_KEYS) as DashboardData;
+  const data = useDataSnapshot() as import('@/types/dashboard').DashboardData;
   const { t } = useTranslation();
   const [internalMinimized, setInternalMinimized] = useState(true);
   const isMinimized = isMinimizedProp !== undefined ? isMinimizedProp : internalMinimized;
@@ -765,10 +711,7 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
   const { theme, toggleTheme, hudColor, cycleHudColor } = useTheme();
   const [gibsPlaying, setGibsPlaying] = useState(false);
   const [potusEnabled, setPotusEnabled] = useState(true);
-  const [meshtasticScanning, setMeshtasticScanning] = useState(false);
-  const [meshtasticScanMessage, setMeshtasticScanMessage] = useState('');
   const gibsIntervalRef = useRef<ReturnType<typeof setInterval> | null>(null);
-  const meshtasticScanPollRef = useRef<ReturnType<typeof setInterval> | null>(null);
 
   // SAR mode chooser — prompts the first time the user enables the SAR
   // layer, remembers the choice, and auto-detects server-side Mode B.
@@ -854,9 +797,7 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
         });
         if (!res.ok || cancelled) return;
         const body = await res.json();
-        const modeBOn = Boolean(
-          body?.products?.fully_configured ?? body?.products?.enabled,
-        );
+        const modeBOn = Boolean(body?.products?.enabled);
         if (cancelled) return;
         if (modeBOn && sarChoice !== 'b_active') {
           try {
@@ -990,67 +931,6 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
     return { meshtasticCount: mesh, aprsCount: aprs };
   }, [data?.sigint, data?.sigint_totals]);
 
-  const stopMeshtasticScanPoll = useCallback(() => {
-    if (meshtasticScanPollRef.current) {
-      clearInterval(meshtasticScanPollRef.current);
-      meshtasticScanPollRef.current = null;
-    }
-  }, []);
-
-  useEffect(() => () => stopMeshtasticScanPoll(), [stopMeshtasticScanPoll]);
-
-  const scanMeshtasticPlanet = useCallback(
-    async (event: React.MouseEvent) => {
-      event.stopPropagation();
-      if (meshtasticScanning) return;
-
-      setMeshtasticScanning(true);
-      setMeshtasticScanMessage('Starting global node scan…');
-      stopMeshtasticScanPoll();
-
-      try {
-        const res = await fetch(`${API_BASE}/api/sigint/meshtastic/scan`, { method: 'POST' });
-        const json = await res.json().catch(() => ({}));
-        if (!res.ok || json.ok === false) {
-          setMeshtasticScanMessage(
-            typeof json.status === 'string' ? json.status : 'Meshtastic scan could not start.',
-          );
-          setMeshtasticScanning(false);
-          return;
-        }
-
-        setMeshtasticScanMessage('Scanning planet (~90s)…');
-        let polls = 0;
-        meshtasticScanPollRef.current = setInterval(async () => {
-          polls += 1;
-          try {
-            const statusRes = await fetch(`${API_BASE}/api/sigint/meshtastic/status`);
-            if (!statusRes.ok) return;
-            const status = await statusRes.json();
-            if (!status.scan_in_progress) {
-              stopMeshtasticScanPoll();
-              setMeshtasticScanning(false);
-              const count = Number(status.node_count || 0);
-              setMeshtasticScanMessage(
-                count > 0 ? `Scan complete — ${count.toLocaleString()} nodes on map.` : 'Scan complete.',
-              );
-            } else if (polls >= 40) {
-              stopMeshtasticScanPoll();
-              setMeshtasticScanning(false);
-              setMeshtasticScanMessage('Scan still running in background. Count will update when finished.');
-            }
-          } catch {
-            // keep polling until timeout
-          }
-        }, 3000);
-      } catch {
-        setMeshtasticScanning(false);
-        setMeshtasticScanMessage('Meshtastic scan request failed.');
-      }
-    },
-    [meshtasticScanning, stopMeshtasticScanPoll],
-  );
-
   const cctvCount = Number(data?.cctv_total || data?.cctv?.length || 0);
   const satnogsCount = Number(data?.satnogs_total || data?.satnogs_stations?.length || 0);
   const tinygsCount = Number(data?.tinygs_total || data?.tinygs_satellites?.length || 0);
@@ -1991,31 +1871,6 @@ const WorldviewLeftPanel = React.memo(function WorldviewLeftPanel({
                                     )}
                                   </div>
                                 </div>
-                                {active && layer.id === 'sigint_meshtastic' && (
-                                  <div
-                                    className="ml-7 mt-2 flex flex-col gap-1.5"
-                                    onClick={(e) => e.stopPropagation()}
-                                  >
-                                    <button
-                                      type="button"
-                                      onClick={scanMeshtasticPlanet}
-                                      disabled={meshtasticScanning}
-                                      className="inline-flex items-center gap-1.5 self-start px-2 py-1 text-[10px] font-mono tracking-wider border border-cyan-500/40 text-cyan-400 hover:bg-cyan-950/30 disabled:opacity-60 disabled:cursor-not-allowed transition-colors"
-                                      title="Re-fetch all Meshtastic node positions from the global map API"
-                                    >
-                                      <RefreshCw
-                                        size={11}
-                                        className={meshtasticScanning ? 'animate-spin' : ''}
-                                      />
-                                      {meshtasticScanning ? 'SCANNING…' : 'SCAN PLANET'}
-                                    </button>
-                                    {meshtasticScanMessage ? (
-                                      <span className="text-[10px] font-mono text-cyan-500/80 leading-snug">
-                                        {meshtasticScanMessage}
-                                      </span>
-                                    ) : null}
-                                  </div>
-                                )}
                                 {/* GIBS Imagery inline controls */}
                                 {active &&
                                   layer.id === 'gibs_imagery' &&

frontend/src/hooks/useDataPolling.ts

@@ -295,26 +295,13 @@ export function useDataPolling() {
       }, VIEWPORT_FAST_REFETCH_DEBOUNCE_MS);
     };
 
-    // When a layer toggle fires, refetch live tiers immediately and retry a few
-    // times so network-heavy on-enable fetches (FIRMS, PSK, …) can finish in the
-    // background without blocking POST /api/layers on the single API worker.
+    // When a layer toggle fires, immediately refetch slow data so the user
+    // doesn't wait up to 120s for power plants / GDELT / etc. to appear.
     const onLayerToggle = () => {
-      slowEtag.current = null;
-      fastEtag.current = null;
+      slowEtag.current = null;           // invalidate ETag → guarantees fresh payload
       if (slowTimerId) clearTimeout(slowTimerId);
       slowTimerId = null;
-      void fetchFastData();
-      void fetchSlowData();
-      const retryDelaysMs = [1000, 2500, 5000];
-      for (const delay of retryDelaysMs) {
-        setTimeout(() => {
-          if (_pollingPaused) return;
-          slowEtag.current = null;
-          fastEtag.current = null;
-          void fetchSlowData();
-          void fetchFastData();
-        }, delay);
-      }
+      fetchSlowData();
     };
     window.addEventListener(LAYER_TOGGLE_EVENT, onLayerToggle);
     window.addEventListener(VIEWPORT_COMMITTED_EVENT, queueViewportFastRefetch);

frontend/src/lib/agentShellWs.ts

@@ -1,21 +1,4 @@
-export async function mintAgentShellWsToken(): Promise<string | null> {
-  if (typeof window === 'undefined') return null;
-  try {
-    const res = await fetch('/api/agent-shell/ws-token', {
-      method: 'POST',
-      credentials: 'same-origin',
-      cache: 'no-store',
-    });
-    if (!res.ok) return null;
-    const body = (await res.json()) as { token?: string };
-    const token = String(body?.token || '').trim();
-    return token || null;
-  } catch {
-    return null;
-  }
-}
-
-export function resolveAgentShellWsUrl(cwd?: string, wsToken?: string): string {
+export function resolveAgentShellWsUrl(cwd?: string): string {
   if (typeof window === 'undefined') return '';
   const protocol = window.location.protocol === 'https:' ? 'wss' : 'ws';
   const host = window.location.hostname || '127.0.0.1';
@@ -28,8 +11,6 @@ export function resolveAgentShellWsUrl(cwd?: string, wsToken?: string): string {
   const params = new URLSearchParams();
   const trimmed = String(cwd || '').trim();
   if (trimmed) params.set('cwd', trimmed);
-  const token = String(wsToken || '').trim();
-  if (token) params.set('ws_token', token);
   const query = params.toString();
   return `${protocol}://${host}:${port}/api/agent-shell/ws${query ? `?${query}` : ''}`;
 }

frontend/src/lib/trackedData.ts

@@ -3,10 +3,8 @@ export const trackedCategories: string[] = [
   'Celebrity',
   'Formula 1',
   'Government',
-  'Oligarch',
   'Other',
   'People',
-  'Royal',
   'Sports',
   'State/Law',
   'Test Aircraft',

scripts/apply_plane_alert_to_tracked_names.py

@@ -1,237 +0,0 @@
-#!/usr/bin/env python3
-"""Merge curated plane-alert-db rows into backend/data/tracked_names.json.
-
-Only real people, companies, and organizations — never plane-alert joke tags
-(The Gambler, Genomes, Aaaaaaaand its gone, etc.).
-"""
-from __future__ import annotations
-
-import csv
-import json
-import re
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parents[1]
-SB_PATH = ROOT / "backend" / "data" / "tracked_names.json"
-PAD = Path.home() / "Downloads" / "plane-alert-db-main" / "plane-alert-db-main"
-
-STRICT_CATS = {
-    "Don't you know who I am?",
-    "Oligarch",
-    "Royal Aircraft",
-    "Football",
-}
-
-GENERIC_TAGS = {
-    "bizjet", "bizjets", "pusher prop", "man made climate change", "government",
-    "royalty", "pga", "nfl", "nba", "basketball", "war eagle", "volunteers",
-    "original nuttah", "jumpers for goalposts", "money money money", "safe return",
-    "do a barrel roll", "biplane", "aerospace", "medical", "defense",
-    "the gambler", "the house always wins", "house always wins", "snake eyes",
-    "bunch of bankers", "scrooge mcduck", "aaaaaaaand its gone", "aaaaaaand its gone",
-    "genomes", "football", "zoomies", "you can't see me", "too much money",
-    "venture capital", "honda jet", "basic cable", "as seen on tv", "joe cool",
-}
-
-COMPANY_HINTS = re.compile(
-    r"\b(inc|llc|ltd|corp|company|co\.|bank|group|holdings|international|"
-    r"university|airlines|aviation|systems|foundation|tribe|resorts|casino|"
-    r"palace|entertainment|insurance|credit union|banco|sa|ag|gmbh|plc)\b",
-    re.I,
-)
-
-MERGE_ALIASES: dict[str, str] = {
-    "falcon landing llc": "Elon Musk",
-    "christian ronaldo": "Cristiano Ronaldo",
-    "elon musk": "Elon Musk",
-    "marc benioff": "Mark Benioff",
-    "p. diddy": "P. Diddy",
-    "baller": "P. Diddy",
-    "empire state of mind": "Jay Z",
-    "judy sheindlin": "Judge Judy",
-    "doge": "Vivek Ramaswamy",
-    "a&m records": "Jerry Moss",
-    "wings of grace": "Folorunso Alakija",
-    "reliance commercial dealers ltd": "Mukesh Ambani",
-    "monaco royal family": "Monaco Royal Family",
-    "the royal squadron": "UK Royal Family (RAF)",
-    "the kings helicopter flight": "UK Royal Family (RAF)",
-}
-
-
-def norm_reg(s: str) -> str:
-    return (s or "").strip().upper()
-
-
-def row_get(row: dict[str, str], *keys: str) -> str:
-    for key in keys:
-        if row.get(key):
-            return str(row[key]).strip()
-    return ""
-
-
-def sb_category(cat: str, display: str, operator: str) -> str:
-    if cat == "Oligarch":
-        return "Oligarch"
-    if cat in {"Royal Aircraft"} or "royal" in display.lower():
-        return "Royal"
-    if cat == "Football":
-        return "Sports"
-    if COMPANY_HINTS.search(operator) or COMPANY_HINTS.search(display):
-        return "Business"
-    return "Celebrity"
-
-
-def is_likely_person_name(text: str) -> bool:
-    t = text.strip()
-    if not t or t.lower() in GENERIC_TAGS:
-        return False
-    if any(ch in t for ch in "?!"):
-        return False
-    if COMPANY_HINTS.search(t):
-        return False
-    words = t.split()
-    if len(words) < 2 or len(words) > 5:
-        return False
-    # Require each word to look name-like (Title case or Mc/Mac/O').
-    for w in words:
-        if not re.match(r"^[A-Z][\w'.-]*$|^(Mc|Mac|O')[A-Z]", w):
-            return False
-    return True
-
-
-def pick_display_name(operator: str, tag1: str, tag2: str, tag3: str, cat: str) -> str | None:
-    op_key = operator.strip().lower()
-    if op_key in MERGE_ALIASES:
-        return MERGE_ALIASES[op_key]
-
-    op = operator.strip()
-    if cat == "Football":
-        return op or None
-
-    if cat == "Royal Aircraft":
-        return op or None
-
-    if cat == "Oligarch":
-        if is_likely_person_name(op):
-            return op
-        for tag in (tag2, tag3, tag1):
-            if is_likely_person_name(tag):
-                return tag.strip()
-        return op or None
-
-    if cat == "Don't you know who I am?":
-        if is_likely_person_name(op):
-            return op
-        for tag in (tag2, tag3, tag1):
-            if is_likely_person_name(tag):
-                return tag.strip()
-        if op and not op.lower() in GENERIC_TAGS:
-            return op
-        return None
-
-    return None
-
-
-def load_rows() -> list[dict[str, str]]:
-    rows: list[dict[str, str]] = []
-    for fname in ("plane-alert-db.csv", "plane-alert-civ.csv"):
-        path = PAD / fname
-        if not path.exists():
-            continue
-        with path.open(encoding="utf-8", errors="replace") as f:
-            rows.extend(list(csv.DictReader(f)))
-    return rows
-
-
-def main() -> None:
-    with SB_PATH.open(encoding="utf-8") as f:
-        sb = json.load(f)
-
-    details: dict = sb.setdefault("details", {})
-    names_list: list[dict[str, str]] = sb.setdefault("names", [])
-    existing_names = {n["name"] for n in names_list}
-
-    sb_regs: set[str] = set()
-    for info in details.values():
-        for reg in info.get("registrations", []):
-            r = norm_reg(reg)
-            if r:
-                sb_regs.add(r)
-
-    added_entries = 0
-    added_regs = 0
-    merged_regs = 0
-
-    seen: set[tuple[str, str]] = set()
-
-    for row in load_rows():
-        cat = row_get(row, "Category")
-        if cat not in STRICT_CATS:
-            continue
-
-        reg = norm_reg(row_get(row, "$Registration", "Registration"))
-        if not reg or (reg, cat) in seen:
-            continue
-        seen.add((reg, cat))
-
-        operator = row_get(row, "$Operator", "Operator")
-        tag1 = row_get(row, "$Tag 1", "Tag 1")
-        tag2 = row_get(row, "#Tag 2", "$#Tag 2")
-        tag3 = row_get(row, "#Tag 3", "$#Tag 3")
-
-        display = pick_display_name(operator, tag1, tag2, tag3, cat)
-        if not display or reg in sb_regs:
-            continue
-
-        category = sb_category(cat, display, operator)
-
-        if display in details:
-            regs = details[display].setdefault("registrations", [])
-            if reg not in regs:
-                regs.append(reg)
-                merged_regs += 1
-                sb_regs.add(reg)
-            continue
-
-        details[display] = {
-            "category": category,
-            "registrations": [reg],
-        }
-        if display not in existing_names:
-            names_list.append({"name": display, "category": category})
-            existing_names.add(display)
-            added_entries += 1
-        added_regs += 1
-        sb_regs.add(reg)
-
-    uk_key = "UK Royal Family (RAF)"
-    uk_regs = ["G-XWBG", "GZ-100", "ZE700", "ZE701", "ZE707", "ZE708", "G-XXEC"]
-    if uk_key in details:
-        details[uk_key]["category"] = "Royal"
-        regs = details[uk_key].setdefault("registrations", [])
-        for r in uk_regs:
-            if r not in regs:
-                regs.append(r)
-                merged_regs += 1
-    else:
-        details[uk_key] = {"category": "Royal", "registrations": uk_regs}
-        if uk_key not in existing_names:
-            names_list.append({"name": uk_key, "category": "Royal"})
-            added_entries += 1
-
-    names_list.sort(key=lambda x: x["name"].lower())
-
-    with SB_PATH.open("w", encoding="utf-8") as f:
-        json.dump(sb, f, indent=2, ensure_ascii=False)
-        f.write("\n")
-
-    print(f"New tracked entries: {added_entries}")
-    print(f"New registrations: {added_regs}")
-    print(f"Merged into existing: {merged_regs}")
-    print(f"Total details entries: {len(details)}")
-    print(f"Total registrations: {sum(len(v.get('registrations',[])) for v in details.values())}")
-
-
-if __name__ == "__main__":
-    main()

scripts/bench_layer_toggle_latency.py

@@ -1,202 +0,0 @@
-#!/usr/bin/env python3
-"""Measure layer-toggle → data-visible latency (UX guardrail for perf work).
-
-Simulates what the dashboard does on toggle:
-  1. POST /api/layers (layer off → on)
-  2. Poll GET /api/live-data/slow until the layer's payload is non-empty
-
-Also reports whether data was already warm in the backend store before toggle
-(via /api/health source counts while the layer is still filtered off in the API).
-
-Usage:
-  python scripts/bench_layer_toggle_latency.py
-  python scripts/bench_layer_toggle_latency.py --base http://127.0.0.1:8000
-"""
-from __future__ import annotations
-
-import argparse
-import json
-import sys
-import time
-import urllib.error
-import urllib.request
-from dataclasses import dataclass
-from typing import Any, Callable
-
-# layer_key → JSON field in /api/live-data/slow + how to count "visible"
-LAYER_PROBE: dict[str, tuple[str, Callable[[Any], int]]] = {
-    "cctv": ("cctv", lambda _v: 0),  # fast tier — see FAST_LAYER_PROBE
-    "firms": ("firms_fires", lambda v: len(v) if isinstance(v, list) else 0),
-    "datacenters": ("datacenters", lambda v: len(v) if isinstance(v, list) else 0),
-    "power_plants": ("power_plants", lambda v: len(v) if isinstance(v, list) else 0),
-    "psk_reporter": ("psk_reporter", lambda v: len(v) if isinstance(v, list) else 0),
-}
-
-FAST_LAYER_PROBE = {
-    "cctv": ("cctv", lambda v: len(v) if isinstance(v, list) else 0),
-}
-
-HEALTH_SOURCE_KEY = {
-    "cctv": "cctv",
-    "firms": "firms_fires",
-    "datacenters": "datacenters",
-    "power_plants": "power_plants",
-    "psk_reporter": "psk_reporter",
-}
-
-
-@dataclass
-class ToggleResult:
-    layer: str
-    warm_store_count: int | None
-    time_to_visible_ms: float | None
-    visible_count: int
-    timed_out: bool
-    on_enable_fetch: bool
-    notes: str
-
-
-def _request(method: str, url: str, body: dict | None = None, timeout: float = 30.0) -> tuple[int, Any]:
-    data = None
-    headers = {"Accept": "application/json"}
-    if body is not None:
-        data = json.dumps(body).encode()
-        headers["Content-Type"] = "application/json"
-    req = urllib.request.Request(url, data=data, headers=headers, method=method)
-    with urllib.request.urlopen(req, timeout=timeout) as resp:
-        raw = resp.read()
-        return resp.status, json.loads(raw) if raw else None
-
-
-def get_health(base: str) -> dict:
-    _, payload = _request("GET", f"{base}/api/health")
-    return payload or {}
-
-
-def get_slow(base: str) -> dict:
-    _, payload = _request("GET", f"{base}/api/live-data/slow")
-    return payload or {}
-
-
-def get_fast(base: str) -> dict:
-    _, payload = _request("GET", f"{base}/api/live-data/fast")
-    return payload or {}
-
-
-def set_layer(base: str, layers: dict[str, bool]) -> None:
-    _request("POST", f"{base}/api/layers", {"layers": layers})
-
-
-def count_visible(payload: dict, field: str, counter: Callable[[Any], int]) -> int:
-    return counter(payload.get(field))
-
-
-ON_ENABLE_IMMEDIATE = {"datacenters", "fishing_activity"}
-
-
-def measure_layer(base: str, layer: str, timeout_s: float = 120.0) -> ToggleResult:
-    health = get_health(base)
-    warm = None
-    hk = HEALTH_SOURCE_KEY.get(layer)
-    if hk and isinstance(health.get("sources"), dict):
-        warm = health["sources"].get(hk)
-
-    # Ensure layer is off (frontend default for these probes)
-    set_layer(base, {layer: False})
-    time.sleep(0.25)
-
-    # Confirm API filters it off while toggled off
-    if layer in FAST_LAYER_PROBE:
-        field, counter = FAST_LAYER_PROBE[layer]
-        off_payload = get_fast(base)
-    else:
-        field, counter = LAYER_PROBE[layer]
-        off_payload = get_slow(base)
-    off_count = count_visible(off_payload, field, counter)
-
-    # Toggle on — mirrors dashboard POST + immediate slow/fast refetch
-    t0 = time.perf_counter()
-    set_layer(base, {layer: True})
-
-    visible_count = 0
-    timed_out = True
-    while (time.perf_counter() - t0) < timeout_s:
-        if layer in FAST_LAYER_PROBE:
-            payload = get_fast(base)
-        else:
-            payload = get_slow(base)
-        visible_count = count_visible(payload, field, counter)
-        if visible_count > 0:
-            timed_out = False
-            break
-        time.sleep(0.25)
-
-    elapsed_ms = None if timed_out else (time.perf_counter() - t0) * 1000.0
-
-    notes_parts = []
-    if off_count > 0:
-        notes_parts.append(f"unexpected visible while off ({off_count})")
-    if warm and warm > 0 and (timed_out or (elapsed_ms or 0) < 500):
-        notes_parts.append("warm store — toggle likely instant from prefetch")
-    elif warm == 0 and timed_out:
-        notes_parts.append("cold store — would feel broken to user")
-    elif timed_out:
-        notes_parts.append(f"no warm store signal; waited {timeout_s:.0f}s")
-
-    return ToggleResult(
-        layer=layer,
-        warm_store_count=warm,
-        time_to_visible_ms=elapsed_ms,
-        visible_count=visible_count,
-        timed_out=timed_out,
-        on_enable_fetch=layer in ON_ENABLE_IMMEDIATE,
-        notes="; ".join(notes_parts) or "ok",
-    )
-
-
-def main() -> int:
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--base", default="http://127.0.0.1:8000")
-    parser.add_argument("--timeout", type=float, default=120.0)
-    parser.add_argument("--layers", nargs="*", default=list(LAYER_PROBE.keys()))
-    args = parser.parse_args()
-
-    print(f"Backend: {args.base}")
-    try:
-        health = get_health(args.base)
-    except urllib.error.URLError as exc:
-        print(f"Health check failed: {exc}", file=sys.stderr)
-        return 1
-
-    print(f"Version: {health.get('version')}  uptime: {health.get('uptime_seconds')}s")
-    print(f"Runtime profile: {(health.get('runtime') or {}).get('profile')}")
-    print()
-    print(f"{'layer':<14} {'warm_store':>10} {'visible_ms':>11} {'count':>8} {'on_enable':>10}  notes")
-    print("-" * 90)
-
-    results: list[ToggleResult] = []
-    for layer in args.layers:
-        try:
-            r = measure_layer(args.base, layer, timeout_s=args.timeout)
-        except urllib.error.URLError as exc:
-            print(f"{layer:<14} ERROR: {exc}")
-            continue
-        results.append(r)
-        ms = f"{r.time_to_visible_ms:.0f}" if r.time_to_visible_ms is not None else f">{args.timeout:.0f}s"
-        warm = str(r.warm_store_count) if r.warm_store_count is not None else "?"
-        on_en = "yes" if r.on_enable_fetch else "no"
-        print(f"{layer:<14} {warm:>10} {ms:>11} {r.visible_count:>8} {on_en:>10}  {r.notes}")
-
-    print()
-    instant = [r for r in results if r.time_to_visible_ms is not None and r.time_to_visible_ms < 500]
-    slow = [r for r in results if r.timed_out or (r.time_to_visible_ms or 0) >= 5000]
-    print(f"Summary: {len(instant)}/{len(results)} toggles visible in <500ms; {len(slow)} slow or timed out")
-    if slow:
-        print("Layers that need on-enable fetch or prefetch to avoid UX pain:")
-        for r in slow:
-            print(f"  - {r.layer}: {r.notes}")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scripts/compare_plane_alert_db.py

@@ -1,143 +0,0 @@
-#!/usr/bin/env python3
-"""Compare plane-alert-db CSVs to Shadowbroker tracked_names.json."""
-from __future__ import annotations
-
-import csv
-import json
-import re
-from pathlib import Path
-
-SB = Path(__file__).resolve().parents[1] / "backend" / "data" / "tracked_names.json"
-PAD = Path.home() / "Downloads" / "plane-alert-db-main" / "plane-alert-db-main"
-
-CELEB_CATS = {
-    "Don't you know who I am?",
-    "As Seen on TV",
-    "Joe Cool",
-    "Vanity Plate",
-    "Football",
-    "Head of State",
-    "Royal Aircraft",
-    "Oligarch",
-    "Bizjets",
-}
-
-PURE_CELEB_CATS = {
-    "Don't you know who I am?",
-    "As Seen on TV",
-    "Joe Cool",
-    "Vanity Plate",
-    "Football",
-}
-
-
-def norm_name(s: str) -> str:
-    return re.sub(r"\s+", " ", (s or "").strip().lower())
-
-
-def load_csv(path: Path) -> list[dict[str, str]]:
-    rows: list[dict[str, str]] = []
-    with path.open(encoding="utf-8", errors="replace") as f:
-        for row in csv.DictReader(f):
-            rows.append(row)
-    return rows
-
-
-def row_field(row: dict[str, str], *keys: str) -> str:
-    for key in keys:
-        if row.get(key):
-            return str(row[key]).strip()
-    return ""
-
-
-def main() -> None:
-    with SB.open(encoding="utf-8") as f:
-        sb = json.load(f)
-
-    sb_regs: set[str] = set()
-    sb_names: set[str] = set()
-    for name, info in sb.get("details", {}).items():
-        sb_names.add(norm_name(name))
-        for reg in info.get("registrations", []):
-            r = reg.strip().upper()
-            if r:
-                sb_regs.add(r)
-
-    rows: list[dict[str, str]] = []
-    for fname in ("plane-alert-db.csv", "plane-alert-civ.csv"):
-        path = PAD / fname
-        if path.exists():
-            rows.extend(load_csv(path))
-
-    seen: set[tuple[str, str, str]] = set()
-    new_by_cat: dict[str, list[dict[str, str]]] = {}
-
-    for row in rows:
-        cat = row_field(row, "Category")
-        if cat not in CELEB_CATS:
-            continue
-        reg = row_field(row, "$Registration", "Registration").upper()
-        op = row_field(row, "$Operator", "Operator")
-        icao = row_field(row, "$ICAO", "ICAO").upper()
-        if not reg and not op:
-            continue
-        key = (reg, norm_name(op), cat)
-        if key in seen:
-            continue
-        seen.add(key)
-
-        in_sb = False
-        if reg and reg in sb_regs:
-            in_sb = True
-        if norm_name(op) in sb_names:
-            in_sb = True
-        if not in_sb and op:
-            opn = norm_name(op)
-            for sn in sb_names:
-                if len(sn) >= 6 and (sn in opn or opn in sn):
-                    in_sb = True
-                    break
-
-        if in_sb:
-            continue
-
-        entry = {
-            "registration": reg,
-            "operator": op,
-            "category": cat,
-            "type": row_field(row, "$Type", "Type"),
-            "icao": icao,
-            "tag1": row_field(row, "$Tag 1", "Tag 1"),
-        }
-        new_by_cat.setdefault(cat, []).append(entry)
-
-    print("=== Shadowbroker tracked ===")
-    print(f"  names in details: {len(sb_names)}")
-    print(f"  registrations: {len(sb_regs)}")
-    print()
-    print("=== NEW celebrity/VIP-ish entries (not in Shadowbroker) ===")
-
-    total = 0
-    for cat in sorted(new_by_cat, key=lambda c: -len(new_by_cat[c])):
-        items = new_by_cat[cat]
-        total += len(items)
-        print(f"\n## {cat} ({len(items)})")
-        for e in sorted(items, key=lambda x: x["operator"])[:30]:
-            reg = e["registration"] or "(no reg)"
-            tag = f" | {e['tag1']}" if e["tag1"] else ""
-            print(f"  {reg:12} {e['operator'][:60]}{tag}")
-        if len(items) > 30:
-            print(f"  ... and {len(items) - 30} more")
-
-    print(f"\n=== TOTAL NEW (all VIP categories): {total} ===")
-
-    pure_items = [e for c in PURE_CELEB_CATS for e in new_by_cat.get(c, [])]
-    print(f"\n=== HIGH-SIGNAL CELEB / NOTABLE ({len(pure_items)}) ===")
-    for e in sorted(pure_items, key=lambda x: (x["category"], x["operator"])):
-        reg = e["registration"] or "????"
-        tag = f" ({e['tag1']})" if e["tag1"] else ""
-        print(f"[{e['category']}] {reg} — {e['operator']}{tag}")
-
-
-if __name__ == "__main__":
-    main()

scripts/extract_plane_alert_additions.py

@@ -1,223 +0,0 @@
-#!/usr/bin/env python3
-"""Extract plane-alert-db entries missing from tracked_names.json."""
-from __future__ import annotations
-
-import csv
-import json
-import re
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parents[1]
-SB = ROOT / "backend" / "data" / "tracked_names.json"
-PAD = Path.home() / "Downloads" / "plane-alert-db-main" / "plane-alert-db-main"
-
-# Categories to import into tracked_names
-IMPORT_CATS = {
-    "Don't you know who I am?",
-    "Oligarch",
-    "Royal Aircraft",
-    "Football",
-    "Head of State",
-    "Dictator Alert",
-}
-
-# As Seen on TV / Bizjets only when operator looks like a person (heuristic)
-PERSON_CATS = {"As Seen on TV", "Bizjets", "Vanity Plate"}
-
-# Skip obvious corps / generic operators
-CORP_RE = re.compile(
-    r"\b(inc|llc|ltd|corp|company|co\.|group|holdings|university|air force|"
-    r"airlines|aviation|services|systems|international|global|partners|"
-    r"foundation|bank|pharma|laboratories|transportation|motors|enterprises)\b",
-    re.I,
-)
-
-CELEB_HINTS = re.compile(
-    r"\b(actor|actress|singer|rapper|musician|celebrity|nfl|nba|f1|formula|"
-    r"royal|prince|princess|king|queen|duke|sheik|sultan|oligarch|billionaire|"
-    r"mogul|tycoon|founder|ceo|president|senator|governor|judge|athlete|"
-    r"footballer|golfer|tennis|director|producer|host|comedian|model|"
-    r"influencer|youtuber|podcast|chef|author|writer|artist|designer)\b",
-    re.I,
-)
-
-KNOWN_PERSON_NAMES = {
-    "elon musk", "jay z", "jay-z", "kanye", "west", "kim kardashian", "taylor swift",
-    "beyonce", "drake", "rihanna", "oprah", "gates", "bezos", "zuckerberg",
-    "buffett", "dalio", "icahn", "ackman", "soros", "thiel", "musk", "cruise",
-    "dicaprio", "pitt", "jolie", "clooney", "hanks", "spielberg", "lucas",
-    "branson", "trump", "biden", "obama", "clinton", "bush", "romney",
-    "ramaswamy", "benioff", "blavatnik", "abramovich", "abramov", "potanin",
-    "fridman", "deripaska", "kerimov", "tinkov", "mordashov", "rybolovlev",
-    "lisin", "vekselberg", "medvedchuk", "alekperov", "mikhelson", "diddy",
-    "combs", "sean combs", "ronaldo", "messi", "mbappe", "beckham", "jordan",
-    "lebron", "brady", "mahomes", "kroenke", "kraft", "jones", "snyder",
-    "sheindlin", "judge judy", "elton john", "moss", "ambani", "adani",
-    "lowry", "ecclestone", "hamilton", "verstappen", "schumacher", "woods",
-    "nicklaus", "federer", "nadal", "djokovic", "osaka", "williams", "serena",
-    "venus", "sharapova", "mcgregor", "mayweather", "paul", "logan paul",
-    "jake paul", "mrbeast", "pewdiepie", "charlie munger", "larry ellison",
-    "michael dell", "tim cook", "satya nadella", "sundar pichai", "jensen huang",
-    "gisele", "tom brady", "gwyneth", "howard stern", "howard marks",
-    "steven cohen", "ken griffin", "david tepper", "ray dalio", "peter thiel",
-    "paul allen", "steve ballmer", "mark cuban", "richard branson", "larry page",
-    "sergey brin", "eric schmidt", "reid hoffman", "marc andreessen",
-    "chamath", "naval", "andretti", "penske", "hendrick", "rick hendrick",
-}
-
-
-def norm_reg(s: str) -> str:
-    return (s or "").strip().upper()
-
-
-def norm_name(s: str) -> str:
-    return re.sub(r"\s+", " ", (s or "").strip())
-
-
-def looks_like_person(operator: str, tag1: str, tag2: str, tag3: str) -> bool:
-    blob = " ".join([operator, tag1, tag2, tag3]).strip()
-    if not blob or len(blob) < 3:
-        return False
-    low = blob.lower()
-    if CORP_RE.search(low) and not any(h in low for h in KNOWN_PERSON_NAMES):
-        # allow "Falcon Landing LLC" when tag says Elon Musk
-        if not any(h in low for h in KNOWN_PERSON_NAMES):
-            return False
-    if any(h in low for h in KNOWN_PERSON_NAMES):
-        return True
-    if CELEB_HINTS.search(low):
-        return True
-    # Two+ capitalized words, no corp suffix — weak person signal
-    words = operator.split()
-    if 2 <= len(words) <= 4 and operator == operator.title() and not CORP_RE.search(low):
-        return True
-    return False
-
-
-def sb_category_for(cat: str, operator: str) -> str:
-    low = operator.lower()
-    if cat in {"Oligarch", "Dictator Alert"}:
-        return "Oligarch"
-    if cat == "Royal Aircraft" or "royal" in low:
-        return "Royal"
-    if cat == "Football":
-        return "Sports"
-    if cat in {"Head of State"}:
-        return "Government"
-    if any(x in low for x in ("nfl", "nba", "mlb", "football", "basketball", "soccer", "f1", "formula")):
-        return "Sports"
-    return "Celebrity"
-
-
-def row_get(row: dict[str, str], *keys: str) -> str:
-    for k in keys:
-        if row.get(k):
-            return str(row[k]).strip()
-    return ""
-
-
-def main() -> None:
-    with SB.open(encoding="utf-8") as f:
-        sb = json.load(f)
-
-    sb_regs: set[str] = set()
-    sb_names: dict[str, str] = {}
-    for name, info in sb.get("details", {}).items():
-        for reg in info.get("registrations", []):
-            r = norm_reg(reg)
-            if r:
-                sb_regs.add(r)
-                sb_names[r] = name
-
-    additions: dict[str, dict] = {}
-    merge: dict[str, list[str]] = {}
-
-    csv_paths = [
-        PAD / "plane-alert-db.csv",
-        PAD / "plane-alert-civ.csv",
-        PAD / "plane-alert-gov.csv",
-        PAD / "plane-alert-mil.csv",
-    ]
-
-    seen: set[tuple[str, str]] = set()
-    person_hits = 0
-
-    for path in csv_paths:
-        if not path.exists():
-            continue
-        with path.open(encoding="utf-8", errors="replace") as f:
-            for row in csv.DictReader(f):
-                cat = row_get(row, "Category")
-                reg = norm_reg(row_get(row, "$Registration", "Registration"))
-                op = norm_reg(row_get(row, "$Operator", "Operator"))
-                op_display = norm_name(row_get(row, "$Operator", "Operator"))
-                tag1 = row_get(row, "$Tag 1", "Tag 1")
-                tag2 = row_get(row, "#Tag 2", "$#Tag 2")
-                tag3 = row_get(row, "#Tag 3", "$#Tag 3")
-
-                if not reg:
-                    continue
-                if (reg, cat) in seen:
-                    continue
-                seen.add((reg, cat))
-
-                include = cat in IMPORT_CATS
-                if not include and cat in PERSON_CATS:
-                    if looks_like_person(op_display, tag1, tag2, tag3):
-                        include = True
-                        person_hits += 1
-
-                if not include:
-                    continue
-                if reg in sb_regs:
-                    continue
-
-                # Prefer tag person name over shell company
-                display = op_display
-                for tag in (tag1, tag2, tag3):
-                    if tag and any(h in tag.lower() for h in KNOWN_PERSON_NAMES):
-                        display = tag
-                        break
-                    if tag and len(tag.split()) <= 4 and tag[0].isupper() and "llc" not in tag.lower():
-                        if cat == "Don't you know who I am?" and tag not in {"Bizjet", "Pusher Prop"}:
-                            display = tag
-
-                key = display
-                if key in sb.get("details", {}):
-                    merge.setdefault(key, []).append(reg)
-                else:
-                    entry = additions.setdefault(
-                        key,
-                        {"category": sb_category_for(cat, display), "registrations": []},
-                    )
-                    if reg not in entry["registrations"]:
-                        entry["registrations"].append(reg)
-
-    print(f"New named entries: {len(additions)}")
-    print(f"Merge into existing: {len(merge)}")
-    print(f"Person-heuristic hits (ASTV/Bizjets): {person_hits}")
-    print()
-
-    by_cat: dict[str, list[tuple[str, list[str]]]] = {}
-    for name, info in sorted(additions.items()):
-        by_cat.setdefault(info["category"], []).append((name, info["registrations"]))
-
-    for cat in sorted(by_cat):
-        items = by_cat[cat]
-        print(f"## {cat} ({len(items)})")
-        for name, regs in items[:40]:
-            print(f"  {name}: {', '.join(regs)}")
-        if len(items) > 40:
-            print(f"  ... +{len(items)-40} more")
-        print()
-
-    out = ROOT / "scripts" / "plane_alert_additions.json"
-    out.write_text(
-        json.dumps({"additions": additions, "merge": merge}, indent=2),
-        encoding="utf-8",
-    )
-    print(f"Wrote {out}")
-
-
-if __name__ == "__main__":
-    main()

scripts/finalize_plane_alert_import.py

@@ -1,190 +0,0 @@
-#!/usr/bin/env python3
-"""Sync plane_alert_db.json from upstream CSV and add explicit celeb/royal tails.
-
-Does NOT import plane-alert joke tags into tracked_names.json.
-"""
-from __future__ import annotations
-
-import csv
-import json
-import subprocess
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parents[1]
-SB_PATH = ROOT / "backend" / "data" / "tracked_names.json"
-PADB_PATH = ROOT / "backend" / "data" / "plane_alert_db.json"
-PAD = Path.home() / "Downloads" / "plane-alert-db-main" / "plane-alert-db-main"
-
-MANUAL_TRACKED: list[tuple[str, str, list[str]]] = [
-    ("Michael Dell", "Celebrity", ["N28ZD"]),
-    ("Lady Moura", "Celebrity", ["VP-CNR"]),
-    ("Lewis Hamilton", "Celebrity", ["G-OFOM"]),
-    ("Mario Andretti", "Celebrity", ["N500MA"]),
-    ("Frank Lowry", "Celebrity", ["N613LF"]),
-    ("Mukesh Ambani", "Celebrity", ["VT-AKV"]),
-    ("Judge Judy", "Celebrity", ["N555QB"]),
-    ("Monaco Royal Family", "Royal", ["3A-MGA"]),
-    ("PGA Tour", "Sports", ["N795HG"]),
-]
-
-
-def norm_reg(s: str) -> str:
-    return (s or "").strip().upper()
-
-
-def norm_icao(s: str) -> str:
-    return (s or "").strip().upper()
-
-
-def row_get(row: dict[str, str], *keys: str) -> str:
-    for key in keys:
-        if row.get(key):
-            return str(row[key]).strip()
-    return ""
-
-
-def load_git_baseline() -> set[str]:
-    raw = subprocess.check_output(
-        ["git", "-C", str(ROOT), "show", "HEAD:backend/data/tracked_names.json"],
-    )
-    data = json.loads(raw)
-    return set(data.get("details", {}).keys())
-
-
-def wiki_from_link(link: str) -> str:
-    if not link:
-        return ""
-    if "wikipedia.org/wiki/" in link:
-        return link.rsplit("/wiki/", 1)[-1].split("#")[0]
-    return ""
-
-
-def steal_reg(details: dict, reg: str, protect_names: set[str]) -> None:
-    reg = norm_reg(reg)
-    for name, info in list(details.items()):
-        if name in protect_names:
-            continue
-        regs = info.get("registrations", [])
-        kept = [r for r in regs if norm_reg(r) != reg]
-        if len(kept) != len(regs):
-            if kept:
-                info["registrations"] = kept
-            else:
-                del details[name]
-
-
-def ensure_entry(
-    details: dict,
-    names_list: list,
-    name: str,
-    category: str,
-    reg: str,
-    *,
-    protect_names: set[str] | None = None,
-) -> bool:
-    reg = norm_reg(reg)
-    if not reg:
-        return False
-    steal_reg(details, reg, protect_names or set())
-    entry = details.setdefault(name, {"category": category, "registrations": []})
-    entry["category"] = category
-    if reg not in {norm_reg(r) for r in entry["registrations"]}:
-        entry["registrations"].append(reg)
-    if name not in {n["name"] for n in names_list}:
-        names_list.append({"name": name, "category": category})
-    return True
-
-
-def sync_plane_alert_db() -> tuple[int, int]:
-    if not PADB_PATH.exists():
-        return 0, 0
-    with PADB_PATH.open(encoding="utf-8") as f:
-        db: dict = json.load(f)
-
-    updated = 0
-    added = 0
-    for fname in (
-        "plane-alert-db.csv",
-        "plane-alert-civ.csv",
-        "plane-alert-gov.csv",
-        "plane-alert-mil.csv",
-        "plane-alert-pol.csv",
-    ):
-        path = PAD / fname
-        if not path.exists():
-            continue
-        with path.open(encoding="utf-8", errors="replace") as f:
-            for row in csv.DictReader(f):
-                icao = norm_icao(row_get(row, "$ICAO", "ICAO"))
-                if not icao:
-                    continue
-                reg = row_get(row, "$Registration", "Registration")
-                operator = row_get(row, "$Operator", "Operator")
-                ac_type = row_get(row, "$Type", "Type")
-                category = row_get(row, "Category")
-                tag1 = row_get(row, "$Tag 1", "Tag 1")
-                tag2 = row_get(row, "#Tag 2", "$#Tag 2")
-                tag3 = row_get(row, "#Tag 3", "$#Tag 3")
-                link = row_get(row, "$#Link", "#Link", "$#Link ")
-                tags = ", ".join(t for t in (tag1, tag2, tag3) if t)
-
-                record = {
-                    "registration": reg,
-                    "operator": operator,
-                    "ac_type": ac_type,
-                    "category": category,
-                    "tags": tags,
-                    "link": link,
-                }
-                wiki = wiki_from_link(link)
-                if wiki:
-                    record["wiki"] = wiki
-
-                if icao in db:
-                    if db[icao] != record:
-                        db[icao] = record
-                        updated += 1
-                else:
-                    db[icao] = record
-                    added += 1
-
-    with PADB_PATH.open("w", encoding="utf-8") as f:
-        json.dump(db, f, indent=4, ensure_ascii=False)
-        f.write("\n")
-    return added, updated
-
-
-def main() -> None:
-    baseline_keys = load_git_baseline()
-
-    with SB_PATH.open(encoding="utf-8") as f:
-        sb = json.load(f)
-
-    details: dict = sb.setdefault("details", {})
-    names_list: list[dict] = sb.setdefault("names", [])
-
-    manual_added = 0
-    for name, category, regs in MANUAL_TRACKED:
-        for reg in regs:
-            if ensure_entry(details, names_list, name, category, reg, protect_names=baseline_keys):
-                manual_added += 1
-
-    for key in baseline_keys:
-        if key not in details:
-            raise RuntimeError(f"Baseline tracked name lost: {key}")
-
-    names_list.sort(key=lambda x: x["name"].lower())
-    with SB_PATH.open("w", encoding="utf-8") as f:
-        json.dump(sb, f, indent=2, ensure_ascii=False)
-        f.write("\n")
-
-    pad_added, pad_updated = sync_plane_alert_db()
-
-    print(f"Manual celeb regs added: {manual_added}")
-    print(f"plane_alert_db.json: +{pad_added} updated {pad_updated}")
-    print(f"tracked_names details: {len(details)}")
-    print(f"tracked_names registrations: {sum(len(v.get('registrations',[])) for v in details.values())}")
-
-
-if __name__ == "__main__":
-    main()