mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-06-25 15:30:07 +02:00
Compare commits
40 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| f1d62844f7 | |||
| 8c4619179e | |||
| 9ad0a5ffce | |||
| a0c79c2044 | |||
| f30ffe2c77 | |||
| 0690d94c37 | |||
| 53ed63ffcf | |||
| 4968f706bb | |||
| 91c76ad1bd | |||
| 013849ad1f | |||
| 559a1bd330 | |||
| cfbeabda1e | |||
| 9c5a4054f6 | |||
| 71a2ef4ce7 | |||
| 51f377f03d | |||
| 5ede669a12 | |||
| 8fcb01276c | |||
| 10dc9450be | |||
| bef462cdcf | |||
| 5135b771f5 | |||
| 7151563a41 | |||
| 52a28967a0 | |||
| 96182fe66d | |||
| 174031479c | |||
| f1cd9eb4b9 | |||
| c266c5ff5e | |||
| 52a0968092 | |||
| 89d6bb8fb9 | |||
| d48a0cdace | |||
| df76f6f147 | |||
| 776c89bfcf | |||
| d3006df57a | |||
| e78e4d186d | |||
| d1e1be4016 | |||
| 0afb85e241 | |||
| 039a0f9d0c | |||
| b9b99c1fa8 | |||
| a8fd33a758 | |||
| 7346129d0e | |||
| eb8f39f84e |
@@ -26,6 +26,20 @@ AIS_API_KEY=
|
||||
# Telegram OSINT map layer — scrapes public t.me/s channel previews (no bot token).
|
||||
# TELEGRAM_OSINT_ENABLED=true
|
||||
# TELEGRAM_OSINT_CHANNELS=osintdefender,insiderpaper,aljazeeraenglish,nexta_live,war_monitor
|
||||
# TELEGRAM_OSINT_TRANSLATE=true
|
||||
# TELEGRAM_OSINT_TRANSLATE_TO=en
|
||||
|
||||
# Strategic Risk Analytics (experimental derived OSINT — off by default)
|
||||
# GT_ANALYTICS_ENABLED=false
|
||||
# GT_ANALYTICS_PROFILE=lean
|
||||
# On 1 vCPU nodes (fleet VPS), leave disabled or set profile=lean. Scheduled ingest
|
||||
# and Louvain clustering stay off until GT_ANALYTICS_ACK_LOW_CPU=true.
|
||||
# GT_ANALYTICS_ACK_LOW_CPU=false
|
||||
# GT_ANALYTICS_BASE_PRIOR=0.15
|
||||
# GT_ANALYTICS_HIGH_RISK_THRESHOLD=0.6
|
||||
# GT_ANALYTICS_SIGNAL_WEIGHTS=payroll_loan=3.0,purge=3.5,troop_movement=3.0
|
||||
# GT_ANALYTICS_WATCHED_CHANNELS=osintdefender,war_monitor,nexta_live
|
||||
# GT_ANALYTICS_LOUVAIN_INTERVAL_MINUTES=30
|
||||
|
||||
# Admin key to protect sensitive endpoints (settings, updates).
|
||||
# If blank, loopback/localhost requests still work for local single-host dev.
|
||||
@@ -48,6 +62,9 @@ ADMIN_KEY=
|
||||
# Free MAP_KEY from https://firms.modaps.eosdis.nasa.gov/
|
||||
# FIRMS_MAP_KEY=
|
||||
|
||||
# Airframes.io ACARS/VDL datalink (plane dossier messages). Dashboard → API Key.
|
||||
# AIRFRAMES_API_KEY=
|
||||
|
||||
# Ukraine air raid alerts — free token from https://alerts.in.ua/
|
||||
# ALERTS_IN_UA_TOKEN=
|
||||
|
||||
|
||||
@@ -177,6 +177,8 @@ frontend/eslint-report.json
|
||||
.git_backup/
|
||||
local-artifacts/
|
||||
release-secrets/
|
||||
release-staging/
|
||||
.tmp-release-inspect/
|
||||
shadowbroker_repo/
|
||||
frontend/src/components.bak/
|
||||
frontend/src/components/map/icons/backups/
|
||||
@@ -261,6 +263,11 @@ frontend/.desktop-export-stash-*/
|
||||
backend/data/wormhole_stderr.log
|
||||
backend/data/wormhole_stdout.log
|
||||
|
||||
# Hermes Agent (operator-local runtime install — not project source)
|
||||
.hermes/
|
||||
**/.hermes/
|
||||
hermes-agent/
|
||||
|
||||
# Runtime caches that already slip through the backend/data/* blanket
|
||||
# (these are caught by the wildcard but listing for clarity)
|
||||
|
||||
|
||||
@@ -34,6 +34,7 @@ These sources have their own terms; consult each link before redistributing.
|
||||
| Source | URL | License / Terms | Notes |
|
||||
|---|---|---|---|
|
||||
| OpenSky Network | https://opensky-network.org | OpenSky API terms | Commercial and private aircraft tracking |
|
||||
| Airframes.io | https://airframes.io | Airframes API terms | Optional ACARS/VDL datalink messages in aircraft dossiers |
|
||||
| CelesTrak | https://celestrak.org | Public domain / no restrictions | Satellite TLE data |
|
||||
| USGS Earthquake Hazards | https://earthquake.usgs.gov | Public domain (US Federal) | Seismic events |
|
||||
| NASA FIRMS | https://firms.modaps.eosdis.nasa.gov | NASA Open Data | Fire/thermal anomalies (VIIRS) |
|
||||
|
||||
@@ -19,26 +19,42 @@
|
||||
|
||||
**ShadowBroker** is a decentralized intelligence platform that aggregates real-time, multi-domain OSINT telemetry from 60+ live intelligence feeds into a single dark-ops map interface. Aircraft, ships, satellites, conflict zones, CCTV networks, GPS jamming, internet-connected devices, police scanners, mesh radio nodes, and breaking geopolitical events — all updating in real time on one screen as well as an obfuscated communications protocol and information exchange infrastructure.
|
||||
|
||||
<details>
|
||||
<summary>🛰️ Project Description</summary>
|
||||
|
||||
Built with **Next.js**, **MapLibre GL**, **FastAPI**, and **Python**. 40+ toggleable data layers, including SAR ground-change detection, **Telegram OSINT** (public channel previews geoparsed onto the map), a **server-side recon toolkit** (DNS, WHOIS, sanctions, BGP, IP sweep, and more), supply-chain risk overlays, and malware/C2 + CISA KEV cyber threat feeds. Multiple visual modes (DEFAULT / SATELLITE / FLIR / NVG / CRT). Right-click any point on Earth for a country dossier, head-of-state lookup, entity-graph expansion, and the latest Sentinel-2 satellite photo. ShadowBroker has no accounts, product telemetry, or analytics; the dashboard talks to your self-hosted backend. Sensitive recon and Shodan queries never hit third-party APIs from the browser — they are proxied through the backend with SSRF guards and local-operator auth. The **OpenClaw / agent command channel** exposes the same recon backends plus full telemetry search — no separate API integration required.
|
||||
|
||||
Designed for analysts, researchers, radio operators, and anyone who wants to see what the world looks like when every public signal is on the same map.
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Why This Exists
|
||||
<details>
|
||||
<summary>🌍 Why This Exists</summary>
|
||||
|
||||
A surprising amount of global telemetry is already public — aircraft ADS-B broadcasts, maritime AIS signals, satellite orbital data, earthquake sensors, mesh radio networks, police scanner feeds, environmental monitoring stations, internet infrastructure telemetry, and more. This data is scattered across dozens of tools and APIs. ShadowBroker combines all of it into a single interface.
|
||||
|
||||
The project does not introduce new surveillance capabilities — it aggregates and visualizes existing public datasets. It is fully open-source so anyone can audit exactly what data is accessed and how. ShadowBroker does not include product telemetry, analytics, or accounts. Operator-supplied keys stay in your local deployment, but live OSINT features necessarily make outbound requests to the public data providers you enable or query.
|
||||
|
||||
### Shodan & Recon (security-first)
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
|
||||
<details>
|
||||
<summary>📡 Shodan & Recon (security-first)</summary>
|
||||
|
||||
ShadowBroker includes an optional **Shodan connector** for operator-supplied API access (`SHODAN_API_KEY`) and a **Recon Toolkit** panel for keyless OSINT lookups. Both run **server-side only**: the browser calls your self-hosted `/api/osint/*` and `/api/tools/shodan/*` routes; outbound requests are made by the backend after SSRF validation. Recon requires **local-operator** access (same trust model as layer toggles and admin routes). Shodan results render as a separate map overlay and remain subject to Shodan’s terms of service.
|
||||
|
||||
> **Not included:** embedded live-news YouTube grids or a built-in Gemini AI analyst panel — use the **OpenClaw / agent channel** for AI-assisted analysis instead.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Interesting Use Cases
|
||||
|
||||
<details>
|
||||
<summary>🗺️ Interesting Use Cases</summary>
|
||||
|
||||
* **Track Air Force One**, the private jets of billionaires and dictators, and every military tanker, ISR, and fighter broadcasting ADS-B. Air Force One and all of the accompanying Presidential/Vice Presidential planes are highlighted and monitored from the moment they leave the ground.
|
||||
* **Connect an AI agent as a co-analyst** through ShadowBroker's HMAC-signed agentic command channel — supports OpenClaw and any other agent that speaks the protocol (Claude, GPT, LangChain, custom). The agent gets full read/write access to all 40+ data layers, compact cross-layer search (`search_telemetry`, `search_news`), the full recon toolkit (`osint_lookup` for IP/DNS/WHOIS/sanctions/CVE/etc.), entity-graph expansion, pin placement, map control, SAR ground-change, mesh networking, and alert delivery. It sees everything the operator sees and can take actions on the map in real time.
|
||||
@@ -64,10 +80,12 @@ ShadowBroker includes an optional **Shodan connector** for operator-supplied API
|
||||
* **Monitor Telegram OSINT channels** — public `t.me/s` war/conflict feeds (OSINTdefender, NEXTA, etc.) scraped hourly, risk-scored, geoparsed to metro anchors, and plotted as clickable map pins with inline media
|
||||
* **Overlay global submarine cables** — static TeleGeography-derived cable routes (opt-in layer)
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Quick Start (Docker)
|
||||
|
||||
⚡ Quick Start (Docker)</summary>
|
||||
|
||||
### From GitHub (default — uses GHCR images)
|
||||
|
||||
@@ -91,15 +109,20 @@ Both paths produce identical containers — same source, same CI, same images by
|
||||
|
||||
Open `http://localhost:3000` to view the dashboard! *(Requires [Docker Desktop](https://www.docker.com/products/docker-desktop/) or Docker Engine)*
|
||||
|
||||
> **Join the private InfoNet swarm (sb-testnet-0):** Click **NODE** in the dashboard, or run `./meshnode.sh` for a headless participant. No manual peer list — fleet defaults discover the seed and pull the signed manifest automatically. Set `MESH_INFONET_FLEET_JOIN=false` in `.env` for a private solo node.
|
||||
|
||||
> **Backend port already in use?** The browser only needs port `3000`, but the backend API is also published on host port `8000` for local diagnostics. If another app already uses `8000`, create or edit `.env` next to `docker-compose.yml` and set `BACKEND_PORT=8001`, then run `docker compose up -d`.
|
||||
|
||||
> **Blank news/UAP/bases/wastewater after several minutes?** Check for backend OOM restarts with `docker events --since 30m --filter container=shadowbroker-backend --filter event=oom`. The default compose file gives the backend 4GB; if your host has less memory, reduce enabled feeds or set `BACKEND_MEMORY_LIMIT=3G` and expect slower/heavier layers to warm more gradually.
|
||||
|
||||
> **Podman users:** Podman works, but `podman compose` is a wrapper and still needs a Compose provider installed. On Windows/WSL, if you see `looking up compose provider failed`, install `podman-compose` and run `podman-compose pull` followed by `podman-compose up -d` from inside the cloned `Shadowbroker` folder. On Linux/macOS/WSL shells you can also use `./compose.sh --engine podman pull` and `./compose.sh --engine podman up -d`.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 🔄 **How to Update**
|
||||
<details>
|
||||
<summary>🔄 How to Update</summary>
|
||||
|
||||
ShadowBroker uses pre-built Docker images — no local building required. Updating takes seconds:
|
||||
|
||||
@@ -157,9 +180,13 @@ docker compose up -d
|
||||
* **Prune old images:** `docker image prune -f`
|
||||
* **Check logs:** `docker compose logs -f backend`
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
### **☸️ Kubernetes / Helm (Advanced)**
|
||||
<details>
|
||||
<summary>☸️ Kubernetes / Helm (Advanced)</summary>
|
||||
|
||||
For high-availability deployments or home-lab clusters, ShadowBroker supports deployment via **Helm**. This chart is based on the `bjw-s-labs` template and provides a robust, modular setup for both the backend and frontend.
|
||||
|
||||
@@ -187,9 +214,13 @@ helm install shadowbroker ./helm/chart --create-namespace --namespace shadowbrok
|
||||
|
||||
*Special thanks to [@chr0n1x](https://github.com/chr0n1x) for contributing the initial Kubernetes architecture.*
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Experimental Testnet — No Privacy Guarantee
|
||||
<details>
|
||||
<summary>🌐 Experimental Testnet — No Privacy Guarantee</summary>
|
||||
|
||||
|
||||
ShadowBroker v0.9.7 ships **InfoNet** (decentralized intelligence mesh + Sovereign Shell governance economy), an **agentic AI command channel** (supports OpenClaw and any HMAC-signing agent), **Time Machine snapshot playback**, and **SAR satellite ground-change detection**. This is an **experimental testnet** — not a private messenger and not a production governance system.
|
||||
|
||||
@@ -210,10 +241,12 @@ ShadowBroker v0.9.7 ships **InfoNet** (decentralized intelligence mesh + Soverei
|
||||
> sentence above is mapped there to the code path that enforces it (or
|
||||
> doesn't).**
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
|
||||
## ✨ Features
|
||||
<details>
|
||||
<summary>✨ Features</summary>
|
||||
|
||||
### 🧅 InfoNet — Decentralized Intelligence Mesh + Sovereign Shell (expanded in v0.9.7)
|
||||
|
||||
@@ -452,9 +485,12 @@ Settings → API Keys is now a read-only registry. Key values never reach the br
|
||||
|
||||
OpenSky API credentials are now a **critical-warn** environment requirement: the startup environment check flags missing OpenSky OAuth2 credentials with a strong warning, and the changelog modal links directly to the free registration page. Without them, the flights layer falls back to ADS-B-only coverage with significant gaps in Africa, Asia, and Latin America.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
<details>
|
||||
<summary>🏗️ Architecture</summary>
|
||||
|
||||
ShadowBroker v0.9.7 is composed of three vertically-stacked planes — the **Operator UI**, the **Backend Service Plane**, and the **Decentralized Layer (InfoNet)** — plus two cross-cutting bridges (the **Time Machine** and the **Agentic AI Channel**, which is the protocol that OpenClaw and any other compatible agent connects through) and a **Privacy Core** Rust crate that backstops both the legacy mesh and the future shielded coin / DEX work.
|
||||
|
||||
@@ -563,9 +599,12 @@ ShadowBroker v0.9.7 is composed of three vertically-stacked planes — the **Ope
|
||||
Desktop: Tauri shell → packaged backend-runtime + Next.js frontend
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 📊 Data Sources & APIs
|
||||
<details>
|
||||
<summary>📊 Data Sources & APIs</summary>
|
||||
|
||||
| Source | Data | Update Frequency | API Key Required |
|
||||
|---|---|---|---|
|
||||
@@ -624,9 +663,12 @@ ShadowBroker v0.9.7 is composed of three vertically-stacked planes — the **Ope
|
||||
|
||||
**Outbound privacy & audit (#348–#366):** Each self-hosted install uses its own backend IP and per-install User-Agent handle. See [docs/OUTBOUND_DATA.md](docs/OUTBOUND_DATA.md) for what contacts third parties, opt-in/env controls, and accepted tradeoffs (CCTV Referer, basemap CDN, LiveUAMap, etc.).
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Getting Started
|
||||
<details>
|
||||
<summary>🚀 Getting Started</summary>
|
||||
|
||||
### 🐳 Docker Setup (Recommended for Self-Hosting)
|
||||
|
||||
@@ -697,10 +739,12 @@ If you are in a bash-compatible shell, the included wrapper can auto-detect Dock
|
||||
./compose.sh --engine podman pull
|
||||
./compose.sh --engine podman up -d
|
||||
```
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
### 🐋 Standalone Deploy (Portainer, Uncloud, NAS, etc.)
|
||||
<details>
|
||||
<summary>🐋 Standalone Deploy (Portainer, Uncloud, NAS, etc.)</summary>
|
||||
|
||||
No need to clone the repo. Use the pre-built images from GitHub Container Registry. GitLab registry images may be used as a mirror if you publish them there.
|
||||
|
||||
@@ -752,9 +796,12 @@ volumes:
|
||||
>
|
||||
> `BACKEND_URL` is a plain runtime environment variable (not a build-time `NEXT_PUBLIC_*`), so you can change it in Portainer, Uncloud, or any compose editor without rebuilding the image. Set it to the address where your backend is reachable from inside the Docker network (e.g. `http://backend:8000`, `http://192.168.1.50:8000`).
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
### 📦 Quick Start (No Code Required)
|
||||
<details>
|
||||
<summary>📦 Quick Start (No Code Required)</summary>
|
||||
|
||||
If you just want to run the dashboard without dealing with terminal commands:
|
||||
|
||||
@@ -772,9 +819,12 @@ Local launcher notes:
|
||||
- For DM root witness, transparency, and operator monitoring rollout, start with `docs/mesh/wormhole-dm-root-operations-runbook.md`.
|
||||
- For sample DM root ops bridge assets, also see `scripts/mesh/poll-dm-root-health-alerts.mjs`, `scripts/mesh/export-dm-root-health-prometheus.mjs`, `scripts/mesh/publish-external-root-witness-package.mjs`, `scripts/mesh/smoke-external-root-witness-flow.mjs`, `scripts/mesh/smoke-root-transparency-publication-flow.mjs`, `scripts/mesh/smoke-dm-root-deployment-flow.mjs`, `scripts/mesh/sync-dm-root-external-assurance.mjs`, and `docs/mesh/examples/`.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
### 💻 Developer Setup
|
||||
<details>
|
||||
<summary>💻 Developer Setup</summary>
|
||||
|
||||
If you want to modify the code or run from source:
|
||||
|
||||
@@ -862,9 +912,13 @@ AIS-catcher decodes VHF radio signals on 161.975 MHz and 162.025 MHz and POSTs d
|
||||
|
||||
**Note:** AIS range depends on your antenna — typically 20-40 nautical miles with a basic setup, 60+ nm with a marine VHF antenna at elevation.
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 🎛️ Data Layers
|
||||
<details>
|
||||
<summary>🎛️ Data Layers</summary>
|
||||
|
||||
All 41 layers are independently toggleable from the left panel:
|
||||
|
||||
@@ -927,9 +981,12 @@ All 41 layers are independently toggleable from the left panel:
|
||||
|
||||
† `osint_sweep` (active InternetDB scan) requires `OPENCLAW_ACCESS_TIER=full`.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Performance
|
||||
<details>
|
||||
<summary>🔧 Performance</summary>
|
||||
|
||||
The platform is optimized for handling massive real-time datasets:
|
||||
|
||||
@@ -943,9 +1000,13 @@ The platform is optimized for handling massive real-time datasets:
|
||||
* **React.memo** — Heavy components wrapped to prevent unnecessary re-renders
|
||||
* **Coordinate Precision** — Lat/lng rounded to 5 decimals (~1m) to reduce JSON size
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
<details>
|
||||
<summary>📁 Project Structure</summary>
|
||||
|
||||
|
||||
```
|
||||
Shadowbroker/
|
||||
@@ -1043,9 +1104,12 @@ Shadowbroker/
|
||||
│ └── package.json
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🔑 Environment Variables
|
||||
<details>
|
||||
<summary>🔑 Environment Variables</summary>
|
||||
|
||||
### Backend (`backend/.env`)
|
||||
|
||||
@@ -1100,9 +1164,13 @@ Then confirm authenticated `GET /api/wormhole/status` or `GET /api/settings/worm
|
||||
|
||||
**How it works:** The frontend proxies all `/api/*` requests through the Next.js server to `BACKEND_URL` using Docker's internal networking. Browsers only talk to port 3000; the backend host port is only for local diagnostics. For local dev without Docker, `BACKEND_URL` defaults to `http://localhost:8000`.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributors
|
||||
|
||||
<details>
|
||||
<summary>🤝 Contributors</summary>
|
||||
|
||||
ShadowBroker is built in the open. These people shipped real code:
|
||||
|
||||
@@ -1118,6 +1186,8 @@ ShadowBroker is built in the open. These people shipped real code:
|
||||
| [@suranyami](https://github.com/suranyami) | Parallel multi-arch Docker builds (11min → 3min) + runtime BACKEND_URL fix | #35, #44 |
|
||||
| [@chr0n1x](https://github.com/chr0n1x) | Kubernetes / Helm chart architecture for HA deployments | — |
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Disclaimer
|
||||
|
||||
+20
-1
@@ -120,6 +120,9 @@ AIS_API_KEY= # https://aisstream.io/ — free tier WebSocket key
|
||||
# Free MAP_KEY from https://firms.modaps.eosdis.nasa.gov/map/#d:24hrs;@0.0,0.0,3.0z
|
||||
# FIRMS_MAP_KEY=
|
||||
|
||||
# Airframes.io ACARS/VDL datalink (plane dossier messages). Dashboard → API Key.
|
||||
# AIRFRAMES_API_KEY=
|
||||
|
||||
# Ukraine frontline mirror (GitHub). Default follows cyterat/deepstate-map-data@main.
|
||||
# Pin an immutable commit SHA so ingest cannot silently change if main is force-pushed (#362).
|
||||
# Example (verify on GitHub before use): main @ b479954e94696bc5622c7818fd20a64a699f4fe8
|
||||
@@ -227,7 +230,23 @@ AIS_API_KEY= # https://aisstream.io/ — free tier WebSocket key
|
||||
# MESH_GATE_SESSION_STREAM_MAX_GATES=16
|
||||
# MESH_BOOTSTRAP_DISABLED=false
|
||||
# MESH_BOOTSTRAP_MANIFEST_PATH=data/bootstrap_peers.json
|
||||
# MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY=
|
||||
# Swarm discovery (signed peer manifest). Participants need only the public key;
|
||||
# the seed operator sets MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY (never commit it).
|
||||
# Generate a fleet keypair: uv run python backend/scripts/bootstrap_manifest_helper.py generate-keypair
|
||||
# Public sb-testnet fleet defaults (auto-used when MESH_INFONET_FLEET_JOIN=true).
|
||||
# MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY=ul1d0kj/ODPIp0OhHzX8eLAVXzJ3CVvzW1vn2IC6q3I=
|
||||
# MESH_INFONET_FLEET_JOIN=true
|
||||
# MESH_INFONET_FLEET_JOIN_DISABLED=false
|
||||
# MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY= # seed only
|
||||
# MESH_BOOTSTRAP_SIGNER_ID=shadowbroker-seed
|
||||
# MESH_PEER_REGISTRY_ENABLED=true # seed only (auto-enabled when private key is set)
|
||||
# Headless relay compose sets MESH_INFONET_RELAY_AUTO_WORMHOLE=true; seed nodes with
|
||||
# MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY also auto-enable Tor wormhole on startup.
|
||||
# MESH_INFONET_RELAY_AUTO_WORMHOLE=false
|
||||
# MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED=false
|
||||
# MESH_SWARM_MANIFEST_TTL_S=14400
|
||||
# MESH_SWARM_MANIFEST_PULL_INTERVAL_S=300
|
||||
# MESH_PEER_REGISTRY_STALE_S=604800
|
||||
# Infonet/Wormhole fails closed to onion/RNS by default. Only enable clearnet
|
||||
# sync for local relay development or an explicitly public testnet.
|
||||
# MESH_INFONET_ALLOW_CLEARNET_SYNC=false
|
||||
|
||||
+2
-1
@@ -27,6 +27,7 @@ WORKDIR /app
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates \
|
||||
curl \
|
||||
git \
|
||||
tor \
|
||||
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
|
||||
&& apt-get install -y --no-install-recommends nodejs \
|
||||
@@ -72,7 +73,7 @@ ENV PRIVACY_CORE_LIB=/app/libprivacy_core.so
|
||||
# Create a non-root user for security
|
||||
# Grant write access to /app so the auto-updater can extract files
|
||||
# Pre-create /app/data so mounted volumes inherit correct ownership
|
||||
RUN adduser --system --uid 1001 backenduser \
|
||||
RUN adduser --system --uid 1001 --home /app backenduser \
|
||||
&& mkdir -p /app/data \
|
||||
&& chown -R backenduser /app \
|
||||
&& chmod -R u+w /app
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Strategic Risk Analytics — game-theoretic early warning layer."""
|
||||
|
||||
from analytics.backtest import (
|
||||
DEFAULT_BACKTEST_ALERT_THRESHOLD,
|
||||
BacktestReport,
|
||||
run_historical_backtest,
|
||||
tune_alert_threshold,
|
||||
)
|
||||
from analytics.gt_early_warning import GT_EarlyWarning
|
||||
from analytics.integration import get_gt_engine, process_feed_item, refresh_from_latest_data
|
||||
|
||||
__all__ = [
|
||||
"BacktestReport",
|
||||
"DEFAULT_BACKTEST_ALERT_THRESHOLD",
|
||||
"GT_EarlyWarning",
|
||||
"get_gt_engine",
|
||||
"process_feed_item",
|
||||
"refresh_from_latest_data",
|
||||
"run_historical_backtest",
|
||||
"tune_alert_threshold",
|
||||
]
|
||||
@@ -0,0 +1,287 @@
|
||||
"""Historical backtesting for Strategic Risk Analytics.
|
||||
|
||||
This is **benchmark validation**, not forward-weeks prediction on live feeds.
|
||||
|
||||
The suite scores whether costly-signal patterns + Bayesian updating correctly
|
||||
classify curated pre-crisis text snippets (positive cases) vs cheap-talk
|
||||
controls (negative cases) at a tuned alert threshold. A high accuracy on this
|
||||
labeled corpus does **not** imply the engine will score 100% on messy,
|
||||
adversarial, or weeks-ahead production telemetry — opponents adapt, labels are
|
||||
easier here than in the wild, and the window is retrospective.
|
||||
|
||||
Reports accuracy and a conservative Wilson 95% confidence lower bound on the
|
||||
benchmark only. Treat 100% here as "classifier fits the benchmark," not "ship
|
||||
it for multi-week forecasting." For live week-over-week scoring with delayed
|
||||
labels, see ``rolling_backtest.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
from analytics.gt_early_warning import GT_EarlyWarning
|
||||
from analytics.historical_events import (
|
||||
HistoricalCase,
|
||||
default_historical_cases,
|
||||
expanded_historical_cases,
|
||||
)
|
||||
from analytics.settings import GTAnalyticsSettings
|
||||
|
||||
DomainName = Literal["financial", "unrest", "conflict"]
|
||||
|
||||
# Validated on expanded suite (82 cases, Wilson lower >= 0.95 at 100% accuracy).
|
||||
DEFAULT_BACKTEST_ALERT_THRESHOLD = 0.26
|
||||
MAX_BACKTEST_ALERT_THRESHOLD = 0.39
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CaseResult:
|
||||
case_id: str
|
||||
name: str
|
||||
kind: str
|
||||
region: str
|
||||
domain: str
|
||||
expected_alert: bool
|
||||
alerted: bool
|
||||
correct: bool
|
||||
peak_domain_risk: float
|
||||
peak_composite_risk: float
|
||||
costly_signals: list[str]
|
||||
tags: tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BacktestReport:
|
||||
total_cases: int
|
||||
correct: int
|
||||
accuracy: float
|
||||
confidence_rate: float
|
||||
wilson_lower_95: float
|
||||
wilson_upper_95: float
|
||||
true_positives: int
|
||||
true_negatives: int
|
||||
false_positives: int
|
||||
false_negatives: int
|
||||
sensitivity: float
|
||||
specificity: float
|
||||
alert_threshold: float
|
||||
target_confidence: float
|
||||
meets_target: bool
|
||||
case_results: tuple[CaseResult, ...]
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"total_cases": self.total_cases,
|
||||
"correct": self.correct,
|
||||
"accuracy": round(self.accuracy, 4),
|
||||
"confidence_rate": round(self.confidence_rate, 4),
|
||||
"wilson_lower_95": round(self.wilson_lower_95, 4),
|
||||
"wilson_upper_95": round(self.wilson_upper_95, 4),
|
||||
"true_positives": self.true_positives,
|
||||
"true_negatives": self.true_negatives,
|
||||
"false_positives": self.false_positives,
|
||||
"false_negatives": self.false_negatives,
|
||||
"sensitivity": round(self.sensitivity, 4),
|
||||
"specificity": round(self.specificity, 4),
|
||||
"alert_threshold": self.alert_threshold,
|
||||
"target_confidence": self.target_confidence,
|
||||
"meets_target": self.meets_target,
|
||||
"cases": [
|
||||
{
|
||||
"case_id": row.case_id,
|
||||
"name": row.name,
|
||||
"kind": row.kind,
|
||||
"correct": row.correct,
|
||||
"alerted": row.alerted,
|
||||
"peak_domain_risk": round(row.peak_domain_risk, 4),
|
||||
"peak_composite_risk": round(row.peak_composite_risk, 4),
|
||||
"costly_signals": row.costly_signals,
|
||||
}
|
||||
for row in self.case_results
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def wilson_interval(
|
||||
successes: int,
|
||||
total: int,
|
||||
z: float = 1.96,
|
||||
) -> tuple[float, float]:
|
||||
"""Wilson score interval for a binomial proportion (95% default)."""
|
||||
if total <= 0:
|
||||
return 0.0, 0.0
|
||||
phat = successes / total
|
||||
z2 = z * z
|
||||
denom = 1.0 + z2 / total
|
||||
center = (phat + z2 / (2.0 * total)) / denom
|
||||
margin = (
|
||||
z
|
||||
* math.sqrt((phat * (1.0 - phat) + z2 / (4.0 * total)) / total)
|
||||
/ denom
|
||||
)
|
||||
return max(0.0, center - margin), min(1.0, center + margin)
|
||||
|
||||
|
||||
def _domain_risk(engine: GT_EarlyWarning, region: str, domain: str) -> float:
|
||||
if domain in ("financial", "unrest", "conflict"):
|
||||
return engine.get_prior(region, domain)
|
||||
return engine.composite_risk(region)
|
||||
|
||||
|
||||
def _evaluate_case(
|
||||
case: HistoricalCase,
|
||||
*,
|
||||
settings: GTAnalyticsSettings,
|
||||
alert_threshold: float,
|
||||
) -> CaseResult:
|
||||
engine = GT_EarlyWarning(settings)
|
||||
peak_domain = float(settings.base_prior)
|
||||
peak_composite = float(settings.base_prior)
|
||||
detected_signals: set[str] = set()
|
||||
|
||||
for item in case.to_feed_dicts():
|
||||
result = engine.process_feed_item(item)
|
||||
for sig in (result or {}).get("signals") or {}:
|
||||
detected_signals.add(str(sig))
|
||||
domain_risk = _domain_risk(engine, case.region, case.domain)
|
||||
composite = engine.composite_risk(case.region)
|
||||
peak_domain = max(peak_domain, domain_risk)
|
||||
peak_composite = max(peak_composite, composite)
|
||||
|
||||
# Domain-specific score for labeled events; composite as secondary for conflict.
|
||||
score = peak_domain
|
||||
if case.domain == "conflict":
|
||||
score = max(peak_domain, peak_composite * 0.95)
|
||||
alerted = score >= alert_threshold
|
||||
expected_alert = case.kind == "positive"
|
||||
|
||||
return CaseResult(
|
||||
case_id=case.case_id,
|
||||
name=case.name,
|
||||
kind=case.kind,
|
||||
region=case.region,
|
||||
domain=case.domain,
|
||||
expected_alert=expected_alert,
|
||||
alerted=alerted,
|
||||
correct=alerted == expected_alert,
|
||||
peak_domain_risk=peak_domain,
|
||||
peak_composite_risk=peak_composite,
|
||||
costly_signals=sorted(detected_signals),
|
||||
tags=case.tags,
|
||||
)
|
||||
|
||||
|
||||
def run_historical_backtest(
|
||||
cases: tuple[HistoricalCase, ...] | None = None,
|
||||
*,
|
||||
settings: GTAnalyticsSettings | None = None,
|
||||
alert_threshold: float | None = None,
|
||||
target_confidence: float = 0.80,
|
||||
use_expanded_suite: bool = True,
|
||||
) -> BacktestReport:
|
||||
"""
|
||||
Run labeled historical cases and compute accuracy + Wilson 95% CI.
|
||||
|
||||
``confidence_rate`` is the conservative Wilson lower bound — the metric
|
||||
used for pass/fail against ``target_confidence``.
|
||||
"""
|
||||
cfg = settings or GTAnalyticsSettings(enabled=True)
|
||||
threshold = float(
|
||||
alert_threshold
|
||||
if alert_threshold is not None
|
||||
else DEFAULT_BACKTEST_ALERT_THRESHOLD
|
||||
)
|
||||
if cases is not None:
|
||||
suite = cases
|
||||
elif use_expanded_suite:
|
||||
suite = expanded_historical_cases()
|
||||
else:
|
||||
suite = default_historical_cases()
|
||||
|
||||
results = tuple(
|
||||
_evaluate_case(case, settings=cfg, alert_threshold=threshold) for case in suite
|
||||
)
|
||||
|
||||
tp = sum(1 for r in results if r.expected_alert and r.alerted)
|
||||
tn = sum(1 for r in results if not r.expected_alert and not r.alerted)
|
||||
fp = sum(1 for r in results if not r.expected_alert and r.alerted)
|
||||
fn = sum(1 for r in results if r.expected_alert and not r.alerted)
|
||||
correct = tp + tn
|
||||
total = len(results)
|
||||
accuracy = correct / total if total else 0.0
|
||||
lower, upper = wilson_interval(correct, total)
|
||||
|
||||
pos_total = sum(1 for r in results if r.expected_alert)
|
||||
neg_total = total - pos_total
|
||||
sensitivity = tp / pos_total if pos_total else 0.0
|
||||
specificity = tn / neg_total if neg_total else 0.0
|
||||
|
||||
return BacktestReport(
|
||||
total_cases=total,
|
||||
correct=correct,
|
||||
accuracy=accuracy,
|
||||
confidence_rate=lower,
|
||||
wilson_lower_95=lower,
|
||||
wilson_upper_95=upper,
|
||||
true_positives=tp,
|
||||
true_negatives=tn,
|
||||
false_positives=fp,
|
||||
false_negatives=fn,
|
||||
sensitivity=sensitivity,
|
||||
specificity=specificity,
|
||||
alert_threshold=threshold,
|
||||
target_confidence=target_confidence,
|
||||
meets_target=lower >= target_confidence,
|
||||
case_results=results,
|
||||
)
|
||||
|
||||
|
||||
def tune_alert_threshold(
|
||||
cases: tuple[HistoricalCase, ...] | None = None,
|
||||
*,
|
||||
settings: GTAnalyticsSettings | None = None,
|
||||
min_threshold: float = 0.20,
|
||||
max_threshold: float = 0.65,
|
||||
step: float = 0.01,
|
||||
target_confidence: float = 0.95,
|
||||
) -> tuple[float, BacktestReport]:
|
||||
"""Grid-search alert threshold to maximize Wilson lower bound."""
|
||||
if cases is not None:
|
||||
suite = cases
|
||||
else:
|
||||
suite = expanded_historical_cases()
|
||||
best_threshold = min_threshold
|
||||
best_report = run_historical_backtest(
|
||||
suite,
|
||||
settings=settings,
|
||||
alert_threshold=min_threshold,
|
||||
target_confidence=target_confidence,
|
||||
)
|
||||
|
||||
steps = int(round((max_threshold - min_threshold) / step))
|
||||
for i in range(steps + 1):
|
||||
threshold = min_threshold + i * step
|
||||
report = run_historical_backtest(
|
||||
suite,
|
||||
settings=settings,
|
||||
alert_threshold=threshold,
|
||||
target_confidence=target_confidence,
|
||||
)
|
||||
better_confidence = report.confidence_rate > best_report.confidence_rate
|
||||
tied_confidence = math.isclose(
|
||||
report.confidence_rate, best_report.confidence_rate, rel_tol=0.0, abs_tol=1e-9
|
||||
)
|
||||
better_accuracy = report.accuracy > best_report.accuracy
|
||||
tied_accuracy = math.isclose(
|
||||
report.accuracy, best_report.accuracy, rel_tol=0.0, abs_tol=1e-9
|
||||
)
|
||||
prefer_higher_threshold = (
|
||||
tied_confidence and tied_accuracy and threshold > best_threshold
|
||||
)
|
||||
if better_confidence or (tied_confidence and better_accuracy) or prefer_higher_threshold:
|
||||
best_threshold = threshold
|
||||
best_report = report
|
||||
|
||||
return best_threshold, best_report
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Daily GT risk readings for micro rolling averages."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import date, datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DAILY_DIR = Path(__file__).parent.parent / "data" / "gt_rolling" / "daily"
|
||||
_store_lock = threading.Lock()
|
||||
|
||||
|
||||
def daily_store_dir() -> Path:
|
||||
override = str(os.environ.get("GT_DAILY_STORE_DIR", "")).strip()
|
||||
if override:
|
||||
return Path(override)
|
||||
return _DAILY_DIR
|
||||
|
||||
|
||||
def utc_today() -> date:
|
||||
return datetime.now(timezone.utc).date()
|
||||
|
||||
|
||||
def date_id(when: date | datetime | None = None) -> str:
|
||||
if when is None:
|
||||
when = utc_today()
|
||||
if isinstance(when, datetime):
|
||||
when = when.date()
|
||||
return when.isoformat()
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailyRegionReading:
|
||||
region: str
|
||||
composite_risk: float
|
||||
financial: float
|
||||
unrest: float
|
||||
conflict: float
|
||||
peak_score: float
|
||||
readings: int = 1
|
||||
last_captured_at: str = ""
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, raw: dict[str, Any]) -> DailyRegionReading:
|
||||
return cls(
|
||||
region=str(raw.get("region") or "").strip().lower(),
|
||||
composite_risk=float(raw.get("composite_risk") or 0.0),
|
||||
financial=float(raw.get("financial") or 0.0),
|
||||
unrest=float(raw.get("unrest") or 0.0),
|
||||
conflict=float(raw.get("conflict") or 0.0),
|
||||
peak_score=float(raw.get("peak_score") or 0.0),
|
||||
readings=int(raw.get("readings") or 1),
|
||||
last_captured_at=str(raw.get("last_captured_at") or ""),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DailySnapshot:
|
||||
date: str
|
||||
regions: dict[str, DailyRegionReading] = field(default_factory=dict)
|
||||
last_updated_at: str = ""
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"date": self.date,
|
||||
"last_updated_at": self.last_updated_at,
|
||||
"regions": {key: row.to_dict() for key, row in self.regions.items()},
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, raw: dict[str, Any]) -> DailySnapshot:
|
||||
regions: dict[str, DailyRegionReading] = {}
|
||||
for key, row in (raw.get("regions") or {}).items():
|
||||
if isinstance(row, dict):
|
||||
reading = DailyRegionReading.from_dict(row)
|
||||
regions[str(key).strip().lower()] = reading
|
||||
return cls(
|
||||
date=str(raw.get("date") or ""),
|
||||
regions=regions,
|
||||
last_updated_at=str(raw.get("last_updated_at") or ""),
|
||||
)
|
||||
|
||||
|
||||
def _daily_path(day_id: str) -> Path:
|
||||
safe = day_id.replace("/", "-").replace("..", "")
|
||||
return daily_store_dir() / f"{safe}.json"
|
||||
|
||||
|
||||
def _ensure_dir() -> None:
|
||||
daily_store_dir().mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def list_daily_ids(*, newest_first: bool = True, limit: int | None = None) -> list[str]:
|
||||
_ensure_dir()
|
||||
ids = sorted(
|
||||
(path.stem for path in daily_store_dir().glob("*.json")),
|
||||
reverse=newest_first,
|
||||
)
|
||||
if limit is not None:
|
||||
return ids[:limit]
|
||||
return ids
|
||||
|
||||
|
||||
def load_daily(day: date | str | None = None) -> DailySnapshot | None:
|
||||
day_id = date_id(day) if day is not None else date_id()
|
||||
path = _daily_path(day_id)
|
||||
if not path.is_file():
|
||||
return None
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, dict):
|
||||
return None
|
||||
return DailySnapshot.from_dict(raw)
|
||||
except (OSError, json.JSONDecodeError, TypeError, ValueError):
|
||||
logger.exception("Failed to load GT daily reading %s", day_id)
|
||||
return None
|
||||
|
||||
|
||||
def save_daily(snapshot: DailySnapshot) -> None:
|
||||
_ensure_dir()
|
||||
path = _daily_path(snapshot.date)
|
||||
tmp = path.with_suffix(".json.tmp")
|
||||
payload = json.dumps(snapshot.to_dict(), indent=2, sort_keys=True)
|
||||
with _store_lock:
|
||||
tmp.write_text(payload, encoding="utf-8")
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def utc_now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
@@ -0,0 +1,206 @@
|
||||
"""Normalize Shadowbroker feed records into GT analytics feed items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, Iterable
|
||||
|
||||
_DOMAIN_CONFLICT = "conflict"
|
||||
_DOMAIN_UNREST = "unrest"
|
||||
_DOMAIN_FINANCIAL = "financial"
|
||||
|
||||
_CONFLICT_HINTS = re.compile(
|
||||
r"\b(war|missile|strike|attack|military|invasion|troop|shelling|drone|bomb|nuclear)\b",
|
||||
re.I,
|
||||
)
|
||||
_UNREST_HINTS = re.compile(
|
||||
r"\b(protest|rally|strike|riot|unrest|mobiliz|demonstrat|curfew|purge|coup)\b",
|
||||
re.I,
|
||||
)
|
||||
_FINANCIAL_HINTS = re.compile(
|
||||
r"\b(payroll|loan|default|bankruptcy|liquidity|sanction|supply\s+chain|delay|shortage)\b",
|
||||
re.I,
|
||||
)
|
||||
|
||||
|
||||
def _clean_region(value: Any) -> str:
|
||||
region = str(value or "").strip().lower()
|
||||
return region or "global"
|
||||
|
||||
|
||||
def _infer_domain(text: str, explicit: str | None = None) -> str:
|
||||
if explicit in {_DOMAIN_CONFLICT, _DOMAIN_UNREST, _DOMAIN_FINANCIAL}:
|
||||
return explicit
|
||||
if _CONFLICT_HINTS.search(text):
|
||||
return _DOMAIN_CONFLICT
|
||||
if _UNREST_HINTS.search(text):
|
||||
return _DOMAIN_UNREST
|
||||
if _FINANCIAL_HINTS.search(text):
|
||||
return _DOMAIN_FINANCIAL
|
||||
return _DOMAIN_FINANCIAL
|
||||
|
||||
|
||||
def _text_from_record(
|
||||
record: dict[str, Any],
|
||||
*,
|
||||
prefer_translation: bool = False,
|
||||
) -> str:
|
||||
"""Build ingest text; prefer English translations for Telegram OSINT when set."""
|
||||
if prefer_translation:
|
||||
translated_parts = [
|
||||
record.get("title_translated"),
|
||||
record.get("description_translated"),
|
||||
]
|
||||
translated = "\n".join(
|
||||
str(p).strip() for p in translated_parts if p and str(p).strip()
|
||||
)
|
||||
if translated:
|
||||
return translated
|
||||
|
||||
parts = [
|
||||
record.get("title"),
|
||||
record.get("description"),
|
||||
record.get("text"),
|
||||
record.get("summary"),
|
||||
]
|
||||
return "\n".join(str(p).strip() for p in parts if p and str(p).strip())
|
||||
|
||||
|
||||
_HASHTAG_REGION = re.compile(r"#([a-z][a-z0-9_-]{2,})", re.I)
|
||||
|
||||
|
||||
def _region_from_hashtags(text: str) -> str | None:
|
||||
"""Map common theater hashtags (#Ukraine) to dossier/heatmap region keys."""
|
||||
for match in _HASHTAG_REGION.finditer(text or ""):
|
||||
tag = match.group(1).lower()
|
||||
if tag in {
|
||||
"ukraine",
|
||||
"russia",
|
||||
"israel",
|
||||
"iran",
|
||||
"gaza",
|
||||
"syria",
|
||||
"taiwan",
|
||||
"china",
|
||||
"belfast",
|
||||
"uk",
|
||||
"usa",
|
||||
}:
|
||||
return tag
|
||||
return None
|
||||
|
||||
|
||||
def _region_from_record(record: dict[str, Any], *, text: str = "") -> str:
|
||||
for key in ("geotag", "region", "country", "location"):
|
||||
if record.get(key):
|
||||
return _clean_region(record[key])
|
||||
hashtag_region = _region_from_hashtags(text)
|
||||
if hashtag_region:
|
||||
return hashtag_region
|
||||
coords = record.get("coords")
|
||||
if isinstance(coords, (list, tuple)) and len(coords) >= 2:
|
||||
try:
|
||||
lat = float(coords[0])
|
||||
lng = float(coords[1])
|
||||
return f"{lat:.2f},{lng:.2f}"
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return "global"
|
||||
|
||||
|
||||
def _entities_from_record(record: dict[str, Any]) -> list[str]:
|
||||
entities: list[str] = []
|
||||
for key in ("entities", "tags", "keywords"):
|
||||
raw = record.get(key)
|
||||
if isinstance(raw, list):
|
||||
entities.extend(str(v).strip() for v in raw if str(v).strip())
|
||||
elif isinstance(raw, str) and raw.strip():
|
||||
entities.extend(part.strip() for part in raw.split(",") if part.strip())
|
||||
channel = str(record.get("channel") or "").strip()
|
||||
if channel:
|
||||
entities.append(f"channel:{channel}")
|
||||
source = str(record.get("source") or "").strip()
|
||||
if source:
|
||||
entities.append(f"source:{source}")
|
||||
return entities
|
||||
|
||||
|
||||
def normalize_feed_item(record: dict[str, Any], *, source_type: str = "generic") -> dict[str, Any]:
|
||||
"""Map a news/Telegram/GDELT record into the GT engine schema."""
|
||||
prefer_translation = source_type == "telegram_osint"
|
||||
text = _text_from_record(record, prefer_translation=prefer_translation)
|
||||
if prefer_translation and not text.strip():
|
||||
text = _text_from_record(record, prefer_translation=False)
|
||||
region = _region_from_record(record, text=text)
|
||||
domain = _infer_domain(text, record.get("domain"))
|
||||
coords = record.get("coords")
|
||||
lat = lng = None
|
||||
if isinstance(coords, (list, tuple)) and len(coords) >= 2:
|
||||
try:
|
||||
lat = float(coords[0])
|
||||
lng = float(coords[1])
|
||||
except (TypeError, ValueError):
|
||||
lat = lng = None
|
||||
|
||||
return {
|
||||
"id": record.get("id") or record.get("link"),
|
||||
"text": text,
|
||||
"source": str(record.get("source") or source_type),
|
||||
"source_type": source_type,
|
||||
"region": region,
|
||||
"domain": domain,
|
||||
"entities": _entities_from_record(record),
|
||||
"coords": [lat, lng] if lat is not None and lng is not None else None,
|
||||
"published": record.get("published"),
|
||||
"risk_score": record.get("risk_score"),
|
||||
}
|
||||
|
||||
|
||||
def iter_telegram_posts(payload: dict[str, Any] | None) -> Iterable[dict[str, Any]]:
|
||||
from services.telegram_translate import apply_post_translation, telegram_translate_enabled
|
||||
|
||||
posts = list((payload or {}).get("posts") or [])
|
||||
for post in posts:
|
||||
if not isinstance(post, dict):
|
||||
continue
|
||||
if not (post.get("description") or post.get("title")):
|
||||
continue
|
||||
enriched = (
|
||||
apply_post_translation(post)
|
||||
if telegram_translate_enabled()
|
||||
else post
|
||||
)
|
||||
yield normalize_feed_item(enriched, source_type="telegram_osint")
|
||||
|
||||
|
||||
def iter_news_items(payload: list[dict[str, Any]] | None) -> Iterable[dict[str, Any]]:
|
||||
for item in list(payload or []):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
yield normalize_feed_item(item, source_type="news")
|
||||
for article in list(item.get("articles") or []):
|
||||
if isinstance(article, dict):
|
||||
yield normalize_feed_item(article, source_type="news_cluster")
|
||||
|
||||
|
||||
def iter_gdelt_features(payload: list[dict[str, Any]] | None) -> Iterable[dict[str, Any]]:
|
||||
for feature in list(payload or []):
|
||||
if not isinstance(feature, dict):
|
||||
continue
|
||||
props = dict(feature.get("properties") or {})
|
||||
geometry = dict(feature.get("geometry") or {})
|
||||
coords = None
|
||||
if geometry.get("type") == "Point":
|
||||
raw = geometry.get("coordinates")
|
||||
if isinstance(raw, (list, tuple)) and len(raw) >= 2:
|
||||
coords = [float(raw[1]), float(raw[0])]
|
||||
record = {
|
||||
"title": props.get("name") or props.get("title"),
|
||||
"description": props.get("snippet") or props.get("description"),
|
||||
"source": props.get("source") or "gdelt",
|
||||
"coords": coords,
|
||||
"published": props.get("date") or props.get("published"),
|
||||
"region": props.get("location") or props.get("country"),
|
||||
}
|
||||
if record["title"] or record["description"]:
|
||||
yield normalize_feed_item(record, source_type="gdelt")
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Top strategic-risk alerts — ranked regions with map coordinates."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from analytics.integration import get_gt_engine
|
||||
from analytics.settings import get_gt_settings
|
||||
|
||||
|
||||
def _peak_score(props: dict[str, Any]) -> float:
|
||||
composite = float(props.get("risk") or 0.0)
|
||||
financial = float(props.get("financial") or 0.0)
|
||||
unrest = float(props.get("unrest") or 0.0)
|
||||
conflict = float(props.get("conflict") or 0.0)
|
||||
return max(composite, financial, unrest, conflict)
|
||||
|
||||
|
||||
def _valid_coords(coords: Any) -> tuple[float, float] | None:
|
||||
if not isinstance(coords, (list, tuple)) or len(coords) < 2:
|
||||
return None
|
||||
try:
|
||||
lng = float(coords[0])
|
||||
lat = float(coords[1])
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not (-90.0 <= lat <= 90.0 and -180.0 <= lng <= 180.0):
|
||||
return None
|
||||
if abs(lat) < 0.001 and abs(lng) < 0.001:
|
||||
return None
|
||||
return lat, lng
|
||||
|
||||
|
||||
def _region_label(region: str) -> str:
|
||||
text = str(region or "").strip()
|
||||
if not text:
|
||||
return "unknown"
|
||||
if "," in text:
|
||||
parts = [piece.strip() for piece in text.split(",") if piece.strip()]
|
||||
if len(parts) >= 2:
|
||||
try:
|
||||
lat = float(parts[0])
|
||||
lng = float(parts[-1])
|
||||
return f"{lat:.2f}°, {lng:.2f}°"
|
||||
except ValueError:
|
||||
pass
|
||||
return text.replace("_", " ")
|
||||
|
||||
|
||||
def parse_heatmap_alerts(
|
||||
heatmap: dict[str, Any] | None,
|
||||
*,
|
||||
limit: int = 8,
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
"""Return ranked alerts and count of regions plottable on the map."""
|
||||
features = (heatmap or {}).get("features") or []
|
||||
rows: list[dict[str, Any]] = []
|
||||
|
||||
for feature in features:
|
||||
if not isinstance(feature, dict):
|
||||
continue
|
||||
geometry = feature.get("geometry") or {}
|
||||
coords = _valid_coords(geometry.get("coordinates"))
|
||||
if coords is None:
|
||||
continue
|
||||
lat, lng = coords
|
||||
props = feature.get("properties") or {}
|
||||
region = str(props.get("region") or "").strip().lower()
|
||||
if not region:
|
||||
continue
|
||||
score = _peak_score(props)
|
||||
rows.append(
|
||||
{
|
||||
"region": region,
|
||||
"region_label": _region_label(region),
|
||||
"risk": round(float(props.get("risk") or 0.0), 4),
|
||||
"financial": round(float(props.get("financial") or 0.0), 4),
|
||||
"unrest": round(float(props.get("unrest") or 0.0), 4),
|
||||
"conflict": round(float(props.get("conflict") or 0.0), 4),
|
||||
"contagion": round(float(props.get("contagion") or 0.0), 4),
|
||||
"score": round(score, 4),
|
||||
"lat": lat,
|
||||
"lng": lng,
|
||||
"ignition": bool(props.get("micro_ignition")),
|
||||
"risk_3d_avg": props.get("risk_3d_avg"),
|
||||
"risk_delta": props.get("risk_delta"),
|
||||
"updates": int(props.get("updates") or 0),
|
||||
}
|
||||
)
|
||||
|
||||
rows.sort(
|
||||
key=lambda row: (
|
||||
bool(row.get("ignition")),
|
||||
float(row.get("risk_delta") or 0.0),
|
||||
float(row.get("score") or 0.0),
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
return rows[: max(1, limit)], len(rows)
|
||||
|
||||
|
||||
def top_gt_alerts(*, limit: int = 8) -> dict[str, Any]:
|
||||
"""Ranked top regions for API / OpenClaw."""
|
||||
settings = get_gt_settings()
|
||||
engine = get_gt_engine()
|
||||
heatmap: dict[str, Any] = {"type": "FeatureCollection", "features": []}
|
||||
engine_regions = 0
|
||||
|
||||
if engine is not None:
|
||||
heatmap = engine.get_risk_heatmap()
|
||||
with engine._lock: # noqa: SLF001 — intentional meta read
|
||||
engine_regions = len(engine._regions)
|
||||
|
||||
alerts, plotted = parse_heatmap_alerts(heatmap, limit=limit)
|
||||
tracked = len(heatmap.get("features") or [])
|
||||
|
||||
return {
|
||||
"alerts": alerts,
|
||||
"tracked_regions": tracked,
|
||||
"engine_regions": engine_regions,
|
||||
"plotted_regions": plotted,
|
||||
"max_regions": settings.max_heatmap_features,
|
||||
"note": (
|
||||
"Layer count is tracked GT regions (cap "
|
||||
f"{settings.max_heatmap_features}), not raw feed events. "
|
||||
"Only regions with valid coordinates appear on the map."
|
||||
),
|
||||
}
|
||||
@@ -0,0 +1,593 @@
|
||||
"""Game-theoretic early warning analytics with Bayesian updating and contagion graph."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, DefaultDict
|
||||
|
||||
import networkx as nx
|
||||
import numpy as np
|
||||
|
||||
from analytics.settings import GTAnalyticsSettings, get_gt_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DomainName = str # financial | unrest | conflict
|
||||
|
||||
_DOMAINS: tuple[DomainName, ...] = ("financial", "unrest", "conflict")
|
||||
|
||||
_DEFAULT_LIKELIHOODS: dict[DomainName, dict[str, float]] = {
|
||||
"financial": {"distress": 0.75, "normal": 0.25},
|
||||
"unrest": {"distress": 0.82, "normal": 0.22},
|
||||
"conflict": {"distress": 0.78, "normal": 0.18},
|
||||
}
|
||||
|
||||
_DEFAULT_SIGNAL_WEIGHTS: dict[str, float] = {
|
||||
"payroll_loan": 3.0,
|
||||
"supply_delay": 2.2,
|
||||
"elite_relocation": 2.8,
|
||||
"purge": 3.5,
|
||||
"protest_mobilize": 2.5,
|
||||
"gps_jamming": 2.7,
|
||||
"troop_movement": 3.0,
|
||||
"bank_run": 3.2,
|
||||
"sanctions_escalation": 2.4,
|
||||
"ceasefire_break": 2.6,
|
||||
}
|
||||
|
||||
# Costly-signal regex patterns (cheap talk filtered by absence of match).
|
||||
_SIGNAL_PATTERNS: dict[str, list[re.Pattern[str]]] = {
|
||||
"payroll_loan": [
|
||||
re.compile(r"payroll\s+loan", re.I),
|
||||
re.compile(r"merchant\s+cash\s+advance", re.I),
|
||||
re.compile(r"working\s+capital\s+loan", re.I),
|
||||
],
|
||||
"supply_delay": [
|
||||
re.compile(r"supply\s+(chain\s+)?delay", re.I),
|
||||
re.compile(r"shipping\s+delay", re.I),
|
||||
re.compile(r"logistics\s+backlog", re.I),
|
||||
re.compile(r"port\s+congestion", re.I),
|
||||
],
|
||||
"elite_relocation": [
|
||||
re.compile(r"elite\s+(asset\s+)?relocation", re.I),
|
||||
re.compile(r"oligarch\s+jet", re.I),
|
||||
re.compile(r"private\s+jet\s+exodus", re.I),
|
||||
re.compile(r"capital\s+flight", re.I),
|
||||
],
|
||||
"purge": [
|
||||
re.compile(r"\bpurge\b", re.I),
|
||||
re.compile(r"political\s+purge", re.I),
|
||||
re.compile(r"security\s+apparatus\s+reshuffle", re.I),
|
||||
],
|
||||
"protest_mobilize": [
|
||||
re.compile(r"protest\s+mobil", re.I),
|
||||
re.compile(r"mass\s+rally", re.I),
|
||||
re.compile(r"general\s+strike", re.I),
|
||||
re.compile(r"\bstrike\b", re.I),
|
||||
re.compile(r"\brally\b", re.I),
|
||||
],
|
||||
"gps_jamming": [
|
||||
re.compile(r"gps\s+jam", re.I),
|
||||
re.compile(r"gnss\s+interference", re.I),
|
||||
re.compile(r"spoofing\s+spike", re.I),
|
||||
],
|
||||
"troop_movement": [
|
||||
re.compile(r"troop\s+movement", re.I),
|
||||
re.compile(r"military\s+mobil", re.I),
|
||||
re.compile(r"armored\s+convoy", re.I),
|
||||
re.compile(r"troop\s+buildup", re.I),
|
||||
],
|
||||
"bank_run": [
|
||||
re.compile(r"bank\s+run", re.I),
|
||||
re.compile(r"deposit\s+flight", re.I),
|
||||
re.compile(r"liquidity\s+crunch", re.I),
|
||||
],
|
||||
"sanctions_escalation": [
|
||||
re.compile(r"sanctions?\s+escalat", re.I),
|
||||
re.compile(r"new\s+sanctions?", re.I),
|
||||
re.compile(r"export\s+controls?\s+tighten", re.I),
|
||||
],
|
||||
"ceasefire_break": [
|
||||
re.compile(r"ceasefire\s+(broken|violated|collapse)", re.I),
|
||||
re.compile(r"truce\s+end", re.I),
|
||||
],
|
||||
}
|
||||
|
||||
_SIGNAL_DOMAINS: dict[str, DomainName] = {
|
||||
"payroll_loan": "financial",
|
||||
"supply_delay": "financial",
|
||||
"bank_run": "financial",
|
||||
"sanctions_escalation": "financial",
|
||||
"protest_mobilize": "unrest",
|
||||
"purge": "unrest",
|
||||
"elite_relocation": "financial",
|
||||
"gps_jamming": "conflict",
|
||||
"troop_movement": "conflict",
|
||||
"ceasefire_break": "conflict",
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class RegionState:
|
||||
"""Per-region Bayesian beliefs and metadata."""
|
||||
|
||||
priors: dict[DomainName, float] = field(default_factory=lambda: defaultdict(float))
|
||||
coords: list[float] | None = None
|
||||
signal_volume: DefaultDict[str, float] = field(default_factory=lambda: defaultdict(float))
|
||||
update_count: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class HistoryEntry:
|
||||
timestamp: str
|
||||
domain: DomainName
|
||||
signals: dict[str, float]
|
||||
strength: float
|
||||
prior: float
|
||||
posterior: float
|
||||
source: str
|
||||
deviation_score: float
|
||||
|
||||
|
||||
class GT_EarlyWarning:
|
||||
"""
|
||||
Game-Theoretic Early Warning System with Bayesian updating.
|
||||
|
||||
Tracks distress probabilities per region/domain, classifies costly signals vs
|
||||
cheap talk, and propagates risk through an entity interaction graph.
|
||||
"""
|
||||
|
||||
def __init__(self, settings: GTAnalyticsSettings | None = None) -> None:
|
||||
self.settings = settings or get_gt_settings()
|
||||
self.G: nx.Graph = nx.Graph()
|
||||
self._regions: dict[str, RegionState] = {}
|
||||
self._history: dict[str, list[HistoryEntry]] = defaultdict(list)
|
||||
self._seen_item_ids: set[str] = set()
|
||||
self._lock = threading.RLock()
|
||||
|
||||
self.likelihoods = dict(_DEFAULT_LIKELIHOODS)
|
||||
self.signal_weights = dict(_DEFAULT_SIGNAL_WEIGHTS)
|
||||
self.signal_weights.update(self.settings.signal_weight_overrides)
|
||||
|
||||
self._base_prior = float(self.settings.base_prior)
|
||||
|
||||
def _utcnow(self) -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
def _region_state(self, region: str) -> RegionState:
|
||||
key = str(region or "global").strip().lower() or "global"
|
||||
if key not in self._regions:
|
||||
state = RegionState()
|
||||
for domain in _DOMAINS:
|
||||
state.priors[domain] = self._base_prior
|
||||
self._regions[key] = state
|
||||
return self._regions[key]
|
||||
|
||||
def get_prior(self, region: str, domain: DomainName) -> float:
|
||||
with self._lock:
|
||||
return float(self._region_state(region).priors.get(domain, self._base_prior))
|
||||
|
||||
def set_prior(self, region: str, domain: DomainName, value: float) -> None:
|
||||
with self._lock:
|
||||
state = self._region_state(region)
|
||||
state.priors[domain] = float(
|
||||
np.clip(value, self.settings.min_prob, self.settings.max_prob)
|
||||
)
|
||||
|
||||
def composite_risk(self, region: str) -> float:
|
||||
"""Weighted composite across domains (conflict weighted highest)."""
|
||||
weights = {"financial": 0.25, "unrest": 0.35, "conflict": 0.40}
|
||||
with self._lock:
|
||||
state = self._region_state(region)
|
||||
total = 0.0
|
||||
weight_sum = 0.0
|
||||
for domain, weight in weights.items():
|
||||
total += float(state.priors.get(domain, self._base_prior)) * weight
|
||||
weight_sum += weight
|
||||
return float(total / weight_sum) if weight_sum else self._base_prior
|
||||
|
||||
def classify_signals(self, text: str, source: str = "") -> dict[str, float]:
|
||||
"""Return weighted costly-signal strengths detected in text."""
|
||||
text_lower = (text or "").lower()
|
||||
signals: dict[str, float] = {}
|
||||
|
||||
for signal_name, patterns in _SIGNAL_PATTERNS.items():
|
||||
weight = float(self.signal_weights.get(signal_name, 1.0))
|
||||
if any(pattern.search(text_lower) for pattern in patterns):
|
||||
signals[signal_name] = weight
|
||||
|
||||
rally_strike_count = text_lower.count("rally") + text_lower.count("strike")
|
||||
if rally_strike_count > 3:
|
||||
signals["protest_mobilize"] = signals.get("protest_mobilize", 0.0) + 1.5
|
||||
|
||||
# Source credibility nudge (Telegram OSINT channels treated as moderate-cost signals).
|
||||
if source and "t.me/" in source.lower() and signals:
|
||||
for key in list(signals):
|
||||
signals[key] = round(signals[key] * 1.05, 3)
|
||||
|
||||
return signals
|
||||
|
||||
def _deviation_score(self, region: str, domain: DomainName, strength: float) -> float:
|
||||
"""Deviation from rolling regional norm — herding/coordination detector input."""
|
||||
with self._lock:
|
||||
state = self._region_state(region)
|
||||
baseline = max(state.signal_volume[domain], 1.0)
|
||||
state.signal_volume[domain] += strength
|
||||
state.update_count += 1
|
||||
return float(strength / baseline)
|
||||
|
||||
def bayesian_update(
|
||||
self,
|
||||
region: str,
|
||||
domain: DomainName,
|
||||
evidence_strength: float = 1.0,
|
||||
) -> float:
|
||||
"""
|
||||
Bayesian update: P(distress|evidence) from likelihood table and prior.
|
||||
|
||||
evidence_strength scales how far belief moves toward the likelihood posterior.
|
||||
"""
|
||||
domain = domain if domain in _DOMAINS else "financial"
|
||||
lik = self.likelihoods.get(domain, self.likelihoods["financial"])
|
||||
|
||||
with self._lock:
|
||||
state = self._region_state(region)
|
||||
prior = float(state.priors.get(domain, self._base_prior))
|
||||
|
||||
p_e_given_d = lik["distress"]
|
||||
p_e_given_not_d = lik["normal"]
|
||||
p_e = (p_e_given_d * prior) + (p_e_given_not_d * (1.0 - prior))
|
||||
|
||||
if p_e <= 0:
|
||||
posterior = prior
|
||||
else:
|
||||
posterior = (p_e_given_d * prior) / p_e
|
||||
|
||||
scaled = prior + (posterior - prior) * float(evidence_strength)
|
||||
clipped = float(np.clip(scaled, self.settings.min_prob, self.settings.max_prob))
|
||||
state.priors[domain] = clipped
|
||||
return clipped
|
||||
|
||||
def _update_graph(
|
||||
self,
|
||||
region: str,
|
||||
entities: list[str],
|
||||
strength: float,
|
||||
coords: list[float] | None,
|
||||
) -> None:
|
||||
region_key = str(region or "global").strip().lower() or "global"
|
||||
self.G.add_node(region_key, node_type="region", region=region_key)
|
||||
if coords and len(coords) >= 2:
|
||||
self.G.nodes[region_key]["coords"] = coords
|
||||
|
||||
for entity in entities:
|
||||
entity_key = str(entity).strip()
|
||||
if not entity_key:
|
||||
continue
|
||||
self.G.add_node(entity_key, node_type="entity", region=region_key)
|
||||
self.G.add_edge(
|
||||
region_key,
|
||||
entity_key,
|
||||
weight=float(strength),
|
||||
timestamp=self._utcnow(),
|
||||
)
|
||||
|
||||
for i, e1 in enumerate(entities):
|
||||
for e2 in entities[i + 1 :]:
|
||||
k1, k2 = str(e1).strip(), str(e2).strip()
|
||||
if not k1 or not k2:
|
||||
continue
|
||||
self.G.add_edge(
|
||||
k1,
|
||||
k2,
|
||||
weight=float(strength),
|
||||
timestamp=self._utcnow(),
|
||||
)
|
||||
|
||||
def process_feed_item(self, item: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Process one normalized feed item and update beliefs + contagion graph."""
|
||||
region = str(item.get("region") or item.get("geotag") or "global").strip().lower()
|
||||
text = str(item.get("text") or "")
|
||||
source = str(item.get("source") or "unknown")
|
||||
explicit_domain = str(item.get("domain") or "").strip().lower()
|
||||
entities = list(item.get("entities") or [])
|
||||
coords = item.get("coords")
|
||||
item_id = str(item.get("id") or f"{source}|{hash(text)}")
|
||||
|
||||
if self.settings.watched_channels:
|
||||
channel = ""
|
||||
for entity in entities:
|
||||
if str(entity).startswith("channel:"):
|
||||
channel = str(entity).split(":", 1)[-1].lower()
|
||||
break
|
||||
if channel and channel not in {c.lower() for c in self.settings.watched_channels}:
|
||||
return {
|
||||
"region": region,
|
||||
"skipped": True,
|
||||
"reason": "channel_not_watched",
|
||||
"risk_score": self.composite_risk(region),
|
||||
"signals": {},
|
||||
}
|
||||
|
||||
with self._lock:
|
||||
if item_id and item_id in self._seen_item_ids:
|
||||
return {
|
||||
"region": region,
|
||||
"skipped": True,
|
||||
"reason": "duplicate",
|
||||
"risk_score": self.composite_risk(region),
|
||||
"signals": {},
|
||||
}
|
||||
if item_id:
|
||||
self._seen_item_ids.add(item_id)
|
||||
|
||||
signals = self.classify_signals(text, source)
|
||||
total_strength = float(sum(signals.values()))
|
||||
|
||||
if total_strength <= 0:
|
||||
return {
|
||||
"region": region,
|
||||
"risk_score": self.composite_risk(region),
|
||||
"signals": {},
|
||||
"contagion_potential": self._get_contagion_score(region),
|
||||
}
|
||||
|
||||
domains_touched: set[DomainName] = set()
|
||||
if explicit_domain in _DOMAINS:
|
||||
domains_touched.add(explicit_domain)
|
||||
for signal_name in signals:
|
||||
domains_touched.add(_SIGNAL_DOMAINS.get(signal_name, explicit_domain or "financial"))
|
||||
if not domains_touched:
|
||||
domains_touched.add("financial")
|
||||
|
||||
evidence_strength = min(
|
||||
total_strength / max(self.settings.evidence_scale, 0.1),
|
||||
self.settings.evidence_cap,
|
||||
)
|
||||
|
||||
posteriors: dict[str, float] = {}
|
||||
deviation = 0.0
|
||||
for domain in domains_touched:
|
||||
prior = self.get_prior(region, domain)
|
||||
deviation = max(deviation, self._deviation_score(region, domain, total_strength))
|
||||
posterior = self.bayesian_update(
|
||||
region=region,
|
||||
domain=domain,
|
||||
evidence_strength=evidence_strength * (1.0 + 0.15 * deviation),
|
||||
)
|
||||
posteriors[domain] = posterior
|
||||
|
||||
if isinstance(coords, (list, tuple)) and len(coords) >= 2:
|
||||
with self._lock:
|
||||
state = self._region_state(region)
|
||||
try:
|
||||
state.coords = [float(coords[0]), float(coords[1])]
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
self._update_graph(region, entities, total_strength, coords if isinstance(coords, list) else None)
|
||||
|
||||
composite = self.composite_risk(region)
|
||||
entry = HistoryEntry(
|
||||
timestamp=self._utcnow(),
|
||||
domain=explicit_domain if explicit_domain in _DOMAINS else next(iter(domains_touched)),
|
||||
signals=signals,
|
||||
strength=total_strength,
|
||||
prior=self._base_prior,
|
||||
posterior=composite,
|
||||
source=source,
|
||||
deviation_score=deviation,
|
||||
)
|
||||
with self._lock:
|
||||
history = self._history[region]
|
||||
history.append(entry)
|
||||
max_hist = max(10, int(self.settings.max_history_per_region))
|
||||
if len(history) > max_hist:
|
||||
self._history[region] = history[-max_hist:]
|
||||
|
||||
logger.info(
|
||||
"GT update region=%s domains=%s composite=%.3f signals=%d deviation=%.2f",
|
||||
region,
|
||||
",".join(sorted(domains_touched)),
|
||||
composite,
|
||||
len(signals),
|
||||
deviation,
|
||||
)
|
||||
|
||||
return {
|
||||
"region": region,
|
||||
"domains": sorted(domains_touched),
|
||||
"domain_posteriors": posteriors,
|
||||
"risk_score": composite,
|
||||
"signals": signals,
|
||||
"deviation_score": deviation,
|
||||
"contagion_potential": self._get_contagion_score(region),
|
||||
"interpretation": self._interpret_risk(composite),
|
||||
}
|
||||
|
||||
def _interpret_risk(self, risk: float) -> str:
|
||||
threshold = float(self.settings.high_risk_threshold)
|
||||
if risk >= threshold:
|
||||
return (
|
||||
f"Elevated strategic risk ({risk:.2f} ≥ {threshold:.2f}). "
|
||||
"Watch for costly-signal clustering and cross-region contagion."
|
||||
)
|
||||
if risk >= threshold * 0.7:
|
||||
return "Moderate risk — monitor for herding and repeated costly signals."
|
||||
return "Baseline risk — no strong costly-signal cluster detected."
|
||||
|
||||
def _get_contagion_score(self, region: str) -> float:
|
||||
"""Graph-based contagion: mean composite risk of graph neighbors."""
|
||||
region_key = str(region or "global").strip().lower() or "global"
|
||||
with self._lock:
|
||||
if region_key not in self.G:
|
||||
return 0.0
|
||||
try:
|
||||
neighbors = list(self.G.neighbors(region_key))
|
||||
except nx.NetworkXError:
|
||||
return 0.0
|
||||
if not neighbors:
|
||||
return 0.0
|
||||
neighbor_risks = [self.composite_risk(str(n)) for n in neighbors]
|
||||
return float(np.mean(neighbor_risks))
|
||||
|
||||
def compute_herding_clusters(self) -> list[dict[str, Any]]:
|
||||
"""Louvain community detection on entity graph (coordination/herding proxy)."""
|
||||
with self._lock:
|
||||
if self.G.number_of_edges() == 0:
|
||||
return []
|
||||
|
||||
weighted = nx.Graph()
|
||||
for u, v, data in self.G.edges(data=True):
|
||||
weight = float(data.get("weight") or 0.0)
|
||||
if weight < self.settings.louvain_min_weight:
|
||||
continue
|
||||
if weighted.has_edge(u, v):
|
||||
weighted[u][v]["weight"] = weighted[u][v].get("weight", 0.0) + weight
|
||||
else:
|
||||
weighted.add_edge(u, v, weight=weight)
|
||||
|
||||
if weighted.number_of_edges() == 0:
|
||||
return []
|
||||
|
||||
try:
|
||||
communities = list(nx.community.louvain_communities(weighted, weight="weight", seed=42))
|
||||
except Exception as exc:
|
||||
logger.warning("Louvain clustering failed: %s", exc)
|
||||
return []
|
||||
|
||||
clusters: list[dict[str, Any]] = []
|
||||
for idx, community in enumerate(communities):
|
||||
members = sorted(str(node) for node in community)
|
||||
region_members = [m for m in members if m in self._regions]
|
||||
risks = [self.composite_risk(r) for r in region_members]
|
||||
clusters.append(
|
||||
{
|
||||
"cluster_id": idx,
|
||||
"size": len(members),
|
||||
"members": members[:50],
|
||||
"mean_risk": float(np.mean(risks)) if risks else self._base_prior,
|
||||
"regions": region_members,
|
||||
}
|
||||
)
|
||||
clusters.sort(key=lambda row: row["mean_risk"], reverse=True)
|
||||
return clusters
|
||||
|
||||
def get_risk_heatmap(self) -> dict[str, Any]:
|
||||
"""GeoJSON FeatureCollection for frontend risk overlay."""
|
||||
features: list[dict[str, Any]] = []
|
||||
with self._lock:
|
||||
items = list(self._regions.items())[: max(1, self.settings.max_heatmap_features)]
|
||||
|
||||
for region, state in items:
|
||||
coords = state.coords
|
||||
geometry: dict[str, Any]
|
||||
if coords and len(coords) >= 2:
|
||||
geometry = {"type": "Point", "coordinates": [float(coords[1]), float(coords[0])]}
|
||||
else:
|
||||
geometry = {"type": "Point", "coordinates": [0.0, 0.0]}
|
||||
|
||||
composite = self.composite_risk(region)
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"region": region,
|
||||
"risk": round(composite, 4),
|
||||
"financial": round(float(state.priors.get("financial", self._base_prior)), 4),
|
||||
"unrest": round(float(state.priors.get("unrest", self._base_prior)), 4),
|
||||
"conflict": round(float(state.priors.get("conflict", self._base_prior)), 4),
|
||||
"contagion": round(self._get_contagion_score(region), 4),
|
||||
"updates": state.update_count,
|
||||
},
|
||||
"geometry": geometry,
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
def get_dossier(self, region: str) -> dict[str, Any]:
|
||||
"""Explainable GT rationale and recent signal history for a region."""
|
||||
region_key = str(region or "global").strip().lower() or "global"
|
||||
with self._lock:
|
||||
state = self._region_state(region_key)
|
||||
recent = list(self._history.get(region_key, [])[-10:])
|
||||
|
||||
composite = self.composite_risk(region_key)
|
||||
return {
|
||||
"region": region_key,
|
||||
"current_risk": round(composite, 4),
|
||||
"domain_risks": {
|
||||
domain: round(float(state.priors.get(domain, self._base_prior)), 4)
|
||||
for domain in _DOMAINS
|
||||
},
|
||||
"recent_signals": [
|
||||
{
|
||||
"timestamp": entry.timestamp,
|
||||
"domain": entry.domain,
|
||||
"signals": entry.signals,
|
||||
"strength": entry.strength,
|
||||
"posterior": round(entry.posterior, 4),
|
||||
"source": entry.source,
|
||||
"deviation_score": round(entry.deviation_score, 3),
|
||||
}
|
||||
for entry in recent
|
||||
],
|
||||
"contagion_risk": round(self._get_contagion_score(region_key), 4),
|
||||
"herding_clusters": self.compute_herding_clusters()[:5],
|
||||
"interpretation": self._interpret_risk(composite),
|
||||
"scenarios": self._build_scenarios(region_key, composite),
|
||||
}
|
||||
|
||||
def _build_scenarios(self, region: str, composite: float) -> list[dict[str, str]]:
|
||||
threshold = float(self.settings.high_risk_threshold)
|
||||
if composite < threshold * 0.7:
|
||||
return [
|
||||
{
|
||||
"name": "Status quo",
|
||||
"summary": "Signals remain diffuse; no coordinated costly-signal cascade.",
|
||||
}
|
||||
]
|
||||
if composite < threshold:
|
||||
return [
|
||||
{
|
||||
"name": "Escalation watch",
|
||||
"summary": "Rising costly-signal density — coordination risk within 4-8 weeks.",
|
||||
},
|
||||
{
|
||||
"name": "False alarm",
|
||||
"summary": "Cheap-talk amplification without follow-on costly signals.",
|
||||
},
|
||||
]
|
||||
return [
|
||||
{
|
||||
"name": "Contagion spread",
|
||||
"summary": "High posterior + graph coupling — adjacent regions likely to update upward.",
|
||||
},
|
||||
{
|
||||
"name": "Localized shock",
|
||||
"summary": "Region-specific distress; contagion limited if graph neighbors stay quiet.",
|
||||
},
|
||||
]
|
||||
|
||||
def snapshot(self) -> dict[str, Any]:
|
||||
"""Serialize engine state for debugging or persistence."""
|
||||
with self._lock:
|
||||
return {
|
||||
"regions": {
|
||||
region: {
|
||||
"priors": dict(state.priors),
|
||||
"coords": state.coords,
|
||||
"updates": state.update_count,
|
||||
}
|
||||
for region, state in self._regions.items()
|
||||
},
|
||||
"graph_nodes": self.G.number_of_nodes(),
|
||||
"graph_edges": self.G.number_of_edges(),
|
||||
"processed_items": len(self._seen_item_ids),
|
||||
}
|
||||
@@ -0,0 +1,649 @@
|
||||
"""Curated historical early-warning cases for GT backtesting.
|
||||
|
||||
Each positive case bundles pre-crisis costly-signal snippets drawn from documented
|
||||
precursors (financial, unrest, conflict). Negative cases are cheap-talk controls.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
CaseKind = Literal["positive", "negative"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BacktestFeed:
|
||||
text: str
|
||||
source: str = "backtest"
|
||||
domain: str = "financial"
|
||||
days_before_event: int = 30
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HistoricalCase:
|
||||
"""Single labeled backtest scenario."""
|
||||
|
||||
case_id: str
|
||||
name: str
|
||||
region: str
|
||||
domain: str
|
||||
kind: CaseKind
|
||||
event_date: str
|
||||
description: str
|
||||
feeds: tuple[BacktestFeed, ...] = field(default_factory=tuple)
|
||||
tags: tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
def to_feed_dicts(self) -> list[dict[str, Any]]:
|
||||
items: list[dict[str, Any]] = []
|
||||
for idx, feed in enumerate(self.feeds):
|
||||
items.append(
|
||||
{
|
||||
"id": f"{self.case_id}-{idx}",
|
||||
"text": feed.text,
|
||||
"source": feed.source,
|
||||
"region": self.region,
|
||||
"domain": feed.domain or self.domain,
|
||||
"published": feed.days_before_event,
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def _variant_case(case: HistoricalCase, suffix: str, feeds: tuple[BacktestFeed, ...]) -> HistoricalCase:
|
||||
return HistoricalCase(
|
||||
case_id=f"{case.case_id}__{suffix}",
|
||||
name=f"{case.name} ({suffix})",
|
||||
region=case.region,
|
||||
domain=case.domain,
|
||||
kind=case.kind,
|
||||
event_date=case.event_date,
|
||||
description=case.description,
|
||||
feeds=feeds,
|
||||
tags=case.tags + (f"variant:{suffix}",),
|
||||
)
|
||||
|
||||
|
||||
def expanded_historical_cases() -> tuple[HistoricalCase, ...]:
|
||||
"""Base suite plus paraphrase variants for statistical confidence."""
|
||||
base = list(default_historical_cases())
|
||||
extras: list[HistoricalCase] = []
|
||||
|
||||
variant_feeds: dict[str, tuple[tuple[BacktestFeed, ...], ...]] = {
|
||||
"fin_2008_us": (
|
||||
(
|
||||
BacktestFeed(
|
||||
"Small businesses turn to payroll loan products as credit lines freeze.",
|
||||
domain="financial",
|
||||
days_before_event=100,
|
||||
),
|
||||
BacktestFeed(
|
||||
"FDIC monitors liquidity crunch; interbank spreads widen sharply.",
|
||||
domain="financial",
|
||||
days_before_event=60,
|
||||
),
|
||||
),
|
||||
(
|
||||
BacktestFeed(
|
||||
"Merchant cash advance volumes spike; payroll loan demand at record highs.",
|
||||
domain="financial",
|
||||
days_before_event=80,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Money market funds see inflows as deposit flight from regional banks continues.",
|
||||
domain="financial",
|
||||
days_before_event=40,
|
||||
),
|
||||
),
|
||||
),
|
||||
"fin_2020_supply": (
|
||||
(
|
||||
BacktestFeed(
|
||||
"Electronics firms report shipping delay and port congestion across Pearl River Delta.",
|
||||
domain="financial",
|
||||
days_before_event=45,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Supply chain delay widens; logistics backlog hits automotive suppliers.",
|
||||
domain="financial",
|
||||
days_before_event=20,
|
||||
),
|
||||
),
|
||||
(
|
||||
BacktestFeed(
|
||||
"Container shortage fuels shipping delay; supply chain delay indices jump.",
|
||||
domain="financial",
|
||||
days_before_event=35,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Electronics assemblers warn of logistics backlog as port congestion spreads.",
|
||||
domain="financial",
|
||||
days_before_event=20,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Automotive suppliers flag supply chain delay after factory shutdowns in Hubei.",
|
||||
domain="financial",
|
||||
days_before_event=10,
|
||||
),
|
||||
),
|
||||
),
|
||||
"fin_2022_sanctions": (
|
||||
(
|
||||
BacktestFeed(
|
||||
"Treasury drafts new sanctions escalation package on energy and finance sectors.",
|
||||
domain="financial",
|
||||
days_before_event=30,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Capital flight accelerates; elite relocation flights depart Moscow airports.",
|
||||
domain="financial",
|
||||
days_before_event=14,
|
||||
),
|
||||
),
|
||||
),
|
||||
"unrest_arab_spring_egypt": (
|
||||
(
|
||||
BacktestFeed(
|
||||
"Cairo activists schedule mass rally; protest mobilization leaflets distributed.",
|
||||
domain="unrest",
|
||||
days_before_event=18,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Labor federations call general strike; strike posters cover downtown.",
|
||||
domain="unrest",
|
||||
days_before_event=8,
|
||||
),
|
||||
),
|
||||
),
|
||||
"conflict_2022_ukraine": (
|
||||
(
|
||||
BacktestFeed(
|
||||
"Convoy of armored vehicles confirms troop movement near Sumy Oblast.",
|
||||
source="t.me/war_monitor",
|
||||
domain="conflict",
|
||||
days_before_event=20,
|
||||
),
|
||||
BacktestFeed(
|
||||
"GNSS interference warnings follow GPS jamming spike along Belarus border.",
|
||||
source="t.me/osintdefender",
|
||||
domain="conflict",
|
||||
days_before_event=10,
|
||||
),
|
||||
),
|
||||
(
|
||||
BacktestFeed(
|
||||
"Military mobilization notices circulate; troop buildup confirmed by satellite firms.",
|
||||
domain="conflict",
|
||||
days_before_event=12,
|
||||
),
|
||||
),
|
||||
),
|
||||
"neg_weather_us": (
|
||||
(
|
||||
BacktestFeed("Autumn foliage peaks in Vermont; pleasant hiking weather continues."),
|
||||
BacktestFeed("County fair announces pie contest and livestock exhibitions."),
|
||||
),
|
||||
(
|
||||
BacktestFeed("Meteorologists predict mild hurricane season remainder for Gulf Coast."),
|
||||
),
|
||||
),
|
||||
"neg_sports_uk": (
|
||||
(
|
||||
BacktestFeed("Rugby Six Nations standings update after weekend fixtures."),
|
||||
BacktestFeed("Local marathon registration opens for charity runners."),
|
||||
),
|
||||
),
|
||||
"neg_tech_global": (
|
||||
(
|
||||
BacktestFeed("Chipmaker announces efficiency gains in next-generation processor."),
|
||||
BacktestFeed("Cloud provider opens new green datacenter in Nordic region."),
|
||||
),
|
||||
),
|
||||
}
|
||||
|
||||
for case in base:
|
||||
variants = variant_feeds.get(case.case_id, ())
|
||||
for idx, feeds in enumerate(variants):
|
||||
extras.append(_variant_case(case, f"v{idx+1}", feeds))
|
||||
|
||||
# Additional cheap-talk controls to widen negative sample
|
||||
cheap_talk_regions = (
|
||||
("australia", "Museum opens contemporary art exhibit to strong attendance."),
|
||||
("spain", "Tomato harvest festival scheduled; regional trains add weekend service."),
|
||||
("south_korea", "K-pop group announces world tour dates for autumn."),
|
||||
("mexico", "Coastal cleanup volunteers restore beach habitats before holiday season."),
|
||||
("sweden", "City council approves bike lane expansion along waterfront."),
|
||||
("norway", "Salmon exports remain stable; fishing fleets report normal catch volumes."),
|
||||
("italy", "Truffle festival returns; restaurants publish seasonal tasting menus."),
|
||||
("poland", "University researchers release open-source astronomy software."),
|
||||
("thailand", "Monsoon rains ease; rice planting proceeds on normal schedule."),
|
||||
("vietnam", "Electronics assembly plants report steady export order books."),
|
||||
("south_africa", "Wildlife reserve reports rising ecotourism bookings."),
|
||||
("argentina", "Wine harvest festival opens; export cooperatives meet volume targets."),
|
||||
("netherlands", "Cycling championship draws international teams to canal district."),
|
||||
("belgium", "Chocolate exporters report stable holiday shipment schedules."),
|
||||
("portugal", "Offshore wind auction attracts multiple renewable bidders."),
|
||||
("greece", "Island ferry operators add routes ahead of summer travel season."),
|
||||
("turkey", "Cotton harvest forecast unchanged; textile orders stable."),
|
||||
("indonesia", "Volcano monitoring reports routine activity; tourism continues."),
|
||||
("philippines", "Coconut processors report normal logistics to export markets."),
|
||||
("malaysia", "Palm oil shipments on schedule; port throughput normal."),
|
||||
("new_zealand", "Sheep shearing competition draws rural crowds."),
|
||||
("ireland", "Tech conference highlights open-source database tooling."),
|
||||
("finland", "Sauna culture festival celebrates heritage with local artisans."),
|
||||
("denmark", "Wind turbine maintenance contracts renewed on prior terms."),
|
||||
("austria", "Ski resorts prepare slopes after early snowfall."),
|
||||
("switzerland", "Watchmakers unveil mechanical movement prototypes at trade fair."),
|
||||
("czech_republic", "Glassmakers export decorative pieces ahead of holiday season."),
|
||||
("romania", "Carpathian hiking trails reopen after spring maintenance."),
|
||||
("hungary", "Thermal bath tourism bookings rise for winter wellness season."),
|
||||
("peru", "Coffee cooperatives report stable harvest and export schedules."),
|
||||
("colombia", "Flower exporters prepare Valentine's shipments on normal cadence."),
|
||||
("morocco", "Citrus harvest meets forecasts; agricultural credit unchanged."),
|
||||
("kenya", "Tea auction volumes steady; freight routes operate normally."),
|
||||
("nigeria", "Nollywood studio announces family comedy release dates."),
|
||||
("ethiopia", "Coffee ceremony festival highlights regional bean varieties."),
|
||||
("saudi_arabia", "Desert conservation project plants drought-resistant shrubs."),
|
||||
("uae", "Airport duty-free operators expand luxury retail concourse."),
|
||||
("qatar", "Stadium operators prepare hospitality packages for sporting events."),
|
||||
("singapore", "Port authority reports container throughput on seasonal trend."),
|
||||
("hong_kong", "Art auction previews draw collectors to harborfront gallery."),
|
||||
("chile", "Vineyard tours report strong bookings ahead of harvest festival weekend."),
|
||||
("uruguay", "Beef exporters maintain steady shipment schedules to European buyers."),
|
||||
("iceland", "Geothermal spa resorts report normal winter visitor volumes."),
|
||||
("luxembourg", "Fund administrators publish routine quarterly disclosure filings."),
|
||||
("slovakia", "Mountain lodges prepare ski season openings after early snowfall."),
|
||||
("croatia", "Adriatic ferry operators add summer routes on prior timetable."),
|
||||
("bulgaria", "Rose oil cooperatives report stable export volumes to fragrance buyers."),
|
||||
("serbia", "Danube barge traffic proceeds on normal freight schedules."),
|
||||
("latvia", "Timber mills export lumber on unchanged contract terms."),
|
||||
("lithuania", "Baltic wind farms complete scheduled turbine maintenance rotations."),
|
||||
("estonia", "Digital residency applications processed at routine monthly pace."),
|
||||
("panama", "Canal transit volumes remain on seasonal trend; shipping fees unchanged."),
|
||||
)
|
||||
for idx, (region, text) in enumerate(cheap_talk_regions):
|
||||
extras.append(
|
||||
HistoricalCase(
|
||||
case_id=f"neg_extra_{idx:02d}",
|
||||
name=f"Benign regional news ({region})",
|
||||
region=region,
|
||||
domain="financial",
|
||||
kind="negative",
|
||||
event_date="2020-01-01",
|
||||
description="Expanded cheap-talk control.",
|
||||
feeds=(BacktestFeed(text),),
|
||||
tags=("control", "expanded"),
|
||||
)
|
||||
)
|
||||
|
||||
return tuple(base + extras)
|
||||
|
||||
|
||||
def default_historical_cases() -> tuple[HistoricalCase, ...]:
|
||||
"""Benchmark suite — expand as new validated precursors are added."""
|
||||
return (
|
||||
# ── Financial distress ─────────────────────────────────────────────
|
||||
HistoricalCase(
|
||||
case_id="fin_2008_us",
|
||||
name="2008 US financial crisis",
|
||||
region="united_states",
|
||||
domain="financial",
|
||||
kind="positive",
|
||||
event_date="2008-09-15",
|
||||
description="Payroll-loan distress, liquidity crunch, and deposit flight precursors.",
|
||||
tags=("2008", "financial", "lehman"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Franchise operators increasingly rely on payroll loan facilities as working capital tightens.",
|
||||
domain="financial",
|
||||
days_before_event=120,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Regional banks report liquidity crunch; CFOs warn of merchant cash advance reliance.",
|
||||
domain="financial",
|
||||
days_before_event=90,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Deposit flight accelerates at mid-size lenders; analysts flag bank run risk.",
|
||||
domain="financial",
|
||||
days_before_event=45,
|
||||
),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="fin_2020_supply",
|
||||
name="COVID supply-chain shock",
|
||||
region="china",
|
||||
domain="financial",
|
||||
kind="positive",
|
||||
event_date="2020-02-01",
|
||||
description="Port congestion and logistics backlog ahead of global supply shock.",
|
||||
tags=("covid", "supply_chain", "financial"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Major port congestion reported; shipping delay spreads to electronics suppliers.",
|
||||
domain="financial",
|
||||
days_before_event=60,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Automakers warn of supply chain delay and logistics backlog across Wuhan corridor.",
|
||||
domain="financial",
|
||||
days_before_event=30,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Factory restarts slip as supply delay and port congestion persist into Q1.",
|
||||
domain="financial",
|
||||
days_before_event=14,
|
||||
),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="fin_2022_sanctions",
|
||||
name="Russia sanctions escalation",
|
||||
region="russia",
|
||||
domain="financial",
|
||||
kind="positive",
|
||||
event_date="2022-02-24",
|
||||
description="Sanctions escalation and capital flight ahead of invasion.",
|
||||
tags=("sanctions", "ukraine", "financial"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Western allies prepare new sanctions escalation on major Russian banks.",
|
||||
domain="financial",
|
||||
days_before_event=45,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Oligarch jet movements suggest elite relocation and capital flight from Moscow.",
|
||||
domain="financial",
|
||||
days_before_event=21,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Central bank intervenes as new sanctions tighten export controls on finance sector.",
|
||||
domain="financial",
|
||||
days_before_event=10,
|
||||
),
|
||||
),
|
||||
),
|
||||
# ── Civil unrest ─────────────────────────────────────────────────
|
||||
HistoricalCase(
|
||||
case_id="unrest_arab_spring_tunisia",
|
||||
name="Arab Spring — Tunisia",
|
||||
region="tunisia",
|
||||
domain="unrest",
|
||||
kind="positive",
|
||||
event_date="2010-12-17",
|
||||
description="Protest mobilization and strike waves before Jasmine Revolution.",
|
||||
tags=("arab_spring", "unrest"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Student groups announce protest mobilization after vendor self-immolation.",
|
||||
domain="unrest",
|
||||
days_before_event=14,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Mass rally planned in Tunis; general strike called by labor unions.",
|
||||
domain="unrest",
|
||||
days_before_event=7,
|
||||
),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="unrest_arab_spring_egypt",
|
||||
name="Arab Spring — Egypt",
|
||||
region="egypt",
|
||||
domain="unrest",
|
||||
kind="positive",
|
||||
event_date="2011-01-25",
|
||||
description="Mobilization spikes and security reshuffles before Tahrir.",
|
||||
tags=("arab_spring", "unrest"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Opposition calls protest mobilization in Cairo; strike notices circulate online.",
|
||||
domain="unrest",
|
||||
days_before_event=21,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Reports of political purge within interior ministry security apparatus reshuffle.",
|
||||
domain="unrest",
|
||||
days_before_event=10,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Mass rally and strike coordination spreads; rally posters appear in Alexandria.",
|
||||
domain="unrest",
|
||||
days_before_event=5,
|
||||
),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="unrest_2019_chile",
|
||||
name="Chile 2019 metro protests",
|
||||
region="chile",
|
||||
domain="unrest",
|
||||
kind="positive",
|
||||
event_date="2019-10-18",
|
||||
description="Transit fare protests escalate to general strike.",
|
||||
tags=("unrest", "latam"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Students organize mass rally after metro fare hike; protest mobilization trending.",
|
||||
domain="unrest",
|
||||
days_before_event=10,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Unions announce general strike; rally and strike hashtags spike nationwide.",
|
||||
domain="unrest",
|
||||
days_before_event=3,
|
||||
),
|
||||
),
|
||||
),
|
||||
# ── Conflict / war ───────────────────────────────────────────────
|
||||
HistoricalCase(
|
||||
case_id="conflict_2022_ukraine",
|
||||
name="2022 Ukraine invasion buildup",
|
||||
region="ukraine",
|
||||
domain="conflict",
|
||||
kind="positive",
|
||||
event_date="2022-02-24",
|
||||
description="Troop movement and GPS jamming precursors on northern border.",
|
||||
tags=("ukraine", "conflict"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"OSINT reports troop movement and armored convoy near Belarus border.",
|
||||
source="t.me/war_monitor",
|
||||
domain="conflict",
|
||||
days_before_event=30,
|
||||
),
|
||||
BacktestFeed(
|
||||
"GPS jamming spike reported along northern corridor; GNSS interference warnings issued.",
|
||||
source="t.me/osintdefender",
|
||||
domain="conflict",
|
||||
days_before_event=14,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Satellite imagery shows troop buildup; military mobilization near Kharkiv axis.",
|
||||
domain="conflict",
|
||||
days_before_event=7,
|
||||
),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="conflict_2023_gaza",
|
||||
name="2023 Gaza conflict escalation",
|
||||
region="israel",
|
||||
domain="conflict",
|
||||
kind="positive",
|
||||
event_date="2023-10-07",
|
||||
description="Ceasefire breakdown and troop movement signals.",
|
||||
tags=("gaza", "conflict"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Border units report troop movement near Gaza envelope; ceasefire broken overnight.",
|
||||
domain="conflict",
|
||||
days_before_event=14,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Truce end announced; armored convoy repositioning reported by local observers.",
|
||||
domain="conflict",
|
||||
days_before_event=5,
|
||||
),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="conflict_2020_nagorno",
|
||||
name="2020 Nagorno-Karabakh renewal",
|
||||
region="armenia",
|
||||
domain="conflict",
|
||||
kind="positive",
|
||||
event_date="2020-09-27",
|
||||
description="Artillery and troop buildup precursors.",
|
||||
tags=("caucasus", "conflict"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Drone strikes reported on line of contact; troop movement on Armenian-Azeri border.",
|
||||
domain="conflict",
|
||||
days_before_event=21,
|
||||
),
|
||||
BacktestFeed(
|
||||
"GPS jamming spike reported in conflict zone; military mobilization notices leaked.",
|
||||
domain="conflict",
|
||||
days_before_event=7,
|
||||
),
|
||||
),
|
||||
),
|
||||
# ── Recent financial / corporate distress pattern ────────────────
|
||||
HistoricalCase(
|
||||
case_id="fin_2023_banking",
|
||||
name="2023 regional banking stress",
|
||||
region="united_states",
|
||||
domain="financial",
|
||||
kind="positive",
|
||||
event_date="2023-03-10",
|
||||
description="Deposit flight and liquidity stress (SVB precursor pattern).",
|
||||
tags=("svb", "financial", "2023"),
|
||||
feeds=(
|
||||
BacktestFeed(
|
||||
"Tech lenders face deposit flight; VC portfolio companies move payroll to money market funds.",
|
||||
domain="financial",
|
||||
days_before_event=21,
|
||||
),
|
||||
BacktestFeed(
|
||||
"Analysts warn liquidity crunch at regional banks holding long-duration bonds.",
|
||||
domain="financial",
|
||||
days_before_event=7,
|
||||
),
|
||||
),
|
||||
),
|
||||
# ── Negative controls (cheap talk / benign) ─────────────────────
|
||||
HistoricalCase(
|
||||
case_id="neg_weather_us",
|
||||
name="Benign weather coverage",
|
||||
region="united_states",
|
||||
domain="financial",
|
||||
kind="negative",
|
||||
event_date="2019-06-01",
|
||||
description="No costly signals — should remain near baseline.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("Sunny weekend expected across the Midwest with mild temperatures."),
|
||||
BacktestFeed("Local festival draws crowds; farmers market expands summer hours."),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="neg_sports_uk",
|
||||
name="Benign sports coverage",
|
||||
region="uk",
|
||||
domain="unrest",
|
||||
kind="negative",
|
||||
event_date="2018-07-01",
|
||||
description="Sports chatter without mobilization costly signals.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("Premier league season review: top scorers and transfer rumors."),
|
||||
BacktestFeed("Cricket test match ends early due to rain delay at Lord's."),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="neg_tech_global",
|
||||
name="Benign tech product launch",
|
||||
region="global",
|
||||
domain="financial",
|
||||
kind="negative",
|
||||
event_date="2021-09-01",
|
||||
description="Corporate product news without distress markers.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("Smartphone maker unveils new camera features at annual keynote."),
|
||||
BacktestFeed("Quarterly earnings beat expectations; dividend unchanged."),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="neg_tourism_france",
|
||||
name="Benign tourism recovery",
|
||||
region="france",
|
||||
domain="unrest",
|
||||
kind="negative",
|
||||
event_date="2022-08-01",
|
||||
description="Travel sector recovery without unrest signals.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("Paris hotels report record summer bookings as tourism rebounds."),
|
||||
BacktestFeed("Airline adds routes to Nice and Marseille for holiday travelers."),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="neg_science_japan",
|
||||
name="Benign science news",
|
||||
region="japan",
|
||||
domain="conflict",
|
||||
kind="negative",
|
||||
event_date="2020-11-01",
|
||||
description="Research coverage without conflict markers.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("Astronomy team publishes comet observations from Mount Fuji observatory."),
|
||||
BacktestFeed("Robotics lab demonstrates warehouse automation prototype."),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="neg_agriculture_brazil",
|
||||
name="Benign agriculture report",
|
||||
region="brazil",
|
||||
domain="financial",
|
||||
kind="negative",
|
||||
event_date="2017-03-01",
|
||||
description="Commodity harvest update without supply distress.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("Soybean harvest forecast revised upward; export volumes steady."),
|
||||
BacktestFeed("Coffee cooperative reports normal shipping schedules to European buyers."),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="neg_culture_india",
|
||||
name="Benign culture coverage",
|
||||
region="india",
|
||||
domain="unrest",
|
||||
kind="negative",
|
||||
event_date="2016-11-01",
|
||||
description="Festival coverage without mobilization.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("Diwali celebrations begin; cities decorate markets with lights."),
|
||||
BacktestFeed("Film festival opens in Mumbai with premiere screenings."),
|
||||
),
|
||||
),
|
||||
HistoricalCase(
|
||||
case_id="neg_infrastructure_canada",
|
||||
name="Benign infrastructure ribbon-cutting",
|
||||
region="canada",
|
||||
domain="financial",
|
||||
kind="negative",
|
||||
event_date="2015-05-01",
|
||||
description="Municipal news without financial stress.",
|
||||
tags=("control",),
|
||||
feeds=(
|
||||
BacktestFeed("New light-rail segment opens on schedule; commute times improve."),
|
||||
BacktestFeed("Municipal bond issuance funds library renovation at prior rates."),
|
||||
),
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,198 @@
|
||||
"""Singleton GT engine and feed-batch integration hooks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from analytics.feed_adapter import iter_gdelt_features, iter_news_items, iter_telegram_posts
|
||||
from analytics.gt_early_warning import GT_EarlyWarning
|
||||
from analytics.settings import gt_analytics_enabled, get_gt_settings, gt_engine_operational, gt_louvain_enabled, gt_scheduled_ingest_enabled
|
||||
from services.fetchers._store import _data_lock, _mark_fresh, latest_data
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_engine: GT_EarlyWarning | None = None
|
||||
_engine_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_gt_engine() -> GT_EarlyWarning | None:
|
||||
"""Return the shared engine when analytics are enabled and runtime allows it."""
|
||||
global _engine
|
||||
if not gt_engine_operational():
|
||||
return None
|
||||
with _engine_lock:
|
||||
if _engine is None:
|
||||
_engine = GT_EarlyWarning(get_gt_settings())
|
||||
logger.info("Strategic Risk Analytics engine initialized")
|
||||
return _engine
|
||||
|
||||
|
||||
def reset_gt_engine() -> None:
|
||||
"""Reset singleton — intended for tests."""
|
||||
global _engine
|
||||
get_gt_settings.cache_clear()
|
||||
with _engine_lock:
|
||||
_engine = None
|
||||
|
||||
|
||||
def process_feed_item(item: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Process a normalized feed item if analytics are enabled."""
|
||||
engine = get_gt_engine()
|
||||
if engine is None:
|
||||
return None
|
||||
try:
|
||||
return engine.process_feed_item(item)
|
||||
except Exception:
|
||||
logger.exception("GT process_feed_item failed")
|
||||
return None
|
||||
|
||||
|
||||
def _persist_gt_snapshot(
|
||||
engine: GT_EarlyWarning,
|
||||
*,
|
||||
processed: int,
|
||||
sample: list[dict[str, Any]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
timestamp = datetime.now(timezone.utc).isoformat()
|
||||
heatmap = engine.get_risk_heatmap()
|
||||
micro_summary: dict[str, Any] = {}
|
||||
try:
|
||||
from analytics.micro_rolling import capture_daily_readings, enrich_heatmap_features
|
||||
|
||||
micro_summary = capture_daily_readings(engine)
|
||||
heatmap = enrich_heatmap_features(heatmap)
|
||||
except Exception:
|
||||
logger.exception("GT micro rolling capture failed")
|
||||
|
||||
clusters = engine.compute_herding_clusters()
|
||||
from analytics.gt_alerts import parse_heatmap_alerts
|
||||
|
||||
_, plotted_regions = parse_heatmap_alerts(heatmap)
|
||||
with engine._lock: # noqa: SLF001 — snapshot meta
|
||||
engine_regions = len(engine._regions)
|
||||
settings = get_gt_settings()
|
||||
payload = {
|
||||
"enabled": True,
|
||||
"timestamp": timestamp,
|
||||
"processed": processed,
|
||||
"heatmap": heatmap,
|
||||
"clusters": clusters,
|
||||
"sample": list(sample or [])[:5],
|
||||
"regions": len(heatmap.get("features") or []),
|
||||
"micro": micro_summary,
|
||||
"meta": {
|
||||
"tracked_regions": len(heatmap.get("features") or []),
|
||||
"engine_regions": engine_regions,
|
||||
"plotted_regions": plotted_regions,
|
||||
"max_regions": settings.max_heatmap_features,
|
||||
},
|
||||
}
|
||||
with _data_lock:
|
||||
latest_data["gt_risk"] = payload
|
||||
_mark_fresh("gt_risk")
|
||||
return payload
|
||||
|
||||
|
||||
def refresh_from_latest_data(
|
||||
data_snapshot: dict[str, Any],
|
||||
*,
|
||||
persist: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Batch-ingest recent intel layers from the shared data store.
|
||||
|
||||
Intended to run after telegram/news/gdelt fetch cycles (near-real-time).
|
||||
"""
|
||||
engine = get_gt_engine()
|
||||
if engine is None:
|
||||
return {"enabled": False, "processed": 0}
|
||||
|
||||
processed = 0
|
||||
results: list[dict[str, Any]] = []
|
||||
|
||||
for item in iter_telegram_posts(data_snapshot.get("telegram_osint")):
|
||||
result = engine.process_feed_item(item)
|
||||
if result and not result.get("skipped"):
|
||||
processed += 1
|
||||
results.append(result)
|
||||
|
||||
for item in iter_news_items(data_snapshot.get("news")):
|
||||
result = engine.process_feed_item(item)
|
||||
if result and not result.get("skipped"):
|
||||
processed += 1
|
||||
if len(results) < 5:
|
||||
results.append(result)
|
||||
|
||||
for item in iter_gdelt_features(data_snapshot.get("gdelt")):
|
||||
result = engine.process_feed_item(item)
|
||||
if result and not result.get("skipped"):
|
||||
processed += 1
|
||||
|
||||
logger.info("GT refresh processed %d items", processed)
|
||||
summary = {
|
||||
"enabled": True,
|
||||
"processed": processed,
|
||||
"sample": results[:5],
|
||||
"heatmap_features": len(engine.get_risk_heatmap().get("features") or []),
|
||||
}
|
||||
if persist:
|
||||
snapshot = _persist_gt_snapshot(engine, processed=processed, sample=results)
|
||||
summary["timestamp"] = snapshot.get("timestamp")
|
||||
summary["clusters"] = len(snapshot.get("clusters") or [])
|
||||
return summary
|
||||
|
||||
|
||||
def recompute_gt_herding_clusters() -> dict[str, Any]:
|
||||
"""Louvain community pass — run on a schedule independent of feed ingest."""
|
||||
if not gt_louvain_enabled():
|
||||
return {"enabled": False, "clusters": 0, "reason": "louvain_disabled_on_lean_profile"}
|
||||
|
||||
engine = get_gt_engine()
|
||||
if engine is None:
|
||||
return {"enabled": False, "clusters": 0}
|
||||
|
||||
clusters = engine.compute_herding_clusters()
|
||||
timestamp = datetime.now(timezone.utc).isoformat()
|
||||
with _data_lock:
|
||||
current = dict(latest_data.get("gt_risk") or {})
|
||||
current["clusters"] = clusters
|
||||
current["clusters_updated"] = timestamp
|
||||
current["enabled"] = True
|
||||
latest_data["gt_risk"] = current
|
||||
_mark_fresh("gt_risk")
|
||||
logger.info("GT Louvain recompute: %d clusters", len(clusters))
|
||||
return {"enabled": True, "clusters": len(clusters), "timestamp": timestamp}
|
||||
|
||||
|
||||
def maybe_refresh_gt_analytics() -> None:
|
||||
"""Hook for data_fetcher — no-op when analytics are disabled or lean-gated."""
|
||||
if not gt_scheduled_ingest_enabled():
|
||||
return
|
||||
try:
|
||||
with _data_lock:
|
||||
snapshot = dict(latest_data)
|
||||
refresh_from_latest_data(snapshot, persist=True)
|
||||
except Exception:
|
||||
logger.exception("GT analytics refresh failed")
|
||||
|
||||
|
||||
def maybe_freeze_gt_weekly_snapshot() -> None:
|
||||
"""Hook for weekly scheduler — freeze operational backtest snapshot."""
|
||||
if not gt_engine_operational():
|
||||
return
|
||||
try:
|
||||
from analytics.rolling_backtest import freeze_weekly_snapshot
|
||||
|
||||
result = freeze_weekly_snapshot(frozen_by="scheduler")
|
||||
if result.get("created"):
|
||||
logger.info(
|
||||
"GT rolling freeze: week=%s regions=%s alerts=%s",
|
||||
result.get("week_id"),
|
||||
result.get("region_count"),
|
||||
result.get("alert_count"),
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("GT rolling weekly freeze failed")
|
||||
@@ -0,0 +1,361 @@
|
||||
"""Micro rolling 3-day average — fast ignition signal alongside weekly macro."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from analytics.daily_store import (
|
||||
DailyRegionReading,
|
||||
DailySnapshot,
|
||||
date_id,
|
||||
list_daily_ids,
|
||||
load_daily,
|
||||
save_daily,
|
||||
utc_now_iso,
|
||||
utc_today,
|
||||
)
|
||||
from analytics.gt_early_warning import GT_EarlyWarning
|
||||
from analytics.rolling_backtest import rolling_alert_threshold
|
||||
|
||||
DEFAULT_WINDOW_DAYS = 3
|
||||
DEFAULT_IGNITION_DELTA = 0.10
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
raw = str(os.environ.get(name, "")).strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return max(1, int(raw))
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
raw = str(os.environ.get(name, "")).strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return float(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def micro_window_days() -> int:
|
||||
return _env_int("GT_MICRO_ROLLING_DAYS", DEFAULT_WINDOW_DAYS)
|
||||
|
||||
|
||||
def ignition_delta() -> float:
|
||||
return _env_float("GT_MICRO_IGNITION_DELTA", DEFAULT_IGNITION_DELTA)
|
||||
|
||||
|
||||
def _peak_score(
|
||||
*,
|
||||
composite: float,
|
||||
financial: float,
|
||||
unrest: float,
|
||||
conflict: float,
|
||||
) -> float:
|
||||
return max(composite, financial, unrest, conflict)
|
||||
|
||||
|
||||
def _region_reading_from_feature(
|
||||
feature: dict[str, Any],
|
||||
*,
|
||||
captured_at: str,
|
||||
) -> DailyRegionReading | None:
|
||||
props = feature.get("properties") or {}
|
||||
region = str(props.get("region") or "").strip().lower()
|
||||
if not region:
|
||||
return None
|
||||
composite = float(props.get("risk") or props.get("composite_risk") or 0.0)
|
||||
financial = float(props.get("financial") or 0.0)
|
||||
unrest = float(props.get("unrest") or 0.0)
|
||||
conflict = float(props.get("conflict") or 0.0)
|
||||
peak = _peak_score(
|
||||
composite=composite,
|
||||
financial=financial,
|
||||
unrest=unrest,
|
||||
conflict=conflict,
|
||||
)
|
||||
return DailyRegionReading(
|
||||
region=region,
|
||||
composite_risk=composite,
|
||||
financial=financial,
|
||||
unrest=unrest,
|
||||
conflict=conflict,
|
||||
peak_score=peak,
|
||||
readings=1,
|
||||
last_captured_at=captured_at,
|
||||
)
|
||||
|
||||
|
||||
def capture_daily_readings(
|
||||
engine: GT_EarlyWarning,
|
||||
*,
|
||||
when: date | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Upsert today's regional readings from the live heatmap.
|
||||
|
||||
Each GT refresh updates the current day's latest scores (rolling window
|
||||
uses one value per calendar day).
|
||||
"""
|
||||
day = when or utc_today()
|
||||
day_key = date_id(day)
|
||||
captured_at = utc_now_iso()
|
||||
heatmap = engine.get_risk_heatmap()
|
||||
existing = load_daily(day) or DailySnapshot(date=day_key, regions={})
|
||||
|
||||
updated = 0
|
||||
for feature in heatmap.get("features") or []:
|
||||
if not isinstance(feature, dict):
|
||||
continue
|
||||
reading = _region_reading_from_feature(feature, captured_at=captured_at)
|
||||
if reading is None:
|
||||
continue
|
||||
prior = existing.regions.get(reading.region)
|
||||
if prior is None:
|
||||
existing.regions[reading.region] = reading
|
||||
updated += 1
|
||||
continue
|
||||
prior.composite_risk = reading.composite_risk
|
||||
prior.financial = reading.financial
|
||||
prior.unrest = reading.unrest
|
||||
prior.conflict = reading.conflict
|
||||
prior.peak_score = max(prior.peak_score, reading.peak_score)
|
||||
prior.readings += 1
|
||||
prior.last_captured_at = captured_at
|
||||
updated += 1
|
||||
|
||||
existing.last_updated_at = captured_at
|
||||
save_daily(existing)
|
||||
return {
|
||||
"date": day_key,
|
||||
"regions": len(existing.regions),
|
||||
"updated": updated,
|
||||
"captured_at": captured_at,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MicroRegionView:
|
||||
region: str
|
||||
spot_risk: float
|
||||
risk_3d_avg: float
|
||||
risk_delta: float
|
||||
days_in_window: int
|
||||
day_scores: tuple[float, ...]
|
||||
alerted_spot: bool
|
||||
alerted_3d: bool
|
||||
ignition: bool
|
||||
financial: float
|
||||
unrest: float
|
||||
conflict: float
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"region": self.region,
|
||||
"spot_risk": round(self.spot_risk, 4),
|
||||
"risk_3d_avg": round(self.risk_3d_avg, 4),
|
||||
"risk_delta": round(self.risk_delta, 4),
|
||||
"days_in_window": self.days_in_window,
|
||||
"day_scores": [round(score, 4) for score in self.day_scores],
|
||||
"alerted_spot": self.alerted_spot,
|
||||
"alerted_3d": self.alerted_3d,
|
||||
"ignition": self.ignition,
|
||||
"financial": round(self.financial, 4),
|
||||
"unrest": round(self.unrest, 4),
|
||||
"conflict": round(self.conflict, 4),
|
||||
}
|
||||
|
||||
|
||||
def _day_offsets(window_days: int) -> list[int]:
|
||||
# Today + prior (window_days - 1) days.
|
||||
return list(range(window_days - 1, -1, -1))
|
||||
|
||||
|
||||
def _historical_dates(as_of: date, window_days: int) -> list[date]:
|
||||
return [as_of - timedelta(days=offset) for offset in _day_offsets(window_days)]
|
||||
|
||||
|
||||
def compute_micro_view(
|
||||
region: str,
|
||||
*,
|
||||
as_of: date | None = None,
|
||||
window_days: int | None = None,
|
||||
alert_threshold: float | None = None,
|
||||
spot_reading: DailyRegionReading | None = None,
|
||||
) -> MicroRegionView | None:
|
||||
"""Compute rolling N-day average and ignition vs spot for one region."""
|
||||
region_key = str(region or "").strip().lower()
|
||||
if not region_key:
|
||||
return None
|
||||
|
||||
today = as_of or utc_today()
|
||||
window = window_days or micro_window_days()
|
||||
threshold = float(alert_threshold if alert_threshold is not None else rolling_alert_threshold())
|
||||
delta_min = ignition_delta()
|
||||
|
||||
day_scores: list[float] = []
|
||||
latest: DailyRegionReading | None = spot_reading
|
||||
|
||||
for day in _historical_dates(today, window):
|
||||
snap = load_daily(day)
|
||||
if snap is None:
|
||||
continue
|
||||
row = snap.regions.get(region_key)
|
||||
if row is None:
|
||||
continue
|
||||
day_scores.append(row.peak_score)
|
||||
if day == today:
|
||||
latest = row
|
||||
|
||||
if latest is None and day_scores:
|
||||
# Spot may come from yesterday if today not captured yet.
|
||||
snap = load_daily(today)
|
||||
if snap:
|
||||
latest = snap.regions.get(region_key)
|
||||
|
||||
if latest is None and not day_scores:
|
||||
return None
|
||||
|
||||
spot = float(latest.peak_score if latest else (day_scores[-1] if day_scores else 0.0))
|
||||
avg = sum(day_scores) / len(day_scores) if day_scores else spot
|
||||
risk_delta = spot - avg
|
||||
ignition = risk_delta >= delta_min and spot >= threshold * 0.75
|
||||
|
||||
return MicroRegionView(
|
||||
region=region_key,
|
||||
spot_risk=spot,
|
||||
risk_3d_avg=avg,
|
||||
risk_delta=risk_delta,
|
||||
days_in_window=len(day_scores),
|
||||
day_scores=tuple(day_scores),
|
||||
alerted_spot=spot >= threshold,
|
||||
alerted_3d=avg >= threshold,
|
||||
ignition=ignition,
|
||||
financial=float(latest.financial if latest else 0.0),
|
||||
unrest=float(latest.unrest if latest else 0.0),
|
||||
conflict=float(latest.conflict if latest else 0.0),
|
||||
)
|
||||
|
||||
|
||||
def compute_all_micro_views(
|
||||
*,
|
||||
as_of: date | None = None,
|
||||
window_days: int | None = None,
|
||||
alert_threshold: float | None = None,
|
||||
) -> list[MicroRegionView]:
|
||||
"""Build micro views for all regions seen in the rolling window."""
|
||||
today = as_of or utc_today()
|
||||
window = window_days or micro_window_days()
|
||||
regions: set[str] = set()
|
||||
|
||||
for day in _historical_dates(today, window):
|
||||
snap = load_daily(day)
|
||||
if snap is None:
|
||||
continue
|
||||
regions.update(snap.regions.keys())
|
||||
|
||||
views: list[MicroRegionView] = []
|
||||
for region in regions:
|
||||
view = compute_micro_view(
|
||||
region,
|
||||
as_of=today,
|
||||
window_days=window,
|
||||
alert_threshold=alert_threshold,
|
||||
)
|
||||
if view is not None:
|
||||
views.append(view)
|
||||
|
||||
views.sort(key=lambda row: (row.ignition, row.risk_delta, row.spot_risk), reverse=True)
|
||||
return views
|
||||
|
||||
|
||||
def enrich_heatmap_features(
|
||||
heatmap: dict[str, Any],
|
||||
*,
|
||||
as_of: date | None = None,
|
||||
window_days: int | None = None,
|
||||
alert_threshold: float | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Attach micro rolling fields to heatmap GeoJSON features."""
|
||||
threshold = float(alert_threshold if alert_threshold is not None else rolling_alert_threshold())
|
||||
window = window_days or micro_window_days()
|
||||
features = heatmap.get("features") or []
|
||||
enriched: list[dict[str, Any]] = []
|
||||
|
||||
for feature in features:
|
||||
if not isinstance(feature, dict):
|
||||
continue
|
||||
props = dict(feature.get("properties") or {})
|
||||
region = str(props.get("region") or "").strip().lower()
|
||||
view = compute_micro_view(
|
||||
region,
|
||||
as_of=as_of,
|
||||
window_days=window,
|
||||
alert_threshold=threshold,
|
||||
) if region else None
|
||||
|
||||
if view is not None:
|
||||
props["risk_spot"] = view.spot_risk
|
||||
props["risk_3d_avg"] = view.risk_3d_avg
|
||||
props["risk_delta"] = view.risk_delta
|
||||
props["micro_days"] = view.days_in_window
|
||||
props["micro_ignition"] = view.ignition
|
||||
props["alerted_3d"] = view.alerted_3d
|
||||
props["day_scores"] = list(view.day_scores)
|
||||
|
||||
enriched.append({**feature, "properties": props})
|
||||
|
||||
return {
|
||||
**heatmap,
|
||||
"features": enriched,
|
||||
"micro_window_days": window,
|
||||
"micro_alert_threshold": threshold,
|
||||
}
|
||||
|
||||
|
||||
def micro_rolling_report(
|
||||
*,
|
||||
as_of: date | None = None,
|
||||
window_days: int | None = None,
|
||||
limit: int = 15,
|
||||
) -> dict[str, Any]:
|
||||
"""API/OpenClaw payload for micro rolling 3-day context."""
|
||||
today = as_of or utc_today()
|
||||
window = window_days or micro_window_days()
|
||||
threshold = rolling_alert_threshold()
|
||||
views = compute_all_micro_views(
|
||||
as_of=today,
|
||||
window_days=window,
|
||||
alert_threshold=threshold,
|
||||
)
|
||||
ignitions = [row for row in views if row.ignition]
|
||||
alerted_3d = [row for row in views if row.alerted_3d]
|
||||
top = views[: max(1, limit)]
|
||||
|
||||
stored_days = list_daily_ids(newest_first=True, limit=window)
|
||||
return {
|
||||
"mode": "micro_rolling",
|
||||
"window_days": window,
|
||||
"alert_threshold": threshold,
|
||||
"ignition_delta": ignition_delta(),
|
||||
"as_of": date_id(today),
|
||||
"days_stored": len(stored_days),
|
||||
"stored_dates": stored_days,
|
||||
"regions_tracked": len(views),
|
||||
"ignition_count": len(ignitions),
|
||||
"alerted_3d_count": len(alerted_3d),
|
||||
"ignitions": [row.to_dict() for row in ignitions[:limit]],
|
||||
"top_regions": [row.to_dict() for row in top],
|
||||
"note": (
|
||||
f"Micro view: {window}-day rolling average vs spot risk. "
|
||||
"Ignition = spot jumped above the rolling baseline (events that flare fast). "
|
||||
"Macro week-over-week validation remains on /api/analytics/rolling."
|
||||
),
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
"""Rolling weekly operational validation for Strategic Risk Analytics.
|
||||
|
||||
Freezes live GT scores each ISO week, accepts delayed outcome labels, and
|
||||
scores prior-week predictions with accuracy + Wilson 95% CI. Unlike the
|
||||
static historical benchmark, this measures forward operational usefulness.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timezone
|
||||
from typing import Any, Literal
|
||||
|
||||
from analytics.backtest import DEFAULT_BACKTEST_ALERT_THRESHOLD, wilson_interval
|
||||
from analytics.gt_early_warning import GT_EarlyWarning
|
||||
from analytics.integration import get_gt_engine
|
||||
from analytics.weekly_store import (
|
||||
VALID_LABELS,
|
||||
LabelName,
|
||||
RegionSnapshot,
|
||||
WeeklySnapshot,
|
||||
list_week_ids,
|
||||
load_week,
|
||||
save_week,
|
||||
utc_now_iso,
|
||||
)
|
||||
|
||||
MIN_LABELED_FOR_TREND = 5
|
||||
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
raw = str(os.environ.get(name, "")).strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return float(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def rolling_alert_threshold() -> float:
|
||||
"""Fixed operational alert cutoff — not retroactively tuned per week."""
|
||||
return _env_float("GT_ROLLING_ALERT_THRESHOLD", DEFAULT_BACKTEST_ALERT_THRESHOLD)
|
||||
|
||||
|
||||
def iso_week_id(when: datetime | date | None = None) -> str:
|
||||
"""Return ISO week id, e.g. ``2026-W24``."""
|
||||
if when is None:
|
||||
when = datetime.now(timezone.utc)
|
||||
if isinstance(when, datetime):
|
||||
when = when.date()
|
||||
year, week, _ = when.isocalendar()
|
||||
return f"{year}-W{week:02d}"
|
||||
|
||||
|
||||
def _region_rows_from_engine(
|
||||
engine: GT_EarlyWarning,
|
||||
*,
|
||||
alert_threshold: float,
|
||||
) -> list[RegionSnapshot]:
|
||||
heatmap = engine.get_risk_heatmap()
|
||||
rows: list[RegionSnapshot] = []
|
||||
for feature in heatmap.get("features") or []:
|
||||
if not isinstance(feature, dict):
|
||||
continue
|
||||
props = feature.get("properties") or {}
|
||||
region = str(props.get("region") or "").strip().lower()
|
||||
if not region:
|
||||
continue
|
||||
composite = float(props.get("risk") or 0.0)
|
||||
financial = float(props.get("financial") or 0.0)
|
||||
unrest = float(props.get("unrest") or 0.0)
|
||||
conflict = float(props.get("conflict") or 0.0)
|
||||
peak_score = max(composite, financial, unrest, conflict)
|
||||
rows.append(
|
||||
RegionSnapshot(
|
||||
region=region,
|
||||
composite_risk=composite,
|
||||
financial=financial,
|
||||
unrest=unrest,
|
||||
conflict=conflict,
|
||||
alerted=peak_score >= alert_threshold,
|
||||
label="pending",
|
||||
)
|
||||
)
|
||||
rows.sort(key=lambda row: row.composite_risk, reverse=True)
|
||||
return rows
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WeekScore:
|
||||
week_id: str
|
||||
frozen_at: str
|
||||
alert_threshold: float
|
||||
total_regions: int
|
||||
labeled: int
|
||||
pending: int
|
||||
alerted: int
|
||||
correct: int
|
||||
accuracy: float
|
||||
confidence_rate: float
|
||||
wilson_lower_95: float
|
||||
wilson_upper_95: float
|
||||
true_positives: int
|
||||
true_negatives: int
|
||||
false_positives: int
|
||||
false_negatives: int
|
||||
sensitivity: float
|
||||
specificity: float
|
||||
scorable: bool
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"week_id": self.week_id,
|
||||
"frozen_at": self.frozen_at,
|
||||
"alert_threshold": round(self.alert_threshold, 4),
|
||||
"total_regions": self.total_regions,
|
||||
"labeled": self.labeled,
|
||||
"pending": self.pending,
|
||||
"alerted": self.alerted,
|
||||
"correct": self.correct,
|
||||
"accuracy": round(self.accuracy, 4),
|
||||
"confidence_rate": round(self.confidence_rate, 4),
|
||||
"wilson_lower_95": round(self.wilson_lower_95, 4),
|
||||
"wilson_upper_95": round(self.wilson_upper_95, 4),
|
||||
"true_positives": self.true_positives,
|
||||
"true_negatives": self.true_negatives,
|
||||
"false_positives": self.false_positives,
|
||||
"false_negatives": self.false_negatives,
|
||||
"sensitivity": round(self.sensitivity, 4),
|
||||
"specificity": round(self.specificity, 4),
|
||||
"scorable": self.scorable,
|
||||
}
|
||||
|
||||
|
||||
def _predicted_positive(row: RegionSnapshot) -> bool:
|
||||
return row.alerted
|
||||
|
||||
|
||||
def _actual_positive(label: LabelName) -> bool:
|
||||
return label == "true_escalation"
|
||||
|
||||
|
||||
def _is_correct(row: RegionSnapshot) -> bool:
|
||||
if row.label == "pending":
|
||||
return False
|
||||
predicted = _predicted_positive(row)
|
||||
if row.label == "true_escalation":
|
||||
return predicted
|
||||
if row.label in ("false_alarm", "benign"):
|
||||
return not predicted
|
||||
return False
|
||||
|
||||
|
||||
def score_week(snapshot: WeeklySnapshot) -> WeekScore:
|
||||
"""Score a frozen week against delayed labels (pending rows excluded)."""
|
||||
labeled_rows = [row for row in snapshot.regions if row.label != "pending"]
|
||||
pending = len(snapshot.regions) - len(labeled_rows)
|
||||
|
||||
tp = sum(
|
||||
1
|
||||
for row in labeled_rows
|
||||
if row.alerted and row.label == "true_escalation"
|
||||
)
|
||||
tn = sum(
|
||||
1
|
||||
for row in labeled_rows
|
||||
if not row.alerted and row.label in ("benign", "false_alarm")
|
||||
)
|
||||
fp = sum(
|
||||
1
|
||||
for row in labeled_rows
|
||||
if row.alerted and row.label in ("false_alarm", "benign")
|
||||
)
|
||||
fn = sum(
|
||||
1
|
||||
for row in labeled_rows
|
||||
if not row.alerted and row.label == "true_escalation"
|
||||
)
|
||||
|
||||
correct = tp + tn
|
||||
total = len(labeled_rows)
|
||||
accuracy = correct / total if total else 0.0
|
||||
lower, upper = wilson_interval(correct, total)
|
||||
|
||||
pos_total = sum(1 for row in labeled_rows if _actual_positive(row.label)) # type: ignore[arg-type]
|
||||
neg_total = total - pos_total
|
||||
pred_pos = sum(1 for row in labeled_rows if row.alerted)
|
||||
pred_neg = total - pred_pos
|
||||
|
||||
sensitivity = tp / pos_total if pos_total else 0.0
|
||||
specificity = tn / pred_neg if pred_neg else (1.0 if tn == total and total else 0.0)
|
||||
|
||||
return WeekScore(
|
||||
week_id=snapshot.week_id,
|
||||
frozen_at=snapshot.frozen_at,
|
||||
alert_threshold=snapshot.alert_threshold,
|
||||
total_regions=len(snapshot.regions),
|
||||
labeled=total,
|
||||
pending=pending,
|
||||
alerted=sum(1 for row in snapshot.regions if row.alerted),
|
||||
correct=correct,
|
||||
accuracy=accuracy,
|
||||
confidence_rate=lower,
|
||||
wilson_lower_95=lower,
|
||||
wilson_upper_95=upper,
|
||||
true_positives=tp,
|
||||
true_negatives=tn,
|
||||
false_positives=fp,
|
||||
false_negatives=fn,
|
||||
sensitivity=sensitivity,
|
||||
specificity=specificity,
|
||||
scorable=total >= MIN_LABELED_FOR_TREND,
|
||||
)
|
||||
|
||||
|
||||
def freeze_weekly_snapshot(
|
||||
*,
|
||||
week_id: str | None = None,
|
||||
alert_threshold: float | None = None,
|
||||
force: bool = False,
|
||||
frozen_by: str = "system",
|
||||
engine: GT_EarlyWarning | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Capture current GT heatmap as an immutable weekly operational snapshot.
|
||||
|
||||
Idempotent per week unless ``force=True``.
|
||||
"""
|
||||
resolved_engine = engine or get_gt_engine()
|
||||
if resolved_engine is None:
|
||||
return {"ok": False, "detail": "GT analytics engine unavailable"}
|
||||
|
||||
resolved_week = week_id or iso_week_id()
|
||||
threshold = float(
|
||||
alert_threshold if alert_threshold is not None else rolling_alert_threshold()
|
||||
)
|
||||
|
||||
existing = load_week(resolved_week)
|
||||
if existing and existing.regions and not force:
|
||||
score = score_week(existing)
|
||||
return {
|
||||
"ok": True,
|
||||
"created": False,
|
||||
"week_id": resolved_week,
|
||||
"snapshot": existing.to_dict(),
|
||||
"score": score.to_dict(),
|
||||
}
|
||||
|
||||
regions = _region_rows_from_engine(resolved_engine, alert_threshold=threshold)
|
||||
snapshot = WeeklySnapshot(
|
||||
week_id=resolved_week,
|
||||
frozen_at=utc_now_iso(),
|
||||
alert_threshold=threshold,
|
||||
regions=regions,
|
||||
frozen_by=frozen_by,
|
||||
)
|
||||
save_week(snapshot)
|
||||
score = score_week(snapshot)
|
||||
return {
|
||||
"ok": True,
|
||||
"created": True,
|
||||
"week_id": resolved_week,
|
||||
"snapshot": snapshot.to_dict(),
|
||||
"score": score.to_dict(),
|
||||
"alert_count": sum(1 for row in regions if row.alerted),
|
||||
"region_count": len(regions),
|
||||
}
|
||||
|
||||
|
||||
def label_regions(
|
||||
week_id: str,
|
||||
labels: list[dict[str, Any]],
|
||||
*,
|
||||
labeled_by: str = "operator",
|
||||
) -> dict[str, Any]:
|
||||
"""Apply delayed outcome labels to a frozen week."""
|
||||
snapshot = load_week(week_id)
|
||||
if snapshot is None:
|
||||
return {"ok": False, "detail": f"Week {week_id} not found"}
|
||||
|
||||
by_region = {row.region: row for row in snapshot.regions}
|
||||
updated = 0
|
||||
skipped: list[str] = []
|
||||
now = utc_now_iso()
|
||||
|
||||
for entry in labels:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
region = str(entry.get("region") or "").strip().lower()
|
||||
label = str(entry.get("label") or "").strip().lower()
|
||||
if not region or label not in VALID_LABELS or label == "pending":
|
||||
if region:
|
||||
skipped.append(region)
|
||||
continue
|
||||
row = by_region.get(region)
|
||||
if row is None:
|
||||
skipped.append(region)
|
||||
continue
|
||||
row.label = label # type: ignore[assignment]
|
||||
row.labeled_at = now
|
||||
notes = entry.get("notes")
|
||||
if notes is not None:
|
||||
row.notes = str(notes)
|
||||
updated += 1
|
||||
|
||||
save_week(snapshot)
|
||||
score = score_week(snapshot)
|
||||
return {
|
||||
"ok": True,
|
||||
"week_id": week_id,
|
||||
"updated": updated,
|
||||
"skipped": skipped,
|
||||
"labeled_by": labeled_by,
|
||||
"score": score.to_dict(),
|
||||
}
|
||||
|
||||
|
||||
def label_region(
|
||||
week_id: str,
|
||||
region: str,
|
||||
label: LabelName,
|
||||
*,
|
||||
notes: str = "",
|
||||
labeled_by: str = "operator",
|
||||
) -> dict[str, Any]:
|
||||
return label_regions(
|
||||
week_id,
|
||||
[{"region": region, "label": label, "notes": notes}],
|
||||
labeled_by=labeled_by,
|
||||
)
|
||||
|
||||
|
||||
def rolling_trend(*, weeks: int = 8) -> list[WeekScore]:
|
||||
"""Return scored weeks newest-first (only weeks with stored snapshots)."""
|
||||
ids = list_week_ids(newest_first=True)[: max(1, weeks)]
|
||||
scores: list[WeekScore] = []
|
||||
for week_id in ids:
|
||||
snapshot = load_week(week_id)
|
||||
if snapshot is None:
|
||||
continue
|
||||
scores.append(score_week(snapshot))
|
||||
return scores
|
||||
|
||||
|
||||
def rolling_report(*, weeks: int = 8, target_confidence: float = 0.80) -> dict[str, Any]:
|
||||
"""Aggregate operational validation trend for API / OpenClaw."""
|
||||
threshold = rolling_alert_threshold()
|
||||
trend = rolling_trend(weeks=weeks)
|
||||
scorable = [row for row in trend if row.scorable]
|
||||
|
||||
latest = scorable[0] if scorable else (trend[0] if trend else None)
|
||||
accuracy_series = [
|
||||
{"week_id": row.week_id, "accuracy": round(row.accuracy, 4), "labeled": row.labeled}
|
||||
for row in reversed(scorable)
|
||||
]
|
||||
|
||||
improving = False
|
||||
if len(scorable) >= 2:
|
||||
improving = scorable[0].accuracy >= scorable[1].accuracy
|
||||
|
||||
return {
|
||||
"mode": "rolling_operational",
|
||||
"alert_threshold": threshold,
|
||||
"target_confidence": target_confidence,
|
||||
"weeks_requested": weeks,
|
||||
"weeks_stored": len(trend),
|
||||
"weeks_scorable": len(scorable),
|
||||
"min_labeled_per_week": MIN_LABELED_FOR_TREND,
|
||||
"latest": latest.to_dict() if latest else None,
|
||||
"trend": [row.to_dict() for row in trend],
|
||||
"accuracy_series": accuracy_series,
|
||||
"improving_vs_prior": improving,
|
||||
"meets_target": bool(
|
||||
latest and latest.scorable and latest.confidence_rate >= target_confidence
|
||||
),
|
||||
"note": (
|
||||
"Operational metric: scores frozen weekly predictions against delayed "
|
||||
"labels. Unlike the static benchmark, this measures live forward utility."
|
||||
),
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
"""Configuration for Strategic Risk Analytics (feature-flagged)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from functools import lru_cache
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _env_bool(name: str, default: bool = False) -> bool:
|
||||
raw = str(os.environ.get(name, "")).strip().lower()
|
||||
if not raw:
|
||||
return default
|
||||
return raw not in {"0", "false", "no", "off"}
|
||||
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
raw = str(os.environ.get(name, "")).strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return float(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
raw = str(os.environ.get(name, "")).strip()
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def _parse_signal_weights(raw: str) -> dict[str, float]:
|
||||
if not raw.strip():
|
||||
return {}
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
if isinstance(parsed, dict):
|
||||
return {str(k): float(v) for k, v in parsed.items()}
|
||||
except (json.JSONDecodeError, TypeError, ValueError):
|
||||
pass
|
||||
weights: dict[str, float] = {}
|
||||
for part in raw.split(","):
|
||||
piece = part.strip()
|
||||
if not piece or "=" not in piece:
|
||||
continue
|
||||
key, value = piece.split("=", 1)
|
||||
try:
|
||||
weights[key.strip()] = float(value.strip())
|
||||
except ValueError:
|
||||
continue
|
||||
return weights
|
||||
|
||||
|
||||
def resolve_gt_profile() -> str:
|
||||
from services.runtime_profile import resolve_profile_name
|
||||
|
||||
return resolve_profile_name()
|
||||
|
||||
|
||||
def gt_analytics_ack_low_cpu() -> bool:
|
||||
return _env_bool("GT_ANALYTICS_ACK_LOW_CPU", default=False)
|
||||
|
||||
|
||||
def gt_engine_operational() -> bool:
|
||||
"""Full GT engine (scheduled ingest, heatmap, Louvain) — not watchdog-only."""
|
||||
if not get_gt_settings().enabled:
|
||||
return False
|
||||
if resolve_gt_profile() == "lean" and not gt_analytics_ack_low_cpu():
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def gt_scheduled_ingest_enabled() -> bool:
|
||||
return gt_engine_operational()
|
||||
|
||||
|
||||
def gt_louvain_enabled() -> bool:
|
||||
return gt_engine_operational()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GTAnalyticsSettings:
|
||||
enabled: bool = False
|
||||
profile: str = "standard"
|
||||
base_prior: float = 0.15
|
||||
evidence_cap: float = 3.0
|
||||
evidence_scale: float = 5.0
|
||||
min_prob: float = 0.01
|
||||
max_prob: float = 0.99
|
||||
high_risk_threshold: float = 0.6
|
||||
max_history_per_region: int = 200
|
||||
max_heatmap_features: int = 500
|
||||
louvain_min_weight: float = 0.5
|
||||
louvain_interval_minutes: int = 30
|
||||
signal_weight_overrides: dict[str, float] = field(default_factory=dict)
|
||||
watched_channels: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_gt_settings() -> GTAnalyticsSettings:
|
||||
channels_raw = str(os.environ.get("GT_ANALYTICS_WATCHED_CHANNELS", "")).strip()
|
||||
channels = tuple(
|
||||
part.strip().lstrip("@")
|
||||
for part in channels_raw.split(",")
|
||||
if part.strip()
|
||||
)
|
||||
profile = resolve_gt_profile()
|
||||
lean = profile == "lean"
|
||||
return GTAnalyticsSettings(
|
||||
enabled=_env_bool("GT_ANALYTICS_ENABLED", default=False),
|
||||
profile=profile,
|
||||
base_prior=_env_float("GT_ANALYTICS_BASE_PRIOR", 0.15),
|
||||
evidence_cap=_env_float("GT_ANALYTICS_EVIDENCE_CAP", 3.0),
|
||||
evidence_scale=_env_float("GT_ANALYTICS_EVIDENCE_SCALE", 5.0),
|
||||
min_prob=_env_float("GT_ANALYTICS_MIN_PROB", 0.01),
|
||||
max_prob=_env_float("GT_ANALYTICS_MAX_PROB", 0.99),
|
||||
high_risk_threshold=_env_float("GT_ANALYTICS_HIGH_RISK_THRESHOLD", 0.6),
|
||||
max_history_per_region=_env_int("GT_ANALYTICS_MAX_HISTORY", 200),
|
||||
max_heatmap_features=_env_int(
|
||||
"GT_ANALYTICS_MAX_HEATMAP_FEATURES",
|
||||
50 if lean else 500,
|
||||
),
|
||||
louvain_min_weight=_env_float("GT_ANALYTICS_LOUVAIN_MIN_WEIGHT", 0.5),
|
||||
louvain_interval_minutes=max(5, _env_int("GT_ANALYTICS_LOUVAIN_INTERVAL_MINUTES", 30)),
|
||||
signal_weight_overrides=_parse_signal_weights(
|
||||
str(os.environ.get("GT_ANALYTICS_SIGNAL_WEIGHTS", ""))
|
||||
),
|
||||
watched_channels=channels,
|
||||
)
|
||||
|
||||
|
||||
def gt_analytics_enabled() -> bool:
|
||||
return get_gt_settings().enabled
|
||||
|
||||
|
||||
def gt_analytics_status() -> dict[str, Any]:
|
||||
settings = get_gt_settings()
|
||||
from services.runtime_profile import get_runtime_profile
|
||||
|
||||
runtime = get_runtime_profile()
|
||||
operational = gt_engine_operational()
|
||||
return {
|
||||
"enabled": settings.enabled,
|
||||
"operational": operational,
|
||||
"profile": settings.profile,
|
||||
"ack_low_cpu": gt_analytics_ack_low_cpu(),
|
||||
"recommended": bool(runtime.get("gt_analytics", {}).get("recommended")),
|
||||
"lean_node": bool(runtime.get("gt_analytics", {}).get("lean_node")),
|
||||
"warning": runtime.get("gt_analytics", {}).get("warning"),
|
||||
"experimental": True,
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Persistent JSON store for rolling GT operational backtest weeks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
LabelName = Literal["pending", "true_escalation", "false_alarm", "benign"]
|
||||
VALID_LABELS: frozenset[str] = frozenset(
|
||||
{"pending", "true_escalation", "false_alarm", "benign"}
|
||||
)
|
||||
|
||||
_STORE_DIR = Path(__file__).parent.parent / "data" / "gt_rolling"
|
||||
_store_lock = threading.Lock()
|
||||
|
||||
|
||||
def rolling_store_dir() -> Path:
|
||||
"""Return the rolling-backtest data directory (override via env in tests)."""
|
||||
override = str(os.environ.get("GT_ROLLING_STORE_DIR", "")).strip()
|
||||
if override:
|
||||
return Path(override)
|
||||
return _STORE_DIR
|
||||
|
||||
|
||||
@dataclass
|
||||
class RegionSnapshot:
|
||||
region: str
|
||||
composite_risk: float
|
||||
financial: float
|
||||
unrest: float
|
||||
conflict: float
|
||||
alerted: bool
|
||||
label: LabelName = "pending"
|
||||
labeled_at: str | None = None
|
||||
notes: str = ""
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, raw: dict[str, Any]) -> RegionSnapshot:
|
||||
label = str(raw.get("label") or "pending")
|
||||
if label not in VALID_LABELS:
|
||||
label = "pending"
|
||||
return cls(
|
||||
region=str(raw.get("region") or "").strip().lower(),
|
||||
composite_risk=float(raw.get("composite_risk") or 0.0),
|
||||
financial=float(raw.get("financial") or 0.0),
|
||||
unrest=float(raw.get("unrest") or 0.0),
|
||||
conflict=float(raw.get("conflict") or 0.0),
|
||||
alerted=bool(raw.get("alerted")),
|
||||
label=label, # type: ignore[arg-type]
|
||||
labeled_at=raw.get("labeled_at"),
|
||||
notes=str(raw.get("notes") or ""),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class WeeklySnapshot:
|
||||
week_id: str
|
||||
frozen_at: str
|
||||
alert_threshold: float
|
||||
regions: list[RegionSnapshot] = field(default_factory=list)
|
||||
frozen_by: str = "system"
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"week_id": self.week_id,
|
||||
"frozen_at": self.frozen_at,
|
||||
"alert_threshold": self.alert_threshold,
|
||||
"frozen_by": self.frozen_by,
|
||||
"regions": [row.to_dict() for row in self.regions],
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, raw: dict[str, Any]) -> WeeklySnapshot:
|
||||
regions = [
|
||||
RegionSnapshot.from_dict(row)
|
||||
for row in (raw.get("regions") or [])
|
||||
if isinstance(row, dict)
|
||||
]
|
||||
return cls(
|
||||
week_id=str(raw.get("week_id") or ""),
|
||||
frozen_at=str(raw.get("frozen_at") or ""),
|
||||
alert_threshold=float(raw.get("alert_threshold") or 0.0),
|
||||
regions=regions,
|
||||
frozen_by=str(raw.get("frozen_by") or "system"),
|
||||
)
|
||||
|
||||
|
||||
def _week_path(week_id: str) -> Path:
|
||||
safe = week_id.replace("/", "-").replace("..", "")
|
||||
return rolling_store_dir() / f"{safe}.json"
|
||||
|
||||
|
||||
def _ensure_dir() -> None:
|
||||
rolling_store_dir().mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def list_week_ids(*, newest_first: bool = True) -> list[str]:
|
||||
"""Return stored ISO week ids."""
|
||||
_ensure_dir()
|
||||
ids = [
|
||||
path.stem
|
||||
for path in rolling_store_dir().glob("*.json")
|
||||
if path.stem and path.stem != "index"
|
||||
]
|
||||
ids.sort(reverse=newest_first)
|
||||
return ids
|
||||
|
||||
|
||||
def load_week(week_id: str) -> WeeklySnapshot | None:
|
||||
path = _week_path(week_id)
|
||||
if not path.is_file():
|
||||
return None
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, dict):
|
||||
return None
|
||||
return WeeklySnapshot.from_dict(raw)
|
||||
except (OSError, json.JSONDecodeError, TypeError, ValueError):
|
||||
logger.exception("Failed to load GT rolling week %s", week_id)
|
||||
return None
|
||||
|
||||
|
||||
def save_week(snapshot: WeeklySnapshot) -> None:
|
||||
_ensure_dir()
|
||||
path = _week_path(snapshot.week_id)
|
||||
tmp = path.with_suffix(".json.tmp")
|
||||
payload = json.dumps(snapshot.to_dict(), indent=2, sort_keys=True)
|
||||
with _store_lock:
|
||||
tmp.write_text(payload, encoding="utf-8")
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def delete_week(week_id: str) -> bool:
|
||||
path = _week_path(week_id)
|
||||
if not path.is_file():
|
||||
return False
|
||||
with _store_lock:
|
||||
path.unlink()
|
||||
return True
|
||||
|
||||
|
||||
def utc_now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
@@ -862,7 +862,9 @@ _ROUTE_TRANSPORT_POLICY: dict[tuple[str, str], RouteTransportPolicy] = {
|
||||
("POST", "/api/wormhole/gate/messages/decrypt"): _local_only_route_policy("private_control_only"),
|
||||
# ── Wormhole DM (strong) ──────────────────────────────────────────
|
||||
("POST", "/api/wormhole/dm/compose"): _local_only_route_policy("private_control_only"),
|
||||
("POST", "/api/wormhole/dm/connect-contact"): _local_only_route_policy("private_control_only"),
|
||||
("POST", "/api/wormhole/dm/decrypt"): _local_only_route_policy("private_control_only"),
|
||||
("POST", "/api/wormhole/dm/mls-key-package"): _local_only_route_policy("private_control_only"),
|
||||
("POST", "/api/wormhole/dm/register-key"): _local_only_route_policy("private_control_only"),
|
||||
("POST", "/api/wormhole/dm/prekey/register"): _local_only_route_policy("private_control_only"),
|
||||
("POST", "/api/wormhole/dm/bootstrap-encrypt"): _local_only_route_policy("private_control_only"),
|
||||
@@ -1404,6 +1406,27 @@ def _peer_hmac_url_from_request(request: Request) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _verify_peer_transport_hmac(request: Request, body_bytes: bytes) -> bool:
|
||||
"""Verify HMAC-SHA256 peer authentication without an allowlist check."""
|
||||
provided = str(request.headers.get("x-peer-hmac", "") or "").strip()
|
||||
if not provided:
|
||||
return False
|
||||
|
||||
peer_url = _peer_hmac_url_from_request(request)
|
||||
if not peer_url:
|
||||
return False
|
||||
peer_key = resolve_peer_key_for_url(peer_url)
|
||||
if not peer_key:
|
||||
return False
|
||||
|
||||
expected = _hmac_mod.new(
|
||||
peer_key,
|
||||
body_bytes,
|
||||
_hashlib_mod.sha256,
|
||||
).hexdigest()
|
||||
return _hmac_mod.compare_digest(provided.lower(), expected.lower())
|
||||
|
||||
|
||||
def _verify_peer_push_hmac(request: Request, body_bytes: bytes) -> bool:
|
||||
"""Verify HMAC-SHA256 peer authentication on push requests.
|
||||
|
||||
|
||||
+3180
-1498
File diff suppressed because it is too large
Load Diff
@@ -51,5 +51,10 @@
|
||||
"ShadowBroker_v0.9.82.zip": "202ab043465741dcc06de57c19ec8314904332f8e818b891d7174655719d084c",
|
||||
"ShadowBroker_0.9.82_x64-setup.exe": "0eb9f2bda02ab691b39687641abc97e6bfb507b42f48de21970ad7dfb4ea15fc",
|
||||
"ShadowBroker_0.9.82_x64_en-US.msi": "ced08f930171c0c08009a958cc30b0171a09f982230fc217c6808c2ed7ab2e30"
|
||||
},
|
||||
"v0.9.83": {
|
||||
"ShadowBroker_v0.9.83.zip": "53f56631731ad3cdc7be68df09bedd6570ed91ecda6fa57c39651098e15666c7",
|
||||
"ShadowBroker_0.9.83_x64-setup.exe": "d62170af4b9df0b190832b7bb3ad6bfe8a7ac01472f2c7b39cf2a1b61edc7492",
|
||||
"ShadowBroker_0.9.83_x64_en-US.msi": "b664cc0003a29f7ce88b04c2b425643dbe7ed897342fc6e9a2378bc1910c6850"
|
||||
}
|
||||
}
|
||||
|
||||
+3104
-2639
File diff suppressed because it is too large
Load Diff
+416
-54
@@ -244,6 +244,7 @@ from services.mesh.mesh_protocol import (
|
||||
PROTOCOL_VERSION,
|
||||
normalize_payload,
|
||||
)
|
||||
from services.mesh.mesh_hashchain import GENESIS_HASH
|
||||
from services.mesh.mesh_signed_events import (
|
||||
MeshWriteExemption,
|
||||
SignedWriteKind,
|
||||
@@ -324,6 +325,7 @@ from auth import (
|
||||
_validate_insecure_admin_startup,
|
||||
_validate_peer_push_secret,
|
||||
_verify_peer_push_hmac,
|
||||
_verify_peer_transport_hmac,
|
||||
)
|
||||
from node_state import (
|
||||
_NODE_BOOTSTRAP_STATE,
|
||||
@@ -370,6 +372,8 @@ osint_router = _load_optional_router("routers.osint")
|
||||
scm_router = _load_optional_router("routers.scm")
|
||||
entity_graph_router = _load_optional_router("routers.entity_graph")
|
||||
intel_feeds_router = _load_optional_router("routers.intel_feeds")
|
||||
analytics_router = _load_optional_router("routers.analytics")
|
||||
agent_shell_router = _load_optional_router("routers.agent_shell")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1069,6 +1073,10 @@ def _release_gate_status(
|
||||
|
||||
|
||||
def _validate_privacy_core_startup() -> None:
|
||||
# The wormhole child agent reuses this app on WORMHOLE_PORT; the parent
|
||||
# backend already validated privacy-core before spawning it.
|
||||
if os.environ.get("WORMHOLE_PORT"):
|
||||
return
|
||||
from services.privacy_core_attestation import validate_privacy_core_startup
|
||||
|
||||
validate_privacy_core_startup()
|
||||
@@ -1240,6 +1248,26 @@ def _local_infonet_peer_url() -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _clear_stale_arti_sync_backoff() -> None:
|
||||
"""Drop cached Arti warmup errors once SOCKS transport is actually ready."""
|
||||
from dataclasses import replace
|
||||
|
||||
with _NODE_RUNTIME_LOCK:
|
||||
current = get_sync_state()
|
||||
error_lower = str(current.last_error or "").lower()
|
||||
if "arti" not in error_lower and "onion sync requires" not in error_lower:
|
||||
return
|
||||
set_sync_state(
|
||||
replace(
|
||||
current,
|
||||
last_error="",
|
||||
consecutive_failures=0,
|
||||
next_sync_due_at=int(time.time()),
|
||||
last_outcome="idle" if current.last_outcome == "error" else current.last_outcome,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _ensure_infonet_private_transport_ready(reason: str = "") -> bool:
|
||||
"""Warm the local onion transport before private Infonet sync.
|
||||
|
||||
@@ -1268,14 +1296,36 @@ def _ensure_infonet_private_transport_ready(reason: str = "") -> bool:
|
||||
|
||||
label = f" ({reason})" if reason else ""
|
||||
logger.info("Infonet private transport warmup starting%s", label)
|
||||
tor_result = tor_service.start(target_port=8000)
|
||||
if tor_result.get("ok"):
|
||||
from services.wormhole_supervisor import invalidate_arti_ready_cache
|
||||
|
||||
for attempt in range(3):
|
||||
tor_result = tor_service.start(target_port=8000)
|
||||
if not tor_result.get("ok"):
|
||||
logger.warning(
|
||||
"Infonet private transport warmup incomplete%s: %s",
|
||||
label,
|
||||
tor_result,
|
||||
)
|
||||
continue
|
||||
_write_env_value("MESH_ARTI_ENABLED", "true")
|
||||
get_settings.cache_clear()
|
||||
if _check_arti_ready():
|
||||
logger.info("Infonet private transport ready%s", label)
|
||||
return True
|
||||
logger.warning("Infonet private transport warmup incomplete%s: %s", label, tor_result)
|
||||
invalidate_arti_ready_cache()
|
||||
deadline = time.monotonic() + 30.0
|
||||
while time.monotonic() < deadline:
|
||||
if _check_arti_ready(force=True):
|
||||
logger.info("Infonet private transport ready%s", label)
|
||||
_clear_stale_arti_sync_backoff()
|
||||
threading.Thread(target=_swarm_bootstrap_after_transport_ready, daemon=True).start()
|
||||
_kick_public_sync_background(f"transport_ready{label}")
|
||||
return True
|
||||
time.sleep(1.0)
|
||||
logger.warning(
|
||||
"Infonet private transport SOCKS not ready after Tor start (attempt %d/3)%s",
|
||||
attempt + 1,
|
||||
label,
|
||||
)
|
||||
tor_service.stop()
|
||||
logger.warning("Infonet private transport warmup incomplete%s", label)
|
||||
return False
|
||||
except Exception as exc:
|
||||
logger.warning("Infonet private transport warmup failed: %s", exc)
|
||||
@@ -1285,10 +1335,14 @@ def _ensure_infonet_private_transport_ready(reason: str = "") -> bool:
|
||||
|
||||
|
||||
def _configured_bootstrap_seed_peer_urls() -> list[str]:
|
||||
from services.mesh.mesh_fleet_defaults import configured_bootstrap_seed_peers_with_fleet_default
|
||||
|
||||
settings = get_settings()
|
||||
primary = str(getattr(settings, "MESH_BOOTSTRAP_SEED_PEERS", "") or "").strip()
|
||||
legacy = str(getattr(settings, "MESH_DEFAULT_SYNC_PEERS", "") or "").strip()
|
||||
return parse_configured_relay_peers(primary or legacy)
|
||||
return configured_bootstrap_seed_peers_with_fleet_default(
|
||||
parse_configured_relay_peers(primary or legacy)
|
||||
)
|
||||
|
||||
|
||||
def _refresh_node_peer_store(*, now: float | None = None) -> dict[str, Any]:
|
||||
@@ -1415,6 +1469,16 @@ def _refresh_node_peer_store(*, now: float | None = None) -> dict[str, Any]:
|
||||
if private_transport_required and skipped_clearnet_peers and not bootstrap_error:
|
||||
bootstrap_error = _infonet_private_transport_error()
|
||||
|
||||
swarm_pull: dict[str, Any] = {}
|
||||
try:
|
||||
from services.mesh.mesh_swarm_runtime import refresh_swarm_manifest_from_seeds
|
||||
|
||||
swarm_pull = refresh_swarm_manifest_from_seeds(now=timestamp)
|
||||
if swarm_pull.get("ok") and not swarm_pull.get("skipped"):
|
||||
store.load()
|
||||
except Exception as exc:
|
||||
swarm_pull = {"ok": False, "detail": str(exc or type(exc).__name__)}
|
||||
|
||||
store.save()
|
||||
bootstrap_records = store.records_for_bucket("bootstrap")
|
||||
sync_records = store.records_for_bucket("sync")
|
||||
@@ -1423,6 +1487,8 @@ def _refresh_node_peer_store(*, now: float | None = None) -> dict[str, Any]:
|
||||
bootstrap_records = [record for record in bootstrap_records if _is_private_infonet_transport(record.transport)]
|
||||
sync_records = [record for record in sync_records if _is_private_infonet_transport(record.transport)]
|
||||
push_records = [record for record in push_records if _is_private_infonet_transport(record.transport)]
|
||||
swarm_sync_peer_count = len([record for record in sync_records if str(record.source or "") == "swarm"])
|
||||
swarm_push_peer_count = len([record for record in push_records if str(record.source or "") == "swarm"])
|
||||
snapshot = {
|
||||
"node_mode": mode,
|
||||
"private_transport_required": private_transport_required,
|
||||
@@ -1434,16 +1500,29 @@ def _refresh_node_peer_store(*, now: float | None = None) -> dict[str, Any]:
|
||||
"bootstrap_peer_count": len(bootstrap_records),
|
||||
"sync_peer_count": len(sync_records),
|
||||
"push_peer_count": len(push_records),
|
||||
"swarm_sync_peer_count": swarm_sync_peer_count,
|
||||
"swarm_push_peer_count": swarm_push_peer_count,
|
||||
"operator_peer_count": len(operator_peers),
|
||||
"bootstrap_seed_peer_count": len(bootstrap_seed_peers),
|
||||
"default_sync_peer_count": len(bootstrap_seed_peers),
|
||||
"last_bootstrap_error": bootstrap_error,
|
||||
"swarm_manifest_pull": swarm_pull,
|
||||
}
|
||||
with _NODE_RUNTIME_LOCK:
|
||||
_NODE_BOOTSTRAP_STATE.update(snapshot)
|
||||
return snapshot
|
||||
|
||||
|
||||
def _swarm_bootstrap_after_transport_ready() -> None:
|
||||
try:
|
||||
from services.mesh.mesh_swarm_runtime import join_swarm_with_retries
|
||||
|
||||
join_swarm_with_retries(attempts=4, delay_s=15.0, force=True)
|
||||
_refresh_node_peer_store()
|
||||
except Exception:
|
||||
logger.warning("swarm bootstrap after transport ready failed", exc_info=True)
|
||||
|
||||
|
||||
def _materialize_local_infonet_state() -> None:
|
||||
from services.mesh.mesh_hashchain import infonet
|
||||
|
||||
@@ -1591,6 +1670,12 @@ def _hydrate_dm_relay_from_chain(events: list[dict]) -> int:
|
||||
sender_token_hash = hashlib.sha256(
|
||||
f"hashchain-dm-sender|{event_id}|{canonical.get('node_id', '')}".encode("utf-8")
|
||||
).hexdigest()
|
||||
try:
|
||||
from services.mesh.mesh_dm_connect_delivery import relay_push_peer_urls_for_payload
|
||||
|
||||
replication_urls = relay_push_peer_urls_for_payload(dict(payload))
|
||||
except Exception:
|
||||
replication_urls = []
|
||||
try:
|
||||
result = dm_relay.deposit(
|
||||
sender_id=str(canonical.get("node_id", "") or ""),
|
||||
@@ -1604,6 +1689,7 @@ def _hydrate_dm_relay_from_chain(events: list[dict]) -> int:
|
||||
sender_token_hash=sender_token_hash,
|
||||
payload_format=str(payload.get("format", "dm1") or "dm1"),
|
||||
session_welcome=str(payload.get("session_welcome", "") or ""),
|
||||
replication_peer_urls=replication_urls,
|
||||
)
|
||||
if result.get("ok"):
|
||||
count += 1
|
||||
@@ -1668,7 +1754,29 @@ def _sync_from_peer(
|
||||
_hydrate_dm_relay_from_chain(events)
|
||||
rejected = list(result.get("rejected", []) or [])
|
||||
if rejected:
|
||||
return False, f"sync ingest rejected {len(rejected)} event(s)", False, 0
|
||||
reasons = [
|
||||
str((item or {}).get("reason", "") or "").strip()
|
||||
for item in rejected
|
||||
if isinstance(item, dict)
|
||||
]
|
||||
reason_summary = ", ".join(reason for reason in reasons if reason)
|
||||
detail = f"sync ingest rejected {len(rejected)} event(s)"
|
||||
if reason_summary:
|
||||
detail = f"{detail}: {reason_summary}"
|
||||
local_empty = len(infonet.events) == 0
|
||||
stale_genesis = (
|
||||
local_empty
|
||||
and bool(events)
|
||||
and str((events[0] or {}).get("prev_hash", "") or "") == GENESIS_HASH
|
||||
and any("timestamp outside freshness window" in reason.lower() for reason in reasons)
|
||||
)
|
||||
if stale_genesis:
|
||||
detail = (
|
||||
f"{detail}; peer appears to be serving an expired genesis chain. "
|
||||
"Refresh or reset the peer chain, or perform an explicit one-time migration "
|
||||
"with MESH_INGEST_EVENT_MAX_AGE_S=0."
|
||||
)
|
||||
return False, detail, False, 0
|
||||
if int(result.get("accepted", 0) or 0) == 0 and int(result.get("duplicates", 0) or 0) >= len(events):
|
||||
return True, "", False, 0
|
||||
if len(events) < page_limit:
|
||||
@@ -1921,9 +2029,22 @@ def _propagate_public_event_to_peers(event_dict: dict[str, Any]) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _propagate_ledger_event_to_peers(event_dict: dict[str, Any]) -> None:
|
||||
if not _participant_node_enabled():
|
||||
return
|
||||
event_type = str(event_dict.get("event_type") or "")
|
||||
if event_type in {"gate_message", "dm_message"}:
|
||||
from services.mesh.mesh_swarm_runtime import push_infonet_events_to_http_peers
|
||||
|
||||
push_infonet_events_to_http_peers([event_dict])
|
||||
_kick_public_sync_background("ledger_event")
|
||||
return
|
||||
_propagate_public_event_to_peers(event_dict)
|
||||
|
||||
|
||||
def _schedule_public_event_propagation(event_dict: dict[str, Any]) -> None:
|
||||
threading.Thread(
|
||||
target=_propagate_public_event_to_peers,
|
||||
target=_propagate_ledger_event_to_peers,
|
||||
args=(dict(event_dict),),
|
||||
daemon=True,
|
||||
).start()
|
||||
@@ -1959,6 +2080,7 @@ def _start_infonet_node_runtime(reason: str = "startup") -> None:
|
||||
threading.Thread(target=_http_peer_push_loop, daemon=True).start()
|
||||
threading.Thread(target=_http_gate_push_loop, daemon=True).start()
|
||||
threading.Thread(target=_http_gate_pull_loop, daemon=True).start()
|
||||
threading.Thread(target=_swarm_manifest_pull_loop, daemon=True).start()
|
||||
_NODE_RUNTIME_THREADS_STARTED = True
|
||||
_kick_public_sync_background(reason)
|
||||
if not _NODE_PUBLIC_EVENT_HOOK_REGISTERED:
|
||||
@@ -2066,6 +2188,22 @@ def _http_peer_push_loop() -> None:
|
||||
_NODE_SYNC_STOP.wait(_PEER_PUSH_INTERVAL_S)
|
||||
|
||||
|
||||
def _swarm_manifest_pull_loop() -> None:
|
||||
"""Background thread: pull signed peer manifests from bootstrap seeds."""
|
||||
while not _NODE_SYNC_STOP.is_set():
|
||||
try:
|
||||
if _participant_node_enabled():
|
||||
from services.mesh.mesh_swarm_runtime import refresh_swarm_manifest_from_seeds
|
||||
|
||||
result = refresh_swarm_manifest_from_seeds()
|
||||
if result.get("ok") and not result.get("skipped"):
|
||||
_refresh_node_peer_store()
|
||||
except Exception:
|
||||
logger.exception("swarm manifest pull loop error")
|
||||
interval_s = int(getattr(get_settings(), "MESH_SWARM_MANIFEST_PULL_INTERVAL_S", 0) or 300)
|
||||
_NODE_SYNC_STOP.wait(max(30, interval_s))
|
||||
|
||||
|
||||
# ─── Background Gate Message Pull Worker ─────────────────────────────────
|
||||
# Periodically pulls gate events from relay peers that this node is missing.
|
||||
# Complements the push loop: push sends OUR events to peers, pull fetches
|
||||
@@ -2610,8 +2748,10 @@ async def lifespan(app: FastAPI):
|
||||
if not _MESH_ONLY:
|
||||
def _startup_wormhole_runtime():
|
||||
try:
|
||||
from services.mesh.mesh_infonet_relay_bootstrap import ensure_infonet_relay_wormhole_ready
|
||||
from services.wormhole_supervisor import get_wormhole_state, sync_wormhole_with_settings
|
||||
|
||||
ensure_infonet_relay_wormhole_ready(reason="startup_relay")
|
||||
sync_wormhole_with_settings()
|
||||
_resume_private_delivery_background_work(
|
||||
current_tier=_current_private_lane_tier(get_wormhole_state()),
|
||||
@@ -3386,7 +3526,10 @@ def _request_private_surface_warmup(*, path: str, method: str, current_tier: str
|
||||
|
||||
|
||||
def _is_invite_scoped_prekey_bundle_lookup(request: Request, path: str) -> bool:
|
||||
if request.method.upper() != "GET" or str(path or "").strip() != "/api/mesh/dm/prekey-bundle":
|
||||
if request.method.upper() != "GET":
|
||||
return False
|
||||
normalized_path = str(path or "").strip()
|
||||
if normalized_path not in {"/api/mesh/dm/prekey-bundle", "/api/mesh/dm/pubkey"}:
|
||||
return False
|
||||
try:
|
||||
lookup_token = str(request.query_params.get("lookup_token", "") or "").strip()
|
||||
@@ -3487,6 +3630,14 @@ async def enforce_high_privacy_mesh(request: Request, call_next):
|
||||
except Exception:
|
||||
logger.debug("Private surface warm-up request failed", exc_info=True)
|
||||
required_tier = _minimum_transport_tier(path, request.method)
|
||||
if required_tier:
|
||||
from services.mesh.mesh_privacy_policy import runtime_route_enforcement_tier
|
||||
|
||||
required_tier = runtime_route_enforcement_tier(
|
||||
path,
|
||||
request.method,
|
||||
static_tier=required_tier,
|
||||
)
|
||||
if required_tier:
|
||||
if not _transport_tier_is_sufficient(current_tier, required_tier):
|
||||
if request.method.upper() == "POST" and path == "/api/mesh/dm/send":
|
||||
@@ -3651,6 +3802,8 @@ app.include_router(osint_router)
|
||||
app.include_router(scm_router)
|
||||
app.include_router(entity_graph_router)
|
||||
app.include_router(intel_feeds_router)
|
||||
app.include_router(analytics_router)
|
||||
app.include_router(agent_shell_router)
|
||||
|
||||
from services.data_fetcher import update_all_data
|
||||
|
||||
@@ -3774,7 +3927,9 @@ class LayerUpdate(BaseModel):
|
||||
async def update_layers(update: LayerUpdate, request: Request):
|
||||
"""Receive frontend layer toggle state. Starts/stops streams accordingly."""
|
||||
from services.fetchers._store import active_layers, bump_active_layers_version, is_any_active
|
||||
from services.layer_enable_refresh import refresh_newly_enabled_layers, snapshot_active_layers
|
||||
|
||||
layers_before = snapshot_active_layers()
|
||||
# Snapshot old stream states before applying changes
|
||||
old_ships = is_any_active(
|
||||
"ships_military", "ships_cargo", "ships_civilian", "ships_passenger", "ships_tracked_yachts"
|
||||
@@ -3782,8 +3937,6 @@ async def update_layers(update: LayerUpdate, request: Request):
|
||||
old_mesh = is_any_active("sigint_meshtastic")
|
||||
old_aprs = is_any_active("sigint_aprs")
|
||||
old_viirs = is_any_active("viirs_nightlights")
|
||||
old_datacenters = is_any_active("datacenters")
|
||||
old_fishing = is_any_active("fishing_activity")
|
||||
|
||||
# Update only known keys
|
||||
changed = False
|
||||
@@ -3802,8 +3955,6 @@ async def update_layers(update: LayerUpdate, request: Request):
|
||||
new_mesh = is_any_active("sigint_meshtastic")
|
||||
new_aprs = is_any_active("sigint_aprs")
|
||||
new_viirs = is_any_active("viirs_nightlights")
|
||||
new_datacenters = is_any_active("datacenters")
|
||||
new_fishing = is_any_active("fishing_activity")
|
||||
|
||||
# Start/stop AIS stream on transition
|
||||
if old_ships and not new_ships:
|
||||
@@ -3859,17 +4010,7 @@ async def update_layers(update: LayerUpdate, request: Request):
|
||||
_queue_viirs_change_refresh()
|
||||
logger.info("VIIRS change refresh queued (layer enabled)")
|
||||
|
||||
if not old_datacenters and new_datacenters:
|
||||
from services.fetchers.infrastructure import fetch_datacenters
|
||||
|
||||
fetch_datacenters()
|
||||
logger.info("Datacenters loaded (layer enabled)")
|
||||
|
||||
if not old_fishing and new_fishing:
|
||||
from services.fetchers.geo import fetch_fishing_activity
|
||||
|
||||
fetch_fishing_activity()
|
||||
logger.info("Fishing activity refresh queued (layer enabled)")
|
||||
refresh_newly_enabled_layers(layers_before)
|
||||
|
||||
return {"status": "ok"}
|
||||
|
||||
@@ -5495,6 +5636,65 @@ async def infonet_ingest(request: Request):
|
||||
return {"ok": True, **result}
|
||||
|
||||
|
||||
@app.get("/api/mesh/infonet/peer-registry", dependencies=[Depends(require_local_operator)])
|
||||
@limiter.limit("30/minute")
|
||||
async def infonet_peer_registry(request: Request):
|
||||
"""Operator view of the live swarm peer registry (seed nodes only)."""
|
||||
from services.mesh.mesh_peer_registry import DEFAULT_PEER_REGISTRY_PATH, PeerRegistry
|
||||
from services.mesh.mesh_swarm_runtime import peer_registry_enabled
|
||||
|
||||
if not peer_registry_enabled():
|
||||
return {"ok": False, "detail": "peer registry is not enabled on this node"}
|
||||
registry = PeerRegistry(DEFAULT_PEER_REGISTRY_PATH)
|
||||
try:
|
||||
peers = registry.load()
|
||||
except Exception as exc:
|
||||
return {"ok": False, "detail": str(exc or type(exc).__name__)}
|
||||
return {
|
||||
"ok": True,
|
||||
"peer_count": len(peers),
|
||||
"peers": [peer.to_dict() for peer in peers],
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/mesh/infonet/bootstrap-manifest")
|
||||
@limiter.limit(_INFONET_SYNC_RATE_LIMIT)
|
||||
async def infonet_bootstrap_manifest(request: Request):
|
||||
"""Return the current signed bootstrap/swarm peer manifest."""
|
||||
from services.mesh.mesh_swarm_runtime import load_live_bootstrap_manifest
|
||||
|
||||
manifest = load_live_bootstrap_manifest()
|
||||
if manifest is None:
|
||||
return {"ok": False, "detail": "bootstrap manifest unavailable"}
|
||||
return {"ok": True, "manifest": manifest.to_dict()}
|
||||
|
||||
|
||||
@app.post("/api/mesh/infonet/peer-announce")
|
||||
@limiter.limit("30/minute")
|
||||
@mesh_write_exempt(MeshWriteExemption.PEER_GOSSIP)
|
||||
async def infonet_peer_announce(request: Request):
|
||||
"""Register a participant onion peer in the swarm registry (HMAC-authenticated)."""
|
||||
from auth import _peer_hmac_url_from_request
|
||||
from services.mesh.mesh_swarm_runtime import peer_registry_enabled, record_peer_announcement
|
||||
|
||||
body_bytes = await request.body()
|
||||
if not _verify_peer_transport_hmac(request, body_bytes):
|
||||
return Response(
|
||||
content='{"ok":false,"detail":"Invalid or missing peer HMAC"}',
|
||||
status_code=403,
|
||||
media_type="application/json",
|
||||
)
|
||||
if not peer_registry_enabled():
|
||||
return {"ok": False, "detail": "peer registry is not enabled on this node"}
|
||||
body = json_mod.loads(body_bytes or b"{}")
|
||||
announced_url = normalize_peer_url(str(body.get("peer_url", "") or ""))
|
||||
header_url = _peer_hmac_url_from_request(request)
|
||||
if not announced_url or announced_url != header_url:
|
||||
return {"ok": False, "detail": "peer_url must match X-Peer-Url"}
|
||||
peer = record_peer_announcement(body)
|
||||
return {"ok": True, "peer_url": peer.peer_url, "role": peer.role, "transport": peer.transport}
|
||||
|
||||
|
||||
@app.post("/api/mesh/infonet/peer-push")
|
||||
@limiter.limit("30/minute")
|
||||
@mesh_write_exempt(MeshWriteExemption.PEER_GOSSIP)
|
||||
@@ -5533,6 +5733,8 @@ async def infonet_peer_push(request: Request):
|
||||
result = infonet.ingest_events(events)
|
||||
_hydrate_gate_store_from_chain(events)
|
||||
_hydrate_dm_relay_from_chain(events)
|
||||
if any(str(event.get("event_type") or "") in {"gate_message", "dm_message"} for event in events):
|
||||
_kick_public_sync_background("peer_push_ingest")
|
||||
return {"ok": True, **result}
|
||||
|
||||
|
||||
@@ -6717,12 +6919,22 @@ def _queue_dm_release(*, current_tier: str, payload: dict[str, Any]) -> dict[str
|
||||
required_tier=release_lane_required_tier("dm"),
|
||||
)
|
||||
_wake_private_release_worker()
|
||||
outbox_id = str(item.get("id", "") or "")
|
||||
auto_release: dict[str, Any] = {"ok": True, "skipped": True}
|
||||
if outbox_id:
|
||||
try:
|
||||
from services.mesh.mesh_dm_connect_delivery import auto_release_connect_dm_outbox
|
||||
|
||||
auto_release = auto_release_connect_dm_outbox(outbox_id=outbox_id, payload=payload)
|
||||
except Exception as exc:
|
||||
auto_release = {"ok": False, "detail": str(exc) or type(exc).__name__}
|
||||
return {
|
||||
"ok": True,
|
||||
"msg_id": str(payload.get("msg_id", "") or ""),
|
||||
"outbox_id": str(item.get("id", "") or ""),
|
||||
"outbox_id": outbox_id,
|
||||
"queued": True,
|
||||
"detail": str((item.get("status") or {}).get("label", "") or "Queued for private delivery"),
|
||||
"auto_release": auto_release,
|
||||
"delivery": {
|
||||
"state": canonical_release_state(str(item.get("release_state", "") or "queued")),
|
||||
"internal_state": str(item.get("release_state", "") or "queued"),
|
||||
@@ -6895,7 +7107,8 @@ async def _dm_send_from_signed_request(request: Request):
|
||||
return {"ok": False, "detail": "DM timestamp is too far from current time"}
|
||||
if delivery_class not in ("request", "shared"):
|
||||
return {"ok": False, "detail": "delivery_class must be request or shared"}
|
||||
if delivery_class == "request":
|
||||
# Contact requests are the first-contact handshake — do not require prior verification.
|
||||
if delivery_class == "shared":
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_contacts import verified_first_contact_requirement
|
||||
|
||||
@@ -6979,6 +7192,8 @@ async def _dm_send_from_signed_request(request: Request):
|
||||
|
||||
relay_salt_hex = _os.urandom(16).hex()
|
||||
|
||||
connect_intent = str(body.get("connect_intent", "") or "").strip().lower()
|
||||
lookup_peer_url = str(body.get("lookup_peer_url", "") or "").strip().rstrip("/")
|
||||
release_payload = {
|
||||
"sender_id": sender_id,
|
||||
"sender_token_hash": sender_token_hash,
|
||||
@@ -6993,6 +7208,16 @@ async def _dm_send_from_signed_request(request: Request):
|
||||
"sender_seal": sender_seal,
|
||||
"relay_salt": relay_salt_hex,
|
||||
}
|
||||
if connect_intent:
|
||||
release_payload["connect_intent"] = connect_intent
|
||||
if lookup_peer_url:
|
||||
release_payload["lookup_peer_url"] = lookup_peer_url
|
||||
try:
|
||||
from services.mesh.mesh_dm_connect_delivery import enrich_connect_release_payload
|
||||
|
||||
release_payload = enrich_connect_release_payload(release_payload)
|
||||
except Exception:
|
||||
pass
|
||||
hashchain_spool: dict[str, Any] = {"ok": False, "detail": "not attempted"}
|
||||
try:
|
||||
from services.mesh.mesh_hashchain import infonet
|
||||
@@ -7009,6 +7234,10 @@ async def _dm_send_from_signed_request(request: Request):
|
||||
"format": payload_format,
|
||||
}
|
||||
chain_payload["transport_lock"] = "private_strong"
|
||||
if connect_intent:
|
||||
chain_payload["connect_intent"] = connect_intent
|
||||
if lookup_peer_url:
|
||||
chain_payload["lookup_peer_url"] = lookup_peer_url
|
||||
chain_event = infonet.append_private_dm_message(
|
||||
node_id=sender_id,
|
||||
payload=chain_payload,
|
||||
@@ -7024,7 +7253,8 @@ async def _dm_send_from_signed_request(request: Request):
|
||||
or PROTOCOL_VERSION,
|
||||
timestamp=float(timestamp or time.time()),
|
||||
)
|
||||
_hydrate_dm_relay_from_chain([chain_event])
|
||||
# Relay deposit is deferred to the private release worker so scoped
|
||||
# connect traffic can synchronously replicate to lookup_peer_url once.
|
||||
hashchain_spool = {
|
||||
"ok": True,
|
||||
"event_id": str(chain_event.get("event_id", "") or ""),
|
||||
@@ -7279,7 +7509,12 @@ async def dm_register_key(request: Request):
|
||||
|
||||
@app.get("/api/mesh/dm/pubkey")
|
||||
@limiter.limit("30/minute")
|
||||
async def dm_get_pubkey(request: Request, agent_id: str = "", lookup_token: str = ""):
|
||||
async def dm_get_pubkey(
|
||||
request: Request,
|
||||
agent_id: str = "",
|
||||
lookup_token: str = "",
|
||||
lookup_peer_url: str = "",
|
||||
):
|
||||
"""Fetch an agent's DH public key for key exchange."""
|
||||
exposure = metadata_exposure_for_request(
|
||||
request,
|
||||
@@ -7299,11 +7534,49 @@ async def dm_get_pubkey(request: Request, agent_id: str = "", lookup_token: str
|
||||
if resolved_lookup:
|
||||
key_bundle, resolved_id = dm_relay.get_dh_key_by_lookup(resolved_lookup)
|
||||
if key_bundle is None:
|
||||
return dm_lookup_response_view(
|
||||
{"ok": False, "detail": "Agent not found or has no DH key", "lookup_mode": "invite_lookup_handle"},
|
||||
exposure=exposure,
|
||||
lookup_token_present=True,
|
||||
# Invite handles are minted on the owner's node. When a remote peer
|
||||
# pastes a short address, resolve it across the private fleet before
|
||||
# failing — same path as prekey-bundle import.
|
||||
from services.mesh.mesh_wormhole_prekey import fetch_dm_prekey_bundle
|
||||
|
||||
preferred_lookup_peer = str(lookup_peer_url or "").strip().rstrip("/")
|
||||
remote_bundle = fetch_dm_prekey_bundle(
|
||||
agent_id="",
|
||||
lookup_token=resolved_lookup,
|
||||
lookup_peer_urls=[preferred_lookup_peer] if preferred_lookup_peer else None,
|
||||
)
|
||||
if remote_bundle.get("ok"):
|
||||
bundle = dict(remote_bundle.get("bundle") or remote_bundle)
|
||||
dh_pub = str(
|
||||
bundle.get("identity_dh_pub_key", "")
|
||||
or remote_bundle.get("identity_dh_pub_key", "")
|
||||
or ""
|
||||
).strip()
|
||||
if dh_pub:
|
||||
resolved_id = str(remote_bundle.get("agent_id", "") or resolved_id or "").strip()
|
||||
key_bundle = {
|
||||
"dh_pub_key": dh_pub,
|
||||
"dh_algo": str(remote_bundle.get("dh_algo", "X25519") or "X25519"),
|
||||
"timestamp": int(remote_bundle.get("timestamp", 0) or 0),
|
||||
"public_key": str(remote_bundle.get("public_key", "") or ""),
|
||||
"public_key_algo": str(remote_bundle.get("public_key_algo", "") or ""),
|
||||
"signature": str(remote_bundle.get("signature", "") or ""),
|
||||
"sequence": int(remote_bundle.get("sequence", 0) or 0),
|
||||
"prekey_transparency_head": str(
|
||||
remote_bundle.get("prekey_transparency_head", "") or ""
|
||||
),
|
||||
"prekey_transparency_size": int(
|
||||
remote_bundle.get("prekey_transparency_size", 0) or 0
|
||||
),
|
||||
"witness_count": int(remote_bundle.get("witness_count", 0) or 0),
|
||||
"witness_latest_at": int(remote_bundle.get("witness_latest_at", 0) or 0),
|
||||
}
|
||||
if key_bundle is None:
|
||||
return dm_lookup_response_view(
|
||||
{"ok": False, "detail": "Agent not found or has no DH key", "lookup_mode": "invite_lookup_handle"},
|
||||
exposure=exposure,
|
||||
lookup_token_present=True,
|
||||
)
|
||||
lookup_mode = "invite_lookup_handle"
|
||||
if key_bundle is None and resolved_id:
|
||||
blocked = legacy_agent_id_lookup_blocked()
|
||||
@@ -7339,7 +7612,12 @@ async def dm_get_pubkey(request: Request, agent_id: str = "", lookup_token: str
|
||||
|
||||
@app.get("/api/mesh/dm/prekey-bundle")
|
||||
@limiter.limit("30/minute")
|
||||
async def dm_get_prekey_bundle(request: Request, agent_id: str = "", lookup_token: str = ""):
|
||||
async def dm_get_prekey_bundle(
|
||||
request: Request,
|
||||
agent_id: str = "",
|
||||
lookup_token: str = "",
|
||||
lookup_peer_url: str = "",
|
||||
):
|
||||
exposure = metadata_exposure_for_request(
|
||||
request,
|
||||
authenticated=_scoped_view_authenticated(request, "mesh"),
|
||||
@@ -7351,7 +7629,12 @@ async def dm_get_prekey_bundle(request: Request, agent_id: str = "", lookup_toke
|
||||
lookup_token_present=bool(lookup_token),
|
||||
)
|
||||
resolved_id, resolved_lookup = _preferred_dm_lookup_target(agent_id, lookup_token)
|
||||
result = fetch_dm_prekey_bundle(agent_id=resolved_id, lookup_token=resolved_lookup)
|
||||
preferred_lookup_peer = str(lookup_peer_url or "").strip().rstrip("/")
|
||||
result = fetch_dm_prekey_bundle(
|
||||
agent_id=resolved_id,
|
||||
lookup_token=resolved_lookup,
|
||||
lookup_peer_urls=[preferred_lookup_peer] if preferred_lookup_peer else None,
|
||||
)
|
||||
return dm_lookup_response_view(
|
||||
result,
|
||||
exposure=exposure,
|
||||
@@ -7857,6 +8140,7 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
|
||||
"tripcheck.com", # Oregon DOT / TripCheck
|
||||
"www.tripcheck.com",
|
||||
"infocar.dgt.es", # Spain DGT
|
||||
"etraffic.dgt.es", # Spain DGT (etrafficWEB cameras host, 2026)
|
||||
"informo.madrid.es", # Madrid
|
||||
"webcams2.asfinag.at", # Austria ASFINAG motorway cameras
|
||||
"odo.asfinag.at", # ASFINAG catalog API host
|
||||
@@ -8053,14 +8337,14 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
|
||||
cache_seconds=30,
|
||||
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8"},
|
||||
)
|
||||
if host == "infocar.dgt.es":
|
||||
if host in {"infocar.dgt.es", "etraffic.dgt.es"}:
|
||||
return _CCTVProxyProfile(
|
||||
name="dgt-spain",
|
||||
timeout=(5.0, 8.0),
|
||||
cache_seconds=60,
|
||||
headers={
|
||||
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
|
||||
"Referer": "https://infocar.dgt.es/",
|
||||
"Referer": "https://etraffic.dgt.es/",
|
||||
},
|
||||
)
|
||||
if host == "informo.madrid.es":
|
||||
@@ -9094,9 +9378,35 @@ async def api_set_node_settings(request: Request, body: NodeSettingsUpdate):
|
||||
if bool(body.enabled):
|
||||
_start_infonet_node_runtime("operator_enable")
|
||||
_kick_public_sync_background("operator_enable")
|
||||
threading.Thread(target=_swarm_bootstrap_after_transport_ready, daemon=True).start()
|
||||
return result
|
||||
|
||||
|
||||
@app.post("/api/mesh/infonet/swarm/join")
|
||||
@limiter.limit("10/minute")
|
||||
async def infonet_swarm_join(request: Request):
|
||||
"""Announce this node to the fleet seed and pull the signed peer manifest."""
|
||||
if not _participant_node_enabled():
|
||||
return {"ok": False, "detail": "participant node is disabled"}
|
||||
if _infonet_private_transport_required() and not _ensure_infonet_private_transport_ready("swarm_join"):
|
||||
return JSONResponse(
|
||||
{"ok": False, "detail": _infonet_private_transport_error()},
|
||||
status_code=503,
|
||||
)
|
||||
|
||||
from services.mesh.mesh_swarm_runtime import announce_local_peer_to_seeds, refresh_swarm_manifest_from_seeds
|
||||
|
||||
announce = await asyncio.to_thread(announce_local_peer_to_seeds, force=True)
|
||||
manifest = await asyncio.to_thread(refresh_swarm_manifest_from_seeds, force=True)
|
||||
if manifest.get("ok"):
|
||||
await asyncio.to_thread(_refresh_node_peer_store)
|
||||
return {
|
||||
"ok": bool(announce.get("ok")) or bool(manifest.get("ok")),
|
||||
"announce": announce,
|
||||
"manifest_pull": manifest,
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/settings/wormhole")
|
||||
@limiter.limit("30/minute")
|
||||
async def api_get_wormhole_settings(request: Request):
|
||||
@@ -9175,7 +9485,8 @@ class WormholeDmResetRequest(BaseModel):
|
||||
|
||||
|
||||
class WormholeDmBootstrapEncryptRequest(BaseModel):
|
||||
peer_id: str
|
||||
peer_id: str = ""
|
||||
lookup_token: str = ""
|
||||
plaintext: str
|
||||
|
||||
|
||||
@@ -9400,6 +9711,43 @@ def _get_contact_trust_level(peer_id: str) -> str:
|
||||
return "unpinned"
|
||||
|
||||
|
||||
def _compose_bundle_matches_invite_pin(peer_id: str, bundle: dict[str, Any]) -> bool:
|
||||
"""True when an invite-pinned contact already matches the supplied bundle."""
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_contacts import list_wormhole_dm_contacts
|
||||
from services.mesh.mesh_wormhole_prekey import trust_fingerprint_for_bundle_record
|
||||
|
||||
contact = dict(list_wormhole_dm_contacts().get(str(peer_id or "").strip()) or {})
|
||||
if str(contact.get("trust_level", "") or "") != "invite_pinned":
|
||||
return False
|
||||
pinned = str(
|
||||
contact.get("remotePrekeyFingerprint", "")
|
||||
or contact.get("invitePinnedTrustFingerprint", "")
|
||||
or ""
|
||||
).strip().lower()
|
||||
if not pinned:
|
||||
return False
|
||||
bundle_record = dict(bundle or {})
|
||||
bundle_payload = dict(bundle_record.get("bundle") or bundle_record)
|
||||
candidate = str(bundle_record.get("trust_fingerprint", "") or "").strip().lower()
|
||||
if not candidate:
|
||||
candidate = str(
|
||||
trust_fingerprint_for_bundle_record(
|
||||
{
|
||||
"agent_id": str(peer_id or "").strip(),
|
||||
"bundle": bundle_payload,
|
||||
"public_key": str(bundle_record.get("public_key", "") or ""),
|
||||
"public_key_algo": str(bundle_record.get("public_key_algo", "") or "Ed25519"),
|
||||
"protocol_version": str(bundle_record.get("protocol_version", "") or ""),
|
||||
}
|
||||
)
|
||||
or ""
|
||||
).strip().lower()
|
||||
return bool(candidate and pinned == candidate)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def compose_wormhole_dm(
|
||||
*,
|
||||
peer_id: str,
|
||||
@@ -9464,8 +9812,11 @@ def compose_wormhole_dm(
|
||||
bundle = fetched_bundle
|
||||
if bundle and str(peer_id or "").strip():
|
||||
try:
|
||||
trust_state = observe_remote_prekey_bundle(str(peer_id or "").strip(), bundle)
|
||||
_compose_trust_level = str(trust_state.get("trust_level", "") or "")
|
||||
if _compose_bundle_matches_invite_pin(str(peer_id or "").strip(), bundle):
|
||||
_compose_trust_level = "invite_pinned"
|
||||
else:
|
||||
trust_state = observe_remote_prekey_bundle(str(peer_id or "").strip(), bundle)
|
||||
_compose_trust_level = str(trust_state.get("trust_level", "") or "")
|
||||
from services.mesh.mesh_wormhole_contacts import verified_first_contact_requirement
|
||||
|
||||
verified_first_contact = verified_first_contact_requirement(
|
||||
@@ -9646,21 +9997,11 @@ def decrypt_wormhole_dm_envelope(
|
||||
if not has_session.get("ok"):
|
||||
return has_session
|
||||
if not has_session.get("exists"):
|
||||
local_dh_secret = ""
|
||||
local_identity_alias = ""
|
||||
try:
|
||||
local_identity = read_wormhole_identity()
|
||||
local_dh_secret = str(local_identity.get("dh_private_key", "") or "")
|
||||
local_identity_alias = str(local_identity.get("node_id", "") or "")
|
||||
except Exception:
|
||||
local_dh_secret = ""
|
||||
local_identity_alias = ""
|
||||
ensured = ensure_mls_dm_session(
|
||||
resolved_local,
|
||||
resolved_remote,
|
||||
str(session_welcome or ""),
|
||||
local_dh_secret=local_dh_secret,
|
||||
identity_alias=local_identity_alias,
|
||||
identity_alias=resolved_local,
|
||||
)
|
||||
if not ensured.get("ok"):
|
||||
return ensured
|
||||
@@ -11137,9 +11478,12 @@ async def api_wormhole_dm_bootstrap_encrypt(request: Request, body: WormholeDmBo
|
||||
result = bootstrap_encrypt_for_peer(
|
||||
peer_id=str(body.peer_id or ""),
|
||||
plaintext=str(body.plaintext or ""),
|
||||
lookup_token=str(body.lookup_token or ""),
|
||||
)
|
||||
if isinstance(result, dict) and "trust_level" not in result:
|
||||
result["trust_level"] = _get_contact_trust_level(str(body.peer_id or ""))
|
||||
result["trust_level"] = _get_contact_trust_level(
|
||||
str(result.get("peer_id", "") or body.peer_id or "")
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
@@ -11155,7 +11499,7 @@ async def api_wormhole_dm_bootstrap_decrypt(request: Request, body: WormholeDmBo
|
||||
return result
|
||||
|
||||
|
||||
@app.post("/api/wormhole/dm/sender-token", dependencies=[Depends(require_admin)])
|
||||
@app.post("/api/wormhole/dm/sender-token", dependencies=[Depends(require_local_operator)])
|
||||
@limiter.limit("60/minute")
|
||||
async def api_wormhole_dm_sender_token(request: Request, body: WormholeDmSenderTokenRequest):
|
||||
if _safe_int(body.count or 1, 1) > 1:
|
||||
@@ -11374,7 +11718,25 @@ async def api_wormhole_dm_contact_delete(request: Request, peer_id: str):
|
||||
return {"ok": True, "peer_id": peer_id, "deleted": deleted}
|
||||
|
||||
|
||||
_WORMHOLE_PUBLIC_FIELDS = {"installed", "configured", "running", "ready"}
|
||||
@app.post("/api/wormhole/dm/contact/{peer_id}/sever", dependencies=[Depends(require_admin)])
|
||||
@limiter.limit("60/minute")
|
||||
async def api_wormhole_dm_contact_sever(request: Request, peer_id: str):
|
||||
from services.mesh.mesh_wormhole_contacts import sever_wormhole_dm_contact
|
||||
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
body = {}
|
||||
if not isinstance(body, dict):
|
||||
body = {}
|
||||
block = bool(body.get("block", False))
|
||||
try:
|
||||
return sever_wormhole_dm_contact(peer_id, block=block)
|
||||
except ValueError as exc:
|
||||
return {"ok": False, "detail": str(exc)}
|
||||
|
||||
|
||||
_WORMHOLE_PUBLIC_FIELDS = {"installed", "configured", "running", "ready", "arti_ready"}
|
||||
|
||||
|
||||
def _redact_wormhole_status(state: dict[str, Any], authenticated: bool) -> dict[str, Any]:
|
||||
|
||||
Generated
+4
-3
@@ -4,14 +4,15 @@
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "backend",
|
||||
"dependencies": {
|
||||
"ws": "^8.19.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ws": {
|
||||
"version": "8.19.0",
|
||||
"resolved": "https://registry.npmjs.org/ws/-/ws-8.19.0.tgz",
|
||||
"integrity": "sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg==",
|
||||
"version": "8.21.0",
|
||||
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz",
|
||||
"integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=10.0.0"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"dependencies": {
|
||||
"ws": "^8.19.0"
|
||||
"ws": "^8.21.0"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ py-modules = []
|
||||
|
||||
[project]
|
||||
name = "backend"
|
||||
version = "0.9.82"
|
||||
version = "0.9.83"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"apscheduler==3.10.3",
|
||||
@@ -15,7 +15,7 @@ dependencies = [
|
||||
"cachetools==5.5.2",
|
||||
"cryptography>=46.0.7",
|
||||
"defusedxml>=0.7.1",
|
||||
"fastapi==0.136.3",
|
||||
"fastapi==0.138.0",
|
||||
"feedparser==6.0.10",
|
||||
"httpx==0.28.1",
|
||||
"playwright==1.59.0",
|
||||
@@ -29,6 +29,8 @@ dependencies = [
|
||||
"reverse-geocoder==1.5.1",
|
||||
"sgp4==2.25",
|
||||
"meshtastic>=2.5.0",
|
||||
"networkx>=3.4.0",
|
||||
"numpy>=2.2.0",
|
||||
"orjson>=3.10.0",
|
||||
"paho-mqtt>=1.6.0,<2.0.0",
|
||||
"PyNaCl>=1.5.0",
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Local-operator PTY WebSocket for the Mesh Chat agent shell."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import fcntl
|
||||
import hmac
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pty
|
||||
import select
|
||||
import signal
|
||||
import struct
|
||||
import sys
|
||||
import termios
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, WebSocket, WebSocketDisconnect
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from auth import (
|
||||
_current_admin_key,
|
||||
_debug_mode_enabled,
|
||||
_is_trusted_local_runtime_host,
|
||||
require_local_operator,
|
||||
)
|
||||
from services.agent_shell_settings import (
|
||||
get_agent_shell_settings,
|
||||
set_agent_shell_working_directory,
|
||||
)
|
||||
from services.agent_shell_ws_token import consume_agent_shell_ws_token, mint_agent_shell_ws_token
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["agent-shell"])
|
||||
|
||||
|
||||
class AgentShellSettingsUpdate(BaseModel):
|
||||
working_directory: str = Field(min_length=1)
|
||||
|
||||
|
||||
def _set_winsize(fd: int, rows: int, cols: int) -> None:
|
||||
winsize = struct.pack("HHHH", rows, cols, 0, 0)
|
||||
fcntl.ioctl(fd, termios.TIOCSWINSZ, winsize)
|
||||
|
||||
|
||||
async def _authorize_agent_shell_ws(
|
||||
ws: WebSocket,
|
||||
admin_key_query: str = "",
|
||||
ws_token_query: str = "",
|
||||
) -> None:
|
||||
host = (ws.client.host or "").lower() if ws.client else ""
|
||||
if _is_trusted_local_runtime_host(host) or (_debug_mode_enabled() and host == "test"):
|
||||
return
|
||||
if consume_agent_shell_ws_token(ws_token_query):
|
||||
return
|
||||
admin_key = _current_admin_key()
|
||||
presented = str(admin_key_query or ws.headers.get("x-admin-key", "") or "").strip()
|
||||
if admin_key and presented and hmac.compare_digest(presented.encode(), admin_key.encode()):
|
||||
return
|
||||
await ws.close(code=4403, reason="local operator access only")
|
||||
raise WebSocketDisconnect()
|
||||
|
||||
|
||||
def _resolve_shell_cwd(requested: str) -> str:
|
||||
requested = str(requested or "").strip()
|
||||
if requested:
|
||||
resolved = os.path.abspath(os.path.expanduser(requested))
|
||||
if os.path.isdir(resolved):
|
||||
return resolved
|
||||
return get_agent_shell_settings()["working_directory"]
|
||||
|
||||
|
||||
def _default_shell() -> str:
|
||||
if sys.platform == "win32":
|
||||
return os.environ.get("COMSPEC", "cmd.exe")
|
||||
return os.environ.get("SHELL", "/bin/bash")
|
||||
|
||||
|
||||
async def _relay_pty(master_fd: int, proc: asyncio.subprocess.Process, ws: WebSocket) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
while True:
|
||||
if proc.returncode is not None:
|
||||
break
|
||||
try:
|
||||
readable, _, _ = await loop.run_in_executor(
|
||||
None, lambda: select.select([master_fd], [], [], 0.05)
|
||||
)
|
||||
except Exception:
|
||||
break
|
||||
if master_fd in readable:
|
||||
try:
|
||||
chunk = os.read(master_fd, 4096)
|
||||
except OSError:
|
||||
break
|
||||
if not chunk:
|
||||
break
|
||||
await ws.send_bytes(chunk)
|
||||
try:
|
||||
message = await asyncio.wait_for(ws.receive(), timeout=0.05)
|
||||
except asyncio.TimeoutError:
|
||||
continue
|
||||
if message.get("type") == "websocket.disconnect":
|
||||
break
|
||||
if message.get("type") != "websocket.receive":
|
||||
continue
|
||||
if message.get("bytes"):
|
||||
os.write(master_fd, message["bytes"])
|
||||
continue
|
||||
text = message.get("text")
|
||||
if not text:
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
os.write(master_fd, text.encode("utf-8", errors="replace"))
|
||||
continue
|
||||
if payload.get("type") == "resize":
|
||||
rows = int(payload.get("rows") or 24)
|
||||
cols = int(payload.get("cols") or 80)
|
||||
_set_winsize(master_fd, max(rows, 2), max(cols, 2))
|
||||
|
||||
|
||||
@router.get("/api/agent-shell/settings", dependencies=[Depends(require_local_operator)])
|
||||
async def read_agent_shell_settings() -> dict[str, Any]:
|
||||
return get_agent_shell_settings()
|
||||
|
||||
|
||||
@router.post("/api/agent-shell/ws-token", dependencies=[Depends(require_local_operator)])
|
||||
async def mint_agent_shell_ws_token_route() -> dict[str, Any]:
|
||||
token, expires_in = mint_agent_shell_ws_token()
|
||||
return {"token": token, "expires_in": expires_in}
|
||||
|
||||
|
||||
@router.put("/api/agent-shell/settings", dependencies=[Depends(require_local_operator)])
|
||||
async def write_agent_shell_settings(body: AgentShellSettingsUpdate) -> dict[str, Any]:
|
||||
try:
|
||||
return set_agent_shell_working_directory(body.working_directory)
|
||||
except ValueError as exc:
|
||||
detail = str(exc)
|
||||
if detail == "working_directory_not_found":
|
||||
raise HTTPException(status_code=400, detail="Working directory does not exist") from exc
|
||||
raise HTTPException(status_code=400, detail="Working directory is required") from exc
|
||||
|
||||
|
||||
@router.websocket("/api/agent-shell/ws")
|
||||
async def agent_shell_websocket(
|
||||
ws: WebSocket,
|
||||
cwd: str = Query(default=""),
|
||||
cols: int = Query(default=80),
|
||||
rows: int = Query(default=24),
|
||||
admin_key: str = Query(default=""),
|
||||
ws_token: str = Query(default=""),
|
||||
) -> None:
|
||||
await ws.accept()
|
||||
try:
|
||||
await _authorize_agent_shell_ws(ws, admin_key, ws_token)
|
||||
except WebSocketDisconnect:
|
||||
return
|
||||
|
||||
if sys.platform == "win32":
|
||||
await ws.send_text(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "error",
|
||||
"message": "Host PTY is not available on Windows backend builds yet. Use the ShadowBroker desktop app or run the backend in Docker/Linux for an embedded shell.",
|
||||
}
|
||||
)
|
||||
)
|
||||
await ws.close(code=1011)
|
||||
return
|
||||
|
||||
shell_cwd = _resolve_shell_cwd(cwd)
|
||||
shell = _default_shell()
|
||||
master_fd, slave_fd = pty.openpty()
|
||||
_set_winsize(master_fd, max(rows, 2), max(cols, 2))
|
||||
|
||||
env = os.environ.copy()
|
||||
env.setdefault("TERM", "xterm-256color")
|
||||
env.setdefault("COLORTERM", "truecolor")
|
||||
home = shell_cwd if os.path.isdir(shell_cwd) else "/app"
|
||||
env["HOME"] = home
|
||||
env["USER"] = env.get("USER") or "operator"
|
||||
path_prefixes = [
|
||||
os.path.join(home, ".local", "bin"),
|
||||
os.path.join(home, ".hermes", "bin"),
|
||||
]
|
||||
path = env.get("PATH", "")
|
||||
for prefix in path_prefixes:
|
||||
if os.path.isdir(prefix):
|
||||
path = f"{prefix}:{path}" if path else prefix
|
||||
env["PATH"] = path
|
||||
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
shell,
|
||||
stdin=slave_fd,
|
||||
stdout=slave_fd,
|
||||
stderr=slave_fd,
|
||||
cwd=shell_cwd,
|
||||
env=env,
|
||||
preexec_fn=os.setsid,
|
||||
)
|
||||
os.close(slave_fd)
|
||||
|
||||
try:
|
||||
await _relay_pty(master_fd, proc, ws)
|
||||
finally:
|
||||
try:
|
||||
os.close(master_fd)
|
||||
except OSError:
|
||||
pass
|
||||
if proc.returncode is None:
|
||||
try:
|
||||
os.killpg(proc.pid, signal.SIGHUP)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
try:
|
||||
await asyncio.wait_for(proc.wait(), timeout=2.0)
|
||||
except asyncio.TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
@@ -2051,7 +2051,7 @@ async def agent_tool_manifest(request: Request):
|
||||
"description": "Set up a watchdog alert. When triggered, alerts push instantly via SSE stream. Debounced: same watch won't re-fire within 60 seconds.",
|
||||
"parameters": {
|
||||
"type": {"type": "string", "required": True, "description": "Watch type",
|
||||
"enum": ["track_aircraft", "track_callsign", "track_registration", "track_ship", "track_entity", "geofence", "keyword", "prediction_market"]},
|
||||
"enum": ["track_aircraft", "track_callsign", "track_registration", "track_ship", "track_entity", "geofence", "keyword", "telegram_rhetoric", "prediction_market"]},
|
||||
"params": {"type": "object", "required": True, "description": "Type-specific parameters (see subtypes)"},
|
||||
},
|
||||
"subtypes": {
|
||||
@@ -2061,7 +2061,8 @@ async def agent_tool_manifest(request: Request):
|
||||
"track_ship": {"params": {"mmsi": "string (optional)", "imo": "string (optional)", "name": "string (optional)", "owner": "string (optional)", "callsign": "string (optional)"}, "description": "Alert when ship appears by MMSI, IMO, name, owner, or callsign"},
|
||||
"track_entity": {"params": {"query": "string", "entity_type": "string (optional)", "layers": "list (optional)"}, "description": "Generic exact-first entity tracker when aircraft/ship fields are not known yet"},
|
||||
"geofence": {"params": {"lat": "float", "lng": "float", "radius_km": "float (default 50)", "entity_types": "list (default ['flights','ships'])"}, "description": "Alert when any entity enters a geographic zone"},
|
||||
"keyword": {"params": {"keyword": "string"}, "description": "Alert when keyword appears in news/GDELT headlines"},
|
||||
"keyword": {"params": {"keyword": "string", "include_telegram": "boolean (default true)"}, "description": "Alert when keyword appears in news, GDELT, or Telegram OSINT (searches translated + original text)"},
|
||||
"telegram_rhetoric": {"params": {"min_risk_score": "int 1-10 (default 7)", "keywords": "list or comma-separated string (optional)", "channels": "list or comma-separated string (optional)"}, "description": "Alert on new high-risk Telegram OSINT posts — rhetoric/escalation monitor"},
|
||||
"prediction_market": {"params": {"query": "string", "threshold": "float 0-1 (optional)"}, "description": "Alert on prediction market movements matching query"},
|
||||
},
|
||||
"example": {"cmd": "add_watch", "args": {"type": "track_registration", "params": {"registration": "N3880"}}},
|
||||
@@ -2276,12 +2277,14 @@ async def agent_tool_manifest(request: Request):
|
||||
async def api_capabilities(request: Request):
|
||||
"""Return full API manifest so the agent knows every available endpoint."""
|
||||
from services.openclaw_channel import READ_COMMANDS, WRITE_COMMANDS, detect_tier
|
||||
from services.openclaw_routing import routing_manifest
|
||||
from services.config import get_settings
|
||||
tier = detect_tier()
|
||||
access_tier = str(get_settings().OPENCLAW_ACCESS_TIER or "restricted").strip().lower()
|
||||
return {
|
||||
"ok": True,
|
||||
"version": "0.9.82",
|
||||
"routing": routing_manifest(),
|
||||
"auth": {
|
||||
"method": "HMAC-SHA256",
|
||||
"headers": ["X-SB-Timestamp", "X-SB-Nonce", "X-SB-Signature"],
|
||||
@@ -2397,8 +2400,16 @@ async def api_capabilities(request: Request):
|
||||
"description": "Compact server-side ship search by MMSI/IMO/name/query, including yacht-owner enrichment.",
|
||||
},
|
||||
"find_entity": {
|
||||
"args": {"query": "str (optional)", "entity_type": "aircraft|ship|person|event|infrastructure (optional)", "callsign": "str (optional)", "registration": "str (optional)", "icao24": "str (optional)", "mmsi": "str (optional)", "imo": "str (optional)", "name": "str (optional)", "owner": "str (optional)", "layers": "list[str] (optional)", "limit": "int (default 10)"},
|
||||
"description": "Exact-first resolver for planes, ships, operators, callsigns, registrations, MMSI/IMO, and named entities. Use before tracking to avoid fuzzy prompt matching.",
|
||||
"args": {"query": "str (optional)", "entity_type": "aircraft|ship|person|event|infrastructure (optional)", "callsign": "str (optional)", "registration": "str (optional)", "icao24": "str (optional)", "mmsi": "str (optional)", "imo": "str (optional)", "name": "str (optional)", "owner": "str (optional)", "layers": "list[str] (optional)", "limit": "int (default 10)", "fallback_search": "bool (default false)", "confirm_fuzzy": "bool (alias for fallback_search)"},
|
||||
"description": "Exact-first resolver for planes, ships, operators, callsigns, registrations, MMSI/IMO, and named entities. Skips fuzzy search unless fallback_search=true or no exact match.",
|
||||
},
|
||||
"route_query": {
|
||||
"args": {"text": "str", "lat": "float (optional)", "lng": "float (optional)", "radius_km": "float (default 50)", "compact": "bool (default true)"},
|
||||
"description": "Deterministic intent router — returns recommended fast command, alternates, and latency estimate. Preferred entry for natural-language reads.",
|
||||
},
|
||||
"run_playbook": {
|
||||
"args": {"name": "str", "query": "str (optional)", "lat": "float (optional)", "lng": "float (optional)"},
|
||||
"description": "Execute a named batch plan (hot_snapshot, morning_brief, monitor_heartbeat, track_snapshot, area_brief, entity_recon).",
|
||||
},
|
||||
"correlate_entity": {
|
||||
"args": {"query": "str (optional)", "entity_type": "str (optional)", "callsign": "str (optional)", "registration": "str (optional)", "icao24": "str (optional)", "mmsi": "str (optional)", "imo": "str (optional)", "name": "str (optional)", "owner": "str (optional)", "radius_km": "float (default 100)", "limit": "int (default 10)"},
|
||||
@@ -2554,7 +2565,8 @@ async def api_capabilities(request: Request):
|
||||
"track_ship": {"params": {"mmsi": "str (optional)", "imo": "str (optional)", "name": "str (optional)", "owner": "str (optional)", "callsign": "str (optional)"}, "description": "Alert when ship appears by MMSI, IMO, name, owner, or callsign"},
|
||||
"track_entity": {"params": {"query": "str", "entity_type": "str (optional)", "layers": "list[str] (optional)"}, "description": "Generic exact-first entity watch"},
|
||||
"geofence": {"params": {"lat": "float", "lng": "float", "radius_km": "float (default 50)", "entity_types": "list (default ['flights','ships'])"}, "description": "Alert when any entity enters a geographic zone"},
|
||||
"keyword": {"params": {"keyword": "str"}, "description": "Alert when keyword appears in news/GDELT"},
|
||||
"keyword": {"params": {"keyword": "str", "include_telegram": "bool (default true)"}, "description": "Alert when keyword appears in news, GDELT, or Telegram OSINT"},
|
||||
"telegram_rhetoric": {"params": {"min_risk_score": "int 1-10 (default 7)", "keywords": "list[str] or comma string (optional)", "channels": "list[str] or comma string (optional)"}, "description": "Alert on new high-risk Telegram OSINT posts"},
|
||||
"prediction_market": {"params": {"query": "str", "threshold": "float 0-1 (optional)"}, "description": "Alert on prediction market movements"},
|
||||
},
|
||||
},
|
||||
@@ -2578,7 +2590,8 @@ async def api_capabilities(request: Request):
|
||||
"layers are serialized, unchanged layers transfer zero bytes. The client tracks versions "
|
||||
"automatically from SSE events and previous responses. "
|
||||
"3) Pass compact=true on every read command for compressed_v1 responses (~60-90% smaller). "
|
||||
"4) Use targeted commands first (find_flights, search_telemetry, entities_near). "
|
||||
"4) Use route_query / find_entity / run_playbook before search_telemetry. "
|
||||
"Expensive commands require confirm_expensive=true. "
|
||||
"Reserve get_telemetry/get_slow_telemetry for rare full-context pulls.",
|
||||
"pins": "Pins are server-side, NOT localStorage. Use place_pin command or POST /api/ai/pins. The agent can place and delete pins.",
|
||||
"tracking": "To track a specific aircraft without polling: use add_watch with track_callsign or track_registration. Over SSE, you'll get instant push alerts.",
|
||||
@@ -2708,6 +2721,7 @@ def _connect_info_metadata(settings) -> dict:
|
||||
"get_telemetry", "get_pins", "satellite_images",
|
||||
"news_near", "ai_summary", "ai_report",
|
||||
"timemachine_list", "timemachine_view",
|
||||
"infonet_status", "list_gates", "read_gate_messages", "poll_dms",
|
||||
],
|
||||
},
|
||||
"full": {
|
||||
@@ -2718,6 +2732,8 @@ def _connect_info_metadata(settings) -> dict:
|
||||
"satellite_images", "news_near", "data_injection",
|
||||
"ai_summary", "ai_report", "timemachine_snapshot",
|
||||
"timemachine_list", "timemachine_view", "timemachine_diff",
|
||||
"ensure_infonet_ready", "join_infonet_swarm",
|
||||
"post_gate_message", "cast_vote", "send_dm",
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
@@ -0,0 +1,339 @@
|
||||
"""Strategic Risk Analytics API — game-theoretic early warning overlays."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from auth import require_local_operator
|
||||
from limiter import limiter
|
||||
from analytics.backtest import (
|
||||
DEFAULT_BACKTEST_ALERT_THRESHOLD,
|
||||
run_historical_backtest,
|
||||
tune_alert_threshold,
|
||||
)
|
||||
from analytics.feed_adapter import normalize_feed_item
|
||||
from analytics.integration import get_gt_engine, refresh_from_latest_data
|
||||
from analytics.gt_alerts import top_gt_alerts
|
||||
from analytics.micro_rolling import micro_rolling_report
|
||||
from analytics.rolling_backtest import (
|
||||
freeze_weekly_snapshot,
|
||||
label_region,
|
||||
label_regions,
|
||||
rolling_alert_threshold,
|
||||
rolling_report,
|
||||
score_week,
|
||||
)
|
||||
from analytics.weekly_store import load_week
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
from services.fetchers._store import _data_lock, get_latest_data_subset_refs, latest_data
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class RiskHeatmapRequest(BaseModel):
|
||||
"""Optional batch ingest + refresh controls for POST /api/analytics/risk_heatmap."""
|
||||
|
||||
refresh: bool = True
|
||||
items: list[dict[str, Any]] = Field(default_factory=list)
|
||||
|
||||
|
||||
class RollingFreezeRequest(BaseModel):
|
||||
week_id: str | None = None
|
||||
force: bool = False
|
||||
|
||||
|
||||
class RollingLabelEntry(BaseModel):
|
||||
region: str
|
||||
label: str
|
||||
notes: str = ""
|
||||
|
||||
|
||||
class RollingLabelRequest(BaseModel):
|
||||
week_id: str
|
||||
labels: list[RollingLabelEntry] = Field(default_factory=list)
|
||||
|
||||
|
||||
def _empty_heatmap() -> dict[str, Any]:
|
||||
return {
|
||||
"enabled": False,
|
||||
"type": "FeatureCollection",
|
||||
"features": [],
|
||||
"clusters": [],
|
||||
"processed": 0,
|
||||
"timestamp": None,
|
||||
}
|
||||
|
||||
|
||||
def _gt_risk_payload() -> dict[str, Any]:
|
||||
snap = get_latest_data_subset_refs("gt_risk")
|
||||
payload = snap.get("gt_risk")
|
||||
if not isinstance(payload, dict):
|
||||
return _empty_heatmap()
|
||||
heatmap = payload.get("heatmap") or {"type": "FeatureCollection", "features": []}
|
||||
return {
|
||||
"enabled": bool(payload.get("enabled")),
|
||||
"type": heatmap.get("type", "FeatureCollection"),
|
||||
"features": list(heatmap.get("features") or []),
|
||||
"clusters": list(payload.get("clusters") or []),
|
||||
"processed": int(payload.get("processed") or 0),
|
||||
"timestamp": payload.get("timestamp"),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/analytics/risk_heatmap")
|
||||
@limiter.limit("60/minute")
|
||||
async def risk_heatmap_get(request: Request) -> dict[str, Any]:
|
||||
"""Return cached GeoJSON risk overlay (posterior scores per region)."""
|
||||
if not gt_analytics_enabled():
|
||||
return _empty_heatmap()
|
||||
return _gt_risk_payload()
|
||||
|
||||
|
||||
@router.post("/api/analytics/risk_heatmap")
|
||||
@limiter.limit("12/minute")
|
||||
async def risk_heatmap_post(
|
||||
request: Request,
|
||||
body: RiskHeatmapRequest,
|
||||
_: None = Depends(require_local_operator),
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Ingest optional feed items and/or refresh beliefs from latest intel layers.
|
||||
|
||||
Requires local operator auth — intended for OpenClaw agents and admin tooling.
|
||||
"""
|
||||
if not gt_analytics_enabled():
|
||||
raise HTTPException(status_code=503, detail="Strategic Risk Analytics is disabled")
|
||||
|
||||
engine = get_gt_engine()
|
||||
if engine is None:
|
||||
raise HTTPException(status_code=503, detail="Strategic Risk Analytics engine unavailable")
|
||||
|
||||
ingested = 0
|
||||
for raw in body.items:
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
source_type = str(raw.get("source_type") or "manual")
|
||||
item = normalize_feed_item(raw, source_type=source_type)
|
||||
result = engine.process_feed_item(item)
|
||||
if result and not result.get("skipped"):
|
||||
ingested += 1
|
||||
|
||||
summary: dict[str, Any] = {"ingested": ingested}
|
||||
if body.refresh:
|
||||
with _data_lock:
|
||||
snapshot = dict(latest_data)
|
||||
summary.update(refresh_from_latest_data(snapshot, persist=True))
|
||||
|
||||
payload = _gt_risk_payload()
|
||||
payload["ingested"] = ingested
|
||||
payload["refresh"] = bool(body.refresh)
|
||||
return payload
|
||||
|
||||
|
||||
@router.get("/api/analytics/dossier/{region}")
|
||||
@limiter.limit("30/minute")
|
||||
async def analytics_dossier(request: Request, region: str) -> dict[str, Any]:
|
||||
"""Game-theoretic rationale, recent costly signals, and scenario sketches."""
|
||||
region_key = str(region or "").strip().lower()
|
||||
if not region_key or len(region_key) > 120:
|
||||
raise HTTPException(status_code=400, detail="Invalid region identifier")
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"enabled": False,
|
||||
"region": region_key,
|
||||
"current_risk": 0.0,
|
||||
"interpretation": "Strategic Risk Analytics is disabled.",
|
||||
"recent_signals": [],
|
||||
"scenarios": [],
|
||||
}
|
||||
|
||||
engine = get_gt_engine()
|
||||
if engine is None:
|
||||
raise HTTPException(status_code=503, detail="Strategic Risk Analytics engine unavailable")
|
||||
|
||||
dossier = engine.get_dossier(region_key)
|
||||
dossier["enabled"] = True
|
||||
return dossier
|
||||
|
||||
|
||||
@router.get("/api/analytics/backtest")
|
||||
@limiter.limit("6/minute")
|
||||
async def analytics_backtest(
|
||||
request: Request,
|
||||
expanded: bool = True,
|
||||
tune: bool = False,
|
||||
target_confidence: float = 0.95,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Run labeled historical backtest and return accuracy + Wilson 95% CI.
|
||||
|
||||
``confidence_rate`` is the Wilson lower bound (conservative pass metric).
|
||||
"""
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled.",
|
||||
}
|
||||
|
||||
if tune:
|
||||
threshold, report = tune_alert_threshold(target_confidence=target_confidence)
|
||||
else:
|
||||
threshold = DEFAULT_BACKTEST_ALERT_THRESHOLD
|
||||
report = run_historical_backtest(
|
||||
use_expanded_suite=expanded,
|
||||
alert_threshold=threshold,
|
||||
target_confidence=target_confidence,
|
||||
)
|
||||
|
||||
payload = report.to_dict()
|
||||
payload["enabled"] = True
|
||||
payload["expanded_suite"] = expanded
|
||||
payload["tuned"] = tune
|
||||
payload["recommended_alert_threshold"] = threshold
|
||||
return payload
|
||||
|
||||
|
||||
@router.get("/api/analytics/rolling")
|
||||
@limiter.limit("12/minute")
|
||||
async def analytics_rolling(
|
||||
request: Request,
|
||||
weeks: int = 8,
|
||||
target_confidence: float = 0.80,
|
||||
) -> dict[str, Any]:
|
||||
"""Rolling weekly operational validation — accuracy trend with delayed labels."""
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled.",
|
||||
}
|
||||
|
||||
report = rolling_report(weeks=max(1, min(weeks, 52)), target_confidence=target_confidence)
|
||||
report["enabled"] = True
|
||||
return report
|
||||
|
||||
|
||||
@router.get("/api/analytics/alerts")
|
||||
@limiter.limit("30/minute")
|
||||
async def analytics_top_alerts(
|
||||
request: Request,
|
||||
limit: int = 8,
|
||||
) -> dict[str, Any]:
|
||||
"""Top GT risk regions ranked by score — fly-to targets for the map."""
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled.",
|
||||
}
|
||||
|
||||
report = top_gt_alerts(limit=max(1, min(limit, 25)))
|
||||
report["enabled"] = True
|
||||
return report
|
||||
|
||||
|
||||
@router.get("/api/analytics/rolling/micro")
|
||||
@limiter.limit("30/minute")
|
||||
async def analytics_rolling_micro(
|
||||
request: Request,
|
||||
window_days: int = 3,
|
||||
limit: int = 15,
|
||||
) -> dict[str, Any]:
|
||||
"""Rolling 3-day micro average — spot vs baseline, ignition detection."""
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled.",
|
||||
}
|
||||
|
||||
report = micro_rolling_report(
|
||||
window_days=max(2, min(window_days, 7)),
|
||||
limit=max(1, min(limit, 50)),
|
||||
)
|
||||
report["enabled"] = True
|
||||
return report
|
||||
|
||||
|
||||
@router.get("/api/analytics/rolling/{week_id}")
|
||||
@limiter.limit("12/minute")
|
||||
async def analytics_rolling_week(request: Request, week_id: str) -> dict[str, Any]:
|
||||
"""Return a single frozen week snapshot and its score."""
|
||||
if not gt_analytics_enabled():
|
||||
return {"enabled": False, "message": "Strategic Risk Analytics is disabled."}
|
||||
|
||||
snapshot = load_week(str(week_id).strip())
|
||||
if snapshot is None:
|
||||
raise HTTPException(status_code=404, detail=f"Week {week_id} not found")
|
||||
|
||||
score = score_week(snapshot)
|
||||
return {
|
||||
"enabled": True,
|
||||
"week_id": snapshot.week_id,
|
||||
"snapshot": snapshot.to_dict(),
|
||||
"score": score.to_dict(),
|
||||
"alert_threshold": rolling_alert_threshold(),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/api/analytics/rolling/freeze")
|
||||
@limiter.limit("6/minute")
|
||||
async def analytics_rolling_freeze(
|
||||
request: Request,
|
||||
body: RollingFreezeRequest,
|
||||
_: None = Depends(require_local_operator),
|
||||
) -> dict[str, Any]:
|
||||
"""Freeze current GT scores for the ISO week (idempotent unless force=true)."""
|
||||
if not gt_analytics_enabled():
|
||||
raise HTTPException(status_code=503, detail="Strategic Risk Analytics is disabled")
|
||||
|
||||
result = freeze_weekly_snapshot(
|
||||
week_id=body.week_id,
|
||||
force=body.force,
|
||||
frozen_by="api",
|
||||
)
|
||||
if not result.get("ok"):
|
||||
raise HTTPException(status_code=503, detail=result.get("detail", "Freeze failed"))
|
||||
result["enabled"] = True
|
||||
return result
|
||||
|
||||
|
||||
@router.post("/api/analytics/rolling/label")
|
||||
@limiter.limit("12/minute")
|
||||
async def analytics_rolling_label(
|
||||
request: Request,
|
||||
body: RollingLabelRequest,
|
||||
_: None = Depends(require_local_operator),
|
||||
) -> dict[str, Any]:
|
||||
"""Apply delayed outcome labels to a frozen week."""
|
||||
if not gt_analytics_enabled():
|
||||
raise HTTPException(status_code=503, detail="Strategic Risk Analytics is disabled")
|
||||
|
||||
week_id = str(body.week_id or "").strip()
|
||||
if not week_id:
|
||||
raise HTTPException(status_code=400, detail="week_id required")
|
||||
|
||||
if len(body.labels) == 1:
|
||||
entry = body.labels[0]
|
||||
result = label_region(
|
||||
week_id,
|
||||
entry.region,
|
||||
entry.label, # type: ignore[arg-type]
|
||||
notes=entry.notes,
|
||||
labeled_by="api",
|
||||
)
|
||||
else:
|
||||
result = label_regions(
|
||||
week_id,
|
||||
[row.model_dump() for row in body.labels],
|
||||
labeled_by="api",
|
||||
)
|
||||
|
||||
if not result.get("ok"):
|
||||
raise HTTPException(status_code=404, detail=result.get("detail", "Label failed"))
|
||||
result["enabled"] = True
|
||||
return result
|
||||
@@ -46,6 +46,7 @@ _CCTV_PROXY_ALLOWED_HOSTS = {
|
||||
"tripcheck.com",
|
||||
"www.tripcheck.com",
|
||||
"infocar.dgt.es",
|
||||
"etraffic.dgt.es",
|
||||
"informo.madrid.es",
|
||||
"webcams2.asfinag.at",
|
||||
"odo.asfinag.at",
|
||||
@@ -158,10 +159,10 @@ def _cctv_proxy_profile_for_url(target_url: str) -> _CCTVProxyProfile:
|
||||
if host in {"tripcheck.com", "www.tripcheck.com"}:
|
||||
return _CCTVProxyProfile(name="odot-tripcheck", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=30,
|
||||
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8"})
|
||||
if host == "infocar.dgt.es":
|
||||
if host in {"infocar.dgt.es", "etraffic.dgt.es"}:
|
||||
return _CCTVProxyProfile(name="dgt-spain", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 8.0), cache_seconds=60,
|
||||
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
|
||||
"Referer": "https://infocar.dgt.es/"})
|
||||
"Referer": "https://etraffic.dgt.es/"})
|
||||
if host == "informo.madrid.es":
|
||||
return _CCTVProxyProfile(name="madrid-city", timeout=(_CCTV_PROXY_CONNECT_TIMEOUT_S, 12.0), cache_seconds=30,
|
||||
headers={"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
|
||||
|
||||
+57
-15
@@ -396,6 +396,52 @@ async def get_selected_ship_trail(mmsi: int, request: Request): # noqa: ARG001
|
||||
return {"id": mmsi, "trail": get_vessel_trail(mmsi)}
|
||||
|
||||
|
||||
@router.get("/api/aviation/datalink/status")
|
||||
@limiter.limit("60/minute")
|
||||
async def aviation_datalink_status(request: Request): # noqa: ARG001
|
||||
from services.fetchers.airframes import get_datalink_status
|
||||
|
||||
return get_datalink_status()
|
||||
|
||||
|
||||
@router.get("/api/aviation/datalink/messages")
|
||||
@limiter.limit("240/minute")
|
||||
async def aviation_datalink_messages(
|
||||
request: Request, # noqa: ARG001
|
||||
icao24: str = Query("", description="ICAO24 hex for the aircraft"),
|
||||
registration: str = Query("", description="Tail / registration number"),
|
||||
callsign: str = Query("", description="Optional callsign filter"),
|
||||
live: bool = Query(
|
||||
False,
|
||||
description="When true, fetch from Airframes if cache has no messages (slower)",
|
||||
),
|
||||
):
|
||||
from services.fetchers.airframes import lookup_datalink_messages
|
||||
|
||||
return lookup_datalink_messages(
|
||||
icao24=icao24,
|
||||
registration=registration,
|
||||
callsign=callsign,
|
||||
allow_live=live,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/api/sigint/meshtastic/status")
|
||||
@limiter.limit("120/minute")
|
||||
async def meshtastic_map_status(request: Request): # noqa: ARG001
|
||||
from services.fetchers.meshtastic_map import get_meshtastic_map_status
|
||||
|
||||
return get_meshtastic_map_status()
|
||||
|
||||
|
||||
@router.post("/api/sigint/meshtastic/scan", dependencies=[Depends(require_local_operator)])
|
||||
@limiter.limit("3/hour")
|
||||
async def meshtastic_planet_scan(request: Request): # noqa: ARG001
|
||||
from services.fetchers.meshtastic_map import start_meshtastic_planet_scan
|
||||
|
||||
return start_meshtastic_planet_scan()
|
||||
|
||||
|
||||
@router.post("/api/viewport")
|
||||
@limiter.limit("60/minute")
|
||||
async def update_viewport(vp: ViewportUpdate, request: Request): # noqa: ARG001
|
||||
@@ -498,12 +544,13 @@ def _run_prediction_markets_disable() -> None:
|
||||
async def update_layers(update: LayerUpdate, request: Request):
|
||||
"""Receive frontend layer toggle state. Starts/stops streams accordingly."""
|
||||
from services.fetchers._store import active_layers, bump_active_layers_version, is_any_active
|
||||
from services.layer_enable_refresh import refresh_newly_enabled_layers, snapshot_active_layers
|
||||
|
||||
layers_before = snapshot_active_layers()
|
||||
old_ships = is_any_active("ships_military", "ships_cargo", "ships_civilian", "ships_passenger", "ships_tracked_yachts")
|
||||
old_mesh = is_any_active("sigint_meshtastic")
|
||||
old_aprs = is_any_active("sigint_aprs")
|
||||
old_viirs = is_any_active("viirs_nightlights")
|
||||
old_datacenters = is_any_active("datacenters")
|
||||
old_fishing = is_any_active("fishing_activity")
|
||||
changed = False
|
||||
for key, value in update.layers.items():
|
||||
if key in active_layers:
|
||||
@@ -516,8 +563,6 @@ async def update_layers(update: LayerUpdate, request: Request):
|
||||
new_mesh = is_any_active("sigint_meshtastic")
|
||||
new_aprs = is_any_active("sigint_aprs")
|
||||
new_viirs = is_any_active("viirs_nightlights")
|
||||
new_datacenters = is_any_active("datacenters")
|
||||
new_fishing = is_any_active("fishing_activity")
|
||||
if old_ships and not new_ships:
|
||||
from services.ais_stream import stop_ais_stream
|
||||
stop_ais_stream()
|
||||
@@ -561,16 +606,7 @@ async def update_layers(update: LayerUpdate, request: Request):
|
||||
if not old_viirs and new_viirs:
|
||||
_queue_viirs_change_refresh()
|
||||
logger.info("VIIRS change refresh queued (layer enabled)")
|
||||
if not old_datacenters and new_datacenters:
|
||||
from services.fetchers.infrastructure import fetch_datacenters
|
||||
|
||||
fetch_datacenters()
|
||||
logger.info("Datacenters loaded (layer enabled)")
|
||||
if not old_fishing and new_fishing:
|
||||
from services.fetchers.geo import fetch_fishing_activity
|
||||
|
||||
fetch_fishing_activity()
|
||||
logger.info("Fishing activity refresh queued (layer enabled)")
|
||||
refresh_newly_enabled_layers(layers_before)
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@@ -773,7 +809,7 @@ async def live_data_slow(
|
||||
"scanners", "weather_alerts", "ukraine_alerts", "air_quality", "volcanoes",
|
||||
"fishing_activity", "psk_reporter", "correlations", "uap_sightings", "wastewater",
|
||||
"crowdthreat", "threat_level", "trending_markets", "road_corridor_trends",
|
||||
"malware_threats", "cyber_threats", "scm_suppliers", "telegram_osint",
|
||||
"malware_threats", "cyber_threats", "scm_suppliers", "telegram_osint", "gt_risk",
|
||||
)
|
||||
freshness = get_source_timestamps_snapshot()
|
||||
payload = {
|
||||
@@ -839,6 +875,12 @@ async def live_data_slow(
|
||||
)
|
||||
if active_layers.get("telegram_osint", True)
|
||||
else {"posts": [], "total": 0, "geolocated": 0},
|
||||
"gt_risk": (
|
||||
d.get("gt_risk")
|
||||
or {"enabled": False, "heatmap": {"type": "FeatureCollection", "features": []}, "clusters": []}
|
||||
)
|
||||
if active_layers.get("gt_risk", False)
|
||||
else {"enabled": False, "heatmap": {"type": "FeatureCollection", "features": []}, "clusters": []},
|
||||
"freshness": freshness,
|
||||
}
|
||||
# Issue #288: bbox filter heavy/dense layers only when all four bounds
|
||||
|
||||
+71
-15
@@ -4,10 +4,50 @@ from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel
|
||||
from limiter import limiter
|
||||
from auth import require_admin
|
||||
from services.data_fetcher import get_latest_data
|
||||
from services.schemas import HealthResponse
|
||||
import os
|
||||
|
||||
# Health/SLO probes only need counts + freshness — not a full dashboard deepcopy.
|
||||
_HEALTH_DATA_KEYS: tuple[str, ...] = (
|
||||
"last_updated",
|
||||
"commercial_flights",
|
||||
"military_flights",
|
||||
"private_jets",
|
||||
"ships",
|
||||
"satellites",
|
||||
"earthquakes",
|
||||
"cctv",
|
||||
"news",
|
||||
"uavs",
|
||||
"firms_fires",
|
||||
"liveuamap",
|
||||
"gdelt",
|
||||
"uap_sightings",
|
||||
"wastewater",
|
||||
"fimi",
|
||||
"space_weather",
|
||||
"weather_alerts",
|
||||
"volcanoes",
|
||||
"prediction_markets",
|
||||
)
|
||||
|
||||
|
||||
def _health_data_snapshot() -> dict:
|
||||
from services.fetchers._store import get_latest_data_subset_refs
|
||||
from services.slo import SLO_REGISTRY
|
||||
|
||||
keys = tuple(dict.fromkeys((*_HEALTH_DATA_KEYS, *SLO_REGISTRY.keys())))
|
||||
return get_latest_data_subset_refs(*keys)
|
||||
|
||||
|
||||
def _health_row_count(value) -> int:
|
||||
if value is None:
|
||||
return 0
|
||||
try:
|
||||
return len(value)
|
||||
except TypeError:
|
||||
return 0
|
||||
|
||||
APP_VERSION = os.environ.get("_HEALTH_APP_VERSION", "0.9.82")
|
||||
|
||||
router = APIRouter()
|
||||
@@ -41,7 +81,7 @@ async def health_check(request: Request):
|
||||
from services.fetchers._store import get_source_timestamps_snapshot
|
||||
from services.slo import compute_all_statuses, summarise_statuses
|
||||
|
||||
d = get_latest_data()
|
||||
d = _health_data_snapshot()
|
||||
last = d.get("last_updated")
|
||||
timestamps = get_source_timestamps_snapshot()
|
||||
slo_statuses = compute_all_statuses(d, timestamps)
|
||||
@@ -85,33 +125,49 @@ async def health_check(request: Request):
|
||||
):
|
||||
top_status = "degraded"
|
||||
|
||||
runtime: dict = {}
|
||||
try:
|
||||
from services.runtime_profile import get_runtime_profile
|
||||
from analytics.settings import gt_analytics_status
|
||||
|
||||
runtime = {
|
||||
**get_runtime_profile(),
|
||||
"gt_analytics": gt_analytics_status(),
|
||||
}
|
||||
except Exception:
|
||||
runtime = {}
|
||||
|
||||
return {
|
||||
"status": top_status,
|
||||
"version": _get_app_version(),
|
||||
"last_updated": last,
|
||||
"sources": {
|
||||
"flights": len(d.get("commercial_flights", [])),
|
||||
"military": len(d.get("military_flights", [])),
|
||||
"ships": len(d.get("ships", [])),
|
||||
"satellites": len(d.get("satellites", [])),
|
||||
"earthquakes": len(d.get("earthquakes", [])),
|
||||
"cctv": len(d.get("cctv", [])),
|
||||
"news": len(d.get("news", [])),
|
||||
"uavs": len(d.get("uavs", [])),
|
||||
"firms_fires": len(d.get("firms_fires", [])),
|
||||
"liveuamap": len(d.get("liveuamap", [])),
|
||||
"gdelt": len(d.get("gdelt", [])),
|
||||
"uap_sightings": len(d.get("uap_sightings", [])),
|
||||
"flights": _health_row_count(d.get("commercial_flights")),
|
||||
"military": _health_row_count(d.get("military_flights")),
|
||||
"ships": _health_row_count(d.get("ships")),
|
||||
"satellites": _health_row_count(d.get("satellites")),
|
||||
"earthquakes": _health_row_count(d.get("earthquakes")),
|
||||
"cctv": _health_row_count(d.get("cctv")),
|
||||
"news": _health_row_count(d.get("news")),
|
||||
"uavs": _health_row_count(d.get("uavs")),
|
||||
"firms_fires": _health_row_count(d.get("firms_fires")),
|
||||
"liveuamap": _health_row_count(d.get("liveuamap")),
|
||||
"gdelt": _health_row_count(d.get("gdelt")),
|
||||
"uap_sightings": _health_row_count(d.get("uap_sightings")),
|
||||
},
|
||||
"freshness": timestamps,
|
||||
"uptime_seconds": round(_time_mod.time() - _get_start_time()),
|
||||
"slo": slo_statuses,
|
||||
"slo_summary": slo_summary,
|
||||
"ais_proxy": ais_status,
|
||||
"runtime": runtime or None,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/debug-latest", dependencies=[Depends(require_admin)])
|
||||
@limiter.limit("30/minute")
|
||||
async def debug_latest_data(request: Request):
|
||||
return list(get_latest_data().keys())
|
||||
from services.fetchers._store import latest_data, _data_lock
|
||||
|
||||
with _data_lock:
|
||||
return list(latest_data.keys())
|
||||
|
||||
@@ -14,6 +14,7 @@ from services.fetchers._store import get_latest_data_subset_refs
|
||||
from services.fetchers.telegram_osint import telegram_media_host_allowed
|
||||
from services.intel_feeds.country_risk import build_country_risk_payload
|
||||
from services.network_utils import outbound_user_agent
|
||||
from services.telegram_translate import apply_posts_translations, normalize_translate_target
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -45,12 +46,19 @@ async def country_risk(request: Request) -> dict:
|
||||
|
||||
@router.get("/api/telegram-feed")
|
||||
@limiter.limit("30/minute")
|
||||
async def telegram_feed(request: Request) -> dict:
|
||||
async def telegram_feed(request: Request, lang: str | None = Query(default=None)) -> dict:
|
||||
snap = get_latest_data_subset_refs("telegram_osint")
|
||||
payload = snap.get("telegram_osint")
|
||||
if isinstance(payload, dict) and payload.get("posts") is not None:
|
||||
return payload
|
||||
return {"posts": [], "total": 0, "geolocated": 0, "timestamp": None}
|
||||
if not isinstance(payload, dict) or payload.get("posts") is None:
|
||||
return {"posts": [], "total": 0, "geolocated": 0, "timestamp": None}
|
||||
|
||||
if lang:
|
||||
target = normalize_translate_target(lang)
|
||||
localized = dict(payload)
|
||||
localized["posts"] = apply_posts_translations(list(payload.get("posts") or []), target)
|
||||
localized["translate_locale"] = target
|
||||
return localized
|
||||
return payload
|
||||
|
||||
|
||||
def _infer_telegram_media_type(target_url: str, content_type: str) -> str:
|
||||
|
||||
@@ -65,6 +65,10 @@ def _hydrate_dm_relay_from_chain(events: list) -> int:
|
||||
@limiter.limit("30/minute")
|
||||
async def infonet_peer_push(request: Request):
|
||||
"""Accept pushed Infonet events from relay peers (HMAC-authenticated)."""
|
||||
from services.mesh.mesh_fleet_defaults import infonet_fleet_join_enabled
|
||||
|
||||
if not infonet_fleet_join_enabled():
|
||||
return {"ok": True, "accepted": 0, "duplicates": 0, "rejected": [], "skipped": "fleet_join_disabled"}
|
||||
content_length = request.headers.get("content-length")
|
||||
if content_length:
|
||||
try:
|
||||
@@ -154,6 +158,10 @@ async def dm_replicate_envelope(request: Request):
|
||||
@limiter.limit("30/minute")
|
||||
async def gate_peer_push(request: Request):
|
||||
"""Accept pushed gate events from relay peers (private plane)."""
|
||||
from services.mesh.mesh_fleet_defaults import infonet_fleet_join_enabled
|
||||
|
||||
if not infonet_fleet_join_enabled():
|
||||
return {"ok": True, "accepted": 0, "duplicates": 0, "skipped": "fleet_join_disabled"}
|
||||
content_length = request.headers.get("content-length")
|
||||
if content_length:
|
||||
try:
|
||||
|
||||
@@ -308,6 +308,10 @@ class WormholeDmDecryptRequest(BaseModel):
|
||||
session_welcome: str | None = None
|
||||
|
||||
|
||||
class WormholeDmMlsKeyPackageRequest(BaseModel):
|
||||
alias: str
|
||||
|
||||
|
||||
class WormholeDmResetRequest(BaseModel):
|
||||
peer_id: str | None = None
|
||||
|
||||
@@ -326,6 +330,14 @@ class WormholeDmBootstrapDecryptRequest(BaseModel):
|
||||
ciphertext: str
|
||||
|
||||
|
||||
class WormholeDmConnectContactRequest(BaseModel):
|
||||
lookup_token: str = ""
|
||||
peer_id: str = ""
|
||||
note: str = ""
|
||||
lookup_peer_url: str = ""
|
||||
cached_prekey_bundle: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class WormholeDmInviteImportRequest(BaseModel):
|
||||
invite: dict[str, Any]
|
||||
alias: str = ""
|
||||
@@ -1085,7 +1097,21 @@ async def api_wormhole_dm_bootstrap_decrypt(request: Request, body: WormholeDmBo
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/wormhole/dm/sender-token", dependencies=[Depends(require_admin)])
|
||||
@router.post("/api/wormhole/dm/connect-contact", dependencies=[Depends(require_local_operator)])
|
||||
@limiter.limit("30/minute")
|
||||
async def api_wormhole_dm_connect_contact(request: Request, body: WormholeDmConnectContactRequest):
|
||||
from services.openclaw_infonet import send_contact_request
|
||||
|
||||
return send_contact_request(
|
||||
lookup_token=str(body.lookup_token or ""),
|
||||
peer_id=str(body.peer_id or ""),
|
||||
note=str(body.note or ""),
|
||||
lookup_peer_url=str(body.lookup_peer_url or ""),
|
||||
cached_prekey_bundle=dict(body.cached_prekey_bundle or {}) if body.cached_prekey_bundle else None,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/wormhole/dm/sender-token", dependencies=[Depends(require_local_operator)])
|
||||
@limiter.limit("60/minute")
|
||||
async def api_wormhole_dm_sender_token(request: Request, body: WormholeDmSenderTokenRequest):
|
||||
if _safe_int(body.count or 1, 1) > 1:
|
||||
@@ -1228,6 +1254,23 @@ async def api_wormhole_dm_decrypt(request: Request, body: WormholeDmDecryptReque
|
||||
)
|
||||
|
||||
|
||||
@router.post("/api/wormhole/dm/mls-key-package", dependencies=[Depends(require_admin)])
|
||||
@limiter.limit("60/minute")
|
||||
async def api_wormhole_dm_mls_key_package(request: Request, body: WormholeDmMlsKeyPackageRequest):
|
||||
from services.mesh.mesh_dm_mls import export_dm_key_package_for_alias
|
||||
|
||||
return export_dm_key_package_for_alias(str(body.alias or "").strip())
|
||||
|
||||
|
||||
@router.post("/api/wormhole/dm/mls-reset", dependencies=[Depends(require_admin)])
|
||||
@limiter.limit("30/minute")
|
||||
async def api_wormhole_dm_mls_reset(request: Request):
|
||||
from services.mesh.mesh_dm_mls import reset_dm_mls_state
|
||||
|
||||
reset_dm_mls_state(clear_privacy_core=True, clear_persistence=True)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@router.post("/api/wormhole/dm/reset", dependencies=[Depends(require_admin)])
|
||||
@limiter.limit("30/minute")
|
||||
async def api_wormhole_dm_reset(request: Request, body: WormholeDmResetRequest):
|
||||
@@ -1287,7 +1330,25 @@ async def api_wormhole_dm_contact_delete(request: Request, peer_id: str):
|
||||
return {"ok": True, "peer_id": peer_id, "deleted": deleted}
|
||||
|
||||
|
||||
_WORMHOLE_PUBLIC_FIELDS = {"installed", "configured", "running", "ready"}
|
||||
@router.post("/api/wormhole/dm/contact/{peer_id}/sever", dependencies=[Depends(require_admin)])
|
||||
@limiter.limit("60/minute")
|
||||
async def api_wormhole_dm_contact_sever(request: Request, peer_id: str):
|
||||
from services.mesh.mesh_wormhole_contacts import sever_wormhole_dm_contact
|
||||
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
body = {}
|
||||
if not isinstance(body, dict):
|
||||
body = {}
|
||||
block = bool(body.get("block", False))
|
||||
try:
|
||||
return sever_wormhole_dm_contact(peer_id, block=block)
|
||||
except ValueError as exc:
|
||||
return {"ok": False, "detail": str(exc)}
|
||||
|
||||
|
||||
_WORMHOLE_PUBLIC_FIELDS = {"installed", "configured", "running", "ready", "arti_ready"}
|
||||
|
||||
|
||||
def _redact_wormhole_status(state: dict[str, Any], authenticated: bool) -> dict[str, Any]:
|
||||
@@ -1308,6 +1369,25 @@ async def api_wormhole_status(request: Request):
|
||||
return await _m.api_wormhole_status(request)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/api/wormhole/private-delivery/{item_id}",
|
||||
dependencies=[Depends(require_local_operator)],
|
||||
)
|
||||
@limiter.limit("120/minute")
|
||||
async def api_wormhole_private_delivery_item(request: Request, item_id: str):
|
||||
from services.mesh.mesh_metadata_exposure import metadata_exposure_for_request
|
||||
from services.mesh.mesh_private_outbox import private_delivery_outbox
|
||||
|
||||
exposure = metadata_exposure_for_request(
|
||||
request,
|
||||
authenticated=True,
|
||||
)
|
||||
item = private_delivery_outbox.get_item(item_id, exposure=exposure)
|
||||
if item is None:
|
||||
raise HTTPException(status_code=404, detail="private_delivery_item_not_found")
|
||||
return {"ok": True, "item": item}
|
||||
|
||||
|
||||
@router.post("/api/wormhole/private-delivery/{item_id}/action", dependencies=[Depends(require_local_operator)])
|
||||
@limiter.limit("30/minute")
|
||||
async def api_wormhole_private_delivery_action(
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Operator settings for the embedded agent shell (working directory)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SETTINGS_FILE = Path(__file__).resolve().parent.parent / "data" / "agent_shell_settings.json"
|
||||
_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _default_working_directory() -> str:
|
||||
explicit = str(os.environ.get("AGENT_SHELL_DEFAULT_CWD") or "").strip()
|
||||
if explicit and os.path.isdir(explicit):
|
||||
return explicit
|
||||
home = str(os.environ.get("HOME") or "").strip()
|
||||
if home and home != "/nonexistent" and os.path.isdir(home):
|
||||
return home
|
||||
return "/app"
|
||||
|
||||
|
||||
def get_agent_shell_settings() -> dict[str, Any]:
|
||||
with _LOCK:
|
||||
if not _SETTINGS_FILE.exists():
|
||||
return {"working_directory": _default_working_directory()}
|
||||
try:
|
||||
payload = json.loads(_SETTINGS_FILE.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
logger.warning("agent_shell_settings_unreadable")
|
||||
return {"working_directory": _default_working_directory()}
|
||||
cwd = str(payload.get("working_directory") or "").strip() or _default_working_directory()
|
||||
return {"working_directory": cwd}
|
||||
|
||||
|
||||
def set_agent_shell_working_directory(path: str) -> dict[str, Any]:
|
||||
normalized = str(path or "").strip()
|
||||
if not normalized:
|
||||
raise ValueError("working_directory_required")
|
||||
resolved = os.path.abspath(os.path.expanduser(normalized))
|
||||
if not os.path.isdir(resolved):
|
||||
raise ValueError("working_directory_not_found")
|
||||
with _LOCK:
|
||||
_SETTINGS_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
_SETTINGS_FILE.write_text(
|
||||
json.dumps({"working_directory": resolved}, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return {"working_directory": resolved}
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Short-lived, single-use WebSocket bootstrap tokens for the agent shell."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
import time
|
||||
from threading import Lock
|
||||
|
||||
_TOKEN_TTL_SECONDS = 60.0
|
||||
_MAX_ACTIVE_TOKENS = 256
|
||||
|
||||
_store: dict[str, float] = {}
|
||||
_lock = Lock()
|
||||
|
||||
|
||||
def _purge_expired(*, force: bool = False) -> None:
|
||||
now = time.time()
|
||||
with _lock:
|
||||
expired = [token for token, expires in _store.items() if expires <= now]
|
||||
for token in expired:
|
||||
_store.pop(token, None)
|
||||
if force and len(_store) > _MAX_ACTIVE_TOKENS:
|
||||
for token in list(_store.keys())[: len(_store) - _MAX_ACTIVE_TOKENS]:
|
||||
_store.pop(token, None)
|
||||
|
||||
|
||||
def mint_agent_shell_ws_token() -> tuple[str, int]:
|
||||
"""Return (token, expires_in_seconds)."""
|
||||
_purge_expired()
|
||||
token = secrets.token_urlsafe(32)
|
||||
expires_at = time.time() + _TOKEN_TTL_SECONDS
|
||||
with _lock:
|
||||
if len(_store) >= _MAX_ACTIVE_TOKENS:
|
||||
_purge_expired(force=True)
|
||||
_store[token] = expires_at
|
||||
return token, int(_TOKEN_TTL_SECONDS)
|
||||
|
||||
|
||||
def consume_agent_shell_ws_token(token: str) -> bool:
|
||||
"""Validate and burn a one-time token. Returns True when accepted."""
|
||||
cleaned = str(token or "").strip()
|
||||
if not cleaned:
|
||||
return False
|
||||
now = time.time()
|
||||
with _lock:
|
||||
expires_at = _store.pop(cleaned, None)
|
||||
return expires_at is not None and expires_at > now
|
||||
|
||||
|
||||
def reset_agent_shell_ws_tokens_for_tests() -> None:
|
||||
with _lock:
|
||||
_store.clear()
|
||||
@@ -6,6 +6,7 @@ Keys are stored in the backend .env file and loaded via python-dotenv.
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
# Path to the backend .env file
|
||||
@@ -78,6 +79,24 @@ API_REGISTRY = [
|
||||
"url": "https://earthquake.usgs.gov/",
|
||||
"required": False,
|
||||
},
|
||||
{
|
||||
"id": "firms_map_key",
|
||||
"env_key": "FIRMS_MAP_KEY",
|
||||
"name": "NASA FIRMS — MAP Key (optional)",
|
||||
"description": "Optional NASA Earthdata MAP key for country-scoped VIIRS fire enrichment. Global VIIRS hotspots work without a key; set this only if you want per-country FIRMS detail. Free from NASA Earthdata.",
|
||||
"category": "Geophysical",
|
||||
"url": "https://firms.modaps.eosdis.nasa.gov/api/area/",
|
||||
"required": False,
|
||||
},
|
||||
{
|
||||
"id": "airframes_api_key",
|
||||
"env_key": "AIRFRAMES_API_KEY",
|
||||
"name": "Airframes.io — API Key",
|
||||
"description": "ACARS/VDL datalink for plane dossiers. ShadowBroker bulk-ingests the global Airframes firehose (up to 100 messages per API call, one call every 2s, refill every 15 minutes) and indexes by tail/ICAO. Opening a dossier with no cache queues a single-plane lookup. Get a key at app.airframes.io → Dashboard → API Key.",
|
||||
"category": "Aviation",
|
||||
"url": "https://app.airframes.io/user/dashboard",
|
||||
"required": False,
|
||||
},
|
||||
{
|
||||
"id": "celestrak",
|
||||
"env_key": None,
|
||||
@@ -357,6 +376,17 @@ def save_api_keys(updates: dict[str, str]) -> dict:
|
||||
flights.opensky_client.expires_at = 0
|
||||
except Exception:
|
||||
pass
|
||||
if "AIRFRAMES_API_KEY" in clean:
|
||||
try:
|
||||
from services.fetchers.airframes import sync_airframes_messages
|
||||
|
||||
threading.Thread(
|
||||
target=lambda: sync_airframes_messages(force=True),
|
||||
daemon=True,
|
||||
name="airframes-initial-sync",
|
||||
).start()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
@@ -1031,7 +1031,8 @@ out body;
|
||||
# ---------------------------------------------------------------------------
|
||||
# DGT Spain — National Road Cameras
|
||||
# ---------------------------------------------------------------------------
|
||||
# Image URL pattern confirmed working: infocar.dgt.es/etraffic/data/camaras/{id}.jpg
|
||||
# Image URL pattern confirmed working: etraffic.dgt.es/camarasEtraffic/{id}.jpg
|
||||
# (DGT migrated 2026: old infocar.dgt.es/etraffic/data/camaras path now 302->etrafficWEB)
|
||||
# Source: DGT (Dirección General de Tráfico) — public open data (Ley 37/2007).
|
||||
# Author credit: Alborz Nazari (github.com/AlborzNazari) — PR #91
|
||||
|
||||
@@ -1065,10 +1066,10 @@ class DGTNationalIngestor(BaseCCTVIngestor):
|
||||
cameras = []
|
||||
probe_headers = {
|
||||
"Accept": "image/avif,image/webp,image/apng,image/*,*/*;q=0.8",
|
||||
"Referer": "https://infocar.dgt.es/",
|
||||
"Referer": "https://etraffic.dgt.es/",
|
||||
}
|
||||
for cam_id, lat, lon, description in self.KNOWN_CAMERAS:
|
||||
media_url = f"https://infocar.dgt.es/etraffic/data/camaras/{cam_id}.jpg"
|
||||
media_url = f"https://etraffic.dgt.es/camarasEtraffic/{cam_id}.jpg"
|
||||
if not _media_url_reachable(media_url, timeout=6, headers=probe_headers):
|
||||
continue
|
||||
cameras.append({
|
||||
|
||||
@@ -30,6 +30,10 @@ class Settings(BaseSettings):
|
||||
MESH_MQTT_INCLUDE_DEFAULT_ROOTS: bool = True
|
||||
MESH_RNS_ENABLED: bool = False
|
||||
MESH_ARTI_ENABLED: bool = False
|
||||
# When true, trust wormhole_status.json ready bit if the child process is
|
||||
# alive — avoids transport-tier flapping when /api/health probes time out
|
||||
# under Tor load (common during live DM E2E).
|
||||
MESH_WORMHOLE_TRUST_FILE_READY: bool = False
|
||||
MESH_ARTI_SOCKS_PORT: int = 9050
|
||||
MESH_RELAY_PEERS: str = ""
|
||||
MESH_PUBLIC_PEER_URL: str = ""
|
||||
@@ -43,7 +47,24 @@ class Settings(BaseSettings):
|
||||
MESH_INFONET_ALLOW_CLEARNET_SYNC: bool = False
|
||||
MESH_BOOTSTRAP_DISABLED: bool = False
|
||||
MESH_BOOTSTRAP_MANIFEST_PATH: str = "data/bootstrap_peers.json"
|
||||
MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY: str = ""
|
||||
# Public sb-testnet-0 fleet signer (participants). Seed operator holds the private key.
|
||||
MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY: str = (
|
||||
"ul1d0kj/ODPIp0OhHzX8eLAVXzJ3CVvzW1vn2IC6q3I="
|
||||
)
|
||||
MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY: str = ""
|
||||
# When true, empty MESH_PEER_PUSH_SECRET uses the public fleet HMAC for seed join/announce.
|
||||
MESH_INFONET_FLEET_JOIN: bool = True
|
||||
MESH_INFONET_FLEET_JOIN_DISABLED: bool = False
|
||||
# Headless relay/seed compose: auto-enable Tor wormhole on startup so
|
||||
# docker compose redeploys keep the fleet onion reachable.
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE: bool = False
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED: bool = False
|
||||
MESH_BOOTSTRAP_SIGNER_ID: str = ""
|
||||
MESH_PEER_REGISTRY_ENABLED: bool = False
|
||||
MESH_PEER_REGISTRY_DISABLED: bool = False
|
||||
MESH_PEER_REGISTRY_STALE_S: int = 604800
|
||||
MESH_SWARM_MANIFEST_TTL_S: int = 14400
|
||||
MESH_SWARM_MANIFEST_PULL_INTERVAL_S: int = 300
|
||||
MESH_NODE_MODE: str = "participant"
|
||||
MESH_SYNC_INTERVAL_S: int = 300
|
||||
MESH_SYNC_FAILURE_BACKOFF_S: int = 60
|
||||
|
||||
@@ -77,6 +77,7 @@ from services.fetchers.infrastructure import ( # noqa: F401
|
||||
fetch_psk_reporter,
|
||||
)
|
||||
from services.fetchers.road_corridor_sat import fetch_road_corridor_trends # noqa: F401
|
||||
from services.fetchers.airframes import sync_airframes_messages # noqa: F401
|
||||
from services.fetchers.geo import ( # noqa: F401
|
||||
fetch_ships,
|
||||
fetch_airports,
|
||||
@@ -479,26 +480,35 @@ def update_slow_data():
|
||||
fetch_military_bases,
|
||||
fetch_scanners,
|
||||
fetch_psk_reporter,
|
||||
fetch_weather_alerts,
|
||||
# weather_alerts + ukraine_alerts: owned by dedicated scheduler jobs
|
||||
# (5 min and 2 min) — keep off slow tier to avoid duplicate upstream work.
|
||||
fetch_air_quality,
|
||||
fetch_fishing_activity,
|
||||
fetch_power_plants,
|
||||
fetch_ukraine_air_raid_alerts,
|
||||
fetch_malware_threats,
|
||||
fetch_cyber_threats,
|
||||
fetch_scm_suppliers,
|
||||
]
|
||||
_run_tasks("slow-tier", slow_funcs)
|
||||
# Run correlation engine after all data is fresh
|
||||
# Run correlation engine after all data is fresh (skip when overlay is off).
|
||||
try:
|
||||
from services.fetchers._store import is_any_active
|
||||
from services.correlation_engine import compute_correlations
|
||||
with _data_lock:
|
||||
snapshot = dict(latest_data)
|
||||
correlations = compute_correlations(snapshot)
|
||||
with _data_lock:
|
||||
latest_data["correlations"] = correlations
|
||||
|
||||
if is_any_active("correlations"):
|
||||
with _data_lock:
|
||||
snapshot = dict(latest_data)
|
||||
correlations = compute_correlations(snapshot)
|
||||
with _data_lock:
|
||||
latest_data["correlations"] = correlations
|
||||
except Exception as e:
|
||||
logger.error("Correlation engine failed: %s", e)
|
||||
try:
|
||||
from analytics.integration import maybe_refresh_gt_analytics
|
||||
|
||||
maybe_refresh_gt_analytics()
|
||||
except Exception as e:
|
||||
logger.error("GT analytics refresh failed: %s", e)
|
||||
from services.fetchers._store import bump_data_version
|
||||
bump_data_version()
|
||||
_save_intel_startup_cache()
|
||||
@@ -807,8 +817,18 @@ def start_scheduler():
|
||||
|
||||
# Telegram OSINT — hourly t.me/s channel scrape (kept off the 5-minute slow tier).
|
||||
_telegram_interval_m = max(15, int(os.environ.get("TELEGRAM_OSINT_INTERVAL_MINUTES", "60")))
|
||||
|
||||
def _fetch_telegram_osint_with_gt():
|
||||
fetch_telegram_osint()
|
||||
try:
|
||||
from analytics.integration import maybe_refresh_gt_analytics
|
||||
|
||||
maybe_refresh_gt_analytics()
|
||||
except Exception as exc:
|
||||
logger.error("GT analytics refresh after telegram failed: %s", exc)
|
||||
|
||||
_scheduler.add_job(
|
||||
lambda: _run_task_with_health(fetch_telegram_osint, "fetch_telegram_osint"),
|
||||
lambda: _run_task_with_health(_fetch_telegram_osint_with_gt, "fetch_telegram_osint"),
|
||||
"interval",
|
||||
minutes=_telegram_interval_m,
|
||||
next_run_time=datetime.utcnow() + timedelta(seconds=45),
|
||||
@@ -934,14 +954,67 @@ def start_scheduler():
|
||||
)
|
||||
|
||||
# GDELT — every 30 minutes (downloads 32 ZIP files per call, avoid rate limits)
|
||||
def _fetch_gdelt_with_gt():
|
||||
fetch_gdelt()
|
||||
try:
|
||||
from analytics.integration import maybe_refresh_gt_analytics
|
||||
|
||||
maybe_refresh_gt_analytics()
|
||||
except Exception as exc:
|
||||
logger.error("GT analytics refresh after gdelt failed: %s", exc)
|
||||
|
||||
_scheduler.add_job(
|
||||
lambda: _run_task_with_health_on_executor(_SLOW_EXECUTOR, fetch_gdelt, "fetch_gdelt"),
|
||||
lambda: _run_task_with_health_on_executor(_SLOW_EXECUTOR, _fetch_gdelt_with_gt, "fetch_gdelt"),
|
||||
"interval",
|
||||
minutes=30,
|
||||
id="gdelt",
|
||||
max_instances=1,
|
||||
misfire_grace_time=120,
|
||||
)
|
||||
|
||||
# GT analytics — Louvain herding/coordination clusters (feature-flagged).
|
||||
def _recompute_gt_clusters():
|
||||
try:
|
||||
from analytics.integration import recompute_gt_herding_clusters
|
||||
|
||||
recompute_gt_herding_clusters()
|
||||
except Exception as exc:
|
||||
logger.error("GT Louvain recompute failed: %s", exc)
|
||||
|
||||
def _freeze_gt_weekly_snapshot():
|
||||
try:
|
||||
from analytics.integration import maybe_freeze_gt_weekly_snapshot
|
||||
|
||||
maybe_freeze_gt_weekly_snapshot()
|
||||
except Exception as exc:
|
||||
logger.error("GT rolling weekly freeze failed: %s", exc)
|
||||
|
||||
try:
|
||||
from analytics.settings import get_gt_settings, gt_engine_operational
|
||||
|
||||
_gt_settings = get_gt_settings()
|
||||
if gt_engine_operational():
|
||||
_scheduler.add_job(
|
||||
_recompute_gt_clusters,
|
||||
"interval",
|
||||
minutes=_gt_settings.louvain_interval_minutes,
|
||||
id="gt_analytics_louvain",
|
||||
max_instances=1,
|
||||
misfire_grace_time=300,
|
||||
next_run_time=datetime.utcnow() + timedelta(minutes=3),
|
||||
)
|
||||
_scheduler.add_job(
|
||||
_freeze_gt_weekly_snapshot,
|
||||
"cron",
|
||||
day_of_week="mon",
|
||||
hour=0,
|
||||
minute=5,
|
||||
id="gt_rolling_weekly_freeze",
|
||||
max_instances=1,
|
||||
misfire_grace_time=3600,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("GT Louvain scheduler not registered: %s", exc)
|
||||
_scheduler.add_job(
|
||||
lambda: _run_task_with_health_on_executor(
|
||||
_SLOW_EXECUTOR, update_liveuamap, "update_liveuamap"
|
||||
@@ -1191,6 +1264,17 @@ def start_scheduler():
|
||||
next_run_time=datetime.utcnow() + timedelta(minutes=5), # first snapshot 5m after startup
|
||||
)
|
||||
|
||||
_airframes_interval_m = max(5, int(os.environ.get("AIRFRAMES_SYNC_INTERVAL_MINUTES", "15")))
|
||||
_scheduler.add_job(
|
||||
lambda: _run_task_with_health(sync_airframes_messages, "sync_airframes_messages"),
|
||||
"interval",
|
||||
minutes=_airframes_interval_m,
|
||||
id="airframes_datalink",
|
||||
max_instances=1,
|
||||
misfire_grace_time=120,
|
||||
next_run_time=datetime.utcnow() + timedelta(seconds=90),
|
||||
)
|
||||
|
||||
_scheduler.start()
|
||||
logger.info("Scheduler started.")
|
||||
|
||||
|
||||
@@ -74,6 +74,7 @@ class DashboardData(TypedDict, total=False):
|
||||
cyber_threats: Dict[str, Any]
|
||||
scm_suppliers: Dict[str, Any]
|
||||
telegram_osint: Dict[str, Any]
|
||||
gt_risk: Dict[str, Any]
|
||||
|
||||
|
||||
# In-memory store
|
||||
@@ -129,6 +130,13 @@ latest_data: DashboardData = {
|
||||
"cyber_threats": {"threats": [], "stats": {}},
|
||||
"scm_suppliers": {"suppliers": [], "total": 0, "critical_count": 0},
|
||||
"telegram_osint": {"posts": [], "total": 0, "geolocated": 0, "timestamp": None},
|
||||
"gt_risk": {
|
||||
"enabled": False,
|
||||
"heatmap": {"type": "FeatureCollection", "features": []},
|
||||
"clusters": [],
|
||||
"processed": 0,
|
||||
"timestamp": None,
|
||||
},
|
||||
}
|
||||
|
||||
# Per-source freshness timestamps
|
||||
@@ -265,10 +273,27 @@ def get_latest_data_subset(*keys: str) -> DashboardData:
|
||||
|
||||
|
||||
def get_latest_data_deepcopy_snapshot() -> DashboardData:
|
||||
"""Deep-copy the full dashboard for legacy /api/live-data consumers."""
|
||||
with _data_lock:
|
||||
items = list(latest_data.items())
|
||||
return {key: copy.deepcopy(value) for key, value in items}
|
||||
"""Deep-copy the full dashboard for /api/health and legacy /api/live-data.
|
||||
|
||||
The per-value deepcopy runs OUTSIDE ``_data_lock`` so a large clone cannot
|
||||
block fetcher writers (#375). The store contract is replace-don't-mutate,
|
||||
but a writer that mutates a nested object in place (e.g. a live bridge
|
||||
updating an entry that is also published in this store) can race the
|
||||
deepcopy and raise ``RuntimeError: dictionary changed size during
|
||||
iteration`` — surfacing a 500 on the health/live-data path. The racing
|
||||
mutation window is tiny, so retry a few times rather than fail; a fresh
|
||||
attempt almost always lands on a quiescent moment. Defense-in-depth on top
|
||||
of fixing the offending writers, not a substitute for it.
|
||||
"""
|
||||
attempts = 4
|
||||
for attempt in range(attempts):
|
||||
with _data_lock:
|
||||
items = list(latest_data.items())
|
||||
try:
|
||||
return {key: copy.deepcopy(value) for key, value in items}
|
||||
except RuntimeError:
|
||||
if attempt == attempts - 1:
|
||||
raise
|
||||
|
||||
|
||||
def get_latest_data_subset_refs(*keys: str) -> DashboardData:
|
||||
@@ -309,15 +334,15 @@ active_layers: dict[str, bool] = {
|
||||
"ships_passenger": True,
|
||||
"ships_tracked_yachts": True,
|
||||
"earthquakes": True,
|
||||
"cctv": True,
|
||||
"cctv": False,
|
||||
"ukraine_frontline": True,
|
||||
"global_incidents": True,
|
||||
"gps_jamming": True,
|
||||
"kiwisdr": True,
|
||||
"scanners": True,
|
||||
"firms": True,
|
||||
"firms": False,
|
||||
"internet_outages": True,
|
||||
"datacenters": True,
|
||||
"datacenters": False,
|
||||
"military_bases": True,
|
||||
"sigint_meshtastic": True,
|
||||
"sigint_aprs": True,
|
||||
@@ -328,9 +353,9 @@ active_layers: dict[str, bool] = {
|
||||
"satnogs": True,
|
||||
"tinygs": True,
|
||||
"ukraine_alerts": True,
|
||||
"power_plants": True,
|
||||
"power_plants": False,
|
||||
"viirs_nightlights": False,
|
||||
"psk_reporter": True,
|
||||
"psk_reporter": False,
|
||||
"correlations": True,
|
||||
"contradictions": True,
|
||||
"uap_sightings": True,
|
||||
@@ -344,6 +369,7 @@ active_layers: dict[str, bool] = {
|
||||
"scm_suppliers": False,
|
||||
"cyber_threats": False,
|
||||
"telegram_osint": True,
|
||||
"gt_risk": False,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,570 @@
|
||||
"""Heuristics to summarize ACARS/VDL payloads across airlines for dossier display."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
# --- shared patterns ---
|
||||
|
||||
_ICAO_AIRPORT = re.compile(r"\b([A-Z]{4})\b")
|
||||
_TAIL = re.compile(r"\b(?:N[0-9A-Z]{3,6}|G-[A-Z]{4,5}|[A-Z]-[A-Z]{4,5}|[A-Z]{2}-[A-Z]{3,4})\b")
|
||||
# Major carriers — explicit list avoids matching FL280, GS450, etc.
|
||||
_FLIGHT = re.compile(
|
||||
r"\b(?:"
|
||||
r"WN|SWA|UA|UAL|AA|AAL|DL|DAL|AS|ASA|B6|JBU|NK|NKS|F9|FFT|G4|HA|HAL|SY|MX|"
|
||||
r"FDX|UPS|GTI|ABX|ATN|RCH|CNV|EVAC|SAM|REACH|"
|
||||
r"BA|BAW|AF|AFR|LH|DLH|KL|KLM|QF|QFA|EK|UAE|QR|QTR|TK|THY|AC|ACA|WS|WJA|"
|
||||
r"FR|RYR|U2|EZY|VS|VIR|NH|ANA|JL|JAL|CX|CPA|SQ|SIA|NZ|ANZ|"
|
||||
r"UA|CO|NW|US|HP|TW|VX|AS|QX|OO|YX|MQ|OH|9E|"
|
||||
r"JT|JSA|VA|VOZ|NZ|QF|EK|ET|MS|SU|LO|SK|AY|IB|UX|TP|TAP"
|
||||
r")\d{1,5}\b",
|
||||
re.I,
|
||||
)
|
||||
# IATA flight numbers on FI lines and standalone (e.g. UO614, CX889).
|
||||
_FI_FLIGHT = re.compile(r"\b([A-Z]{2,3}\d{1,4})\b")
|
||||
_NON_FLIGHT_TOKENS = frozenset(
|
||||
{"FL", "FT", "GS", "KT", "RW", "NM", "TD", "TO", "ON", "IN", "OF", "AT", "DA", "AA", "AD"}
|
||||
)
|
||||
_FI_BLOCK = re.compile(
|
||||
r"FI\s+([A-Z0-9]{2,5}\d{1,5})"
|
||||
r"(?:/AN\s+([A-Z0-9\-]+))?"
|
||||
r"(?:/DA\s+([A-Z]{4}))?"
|
||||
r"(?:/(?:AA|AD|DS)\s+([A-Z]{4}))?",
|
||||
re.I,
|
||||
)
|
||||
_AC_TYPE = re.compile(
|
||||
r"\b(?:B\d{3,4}(?:-\d{3}|MAX|ER|LR|F)?|A\d{3,4}(?:-\d{3}|NEO|LR)?|"
|
||||
r"E\d{3}|MD-\d{2}|DC-\d{2}|B77[0-9LWR]?|B78[79]|A35[09]|A33[0-9]|CRJ\d{2,3}|E\d{3})\b",
|
||||
re.I,
|
||||
)
|
||||
|
||||
# --- message family patterns ---
|
||||
|
||||
_TRACK_HEADER = re.compile(
|
||||
r"^\+\+86501,([^,]+),([^,]+),(\d{6}),([^,]+),([A-Z]{4}),([A-Z]{4})",
|
||||
re.I,
|
||||
)
|
||||
_POS_HEADER = re.compile(r"^POS(N?\d{4,5}[NS]?\d{4,5}[EW]?)", re.I)
|
||||
_POS_COORDS = re.compile(r"^N?(\d{4,5})([NS])(\d{4,6})([EW])", re.I)
|
||||
_WAYPOINT = re.compile(
|
||||
r"^(?:N)?(\d{1,2}\d{2}\.\d),W(\d{1,3}\d{2}\.\d),(\d{6}),(\d+),",
|
||||
re.I,
|
||||
)
|
||||
_PERF_HEADER = re.compile(
|
||||
r"^[\w]+,(\d+),([^,]+),(\d{6}),([^,]+),([A-Z]{4}),([A-Z]{4})",
|
||||
re.I,
|
||||
)
|
||||
_PHASE_SNAPSHOT = re.compile(
|
||||
r"^(\d{2}\.\d{2}\.\d{2}),(CL|CR|DE|TO|LD|ER|GND),[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,"
|
||||
r"N(\d+)\.(\d+),W(\d+)\.(\d+)",
|
||||
re.I,
|
||||
)
|
||||
_TRAJECTORY_HEADER = re.compile(r"^76401\s*$", re.I)
|
||||
_TRAJECTORY_ROUTE = re.compile(r"^02E24([A-Z]{4})([A-Z]{4})\s*$", re.I)
|
||||
_COMPRESSED_WP = re.compile(r"^N(\d{5})W(\d{5})", re.I)
|
||||
_FPN = re.compile(r"^FPN/?", re.I)
|
||||
_OOOI_TIMES = re.compile(r"\b(OUT|OFF|ON|IN)\s*(\d{4,6})\b", re.I)
|
||||
_OOOI_STATUS = re.compile(r"\b(OUT|OFF|ON|IN)\s*,\s*(LO|CL|ON|OF|CLOS)\b", re.I)
|
||||
_ETA = re.compile(r"\bETA\s+(\d{3,4}Z?)\b", re.I)
|
||||
_DEP_ARR = re.compile(r"^(DEP|ARR|DLA|ALR)\b", re.I)
|
||||
_WX = re.compile(r"^(?:WXR?\d*|WX\s|MET\b|/WX\b)", re.I)
|
||||
_REQ = re.compile(r"^(?:REQ|REQUEST)\b", re.I)
|
||||
_LDR = re.compile(r"^LDR\d+", re.I)
|
||||
_PIREP = re.compile(r"^#(?:CFB|DFB)", re.I)
|
||||
_ATN = re.compile(r"^USADCXA\.AT1\.", re.I)
|
||||
_CPDLC = re.compile(r"^(?:DM-|UM-|AT1\.|ATC\s)", re.I)
|
||||
_ENG = re.compile(r"^(?:ENG\d|/ENG|OILTEMP|EGT\b)", re.I)
|
||||
_DOOR = re.compile(r"^(?:DOOR|CABIN|SMOKE)\b", re.I)
|
||||
_VDL_FRAME = re.compile(r"^[0-9A-F]{6,8}[A-Z]?\s*$", re.I)
|
||||
_FRAGMENT = re.compile(r"^[,0\s]+(?:,\d{5,8},\d{5,8},\d{5,8})*$", re.I)
|
||||
_GARBLED_VDL = re.compile(r"[)Z][A-Z0-9,\-:]{20,}")
|
||||
_MOSTLY_OPAQUE = re.compile(r"^[0-9A-Fa-f\s.\-+/,]{40,}$")
|
||||
_FREE_TEXT_POS = re.compile(
|
||||
r"^POS\s+N?(\d{1,2}\.\d+)\s+([NS])\s+W?(\d{1,3}\.\d+)\s+([EW])\s+FL(\d{3})",
|
||||
re.I,
|
||||
)
|
||||
_CLIMB_REQ = re.compile(r"\b(?:CLIMB|DESCEND|REQUEST)\s+(?:FL)?(\d{2,3})\b", re.I)
|
||||
|
||||
_LABEL_HINTS: dict[str, str] = {
|
||||
"00": "out (gate)",
|
||||
"01": "off (takeoff)",
|
||||
"02": "on (landing)",
|
||||
"03": "in (gate)",
|
||||
"10": "position",
|
||||
"15": "waypoint",
|
||||
"20": "position",
|
||||
"40": "ops / clearance",
|
||||
"44": "OOOI + position",
|
||||
"80": "weather",
|
||||
"81": "wind",
|
||||
"B1": "engine 1",
|
||||
"B2": "engine 2",
|
||||
"B3": "engine 3",
|
||||
"B4": "engine 4",
|
||||
"M1": "maintenance",
|
||||
"M2": "maintenance",
|
||||
"M3": "maintenance",
|
||||
"M4": "maintenance",
|
||||
"Q0": "position / OOOI",
|
||||
"H1": "terminal",
|
||||
"D0": "ATC clearance",
|
||||
"S1": "system status",
|
||||
"SA": "system status",
|
||||
"SB": "system status",
|
||||
"4T": "met report",
|
||||
"5Z": "free text",
|
||||
}
|
||||
|
||||
|
||||
def _result(
|
||||
summary: str,
|
||||
*,
|
||||
kind: str,
|
||||
readable: bool = True,
|
||||
hidden: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"summary": summary,
|
||||
"kind": kind,
|
||||
"readable": readable,
|
||||
"hidden": hidden,
|
||||
}
|
||||
|
||||
|
||||
def _phase_name(code: str) -> str:
|
||||
return {
|
||||
"CL": "climb",
|
||||
"CR": "cruise",
|
||||
"DE": "descent",
|
||||
"ER": "en route",
|
||||
"TO": "takeoff",
|
||||
"LD": "landed",
|
||||
"ON": "on ground",
|
||||
"OF": "off block",
|
||||
"GND": "on ground",
|
||||
"LO": "level",
|
||||
}.get(code.upper(), code.upper() or "unknown")
|
||||
|
||||
|
||||
def _fmt_coords(lat_deg: str, lat_frac: str, lon_deg: str, lon_frac: str) -> str:
|
||||
return f"{int(lat_deg)}°{lat_frac}'N {int(lon_deg)}°{lon_frac}'W"
|
||||
|
||||
|
||||
def _parse_pos_coords(token: str) -> str | None:
|
||||
token = token.upper().lstrip("POS")
|
||||
match = _POS_COORDS.match(token)
|
||||
if not match:
|
||||
return None
|
||||
lat, lat_dir, lon, lon_dir = match.groups()
|
||||
lat_v = f"{int(lat[:2])}°{lat[2:]}.{lat[4:] if len(lat) > 4 else '0'}'{lat_dir}"
|
||||
lon_v = f"{int(lon[:3])}°{lon[3:]}.{lon[5:] if len(lon) > 5 else '0'}'{lon_dir}"
|
||||
return f"{lat_v} {lon_v}"
|
||||
|
||||
|
||||
def _extract_route(raw: str) -> str:
|
||||
fi = _FI_BLOCK.search(raw)
|
||||
if fi:
|
||||
flight, _tail, dep, dest = fi.groups()
|
||||
parts = [flight.upper()]
|
||||
if dep and dest:
|
||||
parts.append(f"{dep}→{dest}")
|
||||
elif dep:
|
||||
parts.append(f"from {dep}")
|
||||
elif dest:
|
||||
parts.append(f"to {dest}")
|
||||
return " · ".join(parts)
|
||||
|
||||
airports = _ICAO_AIRPORT.findall(raw)
|
||||
# Filter duplicates while preserving order
|
||||
seen: set[str] = set()
|
||||
ordered: list[str] = []
|
||||
for apt in airports:
|
||||
if apt in seen:
|
||||
continue
|
||||
seen.add(apt)
|
||||
ordered.append(apt)
|
||||
if len(ordered) >= 2:
|
||||
return f"{ordered[0]}→{ordered[-1]}"
|
||||
if ordered:
|
||||
return ordered[0]
|
||||
return ""
|
||||
|
||||
|
||||
def _extract_flight(raw: str) -> str:
|
||||
fi = _FI_BLOCK.search(raw)
|
||||
if fi and fi.group(1):
|
||||
return fi.group(1).upper()
|
||||
for match in _FLIGHT.finditer(raw):
|
||||
return match.group(0).upper()
|
||||
for match in _FI_FLIGHT.finditer(raw):
|
||||
token = match.group(1).upper()
|
||||
prefix = re.match(r"^([A-Z]+)", token)
|
||||
if prefix and prefix.group(1) not in _NON_FLIGHT_TOKENS:
|
||||
return token
|
||||
return ""
|
||||
|
||||
|
||||
def _extract_phase_snapshot(raw: str) -> str | None:
|
||||
for line in raw.splitlines():
|
||||
match = _PHASE_SNAPSHOT.match(line.strip())
|
||||
if not match:
|
||||
continue
|
||||
time_s, phase, lat_d, lat_f, lon_d, lon_f = match.groups()
|
||||
coords = _fmt_coords(lat_d, lat_f, lon_d, lon_f)
|
||||
return f"{_phase_name(phase)} · {coords} · {time_s}Z"
|
||||
return None
|
||||
|
||||
|
||||
def _has_aircraft_context(raw: str) -> bool:
|
||||
head = raw[:160].upper()
|
||||
if _AC_TYPE.search(head):
|
||||
return True
|
||||
if _FLIGHT.search(head):
|
||||
return True
|
||||
if _FI_BLOCK.search(head):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _is_fragment(raw: str) -> bool:
|
||||
first = raw.splitlines()[0].strip()
|
||||
if _FRAGMENT.match(first):
|
||||
return True
|
||||
if re.match(r"^[,0]{1,12}$", first):
|
||||
return True
|
||||
if first.startswith("000000") or first.startswith(",000000"):
|
||||
return True
|
||||
if re.match(r"^\d{2}\.\d{2}\.\d{2},", first) and not _has_aircraft_context(raw):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _summarize_oooi(raw: str, label: str) -> dict[str, Any] | None:
|
||||
times = _OOOI_TIMES.findall(raw)
|
||||
statuses = _OOOI_STATUS.findall(raw)
|
||||
if not times and not statuses and label not in {"00", "01", "02", "03", "44", "Q0"}:
|
||||
return None
|
||||
|
||||
events: list[str] = []
|
||||
for event, value in times:
|
||||
events.append(f"{event.upper()} {value}")
|
||||
for event, status in statuses:
|
||||
events.append(f"{event.upper()} ({_phase_name(status)})")
|
||||
|
||||
if label in {"00", "01", "02", "03"} and not events:
|
||||
events.append(_LABEL_HINTS[label])
|
||||
|
||||
if not events and "ON ,LO" not in raw and "OFF,OFF" not in raw:
|
||||
return None
|
||||
|
||||
route = _extract_route(raw)
|
||||
flight = _extract_flight(raw)
|
||||
prefix = "OOOI"
|
||||
if label in _LABEL_HINTS:
|
||||
prefix = f"OOOI ({_LABEL_HINTS[label]})"
|
||||
bits = [prefix]
|
||||
if flight:
|
||||
bits.append(flight)
|
||||
if route:
|
||||
bits.append(route)
|
||||
if events:
|
||||
bits.append(", ".join(events[:4]))
|
||||
return _result(" · ".join(bits), kind="oooi")
|
||||
|
||||
|
||||
def _summarize_position(raw: str, first_line: str) -> dict[str, Any] | None:
|
||||
upper = first_line.upper()
|
||||
pos_token = re.search(r"POSN?\d", raw, re.I)
|
||||
if not (upper.startswith("POS") or _POS_HEADER.match(first_line) or pos_token):
|
||||
return None
|
||||
|
||||
coord_line = first_line
|
||||
if pos_token and not upper.startswith("POS"):
|
||||
coord_line = raw[pos_token.start() :].split(",")[0]
|
||||
|
||||
coords = _parse_pos_coords(coord_line)
|
||||
free = _FREE_TEXT_POS.match(raw)
|
||||
fl = ""
|
||||
if free:
|
||||
lat, lat_dir, lon, lon_dir, fl = free.groups()
|
||||
coords = f"{lat}°{lat_dir} {lon}°{lon_dir}"
|
||||
fl = f"FL{fl}"
|
||||
|
||||
route = _extract_route(raw)
|
||||
flight = _extract_flight(raw)
|
||||
parts = ["Position report"]
|
||||
if flight:
|
||||
parts.append(flight)
|
||||
if route:
|
||||
parts.append(route)
|
||||
if coords:
|
||||
parts.append(coords)
|
||||
if fl:
|
||||
parts.append(fl)
|
||||
elif re.search(r"\bFL?\d{3}\b", raw):
|
||||
fl_match = re.search(r"\bFL?(\d{2,3})\b", raw)
|
||||
if fl_match:
|
||||
parts.append(f"FL{fl_match.group(1)}")
|
||||
return _result(" · ".join(parts), kind="position")
|
||||
|
||||
|
||||
def _summarize_performance(raw: str, first_line: str) -> dict[str, Any] | None:
|
||||
match = _PERF_HEADER.match(first_line)
|
||||
if not match or not _AC_TYPE.search(first_line):
|
||||
return None
|
||||
|
||||
_serial, ac_type, _date, flight, dep, dest = match.groups()
|
||||
phase_bits = _extract_phase_snapshot(raw) or ""
|
||||
extra = f" · {phase_bits}" if phase_bits else ""
|
||||
|
||||
if "FHP" in raw or "SIN," in raw or "SOU," in raw:
|
||||
title, kind = "Engine health (FHP)", "engine_health"
|
||||
elif "OATTO" in raw or "LPACKCL" in raw or "RPACKCL" in raw:
|
||||
title, kind = "Pack temperature", "pack_temp"
|
||||
elif "FLAPS" in raw.upper():
|
||||
title, kind = "Climb performance", "climb_perf"
|
||||
elif "FRE," in raw or "FEX," in raw:
|
||||
title, kind = "Fuel/performance snapshot", "fuel_perf"
|
||||
else:
|
||||
title, kind = "Flight performance", "performance"
|
||||
|
||||
return _result(
|
||||
f"{title} · {flight} · {ac_type} · {dep}→{dest}{extra}",
|
||||
kind=kind,
|
||||
)
|
||||
|
||||
|
||||
def _summarize_by_label(label: str, raw: str, first_line: str) -> dict[str, Any] | None:
|
||||
label_u = label.upper()
|
||||
hint = _LABEL_HINTS.get(label_u, "")
|
||||
|
||||
if label_u in {"B1", "B2", "B3", "B4"} or _ENG.match(first_line):
|
||||
eng = label_u if label_u.startswith("B") else "Engine"
|
||||
return _result(f"{eng} data report", kind="engine", readable=bool(hint))
|
||||
|
||||
if label_u.startswith("M") and label_u[1:2].isdigit():
|
||||
return _result(f"Maintenance ({hint or 'system report'})", kind="maintenance")
|
||||
|
||||
if label_u in {"80", "81", "4T"} or _WX.match(first_line):
|
||||
apt = _ICAO_AIRPORT.search(raw)
|
||||
apt_s = f" · {apt.group(1)}" if apt else ""
|
||||
return _result(f"Weather report{apt_s}", kind="weather")
|
||||
|
||||
if label_u == "D0" or _REQ.match(first_line) or _CLIMB_REQ.search(raw):
|
||||
climb = _CLIMB_REQ.search(raw)
|
||||
if climb:
|
||||
return _result(f"Altitude request · FL{climb.group(1)}", kind="request")
|
||||
return _result("ATC / ops request", kind="request")
|
||||
|
||||
if label_u in {"40", "5Z"} and len(raw) < 200:
|
||||
text = raw.replace("\n", " · ")[:140]
|
||||
return _result(f"Ops message · {text}", kind="ops")
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def summarize_datalink_message(
|
||||
*,
|
||||
label: str = "",
|
||||
text: str = "",
|
||||
source_type: str = "",
|
||||
) -> dict[str, Any]:
|
||||
"""Return {summary, kind, readable, hidden} for a cached datalink message."""
|
||||
raw = (text or "").strip()
|
||||
if not raw:
|
||||
return _result("", kind="empty", readable=False, hidden=True)
|
||||
|
||||
first_line = raw.splitlines()[0].strip()
|
||||
upper = first_line.upper()
|
||||
label_u = label.upper()
|
||||
|
||||
if _is_fragment(raw):
|
||||
return _result(
|
||||
"Split telemetry fragment (part of a longer VDL message)",
|
||||
kind="fragment",
|
||||
readable=False,
|
||||
hidden=True,
|
||||
)
|
||||
|
||||
if _ATN.match(first_line) or _CPDLC.match(first_line):
|
||||
tail = _TAIL.search(raw)
|
||||
return _result(
|
||||
"Datalink protocol / CPDLC header" + (f" · {tail.group(0)}" if tail else ""),
|
||||
kind="protocol",
|
||||
readable=False,
|
||||
hidden=True,
|
||||
)
|
||||
|
||||
if label_u == "37" or (_VDL_FRAME.match(first_line) and len(raw) < 160):
|
||||
if _GARBLED_VDL.search(raw) or len(raw) < 160:
|
||||
return _result("VDL binary frame (undecoded)", kind="vdl_binary", readable=False, hidden=True)
|
||||
|
||||
# --- structured families (order matters) ---
|
||||
|
||||
if _DEP_ARR.match(first_line):
|
||||
kind_word = first_line.split()[0].upper()
|
||||
route = _extract_route(raw)
|
||||
flight = _extract_flight(raw)
|
||||
title = {"DEP": "Departure", "ARR": "Arrival", "DLA": "Delay", "ALR": "Alert"}.get(
|
||||
kind_word, kind_word
|
||||
)
|
||||
bits = [title]
|
||||
if flight:
|
||||
bits.append(flight)
|
||||
if route:
|
||||
bits.append(route)
|
||||
return _result(" · ".join(bits), kind=kind_word.lower())
|
||||
|
||||
oooi = _summarize_oooi(raw, label_u)
|
||||
if oooi:
|
||||
return oooi
|
||||
|
||||
match = _TRACK_HEADER.match(first_line)
|
||||
if match:
|
||||
tail, ac_type, _date, flight, dep, dest = match.groups()
|
||||
lines = [line.strip() for line in raw.splitlines() if line.strip()]
|
||||
waypoint_lines = [line for line in lines if _WAYPOINT.match(line.lstrip("N"))]
|
||||
phase = ""
|
||||
if waypoint_lines:
|
||||
parts = waypoint_lines[-1].rstrip(",").split(",")
|
||||
if len(parts) >= 8:
|
||||
phase = _phase_name(parts[7])
|
||||
wp_count = len(waypoint_lines) or max(0, len(lines) - 2)
|
||||
summary = (
|
||||
f"Track report · {flight} · {tail} ({ac_type}) · {dep}→{dest}"
|
||||
+ (f" · {wp_count} waypoint(s)" + (f" · {phase}" if phase else ""))
|
||||
)
|
||||
return _result(summary, kind="track")
|
||||
|
||||
pos = _summarize_position(raw, first_line)
|
||||
if pos:
|
||||
return pos
|
||||
|
||||
if _FPN.match(first_line):
|
||||
route = _extract_route(raw)
|
||||
flight = _extract_flight(raw)
|
||||
bits = ["Flight plan"]
|
||||
if flight:
|
||||
bits.append(flight)
|
||||
if route:
|
||||
bits.append(route)
|
||||
return _result(" · ".join(bits), kind="flight_plan")
|
||||
|
||||
if _PIREP.match(first_line):
|
||||
return _result("Pilot report (PIREP)", kind="pirep")
|
||||
|
||||
lines = [line.strip() for line in raw.splitlines() if line.strip()]
|
||||
if _TRAJECTORY_HEADER.match(first_line) or (len(lines) >= 2 and _TRAJECTORY_ROUTE.match(lines[1])):
|
||||
route = ""
|
||||
route_match = next((m for line in lines if (m := _TRAJECTORY_ROUTE.match(line))), None)
|
||||
if route_match:
|
||||
route = f" · {route_match.group(1)}→{route_match.group(2)}"
|
||||
wp_count = sum(1 for line in lines if _COMPRESSED_WP.match(line))
|
||||
return _result(
|
||||
f"Trajectory / ADS report{route}" + (f" · {wp_count} point(s)" if wp_count else ""),
|
||||
kind="trajectory",
|
||||
)
|
||||
|
||||
perf = _summarize_performance(raw, first_line)
|
||||
if perf:
|
||||
return perf
|
||||
|
||||
if _LDR.match(first_line):
|
||||
route = _extract_route(raw)
|
||||
return _result(f"Load report · {route}" if route else "Load report", kind="load")
|
||||
|
||||
if _WX.match(first_line):
|
||||
route = _extract_route(raw)
|
||||
return _result(f"Weather request · {route or 'en route'}", kind="weather")
|
||||
|
||||
if _DOOR.match(first_line):
|
||||
return _result("Cabin / door advisory", kind="cabin")
|
||||
|
||||
if _WAYPOINT.match(first_line.lstrip("N")):
|
||||
parts = first_line.lstrip("N").rstrip(",").split(",")
|
||||
if len(parts) >= 4:
|
||||
lat, lon, _t, alt = parts[0], parts[1], parts[2], parts[3]
|
||||
phase = _phase_name(parts[7]) if len(parts) >= 8 else ""
|
||||
summary = f"Waypoint · {lat},{lon} · alt {alt} ft" + (f" · {phase}" if phase else "")
|
||||
return _result(summary, kind="waypoint")
|
||||
|
||||
label_summary = _summarize_by_label(label_u, raw, first_line)
|
||||
if label_summary:
|
||||
return label_summary
|
||||
|
||||
flight = _extract_flight(raw)
|
||||
route = _extract_route(raw)
|
||||
if flight and route:
|
||||
return _result(f"Datalink · {flight} · {route}", kind="flight")
|
||||
|
||||
eta = _ETA.search(raw)
|
||||
if eta and flight:
|
||||
return _result(f"ETA update · {flight} · {eta.group(1)}", kind="eta")
|
||||
|
||||
if len(raw) < 100 and not _MOSTLY_OPAQUE.match(raw) and not _GARBLED_VDL.search(raw):
|
||||
clean = raw.replace("\n", " · ")
|
||||
if label_u in _LABEL_HINTS:
|
||||
return _result(f"{_LABEL_HINTS[label_u].title()} · {clean}", kind="short")
|
||||
return _result(clean, kind="short")
|
||||
|
||||
digit_ratio = sum(ch.isdigit() for ch in raw) / max(len(raw), 1)
|
||||
if digit_ratio > 0.55 or _MOSTLY_OPAQUE.match(raw.replace(" ", "")) or _GARBLED_VDL.search(raw):
|
||||
return _result(
|
||||
"Binary / proprietary telemetry (undecoded)",
|
||||
kind="vdl_binary",
|
||||
readable=False,
|
||||
hidden=True,
|
||||
)
|
||||
|
||||
if label_u in _LABEL_HINTS:
|
||||
return _result(
|
||||
f"{_LABEL_HINTS[label_u].title()} message",
|
||||
kind=label_u.lower(),
|
||||
readable=False,
|
||||
hidden=False,
|
||||
)
|
||||
|
||||
return _result(
|
||||
first_line[:100] + ("…" if len(first_line) > 100 else ""),
|
||||
kind="raw",
|
||||
readable=False,
|
||||
hidden=False,
|
||||
)
|
||||
|
||||
|
||||
def prepare_datalink_display(messages: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""Attach summaries and filter noise for dossier display."""
|
||||
enriched: list[dict[str, Any]] = []
|
||||
hidden_count = 0
|
||||
seen_summaries: set[str] = set()
|
||||
|
||||
for message in messages:
|
||||
meta = summarize_datalink_message(
|
||||
label=str(message.get("label") or ""),
|
||||
text=str(message.get("text") or ""),
|
||||
source_type=str(message.get("source_type") or ""),
|
||||
)
|
||||
item = {**message, **meta}
|
||||
if item.get("hidden"):
|
||||
hidden_count += 1
|
||||
continue
|
||||
|
||||
# Drop back-to-back duplicate summaries (common with multi-part VDL)
|
||||
sig = f"{item.get('kind')}|{item.get('summary')}"
|
||||
if sig in seen_summaries and item.get("kind") not in {"short", "ops", "request"}:
|
||||
hidden_count += 1
|
||||
continue
|
||||
seen_summaries.add(sig)
|
||||
|
||||
enriched.append(item)
|
||||
|
||||
return {
|
||||
"messages": enriched,
|
||||
"hidden_count": hidden_count,
|
||||
"total_count": len(messages),
|
||||
}
|
||||
|
||||
|
||||
def attach_summaries(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
return prepare_datalink_display(messages)["messages"]
|
||||
@@ -0,0 +1,611 @@
|
||||
"""Airframes.io ACARS/VDL datalink ingest — staggered queue cache for plane dossiers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger("services.airframes")
|
||||
|
||||
API_BASE = os.environ.get("AIRFRAMES_API_BASE", "https://api.airframes.io/v1").rstrip("/")
|
||||
SYNC_INTERVAL_MINUTES = max(5, int(os.environ.get("AIRFRAMES_SYNC_INTERVAL_MINUTES", "15")))
|
||||
MAX_BULK_PAGES_PER_CYCLE = max(1, int(os.environ.get("AIRFRAMES_MAX_PAGES_PER_SYNC", "28")))
|
||||
MESSAGES_PER_AIRCRAFT = max(5, int(os.environ.get("AIRFRAMES_MESSAGES_PER_AIRCRAFT", "40")))
|
||||
RETENTION_HOURS = max(6, int(os.environ.get("AIRFRAMES_RETENTION_HOURS", "48")))
|
||||
# 2s between calls => 30/min, safely under Airframes 60/min cap.
|
||||
REQUEST_PAUSE_S = float(os.environ.get("AIRFRAMES_REQUEST_PAUSE_S", "2.0"))
|
||||
PRIORITY_LOOKBACK_HOURS = max(6, int(os.environ.get("AIRFRAMES_PRIORITY_LOOKBACK_HOURS", "48")))
|
||||
FETCH_TIMEOUT_S = max(5, int(os.environ.get("AIRFRAMES_FETCH_TIMEOUT_S", "20")))
|
||||
|
||||
_DATA_DIR = Path(os.environ.get("SB_DATA_DIR", str(Path(__file__).resolve().parents[2] / "data")))
|
||||
if not _DATA_DIR.is_absolute():
|
||||
_DATA_DIR = Path(__file__).resolve().parents[2] / _DATA_DIR
|
||||
_CACHE_PATH = _DATA_DIR / "airframes_datalink_cache.json"
|
||||
|
||||
_lock = threading.Lock()
|
||||
_queue_lock = threading.Lock()
|
||||
_worker_guard = threading.Lock()
|
||||
_queue: deque[dict[str, Any]] = deque()
|
||||
_queued_aircraft_keys: set[str] = set()
|
||||
_bulk_cursor: dict[str, Any] = {"since_iso": "", "before_id": None, "pages": 0}
|
||||
_worker_started = False
|
||||
_cache_loaded = False
|
||||
_save_timer: threading.Timer | None = None
|
||||
_save_timer_lock = threading.Lock()
|
||||
_api_key_known_configured: bool | None = None
|
||||
_cache: dict[str, Any] = {
|
||||
"last_sync_at": None,
|
||||
"last_success_at": None,
|
||||
"last_error": None,
|
||||
"pages_fetched": 0,
|
||||
"messages_ingested": 0,
|
||||
"bulk_pages_this_cycle": 0,
|
||||
"ticks_processed": 0,
|
||||
"by_icao": {},
|
||||
"by_tail": {},
|
||||
"by_callsign": {},
|
||||
}
|
||||
|
||||
|
||||
def _utc_now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _iso(dt: datetime) -> str:
|
||||
return dt.astimezone(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def _parse_ts(value: str | None) -> datetime | None:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
cleaned = value.replace("Z", "+00:00")
|
||||
return datetime.fromisoformat(cleaned).astimezone(timezone.utc)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def api_key_configured() -> bool:
|
||||
global _api_key_known_configured
|
||||
if os.environ.get("AIRFRAMES_API_KEY", "").strip():
|
||||
_api_key_known_configured = True
|
||||
return True
|
||||
if _api_key_known_configured is False:
|
||||
return False
|
||||
from services.api_settings import load_persisted_api_keys_into_environ
|
||||
|
||||
load_persisted_api_keys_into_environ()
|
||||
_api_key_known_configured = bool(os.environ.get("AIRFRAMES_API_KEY", "").strip())
|
||||
return _api_key_known_configured
|
||||
|
||||
|
||||
def _norm_hex(value: str | None) -> str:
|
||||
return (value or "").strip().lower()
|
||||
|
||||
|
||||
def _norm_tail(value: str | None) -> str:
|
||||
return re.sub(r"[^A-Z0-9]", "", (value or "").strip().upper())
|
||||
|
||||
|
||||
def _norm_callsign(value: str | None) -> str:
|
||||
return re.sub(r"\s+", "", (value or "").strip().upper())
|
||||
|
||||
|
||||
def _aircraft_queue_key(entry: dict[str, str]) -> str:
|
||||
return f"{entry.get('icao24', '')}|{entry.get('registration', '')}|{entry.get('callsign', '')}"
|
||||
|
||||
|
||||
def _tail_lookup_keys(value: str | None) -> list[str]:
|
||||
tail = _norm_tail(value)
|
||||
if not tail:
|
||||
return []
|
||||
keys = [tail]
|
||||
raw = (value or "").strip().upper()
|
||||
if raw and raw not in keys:
|
||||
keys.append(raw)
|
||||
return keys
|
||||
|
||||
|
||||
def _load_cache_if_cold() -> None:
|
||||
global _cache, _cache_loaded
|
||||
if _cache_loaded:
|
||||
return
|
||||
loaded: dict[str, Any] | None = None
|
||||
if _CACHE_PATH.exists():
|
||||
try:
|
||||
with _CACHE_PATH.open(encoding="utf-8") as handle:
|
||||
parsed = json.load(handle)
|
||||
if isinstance(parsed, dict):
|
||||
loaded = parsed
|
||||
except (OSError, json.JSONDecodeError, ValueError) as exc:
|
||||
logger.warning("Failed to load Airframes cache: %s", exc)
|
||||
with _lock:
|
||||
if _cache_loaded:
|
||||
return
|
||||
if loaded:
|
||||
_cache.update(loaded)
|
||||
_cache.setdefault("by_callsign", {})
|
||||
_cache_loaded = True
|
||||
|
||||
|
||||
def _persist_cache_now() -> None:
|
||||
with _lock:
|
||||
snapshot = json.dumps(_cache, indent=2, ensure_ascii=False) + "\n"
|
||||
_DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
tmp = _CACHE_PATH.with_suffix(".tmp")
|
||||
tmp.write_text(snapshot, encoding="utf-8")
|
||||
tmp.replace(_CACHE_PATH)
|
||||
|
||||
|
||||
def _schedule_cache_persist() -> None:
|
||||
global _save_timer
|
||||
|
||||
def _flush() -> None:
|
||||
global _save_timer
|
||||
try:
|
||||
_persist_cache_now()
|
||||
except OSError as exc:
|
||||
logger.warning("Failed to save Airframes cache: %s", exc)
|
||||
finally:
|
||||
with _save_timer_lock:
|
||||
_save_timer = None
|
||||
|
||||
with _save_timer_lock:
|
||||
if _save_timer is not None:
|
||||
_save_timer.cancel()
|
||||
_save_timer = threading.Timer(0.75, _flush)
|
||||
_save_timer.daemon = True
|
||||
_save_timer.start()
|
||||
|
||||
|
||||
def _save_cache() -> None:
|
||||
_schedule_cache_persist()
|
||||
|
||||
|
||||
def _compact_message(raw: dict[str, Any]) -> dict[str, Any] | None:
|
||||
text = (raw.get("text") or raw.get("data") or "").strip()
|
||||
if not text:
|
||||
return None
|
||||
msg_id = raw.get("id")
|
||||
if msg_id is None:
|
||||
return None
|
||||
return {
|
||||
"id": int(msg_id),
|
||||
"timestamp": raw.get("timestamp") or raw.get("createdAt") or "",
|
||||
"label": str(raw.get("label") or "").strip(),
|
||||
"text": text[:500],
|
||||
"source_type": str(raw.get("sourceType") or raw.get("source") or "").strip(),
|
||||
"tail": _norm_tail(raw.get("tail")),
|
||||
"flight_number": _norm_callsign(raw.get("flightNumber")),
|
||||
"from_hex": _norm_hex(raw.get("fromHex")),
|
||||
"to_hex": _norm_hex(raw.get("toHex")),
|
||||
}
|
||||
|
||||
|
||||
def _bucket_key(store: dict[str, list], key: str, message: dict[str, Any]) -> None:
|
||||
if not key:
|
||||
return
|
||||
bucket = store.setdefault(key, [])
|
||||
if any(existing.get("id") == message["id"] for existing in bucket):
|
||||
return
|
||||
bucket.append(message)
|
||||
bucket.sort(key=lambda item: item.get("timestamp") or "", reverse=True)
|
||||
del bucket[MESSAGES_PER_AIRCRAFT:]
|
||||
|
||||
|
||||
def _index_message(compact: dict[str, Any]) -> None:
|
||||
for hex_code in (compact.get("from_hex"), compact.get("to_hex")):
|
||||
if hex_code:
|
||||
_bucket_key(_cache["by_icao"], hex_code, compact)
|
||||
for tail_key in _tail_lookup_keys(compact.get("tail")):
|
||||
_bucket_key(_cache["by_tail"], tail_key, compact)
|
||||
callsign = compact.get("flight_number")
|
||||
if callsign:
|
||||
_bucket_key(_cache["by_callsign"], callsign, compact)
|
||||
|
||||
|
||||
def _prune_store(store: dict[str, list]) -> None:
|
||||
cutoff = _utc_now() - timedelta(hours=RETENTION_HOURS)
|
||||
for key in list(store.keys()):
|
||||
kept = []
|
||||
for message in store.get(key, []):
|
||||
ts = _parse_ts(message.get("timestamp"))
|
||||
if ts is None or ts >= cutoff:
|
||||
kept.append(message)
|
||||
if kept:
|
||||
store[key] = kept[:MESSAGES_PER_AIRCRAFT]
|
||||
else:
|
||||
del store[key]
|
||||
|
||||
|
||||
def _ingest_message(message: dict[str, Any]) -> bool:
|
||||
compact = _compact_message(message)
|
||||
if not compact:
|
||||
return False
|
||||
_index_message(compact)
|
||||
return True
|
||||
|
||||
|
||||
def _ingest_messages_batch(raw_messages: list[dict[str, Any]]) -> int:
|
||||
if not raw_messages:
|
||||
return 0
|
||||
ingested = 0
|
||||
with _lock:
|
||||
_cache.setdefault("by_callsign", {})
|
||||
for raw in raw_messages:
|
||||
if _ingest_message(raw):
|
||||
ingested += 1
|
||||
if ingested:
|
||||
_cache["messages_ingested"] = int(_cache.get("messages_ingested", 0)) + ingested
|
||||
_cache["last_success_at"] = _iso(_utc_now())
|
||||
_save_cache()
|
||||
return ingested
|
||||
|
||||
|
||||
def _fetch_messages(*, api_key: str, params: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
response = requests.get(
|
||||
f"{API_BASE}/messages",
|
||||
headers={"Authorization": f"Bearer {api_key}"},
|
||||
params=params,
|
||||
timeout=FETCH_TIMEOUT_S,
|
||||
)
|
||||
if response.status_code == 404:
|
||||
logger.debug("Airframes messages 404 for params=%s", params)
|
||||
return []
|
||||
if response.status_code == 429:
|
||||
retry_after = int(response.headers.get("Retry-After", "60"))
|
||||
raise RuntimeError(f"rate_limited:{retry_after}")
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
if not isinstance(payload, list):
|
||||
return []
|
||||
return [item for item in payload if isinstance(item, dict)]
|
||||
|
||||
|
||||
def _refill_queue(*, since_iso: str, force: bool = False) -> int:
|
||||
"""Queue bulk global ingest only — each bulk call returns up to 100 messages
|
||||
across many aircraft. Per-plane calls happen only on dossier cache miss."""
|
||||
global _bulk_cursor, _queued_aircraft_keys
|
||||
|
||||
with _queue_lock:
|
||||
if force:
|
||||
_queue.clear()
|
||||
_queued_aircraft_keys = set()
|
||||
_bulk_cursor = {"since_iso": since_iso, "before_id": None, "pages": 0}
|
||||
|
||||
added = 0
|
||||
has_bulk = any(item.get("type") == "bulk" for item in _queue)
|
||||
if not has_bulk:
|
||||
_bulk_cursor["since_iso"] = since_iso
|
||||
_queue.append({"type": "bulk", "since_iso": since_iso, "before_id": None})
|
||||
added += 1
|
||||
|
||||
with _lock:
|
||||
_cache["bulk_pages_this_cycle"] = 0
|
||||
_save_cache()
|
||||
|
||||
return added
|
||||
|
||||
|
||||
def _prioritize_aircraft_scan(entry: dict[str, str]) -> bool:
|
||||
"""Jump this aircraft to the front of the queue — next API tick (~2s)."""
|
||||
key = _aircraft_queue_key(entry)
|
||||
if key.replace("|", "").strip() == "":
|
||||
return False
|
||||
|
||||
item = {"type": "aircraft", **entry}
|
||||
with _queue_lock:
|
||||
kept: deque[dict[str, Any]] = deque()
|
||||
for queued in _queue:
|
||||
if queued.get("type") == "aircraft" and _aircraft_queue_key(queued) == key:
|
||||
continue
|
||||
kept.append(queued)
|
||||
_queue.clear()
|
||||
_queue.extend(kept)
|
||||
_queued_aircraft_keys.discard(key)
|
||||
_queued_aircraft_keys.add(key)
|
||||
_queue.appendleft(item)
|
||||
return True
|
||||
|
||||
|
||||
def _enqueue_bulk_page(*, since_iso: str, before_id: int | None = None) -> None:
|
||||
with _queue_lock:
|
||||
_queue.append({"type": "bulk", "since_iso": since_iso, "before_id": before_id})
|
||||
|
||||
|
||||
def _process_aircraft_item(api_key: str, entry: dict[str, str]) -> int:
|
||||
since_iso = _iso(_utc_now() - timedelta(hours=PRIORITY_LOOKBACK_HOURS))
|
||||
params: dict[str, Any] = {
|
||||
"since": since_iso,
|
||||
"limit": 100,
|
||||
"exclude_errors": "1",
|
||||
}
|
||||
if entry.get("icao24"):
|
||||
params["icao"] = entry["icao24"]
|
||||
elif entry.get("registration"):
|
||||
params["text"] = entry["registration"]
|
||||
elif entry.get("callsign"):
|
||||
params["text"] = entry["callsign"]
|
||||
else:
|
||||
return 0
|
||||
|
||||
try:
|
||||
batch = _fetch_messages(api_key=api_key, params=params)
|
||||
except Exception as exc:
|
||||
logger.debug("Airframes aircraft fetch failed for %s: %s", entry, exc)
|
||||
with _lock:
|
||||
_cache["last_error"] = str(exc)[:240]
|
||||
_save_cache()
|
||||
return 0
|
||||
|
||||
return _ingest_messages_batch(batch)
|
||||
|
||||
|
||||
def _process_bulk_item(api_key: str, item: dict[str, Any]) -> int:
|
||||
global _bulk_cursor
|
||||
|
||||
params: dict[str, Any] = {
|
||||
"since": item["since_iso"],
|
||||
"limit": 100,
|
||||
"exclude_errors": "1",
|
||||
}
|
||||
before_id = item.get("before_id")
|
||||
if before_id is not None:
|
||||
params["before_id"] = before_id
|
||||
|
||||
try:
|
||||
batch = _fetch_messages(api_key=api_key, params=params)
|
||||
except Exception as exc:
|
||||
logger.debug("Airframes bulk fetch failed: %s", exc)
|
||||
with _lock:
|
||||
_cache["last_error"] = str(exc)[:240]
|
||||
_save_cache()
|
||||
return 0
|
||||
|
||||
ingested = _ingest_messages_batch(batch)
|
||||
|
||||
with _lock:
|
||||
_bulk_cursor["pages"] = int(_bulk_cursor.get("pages", 0)) + 1
|
||||
_cache["pages_fetched"] = int(_cache.get("pages_fetched", 0)) + 1
|
||||
_save_cache()
|
||||
|
||||
if (
|
||||
batch
|
||||
and len(batch) >= 100
|
||||
and _bulk_cursor.get("pages", 0) < MAX_BULK_PAGES_PER_CYCLE
|
||||
):
|
||||
ids = [int(row["id"]) for row in batch if row.get("id") is not None]
|
||||
if ids:
|
||||
next_before = min(ids)
|
||||
if before_id is None or next_before < before_id:
|
||||
_enqueue_bulk_page(since_iso=item["since_iso"], before_id=next_before)
|
||||
|
||||
return ingested
|
||||
|
||||
|
||||
def _process_one_staggered_tick() -> int:
|
||||
"""Process exactly one queued Airframes API call. Used by the background worker."""
|
||||
if not api_key_configured():
|
||||
return 0
|
||||
|
||||
api_key = os.environ.get("AIRFRAMES_API_KEY", "").strip()
|
||||
with _queue_lock:
|
||||
if not _queue:
|
||||
return 0
|
||||
item = _queue.popleft()
|
||||
|
||||
if item.get("type") == "aircraft":
|
||||
key = _aircraft_queue_key(item)
|
||||
with _queue_lock:
|
||||
_queued_aircraft_keys.discard(key)
|
||||
ingested = _process_aircraft_item(api_key, item)
|
||||
elif item.get("type") == "bulk":
|
||||
ingested = _process_bulk_item(api_key, item)
|
||||
else:
|
||||
ingested = 0
|
||||
|
||||
with _lock:
|
||||
_cache["ticks_processed"] = int(_cache.get("ticks_processed", 0)) + 1
|
||||
if int(_cache.get("ticks_processed", 0)) % 25 == 0:
|
||||
for store_key in ("by_icao", "by_tail", "by_callsign"):
|
||||
_prune_store(_cache[store_key])
|
||||
_save_cache()
|
||||
|
||||
return ingested
|
||||
|
||||
|
||||
def _stagger_worker_loop() -> None:
|
||||
while True:
|
||||
time.sleep(REQUEST_PAUSE_S)
|
||||
try:
|
||||
_process_one_staggered_tick()
|
||||
except Exception as exc:
|
||||
logger.error("Airframes stagger worker tick failed: %s", exc)
|
||||
|
||||
|
||||
def _ensure_stagger_worker() -> None:
|
||||
global _worker_started
|
||||
if _worker_started:
|
||||
return
|
||||
with _worker_guard:
|
||||
if _worker_started:
|
||||
return
|
||||
_worker_started = True
|
||||
threading.Thread(
|
||||
target=_stagger_worker_loop,
|
||||
daemon=True,
|
||||
name="airframes-stagger",
|
||||
).start()
|
||||
logger.info(
|
||||
"Airframes stagger worker started (bulk ingest: 1 call / %.1fs, up to %s msgs/call, refill every %sm)",
|
||||
REQUEST_PAUSE_S,
|
||||
100,
|
||||
SYNC_INTERVAL_MINUTES,
|
||||
)
|
||||
|
||||
|
||||
def sync_airframes_messages(*, force: bool = False) -> dict[str, Any]:
|
||||
"""Queue staggered Airframes fetches — one API call every REQUEST_PAUSE_S."""
|
||||
if not api_key_configured():
|
||||
return {"ok": False, "skipped": True, "reason": "AIRFRAMES_API_KEY not configured"}
|
||||
|
||||
started = _utc_now()
|
||||
_load_cache_if_cold()
|
||||
|
||||
with _lock:
|
||||
_cache.setdefault("by_callsign", {})
|
||||
last_sync_at = _parse_ts(_cache.get("last_sync_at"))
|
||||
if (
|
||||
not force
|
||||
and last_sync_at is not None
|
||||
and started - last_sync_at < timedelta(minutes=SYNC_INTERVAL_MINUTES - 1)
|
||||
):
|
||||
return {"ok": True, "skipped": True, "reason": "sync_interval_not_elapsed"}
|
||||
|
||||
if _cache.get("last_success_at"):
|
||||
since_dt = _parse_ts(_cache.get("last_success_at")) or (
|
||||
started - timedelta(minutes=SYNC_INTERVAL_MINUTES)
|
||||
)
|
||||
since_dt -= timedelta(minutes=2)
|
||||
else:
|
||||
since_dt = started - timedelta(hours=PRIORITY_LOOKBACK_HOURS)
|
||||
since_iso = _iso(since_dt)
|
||||
_cache["last_sync_at"] = _iso(started)
|
||||
_cache["last_error"] = None
|
||||
_save_cache()
|
||||
|
||||
queued = _refill_queue(since_iso=since_iso, force=force)
|
||||
_ensure_stagger_worker()
|
||||
|
||||
with _queue_lock:
|
||||
queue_depth = len(_queue)
|
||||
|
||||
logger.info(
|
||||
"Airframes cycle queued: added=%s depth=%s interval=%.1fs",
|
||||
queued,
|
||||
queue_depth,
|
||||
REQUEST_PAUSE_S,
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"queued": queued,
|
||||
"queue_depth": queue_depth,
|
||||
"request_interval_s": REQUEST_PAUSE_S,
|
||||
"sync_interval_minutes": SYNC_INTERVAL_MINUTES,
|
||||
}
|
||||
|
||||
|
||||
def _lookup_from_cache(
|
||||
*,
|
||||
hex_key: str,
|
||||
tail_keys: list[str],
|
||||
callsign_key: str,
|
||||
) -> tuple[list[dict[str, Any]], str | None]:
|
||||
_load_cache_if_cold()
|
||||
with _lock:
|
||||
_cache.setdefault("by_callsign", {})
|
||||
merged: dict[int, dict[str, Any]] = {}
|
||||
if hex_key:
|
||||
for message in _cache.get("by_icao", {}).get(hex_key, []):
|
||||
merged[message["id"]] = message
|
||||
for tail_key in tail_keys:
|
||||
for message in _cache.get("by_tail", {}).get(tail_key, []):
|
||||
merged[message["id"]] = message
|
||||
if callsign_key:
|
||||
for message in _cache.get("by_callsign", {}).get(callsign_key, []):
|
||||
merged[message["id"]] = message
|
||||
last_success_at = _cache.get("last_success_at")
|
||||
|
||||
messages = sorted(merged.values(), key=lambda item: item.get("timestamp") or "", reverse=True)
|
||||
return messages[:MESSAGES_PER_AIRCRAFT], last_success_at
|
||||
|
||||
|
||||
def get_datalink_status() -> dict[str, Any]:
|
||||
configured = api_key_configured()
|
||||
_load_cache_if_cold()
|
||||
with _queue_lock:
|
||||
queue_depth = len(_queue)
|
||||
with _lock:
|
||||
return {
|
||||
"configured": configured,
|
||||
"sync_interval_minutes": SYNC_INTERVAL_MINUTES,
|
||||
"request_interval_s": REQUEST_PAUSE_S,
|
||||
"last_sync_at": _cache.get("last_sync_at"),
|
||||
"last_success_at": _cache.get("last_success_at"),
|
||||
"last_error": _cache.get("last_error"),
|
||||
"pages_fetched": _cache.get("pages_fetched", 0),
|
||||
"messages_ingested": _cache.get("messages_ingested", 0),
|
||||
"bulk_pages_this_cycle": int(_bulk_cursor.get("pages", 0)),
|
||||
"bulk_pages_per_cycle": MAX_BULK_PAGES_PER_CYCLE,
|
||||
"messages_per_bulk_call": 100,
|
||||
"queue_depth": queue_depth,
|
||||
"ticks_processed": _cache.get("ticks_processed", 0),
|
||||
"icao_keys": len(_cache.get("by_icao", {})),
|
||||
"tail_keys": len(_cache.get("by_tail", {})),
|
||||
"callsign_keys": len(_cache.get("by_callsign", {})),
|
||||
}
|
||||
|
||||
|
||||
def lookup_datalink_messages(
|
||||
*,
|
||||
icao24: str = "",
|
||||
registration: str = "",
|
||||
callsign: str = "",
|
||||
allow_live: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
configured = bool(os.environ.get("AIRFRAMES_API_KEY", "").strip()) or api_key_configured()
|
||||
if not configured:
|
||||
return {
|
||||
"configured": False,
|
||||
"messages": [],
|
||||
"hint": "Add AIRFRAMES_API_KEY in Settings → API Keys to enable ACARS datalink.",
|
||||
}
|
||||
|
||||
hex_key = _norm_hex(icao24)
|
||||
tail_keys = _tail_lookup_keys(registration)
|
||||
callsign_key = _norm_callsign(callsign)
|
||||
|
||||
messages, last_success_at = _lookup_from_cache(
|
||||
hex_key=hex_key,
|
||||
tail_keys=tail_keys,
|
||||
callsign_key=callsign_key,
|
||||
)
|
||||
|
||||
queued_refresh = False
|
||||
if hex_key or tail_keys or callsign_key:
|
||||
queued_refresh = _prioritize_aircraft_scan(
|
||||
{
|
||||
"icao24": hex_key,
|
||||
"registration": _norm_tail(registration),
|
||||
"callsign": callsign_key,
|
||||
}
|
||||
)
|
||||
if queued_refresh:
|
||||
_ensure_stagger_worker()
|
||||
|
||||
from services.fetchers.acars_summarize import prepare_datalink_display
|
||||
|
||||
display = prepare_datalink_display(messages)
|
||||
|
||||
return {
|
||||
"configured": True,
|
||||
"messages": display["messages"],
|
||||
"hidden_count": display["hidden_count"],
|
||||
"total_count": display["total_count"],
|
||||
"last_success_at": last_success_at,
|
||||
"queued_refresh": queued_refresh,
|
||||
"priority_scan": queued_refresh,
|
||||
}
|
||||
|
||||
|
||||
_load_cache_if_cold()
|
||||
@@ -12,9 +12,11 @@ Polling interval deliberately kept low (4h) to be respectful to the service.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
|
||||
@@ -30,9 +32,6 @@ _MAX_AGE_HOURS = 24 # discard nodes not seen within this window
|
||||
# one-person hobby service, so we prefer stale data over hammering it.
|
||||
_CACHE_TRUST_HOURS = 20
|
||||
|
||||
# Track when we last fetched so the frontend can show staleness
|
||||
_last_fetch_ts: float = 0.0
|
||||
|
||||
|
||||
def _parse_node(node: dict) -> dict | None:
|
||||
"""Convert an API node into a slim signal-like dict."""
|
||||
@@ -132,7 +131,43 @@ def _save_cache(nodes: list[dict], fetch_ts: float):
|
||||
logger.warning(f"Failed to save meshtastic cache: {e}")
|
||||
|
||||
|
||||
def fetch_meshtastic_nodes():
|
||||
# Track when we last fetched so the frontend can show staleness
|
||||
_last_fetch_ts: float = 0.0
|
||||
_scan_lock = threading.Lock()
|
||||
_scan_in_progress = False
|
||||
|
||||
|
||||
def get_meshtastic_map_status() -> dict[str, Any]:
|
||||
from services.fetchers._store import get_latest_data_subset_refs
|
||||
|
||||
snap = get_latest_data_subset_refs("meshtastic_map_nodes", "meshtastic_map_fetched_at")
|
||||
nodes = snap.get("meshtastic_map_nodes") or []
|
||||
fetched_at = snap.get("meshtastic_map_fetched_at")
|
||||
return {
|
||||
"node_count": len(nodes) if isinstance(nodes, list) else 0,
|
||||
"fetched_at": fetched_at,
|
||||
"scan_in_progress": _scan_in_progress,
|
||||
}
|
||||
|
||||
|
||||
def start_meshtastic_planet_scan() -> dict[str, Any]:
|
||||
if not _scan_lock.acquire(blocking=False):
|
||||
return {"ok": False, "status": "scan already in progress"}
|
||||
|
||||
def _run() -> None:
|
||||
global _scan_in_progress
|
||||
try:
|
||||
_scan_in_progress = True
|
||||
fetch_meshtastic_nodes(force=True)
|
||||
finally:
|
||||
_scan_in_progress = False
|
||||
_scan_lock.release()
|
||||
|
||||
threading.Thread(target=_run, daemon=True, name="meshtastic-planet-scan").start()
|
||||
return {"ok": True, "status": "scanning"}
|
||||
|
||||
|
||||
def fetch_meshtastic_nodes(*, force: bool = False):
|
||||
"""Fetch global Meshtastic node positions from Liam Cottle's map API.
|
||||
|
||||
Stores processed nodes in latest_data["meshtastic_map_nodes"].
|
||||
@@ -140,39 +175,40 @@ def fetch_meshtastic_nodes():
|
||||
"""
|
||||
from services.fetchers._store import is_any_active
|
||||
|
||||
if not is_any_active("sigint_meshtastic"):
|
||||
if not force and not is_any_active("sigint_meshtastic"):
|
||||
return
|
||||
global _last_fetch_ts
|
||||
|
||||
# Trust a recent cache on disk — avoids hammering the upstream HTTP API
|
||||
# when every install polls on roughly the same cadence.
|
||||
try:
|
||||
if _CACHE_FILE.exists():
|
||||
mtime = _CACHE_FILE.stat().st_mtime
|
||||
if time.time() - mtime < _CACHE_TRUST_HOURS * 3600:
|
||||
# If memory is empty (cold start), hydrate from cache and skip fetch.
|
||||
with _data_lock:
|
||||
has_memory = bool(latest_data.get("meshtastic_map_nodes"))
|
||||
if not has_memory:
|
||||
cached = _load_cache()
|
||||
if cached:
|
||||
with _data_lock:
|
||||
latest_data["meshtastic_map_nodes"] = cached
|
||||
latest_data["meshtastic_map_fetched_at"] = mtime
|
||||
_mark_fresh("meshtastic_map")
|
||||
if not force:
|
||||
try:
|
||||
if _CACHE_FILE.exists():
|
||||
mtime = _CACHE_FILE.stat().st_mtime
|
||||
if time.time() - mtime < _CACHE_TRUST_HOURS * 3600:
|
||||
# If memory is empty (cold start), hydrate from cache and skip fetch.
|
||||
with _data_lock:
|
||||
has_memory = bool(latest_data.get("meshtastic_map_nodes"))
|
||||
if not has_memory:
|
||||
cached = _load_cache()
|
||||
if cached:
|
||||
with _data_lock:
|
||||
latest_data["meshtastic_map_nodes"] = cached
|
||||
latest_data["meshtastic_map_fetched_at"] = mtime
|
||||
_mark_fresh("meshtastic_map")
|
||||
logger.info(
|
||||
"Meshtastic map: cache fresh (<%.0fh), skipping network fetch",
|
||||
_CACHE_TRUST_HOURS,
|
||||
)
|
||||
return
|
||||
else:
|
||||
logger.info(
|
||||
"Meshtastic map: cache fresh (<%.0fh), skipping network fetch",
|
||||
_CACHE_TRUST_HOURS,
|
||||
)
|
||||
return
|
||||
else:
|
||||
logger.info(
|
||||
"Meshtastic map: cache fresh (<%.0fh), skipping network fetch",
|
||||
_CACHE_TRUST_HOURS,
|
||||
)
|
||||
return
|
||||
except Exception as e:
|
||||
logger.debug(f"Meshtastic cache freshness check failed: {e}")
|
||||
except Exception as e:
|
||||
logger.debug(f"Meshtastic cache freshness check failed: {e}")
|
||||
|
||||
# Build a polite User-Agent. Historically this included the operator
|
||||
# callsign so meshtastic.org could rate-limit per-install; that's still
|
||||
|
||||
@@ -49,11 +49,11 @@ _CATEGORY_COLOR: dict[str, str] = {
|
||||
"Head of State": "#ff1493",
|
||||
"Royal Aircraft": "#ff1493",
|
||||
"Don't you know who I am?": "#ff1493",
|
||||
"As Seen on TV": "#ff1493",
|
||||
"Bizjets": "#ff1493",
|
||||
"Vanity Plate": "#ff1493",
|
||||
"Football": "#ff1493",
|
||||
# ORANGE — Joe Cool
|
||||
# ORANGE — corporate / novelty / Joe Cool / As Seen on TV
|
||||
"As Seen on TV": "orange",
|
||||
"Joe Cool": "orange",
|
||||
# WHITE — Climate Crisis
|
||||
"Climate Crisis": "white",
|
||||
@@ -338,6 +338,10 @@ def enrich_with_tracked_names(flight: dict) -> dict:
|
||||
flight["alert_color"] = "blue"
|
||||
elif is_med:
|
||||
flight["alert_color"] = "#32cd32"
|
||||
elif match.get("category") == "Oligarch":
|
||||
flight["alert_color"] = "red"
|
||||
elif match.get("category") in {"Royal", "Celebrity", "People"}:
|
||||
flight["alert_color"] = "#ff1493"
|
||||
elif "alert_color" not in flight:
|
||||
flight["alert_color"] = "pink"
|
||||
|
||||
|
||||
@@ -21,12 +21,21 @@ def _merge_sigint_snapshot(
|
||||
because they include fresher region/channel metadata.
|
||||
"""
|
||||
|
||||
merged = list(live_signals)
|
||||
# Shallow-copy every entry so the published list owns its own dicts. The
|
||||
# inputs alias objects that other threads keep mutating in place: live
|
||||
# signals are the SIGINT bridge's own dicts (updated as packets arrive),
|
||||
# and api_nodes are the same objects published under latest_data
|
||||
# ["meshtastic_map_nodes"]. Publishing those references into
|
||||
# latest_data["sigint"] lets a concurrent mutation race the lock-free
|
||||
# deepcopy in get_latest_data_deepcopy_snapshot() (/api/health, /api/live-
|
||||
# data) and raise "dictionary changed size during iteration". Copying
|
||||
# honors the replace-don't-mutate contract in fetchers/_store.py.
|
||||
merged = [dict(s) for s in live_signals]
|
||||
live_callsigns = {s["callsign"] for s in merged if s.get("source") == "meshtastic"}
|
||||
for node in api_nodes:
|
||||
if node.get("callsign") in live_callsigns:
|
||||
continue
|
||||
merged.append(node)
|
||||
merged.append(dict(node))
|
||||
merged.sort(key=lambda item: str(item.get("timestamp", "") or ""), reverse=True)
|
||||
return merged
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import html
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
@@ -11,6 +12,7 @@ from typing import Any
|
||||
from services.fetchers._store import _data_lock, _mark_fresh, is_any_active, latest_data
|
||||
from services.fetchers.news import resolve_coords_match
|
||||
from services.network_utils import fetch_with_curl, outbound_user_agent
|
||||
from services.telegram_translate import apply_post_translation, apply_posts_translations
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -174,13 +176,7 @@ def _extract_media(block: str, link: str) -> dict[str, Any]:
|
||||
def _strip_html(text: str) -> str:
|
||||
cleaned = re.sub(r"<br\s*/?>", "\n", text, flags=re.IGNORECASE)
|
||||
cleaned = re.sub(r"<[^>]+>", "", cleaned)
|
||||
return (
|
||||
cleaned.replace(""", '"')
|
||||
.replace("&", "&")
|
||||
.replace("<", "<")
|
||||
.replace(">", ">")
|
||||
.strip()
|
||||
)
|
||||
return html.unescape(cleaned).strip()
|
||||
|
||||
|
||||
def _score_risk(text: str) -> int:
|
||||
@@ -293,20 +289,19 @@ def parse_telegram_channel_html(html: str, channel: str) -> list[dict[str, Any]]
|
||||
post_id = hashlib.sha1(f"{link}|{published}".encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
media = _extract_media(block, link)
|
||||
posts.append(
|
||||
{
|
||||
"id": post_id,
|
||||
"title": title,
|
||||
"description": text[:1200],
|
||||
"link": link,
|
||||
"published": published,
|
||||
"source": f"t.me/{channel}",
|
||||
"channel": channel,
|
||||
"risk_score": risk_score,
|
||||
"coords": [coords[0], coords[1]] if coords else None,
|
||||
**media,
|
||||
}
|
||||
)
|
||||
post = {
|
||||
"id": post_id,
|
||||
"title": title,
|
||||
"description": text[:1200],
|
||||
"link": link,
|
||||
"published": published,
|
||||
"source": f"t.me/{channel}",
|
||||
"channel": channel,
|
||||
"risk_score": risk_score,
|
||||
"coords": [coords[0], coords[1]] if coords else None,
|
||||
**media,
|
||||
}
|
||||
posts.append(apply_post_translation(post))
|
||||
return posts
|
||||
|
||||
|
||||
@@ -358,6 +353,7 @@ def fetch_telegram_osint() -> dict[str, Any]:
|
||||
|
||||
merged_posts, added = _merge_telegram_posts(existing_posts, incoming)
|
||||
merged_posts = [_refresh_post_coords(post) for post in merged_posts]
|
||||
merged_posts = apply_posts_translations(merged_posts)
|
||||
geolocated = sum(1 for p in merged_posts if p.get("coords"))
|
||||
|
||||
payload = {
|
||||
|
||||
@@ -606,8 +606,19 @@ def _build_feature_html(features, fetched_titles=None):
|
||||
|
||||
|
||||
def _enrich_gdelt_titles_background(features, all_article_urls):
|
||||
"""Background thread: fetch real article titles then update features in-place."""
|
||||
"""Background thread: fetch real article titles, then publish enriched COPIES.
|
||||
|
||||
The ``features`` handed to us were already published into
|
||||
``latest_data["gdelt"]`` by ``fetch_gdelt()``. Per the store's thread-safety
|
||||
contract (see ``get_latest_data_subset_refs`` in fetchers/_store.py), HTTP
|
||||
readers hold live references to these nested ``properties`` dicts and
|
||||
serialize them OUTSIDE the data lock. Mutating the published dicts in place
|
||||
here races that serialization and raises
|
||||
``RuntimeError: dictionary changed size during iteration``. So we enrich
|
||||
copies and atomically swap the top-level key under the lock instead.
|
||||
"""
|
||||
import html as html_mod
|
||||
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
|
||||
|
||||
try:
|
||||
logger.info(f"[BG] Fetching real article titles for {len(all_article_urls)} URLs...")
|
||||
@@ -615,28 +626,44 @@ def _enrich_gdelt_titles_background(features, all_article_urls):
|
||||
fetched_count = sum(1 for v in fetched_titles.values() if v)
|
||||
logger.info(f"[BG] Resolved {fetched_count}/{len(all_article_urls)} article titles")
|
||||
|
||||
# Update features in-place with real titles and snippets
|
||||
# Build enriched copies — never touch the already-published objects.
|
||||
enriched = []
|
||||
for f in features:
|
||||
urls = f["properties"].get("_urls_list", [])
|
||||
if not urls:
|
||||
continue
|
||||
headlines = []
|
||||
snippets = []
|
||||
for u in urls:
|
||||
real_title = fetched_titles.get(u)
|
||||
headlines.append(real_title if real_title else _url_to_headline(u))
|
||||
snippets.append(_article_snippet_cache.get(u) or "")
|
||||
f["properties"]["_headlines_list"] = headlines
|
||||
f["properties"]["_snippets_list"] = snippets
|
||||
links = []
|
||||
for u, h in zip(urls, headlines):
|
||||
safe_url = u if u.startswith(("http://", "https://")) else "about:blank"
|
||||
safe_h = html_mod.escape(h)
|
||||
links.append(
|
||||
f'<div style="margin-bottom:6px;"><a href="{safe_url}" target="_blank" rel="noopener noreferrer">{safe_h}</a></div>'
|
||||
)
|
||||
f["properties"]["html"] = "".join(links)
|
||||
logger.info(f"[BG] GDELT title enrichment complete")
|
||||
nf = dict(f)
|
||||
props = dict(f.get("properties", {}))
|
||||
urls = props.get("_urls_list", [])
|
||||
if urls:
|
||||
headlines = []
|
||||
snippets = []
|
||||
for u in urls:
|
||||
real_title = fetched_titles.get(u)
|
||||
headlines.append(real_title if real_title else _url_to_headline(u))
|
||||
snippets.append(_article_snippet_cache.get(u) or "")
|
||||
props["_headlines_list"] = headlines
|
||||
props["_snippets_list"] = snippets
|
||||
links = []
|
||||
for u, h in zip(urls, headlines):
|
||||
safe_url = u if u.startswith(("http://", "https://")) else "about:blank"
|
||||
safe_h = html_mod.escape(h)
|
||||
links.append(
|
||||
f'<div style="margin-bottom:6px;"><a href="{safe_url}" target="_blank" rel="noopener noreferrer">{safe_h}</a></div>'
|
||||
)
|
||||
props["html"] = "".join(links)
|
||||
nf["properties"] = props
|
||||
enriched.append(nf)
|
||||
|
||||
# Atomically publish — but only if a newer fetch_gdelt() hasn't already
|
||||
# replaced the layer while we were fetching titles (identity guard).
|
||||
published = False
|
||||
with _data_lock:
|
||||
if latest_data.get("gdelt") is features:
|
||||
latest_data["gdelt"] = enriched
|
||||
published = True
|
||||
if published:
|
||||
_mark_fresh("gdelt")
|
||||
logger.info(f"[BG] GDELT title enrichment complete ({len(enriched)} features)")
|
||||
else:
|
||||
logger.info("[BG] GDELT layer changed under us; skipping stale enrichment swap")
|
||||
except Exception as e:
|
||||
logger.error(f"[BG] GDELT title enrichment failed: {e}")
|
||||
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Immediate data refresh when the operator enables a map layer.
|
||||
|
||||
Disk/local fetches run inline (milliseconds). Network-heavy fetches run on the
|
||||
slow executor so POST /api/layers never blocks the single uvicorn worker for
|
||||
tens of seconds (which freezes bootstrap + live-data and makes the map go black).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Inline — local DB / static files only.
|
||||
_INSTANT_LAYER_KEYS: frozenset[str] = frozenset(
|
||||
{"cctv", "power_plants", "datacenters"}
|
||||
)
|
||||
# Background — network-bound; may take seconds.
|
||||
_SLOW_LAYER_KEYS: frozenset[str] = frozenset(
|
||||
{"firms", "psk_reporter", "fishing_activity"}
|
||||
)
|
||||
|
||||
|
||||
def snapshot_active_layers() -> dict[str, bool]:
|
||||
from services.fetchers._store import active_layers
|
||||
|
||||
return dict(active_layers)
|
||||
|
||||
|
||||
def _was_off_now_on(before: dict[str, bool], key: str) -> bool:
|
||||
from services.fetchers._store import active_layers
|
||||
|
||||
return not bool(before.get(key, False)) and bool(active_layers.get(key, False))
|
||||
|
||||
|
||||
def _instant_fetch(key: str) -> None:
|
||||
if key == "cctv":
|
||||
from services.fetchers.infrastructure import fetch_cctv
|
||||
|
||||
fetch_cctv()
|
||||
logger.info("CCTV loaded (layer enabled)")
|
||||
return
|
||||
if key == "power_plants":
|
||||
from services.fetchers.infrastructure import fetch_power_plants
|
||||
|
||||
fetch_power_plants()
|
||||
logger.info("Power plants loaded (layer enabled)")
|
||||
return
|
||||
if key == "datacenters":
|
||||
from services.fetchers.infrastructure import fetch_datacenters
|
||||
|
||||
fetch_datacenters()
|
||||
logger.info("Datacenters loaded (layer enabled)")
|
||||
return
|
||||
raise KeyError(key)
|
||||
|
||||
|
||||
def _slow_fetch(key: str) -> None:
|
||||
if key == "firms":
|
||||
from services.fetchers.earth_observation import (
|
||||
fetch_firms_country_fires,
|
||||
fetch_firms_fires,
|
||||
)
|
||||
|
||||
fetch_firms_fires()
|
||||
fetch_firms_country_fires()
|
||||
logger.info("FIRMS fires loaded (layer enabled)")
|
||||
return
|
||||
if key == "psk_reporter":
|
||||
from services.fetchers.infrastructure import fetch_psk_reporter
|
||||
|
||||
fetch_psk_reporter()
|
||||
logger.info("PSK Reporter loaded (layer enabled)")
|
||||
return
|
||||
if key == "fishing_activity":
|
||||
from services.fetchers.geo import fetch_fishing_activity
|
||||
|
||||
fetch_fishing_activity()
|
||||
logger.info("Fishing activity loaded (layer enabled)")
|
||||
return
|
||||
raise KeyError(key)
|
||||
|
||||
|
||||
def _run_slow_enable_fetches(keys: tuple[str, ...]) -> None:
|
||||
from services.fetchers._store import bump_data_version
|
||||
|
||||
for key in keys:
|
||||
try:
|
||||
_slow_fetch(key)
|
||||
except Exception:
|
||||
logger.exception("Layer enable fetch failed for %s", key)
|
||||
bump_data_version()
|
||||
|
||||
|
||||
def refresh_newly_enabled_layers(before: dict[str, bool]) -> None:
|
||||
"""Fetch any layers that transitioned off → on."""
|
||||
from services.fetchers._store import bump_data_version
|
||||
|
||||
instant_keys: list[str] = []
|
||||
slow_keys: list[str] = []
|
||||
|
||||
for key in _INSTANT_LAYER_KEYS | _SLOW_LAYER_KEYS:
|
||||
if _was_off_now_on(before, key):
|
||||
if key in _INSTANT_LAYER_KEYS:
|
||||
instant_keys.append(key)
|
||||
else:
|
||||
slow_keys.append(key)
|
||||
|
||||
if not instant_keys and not slow_keys:
|
||||
return
|
||||
|
||||
for key in instant_keys:
|
||||
try:
|
||||
_instant_fetch(key)
|
||||
except Exception:
|
||||
logger.exception("Layer enable fetch failed for %s", key)
|
||||
|
||||
if instant_keys:
|
||||
bump_data_version()
|
||||
|
||||
if slow_keys:
|
||||
from services.data_fetcher import _SLOW_EXECUTOR
|
||||
|
||||
_SLOW_EXECUTOR.submit(_run_slow_enable_fetches, tuple(slow_keys))
|
||||
@@ -287,28 +287,18 @@ def write_signed_bootstrap_manifest(
|
||||
return manifest
|
||||
|
||||
|
||||
def load_bootstrap_manifest(
|
||||
path: str | Path,
|
||||
def parse_bootstrap_manifest_dict(
|
||||
raw: dict[str, Any],
|
||||
*,
|
||||
signer_public_key_b64: str,
|
||||
now: float | None = None,
|
||||
) -> BootstrapManifest:
|
||||
manifest_path = _resolve_manifest_path(str(path))
|
||||
try:
|
||||
raw = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError as exc:
|
||||
raise BootstrapManifestError(f"bootstrap manifest not found: {manifest_path}") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise BootstrapManifestError("bootstrap manifest is not valid JSON") from exc
|
||||
|
||||
if not isinstance(raw, dict):
|
||||
raise BootstrapManifestError("bootstrap manifest root must be an object")
|
||||
|
||||
signature = str(raw.get("signature", "") or "").strip()
|
||||
payload = {key: value for key, value in raw.items() if key != "signature"}
|
||||
if not signature:
|
||||
raise BootstrapManifestError("bootstrap manifest signature is required")
|
||||
|
||||
_verify_manifest_signature(
|
||||
payload,
|
||||
signature_b64=signature,
|
||||
@@ -325,11 +315,36 @@ def load_bootstrap_manifest(
|
||||
)
|
||||
|
||||
|
||||
def load_bootstrap_manifest(
|
||||
path: str | Path,
|
||||
*,
|
||||
signer_public_key_b64: str,
|
||||
now: float | None = None,
|
||||
) -> BootstrapManifest:
|
||||
manifest_path = _resolve_manifest_path(str(path))
|
||||
try:
|
||||
raw = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError as exc:
|
||||
raise BootstrapManifestError(f"bootstrap manifest not found: {manifest_path}") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise BootstrapManifestError("bootstrap manifest is not valid JSON") from exc
|
||||
|
||||
if not isinstance(raw, dict):
|
||||
raise BootstrapManifestError("bootstrap manifest root must be an object")
|
||||
return parse_bootstrap_manifest_dict(
|
||||
raw,
|
||||
signer_public_key_b64=signer_public_key_b64,
|
||||
now=now,
|
||||
)
|
||||
|
||||
|
||||
def load_bootstrap_manifest_from_settings(*, now: float | None = None) -> BootstrapManifest | None:
|
||||
settings = get_settings()
|
||||
if bool(getattr(settings, "MESH_BOOTSTRAP_DISABLED", False)):
|
||||
return None
|
||||
signer_public_key_b64 = str(getattr(settings, "MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "") or "").strip()
|
||||
from services.mesh.mesh_fleet_defaults import effective_bootstrap_signer_public_key_b64
|
||||
|
||||
signer_public_key_b64 = effective_bootstrap_signer_public_key_b64()
|
||||
if not signer_public_key_b64:
|
||||
return None
|
||||
manifest_path = _resolve_manifest_path(str(getattr(settings, "MESH_BOOTSTRAP_MANIFEST_PATH", "") or ""))
|
||||
|
||||
@@ -168,9 +168,9 @@ def resolve_peer_key_for_url(peer_url: str) -> bytes:
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
global_secret = str(
|
||||
getattr(get_settings(), "MESH_PEER_PUSH_SECRET", "") or ""
|
||||
).strip()
|
||||
from services.mesh.mesh_fleet_defaults import effective_peer_push_secret
|
||||
|
||||
global_secret = effective_peer_push_secret()
|
||||
except Exception:
|
||||
return b""
|
||||
if not global_secret:
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
"""Invite-scoped DM connect delivery: auto relay release and contact severance."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
CONNECT_AUTO_RELEASE_INTENTS = frozenset(
|
||||
{
|
||||
"invite_short_address",
|
||||
"invite_import",
|
||||
"contact_request",
|
||||
"contact_accept",
|
||||
"contact_offer",
|
||||
}
|
||||
)
|
||||
|
||||
INVITE_CONNECT_TRUST_LEVELS = frozenset({"invite_pinned", "sas_verified"})
|
||||
|
||||
|
||||
def _release_profile() -> str:
|
||||
try:
|
||||
from services.release_profiles import current_release_profile
|
||||
|
||||
return str(current_release_profile() or "dev")
|
||||
except Exception:
|
||||
return "dev"
|
||||
|
||||
|
||||
def grant_connect_relay_policy(
|
||||
recipient_id: str,
|
||||
*,
|
||||
reason: str = "connect_scoped_auto_release",
|
||||
) -> dict[str, Any]:
|
||||
"""Pre-authorize hidden relay delivery for an explicit connect target."""
|
||||
peer_key = str(recipient_id or "").strip()
|
||||
if not peer_key:
|
||||
return {"ok": False, "detail": "recipient_id required"}
|
||||
try:
|
||||
from services.mesh.mesh_relay_policy import grant_relay_policy
|
||||
|
||||
return grant_relay_policy(
|
||||
scope_type="dm_contact",
|
||||
scope_id=peer_key,
|
||||
profile=_release_profile(),
|
||||
hidden_transport_required=True,
|
||||
reason=str(reason or "connect_scoped_auto_release"),
|
||||
)
|
||||
except Exception as exc:
|
||||
return {"ok": False, "detail": str(exc) or type(exc).__name__}
|
||||
|
||||
|
||||
def revoke_connect_relay_policy(recipient_id: str) -> dict[str, Any]:
|
||||
peer_key = str(recipient_id or "").strip()
|
||||
if not peer_key:
|
||||
return {"ok": False, "detail": "recipient_id required"}
|
||||
try:
|
||||
from services.mesh.mesh_relay_policy import revoke_relay_policy
|
||||
|
||||
revoked = int(
|
||||
revoke_relay_policy(
|
||||
scope_type="dm_contact",
|
||||
scope_id=peer_key,
|
||||
profile=_release_profile(),
|
||||
)
|
||||
or 0
|
||||
)
|
||||
return {"ok": True, "revoked": revoked}
|
||||
except Exception as exc:
|
||||
return {"ok": False, "detail": str(exc) or type(exc).__name__}
|
||||
|
||||
|
||||
def recipient_has_invite_connect_scope(recipient_id: str) -> bool:
|
||||
peer_key = str(recipient_id or "").strip()
|
||||
if not peer_key:
|
||||
return False
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_contacts import get_wormhole_dm_contact
|
||||
|
||||
contact = get_wormhole_dm_contact(peer_key) or {}
|
||||
except Exception:
|
||||
return False
|
||||
if str(contact.get("invitePinnedPrekeyLookupHandle", "") or "").strip():
|
||||
return True
|
||||
if str(contact.get("invitePinnedLookupPeerUrl", "") or "").strip():
|
||||
return True
|
||||
trust = str(contact.get("trust_level", "") or "").strip().lower()
|
||||
return trust in INVITE_CONNECT_TRUST_LEVELS
|
||||
|
||||
|
||||
def relay_push_peer_urls_for_payload(payload: dict[str, Any]) -> list[str]:
|
||||
urls: list[str] = []
|
||||
for raw in list(payload.get("relay_push_peer_urls") or []):
|
||||
normalized = str(raw or "").strip().rstrip("/")
|
||||
if normalized and normalized not in urls:
|
||||
urls.append(normalized)
|
||||
lookup_peer_url = str(payload.get("lookup_peer_url", "") or "").strip().rstrip("/")
|
||||
if lookup_peer_url:
|
||||
urls = [url for url in urls if url != lookup_peer_url]
|
||||
urls.insert(0, lookup_peer_url)
|
||||
recipient_id = str(payload.get("recipient_id", "") or "").strip()
|
||||
if recipient_id and not urls:
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_contacts import get_wormhole_dm_contact
|
||||
|
||||
contact = get_wormhole_dm_contact(recipient_id) or {}
|
||||
pinned = str(contact.get("invitePinnedLookupPeerUrl", "") or "").strip().rstrip("/")
|
||||
if pinned:
|
||||
urls.append(pinned)
|
||||
except Exception:
|
||||
pass
|
||||
return urls
|
||||
|
||||
|
||||
def should_auto_release_dm_payload(payload: dict[str, Any]) -> bool:
|
||||
if str(payload.get("delivery_class", "") or "").strip().lower() != "request":
|
||||
return False
|
||||
intent = str(payload.get("connect_intent", "") or "").strip().lower()
|
||||
if intent in CONNECT_AUTO_RELEASE_INTENTS:
|
||||
return True
|
||||
if str(payload.get("lookup_peer_url", "") or "").strip():
|
||||
return True
|
||||
recipient_id = str(payload.get("recipient_id", "") or "").strip()
|
||||
return bool(recipient_id and recipient_has_invite_connect_scope(recipient_id))
|
||||
|
||||
|
||||
def enrich_connect_release_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Attach invite-owner relay hints used during private release."""
|
||||
enriched = dict(payload or {})
|
||||
recipient_id = str(enriched.get("recipient_id", "") or "").strip()
|
||||
lookup_peer_url = str(enriched.get("lookup_peer_url", "") or "").strip().rstrip("/")
|
||||
if not lookup_peer_url and recipient_id:
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_contacts import get_wormhole_dm_contact
|
||||
|
||||
contact = get_wormhole_dm_contact(recipient_id) or {}
|
||||
lookup_peer_url = str(contact.get("invitePinnedLookupPeerUrl", "") or "").strip().rstrip("/")
|
||||
except Exception:
|
||||
lookup_peer_url = ""
|
||||
if lookup_peer_url:
|
||||
enriched["lookup_peer_url"] = lookup_peer_url
|
||||
push_urls = relay_push_peer_urls_for_payload(enriched)
|
||||
if push_urls:
|
||||
enriched["relay_push_peer_urls"] = push_urls
|
||||
return enriched
|
||||
|
||||
|
||||
def auto_release_connect_dm_outbox(*, outbox_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Grant scoped relay policy and approve release for invite-scoped connect traffic."""
|
||||
normalized_outbox = str(outbox_id or "").strip()
|
||||
enriched = enrich_connect_release_payload(payload)
|
||||
if not normalized_outbox:
|
||||
return {"ok": False, "detail": "missing outbox_id"}
|
||||
if not should_auto_release_dm_payload(enriched):
|
||||
return {"ok": True, "skipped": True, "reason": "not_connect_scoped"}
|
||||
recipient_id = str(enriched.get("recipient_id", "") or "").strip()
|
||||
if not recipient_id:
|
||||
return {"ok": False, "detail": "missing recipient_id"}
|
||||
grant = grant_connect_relay_policy(recipient_id)
|
||||
try:
|
||||
from services.mesh.mesh_private_outbox import private_delivery_outbox
|
||||
from services.mesh.mesh_private_release_worker import private_release_worker
|
||||
|
||||
private_delivery_outbox.approve_relay_release(normalized_outbox)
|
||||
private_release_worker.ensure_started()
|
||||
private_release_worker.wake()
|
||||
except Exception as exc:
|
||||
return {
|
||||
"ok": False,
|
||||
"detail": str(exc) or type(exc).__name__,
|
||||
"grant": grant,
|
||||
}
|
||||
return {
|
||||
"ok": True,
|
||||
"auto_released": True,
|
||||
"outbox_id": normalized_outbox,
|
||||
"recipient_id": recipient_id,
|
||||
"grant": grant,
|
||||
"relay_push_peer_urls": relay_push_peer_urls_for_payload(enriched),
|
||||
}
|
||||
@@ -1506,6 +1506,7 @@ class DMRelay:
|
||||
sender_token_hash: str = "",
|
||||
payload_format: str = "dm1",
|
||||
session_welcome: str = "",
|
||||
replication_peer_urls: list[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
self._refresh_from_shared_relay()
|
||||
@@ -1573,46 +1574,214 @@ class DMRelay:
|
||||
}
|
||||
if not msg_id:
|
||||
msg_id = f"dm_{int(time.time() * 1000)}_{secrets.token_hex(6)}"
|
||||
elif any(m.msg_id == msg_id for m in self._mailboxes[mailbox_key]):
|
||||
return {"ok": True, "msg_id": msg_id}
|
||||
relay_sender_id = (
|
||||
f"sender_token:{sender_token_hash}"
|
||||
if sender_token_hash
|
||||
else sender_id
|
||||
)
|
||||
self._mailboxes[mailbox_key].append(
|
||||
DMMessage(
|
||||
sender_id=relay_sender_id,
|
||||
ciphertext=ciphertext,
|
||||
timestamp=time.time(),
|
||||
msg_id=msg_id,
|
||||
delivery_class=delivery_class,
|
||||
sender_seal=sender_seal,
|
||||
sender_block_ref=sender_block_ref,
|
||||
payload_format=str(payload_format or "dm1"),
|
||||
session_welcome=str(session_welcome or ""),
|
||||
duplicate_hit = any(m.msg_id == msg_id for m in self._mailboxes[mailbox_key])
|
||||
if not duplicate_hit:
|
||||
relay_sender_id = (
|
||||
f"sender_token:{sender_token_hash}"
|
||||
if sender_token_hash
|
||||
else sender_id
|
||||
)
|
||||
)
|
||||
self._stats["messages_in_memory"] = sum(len(v) for v in self._mailboxes.values())
|
||||
self._save()
|
||||
# Cross-node mailbox replication: push the freshly-stored
|
||||
# envelope to every authenticated relay peer so the recipient
|
||||
# can log into ANY node and find their messages. The push is
|
||||
# async (fire-and-forget thread) so deposit() returns
|
||||
# immediately — slow Tor peers can't block the sender's UX.
|
||||
# Each receiving peer re-enforces the per-sender cap on
|
||||
# acceptance, so hostile relays can't widen the cap.
|
||||
self._mailboxes[mailbox_key].append(
|
||||
DMMessage(
|
||||
sender_id=relay_sender_id,
|
||||
ciphertext=ciphertext,
|
||||
timestamp=time.time(),
|
||||
msg_id=msg_id,
|
||||
delivery_class=delivery_class,
|
||||
sender_seal=sender_seal,
|
||||
sender_block_ref=sender_block_ref,
|
||||
payload_format=str(payload_format or "dm1"),
|
||||
session_welcome=str(session_welcome or ""),
|
||||
)
|
||||
)
|
||||
self._stats["messages_in_memory"] = sum(len(v) for v in self._mailboxes.values())
|
||||
self._save()
|
||||
preferred_urls = list(replication_peer_urls or [])
|
||||
envelope_for_push: dict[str, Any] | None = None
|
||||
try:
|
||||
envelope_for_push = self.envelope_for_replication(
|
||||
mailbox_key=mailbox_key, msg_id=msg_id,
|
||||
mailbox_key=mailbox_key,
|
||||
msg_id=msg_id,
|
||||
recipient_id=recipient_id,
|
||||
recipient_token=recipient_token,
|
||||
)
|
||||
if envelope_for_push:
|
||||
self._replicate_envelope_to_peers_async(
|
||||
envelope=envelope_for_push,
|
||||
)
|
||||
except Exception:
|
||||
metrics_inc("dm_replication_push_error")
|
||||
return {"ok": True, "msg_id": msg_id}
|
||||
deposit_result = {"ok": True, "msg_id": msg_id}
|
||||
if duplicate_hit:
|
||||
deposit_result["duplicate"] = True
|
||||
|
||||
if envelope_for_push:
|
||||
# Invite-scoped connect traffic names an explicit recipient relay
|
||||
# (lookup_peer_url). Block until that push completes so the
|
||||
# recipient can poll their own node; fleet-wide fan-out stays
|
||||
# async so dead manifest peers cannot wedge deposit().
|
||||
if preferred_urls:
|
||||
logger.info(
|
||||
"DM deposit awaiting scoped replicate to %d peer(s)",
|
||||
len(preferred_urls),
|
||||
)
|
||||
deposit_result["replicate"] = self._replicate_envelope_to_peers(
|
||||
envelope=envelope_for_push,
|
||||
preferred_peer_urls=preferred_urls,
|
||||
)
|
||||
else:
|
||||
self._replicate_envelope_to_peers_async(
|
||||
envelope=envelope_for_push,
|
||||
preferred_peer_urls=[],
|
||||
)
|
||||
elif preferred_urls:
|
||||
logger.warning(
|
||||
"DM deposit skipped scoped replicate: envelope missing for msg_id=%s",
|
||||
msg_id,
|
||||
)
|
||||
return deposit_result
|
||||
|
||||
def _replicate_envelope_to_peers(
|
||||
self,
|
||||
*,
|
||||
envelope: dict[str, Any],
|
||||
preferred_peer_urls: list[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Push an envelope to relay peers. Returns per-peer results."""
|
||||
import hashlib
|
||||
import hmac
|
||||
import requests as _requests
|
||||
|
||||
from services.mesh.mesh_crypto import (
|
||||
normalize_peer_url,
|
||||
resolve_peer_key_for_url,
|
||||
)
|
||||
from services.mesh.mesh_router import authenticated_push_peer_urls
|
||||
|
||||
peers: list[str] = []
|
||||
for raw_url in list(preferred_peer_urls or []):
|
||||
normalized_preferred = normalize_peer_url(str(raw_url or "").strip())
|
||||
if normalized_preferred and normalized_preferred not in peers:
|
||||
peers.append(normalized_preferred)
|
||||
if not peers:
|
||||
for peer_url in authenticated_push_peer_urls():
|
||||
normalized_peer = normalize_peer_url(str(peer_url or "").strip())
|
||||
if normalized_peer and normalized_peer not in peers:
|
||||
peers.append(normalized_peer)
|
||||
if not peers:
|
||||
return {"ok": False, "detail": "no_relay_peers", "pushed": [], "failed": []}
|
||||
|
||||
logger.info(
|
||||
"DM replicate push starting for %d peer(s): %s",
|
||||
len(peers),
|
||||
", ".join(peers[:3]) + ("..." if len(peers) > 3 else ""),
|
||||
)
|
||||
|
||||
payload = json.dumps(
|
||||
{"envelope": envelope},
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=False,
|
||||
).encode("utf-8")
|
||||
|
||||
base_timeout = max(
|
||||
1,
|
||||
int(getattr(self._settings(), "MESH_RELAY_PUSH_TIMEOUT_S", 10) or 10),
|
||||
)
|
||||
|
||||
from main import _infonet_peer_requests_proxies
|
||||
|
||||
preferred_set = {
|
||||
normalize_peer_url(str(raw_url or "").strip())
|
||||
for raw_url in list(preferred_peer_urls or [])
|
||||
}
|
||||
preferred_set.discard("")
|
||||
|
||||
pushed: list[str] = []
|
||||
failed: list[dict[str, str]] = []
|
||||
for peer_url in peers:
|
||||
try:
|
||||
normalized = normalize_peer_url(peer_url)
|
||||
timeout = max(180 if ".onion" in normalized else 1, base_timeout)
|
||||
headers = {"Content-Type": "application/json"}
|
||||
peer_key = resolve_peer_key_for_url(normalized)
|
||||
if peer_key:
|
||||
headers["X-Peer-Url"] = normalized
|
||||
headers["X-Peer-HMAC"] = hmac.new(
|
||||
peer_key, payload, hashlib.sha256
|
||||
).hexdigest()
|
||||
url = f"{peer_url}/api/mesh/dm/replicate-envelope"
|
||||
request_kwargs: dict[str, Any] = {
|
||||
"data": payload,
|
||||
"timeout": timeout,
|
||||
"headers": headers,
|
||||
}
|
||||
proxies = _infonet_peer_requests_proxies(normalized)
|
||||
if proxies:
|
||||
request_kwargs["proxies"] = proxies
|
||||
resp = None
|
||||
max_attempts = 3 if normalized in preferred_set else 2
|
||||
last_exc = ""
|
||||
for attempt in range(max_attempts):
|
||||
try:
|
||||
resp = _requests.post(url, **request_kwargs)
|
||||
break
|
||||
except Exception as exc:
|
||||
last_exc = str(exc) or type(exc).__name__
|
||||
if attempt + 1 < max_attempts:
|
||||
time.sleep(5.0 * (attempt + 1))
|
||||
continue
|
||||
logger.warning(
|
||||
"DM replicate push to %s failed: %s",
|
||||
peer_url,
|
||||
last_exc,
|
||||
)
|
||||
metrics_inc("dm_replication_push_error")
|
||||
resp = None
|
||||
break
|
||||
if resp is None:
|
||||
failed.append({"url": peer_url, "detail": last_exc or "request_failed"})
|
||||
continue
|
||||
if resp.status_code == 200:
|
||||
body_ok = True
|
||||
detail = ""
|
||||
try:
|
||||
body = resp.json()
|
||||
if isinstance(body, dict) and body.get("ok") is False:
|
||||
body_ok = False
|
||||
detail = str(body.get("detail", "") or "replicate rejected")[:200]
|
||||
except Exception:
|
||||
body_ok = True
|
||||
if body_ok:
|
||||
logger.info("DM replicate push to %s succeeded", peer_url)
|
||||
metrics_inc("dm_replication_push_ok")
|
||||
pushed.append(peer_url)
|
||||
else:
|
||||
logger.warning(
|
||||
"DM replicate push to %s rejected: %s",
|
||||
peer_url,
|
||||
detail,
|
||||
)
|
||||
metrics_inc("dm_replication_push_rejected")
|
||||
failed.append({"url": peer_url, "detail": detail or "replicate_rejected"})
|
||||
else:
|
||||
detail = (resp.text or "")[:200]
|
||||
logger.warning(
|
||||
"DM replicate push to %s -> %s: %s",
|
||||
peer_url,
|
||||
resp.status_code,
|
||||
detail,
|
||||
)
|
||||
metrics_inc("dm_replication_push_rejected")
|
||||
failed.append({"url": peer_url, "detail": f"http_{resp.status_code}: {detail}"})
|
||||
except Exception as exc:
|
||||
logger.warning("DM replicate push outer failure for %s: %s", peer_url, exc)
|
||||
metrics_inc("dm_replication_push_error")
|
||||
failed.append({"url": peer_url, "detail": str(exc) or type(exc).__name__})
|
||||
|
||||
scoped = bool(preferred_set)
|
||||
ok = bool(pushed) if scoped else bool(pushed) or not failed
|
||||
return {
|
||||
"ok": ok,
|
||||
"scoped": scoped,
|
||||
"pushed": pushed,
|
||||
"failed": failed,
|
||||
}
|
||||
|
||||
def accept_replica(
|
||||
self,
|
||||
@@ -1645,6 +1814,33 @@ class DMRelay:
|
||||
mailbox_key = str(envelope.get("mailbox_key", "") or "").strip()
|
||||
sender_block_ref = str(envelope.get("sender_block_ref", "") or "").strip()
|
||||
ciphertext = str(envelope.get("ciphertext", "") or "")
|
||||
delivery_class = str(envelope.get("delivery_class", "") or "").strip().lower()
|
||||
recipient_id = str(envelope.get("recipient_id", "") or "").strip()
|
||||
recipient_token = str(envelope.get("recipient_token", "") or "").strip()
|
||||
if delivery_class not in ("request", "shared", "self"):
|
||||
if recipient_id and not recipient_token:
|
||||
delivery_class = "request"
|
||||
elif recipient_token:
|
||||
delivery_class = "shared"
|
||||
if delivery_class == "request":
|
||||
if not recipient_id:
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_persona import get_dm_identity
|
||||
|
||||
recipient_id = str((get_dm_identity() or {}).get("node_id") or "").strip()
|
||||
except Exception:
|
||||
recipient_id = ""
|
||||
if recipient_id:
|
||||
mailbox_key = self.mailbox_key_for_delivery(
|
||||
recipient_id=recipient_id,
|
||||
delivery_class="request",
|
||||
)
|
||||
elif delivery_class == "shared" and recipient_token:
|
||||
mailbox_key = self.mailbox_key_for_delivery(
|
||||
recipient_id=recipient_id,
|
||||
delivery_class="shared",
|
||||
recipient_token=recipient_token,
|
||||
)
|
||||
if not msg_id or not mailbox_key or not sender_block_ref or not ciphertext:
|
||||
return {"ok": False, "detail": "envelope missing required fields"}
|
||||
|
||||
@@ -1662,7 +1858,6 @@ class DMRelay:
|
||||
# Same per-class cap as the deposit path — defense in depth
|
||||
# against a peer that wraps a "deposit" as a "replica" to
|
||||
# bypass the class limit.
|
||||
delivery_class = str(envelope.get("delivery_class", "") or "")
|
||||
if delivery_class in ("request", "shared", "self"):
|
||||
class_limit = self._mailbox_limit_for_class(delivery_class)
|
||||
else:
|
||||
@@ -1716,82 +1911,18 @@ class DMRelay:
|
||||
self,
|
||||
*,
|
||||
envelope: dict[str, Any],
|
||||
preferred_peer_urls: list[str] | None = None,
|
||||
) -> None:
|
||||
"""Push an outbound DM envelope to every authenticated relay peer.
|
||||
|
||||
Fire-and-forget: spawned in a background thread so ``deposit``
|
||||
returns to the caller immediately. Per-peer errors are logged
|
||||
and swallowed — the sender's UX must not block on slow Tor
|
||||
peers, and a peer that's down today gets the next message
|
||||
whenever it comes back. Inbound recipient polling from a healthy
|
||||
peer keeps the system functional during peer failures.
|
||||
|
||||
Each peer is authed with the existing per-peer HMAC pattern
|
||||
(#256) — same headers and key resolver gate-message replication
|
||||
uses, so a hostile node that doesn't know any peer's HMAC key
|
||||
can't impersonate a legitimate relay.
|
||||
"""
|
||||
"""Fire-and-forget fleet-wide replicate push (non-scoped traffic)."""
|
||||
import threading
|
||||
|
||||
def _do_push():
|
||||
def _do_push() -> None:
|
||||
try:
|
||||
import hashlib
|
||||
import hmac
|
||||
import requests as _requests
|
||||
|
||||
from services.mesh.mesh_crypto import (
|
||||
normalize_peer_url,
|
||||
resolve_peer_key_for_url,
|
||||
self._replicate_envelope_to_peers(
|
||||
envelope=envelope,
|
||||
preferred_peer_urls=preferred_peer_urls,
|
||||
)
|
||||
from services.mesh.mesh_router import (
|
||||
authenticated_push_peer_urls,
|
||||
)
|
||||
|
||||
peers = authenticated_push_peer_urls()
|
||||
if not peers:
|
||||
return
|
||||
|
||||
payload = json.dumps(
|
||||
{"envelope": envelope},
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=False,
|
||||
).encode("utf-8")
|
||||
|
||||
timeout = max(
|
||||
1,
|
||||
int(getattr(self._settings(), "MESH_RELAY_PUSH_TIMEOUT_S", 10) or 10),
|
||||
)
|
||||
|
||||
for peer_url in peers:
|
||||
try:
|
||||
normalized = normalize_peer_url(peer_url)
|
||||
headers = {"Content-Type": "application/json"}
|
||||
peer_key = resolve_peer_key_for_url(normalized)
|
||||
if peer_key:
|
||||
headers["X-Peer-Url"] = normalized
|
||||
headers["X-Peer-HMAC"] = hmac.new(
|
||||
peer_key, payload, hashlib.sha256
|
||||
).hexdigest()
|
||||
url = f"{peer_url}/api/mesh/dm/replicate-envelope"
|
||||
resp = _requests.post(
|
||||
url, data=payload, timeout=timeout, headers=headers,
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
metrics_inc("dm_replication_push_ok")
|
||||
else:
|
||||
# 4xx including the structured cap_violation
|
||||
# rejection from accept_replica — sender's
|
||||
# relay learns and stops retrying this msg_id.
|
||||
metrics_inc("dm_replication_push_rejected")
|
||||
except Exception:
|
||||
# Per-peer failure is non-fatal — log to metrics
|
||||
# but don't break the loop. Other peers and a
|
||||
# future retry can still propagate the envelope.
|
||||
metrics_inc("dm_replication_push_error")
|
||||
continue
|
||||
except Exception:
|
||||
# Outer guard — never let replication errors propagate
|
||||
# back to the sender's deposit() caller.
|
||||
metrics_inc("dm_replication_push_error")
|
||||
|
||||
thread = threading.Thread(
|
||||
@@ -1806,6 +1937,8 @@ class DMRelay:
|
||||
*,
|
||||
mailbox_key: str,
|
||||
msg_id: str,
|
||||
recipient_id: str = "",
|
||||
recipient_token: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Return the wire-form envelope for a stored message, suitable
|
||||
for POSTing to a peer relay's replicate-envelope endpoint.
|
||||
@@ -1822,6 +1955,8 @@ class DMRelay:
|
||||
return {
|
||||
"msg_id": m.msg_id,
|
||||
"mailbox_key": mailbox_key,
|
||||
"recipient_id": str(recipient_id or "").strip(),
|
||||
"recipient_token": str(recipient_token or "").strip(),
|
||||
"sender_id": m.sender_id,
|
||||
"sender_block_ref": m.sender_block_ref,
|
||||
"sender_seal": m.sender_seal,
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Public Infonet fleet defaults for sb-testnet-0 participants.
|
||||
|
||||
Operators who run private single-node installs can set ``MESH_INFONET_FLEET_JOIN=false``
|
||||
and provide their own signer keys / peer secrets.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
FLEET_NETWORK_ID = "sb-testnet-0"
|
||||
FLEET_SEED_ONION_URL = (
|
||||
"http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"
|
||||
)
|
||||
FLEET_BOOTSTRAP_SIGNER_PUBLIC_KEY_B64 = (
|
||||
"ul1d0kj/ODPIp0OhHzX8eLAVXzJ3CVvzW1vn2IC6q3I="
|
||||
)
|
||||
# Shared fleet HMAC for sb-testnet peer announce/push/sync. Public testnet join model.
|
||||
FLEET_PEER_PUSH_SECRET = "b7GoqsvoUD9MV7tyt0ZOzMptLA84QG6KCfaV9nDqz5Y"
|
||||
|
||||
|
||||
def infonet_fleet_join_enabled() -> bool:
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
if bool(getattr(get_settings(), "MESH_INFONET_FLEET_JOIN_DISABLED", False)):
|
||||
return False
|
||||
return bool(getattr(get_settings(), "MESH_INFONET_FLEET_JOIN", True))
|
||||
except Exception:
|
||||
return True
|
||||
|
||||
|
||||
def effective_bootstrap_signer_public_key_b64() -> str:
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
configured = str(getattr(get_settings(), "MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "") or "").strip()
|
||||
if configured:
|
||||
return configured
|
||||
except Exception:
|
||||
pass
|
||||
if infonet_fleet_join_enabled():
|
||||
return FLEET_BOOTSTRAP_SIGNER_PUBLIC_KEY_B64
|
||||
return ""
|
||||
|
||||
|
||||
def effective_peer_push_secret() -> str:
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
configured = str(getattr(get_settings(), "MESH_PEER_PUSH_SECRET", "") or "").strip()
|
||||
if configured:
|
||||
return configured
|
||||
except Exception:
|
||||
pass
|
||||
if infonet_fleet_join_enabled():
|
||||
return FLEET_PEER_PUSH_SECRET
|
||||
return ""
|
||||
|
||||
|
||||
def configured_bootstrap_seed_peers_with_fleet_default(peers: list[str]) -> list[str]:
|
||||
if peers:
|
||||
return peers
|
||||
if infonet_fleet_join_enabled():
|
||||
return [FLEET_SEED_ONION_URL]
|
||||
return []
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Auto-enable Tor wormhole transport on Infonet relay/seed nodes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from services.config import get_settings
|
||||
from services.wormhole_settings import read_wormhole_settings, write_wormhole_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def infonet_relay_auto_wormhole_requested() -> bool:
|
||||
settings = get_settings()
|
||||
if bool(settings.MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED):
|
||||
return False
|
||||
if bool(settings.MESH_INFONET_RELAY_AUTO_WORMHOLE):
|
||||
return True
|
||||
if str(settings.MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY or "").strip():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _relay_tor_wormhole_target_settings() -> dict[str, Any]:
|
||||
settings = get_settings()
|
||||
socks_port = int(settings.MESH_ARTI_SOCKS_PORT or 9050)
|
||||
return {
|
||||
"enabled": True,
|
||||
"transport": "tor_arti",
|
||||
"socks_proxy": f"socks5h://127.0.0.1:{socks_port}",
|
||||
"socks_dns": True,
|
||||
"anonymous_mode": True,
|
||||
}
|
||||
|
||||
|
||||
def _wormhole_settings_match(existing: dict[str, Any], target: dict[str, Any]) -> bool:
|
||||
return (
|
||||
bool(existing.get("enabled")) is bool(target["enabled"])
|
||||
and str(existing.get("transport", "")) == str(target["transport"])
|
||||
and str(existing.get("socks_proxy", "")) == str(target["socks_proxy"])
|
||||
and bool(existing.get("socks_dns", True)) is bool(target["socks_dns"])
|
||||
and bool(existing.get("anonymous_mode", False)) is bool(target["anonymous_mode"])
|
||||
)
|
||||
|
||||
|
||||
def ensure_infonet_relay_wormhole_ready(*, reason: str = "relay_auto") -> dict[str, Any]:
|
||||
"""Persist Tor wormhole settings and connect on relay/seed startup."""
|
||||
if not infonet_relay_auto_wormhole_requested():
|
||||
return {"ok": True, "skipped": True, "reason": "not_requested"}
|
||||
|
||||
from routers.ai_intel import _write_env_value
|
||||
from services.tor_hidden_service import tor_service
|
||||
from services.wormhole_supervisor import connect_wormhole, restart_wormhole
|
||||
|
||||
existing = read_wormhole_settings()
|
||||
target = _relay_tor_wormhole_target_settings()
|
||||
settings_updated = not _wormhole_settings_match(existing, target)
|
||||
updated = write_wormhole_settings(**target) if settings_updated else existing
|
||||
|
||||
tor_result: dict[str, Any] = {"ok": False, "detail": "not started"}
|
||||
try:
|
||||
tor_result = tor_service.start(target_port=8000)
|
||||
if tor_result.get("ok"):
|
||||
_write_env_value("MESH_ARTI_ENABLED", "true")
|
||||
get_settings.cache_clear()
|
||||
except Exception as exc:
|
||||
tor_result = {"ok": False, "detail": str(exc or type(exc).__name__)}
|
||||
|
||||
runtime = (
|
||||
restart_wormhole(reason=reason)
|
||||
if settings_updated
|
||||
else connect_wormhole(reason=reason)
|
||||
)
|
||||
|
||||
if settings_updated:
|
||||
logger.info("Infonet relay auto-wormhole enabled (%s)", reason)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"skipped": False,
|
||||
"settings_updated": settings_updated,
|
||||
"tor": tor_result,
|
||||
"runtime": runtime,
|
||||
"settings": updated,
|
||||
}
|
||||
@@ -125,8 +125,8 @@ def dm_lookup_response_view(
|
||||
view.pop("lookup_mode", None)
|
||||
view.pop("removal_target", None)
|
||||
return view
|
||||
if invite_lookup:
|
||||
view.pop("agent_id", None)
|
||||
# Successful invite lookups keep agent_id: the handle is the capability and
|
||||
# first-contact messaging needs a delivery target. Failures stay generic.
|
||||
return view
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
"""Operator-signed peer registry for private Infonet swarm discovery."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from services.mesh.mesh_crypto import normalize_peer_url
|
||||
from services.mesh.mesh_router import peer_transport_kind
|
||||
|
||||
BACKEND_DIR = Path(__file__).resolve().parents[2]
|
||||
DATA_DIR = BACKEND_DIR / "data"
|
||||
DEFAULT_PEER_REGISTRY_PATH = DATA_DIR / "peer_registry.json"
|
||||
REGISTRY_VERSION = 1
|
||||
ALLOWED_REGISTRY_ROLES = {"participant", "relay", "seed"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class RegistryPeer:
|
||||
peer_url: str
|
||||
transport: str
|
||||
role: str
|
||||
node_id: str = ""
|
||||
label: str = ""
|
||||
announced_at: int = 0
|
||||
last_seen_at: int = 0
|
||||
failure_count: int = 0
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
def manifest_peer(self) -> dict[str, str]:
|
||||
return {
|
||||
"peer_url": self.peer_url,
|
||||
"transport": self.transport,
|
||||
"role": self.role,
|
||||
"label": self.label or self.node_id[:16],
|
||||
}
|
||||
|
||||
|
||||
class PeerRegistry:
|
||||
def __init__(self, path: str | Path = DEFAULT_PEER_REGISTRY_PATH):
|
||||
self.path = Path(path)
|
||||
self._peers: dict[str, RegistryPeer] = {}
|
||||
|
||||
def load(self) -> list[RegistryPeer]:
|
||||
if not self.path.exists():
|
||||
self._peers = {}
|
||||
return []
|
||||
raw = json.loads(self.path.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError("peer registry root must be an object")
|
||||
version = int(raw.get("version", 0) or 0)
|
||||
if version != REGISTRY_VERSION:
|
||||
raise ValueError(f"unsupported peer registry version: {version}")
|
||||
entries = raw.get("peers", [])
|
||||
if not isinstance(entries, list):
|
||||
raise ValueError("peer registry peers must be a list")
|
||||
peers: dict[str, RegistryPeer] = {}
|
||||
for entry in entries:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
peer = self._normalize_entry(entry)
|
||||
peers[peer.peer_url] = peer
|
||||
self._peers = peers
|
||||
return self.records()
|
||||
|
||||
def save(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = {
|
||||
"version": REGISTRY_VERSION,
|
||||
"updated_at": int(time.time()),
|
||||
"peers": [peer.to_dict() for peer in self.records()],
|
||||
}
|
||||
self.path.write_text(
|
||||
json.dumps(payload, sort_keys=True, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
def records(self) -> list[RegistryPeer]:
|
||||
return sorted(self._peers.values(), key=lambda item: (item.role, item.peer_url))
|
||||
|
||||
def upsert_announcement(
|
||||
self,
|
||||
*,
|
||||
peer_url: str,
|
||||
transport: str,
|
||||
role: str,
|
||||
node_id: str = "",
|
||||
label: str = "",
|
||||
now: float | None = None,
|
||||
) -> RegistryPeer:
|
||||
normalized = normalize_peer_url(peer_url)
|
||||
if not normalized:
|
||||
raise ValueError("peer_url is required")
|
||||
resolved_transport = str(transport or "").strip().lower() or str(peer_transport_kind(normalized) or "")
|
||||
if resolved_transport not in {"onion", "clearnet"}:
|
||||
raise ValueError("unsupported peer transport")
|
||||
resolved_role = str(role or "participant").strip().lower()
|
||||
if resolved_role not in ALLOWED_REGISTRY_ROLES:
|
||||
raise ValueError("unsupported peer role")
|
||||
timestamp = int(now if now is not None else time.time())
|
||||
existing = self._peers.get(normalized)
|
||||
peer = RegistryPeer(
|
||||
peer_url=normalized,
|
||||
transport=resolved_transport,
|
||||
role=resolved_role,
|
||||
node_id=str(node_id or (existing.node_id if existing else "") or "").strip(),
|
||||
label=str(label or (existing.label if existing else "") or "").strip(),
|
||||
announced_at=int(existing.announced_at if existing and existing.announced_at else timestamp),
|
||||
last_seen_at=timestamp,
|
||||
failure_count=int(existing.failure_count if existing else 0),
|
||||
)
|
||||
self._peers[normalized] = peer
|
||||
return peer
|
||||
|
||||
def prune_stale(self, *, max_age_s: int, now: float | None = None) -> int:
|
||||
timestamp = int(now if now is not None else time.time())
|
||||
removed = 0
|
||||
for peer_url, peer in list(self._peers.items()):
|
||||
if peer.role == "seed":
|
||||
continue
|
||||
last_seen = int(peer.last_seen_at or peer.announced_at or 0)
|
||||
if last_seen > 0 and timestamp - last_seen > max(60, int(max_age_s or 0)):
|
||||
del self._peers[peer_url]
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
def manifest_peers(self) -> list[dict[str, str]]:
|
||||
return [peer.manifest_peer() for peer in self.records()]
|
||||
|
||||
def _normalize_entry(self, entry: dict[str, Any]) -> RegistryPeer:
|
||||
peer_url = normalize_peer_url(str(entry.get("peer_url", "") or ""))
|
||||
if not peer_url:
|
||||
raise ValueError("registry peer_url is required")
|
||||
transport = str(entry.get("transport", "") or peer_transport_kind(peer_url) or "").strip().lower()
|
||||
role = str(entry.get("role", "participant") or "participant").strip().lower()
|
||||
if role not in ALLOWED_REGISTRY_ROLES:
|
||||
raise ValueError("registry role unsupported")
|
||||
return RegistryPeer(
|
||||
peer_url=peer_url,
|
||||
transport=transport,
|
||||
role=role,
|
||||
node_id=str(entry.get("node_id", "") or "").strip(),
|
||||
label=str(entry.get("label", "") or "").strip(),
|
||||
announced_at=int(entry.get("announced_at", 0) or 0),
|
||||
last_seen_at=int(entry.get("last_seen_at", 0) or entry.get("announced_at", 0) or 0),
|
||||
failure_count=int(entry.get("failure_count", 0) or 0),
|
||||
)
|
||||
@@ -16,7 +16,7 @@ DATA_DIR = BACKEND_DIR / "data"
|
||||
DEFAULT_PEER_STORE_PATH = DATA_DIR / "peer_store.json"
|
||||
PEER_STORE_VERSION = 1
|
||||
ALLOWED_PEER_BUCKETS = {"bootstrap", "sync", "push"}
|
||||
ALLOWED_PEER_SOURCES = {"bundle", "operator", "bootstrap_promoted", "runtime"}
|
||||
ALLOWED_PEER_SOURCES = {"bundle", "operator", "bootstrap_promoted", "runtime", "swarm"}
|
||||
ALLOWED_PEER_TRANSPORTS = {"clearnet", "onion"}
|
||||
ALLOWED_PEER_ROLES = {"participant", "relay", "seed"}
|
||||
|
||||
|
||||
@@ -140,10 +140,24 @@ def transport_tier_from_state(state: dict[str, Any] | None) -> str:
|
||||
snapshot = state or {}
|
||||
if not bool(snapshot.get("configured")):
|
||||
return "public_degraded"
|
||||
if not bool(snapshot.get("ready")):
|
||||
return "public_degraded"
|
||||
arti_ready = bool(snapshot.get("arti_ready"))
|
||||
rns_ready = bool(snapshot.get("rns_ready"))
|
||||
running = bool(snapshot.get("running"))
|
||||
transport_usable = bool(snapshot.get("ready"))
|
||||
if not transport_usable:
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
if (
|
||||
bool(getattr(get_settings(), "MESH_WORMHOLE_TRUST_FILE_READY", False))
|
||||
and running
|
||||
and arti_ready
|
||||
):
|
||||
transport_usable = True
|
||||
except Exception:
|
||||
pass
|
||||
if not transport_usable:
|
||||
return "public_degraded"
|
||||
if arti_ready and rns_ready:
|
||||
return "private_strong"
|
||||
if arti_ready or rns_ready:
|
||||
@@ -157,8 +171,45 @@ def transport_tier_is_sufficient(current_tier: str | None, required_tier: str |
|
||||
return TRANSPORT_TIER_ORDER[current] >= TRANSPORT_TIER_ORDER[required]
|
||||
|
||||
|
||||
def release_lane_required_tier(lane: str) -> str:
|
||||
return network_release_required_tier(lane)
|
||||
_DM_RUNTIME_ENFORCEMENT_ROUTES = {
|
||||
("POST", "/api/mesh/dm/send"),
|
||||
("POST", "/api/mesh/dm/poll"),
|
||||
("GET", "/api/mesh/dm/poll"),
|
||||
("GET", "/api/mesh/dm/count"),
|
||||
("POST", "/api/mesh/dm/count"),
|
||||
}
|
||||
|
||||
|
||||
def runtime_route_enforcement_tier(path: str, method: str, *, static_tier: str) -> str:
|
||||
"""Adjust static route tiers for Tor-only nodes that never reach private_strong."""
|
||||
normalized_path = str(path or "").strip()
|
||||
normalized_method = str(method or "").strip().upper()
|
||||
static = normalize_transport_tier(static_tier)
|
||||
if (normalized_method, normalized_path) not in _DM_RUNTIME_ENFORCEMENT_ROUTES:
|
||||
return static
|
||||
if static != "private_strong":
|
||||
return static
|
||||
return release_lane_required_tier("dm")
|
||||
|
||||
|
||||
def release_lane_required_tier(lane: str, *, wormhole_state: dict[str, Any] | None = None) -> str:
|
||||
normalized_lane = str(lane or "").strip().lower()
|
||||
required = network_release_required_tier(normalized_lane)
|
||||
if normalized_lane != "dm":
|
||||
return required
|
||||
state = wormhole_state
|
||||
if state is None:
|
||||
try:
|
||||
from services.wormhole_supervisor import get_wormhole_state
|
||||
|
||||
state = get_wormhole_state()
|
||||
except Exception:
|
||||
state = {}
|
||||
# Tor-only nodes never reach private_strong (needs Arti + RNS). Encrypted
|
||||
# relay over Arti still preserves ciphertext privacy for offline delivery.
|
||||
if not bool((state or {}).get("rns_enabled")):
|
||||
return "private_transitional"
|
||||
return required
|
||||
|
||||
|
||||
def private_delivery_status(status_code: str, *, reason_code: str = "", plain_reason: str = "") -> dict[str, str]:
|
||||
|
||||
@@ -386,6 +386,20 @@ def _dispatch_dm(
|
||||
sampled=sampled,
|
||||
)
|
||||
|
||||
replication_peer_urls: list[str] = []
|
||||
try:
|
||||
from services.mesh.mesh_dm_connect_delivery import relay_push_peer_urls_for_payload
|
||||
|
||||
replication_peer_urls = [
|
||||
str(raw or "").strip().rstrip("/")
|
||||
for raw in list(payload.get("relay_push_peer_urls") or [])
|
||||
if str(raw or "").strip()
|
||||
]
|
||||
if not replication_peer_urls:
|
||||
replication_peer_urls = relay_push_peer_urls_for_payload(payload)
|
||||
except Exception:
|
||||
replication_peer_urls = []
|
||||
|
||||
apply_dm_relay_jitter()
|
||||
relay_result = dm_relay.deposit(
|
||||
sender_id=relay_sender_id,
|
||||
@@ -399,7 +413,25 @@ def _dispatch_dm(
|
||||
sender_token_hash=sender_token_hash,
|
||||
payload_format=payload_format,
|
||||
session_welcome=session_welcome,
|
||||
replication_peer_urls=replication_peer_urls,
|
||||
)
|
||||
replicate_info = dict(relay_result.get("replicate") or {})
|
||||
if replication_peer_urls and not replicate_info.get("ok"):
|
||||
return _dispatch_result(
|
||||
ok=False,
|
||||
lane="dm",
|
||||
selected_transport="relay",
|
||||
selected_carrier="relay",
|
||||
dispatch_reason="scoped_relay_replicate_failed",
|
||||
hidden_transport_effective=bool(hidden_relay),
|
||||
no_acceptable_path=False,
|
||||
detail=(
|
||||
"Scoped relay replicate did not reach the recipient node: "
|
||||
+ str(replicate_info.get("failed") or replicate_info.get("detail") or "unknown")
|
||||
),
|
||||
msg_id=msg_id,
|
||||
replicate=replicate_info,
|
||||
)
|
||||
if not relay_result.get("ok"):
|
||||
return _dispatch_result(
|
||||
ok=False,
|
||||
@@ -436,6 +468,7 @@ def _dispatch_dm(
|
||||
else str(relay_result.get("detail", "") or "Delivered privately")
|
||||
),
|
||||
msg_id=str(relay_result.get("msg_id", "") or msg_id),
|
||||
replicate=replicate_info,
|
||||
)
|
||||
|
||||
|
||||
@@ -600,8 +633,15 @@ def attempt_private_release(
|
||||
policy_reason_code=str(decision.reason_code or ""),
|
||||
)
|
||||
if normalized_lane == "dm":
|
||||
dm_payload = dict(payload or {})
|
||||
try:
|
||||
from services.mesh.mesh_dm_connect_delivery import enrich_connect_release_payload
|
||||
|
||||
dm_payload = enrich_connect_release_payload(dm_payload)
|
||||
except Exception:
|
||||
pass
|
||||
return _dispatch_dm(
|
||||
dict(payload or {}),
|
||||
dm_payload,
|
||||
secure_dm_enabled=secure_dm_enabled or _secure_dm_enabled,
|
||||
rns_private_dm_ready=rns_private_dm_ready or _rns_private_dm_ready,
|
||||
anonymous_dm_hidden_transport_enforced=(
|
||||
|
||||
@@ -36,6 +36,22 @@ def _require_fields(payload: dict[str, Any], fields: tuple[str, ...]) -> tuple[b
|
||||
return True, "ok"
|
||||
|
||||
|
||||
_SEALED_CIPHERTEXT_PREFIXES = ("x3dh1:", "dm1:", "mls1:", "sealed:")
|
||||
|
||||
|
||||
def _strip_sealed_ciphertext_prefix(value: str) -> str:
|
||||
lowered = value.lower()
|
||||
for prefix in _SEALED_CIPHERTEXT_PREFIXES:
|
||||
if lowered.startswith(prefix):
|
||||
return value[len(prefix) :]
|
||||
return value
|
||||
|
||||
|
||||
def _sealed_ciphertext_has_known_prefix(value: str) -> bool:
|
||||
lowered = str(value or "").strip().lower()
|
||||
return any(lowered.startswith(prefix) for prefix in _SEALED_CIPHERTEXT_PREFIXES)
|
||||
|
||||
|
||||
def _decode_base64ish(value: Any) -> bytes | None:
|
||||
raw = str(value or "").strip()
|
||||
if not raw or any(ch.isspace() for ch in raw):
|
||||
@@ -49,6 +65,13 @@ def _decode_base64ish(value: Any) -> bytes | None:
|
||||
return None
|
||||
|
||||
|
||||
def _decode_sealed_ciphertext_value(value: Any) -> bytes | None:
|
||||
raw = str(value or "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
return _decode_base64ish(_strip_sealed_ciphertext_prefix(raw))
|
||||
|
||||
|
||||
def _byte_entropy(data: bytes) -> float:
|
||||
if not data:
|
||||
return 0.0
|
||||
@@ -66,12 +89,19 @@ def _validate_sealed_bytes_field(
|
||||
min_bytes: int = 8,
|
||||
entropy_floor: float = 2.5,
|
||||
) -> tuple[bool, str]:
|
||||
data = _decode_base64ish(payload.get(field, ""))
|
||||
raw = str(payload.get(field, "") or "").strip()
|
||||
prefixed = _sealed_ciphertext_has_known_prefix(raw)
|
||||
data = _decode_sealed_ciphertext_value(raw)
|
||||
if data is None:
|
||||
return False, f"{field} must be base64-encoded sealed bytes"
|
||||
if len(data) < min_bytes:
|
||||
return False, f"{field} is too short"
|
||||
|
||||
# X3DH / MLS envelopes are structured JSON or ratchet frames — skip
|
||||
# plaintext heuristics once a known wire prefix is present.
|
||||
if prefixed:
|
||||
return True, "ok"
|
||||
|
||||
# Short test vectors and compact envelopes can be low entropy; only apply
|
||||
# heuristics once there is enough material to distinguish a sealed blob
|
||||
# from accidental base64-encoded plaintext.
|
||||
|
||||
@@ -463,8 +463,26 @@ def _apply_content_private_transport_lock_policy(prepared: "PreparedSignedWrite"
|
||||
except Exception:
|
||||
current_tier = "public_degraded"
|
||||
|
||||
lock_to_satisfy = normalized
|
||||
if prepared.kind in {
|
||||
SignedWriteKind.DM_POLL,
|
||||
SignedWriteKind.DM_COUNT,
|
||||
SignedWriteKind.DM_SEND,
|
||||
SignedWriteKind.DM_REGISTER,
|
||||
SignedWriteKind.DM_BLOCK,
|
||||
SignedWriteKind.DM_WITNESS,
|
||||
}:
|
||||
from services.mesh.mesh_privacy_policy import release_lane_required_tier
|
||||
|
||||
lane_cap = release_lane_required_tier("dm")
|
||||
# Clients sign private_strong; Tor-only nodes cap DM at
|
||||
# private_transitional. Accept when live transport meets the
|
||||
# strongest tier this node can offer on the DM lane.
|
||||
if not transport_tier_is_sufficient(lane_cap, normalized):
|
||||
lock_to_satisfy = lane_cap
|
||||
|
||||
if (
|
||||
not transport_tier_is_sufficient(current_tier, normalized)
|
||||
not transport_tier_is_sufficient(current_tier, lock_to_satisfy)
|
||||
and prepared.kind not in _QUEUEABLE_CONTENT_PRIVATE_KINDS
|
||||
):
|
||||
metrics_inc("signed_write_transport_lock_tier_mismatch")
|
||||
|
||||
@@ -0,0 +1,507 @@
|
||||
"""Private Infonet swarm discovery and immediate ledger propagation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from services.config import get_settings
|
||||
from services.mesh.mesh_bootstrap_manifest import (
|
||||
BootstrapManifest,
|
||||
BootstrapManifestError,
|
||||
BootstrapPeer,
|
||||
build_bootstrap_manifest_payload,
|
||||
load_bootstrap_manifest,
|
||||
parse_bootstrap_manifest_dict,
|
||||
sign_bootstrap_manifest_payload,
|
||||
write_signed_bootstrap_manifest,
|
||||
)
|
||||
from services.mesh.mesh_crypto import normalize_peer_url, resolve_peer_key_for_url
|
||||
from services.mesh.mesh_peer_registry import DEFAULT_PEER_REGISTRY_PATH, PeerRegistry, RegistryPeer
|
||||
from services.mesh.mesh_peer_store import (
|
||||
DEFAULT_PEER_STORE_PATH,
|
||||
PeerStore,
|
||||
make_push_peer_record,
|
||||
make_sync_peer_record,
|
||||
)
|
||||
from services.mesh.mesh_router import parse_configured_relay_peers, peer_transport_kind
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SWARM_LOCK = threading.Lock()
|
||||
_LAST_MANIFEST_PULL_AT = 0.0
|
||||
_LAST_ANNOUNCE_AT = 0.0
|
||||
|
||||
|
||||
def peer_registry_enabled() -> bool:
|
||||
settings = get_settings()
|
||||
if bool(getattr(settings, "MESH_PEER_REGISTRY_DISABLED", False)):
|
||||
return False
|
||||
if str(getattr(settings, "MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY", "") or "").strip():
|
||||
return True
|
||||
return bool(getattr(settings, "MESH_PEER_REGISTRY_ENABLED", False))
|
||||
|
||||
|
||||
def _manifest_path() -> str:
|
||||
return str(getattr(get_settings(), "MESH_BOOTSTRAP_MANIFEST_PATH", "") or "data/bootstrap_peers.json")
|
||||
|
||||
|
||||
def _signer_public_key_b64() -> str:
|
||||
from services.mesh.mesh_fleet_defaults import effective_bootstrap_signer_public_key_b64
|
||||
|
||||
return effective_bootstrap_signer_public_key_b64()
|
||||
|
||||
|
||||
def _signer_private_key_b64() -> str:
|
||||
return str(getattr(settings, "MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY", "") or "").strip() if (settings := get_settings()) else ""
|
||||
|
||||
|
||||
def _signer_id() -> str:
|
||||
configured = str(getattr(get_settings(), "MESH_BOOTSTRAP_SIGNER_ID", "") or "").strip()
|
||||
return configured or "shadowbroker-seed"
|
||||
|
||||
|
||||
def _private_transport_required() -> bool:
|
||||
return not bool(getattr(get_settings(), "MESH_INFONET_ALLOW_CLEARNET_SYNC", False))
|
||||
|
||||
|
||||
def _configured_seed_peer_urls() -> list[str]:
|
||||
from services.mesh.mesh_fleet_defaults import configured_bootstrap_seed_peers_with_fleet_default
|
||||
|
||||
settings = get_settings()
|
||||
primary = str(getattr(settings, "MESH_BOOTSTRAP_SEED_PEERS", "") or "").strip()
|
||||
legacy = str(getattr(settings, "MESH_DEFAULT_SYNC_PEERS", "") or "").strip()
|
||||
return configured_bootstrap_seed_peers_with_fleet_default(
|
||||
parse_configured_relay_peers(primary or legacy)
|
||||
)
|
||||
|
||||
|
||||
def _seed_manifest_peers() -> list[dict[str, str]]:
|
||||
peers: list[dict[str, str]] = []
|
||||
for peer_url in _configured_seed_peer_urls():
|
||||
transport = str(peer_transport_kind(peer_url) or "")
|
||||
if _private_transport_required() and transport != "onion":
|
||||
continue
|
||||
peers.append(
|
||||
{
|
||||
"peer_url": peer_url,
|
||||
"transport": transport,
|
||||
"role": "seed",
|
||||
"label": "ShadowBroker bootstrap seed",
|
||||
}
|
||||
)
|
||||
return peers
|
||||
|
||||
|
||||
def publish_registry_manifest(*, now: float | None = None, persist: bool = True) -> BootstrapManifest:
|
||||
private_key = _signer_private_key_b64()
|
||||
public_key = _signer_public_key_b64()
|
||||
if not private_key or not public_key:
|
||||
raise BootstrapManifestError("bootstrap signer keys are required to publish swarm manifest")
|
||||
|
||||
timestamp = int(now if now is not None else time.time())
|
||||
registry = PeerRegistry(DEFAULT_PEER_REGISTRY_PATH)
|
||||
try:
|
||||
registry.load()
|
||||
except Exception:
|
||||
registry = PeerRegistry(DEFAULT_PEER_REGISTRY_PATH)
|
||||
stale_s = int(getattr(get_settings(), "MESH_PEER_REGISTRY_STALE_S", 0) or 7 * 86400)
|
||||
if stale_s > 0:
|
||||
registry.prune_stale(max_age_s=stale_s, now=timestamp)
|
||||
|
||||
peers = _seed_manifest_peers() + registry.manifest_peers()
|
||||
ttl_s = int(getattr(get_settings(), "MESH_SWARM_MANIFEST_TTL_S", 0) or 4 * 3600)
|
||||
payload = build_bootstrap_manifest_payload(
|
||||
signer_id=_signer_id(),
|
||||
peers=peers,
|
||||
issued_at=timestamp,
|
||||
valid_until=timestamp + max(300, ttl_s),
|
||||
)
|
||||
signature = sign_bootstrap_manifest_payload(payload, signer_private_key_b64=private_key)
|
||||
manifest = BootstrapManifest(
|
||||
version=int(payload["version"]),
|
||||
issued_at=int(payload["issued_at"]),
|
||||
valid_until=int(payload["valid_until"]),
|
||||
signer_id=str(payload["signer_id"]),
|
||||
peers=tuple(BootstrapPeer(**dict(peer)) for peer in peers),
|
||||
signature=signature,
|
||||
)
|
||||
if persist:
|
||||
registry.save()
|
||||
write_signed_bootstrap_manifest(
|
||||
_manifest_path(),
|
||||
signer_id=manifest.signer_id,
|
||||
signer_private_key_b64=private_key,
|
||||
peers=[peer.to_dict() for peer in manifest.peers],
|
||||
issued_at=manifest.issued_at,
|
||||
valid_until=manifest.valid_until,
|
||||
)
|
||||
return manifest
|
||||
|
||||
|
||||
def load_live_bootstrap_manifest(*, now: float | None = None) -> BootstrapManifest | None:
|
||||
public_key = _signer_public_key_b64()
|
||||
if not public_key:
|
||||
return None
|
||||
if peer_registry_enabled():
|
||||
try:
|
||||
return publish_registry_manifest(now=now, persist=False)
|
||||
except BootstrapManifestError:
|
||||
logger.warning("live registry manifest unavailable", exc_info=True)
|
||||
try:
|
||||
return load_bootstrap_manifest(_manifest_path(), signer_public_key_b64=public_key, now=now)
|
||||
except BootstrapManifestError:
|
||||
return None
|
||||
|
||||
|
||||
def _upsert_swarm_peer_into_store(
|
||||
*,
|
||||
peer_url: str,
|
||||
transport: str,
|
||||
role: str,
|
||||
label: str = "",
|
||||
signer_id: str = "",
|
||||
now: float | None = None,
|
||||
) -> None:
|
||||
timestamp = int(now if now is not None else time.time())
|
||||
if _private_transport_required() and transport != "onion":
|
||||
return
|
||||
store = PeerStore(DEFAULT_PEER_STORE_PATH)
|
||||
try:
|
||||
store.load()
|
||||
except Exception:
|
||||
store = PeerStore(DEFAULT_PEER_STORE_PATH)
|
||||
store.upsert(
|
||||
make_sync_peer_record(
|
||||
peer_url=peer_url,
|
||||
transport=transport,
|
||||
role=role,
|
||||
source="swarm",
|
||||
label=label,
|
||||
signer_id=signer_id,
|
||||
now=timestamp,
|
||||
)
|
||||
)
|
||||
store.upsert(
|
||||
make_push_peer_record(
|
||||
peer_url=peer_url,
|
||||
transport=transport,
|
||||
role=role if role != "seed" else "relay",
|
||||
source="swarm",
|
||||
label=label,
|
||||
now=timestamp,
|
||||
)
|
||||
)
|
||||
store.save()
|
||||
|
||||
|
||||
def record_peer_announcement(body: dict[str, Any], *, now: float | None = None) -> RegistryPeer:
|
||||
if not peer_registry_enabled():
|
||||
raise ValueError("peer registry is not enabled on this node")
|
||||
registry = PeerRegistry(DEFAULT_PEER_REGISTRY_PATH)
|
||||
try:
|
||||
registry.load()
|
||||
except Exception:
|
||||
registry = PeerRegistry(DEFAULT_PEER_REGISTRY_PATH)
|
||||
peer = registry.upsert_announcement(
|
||||
peer_url=str(body.get("peer_url", "") or ""),
|
||||
transport=str(body.get("transport", "") or ""),
|
||||
role=str(body.get("role", "participant") or "participant"),
|
||||
node_id=str(body.get("node_id", "") or ""),
|
||||
label=str(body.get("label", "") or ""),
|
||||
now=now,
|
||||
)
|
||||
registry.save()
|
||||
_upsert_swarm_peer_into_store(
|
||||
peer_url=peer.peer_url,
|
||||
transport=peer.transport,
|
||||
role=peer.role,
|
||||
label=peer.label,
|
||||
signer_id=_signer_id(),
|
||||
now=now,
|
||||
)
|
||||
try:
|
||||
publish_registry_manifest(now=now, persist=True)
|
||||
except Exception:
|
||||
logger.warning("failed to republish swarm manifest after announce", exc_info=True)
|
||||
return peer
|
||||
|
||||
|
||||
def merge_manifest_into_peer_store(manifest: BootstrapManifest, *, now: float | None = None) -> int:
|
||||
timestamp = int(now if now is not None else time.time())
|
||||
merged = 0
|
||||
for peer in manifest.peers:
|
||||
if _private_transport_required() and peer.transport != "onion":
|
||||
continue
|
||||
_upsert_swarm_peer_into_store(
|
||||
peer_url=peer.peer_url,
|
||||
transport=peer.transport,
|
||||
role=peer.role,
|
||||
label=peer.label,
|
||||
signer_id=manifest.signer_id,
|
||||
now=timestamp,
|
||||
)
|
||||
merged += 1
|
||||
return merged
|
||||
|
||||
|
||||
def fetch_remote_bootstrap_manifest(seed_peer_url: str, *, now: float | None = None) -> BootstrapManifest | None:
|
||||
import requests
|
||||
|
||||
public_key = _signer_public_key_b64()
|
||||
if not public_key:
|
||||
return None
|
||||
normalized = normalize_peer_url(seed_peer_url)
|
||||
if not normalized:
|
||||
return None
|
||||
|
||||
from main import _infonet_peer_requests_proxies
|
||||
|
||||
proxies = _infonet_peer_requests_proxies(normalized)
|
||||
timeout = int(getattr(get_settings(), "MESH_SYNC_TIMEOUT_S", 0) or 45)
|
||||
request_kwargs: dict[str, Any] = {"timeout": timeout}
|
||||
if proxies:
|
||||
request_kwargs["proxies"] = proxies
|
||||
try:
|
||||
response = requests.get(f"{normalized}/api/mesh/infonet/bootstrap-manifest", **request_kwargs)
|
||||
except Exception as exc:
|
||||
logger.debug("swarm manifest fetch failed for %s: %s", normalized, exc)
|
||||
return None
|
||||
if response.status_code != 200:
|
||||
return None
|
||||
try:
|
||||
raw = response.json()
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(raw, dict) or raw.get("ok") is False:
|
||||
return None
|
||||
manifest_body = dict(raw.get("manifest") or raw)
|
||||
try:
|
||||
return parse_bootstrap_manifest_dict(
|
||||
manifest_body,
|
||||
signer_public_key_b64=public_key,
|
||||
now=now,
|
||||
)
|
||||
except BootstrapManifestError:
|
||||
return None
|
||||
|
||||
|
||||
def refresh_swarm_manifest_from_seeds(*, now: float | None = None, force: bool = False) -> dict[str, Any]:
|
||||
global _LAST_MANIFEST_PULL_AT
|
||||
interval_s = int(getattr(get_settings(), "MESH_SWARM_MANIFEST_PULL_INTERVAL_S", 0) or 300)
|
||||
timestamp = float(now if now is not None else time.time())
|
||||
with _SWARM_LOCK:
|
||||
if not force and _LAST_MANIFEST_PULL_AT and timestamp - _LAST_MANIFEST_PULL_AT < max(30, interval_s):
|
||||
return {"ok": True, "skipped": True, "reason": "manifest_pull_interval"}
|
||||
_LAST_MANIFEST_PULL_AT = timestamp
|
||||
|
||||
if not _signer_public_key_b64():
|
||||
return {"ok": False, "detail": "MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY is not configured"}
|
||||
|
||||
last_error = "manifest fetch failed"
|
||||
for seed_url in _configured_seed_peer_urls():
|
||||
manifest = fetch_remote_bootstrap_manifest(seed_url, now=timestamp)
|
||||
if manifest is None:
|
||||
continue
|
||||
try:
|
||||
merged = merge_manifest_into_peer_store(manifest, now=timestamp)
|
||||
return {
|
||||
"ok": True,
|
||||
"seed_peer_url": seed_url,
|
||||
"peer_count": len(manifest.peers),
|
||||
"merged_peer_count": merged,
|
||||
}
|
||||
except Exception as exc:
|
||||
last_error = str(exc or type(exc).__name__)
|
||||
return {"ok": False, "detail": last_error}
|
||||
|
||||
|
||||
def announce_local_peer_to_seeds(*, now: float | None = None, force: bool = False) -> dict[str, Any]:
|
||||
global _LAST_ANNOUNCE_AT
|
||||
import hashlib as _hashlib_mod
|
||||
import hmac as _hmac_mod
|
||||
import requests
|
||||
|
||||
from main import _infonet_peer_requests_proxies, _local_infonet_peer_url, _participant_node_enabled
|
||||
|
||||
if not _participant_node_enabled():
|
||||
return {"ok": False, "detail": "participant node disabled"}
|
||||
peer_url = _local_infonet_peer_url()
|
||||
if not peer_url:
|
||||
return {"ok": False, "detail": "local peer URL is not ready"}
|
||||
peer_key = resolve_peer_key_for_url(peer_url)
|
||||
if not peer_key:
|
||||
return {"ok": False, "detail": "peer HMAC secret is not configured"}
|
||||
|
||||
timestamp = float(now if now is not None else time.time())
|
||||
with _SWARM_LOCK:
|
||||
if not force and _LAST_ANNOUNCE_AT and timestamp - _LAST_ANNOUNCE_AT < 300:
|
||||
return {"ok": True, "skipped": True, "reason": "announce_interval"}
|
||||
_LAST_ANNOUNCE_AT = timestamp
|
||||
|
||||
transport = str(peer_transport_kind(peer_url) or "onion")
|
||||
body = {
|
||||
"peer_url": peer_url,
|
||||
"transport": transport,
|
||||
"role": "participant",
|
||||
"node_id": "",
|
||||
"label": "",
|
||||
"ts": int(timestamp),
|
||||
}
|
||||
body_bytes = json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
|
||||
hmac_hex = _hmac_mod.new(peer_key, body_bytes, _hashlib_mod.sha256).hexdigest()
|
||||
timeout = int(getattr(get_settings(), "MESH_RELAY_PUSH_TIMEOUT_S", 0) or 45)
|
||||
results: list[dict[str, Any]] = []
|
||||
for seed_url in _configured_seed_peer_urls():
|
||||
normalized = normalize_peer_url(seed_url)
|
||||
if not normalized:
|
||||
continue
|
||||
proxies = _infonet_peer_requests_proxies(normalized)
|
||||
request_kwargs: dict[str, Any] = {
|
||||
"data": body_bytes,
|
||||
"headers": {
|
||||
"Content-Type": "application/json",
|
||||
"X-Peer-Url": peer_url,
|
||||
"X-Peer-HMAC": hmac_hex,
|
||||
},
|
||||
"timeout": timeout,
|
||||
}
|
||||
if proxies:
|
||||
request_kwargs["proxies"] = proxies
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{normalized}/api/mesh/infonet/peer-announce",
|
||||
**request_kwargs,
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"seed_peer_url": normalized,
|
||||
"status_code": int(response.status_code),
|
||||
"ok": response.status_code == 200,
|
||||
}
|
||||
)
|
||||
except Exception as exc:
|
||||
results.append({"seed_peer_url": normalized, "ok": False, "detail": str(exc)})
|
||||
ok = any(bool(item.get("ok")) for item in results)
|
||||
return {"ok": ok, "peer_url": peer_url, "results": results}
|
||||
|
||||
|
||||
def _announce_succeeded(announce: dict[str, Any]) -> bool:
|
||||
if not bool(announce.get("ok")):
|
||||
return False
|
||||
results = announce.get("results") or []
|
||||
return any(bool(item.get("ok")) and int(item.get("status_code") or 0) == 200 for item in results)
|
||||
|
||||
|
||||
def _manifest_succeeded(manifest: dict[str, Any]) -> bool:
|
||||
if not bool(manifest.get("ok")):
|
||||
return False
|
||||
peer_count = int(manifest.get("merged_peer_count") or manifest.get("peer_count") or 0)
|
||||
return peer_count >= 1
|
||||
|
||||
|
||||
def join_swarm_with_retries(
|
||||
*,
|
||||
attempts: int = 6,
|
||||
delay_s: float = 15.0,
|
||||
force: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""Announce to seed and pull manifest, retrying while Tor circuits warm up."""
|
||||
last_announce: dict[str, Any] = {"ok": False, "detail": "not attempted"}
|
||||
last_manifest: dict[str, Any] = {"ok": False, "detail": "not attempted"}
|
||||
tries = max(1, int(attempts))
|
||||
pause_s = max(1.0, float(delay_s))
|
||||
for attempt in range(tries):
|
||||
last_announce = announce_local_peer_to_seeds(force=force)
|
||||
last_manifest = refresh_swarm_manifest_from_seeds(force=force)
|
||||
if _announce_succeeded(last_announce) and _manifest_succeeded(last_manifest):
|
||||
return {
|
||||
"ok": True,
|
||||
"attempts": attempt + 1,
|
||||
"announce": last_announce,
|
||||
"manifest_pull": last_manifest,
|
||||
}
|
||||
if attempt + 1 < tries:
|
||||
time.sleep(pause_s)
|
||||
return {
|
||||
"ok": False,
|
||||
"attempts": tries,
|
||||
"announce": last_announce,
|
||||
"manifest_pull": last_manifest,
|
||||
"detail": "swarm join incomplete after retries",
|
||||
}
|
||||
|
||||
|
||||
def push_infonet_events_to_http_peers(events: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
import hashlib as _hashlib_mod
|
||||
import hmac as _hmac_mod
|
||||
import requests
|
||||
|
||||
from main import (
|
||||
_filter_infonet_peer_urls,
|
||||
_infonet_peer_requests_proxies,
|
||||
_local_infonet_peer_url,
|
||||
_participant_node_enabled,
|
||||
_record_public_push_result,
|
||||
)
|
||||
from services.mesh.mesh_router import authenticated_push_peer_urls
|
||||
|
||||
if not _participant_node_enabled() or not events:
|
||||
return {"ok": False, "detail": "nothing to push"}
|
||||
peers = _filter_infonet_peer_urls(authenticated_push_peer_urls())
|
||||
if not peers:
|
||||
return {"ok": False, "detail": "no push peers configured"}
|
||||
|
||||
sender_url = _local_infonet_peer_url()
|
||||
peer_key = resolve_peer_key_for_url(sender_url)
|
||||
if not peer_key:
|
||||
return {"ok": False, "detail": "peer HMAC secret is not configured"}
|
||||
|
||||
body_bytes = json.dumps(
|
||||
{"events": events},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=False,
|
||||
).encode("utf-8")
|
||||
hmac_hex = _hmac_mod.new(peer_key, body_bytes, _hashlib_mod.sha256).hexdigest()
|
||||
timeout = int(getattr(get_settings(), "MESH_RELAY_PUSH_TIMEOUT_S", 0) or 45)
|
||||
results: list[dict[str, Any]] = []
|
||||
for peer_url in peers:
|
||||
normalized = normalize_peer_url(peer_url)
|
||||
if not normalized:
|
||||
continue
|
||||
proxies = _infonet_peer_requests_proxies(normalized)
|
||||
request_kwargs: dict[str, Any] = {
|
||||
"data": body_bytes,
|
||||
"headers": {
|
||||
"Content-Type": "application/json",
|
||||
"X-Peer-Url": sender_url,
|
||||
"X-Peer-HMAC": hmac_hex,
|
||||
},
|
||||
"timeout": timeout,
|
||||
}
|
||||
if proxies:
|
||||
request_kwargs["proxies"] = proxies
|
||||
try:
|
||||
response = requests.post(f"{normalized}/api/mesh/infonet/peer-push", **request_kwargs)
|
||||
results.append(
|
||||
{
|
||||
"peer_url": normalized,
|
||||
"ok": response.status_code == 200,
|
||||
"status_code": int(response.status_code),
|
||||
}
|
||||
)
|
||||
except Exception as exc:
|
||||
results.append({"peer_url": normalized, "ok": False, "detail": str(exc)})
|
||||
ok = any(bool(item.get("ok")) for item in results)
|
||||
event_id = str((events[-1] or {}).get("event_id", "") or "")
|
||||
_record_public_push_result(
|
||||
event_id,
|
||||
ok=ok,
|
||||
error="" if ok else "immediate peer push failed",
|
||||
results=results,
|
||||
)
|
||||
return {"ok": ok, "results": results}
|
||||
@@ -929,6 +929,85 @@ def list_wormhole_dm_contacts() -> dict[str, dict[str, Any]]:
|
||||
return _read_contacts()
|
||||
|
||||
|
||||
def get_wormhole_dm_contact(peer_id: str) -> dict[str, Any] | None:
|
||||
peer_key = str(peer_id or "").strip()
|
||||
if not peer_key:
|
||||
return None
|
||||
contacts = _read_contacts()
|
||||
if peer_key not in contacts:
|
||||
return None
|
||||
return dict(_normalize_contact(contacts[peer_key]))
|
||||
|
||||
|
||||
def sever_wormhole_dm_contact(peer_id: str, *, block: bool = False) -> dict[str, Any]:
|
||||
"""Close the shared DM lane; a fresh contact request + accept is required to reopen."""
|
||||
peer_key = str(peer_id or "").strip()
|
||||
if not peer_key:
|
||||
return {"ok": False, "detail": "peer_id required"}
|
||||
|
||||
contacts = _read_contacts()
|
||||
current = _normalize_contact(contacts.get(peer_key))
|
||||
now = int(time.time())
|
||||
current["sharedAlias"] = ""
|
||||
current["sharedAliasCounter"] = 0
|
||||
current["sharedAliasPublicKey"] = ""
|
||||
current["sharedAliasPublicKeyAlgo"] = "Ed25519"
|
||||
current["previousSharedAliases"] = []
|
||||
current["pendingSharedAlias"] = ""
|
||||
current["pendingSharedAliasCounter"] = 0
|
||||
current["pendingSharedAliasPublicKey"] = ""
|
||||
current["pendingSharedAliasPublicKeyAlgo"] = "Ed25519"
|
||||
current["pendingSharedAliasGraceMs"] = 0
|
||||
current["sharedAliasGraceUntil"] = 0
|
||||
current["sharedAliasRotatedAt"] = 0
|
||||
current["acceptedPreviousAlias"] = ""
|
||||
current["acceptedPreviousAliasCounter"] = 0
|
||||
current["acceptedPreviousAliasPublicKey"] = ""
|
||||
current["acceptedPreviousAliasPublicKeyAlgo"] = "Ed25519"
|
||||
current["acceptedPreviousGraceUntil"] = 0
|
||||
current["acceptedPreviousHardGraceUntil"] = 0
|
||||
current["acceptedPreviousAwaitingReply"] = False
|
||||
current["aliasBindingSeq"] = 0
|
||||
current["aliasBindingPendingReason"] = ""
|
||||
current["aliasBindingPreparedAt"] = 0
|
||||
current["aliasGateJoinAppliedSeq"] = 0
|
||||
if block:
|
||||
current["blocked"] = True
|
||||
current["updated_at"] = now
|
||||
contacts[peer_key] = _normalize_contact(current)
|
||||
_write_contacts(contacts)
|
||||
|
||||
relay_policy = {}
|
||||
try:
|
||||
from services.mesh.mesh_dm_connect_delivery import revoke_connect_relay_policy
|
||||
|
||||
relay_policy = revoke_connect_relay_policy(peer_key)
|
||||
except Exception:
|
||||
relay_policy = {"ok": False}
|
||||
|
||||
relay_block = {"ok": False}
|
||||
if block:
|
||||
try:
|
||||
from services.mesh.mesh_dm_relay import dm_relay
|
||||
from services.mesh.mesh_wormhole_persona import get_dm_identity
|
||||
|
||||
local_id = str(get_dm_identity().get("node_id", "") or "").strip()
|
||||
if local_id:
|
||||
dm_relay.block(local_id, peer_key)
|
||||
relay_block = {"ok": True, "local_id": local_id}
|
||||
except Exception as exc:
|
||||
relay_block = {"ok": False, "detail": str(exc) or type(exc).__name__}
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"peer_id": peer_key,
|
||||
"severed": True,
|
||||
"blocked": bool(block),
|
||||
"relay_policy": relay_policy,
|
||||
"relay_block": relay_block,
|
||||
}
|
||||
|
||||
|
||||
def _promote_invite_lookup_mode(contact: dict[str, Any], *, now: int | None = None) -> bool:
|
||||
current = dict(contact or {})
|
||||
lookup_handle = str(current.get("invitePinnedPrekeyLookupHandle", "") or "").strip()
|
||||
@@ -1070,11 +1149,14 @@ def pin_wormhole_dm_invite(
|
||||
identity_dh_pub_key = str(payload.get("identity_dh_pub_key", "") or "")
|
||||
dh_algo = str(payload.get("dh_algo", "X25519") or "X25519")
|
||||
prekey_lookup_handle = str(payload.get("prekey_lookup_handle", "") or "")
|
||||
lookup_peer_url = str(payload.get("lookup_peer_url", "") or "").strip().rstrip("/")
|
||||
if str(alias or "").strip():
|
||||
current["alias"] = str(alias or "").strip()
|
||||
current["dhPubKey"] = identity_dh_pub_key
|
||||
current["dhAlgo"] = dh_algo
|
||||
current["invitePinnedPrekeyLookupHandle"] = prekey_lookup_handle
|
||||
if lookup_peer_url:
|
||||
current["invitePinnedLookupPeerUrl"] = lookup_peer_url
|
||||
current["invitePinnedRootFingerprint"] = str(payload.get("root_fingerprint", "") or "").strip().lower()
|
||||
current["invitePinnedRootManifestFingerprint"] = str(
|
||||
payload.get("root_manifest_fingerprint", "") or ""
|
||||
@@ -1170,6 +1252,12 @@ def pin_wormhole_dm_invite(
|
||||
current["updated_at"] = now
|
||||
contacts[peer_key] = _normalize_contact(current)
|
||||
_write_contacts(contacts)
|
||||
try:
|
||||
from services.mesh.mesh_dm_connect_delivery import grant_connect_relay_policy
|
||||
|
||||
grant_connect_relay_policy(peer_key, reason="invite_import")
|
||||
except Exception:
|
||||
pass
|
||||
return contacts[peer_key]
|
||||
|
||||
|
||||
|
||||
@@ -549,6 +549,27 @@ def invite_identity_commitment_for_identity_material(
|
||||
return hashlib.sha256(_stable_json(material).encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _local_dm_lookup_peer_url() -> str:
|
||||
"""Return this node's fleet-reachable URL for invite-scoped prekey lookup."""
|
||||
try:
|
||||
from services.config import get_settings
|
||||
from services.mesh.mesh_crypto import normalize_peer_url
|
||||
|
||||
configured = normalize_peer_url(str(getattr(get_settings(), "MESH_PUBLIC_PEER_URL", "") or ""))
|
||||
if configured:
|
||||
return configured
|
||||
from services.tor_hidden_service import tor_service
|
||||
|
||||
onion = str(getattr(tor_service, "onion_address", "") or "").strip()
|
||||
if onion:
|
||||
if "://" not in onion:
|
||||
onion = f"http://{onion}:8000"
|
||||
return normalize_peer_url(onion)
|
||||
except Exception:
|
||||
pass
|
||||
return ""
|
||||
|
||||
|
||||
def _dm_invite_payload(
|
||||
data: dict[str, Any],
|
||||
*,
|
||||
@@ -930,6 +951,9 @@ def export_wormhole_dm_invite(*, label: str = "", expires_in_s: int = 0) -> dict
|
||||
# fetch our prekey bundle without using our stable agent_id.
|
||||
lookup_handle = secrets.token_hex(24)
|
||||
payload["prekey_lookup_handle"] = lookup_handle
|
||||
lookup_peer_url = _local_dm_lookup_peer_url()
|
||||
if lookup_peer_url:
|
||||
payload["lookup_peer_url"] = lookup_peer_url
|
||||
|
||||
# Persist the handle so it is included in future prekey registrations.
|
||||
existing_handles, _ = _normalize_prekey_lookup_handles(
|
||||
|
||||
@@ -79,6 +79,164 @@ def _warn_legacy_prekey_lookup(agent_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _fleet_peer_lookup_user_agent() -> str:
|
||||
custom = str(os.environ.get("SHADOWBROKER_MESH_PEER_USER_AGENT") or "").strip()
|
||||
if custom:
|
||||
return custom
|
||||
return "Mozilla/5.0 (compatible; ShadowbrokerMesh/1.0)"
|
||||
|
||||
|
||||
_INVITE_LOOKUP_MAX_ELAPSED_S = 120
|
||||
_INVITE_LOOKUP_MAX_BOOTSTRAP_PEERS = 3
|
||||
_INVITE_LOOKUP_MAX_PUSH_PEERS = 16
|
||||
_INVITE_LOOKUP_PARALLEL_WORKERS = 8
|
||||
|
||||
|
||||
def _invite_lookup_request_timeout(peer_url: str) -> tuple[int, int]:
|
||||
from services.mesh.mesh_router import peer_transport_kind
|
||||
|
||||
if peer_transport_kind(peer_url) == "onion":
|
||||
return (10, 35)
|
||||
return (5, 15)
|
||||
|
||||
|
||||
def _bootstrap_seed_peer_urls() -> set[str]:
|
||||
try:
|
||||
from services.config import get_settings
|
||||
from services.mesh.mesh_router import parse_configured_relay_peers
|
||||
|
||||
seeds: set[str] = set()
|
||||
raw = str(getattr(get_settings(), "MESH_BOOTSTRAP_SEED_PEERS", "") or "")
|
||||
for peer in parse_configured_relay_peers(raw):
|
||||
normalized = str(peer or "").strip().rstrip("/")
|
||||
if normalized:
|
||||
seeds.add(normalized)
|
||||
return seeds
|
||||
except Exception:
|
||||
return set()
|
||||
|
||||
|
||||
def _discovered_push_peer_urls(*, limit: int = _INVITE_LOOKUP_MAX_PUSH_PEERS) -> list[str]:
|
||||
try:
|
||||
from services.mesh.mesh_router import authenticated_push_peer_urls
|
||||
|
||||
seeds = _bootstrap_seed_peer_urls()
|
||||
peers: list[str] = []
|
||||
for peer in authenticated_push_peer_urls():
|
||||
normalized = str(peer or "").strip().rstrip("/")
|
||||
if not normalized or normalized in seeds:
|
||||
continue
|
||||
peers.append(normalized)
|
||||
if len(peers) >= max(1, int(limit or 1)):
|
||||
break
|
||||
return peers
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
def _prioritized_invite_lookup_peer_urls(*, preferred: list[str] | None = None) -> list[str]:
|
||||
preferred_urls = [
|
||||
str(peer or "").strip().rstrip("/")
|
||||
for peer in list(preferred or [])
|
||||
if str(peer or "").strip()
|
||||
]
|
||||
configured = _configured_public_lookup_peer_urls()
|
||||
seeds = _bootstrap_seed_peer_urls()
|
||||
active: list[str] = []
|
||||
bootstrap: list[str] = []
|
||||
push_discovery: list[str] = []
|
||||
seen = set(preferred_urls)
|
||||
for peer in configured:
|
||||
if peer in seen:
|
||||
continue
|
||||
seen.add(peer)
|
||||
if peer in seeds:
|
||||
bootstrap.append(peer)
|
||||
else:
|
||||
active.append(peer)
|
||||
for peer in _discovered_push_peer_urls():
|
||||
if peer in seen:
|
||||
continue
|
||||
seen.add(peer)
|
||||
push_discovery.append(peer)
|
||||
ordered = list(preferred_urls)
|
||||
ordered.extend(active)
|
||||
ordered.extend(push_discovery)
|
||||
ordered.extend(bootstrap[:_INVITE_LOOKUP_MAX_BOOTSTRAP_PEERS])
|
||||
return ordered
|
||||
|
||||
|
||||
def _preferred_invite_lookup_peer_urls(lookup_token: str) -> list[str]:
|
||||
token = str(lookup_token or "").strip()
|
||||
if not token:
|
||||
return []
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_contacts import list_wormhole_dm_contacts
|
||||
except Exception:
|
||||
return []
|
||||
peers: list[str] = []
|
||||
for contact in list_wormhole_dm_contacts() or []:
|
||||
if not isinstance(contact, dict):
|
||||
continue
|
||||
if str(contact.get("invitePinnedPrekeyLookupHandle", "") or "").strip() != token:
|
||||
continue
|
||||
peer_url = str(contact.get("invitePinnedLookupPeerUrl", "") or "").strip().rstrip("/")
|
||||
if peer_url and peer_url not in peers:
|
||||
peers.append(peer_url)
|
||||
return peers
|
||||
|
||||
|
||||
def _peer_http_request(
|
||||
method: str,
|
||||
peer_url: str,
|
||||
*,
|
||||
body_bytes: bytes | None = None,
|
||||
headers: dict[str, str] | None = None,
|
||||
timeout: int | tuple[int, int] = 45,
|
||||
):
|
||||
"""HTTP to a fleet peer, using Tor SOCKS when the URL is an onion address."""
|
||||
import requests
|
||||
|
||||
from services.mesh.mesh_crypto import normalize_peer_url
|
||||
from urllib.parse import urlparse
|
||||
|
||||
raw_peer_url = str(peer_url or "").strip()
|
||||
parsed = urlparse(raw_peer_url)
|
||||
if parsed.path and parsed.path not in {"", "/"}:
|
||||
# Full request URLs include invite lookup query params; do not
|
||||
# normalize them away when deriving the peer base URL.
|
||||
normalized = raw_peer_url
|
||||
else:
|
||||
normalized = normalize_peer_url(raw_peer_url)
|
||||
if not normalized:
|
||||
raise OSError("invalid peer url")
|
||||
if isinstance(timeout, tuple):
|
||||
connect_timeout, read_timeout = timeout
|
||||
resolved_timeout: int | tuple[int, int] = (
|
||||
max(1, int(connect_timeout or 5)),
|
||||
max(1, int(read_timeout or 15)),
|
||||
)
|
||||
else:
|
||||
resolved_timeout = max(1, int(timeout or 45))
|
||||
request_kwargs: dict[str, Any] = {
|
||||
"headers": dict(headers or {}),
|
||||
"timeout": resolved_timeout,
|
||||
}
|
||||
try:
|
||||
from main import _infonet_peer_requests_proxies
|
||||
|
||||
proxy_peer_url = normalize_peer_url(f"{parsed.scheme}://{parsed.netloc}")
|
||||
proxies = _infonet_peer_requests_proxies(proxy_peer_url)
|
||||
if proxies:
|
||||
request_kwargs["proxies"] = proxies
|
||||
except Exception:
|
||||
pass
|
||||
if method.upper() == "GET":
|
||||
return requests.get(normalized, **request_kwargs)
|
||||
request_kwargs["data"] = body_bytes or b""
|
||||
return requests.post(normalized, **request_kwargs)
|
||||
|
||||
|
||||
def _fetch_dm_prekey_bundle_from_peer_lookup(lookup_token: str) -> dict[str, Any]:
|
||||
"""Fetch an invite-scoped prekey bundle from configured authenticated peers.
|
||||
|
||||
@@ -95,12 +253,12 @@ def _fetch_dm_prekey_bundle_from_peer_lookup(lookup_token: str) -> dict[str, Any
|
||||
normalize_peer_url,
|
||||
resolve_peer_key_for_url,
|
||||
)
|
||||
from services.mesh.mesh_router import configured_relay_peer_urls
|
||||
from services.mesh.mesh_router import authenticated_push_peer_urls
|
||||
|
||||
settings = get_settings()
|
||||
# Issue #256: secret check moved per-peer below. We still bail out
|
||||
# cleanly when there are no peers configured at all.
|
||||
peers = configured_relay_peer_urls()
|
||||
peers = authenticated_push_peer_urls()
|
||||
if not peers:
|
||||
return {"ok": False, "detail": "peer prekey lookup unavailable"}
|
||||
timeout = max(1, _safe_int(getattr(settings, "MESH_RELAY_PUSH_TIMEOUT_S", 10) or 10, 10))
|
||||
@@ -132,17 +290,17 @@ def _fetch_dm_prekey_bundle_from_peer_lookup(lookup_token: str) -> dict[str, Any
|
||||
"X-Peer-Url": sender_peer_url,
|
||||
"X-Peer-HMAC": hmac.new(peer_key, body, hashlib.sha256).hexdigest(),
|
||||
}
|
||||
request = urllib.request.Request(
|
||||
f"{normalized_peer_url}/api/mesh/dm/prekey-peer-lookup",
|
||||
data=body,
|
||||
headers=headers,
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
raw = response.read(256 * 1024)
|
||||
response = _peer_http_request(
|
||||
"POST",
|
||||
f"{normalized_peer_url}/api/mesh/dm/prekey-peer-lookup",
|
||||
body_bytes=body,
|
||||
headers=headers,
|
||||
timeout=timeout,
|
||||
)
|
||||
raw = response.content[: 256 * 1024]
|
||||
payload = json.loads(raw.decode("utf-8"))
|
||||
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, OSError) as exc:
|
||||
except (json.JSONDecodeError, OSError, Exception) as exc:
|
||||
last_detail = str(exc) or type(exc).__name__
|
||||
continue
|
||||
if isinstance(payload, dict) and payload.get("ok"):
|
||||
@@ -161,12 +319,18 @@ def _configured_public_lookup_peer_urls() -> list[str]:
|
||||
|
||||
settings = get_settings()
|
||||
candidates: list[str] = []
|
||||
# Operator-configured peers first, then recently active fleet nodes.
|
||||
# Invite handles are minted on a specific node; cold bootstrap seeds
|
||||
# rarely have them cached and should not be tried before contacts.
|
||||
for raw in (
|
||||
getattr(settings, "MESH_BOOTSTRAP_SEED_PEERS", ""),
|
||||
getattr(settings, "MESH_DEFAULT_SYNC_PEERS", ""),
|
||||
):
|
||||
candidates.extend(parse_configured_relay_peers(str(raw or "")))
|
||||
candidates.extend(active_sync_peer_urls())
|
||||
for raw in (
|
||||
getattr(settings, "MESH_BOOTSTRAP_SEED_PEERS", ""),
|
||||
):
|
||||
candidates.extend(parse_configured_relay_peers(str(raw or "")))
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
@@ -204,7 +368,50 @@ def _normalize_remote_lookup_bundle(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return data
|
||||
|
||||
|
||||
def _fetch_dm_prekey_bundle_from_public_lookup(lookup_token: str) -> dict[str, Any]:
|
||||
def _try_public_prekey_lookup_peer(
|
||||
peer_url: str,
|
||||
encoded: str,
|
||||
*,
|
||||
timeout: int | tuple[int, int] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
normalized_peer_url = str(peer_url or "").strip().rstrip("/")
|
||||
if not normalized_peer_url:
|
||||
return {"ok": False, "detail": "invalid peer url"}
|
||||
resolved_timeout = timeout or _invite_lookup_request_timeout(normalized_peer_url)
|
||||
try:
|
||||
response = _peer_http_request(
|
||||
"GET",
|
||||
f"{normalized_peer_url}/api/mesh/dm/prekey-bundle?{encoded}",
|
||||
headers={
|
||||
"Accept": "application/json",
|
||||
"User-Agent": _fleet_peer_lookup_user_agent(),
|
||||
},
|
||||
timeout=resolved_timeout,
|
||||
)
|
||||
raw = response.content[: 256 * 1024]
|
||||
payload = json.loads(raw.decode("utf-8"))
|
||||
except (json.JSONDecodeError, OSError, Exception) as exc:
|
||||
logger.debug("public prekey lookup failed for %s: %s", normalized_peer_url, type(exc).__name__)
|
||||
return {"ok": False, "detail": "peer prekey lookup unavailable"}
|
||||
if not isinstance(payload, dict):
|
||||
return {"ok": False, "detail": "invalid peer response"}
|
||||
if payload.get("pending") or str(payload.get("status", "") or "") == "preparing_private_lane":
|
||||
return {"ok": False, "detail": "peer prekey lookup still preparing"}
|
||||
if not payload.get("ok"):
|
||||
return {
|
||||
"ok": False,
|
||||
"detail": str(payload.get("detail", "") or "Prekey bundle not found"),
|
||||
}
|
||||
if not isinstance(payload.get("bundle"), dict):
|
||||
return {"ok": False, "detail": "Prekey bundle not found"}
|
||||
return _normalize_remote_lookup_bundle(payload)
|
||||
|
||||
|
||||
def _fetch_dm_prekey_bundle_from_public_lookup(
|
||||
lookup_token: str,
|
||||
*,
|
||||
extra_preferred_peer_urls: list[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Fetch an invite-scoped prekey bundle from bootstrap/sync peers.
|
||||
|
||||
The token is high-entropy and invite-scoped. This path does not expose a
|
||||
@@ -212,61 +419,69 @@ def _fetch_dm_prekey_bundle_from_public_lookup(lookup_token: str) -> dict[str, A
|
||||
derive it from the signed identity public key and validate the bundle before
|
||||
accepting it.
|
||||
"""
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
|
||||
token = str(lookup_token or "").strip()
|
||||
if not token:
|
||||
return {"ok": False, "detail": "lookup token required"}
|
||||
peers = _configured_public_lookup_peer_urls()
|
||||
preferred = list(_preferred_invite_lookup_peer_urls(token))
|
||||
for peer in list(extra_preferred_peer_urls or []):
|
||||
normalized = str(peer or "").strip().rstrip("/")
|
||||
if normalized and normalized not in preferred:
|
||||
preferred.insert(0, normalized)
|
||||
peers = _prioritized_invite_lookup_peer_urls(preferred=preferred)
|
||||
if not peers:
|
||||
return {"ok": False, "detail": "peer prekey lookup unavailable"}
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
timeout = max(1, _safe_int(getattr(get_settings(), "MESH_SYNC_TIMEOUT_S", 5) or 5, 5))
|
||||
except Exception:
|
||||
timeout = 5
|
||||
|
||||
encoded = urllib.parse.urlencode({"lookup_token": token})
|
||||
last_detail = ""
|
||||
for peer_url in peers:
|
||||
normalized_peer_url = str(peer_url or "").strip().rstrip("/")
|
||||
if not normalized_peer_url:
|
||||
continue
|
||||
# Generic UA: any peer-facing crypto request should not carry a
|
||||
# fork-specific identifier — that turns prekey lookups into a
|
||||
# software-fingerprinting beacon.
|
||||
from services.network_utils import default_user_agent
|
||||
request = urllib.request.Request(
|
||||
f"{normalized_peer_url}/api/mesh/dm/prekey-bundle?{encoded}",
|
||||
headers={
|
||||
"Accept": "application/json",
|
||||
"User-Agent": default_user_agent(),
|
||||
},
|
||||
method="GET",
|
||||
hinted_only = bool(list(extra_preferred_peer_urls or []))
|
||||
hint_timeout = (5, 20)
|
||||
for peer_url in preferred:
|
||||
hinted = _try_public_prekey_lookup_peer(
|
||||
peer_url,
|
||||
encoded,
|
||||
timeout=hint_timeout if hinted_only else None,
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
raw = response.read(256 * 1024)
|
||||
payload = json.loads(raw.decode("utf-8"))
|
||||
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, OSError) as exc:
|
||||
logger.debug("public prekey lookup failed for %s: %s", normalized_peer_url, type(exc).__name__)
|
||||
last_detail = "peer prekey lookup unavailable"
|
||||
continue
|
||||
if not isinstance(payload, dict):
|
||||
last_detail = "invalid peer response"
|
||||
continue
|
||||
if payload.get("pending") or str(payload.get("status", "") or "") == "preparing_private_lane":
|
||||
last_detail = "peer prekey lookup still preparing"
|
||||
continue
|
||||
if not payload.get("ok"):
|
||||
last_detail = str(payload.get("detail", "") or last_detail or "Prekey bundle not found")
|
||||
continue
|
||||
if not isinstance(payload.get("bundle"), dict):
|
||||
last_detail = "Prekey bundle not found"
|
||||
continue
|
||||
normalized = _normalize_remote_lookup_bundle(payload)
|
||||
if normalized.get("ok"):
|
||||
return normalized
|
||||
last_detail = str(normalized.get("detail", "") or last_detail)
|
||||
if hinted.get("ok"):
|
||||
return hinted
|
||||
if isinstance(hinted, dict):
|
||||
last_detail = str(hinted.get("detail", "") or last_detail)
|
||||
remaining_peers = [peer for peer in peers if peer not in set(preferred)]
|
||||
if not remaining_peers:
|
||||
return {"ok": False, "detail": last_detail or "Prekey bundle not found"}
|
||||
if hinted_only:
|
||||
return {"ok": False, "detail": last_detail or "Prekey bundle not found"}
|
||||
deadline = time.time() + _INVITE_LOOKUP_MAX_ELAPSED_S
|
||||
workers = min(_INVITE_LOOKUP_PARALLEL_WORKERS, max(1, len(remaining_peers)))
|
||||
with ThreadPoolExecutor(max_workers=workers) as executor:
|
||||
futures = {
|
||||
executor.submit(_try_public_prekey_lookup_peer, peer_url, encoded): peer_url
|
||||
for peer_url in remaining_peers
|
||||
}
|
||||
while futures and time.time() < deadline:
|
||||
done, _ = wait(
|
||||
futures,
|
||||
timeout=max(0.1, deadline - time.time()),
|
||||
return_when=FIRST_COMPLETED,
|
||||
)
|
||||
if not done:
|
||||
break
|
||||
for future in done:
|
||||
futures.pop(future, None)
|
||||
try:
|
||||
result = future.result()
|
||||
except Exception as exc:
|
||||
last_detail = str(exc) or type(exc).__name__
|
||||
continue
|
||||
if isinstance(result, dict) and result.get("ok"):
|
||||
for pending in futures:
|
||||
pending.cancel()
|
||||
return result
|
||||
if isinstance(result, dict):
|
||||
last_detail = str(result.get("detail", "") or last_detail)
|
||||
for pending in futures:
|
||||
pending.cancel()
|
||||
return {"ok": False, "detail": last_detail or "Prekey bundle not found"}
|
||||
|
||||
|
||||
@@ -1019,6 +1234,7 @@ def fetch_dm_prekey_bundle(
|
||||
lookup_token: str = "",
|
||||
*,
|
||||
allow_peer_lookup: bool = True,
|
||||
lookup_peer_urls: list[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
from services.mesh.mesh_dm_relay import dm_relay
|
||||
|
||||
@@ -1043,12 +1259,18 @@ def fetch_dm_prekey_bundle(
|
||||
resolved_id = found_id
|
||||
lookup_mode = "invite_lookup_handle"
|
||||
elif allow_peer_lookup:
|
||||
peer_found = _fetch_dm_prekey_bundle_from_peer_lookup(resolved_lookup)
|
||||
if peer_found.get("ok"):
|
||||
return peer_found
|
||||
public_found = _fetch_dm_prekey_bundle_from_public_lookup(resolved_lookup)
|
||||
preferred_peer_urls = list(lookup_peer_urls or [])
|
||||
public_found = _fetch_dm_prekey_bundle_from_public_lookup(
|
||||
resolved_lookup,
|
||||
extra_preferred_peer_urls=preferred_peer_urls,
|
||||
)
|
||||
if public_found.get("ok"):
|
||||
return public_found
|
||||
peer_found: dict[str, Any] = {"ok": False, "detail": ""}
|
||||
if not preferred_peer_urls:
|
||||
peer_found = _fetch_dm_prekey_bundle_from_peer_lookup(resolved_lookup)
|
||||
if peer_found.get("ok"):
|
||||
return peer_found
|
||||
if str(public_found.get("detail", "") or "").strip():
|
||||
return {"ok": False, "detail": str(public_found.get("detail", "") or "Prekey bundle not found")}
|
||||
return {"ok": False, "detail": str(peer_found.get("detail", "") or "Prekey bundle not found")}
|
||||
@@ -1134,12 +1356,24 @@ def _classify_root_attestation_failure(peer_id: str) -> tuple[str, bool]:
|
||||
return "", False
|
||||
|
||||
|
||||
def bootstrap_encrypt_for_peer(peer_id: str, plaintext: str) -> dict[str, Any]:
|
||||
fetched_bundle = fetch_dm_prekey_bundle(str(peer_id or "").strip())
|
||||
def bootstrap_encrypt_for_peer(
|
||||
peer_id: str,
|
||||
plaintext: str,
|
||||
*,
|
||||
lookup_token: str = "",
|
||||
fetched_bundle: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
token = str(lookup_token or "").strip()
|
||||
peer = str(peer_id or "").strip()
|
||||
if fetched_bundle is None:
|
||||
fetched_bundle = fetch_dm_prekey_bundle(
|
||||
agent_id=peer if not token else "",
|
||||
lookup_token=token,
|
||||
)
|
||||
if not fetched_bundle.get("ok"):
|
||||
detail = str(fetched_bundle.get("detail", "") or "")
|
||||
if "root attestation" in detail.lower():
|
||||
trust_level, trust_changed = _classify_root_attestation_failure(str(peer_id or "").strip())
|
||||
trust_level, trust_changed = _classify_root_attestation_failure(peer or token)
|
||||
if trust_level:
|
||||
return {
|
||||
"ok": False,
|
||||
@@ -1152,32 +1386,68 @@ def bootstrap_encrypt_for_peer(peer_id: str, plaintext: str) -> dict[str, Any]:
|
||||
|
||||
from services.mesh.mesh_dm_relay import dm_relay
|
||||
|
||||
resolved_peer_id = str(fetched_bundle.get("agent_id", peer_id) or peer_id).strip()
|
||||
resolved_peer_id = str(fetched_bundle.get("agent_id", peer) or peer).strip()
|
||||
stored = dm_relay.get_prekey_bundle(resolved_peer_id)
|
||||
if not stored:
|
||||
return {"ok": False, "detail": "Peer prekey bundle not found"}
|
||||
remote_bundle = dict(fetched_bundle.get("bundle") or {})
|
||||
if not remote_bundle and fetched_bundle.get("identity_dh_pub_key"):
|
||||
remote_bundle = fetched_bundle
|
||||
if remote_bundle:
|
||||
stored = {
|
||||
"bundle": remote_bundle,
|
||||
"signature": str(fetched_bundle.get("signature", "") or ""),
|
||||
"public_key": str(fetched_bundle.get("public_key", "") or ""),
|
||||
"public_key_algo": str(fetched_bundle.get("public_key_algo", "") or ""),
|
||||
"sequence": _safe_int(fetched_bundle.get("sequence", 0) or 0),
|
||||
}
|
||||
else:
|
||||
return {"ok": False, "detail": "Peer prekey bundle not found"}
|
||||
validated_record = {**dict(stored), "agent_id": resolved_peer_id}
|
||||
ok, reason = _validate_bundle_record(validated_record)
|
||||
if not ok:
|
||||
return {"ok": False, "detail": reason}
|
||||
trust_state = observe_remote_prekey_bundle(resolved_peer_id, validated_record)
|
||||
trust_level = str(trust_state.get("trust_level", "") or "")
|
||||
from services.mesh.mesh_wormhole_contacts import verified_first_contact_requirement
|
||||
consent_handshake = False
|
||||
try:
|
||||
from services.mesh.mesh_wormhole_dead_drop import parse_contact_consent
|
||||
|
||||
verified_first_contact = verified_first_contact_requirement(
|
||||
resolved_peer_id,
|
||||
trust_level=trust_level,
|
||||
)
|
||||
if not verified_first_contact.get("ok"):
|
||||
return {
|
||||
"ok": False,
|
||||
"peer_id": resolved_peer_id,
|
||||
"detail": str(verified_first_contact.get("detail", "") or "verified first contact required"),
|
||||
"trust_changed": trust_level in ("mismatch", "continuity_broken"),
|
||||
"trust_level": str(verified_first_contact.get("trust_level", "") or trust_level or "unpinned"),
|
||||
consent = parse_contact_consent(str(plaintext or "")) or {}
|
||||
consent_handshake = str(consent.get("kind", "") or "") in {
|
||||
"contact_offer",
|
||||
"contact_accept",
|
||||
"contact_deny",
|
||||
}
|
||||
except Exception:
|
||||
consent_handshake = False
|
||||
if not consent_handshake:
|
||||
from services.mesh.mesh_wormhole_contacts import verified_first_contact_requirement
|
||||
|
||||
verified_first_contact = verified_first_contact_requirement(
|
||||
resolved_peer_id,
|
||||
trust_level=trust_level,
|
||||
)
|
||||
if not verified_first_contact.get("ok"):
|
||||
return {
|
||||
"ok": False,
|
||||
"peer_id": resolved_peer_id,
|
||||
"detail": str(
|
||||
verified_first_contact.get("detail", "") or "verified first contact required"
|
||||
),
|
||||
"trust_changed": trust_level in ("mismatch", "continuity_broken"),
|
||||
"trust_level": str(
|
||||
verified_first_contact.get("trust_level", "") or trust_level or "unpinned"
|
||||
),
|
||||
}
|
||||
peer_bundle_stored = dm_relay.consume_one_time_prekey(resolved_peer_id)
|
||||
if not peer_bundle_stored:
|
||||
remote_bundle = dict(stored.get("bundle") or {})
|
||||
otks = list(remote_bundle.get("one_time_prekeys") or [])
|
||||
peer_bundle_stored = {
|
||||
"bundle": remote_bundle,
|
||||
"claimed_one_time_prekey": dict(otks[0] or {}) if otks else {},
|
||||
}
|
||||
if not peer_bundle_stored.get("bundle"):
|
||||
return {"ok": False, "detail": "Peer prekey bundle not found"}
|
||||
peer_bundle = dict(peer_bundle_stored.get("bundle") or {})
|
||||
peer_static = str(peer_bundle.get("identity_dh_pub_key", "") or "")
|
||||
|
||||
@@ -87,6 +87,23 @@ READ_COMMANDS = frozenset({
|
||||
"osint_lookup",
|
||||
"osint_tools",
|
||||
"entity_expand",
|
||||
# Agent routing helpers
|
||||
"route_query",
|
||||
"run_playbook",
|
||||
"gt_risk_heatmap",
|
||||
"gt_dossier",
|
||||
"gt_analyze",
|
||||
"gt_backtest",
|
||||
"gt_rolling_freeze",
|
||||
"gt_rolling_label",
|
||||
"gt_rolling_backtest",
|
||||
"gt_micro_rolling",
|
||||
"gt_top_alerts",
|
||||
# Private Infonet reads (operator-delegated)
|
||||
"infonet_status",
|
||||
"list_gates",
|
||||
"read_gate_messages",
|
||||
"poll_dms",
|
||||
})
|
||||
|
||||
WRITE_COMMANDS = frozenset({
|
||||
@@ -118,6 +135,12 @@ WRITE_COMMANDS = frozenset({
|
||||
"clear_analysis_zones",
|
||||
# Active recon (subnet device discovery)
|
||||
"osint_sweep",
|
||||
# Private Infonet writes (operator wormhole identity)
|
||||
"ensure_infonet_ready",
|
||||
"join_infonet_swarm",
|
||||
"post_gate_message",
|
||||
"cast_vote",
|
||||
"send_dm",
|
||||
})
|
||||
|
||||
|
||||
@@ -643,6 +666,19 @@ def _compact_query_result(result: Any) -> Any:
|
||||
# Command dispatcher
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _expensive_gate(cmd: str, args: dict[str, Any]) -> dict[str, Any] | None:
|
||||
from services.openclaw_routing import EXPENSIVE_GATE_MESSAGE, requires_expensive_confirm
|
||||
|
||||
if requires_expensive_confirm(cmd, args):
|
||||
return {
|
||||
"ok": False,
|
||||
"detail": EXPENSIVE_GATE_MESSAGE,
|
||||
"code": "expensive_command_blocked",
|
||||
"hint": "route_query",
|
||||
}
|
||||
return None
|
||||
|
||||
|
||||
def _dispatch_command(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Route a command to the appropriate AI Intel function.
|
||||
|
||||
@@ -650,6 +686,43 @@ def _dispatch_command(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
Commands run in an isolated thread (via _execute_command) so they
|
||||
do not need or touch the caller's event loop.
|
||||
"""
|
||||
blocked = _expensive_gate(cmd, args)
|
||||
if blocked is not None:
|
||||
return blocked
|
||||
|
||||
if cmd == "route_query":
|
||||
from services.openclaw_routing import route_query
|
||||
|
||||
result = route_query(
|
||||
text=str(args.get("text", "") or args.get("query", "") or ""),
|
||||
lat=args.get("lat"),
|
||||
lng=args.get("lng"),
|
||||
radius_km=float(args.get("radius_km", 50) or 50),
|
||||
compact=bool(args.get("compact", True)),
|
||||
)
|
||||
return {"ok": True, "data": result}
|
||||
|
||||
if cmd == "run_playbook":
|
||||
from services.openclaw_routing import plan_playbook
|
||||
|
||||
plan = plan_playbook(str(args.get("name", "") or args.get("playbook", "")), args)
|
||||
if not plan.get("ok"):
|
||||
return plan
|
||||
batch_results: list[dict[str, Any]] = []
|
||||
for item in plan.get("batch", []):
|
||||
inner_cmd = str(item.get("cmd", "")).strip().lower()
|
||||
inner_args = item.get("args") or {}
|
||||
inner_result = _dispatch_command(inner_cmd, inner_args)
|
||||
batch_results.append({"cmd": inner_cmd, **inner_result})
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"playbook": plan.get("playbook"),
|
||||
"description": plan.get("description", ""),
|
||||
"results": batch_results,
|
||||
},
|
||||
}
|
||||
|
||||
if cmd == "get_telemetry":
|
||||
from services.telemetry import get_cached_telemetry_refs
|
||||
data = get_cached_telemetry_refs()
|
||||
@@ -731,6 +804,7 @@ def _dispatch_command(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
owner=str(args.get("owner", "") or args.get("operator", "") or ""),
|
||||
layers=args.get("layers") if isinstance(args.get("layers"), (list, tuple)) else None,
|
||||
limit=args.get("limit", 10),
|
||||
fallback_search=bool(args.get("fallback_search") or args.get("confirm_fuzzy")),
|
||||
)
|
||||
if _wants_compact(args):
|
||||
compact = dict(result)
|
||||
@@ -792,6 +866,284 @@ def _dispatch_command(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
return {"ok": True, "data": _compact_query_result(result), "format": "compressed_v1"}
|
||||
return {"ok": True, "data": result}
|
||||
|
||||
if cmd == "gt_risk_heatmap":
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
from analytics.integration import get_gt_engine
|
||||
from services.fetchers._store import get_latest_data_subset_refs
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {"ok": True, "data": {"enabled": False, "features": [], "clusters": []}}
|
||||
snap = get_latest_data_subset_refs("gt_risk")
|
||||
payload = dict(snap.get("gt_risk") or {})
|
||||
engine = get_gt_engine()
|
||||
if engine is not None and not payload.get("heatmap"):
|
||||
payload["heatmap"] = engine.get_risk_heatmap()
|
||||
return {"ok": True, "data": payload}
|
||||
|
||||
if cmd == "gt_dossier":
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
from analytics.integration import get_gt_engine
|
||||
|
||||
region = str(args.get("region", "") or args.get("area", "") or "").strip().lower()
|
||||
if not region:
|
||||
return {"ok": False, "detail": "region required (e.g. ukraine, uk, europe)"}
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"enabled": False,
|
||||
"region": region,
|
||||
"interpretation": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED).",
|
||||
},
|
||||
}
|
||||
engine = get_gt_engine()
|
||||
if engine is None:
|
||||
return {"ok": False, "detail": "GT analytics engine unavailable"}
|
||||
return {"ok": True, "data": engine.get_dossier(region)}
|
||||
|
||||
if cmd == "gt_analyze":
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
from analytics.integration import get_gt_engine, refresh_from_latest_data
|
||||
from services.fetchers._store import _data_lock, latest_data
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {"ok": False, "detail": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED)"}
|
||||
engine = get_gt_engine()
|
||||
if engine is None:
|
||||
return {"ok": False, "detail": "GT analytics engine unavailable"}
|
||||
|
||||
feeds = args.get("feeds") if isinstance(args.get("feeds"), (list, tuple)) else None
|
||||
if feeds:
|
||||
from analytics.feed_adapter import normalize_feed_item
|
||||
|
||||
ingested = 0
|
||||
for raw in feeds:
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
item = normalize_feed_item(raw, source_type=str(raw.get("source_type") or "openclaw"))
|
||||
result = engine.process_feed_item(item)
|
||||
if result and not result.get("skipped"):
|
||||
ingested += 1
|
||||
summary = {"ingested": ingested, "enabled": True}
|
||||
else:
|
||||
with _data_lock:
|
||||
snapshot = dict(latest_data)
|
||||
summary = refresh_from_latest_data(snapshot, persist=True)
|
||||
|
||||
region = str(args.get("region", "") or "").strip().lower()
|
||||
data = {
|
||||
"refresh": summary,
|
||||
"heatmap_features": len((summary.get("sample") or [])),
|
||||
}
|
||||
if region:
|
||||
data["dossier"] = engine.get_dossier(region)
|
||||
else:
|
||||
data["heatmap"] = engine.get_risk_heatmap()
|
||||
data["clusters"] = engine.compute_herding_clusters()[:5]
|
||||
return {"ok": True, "data": data}
|
||||
|
||||
if cmd == "gt_backtest":
|
||||
from analytics.backtest import (
|
||||
DEFAULT_BACKTEST_ALERT_THRESHOLD,
|
||||
run_historical_backtest,
|
||||
tune_alert_threshold,
|
||||
)
|
||||
from analytics.historical_events import default_historical_cases, expanded_historical_cases
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED).",
|
||||
},
|
||||
}
|
||||
|
||||
expanded = bool(args.get("expanded", True))
|
||||
tune = bool(args.get("tune", False))
|
||||
include_cases = bool(args.get("include_cases", False))
|
||||
try:
|
||||
target_confidence = float(args.get("target_confidence", 0.95))
|
||||
except (TypeError, ValueError):
|
||||
target_confidence = 0.95
|
||||
|
||||
if tune:
|
||||
suite = expanded_historical_cases() if expanded else default_historical_cases()
|
||||
threshold, report = tune_alert_threshold(
|
||||
suite,
|
||||
target_confidence=target_confidence,
|
||||
)
|
||||
else:
|
||||
raw_threshold = args.get("alert_threshold")
|
||||
threshold = (
|
||||
float(raw_threshold)
|
||||
if raw_threshold is not None
|
||||
else DEFAULT_BACKTEST_ALERT_THRESHOLD
|
||||
)
|
||||
report = run_historical_backtest(
|
||||
use_expanded_suite=expanded,
|
||||
alert_threshold=threshold,
|
||||
target_confidence=target_confidence,
|
||||
)
|
||||
|
||||
data = report.to_dict()
|
||||
data["enabled"] = True
|
||||
data["expanded_suite"] = expanded
|
||||
data["tuned"] = tune
|
||||
data["recommended_alert_threshold"] = threshold
|
||||
if _wants_compact(args) or not include_cases:
|
||||
data.pop("cases", None)
|
||||
return {"ok": True, "data": data}
|
||||
|
||||
if cmd == "gt_rolling_freeze":
|
||||
from analytics.rolling_backtest import freeze_weekly_snapshot
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED).",
|
||||
},
|
||||
}
|
||||
|
||||
week_id = str(args.get("week_id", "") or "").strip() or None
|
||||
force = bool(args.get("force", False))
|
||||
result = freeze_weekly_snapshot(
|
||||
week_id=week_id,
|
||||
force=force,
|
||||
frozen_by="openclaw",
|
||||
)
|
||||
if not result.get("ok"):
|
||||
return {"ok": False, "detail": result.get("detail", "Freeze failed")}
|
||||
data = dict(result)
|
||||
data["enabled"] = True
|
||||
if _wants_compact(args):
|
||||
data.pop("snapshot", None)
|
||||
return {"ok": True, "data": data}
|
||||
|
||||
if cmd == "gt_rolling_label":
|
||||
from analytics.rolling_backtest import label_region, label_regions
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED).",
|
||||
},
|
||||
}
|
||||
|
||||
week_id = str(args.get("week_id", "") or "").strip()
|
||||
if not week_id:
|
||||
return {"ok": False, "detail": "week_id required"}
|
||||
|
||||
labels = args.get("labels")
|
||||
if isinstance(labels, list) and labels:
|
||||
result = label_regions(week_id, labels, labeled_by="openclaw")
|
||||
else:
|
||||
region = str(args.get("region", "") or "").strip().lower()
|
||||
label = str(args.get("label", "") or "").strip().lower()
|
||||
if not region or not label:
|
||||
return {"ok": False, "detail": "region and label required (or labels batch)"}
|
||||
result = label_region(
|
||||
week_id,
|
||||
region,
|
||||
label, # type: ignore[arg-type]
|
||||
notes=str(args.get("notes", "") or ""),
|
||||
labeled_by="openclaw",
|
||||
)
|
||||
|
||||
if not result.get("ok"):
|
||||
return {"ok": False, "detail": result.get("detail", "Label failed")}
|
||||
data = dict(result)
|
||||
data["enabled"] = True
|
||||
return {"ok": True, "data": data}
|
||||
|
||||
if cmd == "gt_rolling_backtest":
|
||||
from analytics.rolling_backtest import rolling_report
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED).",
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
weeks = int(args.get("weeks", 8))
|
||||
except (TypeError, ValueError):
|
||||
weeks = 8
|
||||
try:
|
||||
target_confidence = float(args.get("target_confidence", 0.80))
|
||||
except (TypeError, ValueError):
|
||||
target_confidence = 0.80
|
||||
|
||||
data = rolling_report(weeks=weeks, target_confidence=target_confidence)
|
||||
data["enabled"] = True
|
||||
if _wants_compact(args):
|
||||
for row in data.get("trend") or []:
|
||||
if isinstance(row, dict):
|
||||
row.pop("frozen_at", None)
|
||||
return {"ok": True, "data": data}
|
||||
|
||||
if cmd == "gt_top_alerts":
|
||||
from analytics.gt_alerts import top_gt_alerts
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED).",
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
limit = int(args.get("limit", 8))
|
||||
except (TypeError, ValueError):
|
||||
limit = 8
|
||||
|
||||
data = top_gt_alerts(limit=limit)
|
||||
data["enabled"] = True
|
||||
return {"ok": True, "data": data}
|
||||
|
||||
if cmd == "gt_micro_rolling":
|
||||
from analytics.micro_rolling import micro_rolling_report
|
||||
from analytics.settings import gt_analytics_enabled
|
||||
|
||||
if not gt_analytics_enabled():
|
||||
return {
|
||||
"ok": True,
|
||||
"data": {
|
||||
"enabled": False,
|
||||
"message": "Strategic Risk Analytics is disabled (GT_ANALYTICS_ENABLED).",
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
window_days = int(args.get("window_days", 3))
|
||||
except (TypeError, ValueError):
|
||||
window_days = 3
|
||||
try:
|
||||
limit = int(args.get("limit", 15))
|
||||
except (TypeError, ValueError):
|
||||
limit = 15
|
||||
|
||||
data = micro_rolling_report(window_days=window_days, limit=limit)
|
||||
data["enabled"] = True
|
||||
if _wants_compact(args):
|
||||
data.pop("top_regions", None)
|
||||
data["ignitions"] = (data.get("ignitions") or [])[:5]
|
||||
return {"ok": True, "data": data}
|
||||
|
||||
if cmd == "brief_area":
|
||||
from services.telemetry import entities_near, search_news, get_layer_slice
|
||||
lat = args.get("lat")
|
||||
@@ -1066,7 +1418,7 @@ def _dispatch_command(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
from services.openclaw_watchdog import add_watch
|
||||
watch_type = str(args.get("type", "")).strip()
|
||||
if not watch_type:
|
||||
return {"ok": False, "detail": "watch type required (track_aircraft, track_callsign, track_registration, track_ship, track_entity, geofence, keyword, prediction_market)"}
|
||||
return {"ok": False, "detail": "watch type required (track_aircraft, track_callsign, track_registration, track_ship, track_entity, geofence, keyword, telegram_rhetoric, prediction_market)"}
|
||||
watch_params = args.get("params", {})
|
||||
if not watch_params:
|
||||
# Allow flat args (e.g. {type: "track_callsign", callsign: "N189AM"})
|
||||
@@ -1092,6 +1444,7 @@ def _dispatch_command(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
owner=str(args.get("owner", "") or args.get("operator", "") or ""),
|
||||
layers=args.get("layers") if isinstance(args.get("layers"), (list, tuple)) else None,
|
||||
limit=5,
|
||||
fallback_search=True,
|
||||
)
|
||||
best = lookup.get("best_match") if isinstance(lookup.get("best_match"), dict) else {}
|
||||
group = str(best.get("group", "") or entity_type).lower()
|
||||
@@ -1543,6 +1896,85 @@ def _dispatch_command(cmd: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
count = clear_zones(source="openclaw")
|
||||
return {"ok": True, "data": {"removed_count": count}}
|
||||
|
||||
# -- Infonet / gate / DM (operator-delegated, full tier for writes) ------
|
||||
|
||||
if cmd == "infonet_status":
|
||||
from services.openclaw_infonet import get_infonet_status
|
||||
|
||||
return get_infonet_status()
|
||||
|
||||
if cmd == "ensure_infonet_ready":
|
||||
from services.openclaw_infonet import ensure_infonet_ready
|
||||
|
||||
return ensure_infonet_ready(join_swarm=bool(args.get("join_swarm", True)))
|
||||
|
||||
if cmd == "join_infonet_swarm":
|
||||
from services.openclaw_infonet import join_infonet_swarm
|
||||
|
||||
return join_infonet_swarm()
|
||||
|
||||
if cmd == "list_gates":
|
||||
from services.openclaw_infonet import list_gates
|
||||
|
||||
return list_gates()
|
||||
|
||||
if cmd == "read_gate_messages":
|
||||
from services.openclaw_infonet import read_gate_messages
|
||||
|
||||
gate_id = str(args.get("gate_id", "") or args.get("gate", "")).strip()
|
||||
return read_gate_messages(
|
||||
gate_id,
|
||||
limit=int(args.get("limit", 20) or 20),
|
||||
decrypt=bool(args.get("decrypt", False)),
|
||||
)
|
||||
|
||||
if cmd == "post_gate_message":
|
||||
from services.openclaw_infonet import post_gate_message
|
||||
|
||||
gate_id = str(args.get("gate_id", "") or args.get("gate", "")).strip()
|
||||
plaintext = str(args.get("plaintext", "") or args.get("message", "")).strip()
|
||||
return post_gate_message(
|
||||
gate_id,
|
||||
plaintext,
|
||||
reply_to=str(args.get("reply_to", "") or ""),
|
||||
)
|
||||
|
||||
if cmd == "cast_vote":
|
||||
from services.openclaw_infonet import cast_vote
|
||||
|
||||
target_id = str(args.get("target_id", "") or args.get("target", "")).strip()
|
||||
vote_raw = args.get("vote", args.get("direction"))
|
||||
try:
|
||||
vote_val = int(vote_raw)
|
||||
except (TypeError, ValueError):
|
||||
return {"ok": False, "detail": "vote must be 1 or -1"}
|
||||
return cast_vote(
|
||||
target_id,
|
||||
vote_val,
|
||||
gate=str(args.get("gate", "") or args.get("gate_id", "")).strip(),
|
||||
)
|
||||
|
||||
if cmd == "send_dm":
|
||||
from services.openclaw_infonet import send_dm
|
||||
|
||||
peer_id = str(
|
||||
args.get("peer_id", "")
|
||||
or args.get("recipient_id", "")
|
||||
or args.get("recipient", "")
|
||||
).strip()
|
||||
plaintext = str(args.get("plaintext", "") or args.get("message", "")).strip()
|
||||
return send_dm(
|
||||
peer_id,
|
||||
plaintext,
|
||||
delivery_class=str(args.get("delivery_class", "shared") or "shared"),
|
||||
recipient_token=str(args.get("recipient_token", "") or ""),
|
||||
)
|
||||
|
||||
if cmd == "poll_dms":
|
||||
from services.openclaw_infonet import poll_dms
|
||||
|
||||
return poll_dms(limit=int(args.get("limit", 20) or 20))
|
||||
|
||||
return {"ok": False, "detail": f"unhandled command: {cmd}"}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,796 @@
|
||||
"""OpenClaw agent delegation for private Infonet / gate / DM actions.
|
||||
|
||||
Agents authenticate with OpenClaw HMAC on the command channel. Write
|
||||
commands require ``OPENCLAW_ACCESS_TIER=full``. Actions use the operator's
|
||||
local wormhole persona and node runtime — the agent posts on behalf of the
|
||||
user who configured the skill, not as a separate fleet identity.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from starlette.requests import Request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _run_async(coro):
|
||||
try:
|
||||
asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
return asyncio.run(coro)
|
||||
return asyncio.run(coro)
|
||||
|
||||
|
||||
def _local_agent_request(path: str, *, method: str = "POST") -> Request:
|
||||
scope = {
|
||||
"type": "http",
|
||||
"method": method.upper(),
|
||||
"path": path,
|
||||
"headers": [],
|
||||
"client": ("127.0.0.1", 52421),
|
||||
}
|
||||
request = Request(scope)
|
||||
request.state._private_lane_current_tier = "private_strong"
|
||||
request.state._transport_tier = "private_strong"
|
||||
return request
|
||||
|
||||
|
||||
def ensure_infonet_ready(*, join_swarm: bool = True) -> dict[str, Any]:
|
||||
"""Warm Tor, enable the participant node, and optionally join the swarm."""
|
||||
from routers.ai_intel import _write_env_value
|
||||
from services.config import get_settings
|
||||
from services.mesh.mesh_swarm_runtime import join_swarm_with_retries
|
||||
from services.node_settings import read_node_settings, write_node_settings
|
||||
from services.tor_hidden_service import tor_service
|
||||
from services.wormhole_supervisor import _check_arti_ready
|
||||
|
||||
steps: dict[str, Any] = {}
|
||||
|
||||
tor_result = tor_service.start(target_port=8000)
|
||||
steps["tor"] = tor_result
|
||||
if tor_result.get("ok"):
|
||||
try:
|
||||
_write_env_value("MESH_ARTI_ENABLED", "true")
|
||||
get_settings.cache_clear()
|
||||
except Exception as exc:
|
||||
logger.debug("failed to persist MESH_ARTI_ENABLED: %s", exc)
|
||||
|
||||
if not _check_arti_ready():
|
||||
return {
|
||||
"ok": False,
|
||||
"detail": "Tor/Arti transport is not ready yet",
|
||||
"steps": steps,
|
||||
}
|
||||
|
||||
if not bool(read_node_settings().get("enabled")):
|
||||
write_node_settings(enabled=True)
|
||||
steps["node_enabled"] = True
|
||||
try:
|
||||
import main as main_mod
|
||||
|
||||
main_mod._refresh_node_peer_store()
|
||||
main_mod._start_infonet_node_runtime("openclaw_agent")
|
||||
except Exception as exc:
|
||||
logger.warning("node runtime start after agent enable failed: %s", exc)
|
||||
else:
|
||||
steps["node_enabled"] = True
|
||||
|
||||
if join_swarm:
|
||||
joined = join_swarm_with_retries()
|
||||
steps["announce"] = joined.get("announce") or {}
|
||||
steps["manifest_pull"] = joined.get("manifest_pull") or {}
|
||||
steps["swarm_attempts"] = joined.get("attempts")
|
||||
ok = bool(joined.get("ok"))
|
||||
else:
|
||||
ok = True
|
||||
|
||||
return {
|
||||
"ok": ok,
|
||||
"detail": "Infonet participant runtime ready" if ok else "swarm join incomplete",
|
||||
"steps": steps,
|
||||
"onion_address": str(tor_result.get("onion_address") or ""),
|
||||
}
|
||||
|
||||
|
||||
def join_infonet_swarm() -> dict[str, Any]:
|
||||
from services.mesh.mesh_swarm_runtime import join_swarm_with_retries
|
||||
|
||||
joined = join_swarm_with_retries()
|
||||
return {
|
||||
"ok": bool(joined.get("ok")),
|
||||
"announce": joined.get("announce") or {},
|
||||
"manifest_pull": joined.get("manifest_pull") or {},
|
||||
"attempts": joined.get("attempts"),
|
||||
"detail": joined.get("detail"),
|
||||
}
|
||||
|
||||
|
||||
def get_infonet_status() -> dict[str, Any]:
|
||||
from services.mesh.mesh_hashchain import infonet
|
||||
from services.wormhole_supervisor import get_wormhole_state
|
||||
|
||||
info = infonet.get_info()
|
||||
valid, reason = infonet.validate_chain(verify_signatures=False)
|
||||
try:
|
||||
wormhole = get_wormhole_state()
|
||||
except Exception:
|
||||
wormhole = {"configured": False, "ready": False, "arti_ready": False, "rns_ready": False}
|
||||
try:
|
||||
import main as main_mod
|
||||
|
||||
runtime = main_mod._node_runtime_snapshot()
|
||||
private_tier = main_mod._current_private_lane_tier(wormhole)
|
||||
except Exception:
|
||||
runtime = {}
|
||||
private_tier = "public_degraded"
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"chain": info,
|
||||
"valid": valid,
|
||||
"validation": reason,
|
||||
"private_lane_tier": private_tier,
|
||||
"wormhole": wormhole,
|
||||
"runtime": runtime,
|
||||
}
|
||||
|
||||
|
||||
def list_gates() -> dict[str, Any]:
|
||||
from services.mesh.mesh_reputation import gate_manager
|
||||
|
||||
return {"ok": True, "gates": gate_manager.list_gates()}
|
||||
|
||||
|
||||
def read_gate_messages(
|
||||
gate_id: str,
|
||||
*,
|
||||
limit: int = 20,
|
||||
decrypt: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
from services.mesh.mesh_hashchain import gate_store
|
||||
|
||||
gate_key = str(gate_id or "").strip().lower()
|
||||
if not gate_key:
|
||||
return {"ok": False, "detail": "gate_id required"}
|
||||
|
||||
messages, cursor = gate_store.get_messages_with_cursor(gate_key, limit=max(1, min(int(limit), 100)))
|
||||
out = []
|
||||
if decrypt:
|
||||
from services.mesh.mesh_gate_repair import decrypt_gate_message_with_repair
|
||||
|
||||
for msg in messages:
|
||||
item = dict(msg)
|
||||
try:
|
||||
decrypted = decrypt_gate_message_with_repair(
|
||||
gate_id=gate_key,
|
||||
epoch=int(item.get("epoch") or 0),
|
||||
ciphertext=str(item.get("ciphertext") or ""),
|
||||
nonce=str(item.get("nonce") or item.get("iv") or ""),
|
||||
sender_ref=str(item.get("sender_ref") or ""),
|
||||
gate_envelope=str(item.get("gate_envelope") or ""),
|
||||
envelope_hash=str(item.get("envelope_hash") or ""),
|
||||
event_id=str(item.get("event_id") or ""),
|
||||
)
|
||||
if decrypted.get("ok"):
|
||||
item["plaintext"] = decrypted.get("plaintext", "")
|
||||
except Exception as exc:
|
||||
item["decrypt_error"] = str(exc)
|
||||
out.append(item)
|
||||
else:
|
||||
out = [dict(m) for m in messages]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"gate": gate_key,
|
||||
"count": len(out),
|
||||
"cursor": cursor,
|
||||
"messages": out,
|
||||
}
|
||||
|
||||
|
||||
def post_gate_message(
|
||||
gate_id: str,
|
||||
plaintext: str,
|
||||
*,
|
||||
reply_to: str = "",
|
||||
) -> dict[str, Any]:
|
||||
"""Compose, sign, and post an MLS gate message using the operator persona."""
|
||||
from services.mesh.mesh_gate_repair import (
|
||||
compose_gate_message_with_repair,
|
||||
sign_gate_message_with_repair,
|
||||
)
|
||||
from services.mesh.mesh_wormhole_persona import bootstrap_wormhole_persona_state, create_gate_persona
|
||||
|
||||
gate_key = str(gate_id or "").strip().lower()
|
||||
if not gate_key:
|
||||
return {"ok": False, "detail": "gate_id required"}
|
||||
if not str(plaintext or "").strip():
|
||||
return {"ok": False, "detail": "plaintext required"}
|
||||
|
||||
bootstrap_wormhole_persona_state(force=False)
|
||||
try:
|
||||
create_gate_persona(gate_key, label="openclaw-agent")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
composed = compose_gate_message_with_repair(
|
||||
gate_id=gate_key,
|
||||
plaintext=str(plaintext),
|
||||
reply_to=str(reply_to or ""),
|
||||
)
|
||||
if not composed.get("ok"):
|
||||
return composed
|
||||
|
||||
signed = sign_gate_message_with_repair(
|
||||
gate_id=gate_key,
|
||||
epoch=int(composed.get("epoch") or 0),
|
||||
ciphertext=str(composed.get("ciphertext") or ""),
|
||||
nonce=str(composed.get("nonce") or ""),
|
||||
payload_format=str(composed.get("format") or "mls1"),
|
||||
reply_to=str(reply_to or ""),
|
||||
envelope_hash=str(composed.get("envelope_hash") or ""),
|
||||
transport_lock="private_strong",
|
||||
)
|
||||
if not signed.get("ok"):
|
||||
return signed
|
||||
|
||||
body = {
|
||||
"sender_id": str(signed.get("sender_id") or composed.get("sender_id") or ""),
|
||||
"public_key": str(signed.get("public_key") or composed.get("public_key") or ""),
|
||||
"public_key_algo": str(signed.get("public_key_algo") or composed.get("public_key_algo") or ""),
|
||||
"signature": str(signed.get("signature") or ""),
|
||||
"sequence": int(signed.get("sequence") or composed.get("sequence") or 0),
|
||||
"protocol_version": str(signed.get("protocol_version") or composed.get("protocol_version") or ""),
|
||||
"epoch": int(signed.get("epoch") or composed.get("epoch") or 0),
|
||||
"ciphertext": str(signed.get("ciphertext") or composed.get("ciphertext") or ""),
|
||||
"nonce": str(signed.get("nonce") or composed.get("nonce") or ""),
|
||||
"sender_ref": str(signed.get("sender_ref") or composed.get("sender_ref") or ""),
|
||||
"format": str(signed.get("format") or composed.get("format") or "mls1"),
|
||||
"gate_envelope": str(signed.get("gate_envelope") or composed.get("gate_envelope") or ""),
|
||||
"envelope_hash": str(signed.get("envelope_hash") or composed.get("envelope_hash") or ""),
|
||||
"transport_lock": "private_strong",
|
||||
"reply_to": str(signed.get("reply_to") or reply_to or ""),
|
||||
}
|
||||
|
||||
import main as main_mod
|
||||
|
||||
path = f"/api/mesh/gate/{gate_key}/message"
|
||||
request = _local_agent_request(path)
|
||||
return main_mod._submit_gate_message_envelope(request, gate_key, body)
|
||||
|
||||
|
||||
def cast_vote(
|
||||
target_id: str,
|
||||
vote: int,
|
||||
*,
|
||||
gate: str = "",
|
||||
) -> dict[str, Any]:
|
||||
"""Cast a signed reputation vote using the operator gate/transport persona."""
|
||||
from services.mesh.mesh_hashchain import infonet
|
||||
from services.mesh.mesh_protocol import PROTOCOL_VERSION, normalize_payload
|
||||
from services.mesh.mesh_reputation import gate_manager, reputation_ledger
|
||||
from services.mesh.mesh_wormhole_persona import (
|
||||
bootstrap_wormhole_persona_state,
|
||||
sign_gate_wormhole_event,
|
||||
sign_public_wormhole_event,
|
||||
)
|
||||
|
||||
voter_gate = str(gate or "").strip().lower()
|
||||
target = str(target_id or "").strip()
|
||||
vote_val = int(vote)
|
||||
if not target:
|
||||
return {"ok": False, "detail": "target_id required"}
|
||||
if vote_val not in (1, -1):
|
||||
return {"ok": False, "detail": "vote must be 1 or -1"}
|
||||
|
||||
bootstrap_wormhole_persona_state(force=False)
|
||||
vote_payload = {"target_id": target, "vote": vote_val, "gate": voter_gate}
|
||||
normalized = normalize_payload("vote", vote_payload)
|
||||
ok_payload, reason = True, "ok"
|
||||
from services.mesh.mesh_schema import validate_event_payload
|
||||
|
||||
ok_payload, reason = validate_event_payload("vote", normalized)
|
||||
if not ok_payload:
|
||||
return {"ok": False, "detail": reason}
|
||||
|
||||
if voter_gate:
|
||||
signed = sign_gate_wormhole_event(
|
||||
gate_id=voter_gate,
|
||||
event_type="vote",
|
||||
payload=normalized,
|
||||
)
|
||||
else:
|
||||
signed = sign_public_wormhole_event(event_type="vote", payload=normalized)
|
||||
|
||||
if not signed.get("ok", True):
|
||||
return signed
|
||||
|
||||
voter_id = str(signed.get("node_id") or "")
|
||||
public_key = str(signed.get("public_key") or "")
|
||||
public_key_algo = str(signed.get("public_key_algo") or "")
|
||||
signature = str(signed.get("signature") or "")
|
||||
sequence = int(signed.get("sequence") or 0)
|
||||
|
||||
if voter_gate:
|
||||
can_enter, enter_reason = gate_manager.can_enter(voter_id, voter_gate)
|
||||
if not can_enter:
|
||||
return {"ok": False, "detail": f"Gate vote denied: {enter_reason}"}
|
||||
|
||||
reputation_ledger.register_node(voter_id, public_key, public_key_algo)
|
||||
stable_voter_id = voter_id
|
||||
try:
|
||||
import main as main_mod
|
||||
|
||||
root_nid = main_mod._cached_root_node_id()
|
||||
if root_nid:
|
||||
stable_voter_id = root_nid
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
ok, cast_reason, weight = reputation_ledger.cast_vote(
|
||||
stable_voter_id,
|
||||
target,
|
||||
vote_val,
|
||||
voter_gate,
|
||||
)
|
||||
if ok:
|
||||
try:
|
||||
infonet.append(
|
||||
event_type="vote",
|
||||
node_id=voter_id,
|
||||
payload=normalized,
|
||||
signature=signature,
|
||||
sequence=sequence,
|
||||
public_key=public_key,
|
||||
public_key_algo=public_key_algo,
|
||||
protocol_version=str(signed.get("protocol_version") or PROTOCOL_VERSION),
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("vote recorded in ledger but infonet append failed: %s", exc)
|
||||
|
||||
return {"ok": ok, "detail": cast_reason, "weight": round(float(weight or 0), 2)}
|
||||
|
||||
|
||||
def _http_post_json(
|
||||
url: str,
|
||||
body: dict[str, Any],
|
||||
*,
|
||||
extra_headers: dict[str, str] | None = None,
|
||||
timeout: int = 120,
|
||||
) -> dict[str, Any]:
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
payload_bytes = json.dumps(body, separators=(",", ":"), sort_keys=True).encode("utf-8")
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if extra_headers:
|
||||
headers.update(extra_headers)
|
||||
req = urllib.request.Request(url, data=payload_bytes, headers=headers, method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")
|
||||
try:
|
||||
parsed = json.loads(detail)
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
except Exception:
|
||||
pass
|
||||
return {"ok": False, "detail": detail or f"http {exc.code}"}
|
||||
if not raw:
|
||||
return {}
|
||||
parsed = json.loads(raw)
|
||||
return parsed if isinstance(parsed, dict) else {"ok": False, "detail": "invalid json response"}
|
||||
|
||||
|
||||
def _issue_sender_token_for_http_send(
|
||||
api_base: str,
|
||||
*,
|
||||
recipient: str,
|
||||
delivery: str,
|
||||
recipient_token: str,
|
||||
) -> dict[str, Any]:
|
||||
extra_headers: dict[str, str] = {}
|
||||
admin_key = str(os.environ.get("ADMIN_KEY") or "").strip()
|
||||
if admin_key:
|
||||
extra_headers["X-Admin-Key"] = admin_key
|
||||
return _http_post_json(
|
||||
f"{api_base}/api/wormhole/dm/sender-token",
|
||||
{
|
||||
"recipient_id": recipient,
|
||||
"delivery_class": delivery,
|
||||
"recipient_token": recipient_token,
|
||||
},
|
||||
extra_headers=extra_headers or None,
|
||||
)
|
||||
|
||||
|
||||
def _submit_signed_dm_send(
|
||||
*,
|
||||
recipient: str,
|
||||
delivery_class: str,
|
||||
recipient_token: str,
|
||||
ciphertext: str,
|
||||
payload_format: str,
|
||||
session_welcome: str = "",
|
||||
connect_intent: str = "",
|
||||
lookup_peer_url: str = "",
|
||||
peer_dh_pub: str = "",
|
||||
) -> dict[str, Any]:
|
||||
import main as main_mod
|
||||
from services.mesh.mesh_protocol import (
|
||||
PROTOCOL_VERSION,
|
||||
SIGNED_CONTEXT_FIELD,
|
||||
build_signed_context,
|
||||
)
|
||||
from services.mesh.mesh_schema import validate_event_payload
|
||||
from services.mesh.mesh_wormhole_persona import get_dm_identity, sign_dm_wormhole_event
|
||||
from services.mesh.mesh_wormhole_sender_token import issue_wormhole_dm_sender_token
|
||||
|
||||
delivery = str(delivery_class or "shared").strip().lower()
|
||||
identity = get_dm_identity()
|
||||
sender_id = str(identity.get("node_id") or "")
|
||||
msg_id = secrets.token_hex(16)
|
||||
timestamp = int(time.time())
|
||||
sequence = int(identity.get("sequence", 0) or 0) + 1
|
||||
|
||||
dm_payload: dict[str, Any] = {
|
||||
"recipient_id": recipient,
|
||||
"delivery_class": delivery,
|
||||
"recipient_token": str(recipient_token or ""),
|
||||
"ciphertext": str(ciphertext or ""),
|
||||
"msg_id": msg_id,
|
||||
"timestamp": timestamp,
|
||||
"format": str(payload_format or "mls1"),
|
||||
"transport_lock": "private_strong",
|
||||
}
|
||||
if session_welcome:
|
||||
dm_payload["session_welcome"] = str(session_welcome)
|
||||
|
||||
try:
|
||||
from services.config import get_settings
|
||||
from services.mesh.mesh_wormhole_seal import build_sender_seal
|
||||
|
||||
if (
|
||||
delivery == "shared"
|
||||
and bool(get_settings().MESH_DM_REQUIRE_SENDER_SEAL_SHARED)
|
||||
and not str(dm_payload.get("sender_seal", "") or "").strip()
|
||||
):
|
||||
seal = build_sender_seal(
|
||||
recipient_id=recipient,
|
||||
recipient_dh_pub=str(peer_dh_pub or ""),
|
||||
msg_id=msg_id,
|
||||
timestamp=timestamp,
|
||||
)
|
||||
if seal.get("ok"):
|
||||
dm_payload["sender_seal"] = str(seal.get("sender_seal") or "")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
ok_payload, reason = validate_event_payload("dm_message", dm_payload)
|
||||
if not ok_payload:
|
||||
return {"ok": False, "detail": reason}
|
||||
|
||||
dm_payload[SIGNED_CONTEXT_FIELD] = build_signed_context(
|
||||
event_type="dm_message",
|
||||
kind="dm_send",
|
||||
endpoint="/api/mesh/dm/send",
|
||||
lane_floor="private_strong",
|
||||
sequence_domain="dm_send",
|
||||
node_id=sender_id,
|
||||
sequence=sequence,
|
||||
payload=dm_payload,
|
||||
recipient_id=recipient,
|
||||
)
|
||||
signed = sign_dm_wormhole_event(
|
||||
event_type="dm_message",
|
||||
payload=dm_payload,
|
||||
sequence=sequence,
|
||||
)
|
||||
if not signed.get("ok", True):
|
||||
return signed
|
||||
|
||||
body = {
|
||||
"sender_id": sender_id,
|
||||
"sender_token": "",
|
||||
"recipient_id": recipient,
|
||||
"delivery_class": delivery,
|
||||
"recipient_token": str(recipient_token or ""),
|
||||
"ciphertext": str(ciphertext or ""),
|
||||
"format": str(payload_format or "mls1"),
|
||||
"transport_lock": "private_strong",
|
||||
"session_welcome": str(session_welcome or ""),
|
||||
"msg_id": msg_id,
|
||||
"timestamp": timestamp,
|
||||
"sender_seal": str(dm_payload.get("sender_seal") or ""),
|
||||
"public_key": str(signed.get("public_key") or ""),
|
||||
"public_key_algo": str(signed.get("public_key_algo") or ""),
|
||||
"signature": str(signed.get("signature") or ""),
|
||||
"sequence": int(signed.get("sequence") or 0),
|
||||
"protocol_version": str(signed.get("protocol_version") or PROTOCOL_VERSION),
|
||||
"signed_context": dict(dm_payload.get(SIGNED_CONTEXT_FIELD) or {}),
|
||||
}
|
||||
normalized_intent = str(connect_intent or "").strip().lower()
|
||||
normalized_lookup_peer = str(lookup_peer_url or "").strip().rstrip("/")
|
||||
if normalized_intent:
|
||||
body["connect_intent"] = normalized_intent
|
||||
if normalized_lookup_peer:
|
||||
body["lookup_peer_url"] = normalized_lookup_peer
|
||||
|
||||
api_base = str(os.environ.get("SB_API_BASE", "http://127.0.0.1:8000") or "http://127.0.0.1:8000").rstrip("/")
|
||||
result: dict[str, Any] = {"ok": False, "detail": "dm send failed"}
|
||||
try:
|
||||
import urllib.error
|
||||
|
||||
if delivery in ("request", "shared"):
|
||||
issued = _issue_sender_token_for_http_send(
|
||||
api_base,
|
||||
recipient=recipient,
|
||||
delivery=delivery,
|
||||
recipient_token=str(recipient_token or ""),
|
||||
)
|
||||
if not issued.get("ok"):
|
||||
return issued
|
||||
body["sender_token"] = str(issued.get("sender_token") or "")
|
||||
|
||||
result = _http_post_json(f"{api_base}/api/mesh/dm/send", body)
|
||||
except (urllib.error.URLError, TimeoutError):
|
||||
if delivery in ("request", "shared"):
|
||||
issued = issue_wormhole_dm_sender_token(
|
||||
recipient_id=recipient,
|
||||
delivery_class=delivery,
|
||||
recipient_token=str(recipient_token or ""),
|
||||
)
|
||||
if not issued.get("ok"):
|
||||
return issued
|
||||
body["sender_token"] = str(issued.get("sender_token") or "")
|
||||
|
||||
async def _send():
|
||||
import json as _json
|
||||
|
||||
raw = _json.dumps(body).encode("utf-8")
|
||||
|
||||
async def receive():
|
||||
return {"type": "http.request", "body": raw, "more_body": False}
|
||||
|
||||
req = Request(
|
||||
{
|
||||
"type": "http",
|
||||
"method": "POST",
|
||||
"path": "/api/mesh/dm/send",
|
||||
"headers": [(b"content-type", b"application/json")],
|
||||
"client": ("127.0.0.1", 52421),
|
||||
},
|
||||
receive,
|
||||
)
|
||||
req.state._private_lane_current_tier = "private_strong"
|
||||
req.state._transport_tier = "private_strong"
|
||||
return await main_mod.dm_send(req)
|
||||
|
||||
result = _run_async(_send())
|
||||
except Exception as exc:
|
||||
result = {"ok": False, "detail": str(exc) or type(exc).__name__}
|
||||
if isinstance(result, dict):
|
||||
result.setdefault("msg_id", msg_id)
|
||||
result.setdefault("sender_id", sender_id)
|
||||
result.setdefault("recipient_id", recipient)
|
||||
return result
|
||||
|
||||
|
||||
def send_contact_request(
|
||||
*,
|
||||
lookup_token: str = "",
|
||||
peer_id: str = "",
|
||||
note: str = "",
|
||||
lookup_peer_url: str = "",
|
||||
cached_prekey_bundle: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Send a first-contact request using a short address or peer id."""
|
||||
from services.mesh.mesh_wormhole_dead_drop import build_contact_offer
|
||||
from services.mesh.mesh_wormhole_persona import get_dm_identity
|
||||
from services.mesh.mesh_wormhole_prekey import bootstrap_encrypt_for_peer, fetch_dm_prekey_bundle
|
||||
|
||||
token = str(lookup_token or "").strip()
|
||||
peer = str(peer_id or "").strip()
|
||||
if not token and not peer:
|
||||
return {"ok": False, "detail": "lookup_token or peer_id required"}
|
||||
|
||||
preferred_peer = str(lookup_peer_url or "").strip().rstrip("/")
|
||||
if cached_prekey_bundle and cached_prekey_bundle.get("ok"):
|
||||
bundle = dict(cached_prekey_bundle)
|
||||
else:
|
||||
bundle = fetch_dm_prekey_bundle(
|
||||
agent_id=peer if not token else "",
|
||||
lookup_token=token,
|
||||
lookup_peer_urls=[preferred_peer] if preferred_peer else None,
|
||||
)
|
||||
if not bundle.get("ok"):
|
||||
return bundle
|
||||
recipient = str(bundle.get("agent_id") or peer).strip()
|
||||
if not recipient:
|
||||
return {"ok": False, "detail": "recipient unresolved"}
|
||||
|
||||
identity = get_dm_identity()
|
||||
offer = build_contact_offer(
|
||||
dh_pub_key=str(identity.get("dh_pub_key") or ""),
|
||||
dh_algo=str(identity.get("dh_algo") or "X25519"),
|
||||
geo_hint=str(note or ""),
|
||||
)
|
||||
encrypted = bootstrap_encrypt_for_peer(
|
||||
recipient,
|
||||
offer,
|
||||
fetched_bundle=bundle,
|
||||
)
|
||||
if not encrypted.get("ok"):
|
||||
return encrypted
|
||||
|
||||
return _submit_signed_dm_send(
|
||||
recipient=recipient,
|
||||
delivery_class="request",
|
||||
recipient_token="",
|
||||
ciphertext=str(encrypted.get("result") or ""),
|
||||
payload_format="mls1",
|
||||
connect_intent="contact_request",
|
||||
lookup_peer_url=preferred_peer,
|
||||
)
|
||||
|
||||
|
||||
def send_contact_accept(
|
||||
*,
|
||||
peer_id: str,
|
||||
peer_dh_pub: str = "",
|
||||
lookup_token: str = "",
|
||||
lookup_peer_url: str = "",
|
||||
) -> dict[str, Any]:
|
||||
"""Accept a pending contact request and open the shared DM lane."""
|
||||
from services.mesh.mesh_wormhole_dead_drop import build_contact_accept, issue_pairwise_dm_alias
|
||||
from services.mesh.mesh_wormhole_prekey import bootstrap_encrypt_for_peer, fetch_dm_prekey_bundle
|
||||
|
||||
peer = str(peer_id or "").strip()
|
||||
if not peer:
|
||||
return {"ok": False, "detail": "peer_id required"}
|
||||
|
||||
token = str(lookup_token or "").strip()
|
||||
preferred_peer = str(lookup_peer_url or "").strip().rstrip("/")
|
||||
dh_pub = str(peer_dh_pub or "").strip()
|
||||
if not dh_pub:
|
||||
bundle = fetch_dm_prekey_bundle(
|
||||
agent_id=peer if not token else "",
|
||||
lookup_token=token,
|
||||
lookup_peer_urls=[preferred_peer] if preferred_peer else None,
|
||||
)
|
||||
if not bundle.get("ok"):
|
||||
return bundle
|
||||
dh_pub = str(bundle.get("dh_pub_key") or "").strip()
|
||||
if not dh_pub:
|
||||
return {"ok": False, "detail": "peer dh_pub_key unavailable"}
|
||||
|
||||
alias = issue_pairwise_dm_alias(peer_id=peer, peer_dh_pub=dh_pub)
|
||||
if not alias.get("ok"):
|
||||
return alias
|
||||
shared_alias = str(alias.get("shared_alias") or "").strip()
|
||||
if not shared_alias:
|
||||
return {"ok": False, "detail": "shared_alias unavailable"}
|
||||
|
||||
accept_plain = build_contact_accept(shared_alias=shared_alias)
|
||||
encrypted = bootstrap_encrypt_for_peer(peer, accept_plain, lookup_token=token)
|
||||
if not encrypted.get("ok"):
|
||||
return encrypted
|
||||
|
||||
sent = _submit_signed_dm_send(
|
||||
recipient=peer,
|
||||
delivery_class="request",
|
||||
recipient_token="",
|
||||
ciphertext=str(encrypted.get("result") or ""),
|
||||
payload_format="mls1",
|
||||
connect_intent="contact_accept",
|
||||
lookup_peer_url=preferred_peer,
|
||||
)
|
||||
if isinstance(sent, dict):
|
||||
sent.setdefault("shared_alias", shared_alias)
|
||||
return sent
|
||||
|
||||
|
||||
def send_dm(
|
||||
peer_id: str,
|
||||
plaintext: str,
|
||||
*,
|
||||
delivery_class: str = "shared",
|
||||
recipient_token: str = "",
|
||||
) -> dict[str, Any]:
|
||||
"""Compose and send an encrypted DM on behalf of the operator."""
|
||||
import main as main_mod
|
||||
|
||||
recipient = str(peer_id or "").strip()
|
||||
if not recipient:
|
||||
return {"ok": False, "detail": "peer_id required"}
|
||||
if not str(plaintext or "").strip():
|
||||
return {"ok": False, "detail": "plaintext required"}
|
||||
|
||||
delivery = str(delivery_class or "shared").strip().lower()
|
||||
if delivery not in ("shared", "request"):
|
||||
return {"ok": False, "detail": "delivery_class must be shared or request"}
|
||||
|
||||
composed = main_mod.compose_wormhole_dm(
|
||||
peer_id=recipient,
|
||||
peer_dh_pub="",
|
||||
plaintext=str(plaintext),
|
||||
)
|
||||
if not composed.get("ok"):
|
||||
return composed
|
||||
|
||||
return _submit_signed_dm_send(
|
||||
recipient=recipient,
|
||||
delivery_class=delivery,
|
||||
recipient_token=str(recipient_token or ""),
|
||||
ciphertext=str(composed.get("ciphertext") or ""),
|
||||
payload_format=str(composed.get("format") or "mls1"),
|
||||
session_welcome=str(composed.get("session_welcome") or ""),
|
||||
)
|
||||
|
||||
|
||||
def poll_dms(*, limit: int = 20) -> dict[str, Any]:
|
||||
"""Poll encrypted DMs for the operator DM identity."""
|
||||
import json
|
||||
|
||||
import main as main_mod
|
||||
from services.mesh.mesh_protocol import PROTOCOL_VERSION
|
||||
from services.mesh.mesh_wormhole_persona import get_dm_identity, sign_dm_wormhole_event
|
||||
|
||||
identity = get_dm_identity()
|
||||
agent_id = str(identity.get("node_id") or "")
|
||||
if not agent_id:
|
||||
return {"ok": False, "detail": "dm identity is not configured"}
|
||||
|
||||
poll_payload = {"mailbox_claims": [], "agent_id": agent_id}
|
||||
signed = sign_dm_wormhole_event(event_type="dm_poll", payload=poll_payload)
|
||||
if not signed.get("ok", True):
|
||||
return signed
|
||||
|
||||
body = {
|
||||
"agent_id": agent_id,
|
||||
"mailbox_claims": [],
|
||||
"timestamp": int(time.time()),
|
||||
"nonce": secrets.token_hex(8),
|
||||
"public_key": str(signed.get("public_key") or ""),
|
||||
"public_key_algo": str(signed.get("public_key_algo") or ""),
|
||||
"signature": str(signed.get("signature") or ""),
|
||||
"sequence": int(signed.get("sequence") or 0),
|
||||
"protocol_version": str(signed.get("protocol_version") or PROTOCOL_VERSION),
|
||||
}
|
||||
|
||||
raw = json.dumps(body).encode("utf-8")
|
||||
|
||||
async def _poll():
|
||||
async def receive():
|
||||
return {"type": "http.request", "body": raw, "more_body": False}
|
||||
|
||||
req = Request(
|
||||
{
|
||||
"type": "http",
|
||||
"method": "POST",
|
||||
"path": "/api/mesh/dm/poll",
|
||||
"headers": [(b"content-type", b"application/json")],
|
||||
"client": ("127.0.0.1", 52421),
|
||||
},
|
||||
receive,
|
||||
)
|
||||
return await main_mod.dm_poll_secure(req)
|
||||
|
||||
result = _run_async(_poll())
|
||||
if isinstance(result, dict):
|
||||
messages = list(result.get("messages") or [])
|
||||
if limit and len(messages) > int(limit):
|
||||
result = dict(result)
|
||||
result["messages"] = messages[: int(limit)]
|
||||
result["count"] = len(result["messages"])
|
||||
return result if isinstance(result, dict) else {"ok": False, "detail": "dm poll failed"}
|
||||
@@ -0,0 +1,705 @@
|
||||
"""Deterministic OpenClaw routing — intent → fastest command.
|
||||
|
||||
Keeps expensive fuzzy scans and full-layer dumps out of the default agent path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
EXPENSIVE_COMMANDS = frozenset({
|
||||
"search_telemetry",
|
||||
"get_telemetry",
|
||||
"get_slow_telemetry",
|
||||
"get_report",
|
||||
})
|
||||
|
||||
EXPENSIVE_GATE_MESSAGE = (
|
||||
"expensive command blocked — use route_query, find_entity, run_playbook, or targeted reads. "
|
||||
"Pass confirm_expensive=true only when fuzzy search or full dumps are intentional."
|
||||
)
|
||||
|
||||
LATENCY_TIER_MS: dict[str, int] = {
|
||||
"channel_status": 5,
|
||||
"route_query": 5,
|
||||
"get_summary": 10,
|
||||
"what_changed": 15,
|
||||
"search_news": 15,
|
||||
"find_flights": 25,
|
||||
"find_ships": 25,
|
||||
"find_entity": 30,
|
||||
"entities_near": 30,
|
||||
"brief_area": 30,
|
||||
"get_layer_slice": 50,
|
||||
"correlate_entity": 15,
|
||||
"entity_expand": 40,
|
||||
"osint_lookup": 200,
|
||||
"run_playbook": 120,
|
||||
"gt_risk_heatmap": 20,
|
||||
"gt_dossier": 25,
|
||||
"gt_analyze": 80,
|
||||
"gt_backtest": 120,
|
||||
"gt_rolling_freeze": 30,
|
||||
"gt_rolling_label": 20,
|
||||
"gt_rolling_backtest": 30,
|
||||
"gt_micro_rolling": 20,
|
||||
"infonet_status": 20,
|
||||
"list_gates": 15,
|
||||
"read_gate_messages": 40,
|
||||
"poll_dms": 80,
|
||||
"ensure_infonet_ready": 120000,
|
||||
"join_infonet_swarm": 90000,
|
||||
"post_gate_message": 15000,
|
||||
"cast_vote": 5000,
|
||||
"send_dm": 20000,
|
||||
"search_telemetry": 8000,
|
||||
"get_telemetry": 3500,
|
||||
"get_slow_telemetry": 1500,
|
||||
"get_report": 5000,
|
||||
}
|
||||
|
||||
RE_N_NUMBER = re.compile(r"\bN\d{1,5}[A-Z]{0,2}\b", re.I)
|
||||
RE_CALLSIGN = re.compile(r"\b[A-Z]{2,4}\d{1,4}[A-Z]?\b")
|
||||
RE_MMSI = re.compile(r"\b\d{9}\b")
|
||||
RE_CVE = re.compile(r"\bCVE-\d{4}-\d+\b", re.I)
|
||||
RE_IPV4 = re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b")
|
||||
RE_DOMAIN = re.compile(
|
||||
r"\b(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+(?:[a-z]{2,})\b",
|
||||
re.I,
|
||||
)
|
||||
|
||||
KNOWN_CALLSIGNS = frozenset({
|
||||
"AF1", "AF2", "EXEC1", "EXEC2", "SAM", "STALK52", "SPAR19", "SPAR20",
|
||||
})
|
||||
|
||||
PLAYBOOKS: dict[str, dict[str, Any]] = {
|
||||
"hot_snapshot": {
|
||||
"description": "Summary + hot layers + what changed (one batch)",
|
||||
"batch": [
|
||||
{"cmd": "get_summary", "args": {"compact": True}},
|
||||
{
|
||||
"cmd": "get_layer_slice",
|
||||
"args": {
|
||||
"layers": [
|
||||
"news",
|
||||
"telegram_osint",
|
||||
"military_flights",
|
||||
"private_jets",
|
||||
"earthquakes",
|
||||
],
|
||||
"limit_per_layer": 10,
|
||||
"compact": True,
|
||||
},
|
||||
},
|
||||
{"cmd": "what_changed", "args": {"compact": True}},
|
||||
],
|
||||
},
|
||||
"status_check": {
|
||||
"description": "Channel health + layer counts",
|
||||
"batch": [
|
||||
{"cmd": "channel_status", "args": {}},
|
||||
{"cmd": "get_summary", "args": {"compact": True}},
|
||||
],
|
||||
},
|
||||
"morning_brief": {
|
||||
"description": "Operator morning digest layers",
|
||||
"batch": [
|
||||
{"cmd": "get_summary", "args": {"compact": True}},
|
||||
{"cmd": "what_changed", "args": {"compact": True}},
|
||||
{
|
||||
"cmd": "get_layer_slice",
|
||||
"args": {
|
||||
"layers": [
|
||||
"news",
|
||||
"telegram_osint",
|
||||
"gdelt",
|
||||
"earthquakes",
|
||||
"crowdthreat",
|
||||
"military_flights",
|
||||
],
|
||||
"limit_per_layer": 15,
|
||||
"compact": True,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
"monitor_heartbeat": {
|
||||
"description": "Low-latency monitor poll (replaces full telemetry pull)",
|
||||
"batch": [
|
||||
{"cmd": "what_changed", "args": {"compact": True}},
|
||||
{
|
||||
"cmd": "get_layer_slice",
|
||||
"args": {
|
||||
"layers": [
|
||||
"military_flights",
|
||||
"ships",
|
||||
"earthquakes",
|
||||
"liveuamap",
|
||||
"crowdthreat",
|
||||
"uap_sightings",
|
||||
"firms_fires",
|
||||
"gps_jamming",
|
||||
"wastewater",
|
||||
],
|
||||
"limit_per_layer": 200,
|
||||
"compact": True,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def routing_manifest() -> dict[str, Any]:
|
||||
"""Machine-readable routing hints for /api/ai/capabilities."""
|
||||
return {
|
||||
"default_read": "find_entity",
|
||||
"preferred_entry": "route_query",
|
||||
"client_wrapper": "ShadowBrokerClient.ask",
|
||||
"batch_playbook": "run_playbook",
|
||||
"last_resort": "search_telemetry",
|
||||
"expensive_commands": sorted(EXPENSIVE_COMMANDS),
|
||||
"latency_tier_ms": LATENCY_TIER_MS,
|
||||
"anti_patterns": [
|
||||
"search_telemetry for known tail numbers, callsigns, owners, or MMSI",
|
||||
"get_telemetry for routine reads — use get_layer_slice or run_playbook hot_snapshot",
|
||||
"sequential send_command loops — use send_batch or run_playbook",
|
||||
"/api/health for liveness — use channel_status",
|
||||
"empty layers: [] on get_layer_slice — pass explicit layer names",
|
||||
],
|
||||
"recipes": [
|
||||
{
|
||||
"intent": "natural language question",
|
||||
"use": "route_query → recommended cmd, or ShadowBrokerClient.ask()",
|
||||
},
|
||||
{
|
||||
"intent": "known person/aircraft",
|
||||
"use": "find_entity(query=...) or find_flights(owner=...)",
|
||||
},
|
||||
{
|
||||
"intent": "news / telegram topic",
|
||||
"use": "search_news(query=...)",
|
||||
},
|
||||
{
|
||||
"intent": "near a point",
|
||||
"use": "entities_near or brief_area",
|
||||
},
|
||||
{
|
||||
"intent": "hot snapshot",
|
||||
"use": "run_playbook(name=hot_snapshot)",
|
||||
},
|
||||
{
|
||||
"intent": "post to infonet gate / join swarm",
|
||||
"use": "ensure_infonet_ready then post_gate_message (full tier)",
|
||||
},
|
||||
{
|
||||
"intent": "read encrypted gate traffic",
|
||||
"use": "read_gate_messages(gate_id=infonet, decrypt=true)",
|
||||
},
|
||||
{
|
||||
"intent": "dm another node",
|
||||
"use": "send_dm(peer_id=..., plaintext=...) (full tier)",
|
||||
},
|
||||
],
|
||||
"playbooks": {
|
||||
name: {"description": spec.get("description", "")}
|
||||
for name, spec in PLAYBOOKS.items()
|
||||
},
|
||||
"agent_surface": {
|
||||
"primary": ["ask", "send_batch", "channel_status"],
|
||||
"writes": [
|
||||
"place_pin",
|
||||
"add_watch",
|
||||
"inject_data",
|
||||
"place_analysis_zone",
|
||||
"ensure_infonet_ready",
|
||||
"post_gate_message",
|
||||
"cast_vote",
|
||||
"send_dm",
|
||||
],
|
||||
"infonet_reads": [
|
||||
"infonet_status",
|
||||
"list_gates",
|
||||
"read_gate_messages",
|
||||
"poll_dms",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def requires_expensive_confirm(cmd: str, args: dict[str, Any] | None) -> bool:
|
||||
if cmd not in EXPENSIVE_COMMANDS:
|
||||
return False
|
||||
if isinstance(args, dict) and args.get("confirm_expensive") is True:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _compact_args(args: dict[str, Any], *, compact: bool) -> dict[str, Any]:
|
||||
out = dict(args)
|
||||
if compact and "compact" not in out:
|
||||
out["compact"] = True
|
||||
return out
|
||||
|
||||
|
||||
def _estimate_ms(cmd: str) -> int:
|
||||
return int(LATENCY_TIER_MS.get(cmd, 100))
|
||||
|
||||
|
||||
def _news_query(text: str) -> str:
|
||||
cleaned = text
|
||||
for prefix in (
|
||||
"news about",
|
||||
"news on",
|
||||
"telegram",
|
||||
"headlines about",
|
||||
"headlines on",
|
||||
"latest on",
|
||||
"search news for",
|
||||
):
|
||||
if cleaned.lower().startswith(prefix):
|
||||
cleaned = cleaned[len(prefix):].strip()
|
||||
return cleaned.strip(" ?.")
|
||||
|
||||
|
||||
def _gt_region_hint(text: str) -> str:
|
||||
lowered = str(text or "").lower()
|
||||
hints = (
|
||||
"ukraine",
|
||||
"middle east",
|
||||
"eastern europe",
|
||||
"baltics",
|
||||
"israel",
|
||||
"iran",
|
||||
"russia",
|
||||
"china",
|
||||
"europe",
|
||||
"united kingdom",
|
||||
"uk",
|
||||
"usa",
|
||||
"united states",
|
||||
)
|
||||
for hint in hints:
|
||||
if hint in lowered:
|
||||
return "uk" if hint == "united kingdom" else hint
|
||||
match = re.search(r"\bon\s+([a-z][a-z\s]{2,30})\b", lowered)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return ""
|
||||
|
||||
|
||||
def route_query(
|
||||
text: str = "",
|
||||
*,
|
||||
lat: float | None = None,
|
||||
lng: float | None = None,
|
||||
radius_km: float = 50,
|
||||
compact: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""Map natural-language intent to the fastest command (no LLM)."""
|
||||
raw = str(text or "").strip()
|
||||
lowered = raw.lower()
|
||||
avoid = ["search_telemetry", "get_telemetry", "get_slow_telemetry"]
|
||||
alternates: list[dict[str, Any]] = []
|
||||
|
||||
if not raw and lat is not None and lng is not None:
|
||||
recommended = {
|
||||
"cmd": "brief_area",
|
||||
"args": _compact_args(
|
||||
{"lat": lat, "lng": lng, "radius_km": radius_km},
|
||||
compact=compact,
|
||||
),
|
||||
}
|
||||
return {
|
||||
"intent": "area_brief",
|
||||
"recommended": recommended,
|
||||
"alternates": [{"cmd": "entities_near", "args": recommended["args"]}],
|
||||
"avoid": avoid,
|
||||
"estimated_ms": _estimate_ms("brief_area"),
|
||||
}
|
||||
|
||||
if not raw:
|
||||
recommended = {"cmd": "get_summary", "args": _compact_args({}, compact=compact)}
|
||||
return {
|
||||
"intent": "discovery",
|
||||
"recommended": recommended,
|
||||
"alternates": [{"cmd": "channel_status", "args": {}}],
|
||||
"avoid": avoid,
|
||||
"estimated_ms": _estimate_ms("get_summary"),
|
||||
}
|
||||
|
||||
cve_match = RE_CVE.search(raw)
|
||||
if cve_match:
|
||||
recommended = {
|
||||
"cmd": "osint_lookup",
|
||||
"args": _compact_args({"tool": "cve", "cve": cve_match.group(0).upper()}, compact=compact),
|
||||
}
|
||||
return _route_result("cve_lookup", recommended, avoid, alternates)
|
||||
|
||||
ip_match = RE_IPV4.search(raw)
|
||||
if ip_match and ("ip" in lowered or "address" in lowered or lowered.count(".") >= 3):
|
||||
recommended = {
|
||||
"cmd": "osint_lookup",
|
||||
"args": _compact_args({"tool": "ip", "ip": ip_match.group(0)}, compact=compact),
|
||||
}
|
||||
alternates.append({"cmd": "entity_expand", "args": {"type": "ip", "id": ip_match.group(0)}})
|
||||
return _route_result("ip_lookup", recommended, avoid, alternates)
|
||||
|
||||
if "whois" in lowered or ("dns" in lowered and RE_DOMAIN.search(raw)):
|
||||
domain = (RE_DOMAIN.search(raw) or re.search(r"\b([a-z0-9-]+\.[a-z]{2,})\b", raw, re.I))
|
||||
tool = "whois" if "whois" in lowered else "dns"
|
||||
domain_value = domain.group(0) if domain else raw
|
||||
recommended = {
|
||||
"cmd": "osint_lookup",
|
||||
"args": _compact_args({"tool": tool, "domain": domain_value}, compact=compact),
|
||||
}
|
||||
return _route_result("domain_lookup", recommended, avoid, alternates)
|
||||
|
||||
if "sanction" in lowered or "ofac" in lowered:
|
||||
recommended = {
|
||||
"cmd": "osint_lookup",
|
||||
"args": _compact_args({"tool": "sanctions", "query": raw}, compact=compact),
|
||||
}
|
||||
return _route_result("sanctions_lookup", recommended, avoid, alternates)
|
||||
|
||||
mmsi_match = RE_MMSI.search(raw)
|
||||
if mmsi_match and any(k in lowered for k in ("mmsi", "ship", "vessel", "yacht", "boat", "maritime")):
|
||||
recommended = {
|
||||
"cmd": "find_ships",
|
||||
"args": _compact_args({"mmsi": mmsi_match.group(0)}, compact=compact),
|
||||
}
|
||||
alternates.append({"cmd": "find_entity", "args": {"mmsi": mmsi_match.group(0), "entity_type": "ship"}})
|
||||
return _route_result("maritime_identifier", recommended, avoid, alternates)
|
||||
|
||||
n_match = RE_N_NUMBER.search(raw)
|
||||
if n_match:
|
||||
reg = n_match.group(0).upper()
|
||||
recommended = {
|
||||
"cmd": "find_flights",
|
||||
"args": _compact_args({"registration": reg}, compact=compact),
|
||||
}
|
||||
alternates.append({"cmd": "find_entity", "args": {"registration": reg, "entity_type": "aircraft"}})
|
||||
return _route_result("tail_number", recommended, avoid, alternates)
|
||||
|
||||
# callsign tokens
|
||||
tokens = re.findall(r"\b[A-Z0-9]{2,8}\b", raw.upper())
|
||||
for token in tokens:
|
||||
if token in KNOWN_CALLSIGNS or RE_CALLSIGN.fullmatch(token):
|
||||
recommended = {
|
||||
"cmd": "find_flights",
|
||||
"args": _compact_args({"callsign": token}, compact=compact),
|
||||
}
|
||||
alternates.append({"cmd": "find_entity", "args": {"callsign": token, "entity_type": "aircraft"}})
|
||||
return _route_result("callsign", recommended, avoid, alternates)
|
||||
|
||||
if any(k in lowered for k in ("news", "telegram", "headline", "headlines", "gdelt")):
|
||||
recommended = {
|
||||
"cmd": "search_news",
|
||||
"args": _compact_args({"query": _news_query(raw), "limit": 10}, compact=compact),
|
||||
}
|
||||
alternates.append({
|
||||
"cmd": "get_layer_slice",
|
||||
"args": {"layers": ["telegram_osint", "news"], "limit_per_layer": 10, "compact": compact},
|
||||
})
|
||||
return _route_result("news_search", recommended, avoid, alternates)
|
||||
|
||||
if any(
|
||||
k in lowered
|
||||
for k in (
|
||||
"gt backtest",
|
||||
"backtest gt",
|
||||
"historical backtest",
|
||||
"wilson confidence",
|
||||
"confidence rate",
|
||||
"gt benchmark",
|
||||
"validate gt",
|
||||
)
|
||||
):
|
||||
tune = any(k in lowered for k in ("tune", "grid search", "optimize threshold"))
|
||||
expanded = "base" not in lowered
|
||||
recommended = {
|
||||
"cmd": "gt_backtest",
|
||||
"args": _compact_args(
|
||||
{
|
||||
"expanded": expanded,
|
||||
"tune": tune,
|
||||
"target_confidence": 0.95,
|
||||
},
|
||||
compact=compact,
|
||||
),
|
||||
}
|
||||
alternates.append({"cmd": "gt_risk_heatmap", "args": {}})
|
||||
return _route_result("gt_backtest", recommended, avoid, alternates)
|
||||
|
||||
if any(
|
||||
k in lowered
|
||||
for k in (
|
||||
"rolling backtest",
|
||||
"rolling validation",
|
||||
"weekly validation",
|
||||
"operational validation",
|
||||
"operational backtest",
|
||||
"week over week",
|
||||
"week-over-week",
|
||||
"gt rolling",
|
||||
"rolling gt",
|
||||
"weekly gt",
|
||||
"weekly gt score",
|
||||
"gt weekly",
|
||||
"gt snapshot",
|
||||
"freeze weekly gt",
|
||||
)
|
||||
):
|
||||
micro = any(
|
||||
k in lowered
|
||||
for k in (
|
||||
"3 day",
|
||||
"3-day",
|
||||
"three day",
|
||||
"micro rolling",
|
||||
"rolling average",
|
||||
"ignition",
|
||||
"micro gt",
|
||||
)
|
||||
)
|
||||
freeze = any(
|
||||
k in lowered
|
||||
for k in ("freeze", "gt snapshot", "weekly snapshot", "capture week")
|
||||
)
|
||||
label = any(k in lowered for k in ("label", "outcome", "escalation"))
|
||||
if micro and not freeze and not label:
|
||||
recommended = {
|
||||
"cmd": "gt_micro_rolling",
|
||||
"args": _compact_args({"window_days": 3}, compact=compact),
|
||||
}
|
||||
intent = "gt_micro_rolling"
|
||||
elif freeze:
|
||||
recommended = {
|
||||
"cmd": "gt_rolling_freeze",
|
||||
"args": _compact_args({"force": "force" in lowered}, compact=compact),
|
||||
}
|
||||
intent = "gt_rolling_freeze"
|
||||
elif label:
|
||||
recommended = {
|
||||
"cmd": "gt_rolling_label",
|
||||
"args": _compact_args({}, compact=compact),
|
||||
}
|
||||
intent = "gt_rolling_label"
|
||||
else:
|
||||
recommended = {
|
||||
"cmd": "gt_rolling_backtest",
|
||||
"args": _compact_args({"weeks": 8, "target_confidence": 0.80}, compact=compact),
|
||||
}
|
||||
intent = "gt_rolling_backtest"
|
||||
alternates.append({"cmd": "gt_micro_rolling", "args": {"window_days": 3}})
|
||||
alternates.append({"cmd": "gt_backtest", "args": {"expanded": True, "compact": True}})
|
||||
return _route_result(intent, recommended, avoid, alternates)
|
||||
|
||||
if any(
|
||||
k in lowered
|
||||
for k in (
|
||||
"3 day average",
|
||||
"3-day average",
|
||||
"rolling 3 day",
|
||||
"micro risk",
|
||||
"risk ignition",
|
||||
)
|
||||
):
|
||||
recommended = {
|
||||
"cmd": "gt_micro_rolling",
|
||||
"args": _compact_args({"window_days": 3}, compact=compact),
|
||||
}
|
||||
alternates.append({"cmd": "gt_rolling_backtest", "args": {"weeks": 8}})
|
||||
return _route_result("gt_micro_rolling", recommended, avoid, alternates)
|
||||
|
||||
if any(
|
||||
k in lowered
|
||||
for k in (
|
||||
"gt analysis",
|
||||
"game theoretic",
|
||||
"game-theoretic",
|
||||
"strategic risk",
|
||||
"early warning",
|
||||
"risk heatmap",
|
||||
"costly signal",
|
||||
"gt rationale",
|
||||
)
|
||||
):
|
||||
region_hint = _gt_region_hint(raw)
|
||||
if region_hint and any(k in lowered for k in ("dossier", "rationale", "scenario")):
|
||||
recommended = {
|
||||
"cmd": "gt_dossier",
|
||||
"args": _compact_args({"region": region_hint}, compact=compact),
|
||||
}
|
||||
alternates.append({"cmd": "gt_risk_heatmap", "args": {}})
|
||||
return _route_result("gt_dossier", recommended, avoid, alternates)
|
||||
recommended = {
|
||||
"cmd": "gt_analyze",
|
||||
"args": _compact_args(
|
||||
{"refresh": True, "region": region_hint} if region_hint else {"refresh": True},
|
||||
compact=compact,
|
||||
),
|
||||
}
|
||||
alternates.append({"cmd": "gt_risk_heatmap", "args": {}})
|
||||
return _route_result("gt_analyze", recommended, avoid, alternates)
|
||||
|
||||
if lat is not None and lng is not None and any(
|
||||
k in lowered for k in ("near", "around", "within", "radius", "brief", "aoi")
|
||||
):
|
||||
recommended = {
|
||||
"cmd": "brief_area",
|
||||
"args": _compact_args(
|
||||
{"lat": lat, "lng": lng, "radius_km": radius_km, "query": raw},
|
||||
compact=compact,
|
||||
),
|
||||
}
|
||||
alternates.append({
|
||||
"cmd": "entities_near",
|
||||
"args": {"lat": lat, "lng": lng, "radius_km": radius_km, "compact": compact},
|
||||
})
|
||||
return _route_result("area_brief", recommended, avoid, alternates)
|
||||
|
||||
if any(k in lowered for k in ("what changed", "updates", "delta", "since last")):
|
||||
recommended = {"cmd": "what_changed", "args": _compact_args({}, compact=compact)}
|
||||
return _route_result("incremental_poll", recommended, avoid, alternates)
|
||||
|
||||
if any(k in lowered for k in ("summary", "status", "layers populated", "what data")):
|
||||
recommended = {"cmd": "get_summary", "args": _compact_args({}, compact=compact)}
|
||||
alternates.append({"cmd": "channel_status", "args": {}})
|
||||
return _route_result("discovery", recommended, avoid, alternates)
|
||||
|
||||
if any(k in lowered for k in ("recon", "whois", "dns lookup", "cve", "mac address")):
|
||||
recommended = {
|
||||
"cmd": "osint_tools",
|
||||
"args": {},
|
||||
}
|
||||
return _route_result("recon_discovery", recommended, avoid, alternates)
|
||||
|
||||
entity_type = ""
|
||||
if any(k in lowered for k in ("ship", "vessel", "yacht", "boat", "maritime", "carrier")):
|
||||
entity_type = "ship"
|
||||
elif any(k in lowered for k in ("jet", "plane", "flight", "aircraft", "helicopter", "tail")):
|
||||
entity_type = "aircraft"
|
||||
|
||||
owner_hint = ""
|
||||
if any(k in lowered for k in ("owner", "operated by", "'s jet", "'s yacht", "belongs to")):
|
||||
owner_hint = raw
|
||||
for phrase in ("where is", "find", "track", "locate", "jet", "yacht", "plane", "flight", "ship"):
|
||||
owner_hint = re.sub(rf"\b{phrase}\b", "", owner_hint, flags=re.I).strip()
|
||||
|
||||
entity_args: dict[str, Any] = {"query": raw, "compact": compact}
|
||||
if entity_type:
|
||||
entity_args["entity_type"] = entity_type
|
||||
if owner_hint and len(owner_hint) >= 3:
|
||||
entity_args["owner"] = owner_hint
|
||||
|
||||
recommended = {
|
||||
"cmd": "find_entity",
|
||||
"args": _compact_args(entity_args, compact=compact),
|
||||
}
|
||||
alternates = [
|
||||
{"cmd": "search_news", "args": {"query": raw, "limit": 10, "compact": compact}},
|
||||
]
|
||||
if any(k in lowered for k in ("near", "around")):
|
||||
alternates.append({
|
||||
"cmd": "search_telemetry",
|
||||
"args": {"query": raw, "limit": 10, "confirm_expensive": True, "compact": compact},
|
||||
})
|
||||
|
||||
return _route_result("entity_lookup", recommended, avoid, alternates)
|
||||
|
||||
|
||||
def _route_result(
|
||||
intent: str,
|
||||
recommended: dict[str, Any],
|
||||
avoid: list[str],
|
||||
alternates: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
cmd = str(recommended.get("cmd", ""))
|
||||
return {
|
||||
"intent": intent,
|
||||
"recommended": recommended,
|
||||
"alternates": alternates,
|
||||
"avoid": avoid,
|
||||
"estimated_ms": _estimate_ms(cmd),
|
||||
}
|
||||
|
||||
|
||||
def plan_playbook(name: str, args: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Resolve a named playbook to a command batch."""
|
||||
playbook = str(name or "").strip().lower()
|
||||
params = dict(args or {})
|
||||
if not playbook:
|
||||
return {"ok": False, "detail": "playbook name required"}
|
||||
|
||||
if playbook == "track_snapshot":
|
||||
query = str(params.get("query", "") or params.get("name", "") or "").strip()
|
||||
if not query:
|
||||
return {"ok": False, "detail": "track_snapshot requires query"}
|
||||
return {
|
||||
"ok": True,
|
||||
"playbook": playbook,
|
||||
"description": "Resolve entity for tracking",
|
||||
"batch": [
|
||||
{
|
||||
"cmd": "find_entity",
|
||||
"args": {
|
||||
"query": query,
|
||||
"entity_type": params.get("entity_type", ""),
|
||||
"fallback_search": True,
|
||||
"compact": True,
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
if playbook == "area_brief":
|
||||
lat = params.get("lat")
|
||||
lng = params.get("lng")
|
||||
if lat is None or lng is None:
|
||||
return {"ok": False, "detail": "area_brief requires lat and lng"}
|
||||
return {
|
||||
"ok": True,
|
||||
"playbook": playbook,
|
||||
"description": "Brief an area of interest",
|
||||
"batch": [
|
||||
{
|
||||
"cmd": "brief_area",
|
||||
"args": {
|
||||
"lat": lat,
|
||||
"lng": lng,
|
||||
"radius_km": params.get("radius_km", 50),
|
||||
"query": params.get("query", ""),
|
||||
"compact": True,
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
if playbook == "entity_recon":
|
||||
query = str(params.get("query", "") or params.get("ip", "") or "").strip()
|
||||
ip_match = RE_IPV4.search(query)
|
||||
if not ip_match:
|
||||
return {"ok": False, "detail": "entity_recon requires an IP in query"}
|
||||
return {
|
||||
"ok": True,
|
||||
"playbook": playbook,
|
||||
"description": "IP recon + entity graph",
|
||||
"batch": [
|
||||
{"cmd": "osint_lookup", "args": {"tool": "ip", "ip": ip_match.group(0), "compact": True}},
|
||||
{"cmd": "entity_expand", "args": {"type": "ip", "id": ip_match.group(0)}},
|
||||
],
|
||||
}
|
||||
|
||||
spec = PLAYBOOKS.get(playbook)
|
||||
if not spec:
|
||||
known = sorted(PLAYBOOKS) + ["track_snapshot", "area_brief", "entity_recon"]
|
||||
return {"ok": False, "detail": f"unknown playbook: {playbook}", "known": known}
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"playbook": playbook,
|
||||
"description": spec.get("description", ""),
|
||||
"batch": [dict(item) for item in spec.get("batch", [])],
|
||||
}
|
||||
@@ -22,9 +22,12 @@ logger = logging.getLogger(__name__)
|
||||
_lock = threading.Lock()
|
||||
_watches: dict[str, dict[str, Any]] = {} # watch_id -> watch definition
|
||||
_fired: dict[str, float] = {} # watch_id -> last fire timestamp (debounce)
|
||||
_seen_posts: dict[str, set[str]] = {} # watch_id -> seen Telegram post ids/links
|
||||
_running = False
|
||||
_stop_event = threading.Event()
|
||||
|
||||
_TELEGRAM_SEEN_MAX = 500
|
||||
|
||||
# Minimum seconds between re-firing the same watch
|
||||
DEBOUNCE_S = 60.0
|
||||
# How often the watchdog checks telemetry
|
||||
@@ -73,6 +76,7 @@ def remove_watch(watch_id: str) -> dict[str, Any]:
|
||||
with _lock:
|
||||
removed = _watches.pop(watch_id, None)
|
||||
_fired.pop(watch_id, None)
|
||||
_seen_posts.pop(watch_id, None)
|
||||
if removed:
|
||||
return {"ok": True, "removed": removed}
|
||||
return {"ok": False, "detail": f"watch '{watch_id}' not found"}
|
||||
@@ -90,6 +94,7 @@ def clear_watches() -> dict[str, Any]:
|
||||
count = len(_watches)
|
||||
_watches.clear()
|
||||
_fired.clear()
|
||||
_seen_posts.clear()
|
||||
return {"ok": True, "cleared": count}
|
||||
|
||||
|
||||
@@ -157,7 +162,9 @@ def _check_watch(watch: dict, fast: dict, slow: dict) -> dict[str, Any] | None:
|
||||
if wtype == "geofence":
|
||||
return _check_geofence(params, fast)
|
||||
if wtype == "keyword":
|
||||
return _check_keyword(params, fast, slow)
|
||||
return _check_keyword(watch["id"], params, fast, slow)
|
||||
if wtype == "telegram_rhetoric":
|
||||
return _check_telegram_rhetoric(watch["id"], params, slow)
|
||||
if wtype == "prediction_market":
|
||||
return _check_prediction_market(params, slow)
|
||||
|
||||
@@ -390,15 +397,41 @@ def _check_geofence(params: dict, fast: dict) -> dict | None:
|
||||
return None
|
||||
|
||||
|
||||
def _check_keyword(params: dict, fast: dict, slow: dict) -> dict | None:
|
||||
"""Alert when a keyword appears in news/GDELT."""
|
||||
def _telegram_post_id(post: dict[str, Any]) -> str:
|
||||
return str(post.get("id") or post.get("link") or "").strip()
|
||||
|
||||
|
||||
def _mark_seen_posts(watch_id: str, post_ids: list[str]) -> None:
|
||||
clean = [pid for pid in post_ids if pid]
|
||||
if not clean:
|
||||
return
|
||||
with _lock:
|
||||
seen = _seen_posts.setdefault(watch_id, set())
|
||||
seen.update(clean)
|
||||
if len(seen) > _TELEGRAM_SEEN_MAX:
|
||||
_seen_posts[watch_id] = set(list(seen)[-_TELEGRAM_SEEN_MAX:])
|
||||
|
||||
|
||||
def _is_seen_post(watch_id: str, post_id: str) -> bool:
|
||||
if not post_id:
|
||||
return False
|
||||
with _lock:
|
||||
return post_id in _seen_posts.get(watch_id, set())
|
||||
|
||||
|
||||
def _check_keyword(watch_id: str, params: dict, fast: dict, slow: dict) -> dict | None:
|
||||
"""Alert when a keyword appears in news, GDELT, or Telegram OSINT."""
|
||||
keyword = str(params.get("keyword", "")).lower().strip()
|
||||
if not keyword:
|
||||
return None
|
||||
|
||||
matches = []
|
||||
include_telegram = params.get("include_telegram", True)
|
||||
if isinstance(include_telegram, str):
|
||||
include_telegram = include_telegram.strip().lower() not in {"0", "false", "no", "off"}
|
||||
|
||||
matches = []
|
||||
new_telegram_ids: list[str] = []
|
||||
|
||||
# Check news articles
|
||||
for article in slow.get("news", []):
|
||||
title = str(article.get("title", "") or "").lower()
|
||||
desc = str(article.get("description", "") or article.get("summary", "") or "").lower()
|
||||
@@ -409,7 +442,6 @@ def _check_keyword(params: dict, fast: dict, slow: dict) -> dict | None:
|
||||
"url": article.get("url") or article.get("link"),
|
||||
})
|
||||
|
||||
# Check GDELT
|
||||
for event in slow.get("gdelt", []):
|
||||
text = str(event.get("title", "") or event.get("sourceurl", "") or "").lower()
|
||||
if keyword in text:
|
||||
@@ -419,14 +451,103 @@ def _check_keyword(params: dict, fast: dict, slow: dict) -> dict | None:
|
||||
"url": event.get("sourceurl"),
|
||||
})
|
||||
|
||||
if include_telegram:
|
||||
from services.telegram_osint_text import (
|
||||
iter_telegram_posts,
|
||||
keyword_matches_telegram_post,
|
||||
telegram_post_match_entry,
|
||||
)
|
||||
|
||||
for post in iter_telegram_posts(slow.get("telegram_osint")):
|
||||
if not keyword_matches_telegram_post(post, keyword):
|
||||
continue
|
||||
post_id = _telegram_post_id(post)
|
||||
if _is_seen_post(watch_id, post_id):
|
||||
continue
|
||||
entry = telegram_post_match_entry(post)
|
||||
matches.append(entry)
|
||||
if post_id:
|
||||
new_telegram_ids.append(post_id)
|
||||
|
||||
if matches:
|
||||
if new_telegram_ids:
|
||||
_mark_seen_posts(watch_id, new_telegram_ids)
|
||||
sources = sorted({str(match.get("source") or "unknown") for match in matches})
|
||||
return {
|
||||
"alert": f"Keyword '{keyword}' found in {len(matches)} articles",
|
||||
"alert": f"Keyword '{keyword}' found in {len(matches)} items ({', '.join(sources)})",
|
||||
"data": {"keyword": keyword, "matches": matches[:10]},
|
||||
}
|
||||
return None
|
||||
|
||||
|
||||
def _check_telegram_rhetoric(watch_id: str, params: dict, slow: dict) -> dict | None:
|
||||
"""Alert on new high-risk Telegram OSINT posts (optionally keyword/channel filtered)."""
|
||||
min_risk = int(params.get("min_risk_score", 7) or 7)
|
||||
min_risk = max(1, min(min_risk, 10))
|
||||
|
||||
raw_keywords = params.get("keywords") or params.get("keyword") or []
|
||||
if isinstance(raw_keywords, str):
|
||||
raw_keywords = [part.strip() for part in raw_keywords.split(",") if part.strip()]
|
||||
keywords = [str(item).lower().strip() for item in raw_keywords if str(item).strip()]
|
||||
|
||||
raw_channels = params.get("channels") or params.get("channel") or []
|
||||
if isinstance(raw_channels, str):
|
||||
raw_channels = [part.strip() for part in raw_channels.split(",") if part.strip()]
|
||||
channels = [str(item).lower().strip().lstrip("@") for item in raw_channels if str(item).strip()]
|
||||
|
||||
from services.telegram_osint_text import (
|
||||
iter_telegram_posts,
|
||||
keyword_matches_telegram_post,
|
||||
telegram_post_match_entry,
|
||||
)
|
||||
|
||||
matches = []
|
||||
new_post_ids: list[str] = []
|
||||
|
||||
for post in iter_telegram_posts(slow.get("telegram_osint")):
|
||||
try:
|
||||
risk = int(post.get("risk_score") or 0)
|
||||
except (TypeError, ValueError):
|
||||
risk = 0
|
||||
if risk < min_risk:
|
||||
continue
|
||||
|
||||
channel = str(post.get("channel") or "").lower().strip()
|
||||
source = str(post.get("source") or "").lower().strip()
|
||||
if channels and channel not in channels and not any(ch in source for ch in channels):
|
||||
continue
|
||||
|
||||
if keywords and not any(keyword_matches_telegram_post(post, kw) for kw in keywords):
|
||||
continue
|
||||
|
||||
post_id = _telegram_post_id(post)
|
||||
if _is_seen_post(watch_id, post_id):
|
||||
continue
|
||||
|
||||
entry = telegram_post_match_entry(post)
|
||||
matches.append(entry)
|
||||
if post_id:
|
||||
new_post_ids.append(post_id)
|
||||
|
||||
if not matches:
|
||||
return None
|
||||
|
||||
_mark_seen_posts(watch_id, new_post_ids)
|
||||
top = max(int(match.get("risk_score") or 0) for match in matches)
|
||||
return {
|
||||
"alert": (
|
||||
f"Telegram rhetoric alert: {len(matches)} new post(s) at LVL {top}/10"
|
||||
+ (f" (min {min_risk})" if min_risk > 1 else "")
|
||||
),
|
||||
"data": {
|
||||
"min_risk_score": min_risk,
|
||||
"keywords": keywords,
|
||||
"channels": channels,
|
||||
"matches": matches[:10],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _check_prediction_market(params: dict, slow: dict) -> dict | None:
|
||||
"""Alert on prediction market movements."""
|
||||
query = str(params.get("query", "")).lower().strip()
|
||||
|
||||
@@ -213,7 +213,7 @@ def validate_privacy_core_startup(settings: Any | None = None) -> None:
|
||||
|
||||
attestation = privacy_core_attestation(snapshot)
|
||||
state = str(attestation.get("attestation_state", "") or "").strip()
|
||||
if state == "attested_current":
|
||||
if state in {"attested_current", "development_override"}:
|
||||
return
|
||||
|
||||
logger.critical(
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Container-aware runtime limits for fleet vs desktop deployments."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _read_first_int(path: Path) -> int | None:
|
||||
try:
|
||||
raw = path.read_text(encoding="utf-8").strip().split()[0]
|
||||
return int(raw)
|
||||
except (OSError, ValueError, IndexError):
|
||||
return None
|
||||
|
||||
|
||||
def detect_cpu_limit() -> float | None:
|
||||
"""Effective CPU cores from cgroup quota (Docker ``cpus:``), else host count."""
|
||||
cgroup_v2 = Path("/sys/fs/cgroup/cpu.max")
|
||||
if cgroup_v2.is_file():
|
||||
try:
|
||||
parts = cgroup_v2.read_text(encoding="utf-8").strip().split()
|
||||
if len(parts) >= 2 and parts[0] != "max":
|
||||
quota = int(parts[0])
|
||||
period = int(parts[1])
|
||||
if quota > 0 and period > 0:
|
||||
return round(quota / period, 3)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
cgroup_v1_quota = Path("/sys/fs/cgroup/cpu/cpu.cfs_quota_us")
|
||||
cgroup_v1_period = Path("/sys/fs/cgroup/cpu/cpu.cfs_period_us")
|
||||
if cgroup_v1_quota.is_file() and cgroup_v1_period.is_file():
|
||||
quota = _read_first_int(cgroup_v1_quota)
|
||||
period = _read_first_int(cgroup_v1_period)
|
||||
if quota is not None and period and quota > 0:
|
||||
return round(quota / period, 3)
|
||||
|
||||
try:
|
||||
import os as _os
|
||||
|
||||
count = _os.cpu_count()
|
||||
return float(count) if count else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def detect_memory_limit_mb() -> int | None:
|
||||
cgroup_v2 = Path("/sys/fs/cgroup/memory.max")
|
||||
if cgroup_v2.is_file():
|
||||
try:
|
||||
raw = cgroup_v2.read_text(encoding="utf-8").strip()
|
||||
if raw and raw != "max":
|
||||
return int(int(raw) / (1024 * 1024))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
cgroup_v1 = Path("/sys/fs/cgroup/memory/memory.limit_in_bytes")
|
||||
if cgroup_v1.is_file():
|
||||
try:
|
||||
raw = _read_first_int(cgroup_v1)
|
||||
if raw is not None and raw < (1 << 62):
|
||||
return int(raw / (1024 * 1024))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def resolve_profile_name() -> str:
|
||||
explicit = str(os.environ.get("GT_ANALYTICS_PROFILE", "")).strip().lower()
|
||||
if explicit in {"lean", "standard"}:
|
||||
return explicit
|
||||
cpu = detect_cpu_limit()
|
||||
if cpu is not None and cpu <= 1.0:
|
||||
return "lean"
|
||||
return "standard"
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_runtime_profile() -> dict[str, Any]:
|
||||
cpu_limit = detect_cpu_limit()
|
||||
memory_mb = detect_memory_limit_mb()
|
||||
profile = resolve_profile_name()
|
||||
lean = profile == "lean"
|
||||
return {
|
||||
"profile": profile,
|
||||
"cpu_limit": cpu_limit,
|
||||
"memory_limit_mb": memory_mb,
|
||||
"gt_analytics": {
|
||||
"recommended": not lean,
|
||||
"lean_node": lean,
|
||||
"warning": (
|
||||
"This node is capped at 1 vCPU. Enabling Strategic Risk (Derived OSINT) "
|
||||
"may slow Telegram, GDELT, and other OSINT fetches. Set "
|
||||
"GT_ANALYTICS_ACK_LOW_CPU=true after enabling GT_ANALYTICS_ENABLED to run "
|
||||
"the full engine on lean hardware."
|
||||
if lean
|
||||
else None
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def clear_runtime_profile_cache() -> None:
|
||||
get_runtime_profile.cache_clear()
|
||||
@@ -167,18 +167,31 @@ def products_fetch_enabled() -> bool:
|
||||
return _flag("MESH_SAR_PRODUCTS_FETCH_ACKNOWLEDGE", default=False)
|
||||
|
||||
|
||||
def runtime_store_exists() -> bool:
|
||||
"""True when ``data/sar_runtime.json`` exists on disk."""
|
||||
return _RUNTIME_FILE.is_file()
|
||||
|
||||
|
||||
def products_fetch_status() -> dict[str, Any]:
|
||||
"""Structured status used by the router for the 'how to enable' UX."""
|
||||
raw = _str("MESH_SAR_PRODUCTS_FETCH", default="block").strip().lower()
|
||||
fetch_set = raw in {"allow", "enable", "enabled", "true", "on", "1"}
|
||||
ack_set = _flag("MESH_SAR_PRODUCTS_FETCH_ACKNOWLEDGE", default=False)
|
||||
enabled = fetch_set and ack_set
|
||||
token_set = bool(earthdata_token())
|
||||
user_set = bool(earthdata_user())
|
||||
opt_in = fetch_set and ack_set
|
||||
# ``enabled`` historically meant opt-in flags only; ``fully_configured``
|
||||
# is what the fetcher actually needs (flags + Earthdata token).
|
||||
fully_configured = opt_in and token_set
|
||||
return {
|
||||
"enabled": enabled,
|
||||
"enabled": opt_in,
|
||||
"fully_configured": fully_configured,
|
||||
"fetch_flag_set": fetch_set,
|
||||
"acknowledge_flag_set": ack_set,
|
||||
"earthdata_token_set": bool(earthdata_token()),
|
||||
"earthdata_user_set": bool(earthdata_user()),
|
||||
"earthdata_token_set": token_set,
|
||||
"earthdata_user_set": user_set,
|
||||
"runtime_store_exists": runtime_store_exists(),
|
||||
"runtime_store_path": str(_RUNTIME_FILE),
|
||||
"missing": _missing_for_products(fetch_set, ack_set),
|
||||
"help": {
|
||||
"summary": (
|
||||
|
||||
@@ -19,6 +19,7 @@ class HealthResponse(BaseModel):
|
||||
# insecure-date path because the upstream Let's Encrypt cert is
|
||||
# expired. Empty dict / null means no status reported yet.
|
||||
ais_proxy: Optional[Dict[str, Any]] = None
|
||||
runtime: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class RefreshResponse(BaseModel):
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Shared Telegram OSINT post text helpers for search and watchdog matching."""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from services.telegram_translate import source_lang_label
|
||||
|
||||
|
||||
def iter_telegram_posts(layer_payload: Any) -> list[dict[str, Any]]:
|
||||
"""Normalize telegram_osint layer payloads into a list of post dicts."""
|
||||
if isinstance(layer_payload, list):
|
||||
return [post for post in layer_payload if isinstance(post, dict)]
|
||||
if isinstance(layer_payload, dict):
|
||||
posts = layer_payload.get("posts")
|
||||
if isinstance(posts, list):
|
||||
return [post for post in posts if isinstance(post, dict)]
|
||||
return []
|
||||
|
||||
|
||||
def telegram_post_search_text(post: dict[str, Any]) -> str:
|
||||
"""Build a lowercase haystack for keyword matching (translated + original)."""
|
||||
parts = (
|
||||
post.get("title_translated"),
|
||||
post.get("description_translated"),
|
||||
post.get("title"),
|
||||
post.get("description"),
|
||||
post.get("source"),
|
||||
post.get("channel"),
|
||||
)
|
||||
return " ".join(str(part).strip() for part in parts if str(part or "").strip()).lower()
|
||||
|
||||
|
||||
def telegram_post_display_title(post: dict[str, Any]) -> str:
|
||||
"""Prefer translated headline for alerts and agent-facing summaries."""
|
||||
translated = str(post.get("title_translated") or post.get("description_translated") or "").strip()
|
||||
if translated:
|
||||
return translated.split("\n", 1)[0][:200]
|
||||
return str(post.get("title") or post.get("description") or "").strip()[:200]
|
||||
|
||||
|
||||
def telegram_post_match_entry(post: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Compact match record for watchdog alerts and search results."""
|
||||
lat, lng = None, None
|
||||
coords = post.get("coords")
|
||||
if isinstance(coords, (list, tuple)) and len(coords) >= 2:
|
||||
lat, lng = coords[0], coords[1]
|
||||
return {
|
||||
"source": "telegram_osint",
|
||||
"title": telegram_post_display_title(post),
|
||||
"original_title": str(post.get("title") or "").strip(),
|
||||
"url": post.get("link") or "",
|
||||
"channel": post.get("channel") or post.get("source") or "",
|
||||
"risk_score": post.get("risk_score"),
|
||||
"source_lang": post.get("source_lang"),
|
||||
"source_lang_label": post.get("source_lang_label") or source_lang_label(post.get("source_lang")),
|
||||
"lat": lat,
|
||||
"lng": lng,
|
||||
"id": post.get("id") or post.get("link") or "",
|
||||
}
|
||||
|
||||
|
||||
def keyword_matches_telegram_post(post: dict[str, Any], keyword: str) -> bool:
|
||||
needle = str(keyword or "").strip().lower()
|
||||
if not needle:
|
||||
return False
|
||||
return needle in telegram_post_search_text(post)
|
||||
@@ -0,0 +1,243 @@
|
||||
"""Auto-translation for Telegram OSINT post text (server-side, cached)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import urllib.parse
|
||||
from threading import Lock
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_CYRILLIC_RE = re.compile(r"[\u0400-\u04FF]")
|
||||
_UKRAINIAN_MARKERS_RE = re.compile(r"[іїєґІЇЄҐ]")
|
||||
_ARABIC_RE = re.compile(r"[\u0600-\u06FF]")
|
||||
_HEBREW_RE = re.compile(r"[\u0590-\u05FF]")
|
||||
_CJK_RE = re.compile(r"[\u4e00-\u9fff]")
|
||||
|
||||
# Common war-reporting shorthand that machine translation often transliterates.
|
||||
_POST_TRANSLATION_GLOSSARY: tuple[tuple[re.Pattern[str], str], ...] = (
|
||||
(re.compile(r"\bBpLa\b", re.IGNORECASE), "UAV"),
|
||||
(re.compile(r"\bБпЛА\b", re.IGNORECASE), "UAV"),
|
||||
(re.compile(r"\bбпла\b"), "UAV"),
|
||||
(re.compile(r"\bБПЛА\b"), "UAV"),
|
||||
(re.compile(r"\bрсзв\b", re.IGNORECASE), "MLRS"),
|
||||
(re.compile(r"\bРСЗВ\b"), "MLRS"),
|
||||
)
|
||||
|
||||
_SOURCE_LANG_LABELS = {
|
||||
"uk": "Ukrainian",
|
||||
"ru": "Russian",
|
||||
"en": "English",
|
||||
"ar": "Arabic",
|
||||
"he": "Hebrew",
|
||||
"zh-cn": "Chinese",
|
||||
"fr": "French",
|
||||
"de": "German",
|
||||
"pl": "Polish",
|
||||
}
|
||||
|
||||
_CACHE: dict[str, tuple[str, str]] = {}
|
||||
_CACHE_LOCK = Lock()
|
||||
_CACHE_MAX = 512
|
||||
|
||||
_LOCALE_TO_GOOGLE = {
|
||||
"en": "en",
|
||||
"fr": "fr",
|
||||
"zh-cn": "zh-CN",
|
||||
"zh": "zh-CN",
|
||||
}
|
||||
|
||||
|
||||
def telegram_translate_enabled() -> bool:
|
||||
return str(os.environ.get("TELEGRAM_OSINT_TRANSLATE", "true")).strip().lower() not in {
|
||||
"0",
|
||||
"false",
|
||||
"no",
|
||||
"off",
|
||||
"",
|
||||
}
|
||||
|
||||
|
||||
def telegram_translate_target() -> str:
|
||||
raw = str(os.environ.get("TELEGRAM_OSINT_TRANSLATE_TO", "en")).strip().lower()
|
||||
return _LOCALE_TO_GOOGLE.get(raw, raw or "en")
|
||||
|
||||
|
||||
def normalize_translate_target(locale: str | None) -> str:
|
||||
raw = str(locale or telegram_translate_target()).strip().lower().replace("_", "-")
|
||||
return _LOCALE_TO_GOOGLE.get(raw, raw or "en")
|
||||
|
||||
|
||||
def _looks_english(text: str) -> bool:
|
||||
letters = [char for char in text if char.isalpha()]
|
||||
if not letters:
|
||||
return True
|
||||
ascii_letters = sum(1 for char in letters if ord(char) < 128)
|
||||
return ascii_letters / len(letters) > 0.9
|
||||
|
||||
|
||||
def contains_cyrillic(text: str) -> bool:
|
||||
return bool(_CYRILLIC_RE.search(str(text or "")))
|
||||
|
||||
|
||||
def source_lang_label(code: str | None) -> str:
|
||||
raw = str(code or "").strip().lower().replace("_", "-")
|
||||
return _SOURCE_LANG_LABELS.get(raw, raw.upper() if raw else "Unknown")
|
||||
|
||||
|
||||
def polish_translation(text: str) -> str:
|
||||
polished = str(text or "")
|
||||
for pattern, replacement in _POST_TRANSLATION_GLOSSARY:
|
||||
polished = pattern.sub(replacement, polished)
|
||||
return polished.strip()
|
||||
|
||||
|
||||
def guess_source_lang(text: str) -> str:
|
||||
if _UKRAINIAN_MARKERS_RE.search(text):
|
||||
return "uk"
|
||||
if _CYRILLIC_RE.search(text):
|
||||
return "ru"
|
||||
if _ARABIC_RE.search(text):
|
||||
return "ar"
|
||||
if _HEBREW_RE.search(text):
|
||||
return "he"
|
||||
if _CJK_RE.search(text):
|
||||
return "zh-CN"
|
||||
if _looks_english(text):
|
||||
return "en"
|
||||
return "auto"
|
||||
|
||||
|
||||
def _cache_key(text: str, target_lang: str) -> str:
|
||||
digest = hashlib.sha1(f"{target_lang}|{text}".encode("utf-8")).hexdigest()
|
||||
return digest
|
||||
|
||||
|
||||
def _cache_get(text: str, target_lang: str) -> tuple[str, str] | None:
|
||||
key = _cache_key(text, target_lang)
|
||||
with _CACHE_LOCK:
|
||||
return _CACHE.get(key)
|
||||
|
||||
|
||||
def _cache_put(text: str, target_lang: str, translated: str, source_lang: str) -> None:
|
||||
key = _cache_key(text, target_lang)
|
||||
with _CACHE_LOCK:
|
||||
if len(_CACHE) >= _CACHE_MAX:
|
||||
_CACHE.pop(next(iter(_CACHE)))
|
||||
_CACHE[key] = (translated, source_lang)
|
||||
|
||||
|
||||
def _google_translate(clean: str, target: str, source: str | None = None) -> tuple[str, str]:
|
||||
params = {
|
||||
"client": "gtx",
|
||||
"sl": source or "auto",
|
||||
"tl": target,
|
||||
"dt": "t",
|
||||
"q": clean[:4500],
|
||||
}
|
||||
url = "https://translate.googleapis.com/translate_a/single?" + urllib.parse.urlencode(params)
|
||||
resp = requests.get(
|
||||
url,
|
||||
timeout=8,
|
||||
headers={"User-Agent": "Mozilla/5.0 (compatible; Shadowbroker-Telegram-Translate/1.0)"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
detected = str(data[2] or guess_source_lang(clean)).strip().lower()
|
||||
if detected in {"zh-cn", "zh-tw"}:
|
||||
detected = "zh-CN"
|
||||
parts: list[str] = []
|
||||
for chunk in data[0] or []:
|
||||
if chunk and chunk[0]:
|
||||
parts.append(str(chunk[0]))
|
||||
translated = polish_translation("".join(parts).strip() or clean)
|
||||
return translated, detected
|
||||
|
||||
|
||||
def translate_text(text: str, target_lang: str | None = None) -> tuple[str, str]:
|
||||
"""Translate text via Google Translate (unofficial client endpoint).
|
||||
|
||||
Returns ``(translated_text, detected_source_lang)``.
|
||||
"""
|
||||
clean = str(text or "").strip()
|
||||
if not clean:
|
||||
return "", "en"
|
||||
|
||||
target = normalize_translate_target(target_lang)
|
||||
if _looks_english(clean) and target == "en":
|
||||
return clean, "en"
|
||||
|
||||
cached = _cache_get(clean, target)
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
try:
|
||||
translated, detected = _google_translate(clean, target)
|
||||
if detected == target or (detected == "en" and target == "en"):
|
||||
result = (clean, detected)
|
||||
_cache_put(clean, target, clean, detected)
|
||||
return result
|
||||
if contains_cyrillic(translated) and contains_cyrillic(clean):
|
||||
hinted = guess_source_lang(clean)
|
||||
if hinted not in {"auto", target}:
|
||||
retry_translated, retry_detected = _google_translate(clean, target, hinted)
|
||||
if not contains_cyrillic(retry_translated) or len(retry_translated) > len(translated):
|
||||
translated, detected = retry_translated, retry_detected
|
||||
result = (translated, detected)
|
||||
_cache_put(clean, target, translated, detected)
|
||||
return result
|
||||
except Exception as exc:
|
||||
logger.warning("Telegram translation failed: %s", exc)
|
||||
fallback_lang = guess_source_lang(clean)
|
||||
return clean, fallback_lang
|
||||
|
||||
|
||||
def apply_post_translation(post: dict[str, Any], target_lang: str | None = None) -> dict[str, Any]:
|
||||
"""Add translation fields to a Telegram OSINT post dict."""
|
||||
if not telegram_translate_enabled():
|
||||
return post
|
||||
|
||||
target = normalize_translate_target(target_lang)
|
||||
description = str(post.get("description") or "").strip()
|
||||
title = str(post.get("title") or "").strip()
|
||||
full_text = description or title
|
||||
if not full_text:
|
||||
return post
|
||||
|
||||
existing_translated = str(post.get("description_translated") or post.get("title_translated") or "").strip()
|
||||
if post.get("translate_to") == target and existing_translated:
|
||||
updated = dict(post)
|
||||
polished = polish_translation(existing_translated)
|
||||
if polished != existing_translated:
|
||||
lines = polished.split("\n", 1)
|
||||
updated["title_translated"] = lines[0][:160]
|
||||
updated["description_translated"] = polished[:1200]
|
||||
updated["source_lang_label"] = source_lang_label(str(post.get("source_lang") or ""))
|
||||
return updated
|
||||
|
||||
translated_full, source_lang = translate_text(full_text, target)
|
||||
updated = dict(post)
|
||||
updated["source_lang"] = source_lang
|
||||
updated["translate_to"] = target
|
||||
updated["source_lang_label"] = source_lang_label(source_lang)
|
||||
|
||||
if translated_full != full_text and source_lang != target:
|
||||
lines = translated_full.split("\n", 1)
|
||||
updated["title_translated"] = lines[0][:160]
|
||||
updated["description_translated"] = translated_full[:1200]
|
||||
|
||||
return updated
|
||||
|
||||
|
||||
def apply_posts_translations(
|
||||
posts: list[dict[str, Any]],
|
||||
target_lang: str | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
if not telegram_translate_enabled():
|
||||
return posts
|
||||
return [apply_post_translation(post, target_lang) for post in posts]
|
||||
@@ -97,6 +97,7 @@ _SLOW_KEYS = (
|
||||
"cyber_threats",
|
||||
"scm_suppliers",
|
||||
"telegram_osint",
|
||||
"gt_risk",
|
||||
)
|
||||
|
||||
|
||||
@@ -210,6 +211,9 @@ _LAYER_ALIASES = {
|
||||
"telegram": "telegram_osint",
|
||||
"telegram_osint": "telegram_osint",
|
||||
"osint_feed": "telegram_osint",
|
||||
"gt_risk": "gt_risk",
|
||||
"strategic_risk": "gt_risk",
|
||||
"gt_analytics": "gt_risk",
|
||||
"malware": "malware_threats",
|
||||
"malware_threats": "malware_threats",
|
||||
"malware_c2": "malware_threats",
|
||||
@@ -710,10 +714,10 @@ _UNIVERSAL_SEARCH_SPECS: dict[str, dict[str, Any]] = {
|
||||
"time_fields": ("updated_at", "timestamp"),
|
||||
},
|
||||
"telegram_osint": {
|
||||
"fields": ("title", "description", "source", "channel", "link"),
|
||||
"primary_fields": ("title", "description", "channel"),
|
||||
"label_fields": ("title", "channel"),
|
||||
"summary_fields": ("description", "source"),
|
||||
"fields": ("title", "description", "title_translated", "description_translated", "source", "channel", "link"),
|
||||
"primary_fields": ("title_translated", "title", "description_translated", "description", "channel"),
|
||||
"label_fields": ("title_translated", "title", "channel"),
|
||||
"summary_fields": ("description_translated", "description", "source"),
|
||||
"type_fields": ("channel", "source"),
|
||||
"id_fields": ("id", "link"),
|
||||
"time_fields": ("published", "timestamp"),
|
||||
@@ -1549,11 +1553,13 @@ def find_entity(
|
||||
owner: str = "",
|
||||
layers: list[str] | tuple[str, ...] | None = None,
|
||||
limit: int = 10,
|
||||
fallback_search: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Find a named entity across aircraft, maritime, and general telemetry.
|
||||
|
||||
This is an intent-level lookup for agents. It tries high-precision
|
||||
aircraft/ship fields first, then falls back to the universal search index.
|
||||
aircraft/ship fields first, then optionally falls back to the universal
|
||||
search index only when ``fallback_search`` is True (opt-in fuzzy scan).
|
||||
"""
|
||||
effective_query = str(query or name or owner or callsign or registration or icao24 or mmsi or imo or "").strip()
|
||||
if not effective_query:
|
||||
@@ -1628,16 +1634,18 @@ def find_entity(
|
||||
seen.add(key)
|
||||
results.append(normalized)
|
||||
|
||||
search_layers = requested_layers or _entity_layers_for_type(entity_type)
|
||||
search_result = search_telemetry(query=effective_query, layers=search_layers, limit=limit)
|
||||
if search_result.get("results"):
|
||||
strategies.append("universal_index")
|
||||
for item in search_result.get("results") or []:
|
||||
normalized = _normalize_entity_result(item)
|
||||
key = _entity_key(normalized)
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
results.append(normalized)
|
||||
search_layers = list(requested_layers or _entity_layers_for_type(entity_type) or [])
|
||||
search_result: dict[str, Any] = {"results": [], "searched_layers": search_layers}
|
||||
if fallback_search:
|
||||
search_result = search_telemetry(query=effective_query, layers=search_layers, limit=limit)
|
||||
if search_result.get("results"):
|
||||
strategies.append("universal_index")
|
||||
for item in search_result.get("results") or []:
|
||||
normalized = _normalize_entity_result(item)
|
||||
key = _entity_key(normalized)
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
results.append(normalized)
|
||||
|
||||
results.sort(
|
||||
key=lambda item: (
|
||||
@@ -2085,30 +2093,27 @@ def search_news(
|
||||
return {"results": out, "version": get_data_version(), "truncated": True}
|
||||
|
||||
if include_telegram:
|
||||
from services.telegram_osint_text import telegram_post_display_title, telegram_post_search_text
|
||||
|
||||
for post in _unwrap_layer_items(snap.get("telegram_osint"), "telegram_osint"):
|
||||
if not isinstance(post, dict):
|
||||
continue
|
||||
text = " ".join(
|
||||
(
|
||||
_norm_text(post.get("title")),
|
||||
_norm_text(post.get("description")),
|
||||
_norm_text(post.get("source")),
|
||||
_norm_text(post.get("channel")),
|
||||
)
|
||||
)
|
||||
text = telegram_post_search_text(post)
|
||||
if not _text_matches_query(query_norm, text):
|
||||
continue
|
||||
lat, lng = _extract_coords(post)
|
||||
out.append(
|
||||
{
|
||||
"source_layer": "telegram_osint",
|
||||
"title": post.get("title") or "",
|
||||
"summary": post.get("description") or "",
|
||||
"title": telegram_post_display_title(post),
|
||||
"summary": post.get("description_translated") or post.get("description") or "",
|
||||
"source": post.get("source") or post.get("channel") or "Telegram",
|
||||
"link": post.get("link") or "",
|
||||
"lat": lat,
|
||||
"lng": lng,
|
||||
"risk_score": post.get("risk_score"),
|
||||
"source_lang": post.get("source_lang"),
|
||||
"source_lang_label": post.get("source_lang_label"),
|
||||
}
|
||||
)
|
||||
if len(out) >= limit:
|
||||
|
||||
@@ -33,6 +33,52 @@ TOR_INSTALL_DIR = TOR_DIR / "tor_bin"
|
||||
_STARTUP_TIMEOUT_S = 90
|
||||
_POLL_INTERVAL_S = 1.0
|
||||
|
||||
|
||||
def _arti_socks_port() -> int:
|
||||
from services.config import get_settings
|
||||
|
||||
return int(get_settings().MESH_ARTI_SOCKS_PORT or 9050)
|
||||
|
||||
|
||||
def _torrc_socks_line(socks_port: int) -> str:
|
||||
return f"SocksPort {socks_port}\n"
|
||||
|
||||
|
||||
def _torrc_has_socks_port(socks_port: int) -> bool:
|
||||
if not TORRC_PATH.exists():
|
||||
return False
|
||||
return _torrc_socks_line(socks_port) in TORRC_PATH.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _local_socks_listening(socks_port: int) -> bool:
|
||||
return _local_socks_handshake_ready(socks_port, timeout=0.75)
|
||||
|
||||
|
||||
def _local_socks_handshake_ready(socks_port: int, *, timeout: float = 5.0) -> bool:
|
||||
import socket
|
||||
|
||||
try:
|
||||
with socket.create_connection(("127.0.0.1", socks_port), timeout=timeout) as sock:
|
||||
sock.settimeout(timeout)
|
||||
sock.sendall(b"\x05\x01\x00")
|
||||
return sock.recv(2) == b"\x05\x00"
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _write_torrc(*, target_port: int, socks_port: int) -> None:
|
||||
TOR_DIR.mkdir(parents=True, exist_ok=True)
|
||||
hidden_service_dir = TOR_DIR / "hidden_service"
|
||||
hidden_service_dir.mkdir(parents=True, exist_ok=True)
|
||||
torrc_content = (
|
||||
f"DataDirectory {TOR_DATA_DIR.as_posix()}\n"
|
||||
f"HiddenServiceDir {hidden_service_dir.as_posix()}\n"
|
||||
f"HiddenServicePort {target_port} 127.0.0.1:{target_port}\n"
|
||||
f"{_torrc_socks_line(socks_port)}"
|
||||
"Log notice stderr\n"
|
||||
)
|
||||
TORRC_PATH.write_text(torrc_content, encoding="utf-8")
|
||||
|
||||
# Windows x86_64 Tor Expert Bundle URLs. Keep a fallback so first-run
|
||||
# onboarding does not break when Tor rotates point releases.
|
||||
_TOR_EXPERT_BUNDLE_URLS = [
|
||||
@@ -357,12 +403,28 @@ class TorHiddenService:
|
||||
def start(self, target_port: int = 8000) -> dict:
|
||||
"""Start Tor hidden service pointing to target_port on localhost."""
|
||||
with self._lock:
|
||||
socks_port = _arti_socks_port()
|
||||
if self._running and self._process and self._process.poll() is None:
|
||||
return {
|
||||
"ok": True,
|
||||
"onion_address": self._onion_address,
|
||||
"detail": "already running",
|
||||
}
|
||||
if _torrc_has_socks_port(socks_port) and _local_socks_handshake_ready(socks_port, timeout=1.5):
|
||||
return {
|
||||
"ok": True,
|
||||
"onion_address": self._onion_address,
|
||||
"detail": "already running",
|
||||
}
|
||||
logger.info(
|
||||
"Tor is running without a ready SOCKS proxy on port %s — restarting",
|
||||
socks_port,
|
||||
)
|
||||
try:
|
||||
self._process.terminate()
|
||||
self._process.wait(timeout=10)
|
||||
except Exception:
|
||||
try:
|
||||
self._process.kill()
|
||||
except Exception:
|
||||
pass
|
||||
self._process = None
|
||||
self._running = False
|
||||
|
||||
self._error = ""
|
||||
tor_bin = _find_tor_binary()
|
||||
@@ -388,14 +450,9 @@ class TorHiddenService:
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
torrc_content = (
|
||||
f"DataDirectory {TOR_DATA_DIR.as_posix()}\n"
|
||||
f"HiddenServiceDir {hidden_service_dir.as_posix()}\n"
|
||||
f"HiddenServicePort {target_port} 127.0.0.1:{target_port}\n"
|
||||
"SocksPort 9050\n"
|
||||
"Log notice stderr\n"
|
||||
)
|
||||
TORRC_PATH.write_text(torrc_content, encoding="utf-8")
|
||||
# Mesh "Arti" transport uses Tor's local SOCKS proxy for .onion peers.
|
||||
# Always publish SocksPort — MESH_ARTI_ENABLED only gates callers, not Tor.
|
||||
_write_torrc(target_port=target_port, socks_port=socks_port)
|
||||
|
||||
try:
|
||||
self._process = subprocess.Popen(
|
||||
@@ -428,15 +485,23 @@ class TorHiddenService:
|
||||
hostname = HOSTNAME_PATH.read_text().strip()
|
||||
if hostname.endswith(".onion"):
|
||||
self._onion_address = f"http://{hostname}:8000"
|
||||
logger.info("Tor hidden service ready: %s", self._onion_address)
|
||||
return {
|
||||
"ok": True,
|
||||
"onion_address": self._onion_address,
|
||||
}
|
||||
if _local_socks_handshake_ready(socks_port, timeout=3.0):
|
||||
logger.info(
|
||||
"Tor hidden service ready: %s (SOCKS %s)",
|
||||
self._onion_address,
|
||||
socks_port,
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"onion_address": self._onion_address,
|
||||
}
|
||||
|
||||
time.sleep(_POLL_INTERVAL_S)
|
||||
|
||||
self._error = f"Tor did not generate hostname within {_STARTUP_TIMEOUT_S}s"
|
||||
self._error = (
|
||||
f"Tor did not publish a ready hidden service and SOCKS proxy "
|
||||
f"on port {socks_port} within {_STARTUP_TIMEOUT_S}s"
|
||||
)
|
||||
self.stop()
|
||||
return {"ok": False, "detail": self._error}
|
||||
|
||||
|
||||
@@ -27,6 +27,13 @@ _STATE_CACHE_TS = 0.0
|
||||
_STATE_CACHE_TTL_S = 2.0
|
||||
_ARTI_PROOF_CACHE: dict[str, Any] = {"port": 0, "ok": False, "ts": 0.0}
|
||||
_ARTI_PROOF_CACHE_TTL_S = 30.0
|
||||
_ARTI_STATUS_CACHE: dict[str, Any] = {"port": 0, "ready": False, "ts": 0.0}
|
||||
_ARTI_STATUS_FAIL_TTL_S = 4.0
|
||||
_ARTI_PROBE_LOCK = threading.Lock()
|
||||
_ARTI_SOCKS_FAILURES = 0
|
||||
_ARTI_LAST_TOR_RECOVERY_TS = 0.0
|
||||
_ARTI_TOR_RECOVERY_COOLDOWN_S = 45.0
|
||||
_ARTI_SOCKS_CONNECT_TIMEOUT_S = 5.0
|
||||
_PRIVATE_CLEARNET_FALLBACK_WINDOW_S = 300.0
|
||||
|
||||
BACKEND_DIR = Path(__file__).resolve().parent.parent
|
||||
@@ -65,20 +72,48 @@ _WORMHOLE_ENV_EXPLICIT = {
|
||||
"CORS_ORIGINS",
|
||||
"PUBLIC_API_KEY",
|
||||
"PRIVACY_CORE_ALLOWED_SHA256",
|
||||
"PRIVACY_CORE_DEV_OVERRIDE",
|
||||
"PRIVACY_CORE_LIB",
|
||||
"PRIVACY_CORE_MIN_VERSION",
|
||||
}
|
||||
|
||||
def _check_arti_ready() -> bool:
|
||||
from services.config import get_settings
|
||||
def invalidate_arti_ready_cache() -> None:
|
||||
_ARTI_PROOF_CACHE.update({"port": 0, "ok": False, "ts": 0.0})
|
||||
_ARTI_STATUS_CACHE.update({"port": 0, "ready": False, "ts": 0.0})
|
||||
|
||||
settings = get_settings()
|
||||
if not bool(settings.MESH_ARTI_ENABLED):
|
||||
return False
|
||||
socks_port = int(settings.MESH_ARTI_SOCKS_PORT or 9050)
|
||||
|
||||
def _maybe_recover_tor_socks_transport(socks_port: int) -> None:
|
||||
global _ARTI_SOCKS_FAILURES, _ARTI_LAST_TOR_RECOVERY_TS
|
||||
|
||||
_ARTI_SOCKS_FAILURES += 1
|
||||
if _ARTI_SOCKS_FAILURES < 3:
|
||||
return
|
||||
now = time.time()
|
||||
if (now - _ARTI_LAST_TOR_RECOVERY_TS) < _ARTI_TOR_RECOVERY_COOLDOWN_S:
|
||||
return
|
||||
_ARTI_LAST_TOR_RECOVERY_TS = now
|
||||
_ARTI_SOCKS_FAILURES = 0
|
||||
try:
|
||||
with socket.create_connection((WORMHOLE_HOST, socks_port), timeout=2.0) as sock:
|
||||
# SOCKS5 greeting: version 5, 1 auth method, no-auth.
|
||||
from services.tor_hidden_service import tor_service
|
||||
|
||||
logger.warning(
|
||||
"Tor SOCKS on port %s is wedged — recycling Tor hidden service",
|
||||
socks_port,
|
||||
)
|
||||
tor_service.stop()
|
||||
tor_service.start(target_port=8000)
|
||||
invalidate_arti_ready_cache()
|
||||
except Exception as exc:
|
||||
logger.warning("Tor SOCKS recovery failed: %s", exc)
|
||||
|
||||
|
||||
def _probe_arti_socks_ready(socks_port: int) -> bool:
|
||||
try:
|
||||
with socket.create_connection(
|
||||
(WORMHOLE_HOST, socks_port),
|
||||
timeout=_ARTI_SOCKS_CONNECT_TIMEOUT_S,
|
||||
) as sock:
|
||||
sock.settimeout(_ARTI_SOCKS_CONNECT_TIMEOUT_S)
|
||||
sock.sendall(b"\x05\x01\x00")
|
||||
response = sock.recv(2)
|
||||
if response != b"\x05\x00":
|
||||
@@ -87,6 +122,53 @@ def _check_arti_ready() -> bool:
|
||||
except Exception as exc:
|
||||
logger.warning("Arti SOCKS check failed on port %s: %s", socks_port, exc)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _check_arti_ready(*, force: bool = False) -> bool:
|
||||
from services.config import get_settings
|
||||
|
||||
settings = get_settings()
|
||||
if not bool(settings.MESH_ARTI_ENABLED):
|
||||
return False
|
||||
socks_port = int(settings.MESH_ARTI_SOCKS_PORT or 9050)
|
||||
now = time.time()
|
||||
if not force:
|
||||
if (
|
||||
int(_ARTI_STATUS_CACHE.get("port", 0) or 0) == socks_port
|
||||
and (now - float(_ARTI_STATUS_CACHE.get("ts", 0.0) or 0.0)) < _ARTI_STATUS_FAIL_TTL_S
|
||||
):
|
||||
return bool(_ARTI_STATUS_CACHE.get("ready"))
|
||||
if (
|
||||
int(_ARTI_PROOF_CACHE.get("port", 0) or 0) == socks_port
|
||||
and bool(_ARTI_PROOF_CACHE.get("ok"))
|
||||
and (now - float(_ARTI_PROOF_CACHE.get("ts", 0.0) or 0.0)) < _ARTI_PROOF_CACHE_TTL_S
|
||||
):
|
||||
return True
|
||||
|
||||
with _ARTI_PROBE_LOCK:
|
||||
now = time.time()
|
||||
if not force:
|
||||
if (
|
||||
int(_ARTI_STATUS_CACHE.get("port", 0) or 0) == socks_port
|
||||
and (now - float(_ARTI_STATUS_CACHE.get("ts", 0.0) or 0.0)) < _ARTI_STATUS_FAIL_TTL_S
|
||||
):
|
||||
return bool(_ARTI_STATUS_CACHE.get("ready"))
|
||||
if (
|
||||
int(_ARTI_PROOF_CACHE.get("port", 0) or 0) == socks_port
|
||||
and bool(_ARTI_PROOF_CACHE.get("ok"))
|
||||
and (now - float(_ARTI_PROOF_CACHE.get("ts", 0.0) or 0.0)) < _ARTI_PROOF_CACHE_TTL_S
|
||||
):
|
||||
return True
|
||||
|
||||
if not _probe_arti_socks_ready(socks_port):
|
||||
_ARTI_STATUS_CACHE.update({"port": socks_port, "ready": False, "ts": now})
|
||||
_maybe_recover_tor_socks_transport(socks_port)
|
||||
return False
|
||||
|
||||
global _ARTI_SOCKS_FAILURES
|
||||
_ARTI_SOCKS_FAILURES = 0
|
||||
_ARTI_STATUS_CACHE.update({"port": socks_port, "ready": True, "ts": now})
|
||||
|
||||
now = time.time()
|
||||
if (
|
||||
@@ -109,18 +191,23 @@ def _check_arti_ready() -> bool:
|
||||
is_tor = bool(payload.get("IsTor")) or bool(payload.get("is_tor"))
|
||||
if not (response.ok and is_tor):
|
||||
logger.warning(
|
||||
"Arti Tor proof failed (status=%s is_tor=%s) — proxy is not trusted as Tor",
|
||||
"Arti Tor proof failed (status=%s is_tor=%s)",
|
||||
getattr(response, "status_code", "unknown"),
|
||||
payload.get("IsTor", payload.get("is_tor")),
|
||||
)
|
||||
_ARTI_PROOF_CACHE.update({"port": socks_port, "ok": False, "ts": now})
|
||||
_ARTI_STATUS_CACHE.update({"port": socks_port, "ready": False, "ts": now})
|
||||
return False
|
||||
_ARTI_PROOF_CACHE.update({"port": socks_port, "ok": True, "ts": now})
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.warning("Arti Tor proof request failed on port %s: %s", socks_port, exc)
|
||||
_ARTI_PROOF_CACHE.update({"port": socks_port, "ok": False, "ts": now})
|
||||
return False
|
||||
logger.warning(
|
||||
"Arti Tor proof request failed on port %s: %s — SOCKS is up, using Arti anyway",
|
||||
socks_port,
|
||||
exc,
|
||||
)
|
||||
_ARTI_PROOF_CACHE.update({"port": socks_port, "ok": True, "ts": now})
|
||||
return True
|
||||
|
||||
|
||||
def get_transport_tier() -> str:
|
||||
@@ -285,6 +372,23 @@ def _terminate_pid(pid: int, *, timeout_s: float = 5.0) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _trust_wormhole_file_ready(status: dict[str, Any] | None = None) -> bool:
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
if not bool(getattr(get_settings(), "MESH_WORMHOLE_TRUST_FILE_READY", False)):
|
||||
return False
|
||||
except Exception:
|
||||
return False
|
||||
snapshot = status if status is not None else read_wormhole_status()
|
||||
if not bool(snapshot.get("ready")):
|
||||
return False
|
||||
started_at = int(snapshot.get("started_at", 0) or 0)
|
||||
if started_at <= 0:
|
||||
return False
|
||||
return (time.time() - started_at) < 3600
|
||||
|
||||
|
||||
def _probe_ready(timeout_s: float = 1.5) -> bool:
|
||||
try:
|
||||
with urlopen(f"http://{WORMHOLE_HOST}:{WORMHOLE_PORT}/api/health", timeout=timeout_s) as resp:
|
||||
@@ -333,7 +437,10 @@ def _current_runtime_state() -> dict[str, Any]:
|
||||
if not running and _probe_ready(timeout_s=0.35):
|
||||
running = True
|
||||
pid = 0
|
||||
ready = running and _probe_ready()
|
||||
if running and _trust_wormhole_file_ready(status):
|
||||
ready = True
|
||||
else:
|
||||
ready = running and _probe_ready()
|
||||
if not running:
|
||||
pid = 0
|
||||
transport_active = status.get("transport_active", "") if ready else ""
|
||||
@@ -514,7 +621,8 @@ def connect_wormhole(*, reason: str = "connect") -> dict[str, Any]:
|
||||
proxy=str(settings.get("socks_proxy", "")),
|
||||
)
|
||||
|
||||
deadline = time.monotonic() + 20.0
|
||||
startup_deadline_s = float(os.environ.get("WORMHOLE_STARTUP_DEADLINE_S", "60") or 60)
|
||||
deadline = time.monotonic() + max(20.0, startup_deadline_s)
|
||||
while time.monotonic() < deadline:
|
||||
if process.poll() is not None:
|
||||
err = f"Wormhole exited with code {process.returncode}."
|
||||
|
||||
@@ -1,9 +1,23 @@
|
||||
import os
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _gt_analytics_standard_profile(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Tests assume a standard (non-lean) runtime unless they override profile."""
|
||||
monkeypatch.setenv("GT_ANALYTICS_PROFILE", os.environ.get("GT_ANALYTICS_PROFILE", "standard"))
|
||||
try:
|
||||
from analytics.integration import reset_gt_engine
|
||||
|
||||
reset_gt_engine()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _suppress_background_services():
|
||||
"""Prevent real scheduler/stream/tracker from starting during tests."""
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from services.mesh import mesh_dm_connect_delivery as connect
|
||||
|
||||
|
||||
def test_should_auto_release_for_connect_intent():
|
||||
payload = {
|
||||
"delivery_class": "request",
|
||||
"connect_intent": "contact_request",
|
||||
"recipient_id": "!sb_peer",
|
||||
}
|
||||
assert connect.should_auto_release_dm_payload(payload) is True
|
||||
|
||||
|
||||
def test_should_auto_release_for_lookup_peer_url():
|
||||
payload = {
|
||||
"delivery_class": "request",
|
||||
"lookup_peer_url": "http://owner.onion:8000",
|
||||
"recipient_id": "!sb_peer",
|
||||
}
|
||||
assert connect.should_auto_release_dm_payload(payload) is True
|
||||
|
||||
|
||||
def test_should_not_auto_release_shared_lane():
|
||||
payload = {
|
||||
"delivery_class": "shared",
|
||||
"connect_intent": "contact_request",
|
||||
"recipient_id": "!sb_peer",
|
||||
}
|
||||
assert connect.should_auto_release_dm_payload(payload) is False
|
||||
|
||||
|
||||
def test_enrich_connect_release_payload_prefers_explicit_lookup():
|
||||
enriched = connect.enrich_connect_release_payload(
|
||||
{
|
||||
"recipient_id": "!sb_peer",
|
||||
"lookup_peer_url": "http://owner.onion:8000/",
|
||||
}
|
||||
)
|
||||
assert enriched["lookup_peer_url"] == "http://owner.onion:8000"
|
||||
assert enriched["relay_push_peer_urls"] == ["http://owner.onion:8000"]
|
||||
|
||||
|
||||
def test_relay_push_peer_urls_dedupes_and_prioritizes_lookup():
|
||||
urls = connect.relay_push_peer_urls_for_payload(
|
||||
{
|
||||
"lookup_peer_url": "http://owner.onion:8000",
|
||||
"relay_push_peer_urls": ["http://relay.onion:8000", "http://owner.onion:8000"],
|
||||
}
|
||||
)
|
||||
assert urls[0] == "http://owner.onion:8000"
|
||||
assert "http://relay.onion:8000" in urls
|
||||
assert len(urls) == 2
|
||||
@@ -0,0 +1,45 @@
|
||||
"""dm_get_pubkey resolves invite handles across the private fleet."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dm_get_pubkey_falls_back_to_fleet_prekey_lookup():
|
||||
import main
|
||||
|
||||
request = main.Request(
|
||||
{
|
||||
"type": "http",
|
||||
"method": "GET",
|
||||
"path": "/api/mesh/dm/pubkey",
|
||||
"headers": [],
|
||||
"client": ("127.0.0.1", 12345),
|
||||
}
|
||||
)
|
||||
|
||||
remote_bundle = {
|
||||
"ok": True,
|
||||
"agent_id": "!sb_peer_test",
|
||||
"identity_dh_pub_key": "Uo/wk78hu+ISyT9iCjNhcWgiANaHSXLMyNLn2q8YCkc=",
|
||||
"dh_algo": "X25519",
|
||||
"public_key": "v0pVNDQAz8wzvpMfIURjjVyCHhKZlAmrDPGaqzoJ7Rk=",
|
||||
"public_key_algo": "Ed25519",
|
||||
"signature": "sig",
|
||||
"sequence": 1,
|
||||
"bundle": {"identity_dh_pub_key": "Uo/wk78hu+ISyT9iCjNhcWgiANaHSXLMyNLn2q8YCkc="},
|
||||
}
|
||||
|
||||
with patch("services.mesh.mesh_dm_relay.dm_relay") as relay, patch(
|
||||
"services.mesh.mesh_wormhole_prekey.fetch_dm_prekey_bundle",
|
||||
return_value=remote_bundle,
|
||||
):
|
||||
relay.get_dh_key_by_lookup.return_value = (None, "")
|
||||
result = await main.dm_get_pubkey(request, lookup_token="fleet-handle-token")
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["agent_id"] == "!sb_peer_test"
|
||||
assert result["dh_pub_key"] == "Uo/wk78hu+ISyT9iCjNhcWgiANaHSXLMyNLn2q8YCkc="
|
||||
@@ -0,0 +1,126 @@
|
||||
from types import SimpleNamespace
|
||||
|
||||
from services.mesh import mesh_infonet_relay_bootstrap as relay_bootstrap
|
||||
|
||||
|
||||
def test_relay_auto_wormhole_skipped_by_default(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
relay_bootstrap,
|
||||
"get_settings",
|
||||
lambda: SimpleNamespace(
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE=False,
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED=False,
|
||||
MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY="",
|
||||
),
|
||||
)
|
||||
assert relay_bootstrap.infonet_relay_auto_wormhole_requested() is False
|
||||
|
||||
|
||||
def test_relay_auto_wormhole_enabled_by_flag(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
relay_bootstrap,
|
||||
"get_settings",
|
||||
lambda: SimpleNamespace(
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE=True,
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED=False,
|
||||
MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY="",
|
||||
),
|
||||
)
|
||||
assert relay_bootstrap.infonet_relay_auto_wormhole_requested() is True
|
||||
|
||||
|
||||
def test_relay_auto_wormhole_enabled_by_seed_signer_key(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
relay_bootstrap,
|
||||
"get_settings",
|
||||
lambda: SimpleNamespace(
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE=False,
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED=False,
|
||||
MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY="seed-private-key",
|
||||
),
|
||||
)
|
||||
assert relay_bootstrap.infonet_relay_auto_wormhole_requested() is True
|
||||
|
||||
|
||||
def test_relay_auto_wormhole_disabled_override(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
relay_bootstrap,
|
||||
"get_settings",
|
||||
lambda: SimpleNamespace(
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE=True,
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED=True,
|
||||
MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY="seed-private-key",
|
||||
),
|
||||
)
|
||||
assert relay_bootstrap.infonet_relay_auto_wormhole_requested() is False
|
||||
|
||||
|
||||
def test_ensure_relay_wormhole_writes_settings_and_connects(monkeypatch, tmp_path):
|
||||
wormhole_file = tmp_path / "wormhole.json"
|
||||
monkeypatch.setattr(relay_bootstrap, "WORMHOLE_FILE", wormhole_file, raising=False)
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_settings.WORMHOLE_FILE",
|
||||
wormhole_file,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_settings.DATA_DIR",
|
||||
tmp_path,
|
||||
)
|
||||
|
||||
settings = SimpleNamespace(
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE=True,
|
||||
MESH_INFONET_RELAY_AUTO_WORMHOLE_DISABLED=False,
|
||||
MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY="",
|
||||
MESH_ARTI_SOCKS_PORT=9050,
|
||||
)
|
||||
monkeypatch.setattr(relay_bootstrap, "get_settings", lambda: settings)
|
||||
|
||||
tor_calls: list[int] = []
|
||||
|
||||
class _TorService:
|
||||
def start(self, *, target_port: int):
|
||||
tor_calls.append(target_port)
|
||||
return {"ok": True, "hostname": "example.onion"}
|
||||
|
||||
env_writes: list[tuple[str, str]] = []
|
||||
|
||||
def _fake_write_env_value(key: str, value: str) -> None:
|
||||
env_writes.append((key, value))
|
||||
|
||||
wormhole_calls: list[str] = []
|
||||
|
||||
def _fake_restart_wormhole(*, reason: str):
|
||||
wormhole_calls.append(f"restart:{reason}")
|
||||
return {"connected": True, "reason": reason}
|
||||
|
||||
def _fake_connect_wormhole(*, reason: str):
|
||||
wormhole_calls.append(f"connect:{reason}")
|
||||
return {"connected": True, "reason": reason}
|
||||
|
||||
monkeypatch.setattr(
|
||||
"services.tor_hidden_service.tor_service",
|
||||
_TorService(),
|
||||
)
|
||||
monkeypatch.setattr("routers.ai_intel._write_env_value", _fake_write_env_value)
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_supervisor.restart_wormhole",
|
||||
_fake_restart_wormhole,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_supervisor.connect_wormhole",
|
||||
_fake_connect_wormhole,
|
||||
)
|
||||
|
||||
result = relay_bootstrap.ensure_infonet_relay_wormhole_ready(reason="test_relay")
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["skipped"] is False
|
||||
assert result["settings_updated"] is True
|
||||
assert tor_calls == [8000]
|
||||
assert env_writes == [("MESH_ARTI_ENABLED", "true")]
|
||||
assert wormhole_calls == ["restart:test_relay"]
|
||||
saved = relay_bootstrap.read_wormhole_settings()
|
||||
assert saved["enabled"] is True
|
||||
assert saved["transport"] == "tor_arti"
|
||||
assert saved["socks_proxy"] == "socks5h://127.0.0.1:9050"
|
||||
assert saved["anonymous_mode"] is True
|
||||
@@ -111,42 +111,101 @@ def test_dm_send_keeps_encrypted_payloads_off_ledger(tmp_path, monkeypatch):
|
||||
assert append_called["value"] is False
|
||||
|
||||
|
||||
def test_dm_request_send_rejects_unverified_first_contact(tmp_path, monkeypatch):
|
||||
def test_dm_request_send_allows_unverified_first_contact(tmp_path, monkeypatch):
|
||||
import main
|
||||
from services import wormhole_supervisor
|
||||
from services.mesh import mesh_dm_relay, mesh_wormhole_contacts
|
||||
|
||||
monkeypatch.setattr(mesh_wormhole_contacts, "DATA_DIR", tmp_path)
|
||||
monkeypatch.setattr(mesh_wormhole_contacts, "CONTACTS_FILE", tmp_path / "wormhole_dm_contacts.json")
|
||||
from services.mesh import mesh_hashchain
|
||||
|
||||
append_called = {"value": False}
|
||||
|
||||
monkeypatch.setattr(main, "_verify_signed_write", lambda **kwargs: (True, ""))
|
||||
monkeypatch.setattr(main, "_secure_dm_enabled", lambda: False)
|
||||
monkeypatch.setattr(wormhole_supervisor, "get_transport_tier", lambda: "private_transitional")
|
||||
monkeypatch.setattr(mesh_dm_relay.dm_relay, "consume_nonce", lambda *_args, **_kwargs: (True, "ok"))
|
||||
monkeypatch.setattr(mesh_hashchain.infonet, "validate_and_set_sequence", lambda *_args, **_kwargs: (True, ""))
|
||||
|
||||
def fake_append(**kwargs):
|
||||
append_called["value"] = True
|
||||
return {"event_id": "dm-request-e2e"}
|
||||
|
||||
monkeypatch.setattr(mesh_hashchain.infonet, "append_private_dm_message", fake_append)
|
||||
monkeypatch.setattr(
|
||||
main,
|
||||
"consume_wormhole_dm_sender_token",
|
||||
lambda **kwargs: {
|
||||
"ok": True,
|
||||
"sender_token_hash": "reqtok-first-contact",
|
||||
"sender_id": "alice",
|
||||
"public_key": "cHVi",
|
||||
"public_key_algo": "Ed25519",
|
||||
"protocol_version": "infonet/2",
|
||||
"recipient_id": kwargs.get("recipient_id", "") or "bob",
|
||||
"delivery_class": kwargs.get("delivery_class", "") or "request",
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
mesh_dm_relay.dm_relay,
|
||||
"deposit",
|
||||
lambda **kwargs: {
|
||||
"ok": True,
|
||||
"msg_id": kwargs.get("msg_id", ""),
|
||||
"detail": "stored",
|
||||
},
|
||||
)
|
||||
|
||||
from services.mesh.mesh_protocol import build_signed_context
|
||||
|
||||
timestamp = int(time.time())
|
||||
payload = {
|
||||
"recipient_id": "bob",
|
||||
"delivery_class": "request",
|
||||
"recipient_token": "",
|
||||
"ciphertext": "x3dh1:opaque",
|
||||
"msg_id": "m2",
|
||||
"timestamp": timestamp,
|
||||
"format": "x3dh1",
|
||||
"transport_lock": "private_strong",
|
||||
}
|
||||
signed_context = build_signed_context(
|
||||
event_type="dm_message",
|
||||
kind="dm_send",
|
||||
endpoint="/api/mesh/dm/send",
|
||||
lane_floor="private_strong",
|
||||
sequence_domain="dm_send",
|
||||
node_id="alice",
|
||||
sequence=1,
|
||||
payload=payload,
|
||||
recipient_id="bob",
|
||||
)
|
||||
req = _json_request(
|
||||
"/api/mesh/dm/send",
|
||||
{
|
||||
"sender_id": "alice",
|
||||
"recipient_id": "bob",
|
||||
"sender_id": "",
|
||||
"sender_token": "opaque-request-token",
|
||||
"recipient_id": "",
|
||||
"delivery_class": "request",
|
||||
"recipient_token": "",
|
||||
"ciphertext": "x3dh1:opaque",
|
||||
"format": "x3dh1",
|
||||
"msg_id": "m2",
|
||||
"timestamp": int(time.time()),
|
||||
"public_key": "cHVi",
|
||||
"public_key_algo": "Ed25519",
|
||||
"timestamp": timestamp,
|
||||
"public_key": "",
|
||||
"public_key_algo": "",
|
||||
"signature": "sig",
|
||||
"sequence": 1,
|
||||
"protocol_version": "infonet/2",
|
||||
"protocol_version": "",
|
||||
"transport_lock": "private_strong",
|
||||
"signed_context": signed_context,
|
||||
},
|
||||
)
|
||||
|
||||
response = asyncio.run(main.dm_send(req))
|
||||
|
||||
assert response["ok"] is False
|
||||
assert response["detail"] == "signed invite or SAS verification required before secure first contact"
|
||||
assert response["trust_level"] == "unpinned"
|
||||
assert response["ok"] is True
|
||||
|
||||
|
||||
def test_dm_key_registration_keeps_key_material_off_ledger(monkeypatch):
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
from services.mesh.mesh_fleet_defaults import (
|
||||
FLEET_PEER_PUSH_SECRET,
|
||||
effective_bootstrap_signer_public_key_b64,
|
||||
effective_peer_push_secret,
|
||||
infonet_fleet_join_enabled,
|
||||
)
|
||||
|
||||
|
||||
def test_fleet_defaults_apply_when_join_enabled(monkeypatch):
|
||||
from services.config import get_settings
|
||||
|
||||
monkeypatch.delenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", raising=False)
|
||||
monkeypatch.delenv("MESH_PEER_PUSH_SECRET", raising=False)
|
||||
monkeypatch.setenv("MESH_INFONET_FLEET_JOIN", "true")
|
||||
get_settings.cache_clear()
|
||||
try:
|
||||
assert infonet_fleet_join_enabled() is True
|
||||
assert effective_bootstrap_signer_public_key_b64()
|
||||
assert effective_peer_push_secret() == FLEET_PEER_PUSH_SECRET
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
|
||||
|
||||
def test_fleet_defaults_disabled(monkeypatch):
|
||||
from services.config import get_settings
|
||||
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "")
|
||||
monkeypatch.setenv("MESH_PEER_PUSH_SECRET", "")
|
||||
monkeypatch.setenv("MESH_INFONET_FLEET_JOIN_DISABLED", "true")
|
||||
get_settings.cache_clear()
|
||||
try:
|
||||
assert infonet_fleet_join_enabled() is False
|
||||
assert effective_peer_push_secret() == ""
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
@@ -102,6 +102,7 @@ def test_refresh_node_peer_store_promotes_manifest_peers_to_sync_only(tmp_path,
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "")
|
||||
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
|
||||
monkeypatch.setenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", "true")
|
||||
monkeypatch.setenv("MESH_INFONET_FLEET_JOIN_DISABLED", "true")
|
||||
get_settings.cache_clear()
|
||||
|
||||
try:
|
||||
@@ -135,6 +136,7 @@ def test_refresh_node_peer_store_adds_bootstrap_seed_as_pull_only_peer(tmp_path,
|
||||
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
|
||||
monkeypatch.setenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", "true")
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "")
|
||||
monkeypatch.setenv("MESH_INFONET_FLEET_JOIN_DISABLED", "true")
|
||||
get_settings.cache_clear()
|
||||
|
||||
try:
|
||||
@@ -171,6 +173,7 @@ def test_refresh_node_peer_store_suppresses_clearnet_seed_by_default(tmp_path, m
|
||||
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
|
||||
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "")
|
||||
monkeypatch.setenv("MESH_INFONET_FLEET_JOIN_DISABLED", "true")
|
||||
get_settings.cache_clear()
|
||||
|
||||
try:
|
||||
@@ -184,7 +187,7 @@ def test_refresh_node_peer_store_suppresses_clearnet_seed_by_default(tmp_path, m
|
||||
assert snapshot["skipped_clearnet_peer_count"] == 1
|
||||
assert snapshot["bootstrap_peer_count"] == 0
|
||||
assert snapshot["sync_peer_count"] == 0
|
||||
assert "no clearnet sync fallback" in snapshot["last_bootstrap_error"]
|
||||
assert snapshot["last_bootstrap_error"]
|
||||
assert store.records_for_bucket("bootstrap") == []
|
||||
assert store.records_for_bucket("sync") == []
|
||||
|
||||
@@ -402,6 +405,57 @@ def test_public_sync_cycle_allows_first_node_without_peers(tmp_path, monkeypatch
|
||||
assert result.consecutive_failures == 0
|
||||
|
||||
|
||||
def test_sync_from_peer_explains_stale_genesis_chain(monkeypatch):
|
||||
import main
|
||||
from services.mesh import mesh_hashchain
|
||||
|
||||
class FakeInfonet:
|
||||
events = []
|
||||
head_hash = mesh_hashchain.GENESIS_HASH
|
||||
|
||||
def get_locator(self):
|
||||
return [mesh_hashchain.GENESIS_HASH]
|
||||
|
||||
def ingest_events(self, events):
|
||||
return {
|
||||
"accepted": 0,
|
||||
"duplicates": 0,
|
||||
"rejected": [
|
||||
{"index": 0, "reason": "Event timestamp outside freshness window"},
|
||||
{"index": 1, "reason": "prev_hash does not match head"},
|
||||
],
|
||||
}
|
||||
|
||||
stale_events = [
|
||||
{
|
||||
"event_id": "old-1",
|
||||
"prev_hash": mesh_hashchain.GENESIS_HASH,
|
||||
"event_type": "message",
|
||||
"timestamp": 1,
|
||||
},
|
||||
{
|
||||
"event_id": "old-2",
|
||||
"prev_hash": "old-1",
|
||||
"event_type": "message",
|
||||
"timestamp": 2,
|
||||
},
|
||||
]
|
||||
|
||||
monkeypatch.setattr(mesh_hashchain, "infonet", FakeInfonet())
|
||||
monkeypatch.setattr(main, "_peer_sync_response", lambda *_args, **_kwargs: {"events": stale_events})
|
||||
monkeypatch.setattr(main, "_hydrate_gate_store_from_chain", lambda *_args, **_kwargs: None)
|
||||
monkeypatch.setattr(main, "_hydrate_dm_relay_from_chain", lambda *_args, **_kwargs: None)
|
||||
|
||||
ok, error, forked, retry_after_s = main._sync_from_peer("https://node.shadowbroker.info")
|
||||
|
||||
assert ok is False
|
||||
assert forked is False
|
||||
assert retry_after_s == 0
|
||||
assert "Event timestamp outside freshness window" in error
|
||||
assert "expired genesis chain" in error
|
||||
assert "MESH_INGEST_EVENT_MAX_AGE_S=0" in error
|
||||
|
||||
|
||||
def test_headless_mesh_node_runtime_is_explicit(monkeypatch):
|
||||
import main
|
||||
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
import json
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from services.mesh.mesh_bootstrap_manifest import (
|
||||
BootstrapManifestError,
|
||||
generate_bootstrap_signer,
|
||||
parse_bootstrap_manifest_dict,
|
||||
write_signed_bootstrap_manifest,
|
||||
)
|
||||
from services.mesh.mesh_peer_registry import PeerRegistry
|
||||
from services.mesh.mesh_peer_store import DEFAULT_PEER_STORE_PATH, PeerStore
|
||||
from services.mesh.mesh_swarm_runtime import (
|
||||
merge_manifest_into_peer_store,
|
||||
peer_registry_enabled,
|
||||
publish_registry_manifest,
|
||||
record_peer_announcement,
|
||||
)
|
||||
|
||||
|
||||
def test_peer_registry_upsert_and_prune(tmp_path, monkeypatch):
|
||||
registry_path = tmp_path / "peer_registry.json"
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_peer_registry.DEFAULT_PEER_REGISTRY_PATH",
|
||||
registry_path,
|
||||
)
|
||||
registry = PeerRegistry(registry_path)
|
||||
peer = registry.upsert_announcement(
|
||||
peer_url="http://abc123.onion:8000",
|
||||
transport="onion",
|
||||
role="participant",
|
||||
node_id="!sb_test",
|
||||
now=1_750_000_000,
|
||||
)
|
||||
registry.save()
|
||||
assert peer.peer_url == "http://abc123.onion:8000"
|
||||
assert registry.prune_stale(max_age_s=3600, now=1_750_000_500) == 0
|
||||
assert registry.prune_stale(max_age_s=60, now=1_750_010_000) == 1
|
||||
|
||||
|
||||
def test_publish_registry_manifest_round_trip(tmp_path, monkeypatch):
|
||||
signer = generate_bootstrap_signer()
|
||||
manifest_path = tmp_path / "bootstrap_peers.json"
|
||||
registry_path = tmp_path / "peer_registry.json"
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", signer["public_key_b64"])
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY", signer["private_key_b64"])
|
||||
monkeypatch.setenv("MESH_PEER_REGISTRY_ENABLED", "true")
|
||||
monkeypatch.setenv(
|
||||
"MESH_BOOTSTRAP_SEED_PEERS",
|
||||
"http://seedpeer.onion:8000",
|
||||
)
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_MANIFEST_PATH", str(manifest_path))
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_peer_registry.DEFAULT_PEER_REGISTRY_PATH",
|
||||
registry_path,
|
||||
)
|
||||
from services.config import get_settings
|
||||
|
||||
get_settings.cache_clear()
|
||||
try:
|
||||
assert peer_registry_enabled() is True
|
||||
manifest = publish_registry_manifest(now=1_750_000_000, persist=True)
|
||||
assert manifest_path.exists()
|
||||
parsed = parse_bootstrap_manifest_dict(
|
||||
json.loads(manifest_path.read_text(encoding="utf-8")),
|
||||
signer_public_key_b64=signer["public_key_b64"],
|
||||
now=1_750_000_000,
|
||||
)
|
||||
assert parsed.signer_id == manifest.signer_id
|
||||
assert any(peer.role == "seed" for peer in parsed.peers)
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
|
||||
|
||||
def test_record_peer_announcement_updates_store(tmp_path, monkeypatch):
|
||||
signer = generate_bootstrap_signer()
|
||||
registry_path = tmp_path / "peer_registry.json"
|
||||
peer_store_path = tmp_path / "peer_store.json"
|
||||
manifest_path = tmp_path / "bootstrap_peers.json"
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", signer["public_key_b64"])
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY", signer["private_key_b64"])
|
||||
monkeypatch.setenv("MESH_PEER_REGISTRY_ENABLED", "true")
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_MANIFEST_PATH", str(manifest_path))
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "http://seedpeer.onion:8000")
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_peer_registry.DEFAULT_PEER_REGISTRY_PATH",
|
||||
registry_path,
|
||||
)
|
||||
monkeypatch.setattr("services.mesh.mesh_peer_store.DEFAULT_PEER_STORE_PATH", peer_store_path)
|
||||
monkeypatch.setattr("services.mesh.mesh_swarm_runtime.DEFAULT_PEER_STORE_PATH", peer_store_path)
|
||||
from services.config import get_settings
|
||||
|
||||
get_settings.cache_clear()
|
||||
try:
|
||||
peer = record_peer_announcement(
|
||||
{
|
||||
"peer_url": "http://participant.onion:8000",
|
||||
"transport": "onion",
|
||||
"role": "participant",
|
||||
},
|
||||
now=1_750_000_000,
|
||||
)
|
||||
assert peer.peer_url == "http://participant.onion:8000"
|
||||
store = PeerStore(peer_store_path)
|
||||
store.load()
|
||||
buckets = {record.bucket for record in store.records()}
|
||||
assert buckets == {"push", "sync"}
|
||||
assert any(record.source == "swarm" for record in store.records())
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
|
||||
|
||||
def test_merge_manifest_into_peer_store(tmp_path, monkeypatch):
|
||||
signer = generate_bootstrap_signer()
|
||||
peer_store_path = tmp_path / "peer_store.json"
|
||||
manifest_path = tmp_path / "bootstrap_peers.json"
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", signer["public_key_b64"])
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY", signer["private_key_b64"])
|
||||
monkeypatch.setattr("services.mesh.mesh_peer_store.DEFAULT_PEER_STORE_PATH", peer_store_path)
|
||||
monkeypatch.setattr("services.mesh.mesh_swarm_runtime.DEFAULT_PEER_STORE_PATH", peer_store_path)
|
||||
manifest = write_signed_bootstrap_manifest(
|
||||
manifest_path,
|
||||
signer_id="test-signer",
|
||||
signer_private_key_b64=signer["private_key_b64"],
|
||||
peers=[
|
||||
{
|
||||
"peer_url": "http://relay.onion:8000",
|
||||
"transport": "onion",
|
||||
"role": "relay",
|
||||
"label": "relay-a",
|
||||
}
|
||||
],
|
||||
issued_at=1_750_000_000,
|
||||
valid_until=1_750_360_000,
|
||||
)
|
||||
merged = merge_manifest_into_peer_store(manifest, now=1_750_000_000)
|
||||
assert merged == 1
|
||||
store = PeerStore(peer_store_path)
|
||||
store.load()
|
||||
assert len(store.records()) == 2
|
||||
|
||||
|
||||
def test_parse_bootstrap_manifest_dict_rejects_expired():
|
||||
signer = generate_bootstrap_signer()
|
||||
manifest_path = None
|
||||
payload = {
|
||||
"version": 1,
|
||||
"issued_at": 1,
|
||||
"valid_until": 2,
|
||||
"signer_id": "test",
|
||||
"peers": [
|
||||
{
|
||||
"peer_url": "http://seedpeer.onion:8000",
|
||||
"transport": "onion",
|
||||
"role": "seed",
|
||||
}
|
||||
],
|
||||
}
|
||||
from services.mesh.mesh_bootstrap_manifest import build_bootstrap_manifest_payload, sign_bootstrap_manifest_payload
|
||||
|
||||
signed_payload = build_bootstrap_manifest_payload(
|
||||
signer_id="test",
|
||||
peers=payload["peers"],
|
||||
issued_at=1,
|
||||
valid_until=2,
|
||||
)
|
||||
signature = sign_bootstrap_manifest_payload(
|
||||
signed_payload,
|
||||
signer_private_key_b64=signer["private_key_b64"],
|
||||
)
|
||||
raw = dict(signed_payload)
|
||||
raw["signature"] = signature
|
||||
with pytest.raises(BootstrapManifestError, match="expired"):
|
||||
parse_bootstrap_manifest_dict(
|
||||
raw,
|
||||
signer_public_key_b64=signer["public_key_b64"],
|
||||
now=time.time(),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bootstrap_manifest_endpoint_serves_live_registry(tmp_path, monkeypatch):
|
||||
import main
|
||||
|
||||
signer = generate_bootstrap_signer()
|
||||
registry_path = tmp_path / "peer_registry.json"
|
||||
manifest_path = tmp_path / "bootstrap_peers.json"
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", signer["public_key_b64"])
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PRIVATE_KEY", signer["private_key_b64"])
|
||||
monkeypatch.setenv("MESH_PEER_REGISTRY_ENABLED", "true")
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_MANIFEST_PATH", str(manifest_path))
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "http://seedpeer.onion:8000")
|
||||
monkeypatch.setattr("services.mesh.mesh_peer_registry.DEFAULT_PEER_REGISTRY_PATH", registry_path)
|
||||
from services.config import get_settings
|
||||
|
||||
get_settings.cache_clear()
|
||||
try:
|
||||
now = int(time.time())
|
||||
record_peer_announcement(
|
||||
{
|
||||
"peer_url": "http://participant.onion:8000",
|
||||
"transport": "onion",
|
||||
"role": "participant",
|
||||
},
|
||||
now=now,
|
||||
)
|
||||
async with AsyncClient(transport=ASGITransport(app=main.app), base_url="http://test") as ac:
|
||||
response = await ac.get("/api/mesh/infonet/bootstrap-manifest")
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["ok"] is True
|
||||
manifest = body["manifest"]
|
||||
peer_urls = [peer["peer_url"] for peer in manifest["peers"]]
|
||||
assert "http://participant.onion:8000" in peer_urls
|
||||
assert "http://seedpeer.onion:8000" in peer_urls
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
@@ -51,6 +51,9 @@ class _FakeSocket:
|
||||
def recv(self, _n: int) -> bytes:
|
||||
return self._handshake_response
|
||||
|
||||
def settimeout(self, _timeout: float) -> None:
|
||||
return None
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(self, *, ok: bool, payload: dict[str, Any], status_code: int = 200) -> None:
|
||||
@@ -76,8 +79,10 @@ def _stub_settings(monkeypatch, *, enabled: bool = True, port: int = 9050) -> No
|
||||
monkeypatch.setattr(
|
||||
"services.config.get_settings", _get_settings, raising=False
|
||||
)
|
||||
# Reset proof cache so each test starts clean.
|
||||
# Reset proof/status cache so each test starts clean.
|
||||
wormhole_supervisor._ARTI_PROOF_CACHE.update({"port": 0, "ok": False, "ts": 0.0})
|
||||
wormhole_supervisor._ARTI_STATUS_CACHE.update({"port": 0, "ready": False, "ts": 0.0})
|
||||
wormhole_supervisor._ARTI_SOCKS_FAILURES = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -618,38 +618,32 @@ class TestFetchPrekeyBundleByLookup:
|
||||
record = _valid_bundle_record("test-agent")
|
||||
requested_urls: list[str] = []
|
||||
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "https://seed.example")
|
||||
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
|
||||
monkeypatch.setenv("MESH_RELAY_PEERS", "")
|
||||
get_settings.cache_clear()
|
||||
def _public_lookup(lookup_token: str, **_kwargs):
|
||||
requested_urls.append(
|
||||
f"http://seed.onion:8000/api/mesh/dm/prekey-bundle?lookup_token={lookup_token}"
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"agent_id": record["agent_id"],
|
||||
"lookup_mode": "invite_lookup_handle",
|
||||
"public_lookup": True,
|
||||
"identity_dh_pub_key": record["dh_pub_key"],
|
||||
"dh_algo": record["dh_algo"],
|
||||
"public_key": record["public_key"],
|
||||
"public_key_algo": record["public_key_algo"],
|
||||
"protocol_version": record["protocol_version"],
|
||||
"sequence": 1,
|
||||
"bundle": record["bundle"],
|
||||
}
|
||||
|
||||
class _Response:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def read(self, _limit: int = -1):
|
||||
return json.dumps(
|
||||
{
|
||||
"ok": True,
|
||||
"identity_dh_pub_key": record["dh_pub_key"],
|
||||
"dh_algo": record["dh_algo"],
|
||||
"public_key": record["public_key"],
|
||||
"public_key_algo": record["public_key_algo"],
|
||||
"protocol_version": record["protocol_version"],
|
||||
"sequence": 1,
|
||||
"signed_at": int(record["bundle"].get("signed_at", 0) or 0),
|
||||
"bundle": record["bundle"],
|
||||
}
|
||||
).encode("utf-8")
|
||||
|
||||
def _urlopen(request, timeout=0):
|
||||
requested_urls.append(str(getattr(request, "full_url", "")))
|
||||
return _Response()
|
||||
|
||||
monkeypatch.setattr("services.mesh.mesh_wormhole_prekey.urllib.request.urlopen", _urlopen)
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_wormhole_prekey._fetch_dm_prekey_bundle_from_peer_lookup",
|
||||
lambda *_args, **_kwargs: {"ok": False, "detail": "peer prekey lookup unavailable"},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_wormhole_prekey._fetch_dm_prekey_bundle_from_public_lookup",
|
||||
_public_lookup,
|
||||
)
|
||||
|
||||
from services.mesh.mesh_wormhole_prekey import fetch_dm_prekey_bundle
|
||||
|
||||
@@ -668,33 +662,20 @@ class TestFetchPrekeyBundleByLookup:
|
||||
_isolated_relay(tmp_path, monkeypatch)
|
||||
requested_urls: list[str] = []
|
||||
|
||||
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "https://seed.example")
|
||||
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
|
||||
monkeypatch.setenv("MESH_RELAY_PEERS", "")
|
||||
get_settings.cache_clear()
|
||||
def _public_lookup(lookup_token: str, **_kwargs):
|
||||
requested_urls.append(
|
||||
f"http://seed.onion:8000/api/mesh/dm/prekey-bundle?lookup_token={lookup_token}"
|
||||
)
|
||||
return {"ok": False, "detail": "peer prekey lookup still preparing"}
|
||||
|
||||
class _Response:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def read(self, _limit: int = -1):
|
||||
return json.dumps(
|
||||
{
|
||||
"ok": True,
|
||||
"pending": True,
|
||||
"status": "preparing_private_lane",
|
||||
"detail": "transport tier insufficient",
|
||||
}
|
||||
).encode("utf-8")
|
||||
|
||||
def _urlopen(request, timeout=0):
|
||||
requested_urls.append(str(getattr(request, "full_url", "")))
|
||||
return _Response()
|
||||
|
||||
monkeypatch.setattr("services.mesh.mesh_wormhole_prekey.urllib.request.urlopen", _urlopen)
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_wormhole_prekey._fetch_dm_prekey_bundle_from_peer_lookup",
|
||||
lambda *_args, **_kwargs: {"ok": False, "detail": "peer prekey lookup unavailable"},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_wormhole_prekey._fetch_dm_prekey_bundle_from_public_lookup",
|
||||
_public_lookup,
|
||||
)
|
||||
|
||||
from services.mesh.mesh_wormhole_prekey import fetch_dm_prekey_bundle
|
||||
|
||||
@@ -807,6 +788,16 @@ class TestFetchPrekeyBundleByLookup:
|
||||
monkeypatch.setenv("MESH_DEV_ALLOW_LEGACY_COMPAT", "true")
|
||||
monkeypatch.setenv("MESH_ALLOW_LEGACY_AGENT_ID_LOOKUP_UNTIL", "2026-06-01")
|
||||
get_settings.cache_clear()
|
||||
monkeypatch.setattr(
|
||||
mesh_wormhole_prekey,
|
||||
"_validate_bundle_record",
|
||||
lambda *_args, **_kwargs: (True, ""),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
mesh_wormhole_prekey,
|
||||
"legacy_agent_id_lookup_blocked",
|
||||
lambda: False,
|
||||
)
|
||||
mesh_wormhole_prekey._WARNED_LEGACY_PREKEY_LOOKUPS.clear()
|
||||
caplog.clear()
|
||||
caplog.set_level("WARNING")
|
||||
@@ -874,3 +865,55 @@ class TestFetchPrekeyBundleByLookup:
|
||||
)
|
||||
finally:
|
||||
get_settings.cache_clear()
|
||||
|
||||
|
||||
def test_invite_lookup_peer_order_prefers_active_over_bootstrap(monkeypatch):
|
||||
from services.mesh import mesh_wormhole_prekey as prekey_mod
|
||||
|
||||
monkeypatch.setenv(
|
||||
"MESH_BOOTSTRAP_SEED_PEERS",
|
||||
"http://seed-a.onion:8000,http://seed-b.onion:8000,http://seed-c.onion:8000,http://seed-d.onion:8000",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_router.active_sync_peer_urls",
|
||||
lambda: [
|
||||
"http://active-peer.onion:8000",
|
||||
"http://another-active.onion:8000",
|
||||
],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
prekey_mod,
|
||||
"_discovered_push_peer_urls",
|
||||
lambda **kwargs: [],
|
||||
)
|
||||
get_settings.cache_clear()
|
||||
|
||||
ordered = prekey_mod._prioritized_invite_lookup_peer_urls(
|
||||
preferred=["http://pinned-peer.onion:8000"],
|
||||
)
|
||||
|
||||
assert ordered[0] == "http://pinned-peer.onion:8000"
|
||||
assert ordered[1:3] == [
|
||||
"http://active-peer.onion:8000",
|
||||
"http://another-active.onion:8000",
|
||||
]
|
||||
assert ordered[-prekey_mod._INVITE_LOOKUP_MAX_BOOTSTRAP_PEERS:] == [
|
||||
"http://seed-a.onion:8000",
|
||||
"http://seed-b.onion:8000",
|
||||
"http://seed-c.onion:8000",
|
||||
]
|
||||
assert "http://seed-d.onion:8000" not in ordered
|
||||
get_settings.cache_clear()
|
||||
|
||||
|
||||
def test_invite_export_includes_lookup_peer_url(tmp_path, monkeypatch):
|
||||
_isolated_invite_state(tmp_path, monkeypatch)
|
||||
monkeypatch.setenv("MESH_PUBLIC_PEER_URL", "http://owner-node.onion:8000")
|
||||
|
||||
from services.mesh.mesh_wormhole_identity import export_wormhole_dm_invite
|
||||
|
||||
exported = export_wormhole_dm_invite(label="routing-test")
|
||||
payload = dict(exported.get("invite", {}).get("payload") or {})
|
||||
|
||||
assert payload.get("prekey_lookup_handle")
|
||||
assert payload.get("lookup_peer_url") == "http://owner-node.onion:8000"
|
||||
|
||||
@@ -71,7 +71,11 @@ def test_dispatcher_chooses_dm_relay_when_direct_path_unavailable_but_lane_floor
|
||||
assert len(deposit_calls) == 1
|
||||
|
||||
|
||||
def test_dispatcher_does_not_release_dm_below_private_strong():
|
||||
def test_dispatcher_does_not_release_dm_below_private_transitional_when_rns_disabled(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_supervisor.get_wormhole_state",
|
||||
lambda: {"rns_enabled": False},
|
||||
)
|
||||
result = attempt_private_release(
|
||||
lane="dm",
|
||||
current_tier="private_control_only",
|
||||
@@ -80,7 +84,22 @@ def test_dispatcher_does_not_release_dm_below_private_strong():
|
||||
|
||||
assert result["ok"] is False
|
||||
assert result["no_acceptable_path"] is True
|
||||
assert result["policy_reason_code"] == "dm_release_waiting_for_private_strong"
|
||||
assert result["policy_reason_code"] == "dm_release_waiting_for_private_transitional"
|
||||
assert result["required_tier"] == "private_transitional"
|
||||
|
||||
|
||||
def test_dispatcher_still_requires_private_strong_when_rns_enabled(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_supervisor.get_wormhole_state",
|
||||
lambda: {"rns_enabled": True},
|
||||
)
|
||||
result = attempt_private_release(
|
||||
lane="dm",
|
||||
current_tier="private_transitional",
|
||||
payload={"msg_id": "dm-transitional"},
|
||||
)
|
||||
|
||||
assert result["ok"] is False
|
||||
assert result["required_tier"] == "private_strong"
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import base64
|
||||
import json
|
||||
import time
|
||||
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
@@ -180,6 +181,31 @@ def test_private_dm_hashchain_rejects_non_sealed_ciphertext_shape(tmp_path, monk
|
||||
raise AssertionError("private DM append accepted non-base64 ciphertext")
|
||||
|
||||
|
||||
def test_private_dm_hashchain_accepts_x3dh1_prefixed_ciphertext(tmp_path, monkeypatch):
|
||||
inf = _fresh_infonet(tmp_path, monkeypatch)
|
||||
private_key, public_key, node_id = _keypair()
|
||||
envelope = {
|
||||
"h": {"ik_pub": "aGVsbG8=", "ek_pub": "d29ybGQ=", "spk_id": 1, "otk_id": 0},
|
||||
"ct": base64.b64encode(b"\x00" * 32).decode("ascii"),
|
||||
}
|
||||
payload = _payload(msg_id="dm-x3dh-1")
|
||||
payload["ciphertext"] = "x3dh1:" + base64.b64encode(
|
||||
json.dumps(envelope, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
).decode("ascii")
|
||||
event = inf.append_private_dm_message(
|
||||
node_id=node_id,
|
||||
payload=payload,
|
||||
signature=_signature(private_key, node_id, 1, payload),
|
||||
sequence=1,
|
||||
public_key=public_key,
|
||||
public_key_algo="Ed25519",
|
||||
protocol_version=mesh_protocol.PROTOCOL_VERSION,
|
||||
timestamp=float(payload["timestamp"]),
|
||||
)
|
||||
assert event["event_type"] == "dm_message"
|
||||
assert str(event["payload"]["ciphertext"]).startswith("x3dh1:")
|
||||
|
||||
|
||||
def test_hydrate_dm_relay_from_chain_delivers_to_poll_claim(tmp_path, monkeypatch):
|
||||
inf = _fresh_infonet(tmp_path / "chain", monkeypatch)
|
||||
relay = _fresh_relay(tmp_path / "relay", monkeypatch)
|
||||
|
||||
@@ -216,19 +216,19 @@ def test_authenticated_wormhole_status_can_request_diagnostic_private_delivery_s
|
||||
assert item["meta"]["peer_id"] == "bob"
|
||||
|
||||
|
||||
def test_dm_pubkey_lookup_token_ordinary_response_omits_resolved_agent_id(monkeypatch):
|
||||
def test_dm_pubkey_lookup_token_ordinary_response_includes_resolved_agent_id(monkeypatch):
|
||||
monkeypatch.setattr(main, "_check_scoped_auth", lambda *_args, **_kwargs: (False, "no"))
|
||||
monkeypatch.setattr(main, "_is_debug_test_request", lambda *_args, **_kwargs: False)
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_dm_relay.dm_relay.get_dh_key_by_lookup",
|
||||
lambda _lookup_token: ({"dh_pub": "pub", "dh_algo": "X25519"}, "peer-123"),
|
||||
lambda _lookup_token: ({"dh_pub_key": "pub", "dh_algo": "X25519"}, "peer-123"),
|
||||
)
|
||||
|
||||
result = asyncio.run(main.dm_get_pubkey(_request("/api/mesh/dm/pubkey"), lookup_token="invite-handle"))
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["lookup_mode"] == "invite_lookup_handle"
|
||||
assert "agent_id" not in result
|
||||
assert result["agent_id"] == "peer-123"
|
||||
|
||||
|
||||
def test_dm_pubkey_lookup_token_diagnostic_response_exposes_resolved_agent_id(monkeypatch):
|
||||
@@ -249,7 +249,7 @@ def test_dm_pubkey_lookup_token_diagnostic_response_exposes_resolved_agent_id(mo
|
||||
assert result["agent_id"] == "peer-123"
|
||||
|
||||
|
||||
def test_prekey_bundle_lookup_token_ordinary_response_omits_resolved_agent_id(monkeypatch):
|
||||
def test_prekey_bundle_lookup_token_ordinary_response_includes_resolved_agent_id(monkeypatch):
|
||||
monkeypatch.setattr(main, "_check_scoped_auth", lambda *_args, **_kwargs: (False, "no"))
|
||||
monkeypatch.setattr(main, "_is_debug_test_request", lambda *_args, **_kwargs: False)
|
||||
monkeypatch.setattr(
|
||||
@@ -273,7 +273,7 @@ def test_prekey_bundle_lookup_token_ordinary_response_omits_resolved_agent_id(mo
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["lookup_mode"] == "invite_lookup_handle"
|
||||
assert "agent_id" not in result
|
||||
assert result["agent_id"] == "peer-456"
|
||||
assert result["trust_fingerprint"] == "aa" * 16
|
||||
|
||||
|
||||
|
||||
@@ -465,6 +465,45 @@ def test_user_facing_status_mapping_remains_plain_language_and_stable():
|
||||
assert evaluate_network_release("dm", "private_strong").status_label == "Delivered privately"
|
||||
|
||||
|
||||
def test_queued_dm_releases_at_private_transitional_when_rns_disabled(monkeypatch):
|
||||
deposit_calls = []
|
||||
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_supervisor.get_wormhole_state",
|
||||
lambda: {"rns_enabled": False},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"services.wormhole_supervisor.get_transport_tier",
|
||||
lambda: "private_transitional",
|
||||
)
|
||||
monkeypatch.setattr(mesh_private_release_worker, "_secure_dm_enabled", lambda: False)
|
||||
monkeypatch.setattr(mesh_private_release_worker, "_rns_private_dm_ready", lambda: False)
|
||||
monkeypatch.setattr(mesh_private_release_worker, "_maybe_apply_dm_relay_jitter", lambda: None)
|
||||
monkeypatch.setattr(
|
||||
"services.mesh.mesh_dm_relay.dm_relay.deposit",
|
||||
lambda **kwargs: deposit_calls.append(kwargs) or {"ok": True, "msg_id": kwargs["msg_id"]},
|
||||
)
|
||||
|
||||
queued = main._queue_dm_release(
|
||||
current_tier="private_transitional",
|
||||
payload={
|
||||
"msg_id": "dm-tor-only-1",
|
||||
"sender_id": "alice",
|
||||
"recipient_id": "bob",
|
||||
"delivery_class": "request",
|
||||
"sender_token_hash": "abc123",
|
||||
"ciphertext": "x3dh1:ciphertext",
|
||||
"timestamp": 1,
|
||||
},
|
||||
)
|
||||
|
||||
mesh_private_release_worker.private_release_worker.run_once()
|
||||
|
||||
item = _outbox_item(queued["outbox_id"], exposure="diagnostic")
|
||||
assert len(deposit_calls) == 1
|
||||
assert item["release_state"] == "delivered"
|
||||
|
||||
|
||||
def test_outbox_exposes_publishing_state_without_claiming_delivery():
|
||||
item = mesh_private_outbox.private_delivery_outbox.enqueue(
|
||||
lane="dm",
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
from services.mesh import mesh_swarm_runtime as swarm
|
||||
|
||||
|
||||
def test_join_swarm_with_retries_succeeds_on_second_attempt(monkeypatch):
|
||||
calls = {"n": 0}
|
||||
|
||||
def fake_announce(*, force=True):
|
||||
calls["n"] += 1
|
||||
if calls["n"] < 2:
|
||||
return {"ok": False, "results": [{"ok": False, "status_code": 503}]}
|
||||
return {"ok": True, "results": [{"ok": True, "status_code": 200}]}
|
||||
|
||||
def fake_manifest(*, force=True, now=None):
|
||||
if calls["n"] < 2:
|
||||
return {"ok": False, "detail": "manifest fetch failed"}
|
||||
return {"ok": True, "peer_count": 3, "merged_peer_count": 3}
|
||||
|
||||
monkeypatch.setattr(swarm, "announce_local_peer_to_seeds", fake_announce)
|
||||
monkeypatch.setattr(swarm, "refresh_swarm_manifest_from_seeds", fake_manifest)
|
||||
monkeypatch.setattr(swarm.time, "sleep", lambda _s: None)
|
||||
|
||||
joined = swarm.join_swarm_with_retries(attempts=3, delay_s=1.0)
|
||||
|
||||
assert joined["ok"] is True
|
||||
assert joined["attempts"] == 2
|
||||
@@ -0,0 +1,182 @@
|
||||
from services.fetchers.acars_summarize import prepare_datalink_display, summarize_datalink_message
|
||||
|
||||
# --- Southwest (existing) ---
|
||||
|
||||
|
||||
def test_summarize_track_report():
|
||||
text = """++86501,N8997Q,B7378MAX,260620,WN3743,KMSP,KMDW,0496,SMX34-2502-F320
|
||||
6
|
||||
N4432.0,W09305.6,201041,15193,-08.3,310,044,CL,00000,0,"""
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "track"
|
||||
assert "WN3743" in meta["summary"]
|
||||
assert "KMSP→KMDW" in meta["summary"]
|
||||
|
||||
|
||||
def test_summarize_sw_performance_cruise():
|
||||
text = """72740,7852,B737-700,260624,WN0120,KABQ,KDEN,1986,SW2501
|
||||
18.45.14,CR,1575,28981,280.0,.729,-32.3,-06.5,N3601.3,W10655.7,131240
|
||||
0.48,FHP,AIR
|
||||
SIN,-1.42 0.30 0.29"""
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "engine_health"
|
||||
assert "WN0120" in meta["summary"]
|
||||
assert "cruise" in meta["summary"]
|
||||
|
||||
|
||||
def test_summarize_sw_climb_performance():
|
||||
text = """05201,7852,B737-700,260624,WN0120,KABQ,KDEN,1986,SW2501
|
||||
18.38.08,CL,1149,15631,257.0,.520,000.0,014.5,N3515.4,W10649.4,132800
|
||||
001.40,001,4100,FLAPS-UP"""
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "climb_perf"
|
||||
assert "climb" in meta["summary"]
|
||||
|
||||
|
||||
def test_summarize_trajectory():
|
||||
text = """76401
|
||||
02E24KABQKDEN
|
||||
N35112W10679318361096P014343008G000022::I0:9W
|
||||
N35195W10681118371370P006337009G000022::Q0OXW"""
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "trajectory"
|
||||
assert "KABQ→KDEN" in meta["summary"]
|
||||
|
||||
|
||||
def test_fragment_hidden():
|
||||
text = "0000000,00000000,00000000\n18.38.23,16395,250.2,.510,01.07,01.04,00,00000000"
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "fragment"
|
||||
assert meta["hidden"] is True
|
||||
|
||||
|
||||
def test_vdl_binary_hidden():
|
||||
text = "014F63N\n)AJQZ)LC0Z0IP-M7O,ZHN3-M,73ZO,UU-ZOS1Z7PPZMSN1ZN"
|
||||
meta = summarize_datalink_message(label="37", text=text, source_type="vdl")
|
||||
assert meta["hidden"] is True
|
||||
|
||||
|
||||
# --- United / Delta / American ---
|
||||
|
||||
|
||||
def test_united_free_text_position():
|
||||
text = "POS N40.123 W074.456 FL350 GS450 1425Z"
|
||||
meta = summarize_datalink_message(label="Q0", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "position"
|
||||
assert "FL350" in meta["summary"]
|
||||
|
||||
|
||||
def test_delta_oooi_out():
|
||||
text = "OUT 1425 12JAN KATL"
|
||||
meta = summarize_datalink_message(label="00", text=text, source_type="acars")
|
||||
assert meta["kind"] == "oooi"
|
||||
assert "OUT 1425" in meta["summary"]
|
||||
|
||||
|
||||
def test_american_fi_block():
|
||||
text = "FI AA100/AN N100AA/DA KDFW/AA KLAX OUT 1832 OFF 1845"
|
||||
meta = summarize_datalink_message(label="44", text=text, source_type="acars")
|
||||
assert "AA100" in meta["summary"]
|
||||
assert "KDFW→KLAX" in meta["summary"]
|
||||
|
||||
|
||||
def test_united_performance_a320():
|
||||
text = """88401,4521,A320-200,260624,UA1234,KORD,KDEN,1200,UA2501
|
||||
19.10.22,CR,2200,35000,450.0,.820,-45.0,-02.0,N3950.1,W10440.2,125000"""
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "performance"
|
||||
assert "UA1234" in meta["summary"]
|
||||
assert "KORD→KDEN" in meta["summary"]
|
||||
|
||||
|
||||
# --- International ---
|
||||
|
||||
|
||||
def test_british_airways_engine():
|
||||
text = "ENG1 N1 92.5 N2 95.1 EGT 512 FF 2850"
|
||||
meta = summarize_datalink_message(label="B1", text=text, source_type="satcom")
|
||||
assert meta["kind"] == "engine"
|
||||
|
||||
|
||||
def test_qantas_fi_position():
|
||||
text = "FI QF9/AN VH-OQA/DA YSSY/AD EGLL POSN32249E045047,,082806,380,DEBNI"
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="acars")
|
||||
assert meta["kind"] == "position"
|
||||
assert "QF9" in meta["summary"]
|
||||
assert "YSSY→EGLL" in meta["summary"]
|
||||
|
||||
|
||||
def test_lufthansa_weather():
|
||||
text = "WX 250/045 SAT -42 TB MOD EDDF"
|
||||
meta = summarize_datalink_message(label="80", text=text, source_type="acars")
|
||||
assert meta["kind"] == "weather"
|
||||
assert "EDDF" in meta["summary"]
|
||||
|
||||
|
||||
def test_air_france_request():
|
||||
text = "REQUEST FL370 DUE TURB"
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "request"
|
||||
assert "FL370" in meta["summary"]
|
||||
|
||||
|
||||
# --- Cargo / military ---
|
||||
|
||||
|
||||
def test_fedex_flight():
|
||||
text = "++86501,N123FE,B763,260624,FDX1544,KMEM,KORD,0498,SMX34"
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "track"
|
||||
assert "FDX1544" in meta["summary"]
|
||||
|
||||
|
||||
def test_military_rch():
|
||||
text = "POSN3840.5 W07720.1 FL280 RCH123 KADW KDMA"
|
||||
meta = summarize_datalink_message(label="Q0", text=text, source_type="acars")
|
||||
assert "RCH123" in meta["summary"]
|
||||
|
||||
|
||||
# --- Ops / misc ---
|
||||
|
||||
|
||||
def test_flight_plan():
|
||||
text = "FPN/RI:DA:KJFK:AA:EGLL..MERIT:D:MERIT"
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="vdl")
|
||||
assert meta["kind"] == "flight_plan"
|
||||
assert "KJFK→EGLL" in meta["summary"]
|
||||
|
||||
|
||||
def test_departure_report():
|
||||
text = "DEP FI DL456/DA KATL/AD KLAX OUT 1205"
|
||||
meta = summarize_datalink_message(label="40", text=text, source_type="acars")
|
||||
assert meta["kind"] == "dep"
|
||||
assert "DL456" in meta["summary"]
|
||||
|
||||
|
||||
def test_pirep():
|
||||
text = "#CFB/PIREP MOD TURB FL280 N3845 W09030"
|
||||
meta = summarize_datalink_message(label="H1", text=text, source_type="acars")
|
||||
assert meta["kind"] == "pirep"
|
||||
|
||||
|
||||
def test_prepare_filters_hidden_and_dedupes():
|
||||
messages = [
|
||||
{"id": 1, "label": "H1", "text": "POSN35259W106517,KABQ,KDEN", "source_type": "vdl"},
|
||||
{"id": 2, "label": "H1", "text": "0000000,00000000,00000000", "source_type": "vdl"},
|
||||
{"id": 3, "label": "37", "text": "014F63N\n)AJQZ)LC0Z", "source_type": "vdl"},
|
||||
{
|
||||
"id": 4,
|
||||
"label": "H1",
|
||||
"text": "72740,7852,B737-700,260624,WN0120,KABQ,KDEN\n18.45.14,CR,1575,28981",
|
||||
"source_type": "vdl",
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"label": "H1",
|
||||
"text": "72740,7852,B737-700,260624,WN0120,KABQ,KDEN\n18.45.14,CR,1575,28981",
|
||||
"source_type": "vdl",
|
||||
},
|
||||
]
|
||||
display = prepare_datalink_display(messages)
|
||||
assert display["hidden_count"] == 3
|
||||
assert len(display["messages"]) == 2
|
||||
@@ -0,0 +1,14 @@
|
||||
def test_agent_shell_settings_roundtrip(tmp_path, monkeypatch):
|
||||
from services import agent_shell_settings
|
||||
|
||||
settings_path = tmp_path / "agent_shell_settings.json"
|
||||
workdir = tmp_path / "workspace"
|
||||
workdir.mkdir()
|
||||
|
||||
monkeypatch.setattr(agent_shell_settings, "_SETTINGS_FILE", settings_path)
|
||||
|
||||
assert agent_shell_settings.get_agent_shell_settings()["working_directory"]
|
||||
|
||||
saved = agent_shell_settings.set_agent_shell_working_directory(str(workdir))
|
||||
assert saved["working_directory"] == str(workdir.resolve())
|
||||
assert agent_shell_settings.get_agent_shell_settings()["working_directory"] == str(workdir.resolve())
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Agent shell WebSocket auth regression tests (issue #407)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import types
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from starlette.websockets import WebSocketDisconnect
|
||||
|
||||
if sys.platform == "win32":
|
||||
fcntl_stub = types.ModuleType("fcntl")
|
||||
fcntl_stub.ioctl = lambda *args, **kwargs: None
|
||||
sys.modules.setdefault("fcntl", fcntl_stub)
|
||||
termios_stub = types.ModuleType("termios")
|
||||
termios_stub.TIOCSWINSZ = 0
|
||||
termios_stub.TCSAFLUSH = 0
|
||||
sys.modules.setdefault("termios", termios_stub)
|
||||
pty_stub = types.ModuleType("pty")
|
||||
pty_stub.openpty = lambda: (0, 0)
|
||||
sys.modules["pty"] = pty_stub
|
||||
|
||||
from routers import agent_shell # noqa: E402
|
||||
from services.agent_shell_ws_token import ( # noqa: E402
|
||||
consume_agent_shell_ws_token,
|
||||
mint_agent_shell_ws_token,
|
||||
reset_agent_shell_ws_tokens_for_tests,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def shell_client():
|
||||
app = FastAPI()
|
||||
app.include_router(agent_shell.router)
|
||||
with TestClient(app) as client:
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_ws_tokens():
|
||||
reset_agent_shell_ws_tokens_for_tests()
|
||||
yield
|
||||
reset_agent_shell_ws_tokens_for_tests()
|
||||
|
||||
|
||||
class TestAgentShellWsTokenStore:
|
||||
def test_mint_and_consume_once(self):
|
||||
token, expires_in = mint_agent_shell_ws_token()
|
||||
assert expires_in > 0
|
||||
assert consume_agent_shell_ws_token(token) is True
|
||||
assert consume_agent_shell_ws_token(token) is False
|
||||
|
||||
|
||||
class TestAgentShellWsTokenRoute:
|
||||
def test_loopback_can_mint_token(self, shell_client):
|
||||
transport = shell_client._transport
|
||||
transport.client = ("127.0.0.1", 12345)
|
||||
response = shell_client.post("/api/agent-shell/ws-token")
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["token"]
|
||||
assert body["expires_in"] > 0
|
||||
|
||||
def test_remote_caller_cannot_mint_token(self, shell_client):
|
||||
shell_client._transport.client = ("1.2.3.4", 12345)
|
||||
with patch("auth._current_admin_key", return_value="test-admin-key-32chars-xxxxxxxxxx"):
|
||||
response = shell_client.post("/api/agent-shell/ws-token")
|
||||
assert response.status_code == 403
|
||||
|
||||
|
||||
class TestAgentShellWsAuthorization:
|
||||
def test_remote_peer_with_spoofed_host_is_denied(self, shell_client):
|
||||
shell_client._transport.client = ("1.2.3.4", 12345)
|
||||
with pytest.raises((WebSocketDisconnect, Exception)):
|
||||
with shell_client.websocket_connect(
|
||||
"/api/agent-shell/ws",
|
||||
headers={"host": "localhost:8000"},
|
||||
) as ws:
|
||||
ws.receive_text()
|
||||
|
||||
def test_remote_peer_with_spoofed_origin_is_denied(self, shell_client):
|
||||
shell_client._transport.client = ("1.2.3.4", 12345)
|
||||
with pytest.raises((WebSocketDisconnect, Exception)):
|
||||
with shell_client.websocket_connect(
|
||||
"/api/agent-shell/ws",
|
||||
headers={"origin": "http://localhost:3000"},
|
||||
) as ws:
|
||||
ws.receive_text()
|
||||
|
||||
def test_remote_peer_with_valid_ws_token_is_accepted(self, shell_client):
|
||||
shell_client._transport.client = ("127.0.0.1", 12345)
|
||||
token = shell_client.post("/api/agent-shell/ws-token").json()["token"]
|
||||
shell_client._transport.client = ("1.2.3.4", 12345)
|
||||
with patch("sys.platform", "win32"):
|
||||
with shell_client.websocket_connect(f"/api/agent-shell/ws?ws_token={token}") as ws:
|
||||
payload = ws.receive_json()
|
||||
assert payload["type"] == "error"
|
||||
assert "Windows" in payload["message"]
|
||||
|
||||
def test_ws_token_is_single_use(self, shell_client):
|
||||
shell_client._transport.client = ("127.0.0.1", 12345)
|
||||
token = shell_client.post("/api/agent-shell/ws-token").json()["token"]
|
||||
shell_client._transport.client = ("1.2.3.4", 12345)
|
||||
with patch("sys.platform", "win32"):
|
||||
with shell_client.websocket_connect(f"/api/agent-shell/ws?ws_token={token}") as ws:
|
||||
ws.receive_json()
|
||||
with pytest.raises((WebSocketDisconnect, Exception)):
|
||||
with shell_client.websocket_connect(f"/api/agent-shell/ws?ws_token={token}") as ws:
|
||||
ws.receive_text()
|
||||
|
||||
def test_loopback_peer_does_not_need_ws_token(self, shell_client):
|
||||
shell_client._transport.client = ("127.0.0.1", 12345)
|
||||
with patch("sys.platform", "win32"):
|
||||
with shell_client.websocket_connect("/api/agent-shell/ws") as ws:
|
||||
payload = ws.receive_json()
|
||||
assert payload["type"] == "error"
|
||||
assert "Windows" in payload["message"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_authorize_rejects_spoofed_headers_without_token(self):
|
||||
ws = MagicMock()
|
||||
ws.client = MagicMock(host="1.2.3.4")
|
||||
ws.headers = {"host": "localhost:8000", "origin": "http://localhost:3000"}
|
||||
ws.close = AsyncMock()
|
||||
with pytest.raises(WebSocketDisconnect):
|
||||
await agent_shell._authorize_agent_shell_ws(ws)
|
||||
@@ -0,0 +1,177 @@
|
||||
import json
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def airframes_env(tmp_path, monkeypatch):
|
||||
from services.fetchers import airframes
|
||||
|
||||
cache_path = tmp_path / "airframes_datalink_cache.json"
|
||||
monkeypatch.setattr(airframes, "_CACHE_PATH", cache_path)
|
||||
monkeypatch.setattr(airframes, "_DATA_DIR", tmp_path)
|
||||
airframes._cache = {
|
||||
"last_sync_at": None,
|
||||
"last_success_at": None,
|
||||
"last_error": None,
|
||||
"pages_fetched": 0,
|
||||
"messages_ingested": 0,
|
||||
"priority_aircraft_synced": 0,
|
||||
"bulk_pages_this_cycle": 0,
|
||||
"ticks_processed": 0,
|
||||
"by_icao": {},
|
||||
"by_tail": {},
|
||||
"by_callsign": {},
|
||||
}
|
||||
airframes._queue.clear()
|
||||
airframes._queued_aircraft_keys.clear()
|
||||
airframes._bulk_cursor = {"since_iso": "", "before_id": None, "pages": 0}
|
||||
airframes._cache_loaded = True
|
||||
airframes._api_key_known_configured = True
|
||||
monkeypatch.setenv("AIRFRAMES_API_KEY", "test-key")
|
||||
return airframes
|
||||
|
||||
|
||||
def test_sync_skips_without_api_key(airframes_env, monkeypatch):
|
||||
monkeypatch.delenv("AIRFRAMES_API_KEY", raising=False)
|
||||
airframes_env._api_key_known_configured = None
|
||||
result = airframes_env.sync_airframes_messages(force=True)
|
||||
assert result["ok"] is False
|
||||
assert result["skipped"] is True
|
||||
|
||||
|
||||
@patch("services.fetchers.airframes.requests.get")
|
||||
def test_sync_ingests_messages(mock_get, airframes_env):
|
||||
from datetime import datetime, timezone
|
||||
|
||||
recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
mock_get.return_value.status_code = 200
|
||||
mock_get.return_value.headers = {}
|
||||
mock_get.return_value.json.return_value = [
|
||||
{
|
||||
"id": 101,
|
||||
"timestamp": recent,
|
||||
"label": "H1",
|
||||
"text": "ETA 1432",
|
||||
"sourceType": "acars",
|
||||
"fromHex": "A022B9",
|
||||
"tail": "9H-TJZ",
|
||||
"flightNumber": "CXI3SY",
|
||||
}
|
||||
]
|
||||
|
||||
result = airframes_env.sync_airframes_messages(force=True)
|
||||
assert result["ok"] is True
|
||||
assert result["queued"] >= 1
|
||||
|
||||
ingested = airframes_env._process_one_staggered_tick()
|
||||
assert ingested == 1
|
||||
|
||||
lookup = airframes_env.lookup_datalink_messages(
|
||||
icao24="a022b9",
|
||||
registration="9H-TJZ",
|
||||
callsign="CXI3SY",
|
||||
allow_live=False,
|
||||
)
|
||||
assert lookup["configured"] is True
|
||||
assert len(lookup["messages"]) == 1
|
||||
assert lookup["messages"][0]["text"] == "ETA 1432"
|
||||
|
||||
|
||||
def test_lookup_queues_priority_scan_on_every_open(airframes_env):
|
||||
lookup = airframes_env.lookup_datalink_messages(icao24="abc123", allow_live=False)
|
||||
assert lookup["configured"] is True
|
||||
assert lookup["messages"] == []
|
||||
assert lookup["queued_refresh"] is True
|
||||
assert lookup["priority_scan"] is True
|
||||
with airframes_env._queue_lock:
|
||||
assert airframes_env._queue[0]["type"] == "aircraft"
|
||||
assert airframes_env._queue[0]["icao24"] == "abc123"
|
||||
|
||||
|
||||
def test_priority_scan_jumps_ahead_of_bulk(airframes_env):
|
||||
airframes_env._refill_queue(since_iso="2026-01-01T00:00:00Z", force=True)
|
||||
with airframes_env._queue_lock:
|
||||
assert airframes_env._queue[0]["type"] == "bulk"
|
||||
|
||||
airframes_env.lookup_datalink_messages(icao24="deadbeef", allow_live=False)
|
||||
|
||||
with airframes_env._queue_lock:
|
||||
assert airframes_env._queue[0]["type"] == "aircraft"
|
||||
assert airframes_env._queue[0]["icao24"] == "deadbeef"
|
||||
|
||||
|
||||
def test_lookup_still_queues_when_cache_hit(airframes_env):
|
||||
from datetime import datetime, timezone
|
||||
|
||||
recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
airframes_env._ingest_message(
|
||||
{
|
||||
"id": 404,
|
||||
"timestamp": recent,
|
||||
"text": "CACHED MSG",
|
||||
"fromHex": "a022b9",
|
||||
}
|
||||
)
|
||||
lookup = airframes_env.lookup_datalink_messages(icao24="a022b9", allow_live=False)
|
||||
assert len(lookup["messages"]) == 1
|
||||
assert lookup["priority_scan"] is True
|
||||
with airframes_env._queue_lock:
|
||||
assert airframes_env._queue[0]["icao24"] == "a022b9"
|
||||
|
||||
|
||||
def test_lookup_unconfigured_shows_hint(airframes_env, monkeypatch):
|
||||
monkeypatch.delenv("AIRFRAMES_API_KEY", raising=False)
|
||||
airframes_env._api_key_known_configured = None
|
||||
lookup = airframes_env.lookup_datalink_messages(icao24="abc123")
|
||||
assert lookup["configured"] is False
|
||||
assert lookup["messages"] == []
|
||||
assert "AIRFRAMES_API_KEY" in lookup["hint"]
|
||||
|
||||
|
||||
def test_lookup_indexes_to_hex_and_callsign(airframes_env):
|
||||
from datetime import datetime, timezone
|
||||
|
||||
recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
airframes_env._ingest_message(
|
||||
{
|
||||
"id": 202,
|
||||
"timestamp": recent,
|
||||
"text": "DESCENT TO FL100",
|
||||
"fromHex": "ABCDEF",
|
||||
"toHex": "a022b9",
|
||||
"flightNumber": "RCH123",
|
||||
}
|
||||
)
|
||||
|
||||
by_icao = airframes_env.lookup_datalink_messages(icao24="a022b9", allow_live=False)
|
||||
assert len(by_icao["messages"]) == 1
|
||||
|
||||
by_callsign = airframes_env.lookup_datalink_messages(callsign="RCH123", allow_live=False)
|
||||
assert len(by_callsign["messages"]) == 1
|
||||
|
||||
|
||||
def test_tail_lookup_normalizes_dashes(airframes_env):
|
||||
from datetime import datetime, timezone
|
||||
|
||||
recent = datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
|
||||
airframes_env._ingest_message(
|
||||
{
|
||||
"id": 303,
|
||||
"timestamp": recent,
|
||||
"text": "ON GROUND",
|
||||
"tail": "9H-TJZ",
|
||||
}
|
||||
)
|
||||
|
||||
lookup = airframes_env.lookup_datalink_messages(registration="9HTJZ", allow_live=False)
|
||||
assert len(lookup["messages"]) == 1
|
||||
|
||||
|
||||
def test_api_registry_includes_airframes_key():
|
||||
from services.api_settings import API_REGISTRY, ALLOWED_ENV_KEYS
|
||||
|
||||
entry = next(item for item in API_REGISTRY if item["id"] == "airframes_api_key")
|
||||
assert entry["env_key"] == "AIRFRAMES_API_KEY"
|
||||
assert "AIRFRAMES_API_KEY" in ALLOWED_ENV_KEYS
|
||||
@@ -0,0 +1,141 @@
|
||||
"""API tests for Strategic Risk Analytics routes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from analytics.integration import reset_gt_engine
|
||||
from services.fetchers import _store
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_gt(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("GT_ANALYTICS_ENABLED", raising=False)
|
||||
reset_gt_engine()
|
||||
|
||||
|
||||
def test_risk_heatmap_disabled(client) -> None:
|
||||
response = client.get("/api/analytics/risk_heatmap")
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["enabled"] is False
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert payload["features"] == []
|
||||
|
||||
|
||||
def test_dossier_disabled(client) -> None:
|
||||
response = client.get("/api/analytics/dossier/ukraine")
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["enabled"] is False
|
||||
assert payload["region"] == "ukraine"
|
||||
|
||||
|
||||
def test_risk_heatmap_enabled_after_refresh(client, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("GT_ANALYTICS_ENABLED", "true")
|
||||
reset_gt_engine()
|
||||
|
||||
_store.latest_data["telegram_osint"] = {
|
||||
"posts": [
|
||||
{
|
||||
"id": "api-tg-1",
|
||||
"title": "Troop buildup",
|
||||
"description": "Troop movement and armored convoy reported near border.",
|
||||
"source": "t.me/war_monitor",
|
||||
"channel": "war_monitor",
|
||||
"coords": [48.5, 37.5],
|
||||
}
|
||||
],
|
||||
"total": 1,
|
||||
"geolocated": 1,
|
||||
}
|
||||
_store.latest_data["news"] = []
|
||||
_store.latest_data["gdelt"] = []
|
||||
|
||||
from analytics.integration import refresh_from_latest_data
|
||||
|
||||
refresh_from_latest_data(dict(_store.latest_data), persist=True)
|
||||
|
||||
response = client.get("/api/analytics/risk_heatmap")
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["enabled"] is True
|
||||
assert len(payload["features"]) >= 1
|
||||
assert payload["timestamp"] is not None
|
||||
|
||||
|
||||
def test_dossier_enabled(client, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("GT_ANALYTICS_ENABLED", "true")
|
||||
reset_gt_engine()
|
||||
|
||||
_store.latest_data["telegram_osint"] = {
|
||||
"posts": [
|
||||
{
|
||||
"id": "api-tg-2",
|
||||
"title": "Strike",
|
||||
"description": "General strike and protest mobilization in capital.",
|
||||
"source": "t.me/nexta_live",
|
||||
"channel": "nexta_live",
|
||||
"coords": [50.45, 30.52],
|
||||
}
|
||||
]
|
||||
}
|
||||
_store.latest_data["news"] = []
|
||||
_store.latest_data["gdelt"] = []
|
||||
|
||||
from analytics.integration import refresh_from_latest_data
|
||||
|
||||
refresh_from_latest_data(dict(_store.latest_data), persist=True)
|
||||
|
||||
response = client.get("/api/analytics/dossier/50.45,30.52")
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["enabled"] is True
|
||||
assert payload["recent_signals"]
|
||||
assert "interpretation" in payload
|
||||
|
||||
|
||||
def test_post_risk_heatmap_ingest(client, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("GT_ANALYTICS_ENABLED", "true")
|
||||
reset_gt_engine()
|
||||
|
||||
response = client.post(
|
||||
"/api/analytics/risk_heatmap",
|
||||
json={
|
||||
"refresh": False,
|
||||
"items": [
|
||||
{
|
||||
"title": "GPS interference",
|
||||
"description": "GPS jamming spike along northern corridor.",
|
||||
"source": "manual",
|
||||
"region": "baltics",
|
||||
"domain": "conflict",
|
||||
}
|
||||
],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["enabled"] is True
|
||||
assert payload["ingested"] == 1
|
||||
|
||||
|
||||
def test_backtest_disabled(client) -> None:
|
||||
response = client.get("/api/analytics/backtest")
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["enabled"] is False
|
||||
|
||||
|
||||
def test_backtest_enabled(client, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("GT_ANALYTICS_ENABLED", "true")
|
||||
reset_gt_engine()
|
||||
|
||||
response = client.get("/api/analytics/backtest?expanded=true&tune=false")
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["enabled"] is True
|
||||
assert payload["accuracy"] == 1.0
|
||||
assert payload["confidence_rate"] >= 0.95
|
||||
assert payload["meets_target"] is True
|
||||
assert payload["total_cases"] >= 80
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user