CalvinBackup/Shadowbroker
1
0
Fork 0
mirror of https://github.com/BigBodyCobain/Shadowbroker.git synced 2026-07-28 14:38:53 +02:00
Code Issues Packages Projects Releases Wiki Activity

v0.9.6: InfoNet hashchain, Wormhole gate encryption, mesh reputation, 16 community contributors

Browse Source 
Gate messages now propagate via the Infonet hashchain as encrypted blobs — every node syncs them
through normal chain sync while only Gate members with MLS keys can decrypt. Added mesh reputation
system, peer push workers, voluntary Wormhole opt-in for node participation, fork recovery,
killwormhole scripts, obfuscated terminology, and hardened the self-updater to protect encryption
keys and chain state during updates.

New features: Shodan search, train tracking, Sentinel Hub imagery, 8 new intelligence layers,
CCTV expansion to 11,000+ cameras across 6 countries, Mesh Terminal CLI, prediction markets,
desktop-shell scaffold, and comprehensive mesh test suite (215 frontend + backend tests passing).

Community contributors: @wa1id, @AlborzNazari, @adust09, @Xpirix, @imqdcr, @csysp, @suranyami,
@chr0n1x, @johan-martensson, @singularfailure, @smithbh, @OrfeoTerkuci, @deuza, @tm-const,
@Elhard1, @ttulttul
This commit is contained in:
anoracleofra-code
2026-03-26 05:58:04 -06:00
parent d363013742
commit 668ce16dc7
363 changed files with 170430 additions and 23203 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/.env.example
+83 -1
View File
@@ -15,9 +15,91 @@ AIS_API_KEY= # https://aisstream.io/ — free tier WebSocket key
# CORS_ORIGINS=http://192.168.1.50:3000,https://my-domain.com
# Admin key — protects sensitive endpoints (API key management, system update).
# If unset, these endpoints remain open (fine for local dev).
# If unset, endpoints are only accessible from localhost unless ALLOW_INSECURE_ADMIN=true.
# Set this in production and enter the same key in Settings → Admin Key.
# ADMIN_KEY=your-secret-admin-key-here
# Allow insecure admin access without ADMIN_KEY (local dev only).
# ALLOW_INSECURE_ADMIN=false
# User-Agent for Nominatim geocoding requests (per OSM usage policy).
# NOMINATIM_USER_AGENT=ShadowBroker/1.0 (https://github.com/BigBodyCobain/Shadowbroker)
# LTA Singapore traffic cameras — leave blank to skip this data source.
# LTA_ACCOUNT_KEY=
# NASA FIRMS country-scoped fire data — enriches global CSV with conflict-zone hotspots.
# Free MAP_KEY from https://firms.modaps.eosdis.nasa.gov/map/#d:24hrs;@0.0,0.0,3.0z
# FIRMS_MAP_KEY=
# Ukraine air raid alerts from alerts.in.ua — free token from https://alerts.in.ua/
# ALERTS_IN_UA_TOKEN=
# Google Earth Engine service account for VIIRS change detection (optional).
# Download JSON key from https://console.cloud.google.com/iam-admin/serviceaccounts
# pip install earthengine-api
# GEE_SERVICE_ACCOUNT_KEY=
# ── Mesh / Reticulum (RNS) ─────────────────────────────────────
# Full-node / participant-node posture for public Infonet sync.
# MESH_NODE_MODE=participant # participant | relay | perimeter
# MESH_BOOTSTRAP_DISABLED=false
# MESH_BOOTSTRAP_MANIFEST_PATH=data/bootstrap_peers.json
# MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY=
# MESH_RELAY_PEERS= # comma-separated operator-trusted sync/push peers
# MESH_PEER_PUSH_SECRET= # shared-secret push auth for trusted testnet peers
# MESH_SYNC_INTERVAL_S=300
# MESH_SYNC_FAILURE_BACKOFF_S=60
#
# Enable Reticulum bridge for Infonet event gossip.
# MESH_RNS_ENABLED=false
# MESH_RNS_APP_NAME=shadowbroker
# MESH_RNS_ASPECT=infonet
# MESH_RNS_IDENTITY_PATH=
# MESH_RNS_PEERS= # comma-separated destination hashes
# MESH_RNS_DANDELION_HOPS=2
# MESH_RNS_DANDELION_DELAY_MS=400
# MESH_RNS_CHURN_INTERVAL_S=300
# MESH_RNS_MAX_PEERS=32
# MESH_RNS_MAX_PAYLOAD=8192
# MESH_RNS_PEER_BUCKET_PREFIX=4
# MESH_RNS_MAX_PEERS_PER_BUCKET=4
# MESH_RNS_PEER_FAIL_THRESHOLD=3
# MESH_RNS_PEER_COOLDOWN_S=300
# MESH_RNS_SHARD_ENABLED=false
# MESH_RNS_SHARD_DATA_SHARDS=3
# MESH_RNS_SHARD_PARITY_SHARDS=1
# MESH_RNS_SHARD_TTL_S=30
# MESH_RNS_FEC_CODEC=xor
# MESH_RNS_BATCH_MS=200
# MESH_RNS_COVER_INTERVAL_S=0
# MESH_RNS_COVER_SIZE=64
# MESH_RNS_IBF_WINDOW=256
# MESH_RNS_IBF_TABLE_SIZE=64
# MESH_RNS_IBF_MINHASH_SIZE=16
# MESH_RNS_IBF_MINHASH_THRESHOLD=0.25
# MESH_RNS_IBF_WINDOW_JITTER=32
# MESH_RNS_IBF_INTERVAL_S=120
# MESH_RNS_IBF_SYNC_PEERS=3
# MESH_RNS_IBF_QUORUM_TIMEOUT_S=6
# MESH_RNS_IBF_MAX_REQUEST_IDS=64
# MESH_RNS_IBF_MAX_EVENTS=64
# MESH_RNS_SESSION_ROTATE_S=0
# MESH_RNS_IBF_FAIL_THRESHOLD=3
# MESH_RNS_IBF_COOLDOWN_S=120
# MESH_VERIFY_INTERVAL_S=600
# MESH_VERIFY_SIGNATURES=false
# ── Mesh DM Relay ──────────────────────────────────────────────
# MESH_DM_TOKEN_PEPPER=change-me
# ── Self Update ────────────────────────────────────────────────
# MESH_UPDATE_SHA256=
# ── Wormhole (Local Agent) ─────────────────────────────────────
# WORMHOLE_HOST=127.0.0.1
# WORMHOLE_PORT=8787
# WORMHOLE_RELOAD=false
# WORMHOLE_TRANSPORT=direct
# WORMHOLE_SOCKS_PROXY=127.0.0.1:9050
# WORMHOLE_SOCKS_DNS=true
backend/Dockerfile
+21 -15
View File
@@ -1,4 +1,4 @@
FROM python:3.10-slim-bookworm
FROM python:3.11-slim-bookworm
WORKDIR /app
@@ -9,32 +9,38 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/*
# Install UV for Python dependency management
# Install UV for fast, reproducible Python dependency management
ADD https://astral.sh/uv/install.sh /uv-installer.sh
RUN sh /uv-installer.sh && rm /uv-installer.sh
ENV PATH="/root/.local/bin/:$PATH"
# Set environment variable for UV to install dependencies in the system Python environment
# By default UV creates a new venv and installs dependencies there
ENV PATH="/root/.local/bin:$PATH"
# Install into system Python (no venv needed inside container)
ENV UV_PROJECT_ENVIRONMENT=/usr/local
# Copy pyproject.toml from root for dependency management
COPY pyproject.toml .
# Copy lock file for reproducible builds
COPY uv.lock .
# Copy source code
# Copy workspace root files for UV resolution (build context is repo root)
COPY pyproject.toml /workspace/pyproject.toml
COPY uv.lock /workspace/uv.lock
COPY backend/pyproject.toml /workspace/backend/pyproject.toml
# Install Python dependencies using the lockfile
RUN cd /workspace/backend && uv sync --frozen --no-dev \
&& playwright install --with-deps chromium
# Copy backend source code
COPY backend/ .
# Install Python dependencies using UV (this will use the lock file for reproducibility)
RUN uv sync --frozen
RUN uv run playwright install --with-deps chromium
# Install Node.js dependencies (ws module for AIS WebSocket proxy)
# Copy manifests first so this layer is cached unless deps change
COPY backend/package*.json ./
RUN npm ci --omit=dev
# Clean up workspace scaffold
RUN rm -rf /workspace
# Create a non-root user for security
# Grant write access to /app so the auto-updater can extract files
# Pre-create /app/data so mounted volumes inherit correct ownership
RUN adduser --system --uid 1001 backenduser \
&& mkdir -p /app/data \
&& chown -R backenduser /app \
&& chmod -R u+w /app
@@ -45,4 +51,4 @@ USER backenduser
EXPOSE 8000
# Start FastAPI server
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--timeout-keep-alive", "120"]
backend/config/news_feeds.json
+34 -29
View File
@@ -1,5 +1,15 @@
{
"feeds": [
{
"name": "Reuters",
"url": "https://www.reutersagency.com/feed/?best-topics=world",
"weight": 5
},
{
"name": "AP News",
"url": "https://rsshub.app/apnews/topics/world-news",
"weight": 5
},
{
"name": "NPR",
"url": "https://feeds.npr.org/1004/rss.xml",
@@ -26,13 +36,33 @@
"weight": 5
},
{
"name": "NHK",
"url": "https://www3.nhk.or.jp/nhkworld/rss/world.xml",
"name": "The War Zone",
"url": "https://www.twz.com/feed",
"weight": 4
},
{
"name": "Bellingcat",
"url": "https://www.bellingcat.com/feed/",
"weight": 4
},
{
"name": "Guardian",
"url": "https://www.theguardian.com/world/rss",
"weight": 3
},
{
"name": "TASS",
"url": "https://tass.com/rss/v2.xml",
"weight": 2
},
{
"name": "Xinhua",
"url": "http://www.news.cn/english/rss/worldrss.xml",
"weight": 2
},
{
"name": "CNA",
"url": "https://www.channelnewsasia.com/rssfeed/8395986",
"url": "https://www.channelnewsasia.com/api/v1/rss-outbound-feed?_format=xml",
"weight": 3
},
{
@@ -40,16 +70,6 @@
"url": "https://en.mercopress.com/rss/",
"weight": 3
},
{
"name": "FocusTaiwan",
"url": "https://focustaiwan.tw/rss",
"weight": 5
},
{
"name": "Kyodo",
"url": "https://english.kyodonews.net/rss/news.xml",
"weight": 4
},
{
"name": "SCMP",
"url": "https://www.scmp.com/rss/91/feed",
@@ -60,26 +80,11 @@
"url": "https://thediplomat.com/feed/",
"weight": 4
},
{
"name": "Stars and Stripes",
"url": "https://www.stripes.com/feeds/pacific.rss",
"weight": 4
},
{
"name": "Yonhap",
"url": "https://en.yna.co.kr/RSS/news.xml",
"weight": 4
},
{
"name": "Nikkei Asia",
"url": "https://asia.nikkei.com/rss",
"weight": 3
},
{
"name": "Taipei Times",
"url": "https://www.taipeitimes.com/xml/pda.rss",
"weight": 4
},
{
"name": "Asia Times",
"url": "https://asiatimes.com/feed/",
@@ -96,4 +101,4 @@
"weight": 3
}
]
}
}
backend/data/military_bases.json
+4334 -27
View File
File diff suppressed because it is too large Load Diff
backend/data/sat_gp_cache.json
+1 -1
View File
File diff suppressed because one or more lines are too long
backend/data/tracked_names.json
+675 -128
View File
File diff suppressed because it is too large Load Diff
backend/main.py
+7722 -101
View File
File diff suppressed because it is too large Load Diff
backend/out.json
-1
View File
@@ -1 +0,0 @@
{"callsign": "JWZ7", "country": "N625GN", "lng": -111.914754, "lat": 33.620235, "alt": 0, "heading": 0, "type": "tracked_flight", "origin_loc": null, "dest_loc": null, "origin_name": "UNKNOWN", "dest_name": "UNKNOWN", "registration": "N625GN", "model": "GLF5", "icao24": "a82973", "speed_knots": 6.8, "squawk": "1200", "airline_code": "", "aircraft_category": "plane", "alert_operator": "Tilman Fertitta", "alert_category": "People", "alert_color": "pink", "trail": [[33.62024, -111.91475, 0, 1772302052]]}
backend/pytest.ini
+1
View File
@@ -2,3 +2,4 @@
testpaths = tests
python_files = test_*.py
python_functions = test_*
asyncio_default_fixture_loop_scope = function
backend/scripts/bootstrap_manifest_helper.py
+115
View File
@@ -0,0 +1,115 @@
import argparse
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
BACKEND_DIR = ROOT / "backend"
if str(BACKEND_DIR) not in sys.path:
sys.path.insert(0, str(BACKEND_DIR))
from services.mesh.mesh_bootstrap_manifest import ( # noqa: E402
bootstrap_signer_public_key_b64,
generate_bootstrap_signer,
write_signed_bootstrap_manifest,
)
def _load_peers(args: argparse.Namespace) -> list[dict]:
peers: list[dict] = []
if args.peers_file:
raw = json.loads(Path(args.peers_file).read_text(encoding="utf-8"))
if not isinstance(raw, list):
raise ValueError("peers file must be a JSON array")
for entry in raw:
if not isinstance(entry, dict):
raise ValueError("peers file entries must be objects")
peers.append(dict(entry))
for peer_arg in args.peer or []:
parts = [part.strip() for part in str(peer_arg).split(",", 3)]
if len(parts) < 3:
raise ValueError("peer entries must look like url,transport,role[,label]")
peer_url, transport, role = parts[:3]
label = parts[3] if len(parts) > 3 else ""
peers.append(
{
"peer_url": peer_url,
"transport": transport,
"role": role,
"label": label,
}
)
if not peers:
raise ValueError("at least one peer is required")
return peers
def cmd_generate_keypair(_args: argparse.Namespace) -> int:
signer = generate_bootstrap_signer()
print(json.dumps(signer, indent=2))
return 0
def cmd_sign(args: argparse.Namespace) -> int:
peers = _load_peers(args)
manifest = write_signed_bootstrap_manifest(
args.output,
signer_id=args.signer_id,
signer_private_key_b64=args.private_key_b64,
peers=peers,
valid_for_hours=int(args.valid_hours),
)
print(f"Wrote signed bootstrap manifest to {Path(args.output).resolve()}")
print(f"signer_id={manifest.signer_id}")
print(f"valid_until={manifest.valid_until}")
print(f"peer_count={len(manifest.peers)}")
print(f"MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY={bootstrap_signer_public_key_b64(args.private_key_b64)}")
return 0
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Generate and sign Infonet bootstrap manifests for participant nodes."
)
subparsers = parser.add_subparsers(dest="command", required=True)
keygen = subparsers.add_parser("generate-keypair", help="Generate an Ed25519 bootstrap signer keypair")
keygen.set_defaults(func=cmd_generate_keypair)
sign = subparsers.add_parser("sign", help="Sign a bootstrap manifest from peer entries")
sign.add_argument("--output", required=True, help="Output path for bootstrap_peers.json")
sign.add_argument("--signer-id", required=True, help="Manifest signer identifier")
sign.add_argument(
"--private-key-b64",
required=True,
help="Raw Ed25519 private key in base64 returned by generate-keypair",
)
sign.add_argument(
"--peers-file",
help="JSON file containing an array of peer objects with peer_url, transport, role, and optional label",
)
sign.add_argument(
"--peer",
action="append",
help="Inline peer in the form url,transport,role[,label]. May be repeated.",
)
sign.add_argument(
"--valid-hours",
type=int,
default=168,
help="Manifest validity window in hours (default: 168)",
)
sign.set_defaults(func=cmd_sign)
return parser
def main() -> int:
parser = build_parser()
args = parser.parse_args()
return args.func(args)
if __name__ == "__main__":
raise SystemExit(main())
backend/scripts/check-env.ps1
+5
View File
@@ -0,0 +1,5 @@
param(
[string]$Python = "python"
)
& $Python -c "from services.env_check import validate_env; validate_env(strict=False)"
backend/scripts/check-env.sh
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
PYTHON="${PYTHON:-python3}"
"$PYTHON" -c "from services.env_check import validate_env; validate_env(strict=False)"
backend/scripts/diagnostics.py
+45
View File
@@ -0,0 +1,45 @@
from datetime import datetime
from services.data_fetcher import get_latest_data
from services.fetchers._store import source_timestamps, active_layers, source_freshness
from services.fetch_health import get_health_snapshot
def _fmt_ts(ts: str | None) -> str:
if not ts:
return "-"
try:
return datetime.fromisoformat(ts).strftime("%Y-%m-%d %H:%M:%S")
except Exception:
return ts
def main():
data = get_latest_data()
print("=== Diagnostics ===")
print(f"Last updated: {_fmt_ts(data.get('last_updated'))}")
print(
f"Active layers: {sum(1 for v in active_layers.values() if v)} enabled / {len(active_layers)} total"
)
print("\n--- Source Timestamps ---")
for k, v in sorted(source_timestamps.items()):
print(f"{k:20} {_fmt_ts(v)}")
print("\n--- Source Freshness ---")
for k, v in sorted(source_freshness.items()):
last_ok = _fmt_ts(v.get("last_ok"))
last_err = _fmt_ts(v.get("last_error"))
print(f"{k:20} ok={last_ok} err={last_err}")
print("\n--- Fetch Health ---")
health = get_health_snapshot()
for k, v in sorted(health.items()):
print(
f"{k:20} ok={v.get('ok_count', 0)} err={v.get('error_count', 0)} "
f"last_ok={_fmt_ts(v.get('last_ok'))} last_err={_fmt_ts(v.get('last_error'))} "
f"avg_ms={v.get('avg_duration_ms')}"
)
if __name__ == "__main__":
main()
backend/scripts/release_helper.py
+138
View File
@@ -0,0 +1,138 @@
import argparse
import hashlib
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
PACKAGE_JSON = ROOT / "frontend" / "package.json"
def _normalize_version(raw: str) -> str:
version = str(raw or "").strip()
if version.startswith("v"):
version = version[1:]
parts = version.split(".")
if len(parts) != 3 or not all(part.isdigit() for part in parts):
raise ValueError("Version must look like X.Y.Z")
return version
def _read_package_json() -> dict:
return json.loads(PACKAGE_JSON.read_text(encoding="utf-8"))
def _write_package_json(data: dict) -> None:
PACKAGE_JSON.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8")
def current_version() -> str:
return str(_read_package_json().get("version") or "").strip()
def set_version(version: str) -> str:
normalized = _normalize_version(version)
data = _read_package_json()
data["version"] = normalized
_write_package_json(data)
return normalized
def expected_tag(version: str) -> str:
return f"v{_normalize_version(version)}"
def expected_asset(version: str) -> str:
normalized = _normalize_version(version)
return f"ShadowBroker_v{normalized}.zip"
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 128), b""):
digest.update(chunk)
return digest.hexdigest().lower()
def cmd_show(_args: argparse.Namespace) -> int:
version = current_version()
if not version:
print("package.json has no version", file=sys.stderr)
return 1
print(f"package.json version : {version}")
print(f"expected git tag : {expected_tag(version)}")
print(f"expected zip asset : {expected_asset(version)}")
return 0
def cmd_set_version(args: argparse.Namespace) -> int:
version = set_version(args.version)
print(f"Set frontend/package.json version to {version}")
print(f"Next release tag : {expected_tag(version)}")
print(f"Next zip asset : {expected_asset(version)}")
return 0
def cmd_hash(args: argparse.Namespace) -> int:
version = _normalize_version(args.version) if args.version else current_version()
if not version:
print("No version available; pass --version or set frontend/package.json", file=sys.stderr)
return 1
zip_path = Path(args.zip_path).resolve()
if not zip_path.is_file():
print(f"ZIP not found: {zip_path}", file=sys.stderr)
return 1
digest = sha256_file(zip_path)
expected_name = expected_asset(version)
asset_matches = zip_path.name == expected_name
print(f"release version : {version}")
print(f"expected git tag : {expected_tag(version)}")
print(f"zip path : {zip_path}")
print(f"zip name matches : {'yes' if asset_matches else 'no'}")
print(f"expected zip asset : {expected_name}")
print(f"SHA-256 : {digest}")
print("")
print("Updater pin:")
print(f"MESH_UPDATE_SHA256={digest}")
return 0 if asset_matches else 2
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Helper for ShadowBroker release version/tag/asset consistency."
)
subparsers = parser.add_subparsers(dest="command", required=True)
show_parser = subparsers.add_parser("show", help="Show current version, expected tag, and asset")
show_parser.set_defaults(func=cmd_show)
set_version_parser = subparsers.add_parser("set-version", help="Update frontend/package.json version")
set_version_parser.add_argument("version", help="Version like 0.9.6")
set_version_parser.set_defaults(func=cmd_set_version)
hash_parser = subparsers.add_parser(
"hash", help="Compute SHA-256 for a release ZIP and print the updater pin"
)
hash_parser.add_argument("zip_path", help="Path to the release ZIP")
hash_parser.add_argument(
"--version",
help="Release version like 0.9.6. Defaults to frontend/package.json version.",
)
hash_parser.set_defaults(func=cmd_hash)
return parser
def main() -> int:
parser = build_parser()
args = parser.parse_args()
return args.func(args)
if __name__ == "__main__":
raise SystemExit(main())
backend/scripts/repair_wormhole_secure_storage.py
+48
View File
@@ -0,0 +1,48 @@
from __future__ import annotations
import json
from pathlib import Path
from services.mesh import mesh_secure_storage
from services.mesh.mesh_wormhole_contacts import CONTACTS_FILE
from services.mesh.mesh_wormhole_identity import IDENTITY_FILE, _default_identity
from services.mesh.mesh_wormhole_persona import PERSONA_FILE, _default_state as _default_persona_state
from services.mesh.mesh_wormhole_ratchet import STATE_FILE as RATCHET_FILE
def _load_payloads() -> dict[Path, object]:
return {
IDENTITY_FILE: mesh_secure_storage.read_secure_json(IDENTITY_FILE, _default_identity),
PERSONA_FILE: mesh_secure_storage.read_secure_json(PERSONA_FILE, _default_persona_state),
RATCHET_FILE: mesh_secure_storage.read_secure_json(RATCHET_FILE, lambda: {}),
CONTACTS_FILE: mesh_secure_storage.read_secure_json(CONTACTS_FILE, lambda: {}),
}
def main() -> None:
payloads = _load_payloads()
master_key_file = mesh_secure_storage.MASTER_KEY_FILE
backup_key_file = master_key_file.with_suffix(master_key_file.suffix + ".bak")
if master_key_file.exists():
if backup_key_file.exists():
backup_key_file.unlink()
master_key_file.replace(backup_key_file)
for path, payload in payloads.items():
mesh_secure_storage.write_secure_json(path, payload)
print(
json.dumps(
{
"ok": True,
"rewrapped": [str(path.name) for path in payloads.keys()],
"master_key": str(master_key_file),
"backup_master_key": str(backup_key_file) if backup_key_file.exists() else "",
}
)
)
if __name__ == "__main__":
main()
backend/scripts/scan-secrets.sh
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# scan-secrets.sh — Catch keys, secrets, and credentials before they hit git.
#
# Usage:
# ./backend/scripts/scan-secrets.sh # Scan staged files (pre-commit)
# ./backend/scripts/scan-secrets.sh --all # Scan entire working tree
# ./backend/scripts/scan-secrets.sh --staged # Scan staged files only (default)
#
# Exit code: 0 = clean, 1 = secrets found
set -euo pipefail
RED='\033[0;31m'
YELLOW='\033[1;33m'
GREEN='\033[0;32m'
NC='\033[0m'
MODE="${1:---staged}"
FOUND=0
# ── Get file list based on mode ─────────────────────────────────────────
if [[ "$MODE" == "--all" ]]; then
FILELIST=$(mktemp)
{ git ls-files 2>/dev/null; git ls-files --others --exclude-standard 2>/dev/null; } > "$FILELIST"
echo -e "${YELLOW}Scanning entire working tree...${NC}"
else
FILELIST=$(mktemp)
git diff --cached --name-only --diff-filter=ACMR 2>/dev/null > "$FILELIST" || true
if [[ ! -s "$FILELIST" ]]; then
echo -e "${GREEN}No staged files to scan.${NC}"
rm -f "$FILELIST"
exit 0
fi
echo -e "${YELLOW}Scanning $(wc -l < "$FILELIST" | tr -d ' ') staged files...${NC}"
fi
# ── Check 1: Dangerous file extensions ──────────────────────────────────
KEY_EXT='\.key$|\.pem$|\.p12$|\.pfx$|\.jks$|\.keystore$|\.p8$|\.der$'
SECRET_EXT='\.secret$|\.secrets$|\.credential$|\.credentials$'
HITS=$(grep -iE "$KEY_EXT|$SECRET_EXT" "$FILELIST" 2>/dev/null || true)
if [[ -n "$HITS" ]]; then
echo -e "\n${RED}BLOCKED: Key/secret files detected:${NC}"
echo "$HITS" | while read -r f; do echo -e " ${RED}$f${NC}"; done
FOUND=1
fi
# ── Check 2: Dangerous filenames ────────────────────────────────────────
RISKY='id_rsa|id_ed25519|id_ecdsa|private_key|private\.key|secret_key|master\.key'
RISKY+='|serviceaccount|gcloud.*\.json|firebase.*\.json|\.htpasswd'
HITS=$(grep -iE "$RISKY" "$FILELIST" 2>/dev/null || true)
if [[ -n "$HITS" ]]; then
echo -e "\n${RED}BLOCKED: Risky filenames detected:${NC}"
echo "$HITS" | while read -r f; do echo -e " ${RED}$f${NC}"; done
FOUND=1
fi
# ── Check 3: .env files (not .env.example) ──────────────────────────────
HITS=$(grep -E '(^|/)\.env(\.[^e].*)?$' "$FILELIST" 2>/dev/null | grep -v '\.example' || true)
if [[ -n "$HITS" ]]; then
echo -e "\n${RED}BLOCKED: Environment files detected:${NC}"
echo "$HITS" | while read -r f; do echo -e " ${RED}$f${NC}"; done
FOUND=1
fi
# ── Check 4: _domain_keys directory (project-specific) ──────────────────
HITS=$(grep '_domain_keys/' "$FILELIST" 2>/dev/null || true)
if [[ -n "$HITS" ]]; then
echo -e "\n${RED}BLOCKED: Domain keys directory detected:${NC}"
echo "$HITS" | while read -r f; do echo -e " ${RED}$f${NC}"; done
FOUND=1
fi
# ── Check 5: Content scan for embedded secrets (single grep pass) ───────
# Build one mega-pattern and run grep once across all files (fast!)
SECRET_REGEX='PRIVATE KEY-----|'
SECRET_REGEX+='ssh-rsa AAAA[0-9A-Za-z+/]|'
SECRET_REGEX+='ssh-ed25519 AAAA[0-9A-Za-z+/]|'
SECRET_REGEX+='ghp_[0-9a-zA-Z]{36}|' # GitHub PAT
SECRET_REGEX+='github_pat_[0-9a-zA-Z]{22}_[0-9a-zA-Z]{59}|' # GitHub fine-grained
SECRET_REGEX+='gho_[0-9a-zA-Z]{36}|' # GitHub OAuth
SECRET_REGEX+='sk-[0-9a-zA-Z]{48}|' # OpenAI key
SECRET_REGEX+='sk-ant-[0-9a-zA-Z-]{90,}|' # Anthropic key
SECRET_REGEX+='AKIA[0-9A-Z]{16}|' # AWS access key
SECRET_REGEX+='AIzaSy[0-9A-Za-z_-]{33}|' # Google API key
SECRET_REGEX+='xox[bpoas]-[0-9a-zA-Z-]+|' # Slack token
SECRET_REGEX+='npm_[0-9a-zA-Z]{36}|' # npm token
SECRET_REGEX+='pypi-[0-9a-zA-Z-]{50,}' # PyPI token
# Filter to text-like files only (skip binaries by extension + skip this script)
TEXT_FILES=$(grep -ivE '\.(png|jpg|jpeg|gif|ico|svg|woff2?|ttf|eot|pbf|zip|tar|gz|db|sqlite|xlsx|pdf|mp[34]|wav|ogg|webm|webp|avif)$' "$FILELIST" | grep -v 'scan-secrets\.sh$' || true)
if [[ -n "$TEXT_FILES" ]]; then
# Use grep with file list, skip missing/binary, limit output
CONTENT_HITS=$(echo "$TEXT_FILES" | xargs grep -lE "$SECRET_REGEX" 2>/dev/null || true)
if [[ -n "$CONTENT_HITS" ]]; then
echo -e "\n${RED}BLOCKED: Embedded secrets/tokens found in:${NC}"
echo "$CONTENT_HITS" | while read -r f; do
echo -e " ${RED}$f${NC}"
# Show first matching line for context
grep -nE "$SECRET_REGEX" "$f" 2>/dev/null | head -2 | while read -r line; do
echo -e " ${YELLOW}$line${NC}"
done
done
FOUND=1
fi
fi
rm -f "$FILELIST"
# ── Result ──────────────────────────────────────────────────────────────
echo ""
if [[ $FOUND -eq 1 ]]; then
echo -e "${RED}Secret scan FAILED. Add these to .gitignore or remove them before committing.${NC}"
echo -e "${YELLOW}If intentional (e.g. test fixtures): git commit --no-verify${NC}"
exit 1
else
echo -e "${GREEN}Secret scan passed. No keys or secrets detected.${NC}"
exit 0
fi
backend/scripts/setup-venv.ps1
+10
View File
@@ -0,0 +1,10 @@
param(
[string]$Python = "python"
)
$repoRoot = Resolve-Path (Join-Path $PSScriptRoot "..")
$venvPath = Join-Path $repoRoot "venv"
& $Python -m venv $venvPath
$pip = Join-Path $venvPath "Scripts\pip.exe"
& $pip install -r (Join-Path $repoRoot "requirements-dev.txt")
backend/scripts/setup-venv.sh
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -euo pipefail
PYTHON="${PYTHON:-python3}"
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
VENV_DIR="$REPO_ROOT/venv"
"$PYTHON" -m venv "$VENV_DIR"
"$VENV_DIR/bin/pip" install -r "$REPO_ROOT/requirements-dev.txt"
backend/seattle_sample.json
-8
View File
@@ -1,8 +0,0 @@
{
"code" : "dataset.missing",
"error" : true,
"message" : "Not found",
"data" : {
"id" : "xqwu-hwdm"
}
}
backend/services/ais_stream.py
+383 -137
View File
@@ -16,18 +16,19 @@ logger = logging.getLogger(__name__)
AIS_WS_URL = "wss://stream.aisstream.io/v0/stream"
API_KEY = os.environ.get("AIS_API_KEY", "")
# AIS vessel type code classification
# See: https://coast.noaa.gov/data/marinecadastre/ais/VesselTypeCodes2018.pdf
def classify_vessel(ais_type: int, mmsi: int) -> str:
"""Classify a vessel by its AIS type code into a rendering category."""
if 80 <= ais_type <= 89:
return "tanker" # Oil/Chemical/Gas tankers → RED
return "tanker" # Oil/Chemical/Gas tankers → RED
if 70 <= ais_type <= 79:
return "cargo" # Cargo ships, container vessels → RED
return "cargo" # Cargo ships, container vessels → RED
if 60 <= ais_type <= 69:
return "passenger" # Cruise ships, ferries → GRAY
return "passenger" # Cruise ships, ferries → GRAY
if ais_type in (36, 37):
return "yacht" # Sailing/Pleasure craft → DARK BLUE
return "yacht" # Sailing/Pleasure craft → DARK BLUE
if ais_type == 35:
return "military_vessel" # Military → YELLOW
# MMSI-based military detection: military MMSIs often start with certain prefixes
@@ -35,87 +36,286 @@ def classify_vessel(ais_type: int, mmsi: int) -> str:
if mmsi_str.startswith("3380") or mmsi_str.startswith("3381"):
return "military_vessel" # US Navy
if ais_type in (30, 31, 32, 33, 34):
return "other" # Fishing, towing, dredging, diving, etc.
return "other" # Fishing, towing, dredging, diving, etc.
if ais_type in (50, 51, 52, 53, 54, 55, 56, 57, 58, 59):
return "other" # Pilot, SAR, tug, port tender, etc.
return "unknown" # Not yet classified — will update when ShipStaticData arrives
return "other" # Pilot, SAR, tug, port tender, etc.
return "unknown" # Not yet classified — will update when ShipStaticData arrives
# MMSI Maritime Identification Digit (MID) → Country mapping
# First 3 digits of MMSI (for 9-digit MMSIs) encode the flag state
MID_COUNTRY = {
201: "Albania", 202: "Andorra", 203: "Austria", 204: "Portugal", 205: "Belgium",
206: "Belarus", 207: "Bulgaria", 208: "Vatican", 209: "Cyprus", 210: "Cyprus",
211: "Germany", 212: "Cyprus", 213: "Georgia", 214: "Moldova", 215: "Malta",
216: "Armenia", 218: "Germany", 219: "Denmark", 220: "Denmark", 224: "Spain",
225: "Spain", 226: "France", 227: "France", 228: "France", 229: "Malta",
230: "Finland", 231: "Faroe Islands", 232: "United Kingdom", 233: "United Kingdom",
234: "United Kingdom", 235: "United Kingdom", 236: "Gibraltar", 237: "Greece",
238: "Croatia", 239: "Greece", 240: "Greece", 241: "Greece", 242: "Morocco",
243: "Hungary", 244: "Netherlands", 245: "Netherlands", 246: "Netherlands",
247: "Italy", 248: "Malta", 249: "Malta", 250: "Ireland", 251: "Iceland",
252: "Liechtenstein", 253: "Luxembourg", 254: "Monaco", 255: "Portugal",
256: "Malta", 257: "Norway", 258: "Norway", 259: "Norway", 261: "Poland",
263: "Portugal", 264: "Romania", 265: "Sweden", 266: "Sweden", 267: "Slovakia",
268: "San Marino", 269: "Switzerland", 270: "Czech Republic", 271: "Turkey",
272: "Ukraine", 273: "Russia", 274: "North Macedonia", 275: "Latvia",
276: "Estonia", 277: "Lithuania", 278: "Slovenia",
301: "Anguilla", 303: "Alaska", 304: "Antigua", 305: "Antigua",
306: "Netherlands Antilles", 307: "Aruba", 308: "Bahamas", 309: "Bahamas",
310: "Bermuda", 311: "Bahamas", 312: "Belize", 314: "Barbados", 316: "Canada",
319: "Cayman Islands", 321: "Costa Rica", 323: "Cuba", 325: "Dominica",
327: "Dominican Republic", 329: "Guadeloupe", 330: "Grenada", 331: "Greenland",
332: "Guatemala", 334: "Honduras", 336: "Haiti", 338: "United States",
339: "Jamaica", 341: "Saint Kitts", 343: "Saint Lucia", 345: "Mexico",
347: "Martinique", 348: "Montserrat", 350: "Nicaragua", 351: "Panama",
352: "Panama", 353: "Panama", 354: "Panama", 355: "Panama",
356: "Panama", 357: "Panama", 358: "Puerto Rico", 359: "El Salvador",
361: "Saint Pierre", 362: "Trinidad", 364: "Turks and Caicos",
366: "United States", 367: "United States", 368: "United States", 369: "United States",
370: "Panama", 371: "Panama", 372: "Panama", 373: "Panama",
374: "Panama", 375: "Saint Vincent", 376: "Saint Vincent", 377: "Saint Vincent",
378: "British Virgin Islands", 379: "US Virgin Islands",
401: "Afghanistan", 403: "Saudi Arabia", 405: "Bangladesh", 408: "Bahrain",
410: "Bhutan", 412: "China", 413: "China", 414: "China",
416: "Taiwan", 417: "Sri Lanka", 419: "India", 422: "Iran",
423: "Azerbaijan", 425: "Iraq", 428: "Israel", 431: "Japan",
432: "Japan", 434: "Turkmenistan", 436: "Kazakhstan", 437: "Uzbekistan",
438: "Jordan", 440: "South Korea", 441: "South Korea", 443: "Palestine",
445: "North Korea", 447: "Kuwait", 450: "Lebanon", 451: "Kyrgyzstan",
453: "Macao", 455: "Maldives", 457: "Mongolia", 459: "Nepal",
461: "Oman", 463: "Pakistan", 466: "Qatar", 468: "Syria",
470: "UAE", 472: "Tajikistan", 473: "Yemen", 475: "Tonga",
477: "Hong Kong", 478: "Bosnia",
501: "Antarctica", 503: "Australia", 506: "Myanmar",
508: "Brunei", 510: "Micronesia", 511: "Palau", 512: "New Zealand",
514: "Cambodia", 515: "Cambodia", 516: "Christmas Island",
518: "Cook Islands", 520: "Fiji", 523: "Cocos Islands",
525: "Indonesia", 529: "Kiribati", 531: "Laos", 533: "Malaysia",
536: "Northern Mariana Islands", 538: "Marshall Islands",
540: "New Caledonia", 542: "Niue", 544: "Nauru", 546: "French Polynesia",
548: "Philippines", 553: "Papua New Guinea", 555: "Pitcairn",
557: "Solomon Islands", 559: "American Samoa", 561: "Samoa",
563: "Singapore", 564: "Singapore", 565: "Singapore", 566: "Singapore",
567: "Thailand", 570: "Tonga", 572: "Tuvalu", 574: "Vietnam",
576: "Vanuatu", 577: "Vanuatu", 578: "Wallis and Futuna",
601: "South Africa", 603: "Angola", 605: "Algeria", 607: "Benin",
609: "Botswana", 610: "Burundi", 611: "Cameroon", 612: "Cape Verde",
613: "Central African Republic", 615: "Congo", 616: "Comoros",
617: "DR Congo", 618: "Ivory Coast", 619: "Djibouti",
620: "Egypt", 621: "Equatorial Guinea", 622: "Ethiopia",
624: "Eritrea", 625: "Gabon", 626: "Gambia", 627: "Ghana",
629: "Guinea", 630: "Guinea-Bissau", 631: "Kenya", 632: "Lesotho",
633: "Liberia", 634: "Liberia", 635: "Liberia", 636: "Liberia",
637: "Libya", 642: "Madagascar", 644: "Malawi", 645: "Mali",
647: "Mauritania", 649: "Mauritius", 650: "Mozambique",
654: "Namibia", 655: "Niger", 656: "Nigeria", 657: "Guinea",
659: "Rwanda", 660: "Senegal", 661: "Sierra Leone",
662: "Somalia", 663: "South Africa", 664: "Sudan",
667: "Tanzania", 668: "Togo", 669: "Tunisia", 670: "Uganda",
671: "Egypt", 672: "Tanzania", 674: "Zambia", 675: "Zimbabwe",
676: "Comoros", 677: "Tanzania",
201: "Albania",
202: "Andorra",
203: "Austria",
204: "Portugal",
205: "Belgium",
206: "Belarus",
207: "Bulgaria",
208: "Vatican",
209: "Cyprus",
210: "Cyprus",
211: "Germany",
212: "Cyprus",
213: "Georgia",
214: "Moldova",
215: "Malta",
216: "Armenia",
218: "Germany",
219: "Denmark",
220: "Denmark",
224: "Spain",
225: "Spain",
226: "France",
227: "France",
228: "France",
229: "Malta",
230: "Finland",
231: "Faroe Islands",
232: "United Kingdom",
233: "United Kingdom",
234: "United Kingdom",
235: "United Kingdom",
236: "Gibraltar",
237: "Greece",
238: "Croatia",
239: "Greece",
240: "Greece",
241: "Greece",
242: "Morocco",
243: "Hungary",
244: "Netherlands",
245: "Netherlands",
246: "Netherlands",
247: "Italy",
248: "Malta",
249: "Malta",
250: "Ireland",
251: "Iceland",
252: "Liechtenstein",
253: "Luxembourg",
254: "Monaco",
255: "Portugal",
256: "Malta",
257: "Norway",
258: "Norway",
259: "Norway",
261: "Poland",
263: "Portugal",
264: "Romania",
265: "Sweden",
266: "Sweden",
267: "Slovakia",
268: "San Marino",
269: "Switzerland",
270: "Czech Republic",
271: "Turkey",
272: "Ukraine",
273: "Russia",
274: "North Macedonia",
275: "Latvia",
276: "Estonia",
277: "Lithuania",
278: "Slovenia",
301: "Anguilla",
303: "Alaska",
304: "Antigua",
305: "Antigua",
306: "Netherlands Antilles",
307: "Aruba",
308: "Bahamas",
309: "Bahamas",
310: "Bermuda",
311: "Bahamas",
312: "Belize",
314: "Barbados",
316: "Canada",
319: "Cayman Islands",
321: "Costa Rica",
323: "Cuba",
325: "Dominica",
327: "Dominican Republic",
329: "Guadeloupe",
330: "Grenada",
331: "Greenland",
332: "Guatemala",
334: "Honduras",
336: "Haiti",
338: "United States",
339: "Jamaica",
341: "Saint Kitts",
343: "Saint Lucia",
345: "Mexico",
347: "Martinique",
348: "Montserrat",
350: "Nicaragua",
351: "Panama",
352: "Panama",
353: "Panama",
354: "Panama",
355: "Panama",
356: "Panama",
357: "Panama",
358: "Puerto Rico",
359: "El Salvador",
361: "Saint Pierre",
362: "Trinidad",
364: "Turks and Caicos",
366: "United States",
367: "United States",
368: "United States",
369: "United States",
370: "Panama",
371: "Panama",
372: "Panama",
373: "Panama",
374: "Panama",
375: "Saint Vincent",
376: "Saint Vincent",
377: "Saint Vincent",
378: "British Virgin Islands",
379: "US Virgin Islands",
401: "Afghanistan",
403: "Saudi Arabia",
405: "Bangladesh",
408: "Bahrain",
410: "Bhutan",
412: "China",
413: "China",
414: "China",
416: "Taiwan",
417: "Sri Lanka",
419: "India",
422: "Iran",
423: "Azerbaijan",
425: "Iraq",
428: "Israel",
431: "Japan",
432: "Japan",
434: "Turkmenistan",
436: "Kazakhstan",
437: "Uzbekistan",
438: "Jordan",
440: "South Korea",
441: "South Korea",
443: "Palestine",
445: "North Korea",
447: "Kuwait",
450: "Lebanon",
451: "Kyrgyzstan",
453: "Macao",
455: "Maldives",
457: "Mongolia",
459: "Nepal",
461: "Oman",
463: "Pakistan",
466: "Qatar",
468: "Syria",
470: "UAE",
472: "Tajikistan",
473: "Yemen",
475: "Tonga",
477: "Hong Kong",
478: "Bosnia",
501: "Antarctica",
503: "Australia",
506: "Myanmar",
508: "Brunei",
510: "Micronesia",
511: "Palau",
512: "New Zealand",
514: "Cambodia",
515: "Cambodia",
516: "Christmas Island",
518: "Cook Islands",
520: "Fiji",
523: "Cocos Islands",
525: "Indonesia",
529: "Kiribati",
531: "Laos",
533: "Malaysia",
536: "Northern Mariana Islands",
538: "Marshall Islands",
540: "New Caledonia",
542: "Niue",
544: "Nauru",
546: "French Polynesia",
548: "Philippines",
553: "Papua New Guinea",
555: "Pitcairn",
557: "Solomon Islands",
559: "American Samoa",
561: "Samoa",
563: "Singapore",
564: "Singapore",
565: "Singapore",
566: "Singapore",
567: "Thailand",
570: "Tonga",
572: "Tuvalu",
574: "Vietnam",
576: "Vanuatu",
577: "Vanuatu",
578: "Wallis and Futuna",
601: "South Africa",
603: "Angola",
605: "Algeria",
607: "Benin",
609: "Botswana",
610: "Burundi",
611: "Cameroon",
612: "Cape Verde",
613: "Central African Republic",
615: "Congo",
616: "Comoros",
617: "DR Congo",
618: "Ivory Coast",
619: "Djibouti",
620: "Egypt",
621: "Equatorial Guinea",
622: "Ethiopia",
624: "Eritrea",
625: "Gabon",
626: "Gambia",
627: "Ghana",
629: "Guinea",
630: "Guinea-Bissau",
631: "Kenya",
632: "Lesotho",
633: "Liberia",
634: "Liberia",
635: "Liberia",
636: "Liberia",
637: "Libya",
642: "Madagascar",
644: "Malawi",
645: "Mali",
647: "Mauritania",
649: "Mauritius",
650: "Mozambique",
654: "Namibia",
655: "Niger",
656: "Nigeria",
657: "Guinea",
659: "Rwanda",
660: "Senegal",
661: "Sierra Leone",
662: "Somalia",
663: "South Africa",
664: "Sudan",
667: "Tanzania",
668: "Togo",
669: "Tunisia",
670: "Uganda",
671: "Egypt",
672: "Tanzania",
674: "Zambia",
675: "Zimbabwe",
676: "Comoros",
677: "Tanzania",
}
def get_country_from_mmsi(mmsi: int) -> str:
"""Look up flag state from MMSI Maritime Identification Digit."""
mmsi_str = str(mmsi)
@@ -130,8 +330,10 @@ _vessels: dict[int, dict] = {}
_vessels_lock = threading.Lock()
_ws_thread: threading.Thread | None = None
_ws_running = False
_proxy_process = None
import os
CACHE_FILE = os.path.join(os.path.dirname(__file__), "ais_cache.json")
@@ -141,7 +343,7 @@ def _save_cache():
with _vessels_lock:
# Convert int keys to strings for JSON
data = {str(k): v for k, v in _vessels.items()}
with open(CACHE_FILE, 'w') as f:
with open(CACHE_FILE, "w") as f:
json.dump(data, f)
logger.info(f"AIS cache saved: {len(data)} vessels")
except (IOError, OSError) as e:
@@ -154,7 +356,7 @@ def _load_cache():
if not os.path.exists(CACHE_FILE):
return
try:
with open(CACHE_FILE, 'r') as f:
with open(CACHE_FILE, "r") as f:
data = json.load(f)
now = time.time()
stale_cutoff = now - 3600 # Accept vessels up to 1 hour old on restart
@@ -169,41 +371,51 @@ def _load_cache():
logger.error(f"Failed to load AIS cache: {e}")
def get_ais_vessels() -> list[dict]:
"""Return a snapshot of tracked AIS vessels, excluding 'other' type, pruning stale."""
def prune_stale_vessels():
"""Remove vessels not updated in the last 15 minutes. Safe to call from a scheduler."""
now = time.time()
stale_cutoff = now - 900 # 15 minutes
stale_cutoff = now - 900
with _vessels_lock:
# Prune stale vessels
stale_keys = [k for k, v in _vessels.items() if v.get("_updated", 0) < stale_cutoff]
for k in stale_keys:
del _vessels[k]
if stale_keys:
logger.info(f"AIS pruned {len(stale_keys)} stale vessels")
def get_ais_vessels() -> list[dict]:
"""Return a snapshot of tracked AIS vessels, pruning stale."""
prune_stale_vessels()
with _vessels_lock:
result = []
for mmsi, v in _vessels.items():
v_type = v.get("type", "unknown")
# Skip 'other' vessels (fishing, tug, pilot, etc.) to reduce load
if v_type == "other":
continue
# Skip vessels without valid position
if not v.get("lat") or not v.get("lng"):
continue
result.append({
"mmsi": mmsi,
"name": v.get("name", "UNKNOWN"),
"type": v_type,
"lat": round(v.get("lat", 0), 5),
"lng": round(v.get("lng", 0), 5),
"heading": v.get("heading", 0),
"sog": round(v.get("sog", 0), 1),
"cog": round(v.get("cog", 0), 1),
"callsign": v.get("callsign", ""),
"destination": v.get("destination", "") or "UNKNOWN",
"imo": v.get("imo", 0),
"country": get_country_from_mmsi(mmsi),
})
# Sanitize speed: AIS 102.3 kn = "speed not available"
sog = v.get("sog", 0)
if sog >= 102.2:
sog = 0
result.append(
{
"mmsi": mmsi,
"name": v.get("name", "UNKNOWN"),
"type": v_type,
"lat": round(v.get("lat", 0), 5),
"lng": round(v.get("lng", 0), 5),
"heading": v.get("heading", 0),
"sog": round(sog, 1),
"cog": round(v.get("cog", 0), 1),
"callsign": v.get("callsign", ""),
"destination": v.get("destination", "") or "UNKNOWN",
"imo": v.get("imo", 0),
"country": get_country_from_mmsi(mmsi),
}
)
return result
@@ -228,7 +440,9 @@ def ingest_ais_catcher(msgs: list[dict]) -> int:
if lat is not None and lon is not None and lat != 91.0 and lon != 181.0:
vessel["lat"] = lat
vessel["lng"] = lon
vessel["sog"] = msg.get("speed", 0)
# AIS raw value 1023 (102.3 kn) = "speed not available"
raw_speed = msg.get("speed", 0)
vessel["sog"] = 0 if raw_speed >= 102.2 else raw_speed
vessel["cog"] = msg.get("course", 0)
heading = msg.get("heading", 511)
vessel["heading"] = heading if heading != 511 else vessel.get("cog", 0)
@@ -276,31 +490,37 @@ def _ais_stream_loop():
while _ws_running:
try:
logger.info("Starting Node.js AIS Stream Proxy...")
proxy_env = os.environ.copy()
proxy_env["AIS_API_KEY"] = API_KEY
process = subprocess.Popen(
['node', proxy_script, API_KEY],
["node", proxy_script],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
bufsize=1
bufsize=1,
env=proxy_env,
)
_proxy_process = process
with _vessels_lock:
_proxy_process = process
# Drain stderr in a background thread to prevent deadlock
import threading
def _drain_stderr():
for errline in iter(process.stderr.readline, ''):
for errline in iter(process.stderr.readline, ""):
errline = errline.strip()
if errline:
logger.warning(f"AIS proxy stderr: {errline}")
threading.Thread(target=_drain_stderr, daemon=True).start()
logger.info("AIS Stream proxy started — receiving vessel data")
msg_count = 0
ok_streak = 0 # Track consecutive successful messages for backoff reset
last_log_time = time.time()
for raw_msg in iter(process.stdout.readline, ''):
for raw_msg in iter(process.stdout.readline, ""):
if not _ws_running:
process.terminate()
break
@@ -346,14 +566,18 @@ def _ais_stream_loop():
with _vessels_lock:
vessel["lat"] = lat
vessel["lng"] = lng
vessel["sog"] = report.get("Sog", 0)
# AIS raw value 1023 (102.3 kn) = "speed not available"
raw_sog = report.get("Sog", 0)
vessel["sog"] = 0 if raw_sog >= 102.2 else raw_sog
vessel["cog"] = report.get("Cog", 0)
heading = report.get("TrueHeading", 511)
vessel["heading"] = heading if heading != 511 else report.get("Cog", 0)
vessel["_updated"] = time.time()
# Use metadata name if we don't have one yet
if not vessel.get("name") or vessel["name"] == "UNKNOWN":
vessel["name"] = metadata.get("ShipName", "UNKNOWN").strip() or "UNKNOWN"
vessel["name"] = (
metadata.get("ShipName", "UNKNOWN").strip() or "UNKNOWN"
)
# Update static data from ShipStaticData
elif msg_type == "ShipStaticData":
@@ -361,10 +585,14 @@ def _ais_stream_loop():
ais_type = static.get("Type", 0)
with _vessels_lock:
vessel["name"] = (static.get("Name", "") or metadata.get("ShipName", "UNKNOWN")).strip() or "UNKNOWN"
vessel["name"] = (
static.get("Name", "") or metadata.get("ShipName", "UNKNOWN")
).strip() or "UNKNOWN"
vessel["callsign"] = (static.get("CallSign", "") or "").strip()
vessel["imo"] = static.get("ImoNumber", 0)
vessel["destination"] = (static.get("Destination", "") or "").strip().replace("@", "")
vessel["destination"] = (
(static.get("Destination", "") or "").strip().replace("@", "")
)
vessel["ais_type_code"] = ais_type
vessel["type"] = classify_vessel(ais_type, mmsi)
vessel["_updated"] = time.time()
@@ -382,7 +610,9 @@ def _ais_stream_loop():
if now - last_log_time >= 60:
with _vessels_lock:
count = len(_vessels)
logger.info(f"AIS Stream: processed {msg_count} messages, tracking {count} vessels")
logger.info(
f"AIS Stream: processed {msg_count} messages, tracking {count} vessels"
)
_save_cache()
last_log_time = now
@@ -397,23 +627,34 @@ def _ais_stream_loop():
def _run_ais_loop():
"""Thread target: run the AIS loop."""
global _ws_running, _ws_thread, _proxy_process
try:
_ais_stream_loop()
except Exception as e:
logger.error(f"AIS Stream thread crashed: {e}")
finally:
with _vessels_lock:
_ws_running = False
_ws_thread = None
_proxy_process = None
def start_ais_stream():
"""Start the AIS WebSocket stream in a background thread."""
global _ws_thread, _ws_running
if _ws_thread and _ws_thread.is_alive():
with _vessels_lock:
if _ws_running:
logger.info("AIS Stream already running")
return
_ws_running = True
existing_thread = _ws_thread
if existing_thread and existing_thread.is_alive():
logger.info("AIS Stream already running")
return
# Load cached vessel data from disk
_load_cache()
_ws_running = True
_ws_thread = threading.Thread(target=_run_ais_loop, daemon=True, name="ais-stream")
_ws_thread.start()
logger.info("AIS Stream background thread started")
@@ -421,31 +662,36 @@ def start_ais_stream():
def stop_ais_stream():
"""Stop the AIS WebSocket stream and save cache."""
global _ws_running, _proxy_process
_ws_running = False
if _proxy_process and _proxy_process.stdin:
global _ws_running, _ws_thread, _proxy_process
with _vessels_lock:
_ws_running = False
_ws_thread = None
proc = _proxy_process
_proxy_process = None
if proc and proc.stdin:
try:
_proxy_process.stdin.close()
proc.stdin.close()
except Exception:
pass
_save_cache() # Save on shutdown
logger.info("AIS Stream stopping...")
def update_ais_bbox(south: float, west: float, north: float, east: float):
"""Dynamically update the AIS stream bounding box via proxy stdin."""
global _proxy_process
if not _proxy_process or not _proxy_process.stdin:
with _vessels_lock:
proc = _proxy_process
if not proc or not proc.stdin:
return
try:
cmd = json.dumps({
"type": "update_bbox",
"bboxes": [[[south, west], [north, east]]]
})
_proxy_process.stdin.write(cmd + "\n")
_proxy_process.stdin.flush()
logger.info(f"Updated AIS bounding box to: S:{south:.2f} W:{west:.2f} N:{north:.2f} E:{east:.2f}")
cmd = json.dumps({"type": "update_bbox", "bboxes": [[[south, west], [north, east]]]})
proc.stdin.write(cmd + "\n")
proc.stdin.flush()
logger.info(
f"Updated AIS bounding box to: S:{south:.2f} W:{west:.2f} N:{north:.2f} E:{east:.2f}"
)
except Exception as e:
logger.error(f"Failed to update AIS bbox: {e}")
backend/services/api_settings.py
+20 -1
View File
@@ -2,6 +2,7 @@
API Settings management — serves the API key registry and allows updates.
Keys are stored in the backend .env file and loaded via python-dotenv.
"""
import os
import re
from pathlib import Path
@@ -121,6 +122,24 @@ API_REGISTRY = [
"url": "https://openmhz.com/",
"required": False,
},
{
"id": "shodan_api_key",
"env_key": "SHODAN_API_KEY",
"name": "Shodan — Operator API Key",
"description": "Paid Shodan API key for local operator-driven searches and temporary map overlays. Results are attributed to Shodan and are not merged into ShadowBroker core feeds.",
"category": "Reconnaissance",
"url": "https://account.shodan.io/billing",
"required": False,
},
{
"id": "finnhub_api_key",
"env_key": "FINNHUB_API_KEY",
"name": "Finnhub — API Key",
"description": "Free market data API. Defense stock quotes, congressional trading disclosures, and insider transactions. 60 calls/min free tier.",
"category": "Financial",
"url": "https://finnhub.io/register",
"required": False,
},
]
@@ -160,7 +179,7 @@ def update_api_key(env_key: str, new_value: str) -> bool:
valid_keys = {api["env_key"] for api in API_REGISTRY if api.get("env_key")}
if env_key not in valid_keys:
return False
if not isinstance(new_value, str):
return False
if "\n" in new_value or "\r" in new_value:
backend/services/carrier_tracker.py
+142 -75
View File
@@ -15,6 +15,7 @@ import json
import time
import logging
import threading
import random
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Optional
@@ -34,108 +35,127 @@ CARRIER_REGISTRY: Dict[str, dict] = {
"name": "USS Nimitz (CVN-68)",
"wiki": "https://en.wikipedia.org/wiki/USS_Nimitz",
"homeport": "Bremerton, WA",
"homeport_lat": 47.5535, "homeport_lng": -122.6400,
"fallback_lat": 47.5535, "fallback_lng": -122.6400,
"homeport_lat": 47.5535,
"homeport_lng": -122.6400,
"fallback_lat": 47.5535,
"fallback_lng": -122.6400,
"fallback_heading": 90,
"fallback_desc": "Bremerton, WA (Maintenance)"
"fallback_desc": "Bremerton, WA (Maintenance)",
},
"CVN-76": {
"name": "USS Ronald Reagan (CVN-76)",
"wiki": "https://en.wikipedia.org/wiki/USS_Ronald_Reagan",
"homeport": "Bremerton, WA",
"homeport_lat": 47.5580, "homeport_lng": -122.6360,
"fallback_lat": 47.5580, "fallback_lng": -122.6360,
"homeport_lat": 47.5580,
"homeport_lng": -122.6360,
"fallback_lat": 47.5580,
"fallback_lng": -122.6360,
"fallback_heading": 90,
"fallback_desc": "Bremerton, WA (Decommissioning)"
"fallback_desc": "Bremerton, WA (Decommissioning)",
},
# --- Norfolk, VA (Naval Station Norfolk) ---
# Piers run N-S along Willoughby Bay; each carrier gets a distinct berth
"CVN-69": {
"name": "USS Dwight D. Eisenhower (CVN-69)",
"wiki": "https://en.wikipedia.org/wiki/USS_Dwight_D._Eisenhower",
"homeport": "Norfolk, VA",
"homeport_lat": 36.9465, "homeport_lng": -76.3265,
"fallback_lat": 36.9465, "fallback_lng": -76.3265,
"homeport_lat": 36.9465,
"homeport_lng": -76.3265,
"fallback_lat": 36.9465,
"fallback_lng": -76.3265,
"fallback_heading": 0,
"fallback_desc": "Norfolk, VA (Post-deployment maintenance)"
"fallback_desc": "Norfolk, VA (Post-deployment maintenance)",
},
"CVN-78": {
"name": "USS Gerald R. Ford (CVN-78)",
"wiki": "https://en.wikipedia.org/wiki/USS_Gerald_R._Ford",
"homeport": "Norfolk, VA",
"homeport_lat": 36.9505, "homeport_lng": -76.3250,
"fallback_lat": 18.0, "fallback_lng": 39.5,
"homeport_lat": 36.9505,
"homeport_lng": -76.3250,
"fallback_lat": 18.0,
"fallback_lng": 39.5,
"fallback_heading": 0,
"fallback_desc": "Red Sea — Operation Epic Fury (USNI Mar 9)"
"fallback_desc": "Red Sea — Operation Epic Fury (USNI Mar 9)",
},
"CVN-74": {
"name": "USS John C. Stennis (CVN-74)",
"wiki": "https://en.wikipedia.org/wiki/USS_John_C._Stennis",
"homeport": "Norfolk, VA",
"homeport_lat": 36.9540, "homeport_lng": -76.3235,
"fallback_lat": 36.98, "fallback_lng": -76.43,
"homeport_lat": 36.9540,
"homeport_lng": -76.3235,
"fallback_lat": 36.98,
"fallback_lng": -76.43,
"fallback_heading": 0,
"fallback_desc": "Newport News, VA (RCOH refueling overhaul)"
"fallback_desc": "Newport News, VA (RCOH refueling overhaul)",
},
"CVN-75": {
"name": "USS Harry S. Truman (CVN-75)",
"wiki": "https://en.wikipedia.org/wiki/USS_Harry_S._Truman",
"homeport": "Norfolk, VA",
"homeport_lat": 36.9580, "homeport_lng": -76.3220,
"fallback_lat": 36.0, "fallback_lng": 15.0,
"homeport_lat": 36.9580,
"homeport_lng": -76.3220,
"fallback_lat": 36.0,
"fallback_lng": 15.0,
"fallback_heading": 0,
"fallback_desc": "Mediterranean Sea deployment (USNI Mar 9)"
"fallback_desc": "Mediterranean Sea deployment (USNI Mar 9)",
},
"CVN-77": {
"name": "USS George H.W. Bush (CVN-77)",
"wiki": "https://en.wikipedia.org/wiki/USS_George_H.W._Bush",
"homeport": "Norfolk, VA",
"homeport_lat": 36.9620, "homeport_lng": -76.3210,
"fallback_lat": 36.5, "fallback_lng": -74.0,
"homeport_lat": 36.9620,
"homeport_lng": -76.3210,
"fallback_lat": 36.5,
"fallback_lng": -74.0,
"fallback_heading": 0,
"fallback_desc": "Atlantic — Pre-deployment workups (USNI Mar 9)"
"fallback_desc": "Atlantic — Pre-deployment workups (USNI Mar 9)",
},
# --- San Diego, CA (Naval Base San Diego) ---
# Carrier piers along the east shore of San Diego Bay, spread N-S
"CVN-70": {
"name": "USS Carl Vinson (CVN-70)",
"wiki": "https://en.wikipedia.org/wiki/USS_Carl_Vinson",
"homeport": "San Diego, CA",
"homeport_lat": 32.6840, "homeport_lng": -117.1290,
"fallback_lat": 32.6840, "fallback_lng": -117.1290,
"homeport_lat": 32.6840,
"homeport_lng": -117.1290,
"fallback_lat": 32.6840,
"fallback_lng": -117.1290,
"fallback_heading": 180,
"fallback_desc": "San Diego, CA (Homeport)"
"fallback_desc": "San Diego, CA (Homeport)",
},
"CVN-71": {
"name": "USS Theodore Roosevelt (CVN-71)",
"wiki": "https://en.wikipedia.org/wiki/USS_Theodore_Roosevelt_(CVN-71)",
"homeport": "San Diego, CA",
"homeport_lat": 32.6885, "homeport_lng": -117.1280,
"fallback_lat": 32.6885, "fallback_lng": -117.1280,
"homeport_lat": 32.6885,
"homeport_lng": -117.1280,
"fallback_lat": 32.6885,
"fallback_lng": -117.1280,
"fallback_heading": 180,
"fallback_desc": "San Diego, CA (Maintenance)"
"fallback_desc": "San Diego, CA (Maintenance)",
},
"CVN-72": {
"name": "USS Abraham Lincoln (CVN-72)",
"wiki": "https://en.wikipedia.org/wiki/USS_Abraham_Lincoln_(CVN-72)",
"homeport": "San Diego, CA",
"homeport_lat": 32.6925, "homeport_lng": -117.1275,
"fallback_lat": 20.0, "fallback_lng": 64.0,
"homeport_lat": 32.6925,
"homeport_lng": -117.1275,
"fallback_lat": 20.0,
"fallback_lng": 64.0,
"fallback_heading": 0,
"fallback_desc": "Arabian Sea — Operation Epic Fury (USNI Mar 9)"
"fallback_desc": "Arabian Sea — Operation Epic Fury (USNI Mar 9)",
},
# --- Yokosuka, Japan (CFAY) ---
"CVN-73": {
"name": "USS George Washington (CVN-73)",
"wiki": "https://en.wikipedia.org/wiki/USS_George_Washington_(CVN-73)",
"homeport": "Yokosuka, Japan",
"homeport_lat": 35.2830, "homeport_lng": 139.6700,
"fallback_lat": 35.2830, "fallback_lng": 139.6700,
"homeport_lat": 35.2830,
"homeport_lng": 139.6700,
"fallback_lat": 35.2830,
"fallback_lng": 139.6700,
"fallback_heading": 180,
"fallback_desc": "Yokosuka, Japan (Forward deployed)"
"fallback_desc": "Yokosuka, Japan (Forward deployed)",
},
}
@@ -175,7 +195,6 @@ REGION_COORDS: Dict[str, tuple] = {
"coral sea": (-18.0, 155.0),
"gulf of mexico": (25.0, -90.0),
"caribbean": (15.0, -75.0),
# Specific bases / ports
"norfolk": (36.95, -76.33),
"san diego": (32.68, -117.15),
@@ -188,7 +207,6 @@ REGION_COORDS: Dict[str, tuple] = {
"bremerton": (47.56, -122.63),
"puget sound": (47.56, -122.63),
"newport news": (36.98, -76.43),
# Areas of operation
"centcom": (25.0, 55.0),
"indopacom": (20.0, 130.0),
@@ -209,6 +227,11 @@ CACHE_FILE = Path(__file__).parent.parent / "carrier_cache.json"
_carrier_positions: Dict[str, dict] = {}
_positions_lock = threading.Lock()
_last_update: Optional[datetime] = None
_last_gdelt_fetch_at = 0.0
_cached_gdelt_articles: List[dict] = []
_GDELT_FETCH_INTERVAL_SECONDS = 1800
_GDELT_REQUEST_DELAY_SECONDS = 1.25
_GDELT_REQUEST_JITTER_SECONDS = 0.35
def _load_cache() -> Dict[str, dict]:
@@ -260,22 +283,41 @@ def _match_carrier(text: str) -> Optional[str]:
def _fetch_gdelt_carrier_news() -> List[dict]:
"""Search GDELT for recent carrier movement news."""
global _last_gdelt_fetch_at, _cached_gdelt_articles
now = time.time()
if _cached_gdelt_articles and (now - _last_gdelt_fetch_at) < _GDELT_FETCH_INTERVAL_SECONDS:
logger.info("Carrier OSINT: using cached GDELT article set to avoid startup bursts")
return list(_cached_gdelt_articles)
results = []
search_terms = [
"aircraft+carrier+deployed",
"carrier+strike+group+navy",
"USS+Nimitz+carrier", "USS+Ford+carrier", "USS+Eisenhower+carrier",
"USS+Vinson+carrier", "USS+Roosevelt+carrier+navy",
"USS+Lincoln+carrier", "USS+Truman+carrier",
"USS+Reagan+carrier", "USS+Washington+carrier+navy",
"USS+Bush+carrier", "USS+Stennis+carrier",
"USS+Nimitz+carrier",
"USS+Ford+carrier",
"USS+Eisenhower+carrier",
"USS+Vinson+carrier",
"USS+Roosevelt+carrier+navy",
"USS+Lincoln+carrier",
"USS+Truman+carrier",
"USS+Reagan+carrier",
"USS+Washington+carrier+navy",
"USS+Bush+carrier",
"USS+Stennis+carrier",
]
for term in search_terms:
for idx, term in enumerate(search_terms):
try:
url = f"https://api.gdeltproject.org/api/v2/doc/doc?query={term}&mode=artlist&maxrecords=5&format=json&timespan=14d"
raw = fetch_with_curl(url, timeout=8)
if not raw or not hasattr(raw, 'text'):
if getattr(raw, "status_code", 500) == 429:
logger.warning(
"GDELT returned 429 for '%s'; preserving cached carrier OSINT results",
term,
)
continue
if not raw or not hasattr(raw, "text"):
continue
data = raw.json()
articles = data.get("articles", [])
@@ -286,7 +328,14 @@ def _fetch_gdelt_carrier_news() -> List[dict]:
except (ConnectionError, TimeoutError, ValueError, KeyError, OSError) as e:
logger.debug(f"GDELT search failed for '{term}': {e}")
continue
if idx < len(search_terms) - 1:
time.sleep(
_GDELT_REQUEST_DELAY_SECONDS
+ random.uniform(0.0, _GDELT_REQUEST_JITTER_SECONDS)
)
_cached_gdelt_articles = list(results)
_last_gdelt_fetch_at = time.time()
logger.info(f"Carrier OSINT: found {len(results)} GDELT articles")
return results
@@ -316,9 +365,11 @@ def _parse_carrier_positions_from_news(articles: List[dict]) -> Dict[str, dict]:
"desc": title[:100],
"source": "GDELT News API",
"source_url": article.get("url", "https://api.gdeltproject.org"),
"updated": datetime.now(timezone.utc).isoformat()
"updated": datetime.now(timezone.utc).isoformat(),
}
logger.info(f"Carrier update: {CARRIER_REGISTRY[hull]['name']} → {coords} (from: {title[:80]})")
logger.info(
f"Carrier update: {CARRIER_REGISTRY[hull]['name']} → {coords} (from: {title[:80]})"
)
return updates
@@ -336,21 +387,25 @@ def _load_carrier_fallbacks() -> Dict[str, dict]:
"wiki": info["wiki"],
"source": "USNI News Fleet & Marine Tracker",
"source_url": "https://news.usni.org/category/fleet-tracker",
"updated": datetime.now(timezone.utc).isoformat()
"updated": datetime.now(timezone.utc).isoformat(),
}
# Overlay cached positions from previous runs (may have GDELT data)
cached = _load_cache()
for hull, cached_pos in cached.items():
if hull in positions:
if cached_pos.get("source", "").startswith("GDELT") or cached_pos.get("source", "").startswith("News"):
positions[hull].update({
"lat": cached_pos["lat"],
"lng": cached_pos["lng"],
"desc": cached_pos.get("desc", positions[hull]["desc"]),
"source": cached_pos.get("source", "Cached OSINT"),
"updated": cached_pos.get("updated", "")
})
if cached_pos.get("source", "").startswith("GDELT") or cached_pos.get(
"source", ""
).startswith("News"):
positions[hull].update(
{
"lat": cached_pos["lat"],
"lng": cached_pos["lng"],
"desc": cached_pos.get("desc", positions[hull]["desc"]),
"source": cached_pos.get("source", "Cached OSINT"),
"updated": cached_pos.get("updated", ""),
}
)
return positions
@@ -371,7 +426,9 @@ def update_carrier_positions():
if not _carrier_positions:
_carrier_positions.update(positions)
_last_update = datetime.now(timezone.utc)
logger.info(f"Carrier tracker: {len(positions)} carriers loaded from fallback/cache (GDELT enrichment starting...)")
logger.info(
f"Carrier tracker: {len(positions)} carriers loaded from fallback/cache (GDELT enrichment starting...)"
)
# --- Phase 2: slow GDELT enrichment ---
try:
@@ -408,6 +465,7 @@ def _deconflict_positions(result: List[dict]) -> List[dict]:
"""
# Group by rounded lat/lng (within ~0.01° ≈ 1km = same spot)
from collections import defaultdict
groups: dict[str, list[int]] = defaultdict(list)
for i, c in enumerate(result):
key = f"{round(c['lat'], 2)},{round(c['lng'], 2)}"
@@ -454,22 +512,26 @@ def get_carrier_positions() -> List[dict]:
result = []
for hull, pos in _carrier_positions.items():
info = CARRIER_REGISTRY.get(hull, {})
result.append({
"name": pos.get("name", info.get("name", hull)),
"type": "carrier",
"lat": pos["lat"],
"lng": pos["lng"],
"heading": None, # Heading unknown for carriers — OSINT cannot determine true heading
"sog": 0,
"cog": 0,
"country": "United States",
"desc": pos.get("desc", ""),
"wiki": pos.get("wiki", info.get("wiki", "")),
"estimated": True,
"source": pos.get("source", "OSINT estimated position"),
"source_url": pos.get("source_url", "https://news.usni.org/category/fleet-tracker"),
"last_osint_update": pos.get("updated", "")
})
result.append(
{
"name": pos.get("name", info.get("name", hull)),
"type": "carrier",
"lat": pos["lat"],
"lng": pos["lng"],
"heading": None, # Heading unknown for carriers — OSINT cannot determine true heading
"sog": 0,
"cog": 0,
"country": "United States",
"desc": pos.get("desc", ""),
"wiki": pos.get("wiki", info.get("wiki", "")),
"estimated": True,
"source": pos.get("source", "OSINT estimated position"),
"source_url": pos.get(
"source_url", "https://news.usni.org/category/fleet-tracker"
),
"last_osint_update": pos.get("updated", ""),
}
)
return _deconflict_positions(result)
@@ -500,10 +562,13 @@ def _scheduler_loop():
next_run = now.replace(hour=next_hour % 24, minute=0, second=0, microsecond=0)
if next_hour == 24:
from datetime import timedelta
next_run = (now + timedelta(days=1)).replace(hour=0, minute=0, second=0, microsecond=0)
wait_seconds = (next_run - now).total_seconds()
logger.info(f"Carrier tracker: next update at {next_run.isoformat()} ({wait_seconds/3600:.1f}h)")
logger.info(
f"Carrier tracker: next update at {next_run.isoformat()} ({wait_seconds/3600:.1f}h)"
)
# Wait until next scheduled time, or until stop event
if _scheduler_stop.wait(timeout=wait_seconds):
@@ -521,7 +586,9 @@ def start_carrier_tracker():
if _scheduler_thread and _scheduler_thread.is_alive():
return
_scheduler_stop.clear()
_scheduler_thread = threading.Thread(target=_scheduler_loop, daemon=True, name="carrier-tracker")
_scheduler_thread = threading.Thread(
target=_scheduler_loop, daemon=True, name="carrier-tracker"
)
_scheduler_thread.start()
logger.info("Carrier tracker started")
backend/services/cctv_pipeline.py
+773 -371
View File
File diff suppressed because it is too large Load Diff
backend/services/config.py
+122
View File
@@ -0,0 +1,122 @@
"""Typed configuration via pydantic-settings."""
from functools import lru_cache
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
# Admin/security
ADMIN_KEY: str = ""
ALLOW_INSECURE_ADMIN: bool = False
PUBLIC_API_KEY: str = ""
# Data sources
AIS_API_KEY: str = ""
OPENSKY_CLIENT_ID: str = ""
OPENSKY_CLIENT_SECRET: str = ""
LTA_ACCOUNT_KEY: str = ""
# Runtime
CORS_ORIGINS: str = ""
FETCH_SLOW_THRESHOLD_S: float = 5.0
MESH_STRICT_SIGNATURES: bool = True
MESH_DEBUG_MODE: bool = False
MESH_MQTT_EXTRA_ROOTS: str = ""
MESH_MQTT_EXTRA_TOPICS: str = ""
MESH_MQTT_INCLUDE_DEFAULT_ROOTS: bool = True
MESH_RNS_ENABLED: bool = False
MESH_ARTI_ENABLED: bool = False
MESH_ARTI_SOCKS_PORT: int = 9050
MESH_RELAY_PEERS: str = ""
MESH_BOOTSTRAP_DISABLED: bool = False
MESH_BOOTSTRAP_MANIFEST_PATH: str = "data/bootstrap_peers.json"
MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY: str = ""
MESH_NODE_MODE: str = "participant"
MESH_SYNC_INTERVAL_S: int = 300
MESH_SYNC_FAILURE_BACKOFF_S: int = 60
MESH_RELAY_PUSH_TIMEOUT_S: int = 10
MESH_RELAY_MAX_FAILURES: int = 3
MESH_RELAY_FAILURE_COOLDOWN_S: int = 120
MESH_PEER_PUSH_SECRET: str = ""
MESH_RNS_APP_NAME: str = "shadowbroker"
MESH_RNS_ASPECT: str = "infonet"
MESH_RNS_IDENTITY_PATH: str = ""
MESH_RNS_PEERS: str = ""
MESH_RNS_DANDELION_HOPS: int = 2
MESH_RNS_DANDELION_DELAY_MS: int = 400
MESH_RNS_CHURN_INTERVAL_S: int = 300
MESH_RNS_MAX_PEERS: int = 32
MESH_RNS_MAX_PAYLOAD: int = 8192
MESH_RNS_PEER_BUCKET_PREFIX: int = 4
MESH_RNS_MAX_PEERS_PER_BUCKET: int = 4
MESH_RNS_PEER_FAIL_THRESHOLD: int = 3
MESH_RNS_PEER_COOLDOWN_S: int = 300
MESH_RNS_SHARD_ENABLED: bool = False
MESH_RNS_SHARD_DATA_SHARDS: int = 3
MESH_RNS_SHARD_PARITY_SHARDS: int = 1
MESH_RNS_SHARD_TTL_S: int = 30
MESH_RNS_FEC_CODEC: str = "xor" # xor | rs
MESH_RNS_BATCH_MS: int = 200
# Keep a low background cadence on private RNS links so quiet nodes are less
# trivially fingerprintable by silence alone. Set to 0 to disable explicitly.
MESH_RNS_COVER_INTERVAL_S: int = 30
MESH_RNS_COVER_SIZE: int = 64
MESH_RNS_IBF_WINDOW: int = 256
MESH_RNS_IBF_TABLE_SIZE: int = 64
MESH_RNS_IBF_MINHASH_SIZE: int = 16
MESH_RNS_IBF_MINHASH_THRESHOLD: float = 0.25
MESH_RNS_IBF_WINDOW_JITTER: int = 32
MESH_RNS_IBF_INTERVAL_S: int = 120
MESH_RNS_IBF_SYNC_PEERS: int = 3
MESH_RNS_IBF_QUORUM_TIMEOUT_S: int = 6
MESH_RNS_IBF_MAX_REQUEST_IDS: int = 64
MESH_RNS_IBF_MAX_EVENTS: int = 64
MESH_RNS_SESSION_ROTATE_S: int = 1800
MESH_RNS_IBF_FAIL_THRESHOLD: int = 3
MESH_RNS_IBF_COOLDOWN_S: int = 120
MESH_VERIFY_INTERVAL_S: int = 600
MESH_VERIFY_SIGNATURES: bool = True
MESH_DM_SECURE_MODE: bool = True
MESH_DM_TOKEN_PEPPER: str = ""
MESH_DM_ALLOW_LEGACY_GET: bool = False
MESH_DM_PERSIST_SPOOL: bool = False
MESH_DM_REQUIRE_SENDER_SEAL_SHARED: bool = True
MESH_DM_NONCE_TTL_S: int = 300
MESH_DM_NONCE_CACHE_MAX: int = 4096
MESH_DM_REQUEST_MAX_AGE_S: int = 300
MESH_DM_REQUEST_MAILBOX_LIMIT: int = 12
MESH_DM_SHARED_MAILBOX_LIMIT: int = 48
MESH_DM_SELF_MAILBOX_LIMIT: int = 12
MESH_DM_MAX_MSG_BYTES: int = 8192
MESH_DM_ALLOW_SENDER_SEAL: bool = False
# TTL for DH key and prekey bundle registrations — stale entries are pruned.
MESH_DM_KEY_TTL_DAYS: int = 30
# TTL for mailbox binding metadata — shorter = smaller metadata footprint on disk.
MESH_DM_BINDING_TTL_DAYS: int = 7
# When False, mailbox bindings are memory-only (agents re-register on restart).
MESH_DM_METADATA_PERSIST: bool = True
MESH_SCOPED_TOKENS: str = ""
MESH_GATE_SESSION_ROTATE_MSGS: int = 50
MESH_GATE_SESSION_ROTATE_S: int = 3600
# Add a randomized grace window before anonymous gate-session auto-rotation
# so threshold-triggered identity swaps are less trivially correlated.
MESH_GATE_SESSION_ROTATE_JITTER_S: int = 180
# Private gate APIs expose a backward-jittered timestamp view so observers
# cannot trivially align exact send times from response metadata alone.
MESH_GATE_TIMESTAMP_JITTER_S: int = 60
MESH_ALLOW_RAW_SECURE_STORAGE_FALLBACK: bool = False
MESH_PRIVATE_LOG_TTL_S: int = 900
# Clearnet fallback policy for private-tier messages.
# "block" (default) = refuse to send private messages over clearnet.
# "allow" = fall back to clearnet when Tor/RNS is unavailable (weaker privacy).
MESH_PRIVATE_CLEARNET_FALLBACK: str = "block"
# Meshtastic MQTT broker credentials (defaults match public firmware).
MESH_MQTT_USER: str = "meshdev"
MESH_MQTT_PASS: str = "large4cats"
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
@lru_cache
def get_settings() -> Settings:
return Settings()
backend/services/constants.py
+19 -18
View File
@@ -2,32 +2,33 @@
# Centralized magic numbers. Import from here instead of hardcoding.
# ─── Flight Trails ──────────────────────────────────────────────────────────
FLIGHT_TRAIL_MAX_TRACKED = 2000 # Max concurrent tracked trails before LRU eviction
FLIGHT_TRAIL_MAX_TRACKED = 2000 # Max concurrent tracked trails before LRU eviction
FLIGHT_TRAIL_POINTS_PER_FLIGHT = 200 # Max trail points kept per aircraft
TRACKED_TRAIL_TTL_S = 1800 # 30 min - trail TTL for tracked flights
DEFAULT_TRAIL_TTL_S = 300 # 5 min - trail TTL for non-tracked flights
TRACKED_TRAIL_TTL_S = 1800 # 30 min - trail TTL for tracked flights
DEFAULT_TRAIL_TTL_S = 300 # 5 min - trail TTL for non-tracked flights
# ─── Detection Thresholds ──────────────────────────────────────────────────
HOLD_PATTERN_DEGREES = 300 # Total heading change to flag holding pattern
GPS_JAMMING_NACP_THRESHOLD = 8 # NACp below this = degraded GPS signal
GPS_JAMMING_GRID_SIZE = 1.0 # 1 degree grid for aggregation
GPS_JAMMING_MIN_RATIO = 0.25 # 25% degraded aircraft to flag zone
HOLD_PATTERN_DEGREES = 300 # Total heading change to flag holding pattern
GPS_JAMMING_NACP_THRESHOLD = 8 # NACp below this = degraded GPS signal
GPS_JAMMING_GRID_SIZE = 1.0 # 1 degree grid for aggregation
GPS_JAMMING_MIN_RATIO = 0.30 # 30% degraded aircraft to flag zone
GPS_JAMMING_MIN_AIRCRAFT = 5 # Min aircraft in grid cell for statistical significance
# ─── Network & Circuit Breaker ──────────────────────────────────────────────
CIRCUIT_BREAKER_TTL_S = 120 # Skip domain for 2 min after total failure
DOMAIN_FAIL_TTL_S = 300 # Skip requests.get for 5 min, go straight to curl
CONNECT_TIMEOUT_S = 3 # Short connect timeout for fast firewall-block detection
CIRCUIT_BREAKER_TTL_S = 120 # Skip domain for 2 min after total failure
DOMAIN_FAIL_TTL_S = 300 # Skip requests.get for 5 min, go straight to curl
CONNECT_TIMEOUT_S = 3 # Short connect timeout for fast firewall-block detection
# ─── Data Fetcher Intervals ────────────────────────────────────────────────
FAST_FETCH_INTERVAL_S = 60 # Flights, ships, satellites, military
SLOW_FETCH_INTERVAL_MIN = 30 # News, markets, space weather
CCTV_FETCH_INTERVAL_MIN = 1 # CCTV camera pipeline
LIVEUAMAP_FETCH_INTERVAL_HR = 12 # LiveUAMap scraper
FAST_FETCH_INTERVAL_S = 60 # Flights, ships, satellites, military
SLOW_FETCH_INTERVAL_MIN = 30 # News, markets, space weather
CCTV_FETCH_INTERVAL_MIN = 1 # CCTV camera pipeline
LIVEUAMAP_FETCH_INTERVAL_HR = 12 # LiveUAMap scraper
# ─── External API ──────────────────────────────────────────────────────────
OPENSKY_RATE_LIMIT_S = 300 # Only re-fetch OpenSky every 5 minutes
OPENSKY_REQUEST_TIMEOUT_S = 15 # Timeout for OpenSky API calls
ROUTE_FETCH_TIMEOUT_S = 15 # Timeout for adsb.lol route lookups
OPENSKY_RATE_LIMIT_S = 300 # Only re-fetch OpenSky every 5 minutes
OPENSKY_REQUEST_TIMEOUT_S = 15 # Timeout for OpenSky API calls
ROUTE_FETCH_TIMEOUT_S = 15 # Timeout for adsb.lol route lookups
# ─── Internet Outage Detection ─────────────────────────────────────────────
INTERNET_OUTAGE_MIN_SEVERITY = 0.10 # 10% drop minimum to show
INTERNET_OUTAGE_MIN_SEVERITY = 0.10 # 10% drop minimum to show
backend/services/correlation_engine.py
+342
View File
@@ -0,0 +1,342 @@
"""
Emergent Intelligence — Cross-layer correlation engine.
Scans co-located events across multiple data layers and emits composite
alerts that no single source could generate alone.
Correlation types:
- RF Anomaly: GPS jamming + internet outage (both required)
- Military Buildup: Military flights + naval vessels + GDELT conflict events
- Infrastructure Cascade: Internet outage + KiwiSDR offline in same zone
"""
import logging
from collections import defaultdict
logger = logging.getLogger(__name__)
# Grid cell size in degrees — 1° ≈ 111 km at equator.
# Tighter than the previous 2° to reduce false co-locations.
_CELL_SIZE = 1
# Quality gates for RF anomaly correlation — only high-confidence inputs.
# GPS jamming + internet outage overlap in a 111km cell is easily a coincidence
# (IODA returns ~100 regional outages; GPS NACp dips are common in busy airspace).
# Only fire when the evidence is strong enough to indicate deliberate RF interference.
_RF_CORR_MIN_GPS_RATIO = 0.60 # Need strong jamming signal, not marginal NACp dips
_RF_CORR_MIN_OUTAGE_PCT = 40 # Need a serious outage, not routine BGP fluctuation
_RF_CORR_MIN_INDICATORS = 3 # Require 3+ corroborating signals (not just GPS+outage)
def _cell_key(lat: float, lng: float) -> str:
"""Convert lat/lng to a grid cell key."""
clat = int(lat // _CELL_SIZE) * _CELL_SIZE
clng = int(lng // _CELL_SIZE) * _CELL_SIZE
return f"{clat},{clng}"
def _cell_center(key: str) -> tuple[float, float]:
"""Get center lat/lng from a cell key."""
parts = key.split(",")
return float(parts[0]) + _CELL_SIZE / 2, float(parts[1]) + _CELL_SIZE / 2
def _severity(indicator_count: int) -> str:
if indicator_count >= 3:
return "high"
if indicator_count >= 2:
return "medium"
return "low"
def _severity_score(sev: str) -> float:
return {"high": 90, "medium": 60, "low": 30}.get(sev, 0)
def _outage_pct(outage: dict) -> float:
"""Extract outage severity percentage from an outage dict."""
return float(outage.get("severity", 0) or outage.get("severity_pct", 0) or 0)
# ---------------------------------------------------------------------------
# RF Anomaly: GPS jamming + internet outage (both must be present)
# ---------------------------------------------------------------------------
def _detect_rf_anomalies(data: dict) -> list[dict]:
gps_jamming = data.get("gps_jamming") or []
internet_outages = data.get("internet_outages") or []
if not gps_jamming:
return [] # No GPS jamming → no RF anomalies possible
# Build grid of indicators
cells: dict[str, dict] = defaultdict(lambda: {
"gps_jam": False, "gps_ratio": 0.0,
"outage": False, "outage_pct": 0.0,
})
for z in gps_jamming:
lat, lng = z.get("lat"), z.get("lng")
if lat is None or lng is None:
continue
ratio = z.get("ratio", 0)
if ratio < _RF_CORR_MIN_GPS_RATIO:
continue # Skip marginal jamming zones
key = _cell_key(lat, lng)
cells[key]["gps_jam"] = True
cells[key]["gps_ratio"] = max(cells[key]["gps_ratio"], ratio)
for o in internet_outages:
lat = o.get("lat") or o.get("latitude")
lng = o.get("lng") or o.get("lon") or o.get("longitude")
if lat is None or lng is None:
continue
pct = _outage_pct(o)
if pct < _RF_CORR_MIN_OUTAGE_PCT:
continue # Skip minor outages (ISP maintenance noise)
key = _cell_key(float(lat), float(lng))
cells[key]["outage"] = True
cells[key]["outage_pct"] = max(cells[key]["outage_pct"], pct)
# PSK Reporter: presence = healthy RF. Only used as a bonus indicator,
# NOT as a standalone trigger (absence is normal in most cells).
psk_reporter = data.get("psk_reporter") or []
psk_cells: set[str] = set()
for s in psk_reporter:
lat, lng = s.get("lat"), s.get("lon")
if lat is not None and lng is not None:
psk_cells.add(_cell_key(lat, lng))
# When PSK data is unavailable, we can't get a 3rd indicator, so require
# an even higher GPS jamming ratio to compensate (real EW shows 75%+).
psk_available = len(psk_reporter) > 0
alerts: list[dict] = []
for key, c in cells.items():
# GPS jamming is the anchor — required for every RF anomaly alert
if not c["gps_jam"]:
continue
if not c["outage"]:
continue # Both GPS jamming AND outage are always required
indicators = 2 # GPS jamming + outage
drivers: list[str] = [f"GPS jamming {int(c['gps_ratio'] * 100)}%"]
pct = c["outage_pct"]
drivers.append(f"Internet outage{f' {pct:.0f}%' if pct else ''}")
# PSK absence confirms RF environment is disrupted
if psk_available and key not in psk_cells:
indicators += 1
drivers.append("No HF digital activity (PSK Reporter)")
if indicators < _RF_CORR_MIN_INDICATORS:
# Without PSK data, only allow through if GPS ratio is extreme
# (75%+ indicates deliberate, sustained jamming — not noise)
if not psk_available and c["gps_ratio"] >= 0.75 and pct >= 50:
pass # Allow this high-confidence 2-indicator alert through
else:
continue
lat, lng = _cell_center(key)
sev = _severity(indicators)
alerts.append({
"lat": lat,
"lng": lng,
"type": "rf_anomaly",
"severity": sev,
"score": _severity_score(sev),
"drivers": drivers[:3],
"cell_size": _CELL_SIZE,
})
return alerts
# ---------------------------------------------------------------------------
# Military Buildup: flights + ships + GDELT conflict
# ---------------------------------------------------------------------------
def _detect_military_buildups(data: dict) -> list[dict]:
mil_flights = data.get("military_flights") or []
ships = data.get("ships") or []
gdelt = data.get("gdelt") or []
cells: dict[str, dict] = defaultdict(lambda: {
"mil_flights": 0, "mil_ships": 0, "gdelt_events": 0,
})
for f in mil_flights:
lat = f.get("lat") or f.get("latitude")
lng = f.get("lng") or f.get("lon") or f.get("longitude")
if lat is None or lng is None:
continue
try:
key = _cell_key(float(lat), float(lng))
cells[key]["mil_flights"] += 1
except (ValueError, TypeError):
continue
mil_ship_types = {"military_vessel", "military", "warship", "patrol", "destroyer",
"frigate", "corvette", "carrier", "submarine", "cruiser"}
for s in ships:
stype = (s.get("type") or s.get("ship_type") or "").lower()
if not any(mt in stype for mt in mil_ship_types):
continue
lat = s.get("lat") or s.get("latitude")
lng = s.get("lng") or s.get("lon") or s.get("longitude")
if lat is None or lng is None:
continue
try:
key = _cell_key(float(lat), float(lng))
cells[key]["mil_ships"] += 1
except (ValueError, TypeError):
continue
for g in gdelt:
lat = g.get("lat") or g.get("latitude") or g.get("actionGeo_Lat")
lng = g.get("lng") or g.get("lon") or g.get("longitude") or g.get("actionGeo_Long")
if lat is None or lng is None:
continue
try:
key = _cell_key(float(lat), float(lng))
cells[key]["gdelt_events"] += 1
except (ValueError, TypeError):
continue
alerts: list[dict] = []
for key, c in cells.items():
mil_total = c["mil_flights"] + c["mil_ships"]
has_gdelt = c["gdelt_events"] > 0
# Need meaningful military presence AND a conflict indicator
if mil_total < 3 or not has_gdelt:
continue
drivers: list[str] = []
if c["mil_flights"]:
drivers.append(f"{c['mil_flights']} military aircraft")
if c["mil_ships"]:
drivers.append(f"{c['mil_ships']} military vessels")
if c["gdelt_events"]:
drivers.append(f"{c['gdelt_events']} conflict events")
if mil_total >= 11:
sev = "high"
elif mil_total >= 6:
sev = "medium"
else:
sev = "low"
lat, lng = _cell_center(key)
alerts.append({
"lat": lat,
"lng": lng,
"type": "military_buildup",
"severity": sev,
"score": _severity_score(sev),
"drivers": drivers[:3],
"cell_size": _CELL_SIZE,
})
return alerts
# ---------------------------------------------------------------------------
# Infrastructure Cascade: outage + KiwiSDR co-location
#
# Power plants are removed from this detector — with 35K plants globally,
# virtually every 2° cell contains one, making every outage a false hit.
# KiwiSDR receivers (~300 worldwide) are sparse enough to be meaningful:
# an outage in the same cell as a KiwiSDR indicates real infrastructure
# disruption affecting radio monitoring capability.
# ---------------------------------------------------------------------------
def _detect_infra_cascades(data: dict) -> list[dict]:
internet_outages = data.get("internet_outages") or []
kiwisdr = data.get("kiwisdr") or []
if not kiwisdr:
return []
# Build set of cells with KiwiSDR receivers
kiwi_cells: set[str] = set()
for k in kiwisdr:
lat, lng = k.get("lat"), k.get("lon") or k.get("lng")
if lat is not None and lng is not None:
try:
kiwi_cells.add(_cell_key(float(lat), float(lng)))
except (ValueError, TypeError):
pass
if not kiwi_cells:
return []
alerts: list[dict] = []
for o in internet_outages:
lat = o.get("lat") or o.get("latitude")
lng = o.get("lng") or o.get("lon") or o.get("longitude")
if lat is None or lng is None:
continue
try:
key = _cell_key(float(lat), float(lng))
except (ValueError, TypeError):
continue
if key not in kiwi_cells:
continue
pct = _outage_pct(o)
drivers = [f"Internet outage{f' {pct:.0f}%' if pct else ''}",
"KiwiSDR receivers in affected zone"]
lat_c, lng_c = _cell_center(key)
alerts.append({
"lat": lat_c,
"lng": lng_c,
"type": "infra_cascade",
"severity": "medium",
"score": _severity_score("medium"),
"drivers": drivers,
"cell_size": _CELL_SIZE,
})
return alerts
# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------
def compute_correlations(data: dict) -> list[dict]:
"""Run all correlation detectors and return merged alert list."""
alerts: list[dict] = []
try:
alerts.extend(_detect_rf_anomalies(data))
except Exception as e:
logger.error("Correlation engine RF anomaly error: %s", e)
try:
alerts.extend(_detect_military_buildups(data))
except Exception as e:
logger.error("Correlation engine military buildup error: %s", e)
try:
alerts.extend(_detect_infra_cascades(data))
except Exception as e:
logger.error("Correlation engine infra cascade error: %s", e)
rf = sum(1 for a in alerts if a["type"] == "rf_anomaly")
mil = sum(1 for a in alerts if a["type"] == "military_buildup")
infra = sum(1 for a in alerts if a["type"] == "infra_cascade")
if alerts:
logger.info(
"Correlations: %d alerts (%d rf, %d mil, %d infra)",
len(alerts), rf, mil, infra,
)
return alerts
backend/services/data_fetcher.py
+458 -45
View File
@@ -13,10 +13,14 @@ Heavy logic has been extracted into services/fetchers/:
- infrastructure.py — internet outages, data centers, CCTV, KiwiSDR
- geo.py — ships, airports, frontlines, GDELT, LiveUAMap
"""
import logging
import concurrent.futures
from datetime import datetime
import os
import time
from datetime import datetime, timedelta
from dotenv import load_dotenv
load_dotenv()
from apscheduler.schedulers.background import BackgroundScheduler
@@ -25,7 +29,11 @@ from services.cctv_pipeline import init_db
# Shared state — all fetcher modules read/write through this
from services.fetchers._store import (
latest_data, source_timestamps, _mark_fresh, _data_lock, # noqa: F401 — re-exported for main.py
latest_data,
source_timestamps,
_mark_fresh,
_data_lock, # noqa: F401 — re-exported for main.py
get_latest_data_subset,
)
# Domain-specific fetcher modules (already extracted)
@@ -36,24 +44,109 @@ from services.fetchers.satellites import fetch_satellites # noqa: F401
from services.fetchers.news import fetch_news # noqa: F401
# Newly extracted fetcher modules
from services.fetchers.financial import fetch_defense_stocks, fetch_oil_prices # noqa: F401
from services.fetchers.financial import fetch_financial_markets # noqa: F401
from services.fetchers.unusual_whales import fetch_unusual_whales # noqa: F401
from services.fetchers.earth_observation import ( # noqa: F401
fetch_earthquakes, fetch_firms_fires, fetch_space_weather, fetch_weather,
fetch_earthquakes,
fetch_firms_fires,
fetch_firms_country_fires,
fetch_space_weather,
fetch_weather,
fetch_weather_alerts,
fetch_air_quality,
fetch_volcanoes,
fetch_viirs_change_nodes,
)
from services.fetchers.infrastructure import ( # noqa: F401
fetch_internet_outages, fetch_datacenters, fetch_military_bases, fetch_power_plants,
fetch_cctv, fetch_kiwisdr,
fetch_internet_outages,
fetch_ripe_atlas_probes,
fetch_datacenters,
fetch_military_bases,
fetch_power_plants,
fetch_cctv,
fetch_kiwisdr,
fetch_scanners,
fetch_satnogs,
fetch_tinygs,
fetch_psk_reporter,
)
from services.fetchers.geo import ( # noqa: F401
fetch_ships, fetch_airports, find_nearest_airport, cached_airports,
fetch_frontlines, fetch_gdelt, fetch_geopolitics, update_liveuamap,
fetch_ships,
fetch_airports,
find_nearest_airport,
cached_airports,
fetch_frontlines,
fetch_gdelt,
fetch_geopolitics,
update_liveuamap,
fetch_fishing_activity,
)
from services.fetchers.prediction_markets import fetch_prediction_markets # noqa: F401
from services.fetchers.sigint import fetch_sigint # noqa: F401
from services.fetchers.trains import fetch_trains # noqa: F401
from services.fetchers.ukraine_alerts import fetch_ukraine_air_raid_alerts # noqa: F401
from services.fetchers.meshtastic_map import (
fetch_meshtastic_nodes,
load_meshtastic_cache_if_available,
) # noqa: F401
from services.fetchers.fimi import fetch_fimi # noqa: F401
from services.ais_stream import prune_stale_vessels # noqa: F401
logger = logging.getLogger(__name__)
_SLOW_FETCH_S = float(os.environ.get("FETCH_SLOW_THRESHOLD_S", "5"))
# Shared thread pool — reused across all fetch cycles instead of creating/destroying per tick
_SHARED_EXECUTOR = concurrent.futures.ThreadPoolExecutor(
max_workers=20, thread_name_prefix="fetch"
)
# ---------------------------------------------------------------------------
# Scheduler & Orchestration
# ---------------------------------------------------------------------------
def _run_tasks(label: str, funcs: list):
"""Run tasks concurrently and log any exceptions (do not fail silently)."""
if not funcs:
return
futures = {_SHARED_EXECUTOR.submit(func): (func.__name__, time.perf_counter()) for func in funcs}
for future in concurrent.futures.as_completed(futures):
name, start = futures[future]
try:
future.result()
duration = time.perf_counter() - start
from services.fetch_health import record_success
record_success(name, duration_s=duration)
if duration > _SLOW_FETCH_S:
logger.warning(f"{label} task slow: {name} took {duration:.2f}s")
except Exception as e:
duration = time.perf_counter() - start
from services.fetch_health import record_failure
record_failure(name, error=e, duration_s=duration)
logger.exception(f"{label} task failed: {name}")
def _run_task_with_health(func, name: str | None = None):
"""Run a single task with health tracking."""
task_name = name or getattr(func, "__name__", "task")
start = time.perf_counter()
try:
func()
duration = time.perf_counter() - start
from services.fetch_health import record_success
record_success(task_name, duration_s=duration)
if duration > _SLOW_FETCH_S:
logger.warning(f"task slow: {task_name} took {duration:.2f}s")
except Exception as e:
duration = time.perf_counter() - start
from services.fetch_health import record_failure
record_failure(task_name, error=e, duration_s=duration)
logger.exception(f"task failed: {task_name}")
def update_fast_data():
"""Fast-tier: moving entities that need frequent updates (every 60s)."""
logger.info("Fast-tier data update starting...")
@@ -62,50 +155,201 @@ def update_fast_data():
fetch_military_flights,
fetch_ships,
fetch_satellites,
fetch_sigint,
fetch_trains,
fetch_tinygs,
]
with concurrent.futures.ThreadPoolExecutor(max_workers=len(fast_funcs)) as executor:
futures = [executor.submit(func) for func in fast_funcs]
concurrent.futures.wait(futures)
_run_tasks("fast-tier", fast_funcs)
with _data_lock:
latest_data['last_updated'] = datetime.utcnow().isoformat()
latest_data["last_updated"] = datetime.utcnow().isoformat()
from services.fetchers._store import bump_data_version
bump_data_version()
logger.info("Fast-tier update complete.")
def update_slow_data():
"""Slow-tier: contextual + enrichment data that refreshes less often (every 5–10 min)."""
logger.info("Slow-tier data update starting...")
slow_funcs = [
fetch_news,
fetch_prediction_markets,
fetch_earthquakes,
fetch_firms_fires,
fetch_defense_stocks,
fetch_oil_prices,
fetch_firms_country_fires,
fetch_weather,
fetch_space_weather,
fetch_internet_outages,
fetch_ripe_atlas_probes, # runs after IODA to deduplicate
fetch_cctv,
fetch_kiwisdr,
fetch_satnogs,
fetch_frontlines,
fetch_gdelt,
fetch_datacenters,
fetch_military_bases,
fetch_scanners,
fetch_psk_reporter,
fetch_weather_alerts,
fetch_air_quality,
fetch_fishing_activity,
fetch_power_plants,
fetch_ukraine_air_raid_alerts,
]
with concurrent.futures.ThreadPoolExecutor(max_workers=len(slow_funcs)) as executor:
futures = [executor.submit(func) for func in slow_funcs]
concurrent.futures.wait(futures)
_run_tasks("slow-tier", slow_funcs)
# Run correlation engine after all data is fresh
try:
from services.correlation_engine import compute_correlations
with _data_lock:
snapshot = dict(latest_data)
correlations = compute_correlations(snapshot)
with _data_lock:
latest_data["correlations"] = correlations
except Exception as e:
logger.error("Correlation engine failed: %s", e)
from services.fetchers._store import bump_data_version
bump_data_version()
logger.info("Slow-tier update complete.")
def update_all_data():
"""Full refresh — all tiers run IN PARALLEL for fastest startup."""
def update_all_data(*, startup_mode: bool = False):
"""Full refresh.
On startup we prefer cached/DB-backed data first, then let scheduled jobs
perform some heavy top-ups after the app is already responsive.
"""
logger.info("Full data update starting (parallel)...")
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
f0 = pool.submit(fetch_airports)
f1 = pool.submit(update_fast_data)
f2 = pool.submit(update_slow_data)
concurrent.futures.wait([f0, f1, f2])
# Preload Meshtastic map cache immediately (instant, from disk)
load_meshtastic_cache_if_available()
with _data_lock:
meshtastic_seeded = bool(latest_data.get("meshtastic_map_nodes"))
futures = {
_SHARED_EXECUTOR.submit(fetch_airports): ("fetch_airports", time.perf_counter()),
_SHARED_EXECUTOR.submit(update_fast_data): ("update_fast_data", time.perf_counter()),
_SHARED_EXECUTOR.submit(update_slow_data): ("update_slow_data", time.perf_counter()),
_SHARED_EXECUTOR.submit(fetch_volcanoes): ("fetch_volcanoes", time.perf_counter()),
_SHARED_EXECUTOR.submit(fetch_viirs_change_nodes): ("fetch_viirs_change_nodes", time.perf_counter()),
_SHARED_EXECUTOR.submit(fetch_unusual_whales): ("fetch_unusual_whales", time.perf_counter()),
_SHARED_EXECUTOR.submit(fetch_fimi): ("fetch_fimi", time.perf_counter()),
_SHARED_EXECUTOR.submit(fetch_gdelt): ("fetch_gdelt", time.perf_counter()),
_SHARED_EXECUTOR.submit(update_liveuamap): ("update_liveuamap", time.perf_counter()),
}
if not startup_mode or not meshtastic_seeded:
futures[_SHARED_EXECUTOR.submit(fetch_meshtastic_nodes)] = (
"fetch_meshtastic_nodes",
time.perf_counter(),
)
else:
logger.info(
"Startup preload: Meshtastic cache already loaded, deferring remote map refresh to scheduled cadence"
)
for future in concurrent.futures.as_completed(futures):
name, start = futures[future]
try:
future.result()
duration = time.perf_counter() - start
from services.fetch_health import record_success
record_success(name, duration_s=duration)
if duration > _SLOW_FETCH_S:
logger.warning(f"full-refresh task slow: {name} took {duration:.2f}s")
except Exception as e:
duration = time.perf_counter() - start
from services.fetch_health import record_failure
record_failure(name, error=e, duration_s=duration)
logger.exception(f"full-refresh task failed: {name}")
logger.info("Full data update complete.")
_scheduler = None
_STARTUP_CCTV_INGEST_DELAY_S = 30
_FINANCIAL_REFRESH_MINUTES = 30
def _oracle_resolution_sweep():
"""Hourly sweep: check if any markets with active predictions have concluded.
Resolution logic:
- If a market's end_date has passed AND it's no longer in the active API data → resolved
- For binary markets: final probability determines outcome (>50% = yes, <50% = no)
- For multi-outcome: the outcome with highest final probability wins
"""
try:
from services.mesh.mesh_oracle import oracle_ledger
active_titles = oracle_ledger.get_active_markets()
if not active_titles:
return
# Get current market data
with _data_lock:
markets = list(latest_data.get("prediction_markets", []))
# Build lookup of active API markets
api_titles = {m.get("title", "").lower(): m for m in markets}
import time as _time
now = _time.time()
resolved_count = 0
for title in active_titles:
api_market = api_titles.get(title.lower())
# If market still in API and end_date hasn't passed, skip
if api_market:
end_date = api_market.get("end_date")
if end_date:
try:
from datetime import datetime, timezone
dt = datetime.fromisoformat(end_date.replace("Z", "+00:00"))
if dt.timestamp() > now:
continue # Market hasn't ended yet
except Exception:
continue
else:
continue # No end date, can't auto-resolve
# Market has concluded (past end_date or dropped from API)
# Determine outcome from last known data
if api_market:
outcomes = api_market.get("outcomes", [])
if outcomes and len(outcomes) > 2:
# Multi-outcome: highest pct wins
best = max(outcomes, key=lambda o: o.get("pct", 0))
outcome = best.get("name", "")
else:
# Binary: consensus > 50 = yes
pct = api_market.get("consensus_pct") or api_market.get("polymarket_pct") or 50
outcome = "yes" if float(pct) > 50 else "no"
else:
# Market dropped from API entirely — can't determine outcome, skip
logger.warning(
f"Oracle sweep: market '{title}' no longer in API, cannot auto-resolve"
)
continue
if not outcome:
continue
# Resolve both free predictions and market stakes
winners, losers = oracle_ledger.resolve_market(title, outcome)
stake_result = oracle_ledger.resolve_market_stakes(title, outcome)
resolved_count += 1
logger.info(
f"Oracle sweep resolved '{title}' → {outcome}: "
f"{winners}W/{losers}L free, "
f"{stake_result.get('winners', 0)}W/{stake_result.get('losers', 0)}L staked"
)
if resolved_count:
logger.info(f"Oracle sweep complete: {resolved_count} markets resolved")
# Also clean up old data periodically
oracle_ledger.cleanup_old_data()
except Exception as e:
logger.error(f"Oracle resolution sweep error: {e}")
def start_scheduler():
global _scheduler
@@ -113,38 +357,207 @@ def start_scheduler():
_scheduler = BackgroundScheduler(daemon=True)
# Fast tier — every 60 seconds
_scheduler.add_job(update_fast_data, 'interval', seconds=60, id='fast_tier', max_instances=1, misfire_grace_time=30)
_scheduler.add_job(
lambda: _run_task_with_health(update_fast_data, "update_fast_data"),
"interval",
seconds=60,
id="fast_tier",
max_instances=1,
misfire_grace_time=30,
)
# Slow tier — every 5 minutes
_scheduler.add_job(update_slow_data, 'interval', minutes=5, id='slow_tier', max_instances=1, misfire_grace_time=120)
_scheduler.add_job(
lambda: _run_task_with_health(update_slow_data, "update_slow_data"),
"interval",
minutes=5,
id="slow_tier",
max_instances=1,
misfire_grace_time=120,
)
# Very slow — every 15 minutes
_scheduler.add_job(fetch_gdelt, 'interval', minutes=15, id='gdelt', max_instances=1, misfire_grace_time=120)
_scheduler.add_job(update_liveuamap, 'interval', minutes=15, id='liveuamap', max_instances=1, misfire_grace_time=120)
# Weather alerts — every 5 minutes (time-critical, separate from slow tier)
_scheduler.add_job(
lambda: _run_task_with_health(fetch_weather_alerts, "fetch_weather_alerts"),
"interval",
minutes=5,
id="weather_alerts",
max_instances=1,
misfire_grace_time=60,
)
# CCTV pipeline refresh — every 10 minutes
# Instantiate once and reuse — avoids re-creating DB connections on every tick
# Ukraine air raid alerts — every 2 minutes (time-critical)
_scheduler.add_job(
lambda: _run_task_with_health(fetch_ukraine_air_raid_alerts, "fetch_ukraine_air_raid_alerts"),
"interval",
minutes=2,
id="ukraine_alerts",
max_instances=1,
misfire_grace_time=60,
)
# AIS vessel pruning — every 5 minutes (prevents unbounded memory growth)
_scheduler.add_job(
lambda: _run_task_with_health(prune_stale_vessels, "prune_stale_vessels"),
"interval",
minutes=5,
id="ais_prune",
max_instances=1,
misfire_grace_time=60,
)
# GDELT — every 30 minutes (downloads 32 ZIP files per call, avoid rate limits)
_scheduler.add_job(
lambda: _run_task_with_health(fetch_gdelt, "fetch_gdelt"),
"interval",
minutes=30,
id="gdelt",
max_instances=1,
misfire_grace_time=120,
)
_scheduler.add_job(
lambda: _run_task_with_health(update_liveuamap, "update_liveuamap"),
"interval",
minutes=30,
id="liveuamap",
max_instances=1,
misfire_grace_time=120,
)
# CCTV pipeline refresh — runs all ingestors, then refreshes in-memory data.
# Delay the first run slightly so startup serves cached/DB-backed data first.
from services.cctv_pipeline import (
TFLJamCamIngestor, LTASingaporeIngestor,
AustinTXIngestor, NYCDOTIngestor,
TFLJamCamIngestor,
LTASingaporeIngestor,
AustinTXIngestor,
NYCDOTIngestor,
CaltransIngestor,
ColoradoDOTIngestor,
WSDOTIngestor,
GeorgiaDOTIngestor,
IllinoisDOTIngestor,
MichiganDOTIngestor,
WindyWebcamsIngestor,
DGTNationalIngestor,
MadridCityIngestor,
OSMTrafficCameraIngestor,
)
_cctv_ingestors = [
(TFLJamCamIngestor(), "cctv_tfl"),
(LTASingaporeIngestor(), "cctv_lta"),
(AustinTXIngestor(), "cctv_atx"),
(NYCDOTIngestor(), "cctv_nyc"),
(CaltransIngestor(), "cctv_caltrans"),
(ColoradoDOTIngestor(), "cctv_codot"),
(WSDOTIngestor(), "cctv_wsdot"),
(GeorgiaDOTIngestor(), "cctv_gdot"),
(IllinoisDOTIngestor(), "cctv_idot"),
(MichiganDOTIngestor(), "cctv_mdot"),
(WindyWebcamsIngestor(), "cctv_windy"),
(DGTNationalIngestor(), "cctv_dgt"),
(MadridCityIngestor(), "cctv_madrid"),
(OSMTrafficCameraIngestor(), "cctv_osm"),
]
def _run_cctv_ingest_cycle():
from services.fetchers._store import is_any_active
if not is_any_active("cctv"):
return
for ingestor, name in _cctv_ingestors:
_run_task_with_health(ingestor.ingest, name)
# Refresh in-memory CCTV data immediately after ingest
try:
from services.cctv_pipeline import get_all_cameras
from services.fetchers.infrastructure import fetch_cctv
fetch_cctv()
logger.info(f"CCTV ingest cycle complete — {len(get_all_cameras())} cameras in DB")
except Exception as e:
logger.warning(f"CCTV post-ingest refresh failed: {e}")
_scheduler.add_job(
_run_cctv_ingest_cycle,
"interval",
minutes=10,
id="cctv_ingest",
max_instances=1,
misfire_grace_time=120,
next_run_time=datetime.utcnow() + timedelta(seconds=_STARTUP_CCTV_INGEST_DELAY_S),
)
# Financial tickers — every 30 minutes (Yahoo Finance rate-limits aggressively)
def _fetch_financial():
_run_task_with_health(fetch_financial_markets, "fetch_financial_markets")
_scheduler.add_job(
_fetch_financial,
"interval",
minutes=_FINANCIAL_REFRESH_MINUTES,
id="financial_tickers",
max_instances=1,
misfire_grace_time=120,
next_run_time=datetime.utcnow() + timedelta(minutes=_FINANCIAL_REFRESH_MINUTES),
)
# Unusual Whales — every 15 minutes (congress trades, dark pool, flow alerts)
_scheduler.add_job(
lambda: _run_task_with_health(fetch_unusual_whales, "fetch_unusual_whales"),
"interval",
minutes=15,
id="unusual_whales",
max_instances=1,
misfire_grace_time=120,
)
# Meshtastic map API — every 4 hours, fetch global node positions
_scheduler.add_job(
lambda: _run_task_with_health(fetch_meshtastic_nodes, "fetch_meshtastic_nodes"),
"interval",
hours=4,
id="meshtastic_map",
max_instances=1,
misfire_grace_time=600,
)
# Oracle resolution sweep — every hour, check if any markets with predictions have concluded
_scheduler.add_job(
lambda: _run_task_with_health(_oracle_resolution_sweep, "oracle_sweep"),
"interval",
hours=1,
id="oracle_sweep",
max_instances=1,
misfire_grace_time=300,
)
# VIIRS change detection — every 12 hours (monthly composites, no rush)
_scheduler.add_job(
lambda: _run_task_with_health(fetch_viirs_change_nodes, "fetch_viirs_change_nodes"),
"interval",
hours=12,
id="viirs_change",
max_instances=1,
misfire_grace_time=600,
)
# FIMI disinformation index — every 12 hours (weekly editorial feed)
_scheduler.add_job(
lambda: _run_task_with_health(fetch_fimi, "fetch_fimi"),
"interval",
hours=12,
id="fimi",
max_instances=1,
misfire_grace_time=600,
)
_cctv_tfl = TFLJamCamIngestor()
_cctv_lta = LTASingaporeIngestor()
_cctv_atx = AustinTXIngestor()
_cctv_nyc = NYCDOTIngestor()
_now = datetime.now()
_scheduler.add_job(_cctv_tfl.ingest, 'interval', minutes=10, id='cctv_tfl', max_instances=1, misfire_grace_time=120, next_run_time=_now)
_scheduler.add_job(_cctv_lta.ingest, 'interval', minutes=10, id='cctv_lta', max_instances=1, misfire_grace_time=120, next_run_time=_now)
_scheduler.add_job(_cctv_atx.ingest, 'interval', minutes=10, id='cctv_atx', max_instances=1, misfire_grace_time=120, next_run_time=_now)
_scheduler.add_job(_cctv_nyc.ingest, 'interval', minutes=10, id='cctv_nyc', max_instances=1, misfire_grace_time=120, next_run_time=_now)
_scheduler.start()
logger.info("Scheduler started.")
def stop_scheduler():
if _scheduler:
_scheduler.shutdown(wait=False)
def get_latest_data():
with _data_lock:
return dict(latest_data)
return get_latest_data_subset(*latest_data.keys())
backend/services/env_check.py
+225 -11
View File
@@ -2,10 +2,16 @@
Ensures required env vars are present before the scheduler starts.
Logs warnings for optional keys that degrade functionality when missing.
Audits security-critical config for dangerous combinations.
"""
import os
import secrets
import sys
import time
import logging
from pathlib import Path
from services.config import get_settings
logger = logging.getLogger(__name__)
@@ -23,9 +29,200 @@ _OPTIONAL = {
"OPENSKY_CLIENT_ID": "OpenSky OAuth2 — gap-fill flights in Africa/Asia/LatAm",
"OPENSKY_CLIENT_SECRET": "OpenSky OAuth2 — gap-fill flights in Africa/Asia/LatAm",
"LTA_ACCOUNT_KEY": "Singapore LTA traffic cameras (CCTV layer)",
"PUBLIC_API_KEY": "Optional client auth for public endpoints (recommended for exposed deployments)",
}
def _invalid_dm_token_pepper_reason(value: str) -> str:
raw = str(value or "").strip()
lowered = raw.lower()
if not raw:
return "empty"
if lowered in {"change-me", "changeme"}:
return "placeholder"
if len(raw) < 16:
return "too short"
return ""
def _invalid_peer_push_secret_reason(value: str) -> str:
raw = str(value or "").strip()
lowered = raw.lower()
if not raw:
return "empty"
if lowered in {"change-me", "changeme"}:
return "placeholder"
if len(raw) < 16:
return "too short"
return ""
_PEPPER_FILE = Path(__file__).resolve().parents[1] / "data" / "dm_token_pepper.key"
def _ensure_dm_token_pepper(settings) -> str:
token_pepper = str(getattr(settings, "MESH_DM_TOKEN_PEPPER", "") or "").strip()
pepper_reason = _invalid_dm_token_pepper_reason(token_pepper)
if not pepper_reason:
return token_pepper
# Try loading a previously persisted pepper before generating a new one.
try:
from services.mesh.mesh_secure_storage import read_secure_json
stored = read_secure_json(_PEPPER_FILE, lambda: {})
stored_pepper = str(stored.get("pepper", "") or "").strip()
if stored_pepper and not _invalid_dm_token_pepper_reason(stored_pepper):
os.environ["MESH_DM_TOKEN_PEPPER"] = stored_pepper
get_settings.cache_clear()
logger.info("Loaded persisted DM token pepper from %s", _PEPPER_FILE.name)
return stored_pepper
except Exception:
pass
generated = secrets.token_hex(32)
os.environ["MESH_DM_TOKEN_PEPPER"] = generated
get_settings.cache_clear()
log_fn = logger.warning if bool(getattr(settings, "MESH_DEBUG_MODE", False)) else logger.critical
log_fn(
"⚠️ SECURITY: MESH_DM_TOKEN_PEPPER is invalid (%s) — mailbox tokens "
"would be predictably derivable. Auto-generated a random pepper for "
"this session.",
pepper_reason,
)
# Persist so the same pepper survives restarts.
try:
from services.mesh.mesh_secure_storage import write_secure_json
_PEPPER_FILE.parent.mkdir(parents=True, exist_ok=True)
write_secure_json(_PEPPER_FILE, {"pepper": generated, "generated_at": int(time.time())})
logger.info("Persisted auto-generated DM token pepper to %s", _PEPPER_FILE.name)
except Exception:
logger.warning("Could not persist auto-generated DM token pepper to disk — will regenerate on next restart")
return generated
def _peer_push_secret_required(settings) -> bool:
relay_peers = str(getattr(settings, "MESH_RELAY_PEERS", "") or "").strip()
rns_peers = str(getattr(settings, "MESH_RNS_PEERS", "") or "").strip()
return bool(getattr(settings, "MESH_RNS_ENABLED", False) or relay_peers or rns_peers)
def get_security_posture_warnings(settings=None) -> list[str]:
snapshot = settings or get_settings()
warnings: list[str] = []
admin_key = str(getattr(snapshot, "ADMIN_KEY", "") or "").strip()
allow_insecure = bool(getattr(snapshot, "ALLOW_INSECURE_ADMIN", False))
if allow_insecure and not admin_key:
warnings.append(
"ALLOW_INSECURE_ADMIN=true with no ADMIN_KEY leaves admin and Wormhole endpoints unauthenticated."
)
if not bool(getattr(snapshot, "MESH_STRICT_SIGNATURES", True)):
warnings.append(
"MESH_STRICT_SIGNATURES=false is deprecated and ignored; signature enforcement remains mandatory."
)
peer_secret = str(getattr(snapshot, "MESH_PEER_PUSH_SECRET", "") or "").strip()
peer_secret_reason = _invalid_peer_push_secret_reason(peer_secret)
if _peer_push_secret_required(snapshot) and peer_secret_reason:
warnings.append(
"MESH_PEER_PUSH_SECRET is invalid "
f"({peer_secret_reason}) while relay or RNS peers are enabled; private peer authentication, opaque gate forwarding, and voter blinding are not secure-by-default."
)
if os.name != "nt" and bool(getattr(snapshot, "MESH_ALLOW_RAW_SECURE_STORAGE_FALLBACK", False)):
warnings.append(
"MESH_ALLOW_RAW_SECURE_STORAGE_FALLBACK=true stores Wormhole keys in raw local files on this platform."
)
if bool(getattr(snapshot, "MESH_RNS_ENABLED", False)) and int(getattr(snapshot, "MESH_RNS_COVER_INTERVAL_S", 0) or 0) <= 0:
warnings.append(
"MESH_RNS_COVER_INTERVAL_S<=0 disables RNS cover traffic outside high-privacy mode, making quiet-node traffic analysis easier."
)
fallback_policy = str(getattr(snapshot, "MESH_PRIVATE_CLEARNET_FALLBACK", "block") or "block").strip().lower()
if fallback_policy == "allow":
warnings.append(
"MESH_PRIVATE_CLEARNET_FALLBACK=allow — private-tier messages may fall back to clearnet relay when Tor/RNS is unavailable."
)
metadata_persist = bool(getattr(snapshot, "MESH_DM_METADATA_PERSIST", True))
binding_ttl = int(getattr(snapshot, "MESH_DM_BINDING_TTL_DAYS", 7) or 7)
if metadata_persist and binding_ttl > 14:
warnings.append(
f"MESH_DM_BINDING_TTL_DAYS={binding_ttl} with MESH_DM_METADATA_PERSIST=true — long-lived mailbox binding metadata persists communication graph structure on disk."
)
return warnings
def _audit_security_config(settings) -> None:
"""Audit security-critical config combinations and log loud warnings.
This does not block startup (dev ergonomics), but makes dangerous
settings impossible to miss in the logs.
"""
# ── 1. ALLOW_INSECURE_ADMIN without ADMIN_KEY ─────────────────────
admin_key = (getattr(settings, "ADMIN_KEY", "") or "").strip()
allow_insecure = bool(getattr(settings, "ALLOW_INSECURE_ADMIN", False))
if allow_insecure and not admin_key:
logger.critical(
"🚨 SECURITY: ALLOW_INSECURE_ADMIN=true with no ADMIN_KEY — "
"ALL admin/wormhole endpoints are completely unauthenticated. "
"This is acceptable ONLY for local development. "
"Set ADMIN_KEY for any networked or production deployment."
)
# ── 2. Signature enforcement ──────────────────────────────────────
mesh_strict = bool(getattr(settings, "MESH_STRICT_SIGNATURES", True))
if not mesh_strict:
logger.warning(
"⚠️ CONFIG: MESH_STRICT_SIGNATURES=false is deprecated and ignored — "
"runtime signature enforcement remains mandatory."
)
# ── 3. Empty DM token pepper ──────────────────────────────────────
_ensure_dm_token_pepper(settings)
# ── 4. Peer push secret / private-plane integrity ─────────────────
peer_secret = str(getattr(settings, "MESH_PEER_PUSH_SECRET", "") or "").strip()
peer_secret_reason = _invalid_peer_push_secret_reason(peer_secret)
if _peer_push_secret_required(settings) and peer_secret_reason:
log_fn = logger.warning if bool(getattr(settings, "MESH_DEBUG_MODE", False)) else logger.critical
log_fn(
"⚠️ SECURITY: MESH_PEER_PUSH_SECRET is invalid (%s) while relay or RNS peers are enabled — "
"private peer authentication, opaque gate forwarding, and voter blinding are not secure-by-default until it is set to a non-placeholder secret.",
peer_secret_reason,
)
# ── 5. Raw secure-storage fallback on non-Windows ────────────────
if os.name != "nt" and bool(getattr(settings, "MESH_ALLOW_RAW_SECURE_STORAGE_FALLBACK", False)):
log_fn = logger.warning if bool(getattr(settings, "MESH_DEBUG_MODE", False)) else logger.critical
log_fn(
"⚠️ SECURITY: MESH_ALLOW_RAW_SECURE_STORAGE_FALLBACK=true leaves Wormhole keys in raw local files. "
"Use this only for development/CI until a native keyring provider is available."
)
# ── 6. Disabled cover traffic outside forced high-privacy mode ─────────
if bool(getattr(settings, "MESH_RNS_ENABLED", False)) and int(getattr(settings, "MESH_RNS_COVER_INTERVAL_S", 0) or 0) <= 0:
logger.warning(
"⚠️ PRIVACY: MESH_RNS_COVER_INTERVAL_S<=0 disables background RNS cover traffic outside high-privacy mode. "
"Quiet nodes become easier to fingerprint by silence and burst timing."
)
# ── 7. Clearnet fallback policy ──────────────────────────────────
fallback_policy = str(getattr(settings, "MESH_PRIVATE_CLEARNET_FALLBACK", "block") or "block").strip().lower()
if fallback_policy == "allow":
logger.warning(
"⚠️ PRIVACY: MESH_PRIVATE_CLEARNET_FALLBACK=allow — private-tier messages will fall "
"back to clearnet relay when Tor/RNS is unavailable. Set to 'block' for safer defaults."
)
def validate_env(*, strict: bool = True) -> bool:
"""Validate environment variables at startup.
@@ -38,14 +235,20 @@ def validate_env(*, strict: bool = True) -> bool:
"""
all_ok = True
settings = get_settings()
# Required keys — must be set
for key, desc in _REQUIRED.items():
value = os.environ.get(key, "").strip()
value = getattr(settings, key, "")
if isinstance(value, str):
value = value.strip()
if not value:
logger.error(
"❌ REQUIRED env var %s is not set. %s\n"
" Set it in .env or via Docker secrets (%s_FILE).",
key, desc, key,
key,
desc,
key,
)
all_ok = False
@@ -55,21 +258,32 @@ def validate_env(*, strict: bool = True) -> bool:
# Critical-warn keys — app works but security/functionality is degraded
for key, desc in _CRITICAL_WARN.items():
value = os.environ.get(key, "").strip()
value = getattr(settings, key, "")
if isinstance(value, str):
value = value.strip()
if not value:
logger.critical(
"🔓 CRITICAL: env var %s is not set — %s\n"
" This is safe for local dev but MUST be set in production.",
key, desc,
allow_insecure = bool(getattr(settings, "ALLOW_INSECURE_ADMIN", False))
logger.warning(
"⚠️ ADMIN_KEY is not set%s — %s",
" and ALLOW_INSECURE_ADMIN=true" if allow_insecure else "",
desc,
)
if not allow_insecure:
logger.critical(
"🔓 CRITICAL: env var %s is not set — this MUST be set in production.",
key,
)
# Optional keys — warn if missing
for key, desc in _OPTIONAL.items():
value = os.environ.get(key, "").strip()
value = getattr(settings, key, "")
if isinstance(value, str):
value = value.strip()
if not value:
logger.warning(
"⚠️ Optional env var %s is not set — %s", key, desc
)
logger.warning("⚠️ Optional env var %s is not set — %s", key, desc)
# ── Security posture audit ────────────────────────────────────────
_audit_security_config(settings)
if all_ok:
logger.info("✅ Environment validation passed.")
backend/services/fetch_health.py
+94
View File
@@ -0,0 +1,94 @@
"""Fetch health registry — tracks per-source success/failure counts and timings."""
import logging
import threading
from datetime import datetime
from typing import Any, Dict, Optional
from services.fetchers._store import _data_lock, source_freshness
logger = logging.getLogger(__name__)
_health: Dict[str, Dict[str, Any]] = {}
_lock = threading.Lock()
def _now_iso() -> str:
return datetime.utcnow().isoformat()
def _update_source_freshness(source: str, *, ok: bool, error_msg: Optional[str] = None):
"""Mirror health summary into shared store for visibility."""
with _data_lock:
entry = source_freshness.get(source, {})
if ok:
entry["last_ok"] = _now_iso()
else:
entry["last_error"] = _now_iso()
if error_msg:
entry["last_error_msg"] = error_msg[:200]
source_freshness[source] = entry
def record_success(source: str, duration_s: Optional[float] = None, count: Optional[int] = None):
"""Record a successful fetch for a source."""
now = _now_iso()
with _lock:
entry = _health.setdefault(
source,
{
"ok_count": 0,
"error_count": 0,
"last_ok": None,
"last_error": None,
"last_error_msg": None,
"last_duration_ms": None,
"avg_duration_ms": None,
"last_count": None,
},
)
entry["ok_count"] += 1
entry["last_ok"] = now
if duration_s is not None:
dur_ms = round(duration_s * 1000, 1)
entry["last_duration_ms"] = dur_ms
prev_avg = entry["avg_duration_ms"] or 0.0
n = entry["ok_count"]
entry["avg_duration_ms"] = round(((prev_avg * (n - 1)) + dur_ms) / n, 1)
if count is not None:
entry["last_count"] = count
_update_source_freshness(source, ok=True)
def record_failure(source: str, error: Exception, duration_s: Optional[float] = None):
"""Record a failed fetch for a source."""
now = _now_iso()
err_msg = str(error)
with _lock:
entry = _health.setdefault(
source,
{
"ok_count": 0,
"error_count": 0,
"last_ok": None,
"last_error": None,
"last_error_msg": None,
"last_duration_ms": None,
"avg_duration_ms": None,
"last_count": None,
},
)
entry["error_count"] += 1
entry["last_error"] = now
entry["last_error_msg"] = err_msg[:200]
if duration_s is not None:
entry["last_duration_ms"] = round(duration_s * 1000, 1)
_update_source_freshness(source, ok=False, error_msg=err_msg)
def get_health_snapshot() -> Dict[str, Dict[str, Any]]:
"""Return a snapshot of current fetch health state."""
with _lock:
return {k: dict(v) for k, v in _health.items()}
backend/services/fetchers/_store.py
+199 -4
View File
@@ -3,14 +3,68 @@
Central location for latest_data, source_timestamps, and the data lock.
Every fetcher imports from here instead of maintaining its own copy.
"""
import threading
import logging
from datetime import datetime
from typing import Any, Dict, List, Optional, TypedDict
logger = logging.getLogger("services.data_fetcher")
class DashboardData(TypedDict, total=False):
"""Schema for the in-memory data store. Catches key typos at dev time."""
last_updated: Optional[str]
news: List[Dict[str, Any]]
stocks: Dict[str, Any]
oil: Dict[str, Any]
commercial_flights: List[Dict[str, Any]]
private_flights: List[Dict[str, Any]]
private_jets: List[Dict[str, Any]]
flights: List[Dict[str, Any]]
ships: List[Dict[str, Any]]
military_flights: List[Dict[str, Any]]
tracked_flights: List[Dict[str, Any]]
cctv: List[Dict[str, Any]]
weather: Optional[Dict[str, Any]]
earthquakes: List[Dict[str, Any]]
uavs: List[Dict[str, Any]]
frontlines: Optional[Any]
gdelt: List[Dict[str, Any]]
liveuamap: List[Dict[str, Any]]
kiwisdr: List[Dict[str, Any]]
space_weather: Optional[Dict[str, Any]]
internet_outages: List[Dict[str, Any]]
firms_fires: List[Dict[str, Any]]
datacenters: List[Dict[str, Any]]
airports: List[Dict[str, Any]]
gps_jamming: List[Dict[str, Any]]
satellites: List[Dict[str, Any]]
satellite_source: str
prediction_markets: List[Dict[str, Any]]
sigint: List[Dict[str, Any]]
sigint_totals: Dict[str, Any]
mesh_channel_stats: Dict[str, Any]
meshtastic_map_nodes: List[Dict[str, Any]]
meshtastic_map_fetched_at: Optional[float]
weather_alerts: List[Dict[str, Any]]
air_quality: List[Dict[str, Any]]
volcanoes: List[Dict[str, Any]]
fishing_activity: List[Dict[str, Any]]
satnogs_stations: List[Dict[str, Any]]
satnogs_observations: List[Dict[str, Any]]
tinygs_satellites: List[Dict[str, Any]]
ukraine_alerts: List[Dict[str, Any]]
power_plants: List[Dict[str, Any]]
viirs_change_nodes: List[Dict[str, Any]]
fimi: Dict[str, Any]
psk_reporter: List[Dict[str, Any]]
correlations: List[Dict[str, Any]]
# In-memory store
latest_data = {
latest_data: DashboardData = {
"last_updated": None,
"news": [],
"stocks": {},
@@ -32,17 +86,158 @@ latest_data = {
"firms_fires": [],
"datacenters": [],
"military_bases": [],
"power_plants": []
"prediction_markets": [],
"sigint": [],
"sigint_totals": {},
"mesh_channel_stats": {},
"meshtastic_map_nodes": [],
"meshtastic_map_fetched_at": None,
"weather_alerts": [],
"air_quality": [],
"volcanoes": [],
"fishing_activity": [],
"satnogs_stations": [],
"satnogs_observations": [],
"tinygs_satellites": [],
"ukraine_alerts": [],
"power_plants": [],
"viirs_change_nodes": [],
"fimi": {},
"psk_reporter": [],
"correlations": [],
}
# Per-source freshness timestamps
source_timestamps = {}
# Per-source health/freshness metadata (last ok/error)
source_freshness: dict[str, dict] = {}
def _mark_fresh(*keys):
"""Record the current UTC time for one or more data source keys."""
now = datetime.utcnow().isoformat()
for k in keys:
source_timestamps[k] = now
with _data_lock:
for k in keys:
source_timestamps[k] = now
# Thread lock for safe reads/writes to latest_data
_data_lock = threading.Lock()
# Monotonic version counter — incremented on each data update cycle.
# Used for cheap ETag generation instead of MD5-hashing the full response.
_data_version: int = 0
def bump_data_version() -> None:
"""Increment the data version counter after a fetch cycle completes."""
global _data_version
_data_version += 1
def get_data_version() -> int:
"""Return the current data version (for ETag generation)."""
return _data_version
_active_layers_version: int = 0
def bump_active_layers_version() -> None:
"""Increment the active-layer version when frontend toggles change response shape."""
global _active_layers_version
_active_layers_version += 1
def get_active_layers_version() -> int:
"""Return the current active-layer version (for ETag generation)."""
return _active_layers_version
def get_latest_data_subset(*keys: str) -> DashboardData:
"""Return a shallow snapshot of only the requested top-level keys.
This avoids cloning the entire dashboard store for endpoints that only need
a small tier-specific subset.
"""
with _data_lock:
snap: DashboardData = {}
for key in keys:
value = latest_data.get(key)
if isinstance(value, list):
snap[key] = list(value)
elif isinstance(value, dict):
snap[key] = dict(value)
else:
snap[key] = value
return snap
def get_latest_data_subset_refs(*keys: str) -> DashboardData:
"""Return direct top-level references for read-only hot paths.
Writers replace top-level values under the lock instead of mutating them
in place, so readers can safely use these references after releasing the
lock as long as they do not modify them.
"""
with _data_lock:
snap: DashboardData = {}
for key in keys:
snap[key] = latest_data.get(key)
return snap
def get_source_timestamps_snapshot() -> dict[str, str]:
"""Return a stable copy of per-source freshness timestamps."""
with _data_lock:
return dict(source_timestamps)
# ---------------------------------------------------------------------------
# Active layers — frontend POSTs toggles, fetchers check before running.
# Keep these aligned with the dashboard's default layer state so startup does
# not fetch heavyweight feeds the UI starts with disabled.
# ---------------------------------------------------------------------------
active_layers: dict[str, bool] = {
"flights": True,
"private": True,
"jets": True,
"military": True,
"tracked": True,
"satellites": True,
"ships_military": True,
"ships_cargo": True,
"ships_civilian": True,
"ships_passenger": True,
"ships_tracked_yachts": True,
"earthquakes": True,
"cctv": True,
"ukraine_frontline": True,
"global_incidents": True,
"gps_jamming": True,
"kiwisdr": True,
"scanners": True,
"firms": True,
"internet_outages": True,
"datacenters": True,
"military_bases": True,
"sigint_meshtastic": True,
"sigint_aprs": True,
"weather_alerts": True,
"air_quality": True,
"volcanoes": True,
"fishing_activity": True,
"satnogs": True,
"tinygs": True,
"ukraine_alerts": True,
"power_plants": False,
"viirs_nightlights": False,
"psk_reporter": True,
"correlations": True,
}
def is_any_active(*layer_names: str) -> bool:
"""Return True if any of the given layer names is currently active."""
return any(active_layers.get(name, True) for name in layer_names)
backend/services/fetchers/earth_observation.py
+480 -26
View File
@@ -1,8 +1,15 @@
"""Earth-observation fetchers — earthquakes, FIRMS fires, space weather, weather radar."""
"""Earth-observation fetchers — earthquakes, FIRMS fires, space weather, weather radar,
severe weather alerts, air quality, volcanoes."""
import csv
import io
import json
import logging
import os
import time
import heapq
from datetime import datetime
from pathlib import Path
from services.network_utils import fetch_with_curl
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
from services.fetchers.retry import with_retry
@@ -15,6 +22,10 @@ logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
@with_retry(max_retries=1, base_delay=1)
def fetch_earthquakes():
from services.fetchers._store import is_any_active
if not is_any_active("earthquakes"):
return
quakes = []
try:
url = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_day.geojson"
@@ -24,12 +35,16 @@ def fetch_earthquakes():
for f in features[:50]:
mag = f["properties"]["mag"]
lng, lat, depth = f["geometry"]["coordinates"]
quakes.append({
"id": f["id"], "mag": mag,
"lat": lat, "lng": lng,
"place": f["properties"]["place"]
})
except Exception as e:
quakes.append(
{
"id": f["id"],
"mag": mag,
"lat": lat,
"lng": lng,
"place": f["properties"]["place"],
}
)
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching earthquakes: {e}")
with _data_lock:
latest_data["earthquakes"] = quakes
@@ -43,6 +58,10 @@ def fetch_earthquakes():
@with_retry(max_retries=1, base_delay=2)
def fetch_firms_fires():
"""Fetch global fire/thermal anomalies from NASA FIRMS (NOAA-20 VIIRS, 24h, no key needed)."""
from services.fetchers._store import is_any_active
if not is_any_active("firms"):
return
fires = []
try:
url = "https://firms.modaps.eosdis.nasa.gov/data/active_fire/noaa-20-viirs-c2/csv/J1_VIIRS_C2_Global_24h.csv"
@@ -58,18 +77,23 @@ def fetch_firms_fires():
conf = row.get("confidence", "nominal")
daynight = row.get("daynight", "")
bright = float(row.get("bright_ti4", 0))
all_rows.append({
"lat": lat, "lng": lng, "frp": frp,
"brightness": bright, "confidence": conf,
"daynight": daynight,
"acq_date": row.get("acq_date", ""),
"acq_time": row.get("acq_time", ""),
})
all_rows.append(
{
"lat": lat,
"lng": lng,
"frp": frp,
"brightness": bright,
"confidence": conf,
"daynight": daynight,
"acq_date": row.get("acq_date", ""),
"acq_time": row.get("acq_time", ""),
}
)
except (ValueError, TypeError):
continue
fires = heapq.nlargest(5000, all_rows, key=lambda x: x["frp"])
logger.info(f"FIRMS fires: {len(fires)} hotspots (from {response.status_code})")
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching FIRMS fires: {e}")
with _data_lock:
latest_data["firms_fires"] = fires
@@ -77,6 +101,90 @@ def fetch_firms_fires():
_mark_fresh("firms_fires")
# ---------------------------------------------------------------------------
# NASA FIRMS Country-Scoped Fires (enriches global CSV with conflict zones)
# ---------------------------------------------------------------------------
# Conflict-zone countries of interest for higher-detail fire/thermal data
_FIRMS_COUNTRIES = ["ISR", "IRN", "IRQ", "LBN", "SYR", "YEM", "SAU", "UKR", "RUS", "TUR"]
@with_retry(max_retries=1, base_delay=2)
def fetch_firms_country_fires():
"""Fetch country-scoped fire hotspots from NASA FIRMS MAP_KEY API.
Supplements the global CSV feed with more granular data for conflict zones.
Merges results into the existing firms_fires data store (no new frontend key).
Requires FIRMS_MAP_KEY env var (free from NASA Earthdata). Skips if not set.
"""
from services.fetchers._store import is_any_active
if not is_any_active("firms"):
return
map_key = os.environ.get("FIRMS_MAP_KEY", "")
if not map_key:
logger.debug("FIRMS_MAP_KEY not set, skipping country-scoped FIRMS fetch")
return
# Build a set of existing (lat, lng) rounded to 0.01° for dedup
with _data_lock:
existing = set()
for f in latest_data.get("firms_fires", []):
existing.add((round(f["lat"], 2), round(f["lng"], 2)))
new_fires = []
for country in _FIRMS_COUNTRIES:
try:
url = (
f"https://firms.modaps.eosdis.nasa.gov/api/country/csv/"
f"{map_key}/VIIRS_NOAA20_NRT/{country}/1"
)
response = fetch_with_curl(url, timeout=15)
if response.status_code != 200:
logger.debug(f"FIRMS country {country}: HTTP {response.status_code}")
continue
reader = csv.DictReader(io.StringIO(response.text))
for row in reader:
try:
lat = float(row.get("latitude", 0))
lng = float(row.get("longitude", 0))
key = (round(lat, 2), round(lng, 2))
if key in existing:
continue # Already in global data
existing.add(key)
frp = float(row.get("frp", 0))
new_fires.append({
"lat": lat,
"lng": lng,
"frp": frp,
"brightness": float(row.get("bright_ti4", 0)),
"confidence": row.get("confidence", "nominal"),
"daynight": row.get("daynight", ""),
"acq_date": row.get("acq_date", ""),
"acq_time": row.get("acq_time", ""),
})
except (ValueError, TypeError):
continue
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.debug(f"FIRMS country {country} failed: {e}")
if new_fires:
with _data_lock:
current = latest_data.get("firms_fires", [])
merged = current + new_fires
# Keep top 6000 by FRP (slightly more than global-only cap of 5000)
if len(merged) > 6000:
merged = heapq.nlargest(6000, merged, key=lambda x: x["frp"])
latest_data["firms_fires"] = merged
logger.info(f"FIRMS country enrichment: +{len(new_fires)} fires from {len(_FIRMS_COUNTRIES)} countries")
_mark_fresh("firms_fires")
else:
logger.debug("FIRMS country enrichment: no new fires found")
# ---------------------------------------------------------------------------
# Space Weather (NOAA SWPC)
# ---------------------------------------------------------------------------
@@ -84,7 +192,9 @@ def fetch_firms_fires():
def fetch_space_weather():
"""Fetch NOAA SWPC Kp index and recent solar events."""
try:
kp_resp = fetch_with_curl("https://services.swpc.noaa.gov/json/planetary_k_index_1m.json", timeout=10)
kp_resp = fetch_with_curl(
"https://services.swpc.noaa.gov/json/planetary_k_index_1m.json", timeout=10
)
kp_value = None
kp_text = "QUIET"
if kp_resp.status_code == 200:
@@ -102,16 +212,20 @@ def fetch_space_weather():
kp_text = "UNSETTLED"
events = []
ev_resp = fetch_with_curl("https://services.swpc.noaa.gov/json/edited_events.json", timeout=10)
ev_resp = fetch_with_curl(
"https://services.swpc.noaa.gov/json/edited_events.json", timeout=10
)
if ev_resp.status_code == 200:
all_events = ev_resp.json()
for ev in all_events[-10:]:
events.append({
"type": ev.get("type", ""),
"begin": ev.get("begin", ""),
"end": ev.get("end", ""),
"classtype": ev.get("classtype", ""),
})
events.append(
{
"type": ev.get("type", ""),
"begin": ev.get("begin", ""),
"end": ev.get("end", ""),
"classtype": ev.get("classtype", ""),
}
)
with _data_lock:
latest_data["space_weather"] = {
@@ -121,7 +235,7 @@ def fetch_space_weather():
}
_mark_fresh("space_weather")
logger.info(f"Space weather: Kp={kp_value} ({kp_text}), {len(events)} events")
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching space weather: {e}")
@@ -138,7 +252,347 @@ def fetch_weather():
if "radar" in data and "past" in data["radar"]:
latest_time = data["radar"]["past"][-1]["time"]
with _data_lock:
latest_data["weather"] = {"time": latest_time, "host": data.get("host", "https://tilecache.rainviewer.com")}
latest_data["weather"] = {
"time": latest_time,
"host": data.get("host", "https://tilecache.rainviewer.com"),
}
_mark_fresh("weather")
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching weather: {e}")
# ---------------------------------------------------------------------------
# NOAA/NWS Severe Weather Alerts
# ---------------------------------------------------------------------------
@with_retry(max_retries=1, base_delay=2)
def fetch_weather_alerts():
"""Fetch active severe weather alerts from NOAA/NWS (US coverage, GeoJSON polygons)."""
from services.fetchers._store import is_any_active
if not is_any_active("weather_alerts"):
return
alerts = []
try:
url = "https://api.weather.gov/alerts/active?status=actual"
headers = {
"User-Agent": "(ShadowBroker OSINT Dashboard, github.com/BigBodyCobain/Shadowbroker)",
"Accept": "application/geo+json",
}
response = fetch_with_curl(url, timeout=15, headers=headers)
if response.status_code == 200:
features = response.json().get("features", [])
for f in features:
props = f.get("properties", {})
geom = f.get("geometry")
if not geom:
continue # skip zone-only alerts with no polygon
alerts.append(
{
"id": props.get("id", ""),
"event": props.get("event", ""),
"severity": props.get("severity", "Unknown"),
"certainty": props.get("certainty", ""),
"urgency": props.get("urgency", ""),
"headline": props.get("headline", ""),
"description": (props.get("description", "") or "")[:300],
"expires": props.get("expires", ""),
"geometry": geom,
}
)
logger.info(f"Weather alerts: {len(alerts)} active (with polygons)")
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching weather alerts: {e}")
with _data_lock:
latest_data["weather_alerts"] = alerts
if alerts:
_mark_fresh("weather_alerts")
# ---------------------------------------------------------------------------
# Air Quality (OpenAQ v3)
# ---------------------------------------------------------------------------
def _pm25_to_aqi(pm25: float) -> int:
"""Convert PM2.5 concentration (µg/m³) to US EPA AQI."""
breakpoints = [
(0, 12.0, 0, 50),
(12.1, 35.4, 51, 100),
(35.5, 55.4, 101, 150),
(55.5, 150.4, 151, 200),
(150.5, 250.4, 201, 300),
(250.5, 500.4, 301, 500),
]
for c_lo, c_hi, i_lo, i_hi in breakpoints:
if pm25 <= c_hi:
return round(((i_hi - i_lo) / (c_hi - c_lo)) * (pm25 - c_lo) + i_lo)
return 500
@with_retry(max_retries=1, base_delay=2)
def fetch_air_quality():
"""Fetch global air quality stations with PM2.5 data from OpenAQ."""
from services.fetchers._store import is_any_active
if not is_any_active("air_quality"):
return
stations = []
api_key = os.environ.get("OPENAQ_API_KEY", "")
if not api_key:
logger.debug("OPENAQ_API_KEY not set, skipping air quality fetch")
return
try:
url = "https://api.openaq.org/v3/locations?limit=5000&parameter_id=2&order_by=datetime&sort_order=desc"
headers = {"X-API-Key": api_key}
response = fetch_with_curl(url, timeout=30, headers=headers)
if response.status_code == 200:
results = response.json().get("results", [])
for loc in results:
coords = loc.get("coordinates", {})
lat = coords.get("latitude")
lng = coords.get("longitude")
if lat is None or lng is None:
continue
pm25 = None
for p in loc.get("parameters", []):
if p.get("id") == 2:
pm25 = p.get("lastValue")
break
if pm25 is None:
continue
pm25_val = float(pm25)
if pm25_val < 0:
continue
stations.append(
{
"id": loc.get("id"),
"name": loc.get("name", "Unknown"),
"lat": lat,
"lng": lng,
"pm25": round(pm25_val, 1),
"aqi": _pm25_to_aqi(pm25_val),
"country": loc.get("country", {}).get("code", ""),
}
)
logger.info(f"Air quality: {len(stations)} stations")
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching air quality: {e}")
with _data_lock:
latest_data["air_quality"] = stations
if stations:
_mark_fresh("air_quality")
# ---------------------------------------------------------------------------
# Volcanoes (Smithsonian Global Volcanism Program)
# ---------------------------------------------------------------------------
@with_retry(max_retries=2, base_delay=5)
def fetch_volcanoes():
"""Fetch Holocene volcanoes from Smithsonian GVP WFS (static reference data)."""
from services.fetchers._store import is_any_active
if not is_any_active("volcanoes"):
return
volcanoes = []
try:
url = (
"https://webservices.volcano.si.edu/geoserver/GVP-VOTW/wfs"
"?service=WFS&version=2.0.0&request=GetFeature"
"&typeName=GVP-VOTW:E3WebApp_HoloceneVolcanoes"
"&outputFormat=application/json"
)
response = fetch_with_curl(url, timeout=30)
if response.status_code == 200:
features = response.json().get("features", [])
for f in features:
props = f.get("properties", {})
geom = f.get("geometry", {})
coords = geom.get("coordinates", [None, None])
if coords[0] is None:
continue
last_eruption = props.get("LastEruption")
last_eruption_year = None
if last_eruption is not None:
try:
last_eruption_year = int(last_eruption)
except (ValueError, TypeError):
pass
volcanoes.append(
{
"name": props.get("VolcanoName", "Unknown"),
"type": props.get("VolcanoType", ""),
"country": props.get("Country", ""),
"region": props.get("TectonicSetting", ""),
"elevation": props.get("Elevation", 0),
"last_eruption_year": last_eruption_year,
"lat": coords[1],
"lng": coords[0],
}
)
logger.info(f"Volcanoes: {len(volcanoes)} Holocene volcanoes loaded")
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching volcanoes: {e}")
with _data_lock:
latest_data["volcanoes"] = volcanoes
if volcanoes:
_mark_fresh("volcanoes")
# ---------------------------------------------------------------------------
# VIIRS Night Lights Change Detection (Google Earth Engine — optional)
# ---------------------------------------------------------------------------
_VIIRS_CACHE_PATH = Path(__file__).parent.parent.parent / "data" / "viirs_change_nodes.json"
_VIIRS_CACHE_MAX_AGE_S = 86400 # 24 hours
# Conflict-zone AOIs: (name, south, west, north, east)
_VIIRS_AOIS = [
("Gaza Strip", 31.2, 34.2, 31.6, 34.6),
("Kharkiv Oblast", 48.5, 35.0, 50.5, 38.5),
("Donetsk Oblast", 47.0, 36.5, 49.0, 39.5),
("Zaporizhzhia Oblast", 46.5, 34.5, 48.5, 37.0),
("Aleppo", 35.8, 36.5, 36.5, 37.5),
("Khartoum", 15.2, 32.2, 15.9, 32.9),
("Sana'a", 14.9, 43.8, 15.6, 44.5),
("Mosul", 36.0, 42.8, 36.7, 43.5),
("Mariupol", 46.9, 37.2, 47.3, 37.8),
("Southern Lebanon", 33.0, 35.0, 33.5, 36.0),
]
_VIIRS_SEVERITY_THRESHOLDS = [
(-100, -70, "severe"),
(-70, -50, "high"),
(-50, -30, "moderate"),
(30, 100, "growth"),
(100, 500, "rapid_growth"),
]
def _classify_viirs_severity(pct_change: float):
for lo, hi, label in _VIIRS_SEVERITY_THRESHOLDS:
if lo <= pct_change <= hi:
return label
return None
def _load_viirs_stale_cache():
"""Load stale cache if available (when GEE is not configured)."""
if _VIIRS_CACHE_PATH.exists():
try:
cached = json.loads(_VIIRS_CACHE_PATH.read_text(encoding="utf-8"))
with _data_lock:
latest_data["viirs_change_nodes"] = cached
_mark_fresh("viirs_change_nodes")
logger.info(f"VIIRS change nodes: loaded {len(cached)} from stale cache")
except Exception:
pass
@with_retry(max_retries=1, base_delay=5)
def fetch_viirs_change_nodes():
"""Compute VIIRS nighttime radiance change nodes via GEE (optional)."""
from services.fetchers._store import is_any_active
if not is_any_active("viirs_nightlights"):
return
# Check cache freshness first
if _VIIRS_CACHE_PATH.exists():
age = time.time() - _VIIRS_CACHE_PATH.stat().st_mtime
if age < _VIIRS_CACHE_MAX_AGE_S:
try:
cached = json.loads(_VIIRS_CACHE_PATH.read_text(encoding="utf-8"))
with _data_lock:
latest_data["viirs_change_nodes"] = cached
_mark_fresh("viirs_change_nodes")
logger.info(f"VIIRS change nodes: loaded {len(cached)} from cache (age {age:.0f}s)")
return
except Exception as e:
logger.warning(f"VIIRS cache read failed: {e}")
# Try importing earthengine-api (optional dependency)
try:
import ee
except ImportError:
logger.debug("earthengine-api not installed, skipping VIIRS change detection")
_load_viirs_stale_cache()
return
# Authenticate with service account
sa_key_path = os.environ.get("GEE_SERVICE_ACCOUNT_KEY", "")
if not sa_key_path:
logger.debug("GEE_SERVICE_ACCOUNT_KEY not set, skipping VIIRS change detection")
_load_viirs_stale_cache()
return
try:
credentials = ee.ServiceAccountCredentials(None, key_file=sa_key_path)
ee.Initialize(credentials)
except Exception as e:
logger.error(f"GEE authentication failed: {e}")
_load_viirs_stale_cache()
return
# Compute change nodes for each AOI
nodes = []
viirs = ee.ImageCollection("NOAA/VIIRS/DNB/MONTHLY_V1/VCMCFG").select("avg_rad")
for aoi_name, s_lat, w_lng, n_lat, e_lng in _VIIRS_AOIS:
try:
aoi = ee.Geometry.Rectangle([w_lng, s_lat, e_lng, n_lat])
# Most recent available date
now = ee.Date(datetime.utcnow().isoformat()[:10])
# Current: 12-month rolling mean ending now
current = viirs.filterDate(now.advance(-12, "month"), now).mean().clip(aoi)
# Baseline: 12-month mean ending 12 months ago
baseline = viirs.filterDate(
now.advance(-24, "month"), now.advance(-12, "month")
).mean().clip(aoi)
# Floor baseline at 0.5 nW/cm²/sr to avoid div-by-zero in dark areas
baseline_safe = baseline.max(0.5)
# Percentage change
change = current.subtract(baseline).divide(baseline_safe).multiply(100)
# Only keep pixels with >30% absolute change
sig_mask = change.abs().gt(30)
change_masked = change.updateMask(sig_mask)
# Sample up to 200 points per AOI
samples = change_masked.sample(
region=aoi, scale=500, numPixels=200, geometries=True
)
sample_list = samples.getInfo()
for feat in sample_list.get("features", []):
coords = feat["geometry"]["coordinates"]
pct = feat["properties"].get("avg_rad", 0)
severity = _classify_viirs_severity(pct)
if severity is None:
continue
nodes.append({
"lat": round(coords[1], 4),
"lng": round(coords[0], 4),
"mean_change_pct": round(pct, 1),
"severity": severity,
"aoi_name": aoi_name,
})
except Exception as e:
logger.warning(f"VIIRS change detection failed for {aoi_name}: {e}")
continue
# Save to cache
try:
_VIIRS_CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
_VIIRS_CACHE_PATH.write_text(
json.dumps(nodes, separators=(",", ":")), encoding="utf-8"
)
except Exception as e:
logger.warning(f"Failed to write VIIRS cache: {e}")
with _data_lock:
latest_data["viirs_change_nodes"] = nodes
if nodes:
_mark_fresh("viirs_change_nodes")
logger.info(f"VIIRS change nodes: {len(nodes)} nodes from {len(_VIIRS_AOIS)} AOIs")
backend/services/fetchers/emissions.py
+131
View File
@@ -0,0 +1,131 @@
"""
Fuel burn & CO2 emissions estimator for private jets.
Based on manufacturer-published cruise fuel burn rates (GPH at long-range cruise).
1 US gallon of Jet-A produces ~21.1 lbs (9.57 kg) of CO2.
"""
JET_A_CO2_KG_PER_GALLON = 9.57
# ICAO type code -> gallons per hour at long-range cruise
FUEL_BURN_GPH: dict[str, int] = {
# Gulfstream
"GLF6": 430, # G650/G650ER
"G700": 480, # G700
"GLF5": 390, # G550
"GVSP": 400, # GV-SP
"GLF4": 330, # G-IV
# Bombardier
"GL7T": 490, # Global 7500
"GLEX": 430, # Global Express/6000/6500
"GL5T": 420, # Global 5000/5500
"CL35": 220, # Challenger 350
"CL60": 310, # Challenger 604/605
"CL30": 200, # Challenger 300
"CL65": 320, # Challenger 650
# Dassault
"F7X": 350, # Falcon 7X
"F8X": 370, # Falcon 8X
"F900": 285, # Falcon 900/900EX/900LX
"F2TH": 230, # Falcon 2000
"FA50": 240, # Falcon 50
# Cessna
"CITX": 280, # Citation X
"C68A": 195, # Citation Latitude
"C700": 230, # Citation Longitude
"C680": 220, # Citation Sovereign
"C560": 190, # Citation Excel/XLS
"C510": 75, # Citation Mustang
"CJ3": 120, # CJ3
"CJ4": 135, # CJ4
# Boeing
"B737": 850, # BBJ (737)
"B738": 920, # BBJ2 (737-800)
"B752": 1100, # 757-200
"B762": 1400, # 767-200
"B788": 1200, # 787-8
# Airbus
"A318": 780, # ACJ318
"A319": 850, # ACJ319
"A320": 900, # ACJ320
"A343": 1800, # A340-300
"A346": 2100, # A340-600
# Pilatus
"PC24": 115, # PC-24
"PC12": 60, # PC-12
# Embraer
"E55P": 185, # Legacy 500
"E135": 300, # Legacy 600/650
"E50P": 135, # Phenom 300
"E500": 80, # Phenom 100
# Learjet
"LJ60": 195, # Learjet 60
"LJ75": 185, # Learjet 75
"LJ45": 175, # Learjet 45
# Hawker
"H25B": 210, # Hawker 800/800XP
"H25C": 215, # Hawker 900XP
# Beechcraft
"B350": 100, # King Air 350
"B200": 80, # King Air 200/250
}
# Common string names -> ICAO type code
_ALIASES: dict[str, str] = {
"Gulfstream G650": "GLF6", "Gulfstream G650ER": "GLF6", "G650": "GLF6", "G650ER": "GLF6",
"Gulfstream G700": "G700",
"Gulfstream G550": "GLF5", "G550": "GLF5", "G500": "GLF5",
"Gulfstream GV": "GVSP", "Gulfstream G-V": "GVSP", "GV": "GVSP",
"Gulfstream G-IV": "GLF4", "Gulfstream GIV": "GLF4", "G450": "GLF4",
"Global 7500": "GL7T", "Bombardier Global 7500": "GL7T",
"Global 6000": "GLEX", "Global Express": "GLEX", "Bombardier Global 6000": "GLEX",
"Global 5000": "GL5T",
"Challenger 350": "CL35", "Challenger 300": "CL30",
"Challenger 604": "CL60", "Challenger 605": "CL60", "Challenger 650": "CL65",
"Falcon 7X": "F7X", "Dassault Falcon 7X": "F7X",
"Falcon 8X": "F8X", "Dassault Falcon 8X": "F8X",
"Falcon 900": "F900", "Falcon 900LX": "F900", "Falcon 900EX": "F900",
"Falcon 2000": "F2TH",
"Citation X": "CITX", "Citation Latitude": "C68A", "Citation Longitude": "C700",
"Boeing 757-200": "B752", "757-200": "B752", "Boeing 757": "B752",
"Boeing 767-200": "B762", "767-200": "B762", "Boeing 767": "B762",
"Boeing 787-8": "B788", "Boeing 787": "B788",
"Boeing 737": "B737", "737 BBJ": "B737", "BBJ": "B737",
"Airbus A340-300": "A343", "A340-300": "A343", "A340": "A343",
"Airbus A318": "A318",
"Pilatus PC-24": "PC24", "PC-24": "PC24",
"Legacy 500": "E55P", "Legacy 600": "E135", "Phenom 300": "E50P",
"Learjet 60": "LJ60", "Learjet 75": "LJ75",
"Hawker 800": "H25B", "Hawker 900XP": "H25C",
"King Air 350": "B350", "King Air 200": "B200",
}
def get_emissions_info(model: str) -> dict | None:
"""
Given an aircraft model string (ICAO type code or common name),
return emissions info dict or None if unknown.
"""
if not model:
return None
model_clean = model.strip()
# Try direct ICAO code match first
gph = FUEL_BURN_GPH.get(model_clean.upper())
if gph is None:
# Try alias lookup
code = _ALIASES.get(model_clean)
if code:
gph = FUEL_BURN_GPH.get(code)
if gph is None:
# Fuzzy: check if any alias is a substring
model_lower = model_clean.lower()
for alias, code in _ALIASES.items():
if alias.lower() in model_lower or model_lower in alias.lower():
gph = FUEL_BURN_GPH.get(code)
if gph:
break
if gph is None:
return None
return {
"fuel_gph": gph,
"co2_kg_per_hour": round(gph * JET_A_CO2_KG_PER_GALLON, 1),
}
backend/services/fetchers/fimi.py
+274
View File
@@ -0,0 +1,274 @@
"""EUvsDisinfo FIMI (Foreign Information Manipulation & Interference) fetcher.
Parses the EUvsDisinfo RSS feed to extract disinformation narratives,
debunked claims, threat actor mentions, and target country references.
Refreshes every 12 hours (FIMI data updates weekly).
"""
import re
import logging
from datetime import datetime, timezone
import feedparser
from services.network_utils import fetch_with_curl
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
from services.fetchers.retry import with_retry
logger = logging.getLogger("services.data_fetcher")
_FIMI_FEED_URL = "https://euvsdisinfo.eu/feed/"
# ── Threat actor keywords ──────────────────────────────────────────────────
# Map of keyword → canonical actor name. Checked case-insensitively.
_THREAT_ACTORS: dict[str, str] = {
"russia": "Russia",
"russian": "Russia",
"kremlin": "Russia",
"pro-kremlin": "Russia",
"moscow": "Russia",
"china": "China",
"chinese": "China",
"beijing": "China",
"iran": "Iran",
"iranian": "Iran",
"tehran": "Iran",
"north korea": "North Korea",
"pyongyang": "North Korea",
"dprk": "North Korea",
"belarus": "Belarus",
"belarusian": "Belarus",
"minsk": "Belarus",
}
# ── Target country/region keywords ─────────────────────────────────────────
_TARGET_KEYWORDS: dict[str, str] = {
"ukraine": "Ukraine",
"kyiv": "Ukraine",
"moldova": "Moldova",
"georgia": "Georgia",
"tbilisi": "Georgia",
"eu": "EU",
"european union": "EU",
"europe": "Europe",
"nato": "NATO",
"united states": "United States",
"usa": "United States",
"germany": "Germany",
"france": "France",
"poland": "Poland",
"baltic": "Baltics",
"lithuania": "Baltics",
"latvia": "Baltics",
"estonia": "Baltics",
"romania": "Romania",
"czech": "Czech Republic",
"slovakia": "Slovakia",
"armenia": "Armenia",
"africa": "Africa",
"middle east": "Middle East",
"syria": "Syria",
"israel": "Israel",
"serbia": "Serbia",
"india": "India",
"brazil": "Brazil",
}
# ── Disinformation topic keywords (for cross-referencing news) ─────────────
_DISINFO_TOPICS = [
"sanctions",
"energy crisis",
"gas supply",
"nuclear threat",
"nato expansion",
"biolab",
"biological weapon",
"provocation",
"false flag",
"staged",
"nazi",
"genocide",
"referendum",
"regime change",
"coup",
"puppet government",
"election interference",
"election meddling",
"voter fraud",
"migrant invasion",
"refugee crisis",
"civil war",
"food crisis",
"grain deal",
]
# Regex for extracting debunked report URLs from feed HTML
_REPORT_URL_RE = re.compile(
r'https?://euvsdisinfo\.eu/report/[a-z0-9\-]+/?',
re.IGNORECASE,
)
# Regex for extracting the claim title from a report URL slug
_SLUG_RE = re.compile(r'/report/([a-z0-9\-]+)/?$', re.IGNORECASE)
def _slug_to_title(url: str) -> str:
"""Convert a report URL slug to a human-readable title."""
m = _SLUG_RE.search(url)
if not m:
return url
return m.group(1).replace("-", " ").title()
def _count_mentions(text: str, keywords: dict[str, str]) -> dict[str, int]:
"""Count keyword mentions, mapping to canonical names."""
counts: dict[str, int] = {}
text_lower = text.lower()
for kw, canonical in keywords.items():
# Word-boundary match, case-insensitive
pattern = r'\b' + re.escape(kw) + r'\b'
matches = re.findall(pattern, text_lower)
if matches:
counts[canonical] = counts.get(canonical, 0) + len(matches)
return counts
def _extract_disinfo_keywords(text: str) -> list[str]:
"""Return which disinformation topic keywords appear in the text."""
text_lower = text.lower()
found = []
for topic in _DISINFO_TOPICS:
if topic in text_lower:
found.append(topic)
return found
def _is_major_wave(narratives: list[dict], targets: dict[str, int]) -> bool:
"""Heuristic: detect a 'major disinformation wave'.
Triggers when:
- 3+ narratives in the feed mention the same target, OR
- A single target has 10+ total mentions across all narratives, OR
- 5+ distinct debunked claims extracted in one fetch
"""
if not narratives:
return False
# Check per-target narrative count
target_narrative_counts: dict[str, int] = {}
total_claims = 0
for n in narratives:
for t in n.get("targets", []):
target_narrative_counts[t] = target_narrative_counts.get(t, 0) + 1
total_claims += len(n.get("claims", []))
if any(c >= 3 for c in target_narrative_counts.values()):
return True
if any(c >= 10 for c in targets.values()):
return True
if total_claims >= 5:
return True
return False
@with_retry(max_retries=1, base_delay=5)
def fetch_fimi():
"""Fetch and parse the EUvsDisinfo RSS feed."""
try:
resp = fetch_with_curl(_FIMI_FEED_URL, timeout=15)
feed = feedparser.parse(resp.text)
except Exception as e:
logger.warning(f"FIMI feed fetch failed: {e}")
return
if not feed.entries:
logger.warning("FIMI feed: no entries found")
return
narratives = []
all_claims: list[dict] = []
agg_actors: dict[str, int] = {}
agg_targets: dict[str, int] = {}
all_disinfo_kw: set[str] = set()
for entry in feed.entries[:15]: # Cap at 15 entries
title = entry.get("title", "")
link = entry.get("link", "")
published = entry.get("published", "")
summary_html = entry.get("summary", "") or entry.get("description", "")
# Strip HTML tags for text analysis
summary_text = re.sub(r"<[^>]+>", " ", summary_html)
summary_text = re.sub(r"\s+", " ", summary_text).strip()
full_text = f"{title} {summary_text}"
# Extract debunked report URLs
report_urls = list(set(_REPORT_URL_RE.findall(summary_html)))
claims = [{"url": url, "title": _slug_to_title(url)} for url in report_urls]
all_claims.extend(claims)
# Count threat actors
actors = _count_mentions(full_text, _THREAT_ACTORS)
for actor, count in actors.items():
agg_actors[actor] = agg_actors.get(actor, 0) + count
# Count target countries
targets = _count_mentions(full_text, _TARGET_KEYWORDS)
for target, count in targets.items():
agg_targets[target] = agg_targets.get(target, 0) + count
# Extract disinfo topic keywords
disinfo_kw = _extract_disinfo_keywords(full_text)
all_disinfo_kw.update(disinfo_kw)
# Truncate summary for storage
snippet = summary_text[:300] + ("..." if len(summary_text) > 300 else "")
narratives.append({
"title": title,
"link": link,
"published": published,
"snippet": snippet,
"claims": claims,
"actors": list(actors.keys()),
"targets": list(targets.keys()),
"disinfo_keywords": disinfo_kw,
})
# Sort actors and targets by count (descending)
sorted_actors = dict(sorted(agg_actors.items(), key=lambda x: x[1], reverse=True))
sorted_targets = dict(sorted(agg_targets.items(), key=lambda x: x[1], reverse=True))
# Deduplicate claims
seen_urls: set[str] = set()
unique_claims = []
for c in all_claims:
if c["url"] not in seen_urls:
seen_urls.add(c["url"])
unique_claims.append(c)
major_wave = _is_major_wave(narratives, sorted_targets)
fimi_data = {
"narratives": narratives,
"claims": unique_claims,
"threat_actors": sorted_actors,
"targets": sorted_targets,
"disinfo_keywords": sorted(all_disinfo_kw),
"major_wave": major_wave,
"major_wave_target": (
max(sorted_targets, key=sorted_targets.get) if major_wave and sorted_targets else None
),
"last_fetched": datetime.now(timezone.utc).isoformat(),
"source": "EUvsDisinfo",
"source_url": "https://euvsdisinfo.eu",
}
with _data_lock:
latest_data["fimi"] = fimi_data
_mark_fresh("fimi")
logger.info(
f"FIMI fetch complete: {len(narratives)} narratives, "
f"{len(unique_claims)} claims, "
f"{len(sorted_actors)} actors, "
f"major_wave={major_wave}"
)
backend/services/fetchers/financial.py
+144 -80
View File
@@ -1,97 +1,161 @@
"""Financial data fetchers — defense stocks and oil prices.
Uses yfinance batch download to minimise Yahoo Finance requests and avoid rate limiting.
"""
import logging
import yfinance as yf
import math
import random
import time
import os
import urllib.request
import json
import threading
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
from services.fetchers.retry import with_retry
logger = logging.getLogger(__name__)
_YFINANCE_REQUEST_DELAY_SECONDS = 0.5
_YFINANCE_REQUEST_JITTER_SECONDS = 0.2
def _batch_fetch(symbols: list[str], period: str = "5d") -> dict:
"""Fetch multiple tickers in a single yfinance request. Returns {symbol: {price, change_percent, up}}."""
TICKERS_DEFENSE = ["RTX", "LMT", "NOC", "GD", "BA", "PLTR"]
TICKERS_TECH = ["NVDA", "AMD", "TSM", "INTC", "GOOGL", "AMZN", "MSFT", "AAPL", "TSLA", "META", "NFLX", "SMCI", "ARM", "ASML"]
TICKERS_CRYPTO = [
("BTC", "BINANCE:BTCUSDT", "BTC-USD"),
("ETH", "BINANCE:ETHUSDT", "ETH-USD"),
("SOL", "BINANCE:SOLUSDT", "SOL-USD"),
("XRP", "BINANCE:XRPUSDT", "XRP-USD"),
("ADA", "BINANCE:ADAUSDT", "ADA-USD"),
]
# Ticker priority for high-frequency updates (we update these every tick)
PRIORITY_SYMBOLS = ["BTC", "ETH", "NVDA", "PLTR"]
# Persistence for state between short-lived scheduler ticks
_last_fetch_results = {}
_last_fetch_time = 0.0
_rotating_index = 0
_executor = ThreadPoolExecutor(max_workers=10)
def _fetch_finnhub_quote(symbol: str, api_key: str):
"""Fetch from Finnhub. Returns (symbol, data) or (symbol, None)."""
url = f"https://finnhub.io/api/v1/quote?symbol={symbol}&token={api_key}"
try:
hist = yf.download(symbols, period=period, auto_adjust=True, progress=False)
if hist.empty:
return {}
close = hist["Close"]
result = {}
for sym in symbols:
try:
col = close[sym] if len(symbols) > 1 else close
col = col.dropna()
if len(col) < 1:
continue
current = float(col.iloc[-1])
prev = float(col.iloc[0]) if len(col) > 1 else current
change = ((current - prev) / prev * 100) if prev else 0
result[sym] = {
"price": round(current, 2),
"change_percent": round(change, 2),
"up": bool(change >= 0),
}
except Exception as e:
logger.warning(f"Could not parse {sym}: {e}")
return result
req = urllib.request.Request(url)
with urllib.request.urlopen(req, timeout=5) as response:
data = json.loads(response.read().decode())
if "c" not in data or data["c"] == 0:
return symbol, None
current = float(data["c"])
change_p = float(data.get("dp", 0.0) or 0.0)
return symbol, {
"price": round(current, 2),
"change_percent": round(change_p, 2),
"up": bool(change_p >= 0),
}
except Exception as e:
logger.warning(f"Batch fetch failed: {e}")
return {}
logger.debug(f"Finnhub error for {symbol}: {e}")
return symbol, None
_STOCK_TICKERS = ["RTX", "LMT", "NOC", "GD", "BA", "PLTR"]
_OIL_MAP = {"WTI Crude": "CL=F", "Brent Crude": "BZ=F"}
_ALL_TICKERS = _STOCK_TICKERS + list(_OIL_MAP.values())
_MARKET_COOLDOWN_SECONDS = 1800 # fetch at most once every 30 minutes
_last_market_fetch: float = 0.0
def _fetch_yfinance_single(symbol: str, period: str = "2d"):
"""Fetch from yfinance. Returns (symbol, data) or (symbol, None)."""
try:
import yfinance as yf
ticker = yf.Ticker(symbol)
hist = ticker.history(period=period)
if len(hist) >= 1:
current_price = hist["Close"].iloc[-1]
prev_close = hist["Close"].iloc[0] if len(hist) > 1 else current_price
change_percent = ((current_price - prev_close) / prev_close) * 100 if prev_close else 0
current_price_f = float(current_price)
change_percent_f = float(change_percent)
if not math.isfinite(current_price_f) or not math.isfinite(change_percent_f):
return symbol, None
return symbol, {
"price": round(current_price_f, 2),
"change_percent": round(change_percent_f, 2),
"up": bool(change_percent_f >= 0),
}
except Exception as e:
logger.debug(f"Yfinance error for {symbol}: {e}")
return symbol, None
def _fetch_all_market_data():
"""Single yfinance download for all market tickers to avoid rate limiting."""
raw = _batch_fetch(_ALL_TICKERS, period="5d")
stocks = {sym: raw[sym] for sym in _STOCK_TICKERS if sym in raw}
oil = {name: raw[sym] for name, sym in _OIL_MAP.items() if sym in raw}
return stocks, oil
@with_retry(max_retries=1, base_delay=1)
def fetch_financial_markets():
"""Fetches full market list with smart throttling (3s for Finnhub, 60s for yfinance)."""
global _last_fetch_time, _last_fetch_results, _rotating_index
finnhub_key = os.getenv("FINNHUB_API_KEY", "").strip()
use_finnhub = bool(finnhub_key)
now = time.time()
# Throttle logic: 3s for Finnhub, 60s for yfinance fallback
throttle_s = 3.0 if use_finnhub else 60.0
if now - _last_fetch_time < throttle_s and _last_fetch_results:
return # Skip if too frequent
_last_fetch_time = now
# Prepare symbol lists
all_crypto = {label: (f_sym, y_sym) for label, f_sym, y_sym in TICKERS_CRYPTO}
all_stocks = TICKERS_TECH + TICKERS_DEFENSE
subset_to_fetch = []
if use_finnhub:
# Finnhub Free Limit: 60/min.
# Ticking every 3s = 20 ticks/min.
# To stay safe, we fetch only ~3 items per tick.
# Priority items (BTC, ETH) + 1 rotating item.
subset_to_fetch = ["BINANCE:BTCUSDT", "BINANCE:ETHUSDT"]
# Determine rotating ticker
all_other_symbols = []
for sym in all_stocks:
all_other_symbols.append(sym)
for label, (f_sym, y_sym) in all_crypto.items():
if label not in ["BTC", "ETH"]:
all_other_symbols.append(f_sym)
if all_other_symbols:
rotated = all_other_symbols[_rotating_index % len(all_other_symbols)]
subset_to_fetch.append(rotated)
_rotating_index += 1
# Concurrently fetch
futures = [_executor.submit(_fetch_finnhub_quote, s, finnhub_key) for s in subset_to_fetch]
for f in futures:
sym, data = f.result()
if data:
# Map back to readable label if it was crypto
label = sym
for l, (fs, ys) in all_crypto.items():
if fs == sym:
label = l
break
_last_fetch_results[label] = data
else:
# Yahoo Finance Fallback - fetch all (once per minute)
logger.info("Finnhub key missing, using Yahoo Finance 60s update cycle.")
to_fetch = all_stocks + [y_sym for l, (fs, y_sym) in all_crypto.items()]
futures = [_executor.submit(_fetch_yfinance_single, s) for s in to_fetch]
for f in futures:
sym, data = f.result()
if data:
# Map back to readable label if it was crypto
label = sym
for l, (fs, ys) in all_crypto.items():
if ys == sym:
label = l
break
_last_fetch_results[label] = data
@with_retry(max_retries=2, base_delay=10)
def fetch_defense_stocks():
global _last_market_fetch
import time
if time.time() - _last_market_fetch < _MARKET_COOLDOWN_SECONDS:
if not _last_fetch_results:
return
try:
stocks, oil = _fetch_all_market_data()
if stocks:
_last_market_fetch = time.time()
with _data_lock:
latest_data['stocks'] = stocks
if oil:
latest_data['oil'] = oil
_mark_fresh("stocks")
if oil:
_mark_fresh("oil")
logger.info(f"Markets: {len(stocks)} stocks, {len(oil)} oil tickers")
else:
logger.warning("Markets: empty result from yfinance (rate limited?)")
except Exception as e:
logger.error(f"Error fetching market data: {e}")
@with_retry(max_retries=1, base_delay=10)
def fetch_oil_prices():
# Oil is now fetched together with stocks in fetch_defense_stocks to use a single request.
# This function is kept for scheduler compatibility but is a no-op if stocks already ran.
with _data_lock:
if latest_data.get('oil'):
return # Already populated by fetch_defense_stocks
try:
_, oil = _fetch_all_market_data()
if oil:
with _data_lock:
latest_data['oil'] = oil
_mark_fresh("oil")
except Exception as e:
logger.error(f"Error fetching oil: {e}")
latest_data["stocks"] = dict(_last_fetch_results)
latest_data["financial_source"] = "finnhub" if use_finnhub else "yfinance"
_mark_fresh("stocks")
backend/services/fetchers/flights.py
+474 -177
View File
@@ -1,5 +1,7 @@
"""Commercial flight fetching — ADS-B, OpenSky, supplemental sources, routes,
trail accumulation, GPS jamming detection, and holding pattern detection."""
import copy
import re
import os
import time
@@ -8,19 +10,23 @@ import json
import logging
import threading
import concurrent.futures
import random
import requests
from datetime import datetime
from cachetools import TTLCache
from services.network_utils import fetch_with_curl
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
from services.fetchers.plane_alert import enrich_with_plane_alert, enrich_with_tracked_names
from services.fetchers.emissions import get_emissions_info
from services.fetchers.retry import with_retry
from services.constants import GPS_JAMMING_NACP_THRESHOLD, GPS_JAMMING_MIN_RATIO, GPS_JAMMING_MIN_AIRCRAFT
logger = logging.getLogger("services.data_fetcher")
# Pre-compiled regex patterns for airline code extraction (used in hot loop)
_RE_AIRLINE_CODE_1 = re.compile(r'^([A-Z]{3})\d')
_RE_AIRLINE_CODE_2 = re.compile(r'^([A-Z]{3})[A-Z\d]')
_RE_AIRLINE_CODE_1 = re.compile(r"^([A-Z]{3})\d")
_RE_AIRLINE_CODE_2 = re.compile(r"^([A-Z]{3})[A-Z\d]")
# ---------------------------------------------------------------------------
# OpenSky Network API Client (OAuth2)
@@ -39,7 +45,7 @@ class OpenSkyClient:
data = {
"grant_type": "client_credentials",
"client_id": self.client_id,
"client_secret": self.client_secret
"client_secret": self.client_secret,
}
try:
r = requests.post(url, data=data, timeout=10)
@@ -51,13 +57,20 @@ class OpenSkyClient:
return self.token
else:
logger.error(f"OpenSky Auth Failed: {r.status_code} {r.text}")
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
) as e:
logger.error(f"OpenSky Auth Exception: {e}")
return None
opensky_client = OpenSkyClient(
client_id=os.environ.get("OPENSKY_CLIENT_ID", ""),
client_secret=os.environ.get("OPENSKY_CLIENT_SECRET", "")
client_secret=os.environ.get("OPENSKY_CLIENT_SECRET", ""),
)
# Throttling and caching for OpenSky (400 req/day limit)
@@ -68,46 +81,173 @@ cached_opensky_flights = []
# Supplemental ADS-B sources for blind-spot gap-filling
# ---------------------------------------------------------------------------
_BLIND_SPOT_REGIONS = [
{"name": "Yekaterinburg", "lat": 56.8, "lon": 60.6, "radius_nm": 250},
{"name": "Novosibirsk", "lat": 55.0, "lon": 82.9, "radius_nm": 250},
{"name": "Krasnoyarsk", "lat": 56.0, "lon": 92.9, "radius_nm": 250},
{"name": "Vladivostok", "lat": 43.1, "lon": 131.9, "radius_nm": 250},
{"name": "Urumqi", "lat": 43.8, "lon": 87.6, "radius_nm": 250},
{"name": "Chengdu", "lat": 30.6, "lon": 104.1, "radius_nm": 250},
{"name": "Lagos-Accra", "lat": 6.5, "lon": 3.4, "radius_nm": 250},
{"name": "Addis Ababa", "lat": 9.0, "lon": 38.7, "radius_nm": 250},
{"name": "Yekaterinburg", "lat": 56.8, "lon": 60.6, "radius_nm": 250},
{"name": "Novosibirsk", "lat": 55.0, "lon": 82.9, "radius_nm": 250},
{"name": "Krasnoyarsk", "lat": 56.0, "lon": 92.9, "radius_nm": 250},
{"name": "Vladivostok", "lat": 43.1, "lon": 131.9, "radius_nm": 250},
{"name": "Urumqi", "lat": 43.8, "lon": 87.6, "radius_nm": 250},
{"name": "Chengdu", "lat": 30.6, "lon": 104.1, "radius_nm": 250},
{"name": "Lagos-Accra", "lat": 6.5, "lon": 3.4, "radius_nm": 250},
{"name": "Addis Ababa", "lat": 9.0, "lon": 38.7, "radius_nm": 250},
]
_SUPPLEMENTAL_FETCH_INTERVAL = 120
# The blind-spot supplement previously burst several airplanes.live point
# queries in parallel and triggered repeated 429s in real startup logs, so we
# keep it on a long cache interval and pace each regional point query serially.
_SUPPLEMENTAL_FETCH_INTERVAL = 1800
_AIRPLANES_LIVE_DELAY_SECONDS = 1.2
_AIRPLANES_LIVE_DELAY_JITTER_SECONDS = 0.4
last_supplemental_fetch = 0
cached_supplemental_flights = []
# Helicopter type codes (backend classification)
_HELI_TYPES_BACKEND = {
"R22", "R44", "R66", "B06", "B06T", "B204", "B205", "B206", "B212", "B222", "B230",
"B407", "B412", "B427", "B429", "B430", "B505", "B525",
"AS32", "AS35", "AS50", "AS55", "AS65",
"EC20", "EC25", "EC30", "EC35", "EC45", "EC55", "EC75",
"H125", "H130", "H135", "H145", "H155", "H160", "H175", "H215", "H225",
"S55", "S58", "S61", "S64", "S70", "S76", "S92",
"A109", "A119", "A139", "A169", "A189", "AW09",
"MD52", "MD60", "MDHI", "MD90", "NOTR",
"B47G", "HUEY", "GAMA", "CABR", "EXE",
"R22",
"R44",
"R66",
"B06",
"B06T",
"B204",
"B205",
"B206",
"B212",
"B222",
"B230",
"B407",
"B412",
"B427",
"B429",
"B430",
"B505",
"B525",
"AS32",
"AS35",
"AS50",
"AS55",
"AS65",
"EC20",
"EC25",
"EC30",
"EC35",
"EC45",
"EC55",
"EC75",
"H125",
"H130",
"H135",
"H145",
"H155",
"H160",
"H175",
"H215",
"H225",
"S55",
"S58",
"S61",
"S64",
"S70",
"S76",
"S92",
"A109",
"A119",
"A139",
"A169",
"A189",
"AW09",
"MD52",
"MD60",
"MDHI",
"MD90",
"NOTR",
"B47G",
"HUEY",
"GAMA",
"CABR",
"EXE",
}
# Private jet ICAO type designator codes
PRIVATE_JET_TYPES = {
"G150", "G200", "G280", "GLEX", "G500", "G550", "G600", "G650", "G700",
"GLF2", "GLF3", "GLF4", "GLF5", "GLF6", "GL5T", "GL7T", "GV", "GIV",
"CL30", "CL35", "CL60", "BD70", "BD10", "GL5T", "GL7T",
"CRJ1", "CRJ2",
"C25A", "C25B", "C25C", "C500", "C501", "C510", "C525", "C526",
"C550", "C560", "C56X", "C680", "C68A", "C700", "C750",
"FA10", "FA20", "FA50", "FA7X", "FA8X", "F900", "F2TH", "ASTR",
"E35L", "E545", "E550", "E55P", "LEGA", "PH10", "PH30",
"LJ23", "LJ24", "LJ25", "LJ28", "LJ31", "LJ35", "LJ36",
"LJ40", "LJ45", "LJ55", "LJ60", "LJ70", "LJ75",
"H25A", "H25B", "H25C", "HA4T", "BE40", "PRM1",
"HDJT", "PC24", "EA50", "SF50", "GALX",
"G150",
"G200",
"G280",
"GLEX",
"G500",
"G550",
"G600",
"G650",
"G700",
"GLF2",
"GLF3",
"GLF4",
"GLF5",
"GLF6",
"GL5T",
"GL7T",
"GV",
"GIV",
"CL30",
"CL35",
"CL60",
"BD70",
"BD10",
"GL5T",
"GL7T",
"CRJ1",
"CRJ2",
"C25A",
"C25B",
"C25C",
"C500",
"C501",
"C510",
"C525",
"C526",
"C550",
"C560",
"C56X",
"C680",
"C68A",
"C700",
"C750",
"FA10",
"FA20",
"FA50",
"FA7X",
"FA8X",
"F900",
"F2TH",
"ASTR",
"E35L",
"E545",
"E550",
"E55P",
"LEGA",
"PH10",
"PH30",
"LJ23",
"LJ24",
"LJ25",
"LJ28",
"LJ31",
"LJ35",
"LJ36",
"LJ40",
"LJ45",
"LJ55",
"LJ60",
"LJ70",
"LJ75",
"H25A",
"H25B",
"H25C",
"HA4T",
"BE40",
"PRM1",
"HDJT",
"PC24",
"EA50",
"SF50",
"GALX",
}
# Flight trails state
@@ -127,35 +267,59 @@ def _fetch_supplemental_sources(seen_hex: set) -> list:
now = time.time()
if now - last_supplemental_fetch < _SUPPLEMENTAL_FETCH_INTERVAL:
return [f for f in cached_supplemental_flights
if f.get("hex", "").lower().strip() not in seen_hex]
return [
f
for f in cached_supplemental_flights
if f.get("hex", "").lower().strip() not in seen_hex
]
new_supplemental = []
supplemental_hex = set()
def _fetch_airplaneslive(region):
try:
url = (f"https://api.airplanes.live/v2/point/"
f"{region['lat']}/{region['lon']}/{region['radius_nm']}")
url = (
f"https://api.airplanes.live/v2/point/"
f"{region['lat']}/{region['lon']}/{region['radius_nm']}"
)
res = fetch_with_curl(url, timeout=10)
if res.status_code == 200:
data = res.json()
return data.get("ac", [])
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as e:
logger.debug(f"airplanes.live {region['name']} failed: {e}")
return []
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
results = list(pool.map(_fetch_airplaneslive, _BLIND_SPOT_REGIONS))
for region_flights in results:
for idx, region in enumerate(_BLIND_SPOT_REGIONS):
region_flights = _fetch_airplaneslive(region)
for f in region_flights:
h = f.get("hex", "").lower().strip()
if h and h not in seen_hex and h not in supplemental_hex:
f["supplemental_source"] = "airplanes.live"
new_supplemental.append(f)
supplemental_hex.add(h)
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, OSError) as e:
if idx < len(_BLIND_SPOT_REGIONS) - 1:
time.sleep(
_AIRPLANES_LIVE_DELAY_SECONDS
+ random.uniform(0.0, _AIRPLANES_LIVE_DELAY_JITTER_SECONDS)
)
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
OSError,
) as e:
logger.warning(f"airplanes.live supplemental fetch failed: {e}")
ap_count = len(new_supplemental)
@@ -163,8 +327,10 @@ def _fetch_supplemental_sources(seen_hex: set) -> list:
try:
for region in _BLIND_SPOT_REGIONS:
try:
url = (f"https://opendata.adsb.fi/api/v3/lat/"
f"{region['lat']}/lon/{region['lon']}/dist/{region['radius_nm']}")
url = (
f"https://opendata.adsb.fi/api/v3/lat/"
f"{region['lat']}/lon/{region['lon']}/dist/{region['radius_nm']}"
)
res = fetch_with_curl(url, timeout=10)
if res.status_code == 200:
data = res.json()
@@ -174,10 +340,25 @@ def _fetch_supplemental_sources(seen_hex: set) -> list:
f["supplemental_source"] = "adsb.fi"
new_supplemental.append(f)
supplemental_hex.add(h)
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as e:
logger.debug(f"adsb.fi {region['name']} failed: {e}")
time.sleep(1.1)
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
OSError,
) as e:
logger.warning(f"adsb.fi supplemental fetch failed: {e}")
fi_count = len(new_supplemental) - ap_count
@@ -187,8 +368,10 @@ def _fetch_supplemental_sources(seen_hex: set) -> list:
if new_supplemental:
_mark_fresh("supplemental_flights")
logger.info(f"Supplemental: +{len(new_supplemental)} new aircraft from blind-spot "
f"hotspots (airplanes.live: {ap_count}, adsb.fi: {fi_count})")
logger.info(
f"Supplemental: +{len(new_supplemental)} new aircraft from blind-spot "
f"hotspots (airplanes.live: {ap_count}, adsb.fi: {fi_count})"
)
return new_supplemental
@@ -204,18 +387,24 @@ def fetch_routes_background(sampled):
for f in sampled:
c_sign = str(f.get("flight", "")).strip()
if c_sign and c_sign != "UNKNOWN":
callsigns_to_query.append({
"callsign": c_sign,
"lat": f.get("lat", 0),
"lng": f.get("lon", 0)
})
callsigns_to_query.append(
{"callsign": c_sign, "lat": f.get("lat", 0), "lng": f.get("lon", 0)}
)
batch_size = 100
batches = [callsigns_to_query[i:i+batch_size] for i in range(0, len(callsigns_to_query), batch_size)]
batches = [
callsigns_to_query[i : i + batch_size]
for i in range(0, len(callsigns_to_query), batch_size)
]
for batch in batches:
try:
r = fetch_with_curl("https://api.adsb.lol/api/0/routeset", method="POST", json_data={"planes": batch}, timeout=15)
r = fetch_with_curl(
"https://api.adsb.lol/api/0/routeset",
method="POST",
json_data={"planes": batch},
timeout=15,
)
if r.status_code == 200:
route_data = r.json()
route_list = []
@@ -238,7 +427,15 @@ def fetch_routes_background(sampled):
"dest_loc": [dest_apt.get("lon", 0), dest_apt.get("lat", 0)],
}
time.sleep(0.25)
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as e:
logger.debug(f"Route batch request failed: {e}")
finally:
with _routes_lock:
@@ -259,7 +456,9 @@ def _classify_and_publish(all_adsb_flights):
with _routes_lock:
already_running = routes_fetch_in_progress
if not already_running:
threading.Thread(target=fetch_routes_background, args=(all_adsb_flights,), daemon=True).start()
threading.Thread(
target=fetch_routes_background, args=(all_adsb_flights,), daemon=True
).start()
for f in all_adsb_flights:
try:
@@ -308,27 +507,29 @@ def _classify_and_publish(all_adsb_flights):
ac_category = "heli" if model_upper in _HELI_TYPES_BACKEND else "plane"
flights.append({
"callsign": flight_str,
"country": f.get("r", "N/A"),
"lng": float(lng),
"lat": float(lat),
"alt": alt_value,
"heading": heading,
"type": "flight",
"origin_loc": origin_loc,
"dest_loc": dest_loc,
"origin_name": origin_name,
"dest_name": dest_name,
"registration": f.get("r", "N/A"),
"model": f.get("t", "Unknown"),
"icao24": f.get("hex", ""),
"speed_knots": speed_knots,
"squawk": f.get("squawk", ""),
"airline_code": airline_code,
"aircraft_category": ac_category,
"nac_p": f.get("nac_p")
})
flights.append(
{
"callsign": flight_str,
"country": f.get("r", "N/A"),
"lng": float(lng),
"lat": float(lat),
"alt": alt_value,
"heading": heading,
"type": "flight",
"origin_loc": origin_loc,
"dest_loc": dest_loc,
"origin_name": origin_name,
"dest_name": dest_name,
"registration": f.get("r", "N/A"),
"model": f.get("t", "Unknown"),
"icao24": f.get("hex", ""),
"speed_knots": speed_knots,
"squawk": f.get("squawk", ""),
"airline_code": airline_code,
"aircraft_category": ac_category,
"nac_p": f.get("nac_p"),
}
)
except (ValueError, TypeError, KeyError, AttributeError) as loop_e:
logger.error(f"Flight interpolation error: {loop_e}")
continue
@@ -342,80 +543,97 @@ def _classify_and_publish(all_adsb_flights):
for f in flights:
enrich_with_plane_alert(f)
enrich_with_tracked_names(f)
# Attach fuel-burn / CO2 emissions estimate when model is known
model = f.get("model")
if model:
emi = get_emissions_info(model)
if emi:
f["emissions"] = emi
callsign = f.get('callsign', '').strip().upper()
is_commercial_format = bool(re.match(r'^[A-Z]{3}\d{1,4}[A-Z]{0,2}$', callsign))
callsign = f.get("callsign", "").strip().upper()
is_commercial_format = bool(re.match(r"^[A-Z]{3}\d{1,4}[A-Z]{0,2}$", callsign))
if f.get('alert_category'):
f['type'] = 'tracked_flight'
if f.get("alert_category"):
f["type"] = "tracked_flight"
tracked.append(f)
elif f.get('airline_code') or is_commercial_format:
f['type'] = 'commercial_flight'
elif f.get("airline_code") or is_commercial_format:
f["type"] = "commercial_flight"
commercial.append(f)
elif f.get('model', '').upper() in PRIVATE_JET_TYPES:
f['type'] = 'private_jet'
elif f.get("model", "").upper() in PRIVATE_JET_TYPES:
f["type"] = "private_jet"
private_jets.append(f)
else:
f['type'] = 'private_ga'
f["type"] = "private_ga"
private_ga.append(f)
# --- Smart merge: protect against partial API failures ---
prev_commercial_count = len(latest_data.get('commercial_flights', []))
prev_total = prev_commercial_count + len(latest_data.get('private_jets', [])) + len(latest_data.get('private_flights', []))
with _data_lock:
prev_commercial_count = len(latest_data.get("commercial_flights", []))
prev_private_jets_count = len(latest_data.get("private_jets", []))
prev_private_flights_count = len(latest_data.get("private_flights", []))
prev_total = prev_commercial_count + prev_private_jets_count + prev_private_flights_count
new_total = len(commercial) + len(private_jets) + len(private_ga)
if new_total == 0:
logger.warning("No civilian flights found! Skipping overwrite to prevent clearing the map.")
elif prev_total > 100 and new_total < prev_total * 0.5:
logger.warning(f"Flight count dropped from {prev_total} to {new_total} (>50% loss). Keeping previous data to prevent flicker.")
logger.warning(
f"Flight count dropped from {prev_total} to {new_total} (>50% loss). Keeping previous data to prevent flicker."
)
else:
_now = time.time()
def _merge_category(new_list, old_list, max_stale_s=120):
by_icao = {}
for f in old_list:
icao = f.get('icao24', '')
icao = f.get("icao24", "")
if icao:
f.setdefault('_seen_at', _now)
if (_now - f.get('_seen_at', _now)) < max_stale_s:
f.setdefault("_seen_at", _now)
if (_now - f.get("_seen_at", _now)) < max_stale_s:
by_icao[icao] = f
for f in new_list:
icao = f.get('icao24', '')
icao = f.get("icao24", "")
if icao:
f['_seen_at'] = _now
f["_seen_at"] = _now
by_icao[icao] = f
else:
continue
return list(by_icao.values())
with _data_lock:
latest_data['commercial_flights'] = _merge_category(commercial, latest_data.get('commercial_flights', []))
latest_data['private_jets'] = _merge_category(private_jets, latest_data.get('private_jets', []))
latest_data['private_flights'] = _merge_category(private_ga, latest_data.get('private_flights', []))
latest_data["commercial_flights"] = _merge_category(
commercial, latest_data.get("commercial_flights", [])
)
latest_data["private_jets"] = _merge_category(
private_jets, latest_data.get("private_jets", [])
)
latest_data["private_flights"] = _merge_category(
private_ga, latest_data.get("private_flights", [])
)
_mark_fresh("commercial_flights", "private_jets", "private_flights")
with _data_lock:
if flights:
latest_data['flights'] = flights
latest_data["flights"] = flights
# Merge tracked civilian flights with tracked military flights
with _data_lock:
existing_tracked = list(latest_data.get('tracked_flights', []))
existing_tracked = copy.deepcopy(latest_data.get("tracked_flights", []))
fresh_tracked_map = {}
for t in tracked:
icao = t.get('icao24', '').upper()
icao = t.get("icao24", "").upper()
if icao:
fresh_tracked_map[icao] = t
merged_tracked = []
seen_icaos = set()
for old_t in existing_tracked:
icao = old_t.get('icao24', '').upper()
icao = old_t.get("icao24", "").upper()
if icao in fresh_tracked_map:
fresh = fresh_tracked_map[icao]
for key in ('alert_category', 'alert_operator', 'alert_special', 'alert_flag'):
for key in ("alert_category", "alert_operator", "alert_special", "alert_flag"):
if key in old_t and key not in fresh:
fresh[key] = old_t[key]
merged_tracked.append(fresh)
@@ -429,36 +647,47 @@ def _classify_and_publish(all_adsb_flights):
merged_tracked.append(t)
with _data_lock:
latest_data['tracked_flights'] = merged_tracked
logger.info(f"Tracked flights: {len(merged_tracked)} total ({len(fresh_tracked_map)} fresh from civilian)")
latest_data["tracked_flights"] = merged_tracked
logger.info(
f"Tracked flights: {len(merged_tracked)} total ({len(fresh_tracked_map)} fresh from civilian)"
)
# --- Trail Accumulation ---
def _accumulate_trail(f, now_ts, check_route=True):
hex_id = f.get('icao24', '').lower()
hex_id = f.get("icao24", "").lower()
if not hex_id:
return 0, None
if check_route and f.get('origin_name', 'UNKNOWN') != 'UNKNOWN':
f['trail'] = []
if check_route and f.get("origin_name", "UNKNOWN") != "UNKNOWN":
f["trail"] = []
return 0, hex_id
lat, lng, alt = f.get('lat'), f.get('lng'), f.get('alt', 0)
lat, lng, alt = f.get("lat"), f.get("lng"), f.get("alt", 0)
if lat is None or lng is None:
f['trail'] = flight_trails.get(hex_id, {}).get('points', [])
f["trail"] = flight_trails.get(hex_id, {}).get("points", [])
return 0, hex_id
point = [round(lat, 5), round(lng, 5), round(alt, 1), round(now_ts)]
if hex_id not in flight_trails:
flight_trails[hex_id] = {'points': [], 'last_seen': now_ts}
flight_trails[hex_id] = {"points": [], "last_seen": now_ts}
trail_data = flight_trails[hex_id]
if trail_data['points'] and trail_data['points'][-1][0] == point[0] and trail_data['points'][-1][1] == point[1]:
trail_data['last_seen'] = now_ts
if (
trail_data["points"]
and trail_data["points"][-1][0] == point[0]
and trail_data["points"][-1][1] == point[1]
):
trail_data["last_seen"] = now_ts
else:
trail_data['points'].append(point)
trail_data['last_seen'] = now_ts
if len(trail_data['points']) > 200:
trail_data['points'] = trail_data['points'][-200:]
f['trail'] = trail_data['points']
trail_data["points"].append(point)
trail_data["last_seen"] = now_ts
if len(trail_data["points"]) > 200:
trail_data["points"] = trail_data["points"][-200:]
f["trail"] = trail_data["points"]
return 1, hex_id
now_ts = datetime.utcnow().timestamp()
with _data_lock:
military_snapshot = copy.deepcopy(latest_data.get("military_flights", []))
tracked_snapshot = copy.deepcopy(latest_data.get("tracked_flights", []))
raw_flights_snapshot = list(latest_data.get("flights", []))
all_lists = [commercial, private_jets, private_ga, existing_tracked]
seen_hexes = set()
trail_count = 0
@@ -470,97 +699,121 @@ def _classify_and_publish(all_adsb_flights):
if hex_id:
seen_hexes.add(hex_id)
for mf in latest_data.get('military_flights', []):
for mf in military_snapshot:
count, hex_id = _accumulate_trail(mf, now_ts, check_route=False)
trail_count += count
if hex_id:
seen_hexes.add(hex_id)
tracked_hexes = {t.get('icao24', '').lower() for t in latest_data.get('tracked_flights', [])}
tracked_hexes = {t.get("icao24", "").lower() for t in tracked_snapshot}
stale_keys = []
for k, v in flight_trails.items():
cutoff = now_ts - 1800 if k in tracked_hexes else now_ts - 300
if v['last_seen'] < cutoff:
if v["last_seen"] < cutoff:
stale_keys.append(k)
for k in stale_keys:
del flight_trails[k]
if len(flight_trails) > _MAX_TRACKED_TRAILS:
sorted_keys = sorted(flight_trails.keys(), key=lambda k: flight_trails[k]['last_seen'])
sorted_keys = sorted(flight_trails.keys(), key=lambda k: flight_trails[k]["last_seen"])
evict_count = len(flight_trails) - _MAX_TRACKED_TRAILS
for k in sorted_keys[:evict_count]:
del flight_trails[k]
logger.info(f"Trail accumulation: {trail_count} active trails, {len(stale_keys)} pruned, {len(flight_trails)} total")
logger.info(
f"Trail accumulation: {trail_count} active trails, {len(stale_keys)} pruned, {len(flight_trails)} total"
)
# --- GPS Jamming Detection ---
# Uses NACp (Navigation Accuracy Category – Position) from ADS-B to infer
# GPS interference zones, similar to GPSJam.org / Flightradar24.
# NACp < 8 = position accuracy worse than the FAA-mandated 0.05 NM.
#
# Denoising (to suppress false positives from old GA transponders):
# 1. Skip nac_p == 0 ("unknown accuracy") — old transponders that never
# computed accuracy, NOT evidence of jamming. Real jamming shows 1-7.
# 2. Require minimum aircraft per grid cell for statistical validity.
# 3. Subtract 1 from degraded count per cell (GPSJam's technique) so a
# single quirky transponder can't flag an entire zone.
# 4. Require the adjusted ratio to exceed the threshold.
try:
jamming_grid = {}
raw_flights = latest_data.get('flights', [])
raw_flights = raw_flights_snapshot
for rf in raw_flights:
rlat = rf.get('lat')
rlng = rf.get('lng') or rf.get('lon')
rlat = rf.get("lat")
rlng = rf.get("lng") or rf.get("lon")
if rlat is None or rlng is None:
continue
nacp = rf.get('nac_p')
if nacp is None:
nacp = rf.get("nac_p")
if nacp is None or nacp == 0:
continue
grid_key = f"{int(rlat)},{int(rlng)}"
if grid_key not in jamming_grid:
jamming_grid[grid_key] = {"degraded": 0, "total": 0}
jamming_grid[grid_key]["total"] += 1
if nacp < 8:
if nacp < GPS_JAMMING_NACP_THRESHOLD:
jamming_grid[grid_key]["degraded"] += 1
jamming_zones = []
for gk, counts in jamming_grid.items():
if counts["total"] < 3:
if counts["total"] < GPS_JAMMING_MIN_AIRCRAFT:
continue
ratio = counts["degraded"] / counts["total"]
if ratio > 0.25:
adjusted_degraded = max(counts["degraded"] - 1, 0)
if adjusted_degraded == 0:
continue
ratio = adjusted_degraded / counts["total"]
if ratio > GPS_JAMMING_MIN_RATIO:
lat_i, lng_i = gk.split(",")
severity = "low" if ratio < 0.5 else "medium" if ratio < 0.75 else "high"
jamming_zones.append({
"lat": int(lat_i) + 0.5,
"lng": int(lng_i) + 0.5,
"severity": severity,
"ratio": round(ratio, 2),
"degraded": counts["degraded"],
"total": counts["total"]
})
jamming_zones.append(
{
"lat": int(lat_i) + 0.5,
"lng": int(lng_i) + 0.5,
"severity": severity,
"ratio": round(ratio, 2),
"degraded": counts["degraded"],
"total": counts["total"],
}
)
with _data_lock:
latest_data['gps_jamming'] = jamming_zones
latest_data["gps_jamming"] = jamming_zones
if jamming_zones:
logger.info(f"GPS Jamming: {len(jamming_zones)} interference zones detected")
except (ValueError, TypeError, KeyError, ZeroDivisionError) as e:
logger.error(f"GPS Jamming detection error: {e}")
with _data_lock:
latest_data['gps_jamming'] = []
latest_data["gps_jamming"] = []
# --- Holding Pattern Detection ---
try:
holding_count = 0
all_flight_lists = [commercial, private_jets, private_ga,
latest_data.get('tracked_flights', []),
latest_data.get('military_flights', [])]
all_flight_lists = [
commercial,
private_jets,
private_ga,
tracked_snapshot,
military_snapshot,
]
with _trails_lock:
trails_snapshot = {k: v.get('points', [])[:] for k, v in flight_trails.items()}
trails_snapshot = {k: v.get("points", [])[:] for k, v in flight_trails.items()}
for flist in all_flight_lists:
for f in flist:
hex_id = f.get('icao24', '').lower()
hex_id = f.get("icao24", "").lower()
trail = trails_snapshot.get(hex_id, [])
if len(trail) < 6:
f['holding'] = False
f["holding"] = False
continue
pts = trail[-8:]
total_turn = 0.0
prev_bearing = 0.0
for i in range(1, len(pts)):
lat1, lng1 = math.radians(pts[i-1][0]), math.radians(pts[i-1][1])
lat1, lng1 = math.radians(pts[i - 1][0]), math.radians(pts[i - 1][1])
lat2, lng2 = math.radians(pts[i][0]), math.radians(pts[i][1])
dlng = lng2 - lng1
x = math.sin(dlng) * math.cos(lat2)
y = math.cos(lat1) * math.sin(lat2) - math.sin(lat1) * math.cos(lat2) * math.cos(dlng)
y = math.cos(lat1) * math.sin(lat2) - math.sin(lat1) * math.cos(
lat2
) * math.cos(dlng)
bearing = math.degrees(math.atan2(x, y)) % 360
if i > 1:
delta = abs(bearing - prev_bearing)
@@ -568,8 +821,8 @@ def _classify_and_publish(all_adsb_flights):
delta = 360 - delta
total_turn += delta
prev_bearing = bearing
f['holding'] = total_turn > 300
if f['holding']:
f["holding"] = total_turn > 300
if f["holding"]:
holding_count += 1
if holding_count:
logger.info(f"Holding patterns: {holding_count} aircraft circling")
@@ -577,7 +830,7 @@ def _classify_and_publish(all_adsb_flights):
logger.error(f"Holding pattern detection error: {e}")
with _data_lock:
latest_data['last_updated'] = datetime.utcnow().isoformat()
latest_data["last_updated"] = datetime.utcnow().isoformat()
def _fetch_adsb_lol_regions():
@@ -588,7 +841,7 @@ def _fetch_adsb_lol_regions():
{"lat": 35.0, "lon": 105.0, "dist": 2000},
{"lat": -25.0, "lon": 133.0, "dist": 2000},
{"lat": 0.0, "lon": 20.0, "dist": 2500},
{"lat": -15.0, "lon": -60.0, "dist": 2000}
{"lat": -15.0, "lon": -60.0, "dist": 2000},
]
def _fetch_region(r):
@@ -598,7 +851,15 @@ def _fetch_adsb_lol_regions():
if res.status_code == 200:
data = res.json()
return data.get("ac", [])
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as e:
logger.warning(f"Region fetch failed for lat={r['lat']}: {e}")
return []
@@ -632,9 +893,18 @@ def _enrich_with_opensky_and_supplemental(adsb_flights):
token = opensky_client.get_token()
if token:
opensky_regions = [
{"name": "Africa", "bbox": {"lamin": -35.0, "lomin": -20.0, "lamax": 38.0, "lomax": 55.0}},
{"name": "Asia", "bbox": {"lamin": 0.0, "lomin": 30.0, "lamax": 75.0, "lomax": 150.0}},
{"name": "South America", "bbox": {"lamin": -60.0, "lomin": -95.0, "lamax": 15.0, "lomax": -30.0}}
{
"name": "Africa",
"bbox": {"lamin": -35.0, "lomin": -20.0, "lamax": 38.0, "lomax": 55.0},
},
{
"name": "Asia",
"bbox": {"lamin": 0.0, "lomin": 30.0, "lamax": 75.0, "lomax": 150.0},
},
{
"name": "South America",
"bbox": {"lamin": -60.0, "lomin": -95.0, "lamax": 15.0, "lomax": -30.0},
},
]
new_opensky_flights = []
@@ -648,24 +918,38 @@ def _enrich_with_opensky_and_supplemental(adsb_flights):
if os_res.status_code == 200:
os_data = os_res.json()
states = os_data.get("states") or []
logger.info(f"OpenSky: Fetched {len(states)} states for {os_reg['name']}")
logger.info(
f"OpenSky: Fetched {len(states)} states for {os_reg['name']}"
)
for s in states:
new_opensky_flights.append({
"hex": s[0],
"flight": s[1].strip() if s[1] else "UNKNOWN",
"r": s[2],
"lon": s[5],
"lat": s[6],
"alt_baro": (s[7] * 3.28084) if s[7] else 0,
"track": s[10] or 0,
"gs": (s[9] * 1.94384) if s[9] else 0,
"t": "Unknown",
"is_opensky": True
})
new_opensky_flights.append(
{
"hex": s[0],
"flight": s[1].strip() if s[1] else "UNKNOWN",
"r": s[2],
"lon": s[5],
"lat": s[6],
"alt_baro": (s[7] * 3.28084) if s[7] else 0,
"track": s[10] or 0,
"gs": (s[9] * 1.94384) if s[9] else 0,
"t": "Unknown",
"is_opensky": True,
}
)
else:
logger.warning(f"OpenSky API {os_reg['name']} failed: {os_res.status_code}")
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as ex:
logger.warning(
f"OpenSky API {os_reg['name']} failed: {os_res.status_code}"
)
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as ex:
logger.error(f"OpenSky fetching error for {os_reg['name']}: {ex}")
cached_opensky_flights = new_opensky_flights
@@ -688,12 +972,21 @@ def _enrich_with_opensky_and_supplemental(adsb_flights):
seen_hex.add(h)
if gap_fill:
logger.info(f"Gap-fill: added {len(gap_fill)} aircraft to pipeline")
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
OSError,
) as e:
logger.warning(f"Supplemental source fetch failed (non-fatal): {e}")
# Re-publish with enriched data
if len(all_flights) > len(adsb_flights):
logger.info(f"Enrichment: {len(all_flights) - len(adsb_flights)} additional aircraft from OpenSky + supplemental")
logger.info(
f"Enrichment: {len(all_flights) - len(adsb_flights)} additional aircraft from OpenSky + supplemental"
)
_classify_and_publish(all_flights)
except Exception as e:
logger.error(f"OpenSky/supplemental enrichment error: {e}")
@@ -705,6 +998,10 @@ def fetch_flights():
Phase 1 (fast): Fetch adsb.lol → classify → publish immediately (~3-5s)
Phase 2 (background): Merge OpenSky + supplemental → re-publish (~15-30s)
"""
from services.fetchers._store import is_any_active
if not is_any_active("flights", "private", "jets", "tracked", "gps_jamming"):
return
try:
# Phase 1: adsb.lol — fast, parallel, publish immediately
adsb_flights = _fetch_adsb_lol_regions()
backend/services/fetchers/geo.py
+113 -28
View File
@@ -1,7 +1,9 @@
"""Ship and geopolitics fetchers — AIS vessels, carriers, frontlines, GDELT, LiveUAmap."""
"""Ship and geopolitics fetchers — AIS vessels, carriers, frontlines, GDELT, LiveUAmap, fishing."""
import csv
import io
import math
import os
import logging
from services.network_utils import fetch_with_curl
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
@@ -16,6 +18,12 @@ logger = logging.getLogger(__name__)
@with_retry(max_retries=1, base_delay=1)
def fetch_ships():
"""Fetch real-time AIS vessel data and combine with OSINT carrier positions."""
from services.fetchers._store import is_any_active
if not is_any_active(
"ships_military", "ships_cargo", "ships_civilian", "ships_passenger", "ships_tracked_yachts"
):
return
from services.ais_stream import get_ais_vessels
from services.carrier_tracker import get_carrier_positions
@@ -23,19 +31,20 @@ def fetch_ships():
try:
carriers = get_carrier_positions()
ships.extend(carriers)
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Carrier tracker error (non-fatal): {e}")
carriers = []
try:
ais_vessels = get_ais_vessels()
ships.extend(ais_vessels)
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"AIS stream error (non-fatal): {e}")
ais_vessels = []
# Enrich ships with yacht alert data (tracked superyachts)
from services.fetchers.yacht_alert import enrich_with_yacht_alert
for ship in ships:
enrich_with_yacht_alert(ship)
@@ -46,7 +55,7 @@ def fetch_ships():
logger.info(f"Ships: {len(carriers)} carriers + {len(ais_vessels)} AIS vessels")
with _data_lock:
latest_data['ships'] = ships
latest_data["ships"] = ships
_mark_fresh("ships")
@@ -62,16 +71,19 @@ def find_nearest_airport(lat, lng, max_distance_nm=200):
return None
best = None
best_dist = float('inf')
best_dist = float("inf")
lat_r = math.radians(lat)
lng_r = math.radians(lng)
for apt in cached_airports:
apt_lat_r = math.radians(apt['lat'])
apt_lng_r = math.radians(apt['lng'])
apt_lat_r = math.radians(apt["lat"])
apt_lng_r = math.radians(apt["lng"])
dlat = apt_lat_r - lat_r
dlng = apt_lng_r - lng_r
a = math.sin(dlat / 2) ** 2 + math.cos(lat_r) * math.cos(apt_lat_r) * math.sin(dlng / 2) ** 2
a = (
math.sin(dlat / 2) ** 2
+ math.cos(lat_r) * math.cos(apt_lat_r) * math.sin(dlng / 2) ** 2
)
c = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))
dist_nm = 3440.065 * c
@@ -81,9 +93,11 @@ def find_nearest_airport(lat, lng, max_distance_nm=200):
if best and best_dist <= max_distance_nm:
return {
"iata": best['iata'], "name": best['name'],
"lat": best['lat'], "lng": best['lng'],
"distance_nm": round(best_dist, 1)
"iata": best["iata"],
"name": best["name"],
"lat": best["lat"],
"lng": best["lng"],
"distance_nm": round(best_dist, 1),
}
return None
@@ -99,21 +113,23 @@ def fetch_airports():
f = io.StringIO(response.text)
reader = csv.DictReader(f)
for row in reader:
if row['type'] == 'large_airport' and row['iata_code']:
cached_airports.append({
"id": row['ident'],
"name": row['name'],
"iata": row['iata_code'],
"lat": float(row['latitude_deg']),
"lng": float(row['longitude_deg']),
"type": "airport"
})
if row["type"] == "large_airport" and row["iata_code"]:
cached_airports.append(
{
"id": row["ident"],
"name": row["name"],
"iata": row["iata_code"],
"lat": float(row["latitude_deg"]),
"lng": float(row["longitude_deg"]),
"type": "airport",
}
)
logger.info(f"Loaded {len(cached_airports)} large airports into cache.")
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching airports: {e}")
with _data_lock:
latest_data['airports'] = cached_airports
latest_data["airports"] = cached_airports
# ---------------------------------------------------------------------------
@@ -122,28 +138,38 @@ def fetch_airports():
@with_retry(max_retries=1, base_delay=2)
def fetch_frontlines():
"""Fetch Ukraine frontline data (fast — single GitHub API call)."""
from services.fetchers._store import is_any_active
if not is_any_active("ukraine_frontline"):
return
try:
from services.geopolitics import fetch_ukraine_frontlines
frontlines = fetch_ukraine_frontlines()
if frontlines:
with _data_lock:
latest_data['frontlines'] = frontlines
latest_data["frontlines"] = frontlines
_mark_fresh("frontlines")
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching frontlines: {e}")
@with_retry(max_retries=1, base_delay=3)
def fetch_gdelt():
"""Fetch GDELT global military incidents (slow — downloads 32 ZIP files)."""
from services.fetchers._store import is_any_active
if not is_any_active("global_incidents"):
return
try:
from services.geopolitics import fetch_global_military_incidents
gdelt = fetch_global_military_incidents()
if gdelt is not None:
with _data_lock:
latest_data['gdelt'] = gdelt
latest_data["gdelt"] = gdelt
_mark_fresh("gdelt")
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching GDELT: {e}")
@@ -154,13 +180,72 @@ def fetch_geopolitics():
def update_liveuamap():
from services.fetchers._store import is_any_active
if not is_any_active("global_incidents"):
return
logger.info("Running scheduled Liveuamap scraper...")
try:
from services.liveuamap_scraper import fetch_liveuamap
res = fetch_liveuamap()
if res:
with _data_lock:
latest_data['liveuamap'] = res
latest_data["liveuamap"] = res
_mark_fresh("liveuamap")
except Exception as e:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Liveuamap scraper error: {e}")
# ---------------------------------------------------------------------------
# Fishing Activity (Global Fishing Watch)
# ---------------------------------------------------------------------------
@with_retry(max_retries=1, base_delay=5)
def fetch_fishing_activity():
"""Fetch recent fishing events from Global Fishing Watch (~5 day lag)."""
from services.fetchers._store import is_any_active
if not is_any_active("fishing_activity"):
return
token = os.environ.get("GFW_API_TOKEN", "")
if not token:
logger.debug("GFW_API_TOKEN not set, skipping fishing activity fetch")
return
events = []
try:
url = (
"https://gateway.api.globalfishingwatch.org/v3/events"
"?datasets[0]=public-global-fishing-events:latest"
"&limit=500&sort=start&sort-direction=DESC"
)
headers = {"Authorization": f"Bearer {token}"}
response = fetch_with_curl(url, timeout=30, headers=headers)
if response.status_code == 200:
entries = response.json().get("entries", [])
for e in entries:
pos = e.get("position", {})
lat = pos.get("lat")
lng = pos.get("lon")
if lat is None or lng is None:
continue
dur = e.get("event", {}).get("duration", 0) or 0
events.append(
{
"id": e.get("id", ""),
"type": e.get("type", "fishing"),
"lat": lat,
"lng": lng,
"start": e.get("start", ""),
"end": e.get("end", ""),
"vessel_name": (e.get("vessel") or {}).get("name", "Unknown"),
"vessel_flag": (e.get("vessel") or {}).get("flag", ""),
"duration_hrs": round(dur / 3600, 1),
}
)
logger.info(f"Fishing activity: {len(events)} events")
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching fishing activity: {e}")
with _data_lock:
latest_data["fishing_activity"] = events
if events:
_mark_fresh("fishing_activity")
backend/services/fetchers/infrastructure.py
+492 -16
View File
@@ -1,4 +1,5 @@
"""Infrastructure fetchers — internet outages (IODA), data centers, CCTV, KiwiSDR."""
import json
import time
import heapq
@@ -25,6 +26,7 @@ def _geocode_region(region_name: str, country_name: str) -> tuple:
return _region_geocode_cache[cache_key]
try:
import urllib.parse
query = urllib.parse.quote(f"{region_name}, {country_name}")
url = f"https://nominatim.openstreetmap.org/search?q={query}&format=json&limit=1"
response = fetch_with_curl(url, timeout=8, headers={"User-Agent": "ShadowBroker-OSINT/1.0"})
@@ -35,7 +37,7 @@ def _geocode_region(region_name: str, country_name: str) -> tuple:
lon = float(results[0]["lon"])
_region_geocode_cache[cache_key] = (lat, lon)
return (lat, lon)
except Exception:
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError):
pass
_region_geocode_cache[cache_key] = None
return None
@@ -44,6 +46,10 @@ def _geocode_region(region_name: str, country_name: str) -> tuple:
@with_retry(max_retries=1, base_delay=1)
def fetch_internet_outages():
"""Fetch regional internet outage alerts from IODA (Georgia Tech)."""
from services.fetchers._store import is_any_active
if not is_any_active("internet_outages"):
return
RELIABLE_DATASOURCES = {"bgp", "ping-slash24"}
outages = []
try:
@@ -96,7 +102,15 @@ def fetch_internet_outages():
geocoded.append(r)
outages = heapq.nlargest(100, geocoded, key=lambda x: x["severity"])
logger.info(f"Internet outages: {len(outages)} regions affected")
except Exception as e:
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching internet outages: {e}")
with _data_lock:
latest_data["internet_outages"] = outages
@@ -104,6 +118,116 @@ def fetch_internet_outages():
_mark_fresh("internet_outages")
# ---------------------------------------------------------------------------
# RIPE Atlas — complement IODA with probe-level disconnection data
# ---------------------------------------------------------------------------
@with_retry(max_retries=1, base_delay=3)
def fetch_ripe_atlas_probes():
"""Fetch disconnected RIPE Atlas probes and merge into internet_outages (complementing IODA)."""
from services.fetchers._store import is_any_active
if not is_any_active("internet_outages"):
return
try:
# 1. Fetch disconnected probes (status=2) — ~2,000 probes, no auth needed
url_disc = "https://atlas.ripe.net/api/v2/probes/?status=2&page_size=500&format=json"
resp_disc = fetch_with_curl(url_disc, timeout=20)
if resp_disc.status_code != 200:
logger.warning(f"RIPE Atlas probes API returned {resp_disc.status_code}")
return
disc_data = resp_disc.json()
disconnected = disc_data.get("results", [])
# 2. Fetch connected probe count (page_size=1 — we only need the count)
url_conn = "https://atlas.ripe.net/api/v2/probes/?status=1&page_size=1&format=json"
resp_conn = fetch_with_curl(url_conn, timeout=10)
total_connected = 0
if resp_conn.status_code == 200:
total_connected = resp_conn.json().get("count", 0)
# 3. Group disconnected probes by country
country_disc: dict = {}
for p in disconnected:
cc = p.get("country_code", "")
if not cc:
continue
if cc not in country_disc:
country_disc[cc] = []
country_disc[cc].append(p)
# 4. Get IODA-covered countries to avoid double-reporting
with _data_lock:
ioda_outages = list(latest_data.get("internet_outages", []))
ioda_countries = {
o.get("country_code", "").upper()
for o in ioda_outages
if o.get("datasource") != "ripe-atlas"
}
# 5. Build RIPE-only alerts for countries NOT already in IODA
ripe_alerts = []
for cc, probes in country_disc.items():
if cc.upper() in ioda_countries:
continue # IODA already covers this country
if len(probes) < 3:
continue # Too few probes to be meaningful
# Use centroid of disconnected probes as marker location
lats = [
p["geometry"]["coordinates"][1]
for p in probes
if p.get("geometry") and p["geometry"].get("coordinates")
]
lngs = [
p["geometry"]["coordinates"][0]
for p in probes
if p.get("geometry") and p["geometry"].get("coordinates")
]
if not lats:
continue
disc_count = len(probes)
# Severity: scale 10-80 based on disconnected probe count
severity = min(80, 10 + disc_count * 2)
ripe_alerts.append({
"region_code": f"RIPE-{cc}",
"region_name": f"{cc} (Atlas probes)",
"country_code": cc,
"country_name": cc,
"level": "critical" if disc_count >= 10 else "warning",
"datasource": "ripe-atlas",
"severity": severity,
"lat": sum(lats) / len(lats),
"lng": sum(lngs) / len(lngs),
"probe_count": disc_count,
})
# 6. Merge into internet_outages — keep IODA entries, replace old RIPE entries
with _data_lock:
current = latest_data.get("internet_outages", [])
ioda_only = [o for o in current if o.get("datasource") != "ripe-atlas"]
latest_data["internet_outages"] = ioda_only + ripe_alerts
if ripe_alerts:
_mark_fresh("internet_outages")
logger.info(
f"RIPE Atlas: {len(ripe_alerts)} countries with probe disconnections "
f"(from {len(disconnected)} disconnected / ~{total_connected} connected probes)"
)
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching RIPE Atlas probes: {e}")
# ---------------------------------------------------------------------------
# Data Centers (local geocoded JSON)
# ---------------------------------------------------------------------------
@@ -112,6 +236,10 @@ _DC_GEOCODED_PATH = Path(__file__).parent.parent.parent / "data" / "datacenters_
def fetch_datacenters():
"""Load geocoded data centers (5K+ street-level precise locations)."""
from services.fetchers._store import is_any_active
if not is_any_active("datacenters"):
return
dcs = []
try:
if not _DC_GEOCODED_PATH.exists():
@@ -125,17 +253,28 @@ def fetch_datacenters():
continue
if not (-90 <= lat <= 90 and -180 <= lng <= 180):
continue
dcs.append({
"name": entry.get("name", "Unknown"),
"company": entry.get("company", ""),
"street": entry.get("street", ""),
"city": entry.get("city", ""),
"country": entry.get("country", ""),
"zip": entry.get("zip", ""),
"lat": lat, "lng": lng,
})
dcs.append(
{
"name": entry.get("name", "Unknown"),
"company": entry.get("company", ""),
"street": entry.get("street", ""),
"city": entry.get("city", ""),
"country": entry.get("country", ""),
"zip": entry.get("zip", ""),
"lat": lat,
"lng": lng,
}
)
logger.info(f"Data centers: {len(dcs)} geocoded locations loaded")
except Exception as e:
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error loading data centers: {e}")
with _data_lock:
latest_data["datacenters"] = dcs
@@ -222,16 +361,34 @@ def fetch_power_plants():
# CCTV Cameras
# ---------------------------------------------------------------------------
def fetch_cctv():
from services.fetchers._store import is_any_active
if not is_any_active("cctv"):
return
try:
from services.cctv_pipeline import get_all_cameras
cameras = get_all_cameras()
if len(cameras) < 500:
# Serve the current DB snapshot immediately and let the scheduled
# ingest cycle populate/refresh cameras asynchronously.
logger.info(
"CCTV DB currently has %d cameras — serving cached snapshot and waiting for scheduled ingest",
len(cameras),
)
with _data_lock:
latest_data["cctv"] = cameras
_mark_fresh("cctv")
except Exception as e:
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching cctv from DB: {e}")
with _data_lock:
latest_data["cctv"] = []
# ---------------------------------------------------------------------------
@@ -239,13 +396,332 @@ def fetch_cctv():
# ---------------------------------------------------------------------------
@with_retry(max_retries=2, base_delay=2)
def fetch_kiwisdr():
from services.fetchers._store import is_any_active
if not is_any_active("kiwisdr"):
return
try:
from services.kiwisdr_fetcher import fetch_kiwisdr_nodes
nodes = fetch_kiwisdr_nodes()
with _data_lock:
latest_data["kiwisdr"] = nodes
_mark_fresh("kiwisdr")
except Exception as e:
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching KiwiSDR nodes: {e}")
with _data_lock:
latest_data["kiwisdr"] = []
# ---------------------------------------------------------------------------
# SatNOGS Ground Stations + Observations
# ---------------------------------------------------------------------------
@with_retry(max_retries=2, base_delay=2)
def fetch_satnogs():
from services.fetchers._store import is_any_active
if not is_any_active("satnogs"):
return
try:
from services.satnogs_fetcher import fetch_satnogs_stations, fetch_satnogs_observations
stations = fetch_satnogs_stations()
obs = fetch_satnogs_observations()
with _data_lock:
latest_data["satnogs_stations"] = stations
latest_data["satnogs_observations"] = obs
_mark_fresh("satnogs_stations", "satnogs_observations")
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching SatNOGS: {e}")
# ---------------------------------------------------------------------------
# PSK Reporter — HF Digital Mode Spots
# ---------------------------------------------------------------------------
@with_retry(max_retries=2, base_delay=2)
def fetch_psk_reporter():
from services.fetchers._store import is_any_active
if not is_any_active("psk_reporter"):
return
try:
from services.psk_reporter_fetcher import fetch_psk_reporter_spots
spots = fetch_psk_reporter_spots()
with _data_lock:
latest_data["psk_reporter"] = spots
_mark_fresh("psk_reporter")
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching PSK Reporter: {e}")
with _data_lock:
latest_data["psk_reporter"] = []
# ---------------------------------------------------------------------------
# TinyGS LoRa Satellites
# ---------------------------------------------------------------------------
@with_retry(max_retries=2, base_delay=2)
def fetch_tinygs():
from services.fetchers._store import is_any_active
if not is_any_active("tinygs"):
return
try:
from services.tinygs_fetcher import fetch_tinygs_satellites
sats = fetch_tinygs_satellites()
with _data_lock:
latest_data["tinygs_satellites"] = sats
_mark_fresh("tinygs_satellites")
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching TinyGS: {e}")
# ---------------------------------------------------------------------------
# Police Scanners (OpenMHZ) — geocode city+state via local GeoNames DB
# ---------------------------------------------------------------------------
_scanner_geo_cache: dict = {} # city|state -> (lat, lng) — populated once from GeoNames
def _build_scanner_geo_lookup():
"""Build a US city/county→coords lookup from reverse_geocoder's bundled GeoNames CSV."""
if _scanner_geo_cache:
return
try:
import csv, os, reverse_geocoder as rg
geo_file = os.path.join(os.path.dirname(rg.__file__), "rg_cities1000.csv")
# US state abbreviation → admin1 name mapping
_abbr = {
"AL": "Alabama",
"AK": "Alaska",
"AZ": "Arizona",
"AR": "Arkansas",
"CA": "California",
"CO": "Colorado",
"CT": "Connecticut",
"DE": "Delaware",
"FL": "Florida",
"GA": "Georgia",
"HI": "Hawaii",
"ID": "Idaho",
"IL": "Illinois",
"IN": "Indiana",
"IA": "Iowa",
"KS": "Kansas",
"KY": "Kentucky",
"LA": "Louisiana",
"ME": "Maine",
"MD": "Maryland",
"MA": "Massachusetts",
"MI": "Michigan",
"MN": "Minnesota",
"MS": "Mississippi",
"MO": "Missouri",
"MT": "Montana",
"NE": "Nebraska",
"NV": "Nevada",
"NH": "New Hampshire",
"NJ": "New Jersey",
"NM": "New Mexico",
"NY": "New York",
"NC": "North Carolina",
"ND": "North Dakota",
"OH": "Ohio",
"OK": "Oklahoma",
"OR": "Oregon",
"PA": "Pennsylvania",
"RI": "Rhode Island",
"SC": "South Carolina",
"SD": "South Dakota",
"TN": "Tennessee",
"TX": "Texas",
"UT": "Utah",
"VT": "Vermont",
"VA": "Virginia",
"WA": "Washington",
"WV": "West Virginia",
"WI": "Wisconsin",
"WY": "Wyoming",
"DC": "Washington, D.C.",
}
state_full = {v.lower(): k for k, v in _abbr.items()}
state_full["washington, d.c."] = "DC"
county_coords = {} # admin2(county)|state -> (lat, lon) — first city per county
with open(geo_file, "r", encoding="utf-8") as f:
reader = csv.reader(f)
next(reader, None) # skip header
for row in reader:
if len(row) < 6 or row[5] != "US":
continue
lat_s, lon_s, name, admin1, admin2 = row[0], row[1], row[2], row[3], row[4]
st = state_full.get(admin1.lower(), "")
if not st:
continue
coords = (float(lat_s), float(lon_s))
# City name → coords
_scanner_geo_cache[f"{name.lower()}|{st}"] = coords
# County name → coords (keep first match per county, usually the largest city)
if admin2:
county_key = f"{admin2.lower()}|{st}"
if county_key not in county_coords:
county_coords[county_key] = coords
# Also strip " County" suffix for matching
stripped = admin2.lower().replace(" county", "").strip()
stripped_key = f"{stripped}|{st}"
if stripped_key not in county_coords:
county_coords[stripped_key] = coords
# Merge county lookups (don't override city entries)
for k, v in county_coords.items():
if k not in _scanner_geo_cache:
_scanner_geo_cache[k] = v
# Special case: DC
_scanner_geo_cache["washington|DC"] = (38.89511, -77.03637)
logger.info(f"Scanner geo lookup: {len(_scanner_geo_cache)} US entries loaded")
except Exception as e:
logger.warning(f"Failed to build scanner geo lookup: {e}")
def _geocode_scanner(city: str, state: str):
"""Look up city+state coordinates from local GeoNames cache."""
_build_scanner_geo_lookup()
if not city or not state:
return None
st = state.upper()
# Strip trailing state from city (e.g. "Lehigh, PA")
c = city.strip()
if ", " in c:
parts = c.rsplit(", ", 1)
if len(parts[1]) <= 2:
c = parts[0]
name = c.lower()
# Try exact city match
result = _scanner_geo_cache.get(f"{name}|{st}")
if result:
return result
# Strip "County" / "Co" suffix
stripped = name.replace(" county", "").replace(" co", "").strip()
result = _scanner_geo_cache.get(f"{stripped}|{st}")
if result:
return result
# Normalize "St." / "St" → "Saint"
import re
normed = re.sub(r"\bst\.?\s", "saint ", name)
if normed != name:
result = _scanner_geo_cache.get(f"{normed}|{st}")
if result:
return result
# Also try with "s" suffix: "St. Marys" → "Saint Marys" and "Saint Mary's"
for variant in [normed.rstrip("s"), normed.replace("ys", "y's")]:
result = _scanner_geo_cache.get(f"{variant}|{st}")
if result:
return result
# "Prince Georges" → "Prince George's" (apostrophe variants)
if "georges" in name:
key = name.replace("georges", "george's") + "|" + st
result = _scanner_geo_cache.get(key)
if result:
return result
# Multi-location: "Scott and Carver" → try first part
if " and " in name:
first = name.split(" and ")[0].strip()
result = _scanner_geo_cache.get(f"{first}|{st}")
if result:
return result
# Comma-separated list: "Adams, Jackson, Juneau" → try first
if ", " in name:
first = name.split(", ")[0].strip()
result = _scanner_geo_cache.get(f"{first}|{st}")
if result:
return result
# Drop directional prefix: "North Fulton" → "Fulton"
for prefix in ("north ", "south ", "east ", "west "):
if name.startswith(prefix):
result = _scanner_geo_cache.get(f"{name[len(prefix):]}|{st}")
if result:
return result
return None
@with_retry(max_retries=2, base_delay=2)
def fetch_scanners():
from services.fetchers._store import is_any_active
if not is_any_active("scanners"):
return
try:
from services.radio_intercept import get_openmhz_systems
systems = get_openmhz_systems()
scanners = []
for s in systems:
city = s.get("city", "") or s.get("county", "") or ""
state = s.get("state", "")
coords = _geocode_scanner(city, state)
if not coords:
continue
lat, lng = coords
scanners.append(
{
"shortName": s.get("shortName", ""),
"name": s.get("name", "Unknown Scanner"),
"lat": round(lat, 5),
"lng": round(lng, 5),
"city": city,
"state": state,
"clientCount": s.get("clientCount", 0),
"description": s.get("description", ""),
}
)
with _data_lock:
latest_data["scanners"] = scanners
if scanners:
_mark_fresh("scanners")
logger.info(f"Scanners: {len(scanners)}/{len(systems)} geocoded")
except (
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
TypeError,
json.JSONDecodeError,
) as e:
logger.error(f"Error fetching scanners: {e}")
with _data_lock:
latest_data["scanners"] = []
backend/services/fetchers/meshtastic_map.py
+222
View File
@@ -0,0 +1,222 @@
"""Meshtastic Map fetcher — pulls global node positions from meshtastic.liamcottle.net.
Bootstrap + top-up strategy:
- On startup: fetch all nodes with positions to seed the map
- Every 4 hours: refresh from the API
- Persists to JSON cache so data survives restarts
- MQTT bridge provides real-time updates between API fetches
API source: https://meshtastic.liamcottle.net/api/v1/nodes (community project by Liam Cottle)
Polling interval deliberately kept low (4h) to be respectful to the service.
"""
import json
import logging
import time
from datetime import datetime, timezone, timedelta
from pathlib import Path
import requests
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
logger = logging.getLogger("services.data_fetcher")
_API_URL = "https://meshtastic.liamcottle.net/api/v1/nodes"
_CACHE_FILE = Path(__file__).resolve().parent.parent.parent / "data" / "meshtastic_nodes_cache.json"
_FETCH_TIMEOUT = 90 # seconds — response is ~37MB, needs time on slow connections
_MAX_AGE_HOURS = 4 # discard nodes not seen within this window (matches refresh interval)
# Track when we last fetched so the frontend can show staleness
_last_fetch_ts: float = 0.0
def _parse_node(node: dict) -> dict | None:
"""Convert an API node into a slim signal-like dict."""
lat_i = node.get("latitude")
lng_i = node.get("longitude")
if lat_i is None or lng_i is None:
return None
lat = lat_i / 1e7
lng = lng_i / 1e7
# Basic validity
if not (-90 <= lat <= 90 and -180 <= lng <= 180):
return None
if abs(lat) < 0.1 and abs(lng) < 0.1:
return None
callsign = node.get("node_id_hex", "")
if not callsign:
nid = node.get("node_id")
callsign = f"!{int(nid):08x}" if nid else ""
if not callsign:
return None
# Position age from API — reject nodes older than _MAX_AGE_HOURS
pos_updated = node.get("position_updated_at") or node.get("updated_at", "")
if pos_updated:
try:
ts = datetime.fromisoformat(pos_updated.replace("Z", "+00:00"))
if datetime.now(timezone.utc) - ts > timedelta(hours=_MAX_AGE_HOURS):
return None
except (ValueError, TypeError):
pass
else:
return None # no timestamp at all — skip
return {
"callsign": callsign[:20],
"lat": round(lat, 5),
"lng": round(lng, 5),
"source": "meshtastic",
"confidence": 0.5,
"timestamp": pos_updated,
"position_updated_at": pos_updated,
"from_api": True,
"long_name": (node.get("long_name") or "")[:40],
"short_name": (node.get("short_name") or "")[:4],
"hardware": node.get("hardware_model_name", ""),
"role": node.get("role_name", ""),
"battery_level": node.get("battery_level"),
"voltage": node.get("voltage"),
"altitude": node.get("altitude"),
}
def _is_fresh(node: dict) -> bool:
"""Check if a cached node is still within the _MAX_AGE_HOURS window."""
ts_str = node.get("position_updated_at") or node.get("timestamp", "")
if not ts_str:
return False
try:
ts = datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
return datetime.now(timezone.utc) - ts <= timedelta(hours=_MAX_AGE_HOURS)
except (ValueError, TypeError):
return False
def _load_cache() -> list[dict]:
"""Load cached nodes from disk, filtering out stale entries."""
if _CACHE_FILE.exists():
try:
data = json.loads(_CACHE_FILE.read_text(encoding="utf-8"))
nodes = data.get("nodes", [])
fresh = [n for n in nodes if _is_fresh(n)]
logger.info(f"Meshtastic map cache loaded: {len(fresh)} fresh / {len(nodes)} total")
return fresh
except Exception as e:
logger.warning(f"Failed to load meshtastic cache: {e}")
return []
def _save_cache(nodes: list[dict], fetch_ts: float):
"""Persist processed nodes to disk."""
try:
_CACHE_FILE.parent.mkdir(parents=True, exist_ok=True)
_CACHE_FILE.write_text(
json.dumps(
{
"fetched_at": fetch_ts,
"count": len(nodes),
"nodes": nodes,
}
),
encoding="utf-8",
)
except Exception as e:
logger.warning(f"Failed to save meshtastic cache: {e}")
def fetch_meshtastic_nodes():
"""Fetch global Meshtastic node positions from Liam Cottle's map API.
Stores processed nodes in latest_data["meshtastic_map_nodes"].
Persists to JSON cache for restart resilience.
"""
from services.fetchers._store import is_any_active
if not is_any_active("sigint_meshtastic"):
return
global _last_fetch_ts
try:
logger.info("Fetching Meshtastic map nodes from API...")
resp = requests.get(
_API_URL,
timeout=_FETCH_TIMEOUT,
headers={
"User-Agent": "ShadowBroker/1.0 (OSINT dashboard, 4h polling)",
"Accept": "application/json",
},
)
resp.raise_for_status()
raw = resp.json()
raw_nodes = raw.get("nodes", []) if isinstance(raw, dict) else raw
# Parse and filter to only nodes with valid positions
parsed = []
for node in raw_nodes:
sig = _parse_node(node)
if sig:
parsed.append(sig)
_last_fetch_ts = time.time()
_save_cache(parsed, _last_fetch_ts)
with _data_lock:
latest_data["meshtastic_map_nodes"] = parsed
latest_data["meshtastic_map_fetched_at"] = _last_fetch_ts
try:
from services.fetchers.sigint import refresh_sigint_snapshot
refresh_sigint_snapshot()
except Exception as exc:
logger.debug(f"Meshtastic map: SIGINT snapshot refresh skipped: {exc}")
logger.info(
f"Meshtastic map: {len(parsed)} nodes with positions " f"(from {len(raw_nodes)} total)"
)
except Exception as e:
logger.error(f"Meshtastic map fetch failed: {e}")
# Fall back to cache if available and we have nothing in memory
with _data_lock:
if not latest_data.get("meshtastic_map_nodes"):
cached = _load_cache()
if cached:
latest_data["meshtastic_map_nodes"] = cached
latest_data["meshtastic_map_fetched_at"] = (
_CACHE_FILE.stat().st_mtime if _CACHE_FILE.exists() else 0
)
logger.info(
f"Meshtastic map: using {len(cached)} cached nodes (API unavailable)"
)
try:
from services.fetchers.sigint import refresh_sigint_snapshot
refresh_sigint_snapshot()
except Exception as exc:
logger.debug(f"Meshtastic map cache: SIGINT snapshot refresh skipped: {exc}")
_mark_fresh("meshtastic_map")
def load_meshtastic_cache_if_available():
"""On startup, load cached nodes immediately (before first API fetch)."""
global _last_fetch_ts
cached = _load_cache()
if cached:
with _data_lock:
latest_data["meshtastic_map_nodes"] = cached
_last_fetch_ts = _CACHE_FILE.stat().st_mtime if _CACHE_FILE.exists() else 0
latest_data["meshtastic_map_fetched_at"] = _last_fetch_ts
try:
from services.fetchers.sigint import refresh_sigint_snapshot
refresh_sigint_snapshot()
except Exception as exc:
logger.debug(f"Meshtastic preload: SIGINT snapshot refresh skipped: {exc}")
logger.info(f"Meshtastic map: preloaded {len(cached)} nodes from cache")
backend/services/fetchers/military.py
+64 -14
View File
@@ -1,4 +1,5 @@
"""Military flight tracking and UAV detection from ADS-B data."""
import json
import logging
import requests
@@ -13,7 +14,21 @@ logger = logging.getLogger("services.data_fetcher")
# ---------------------------------------------------------------------------
_UAV_TYPE_CODES = {"Q9", "R4", "TB2", "MALE", "HALE", "HERM", "HRON"}
_UAV_CALLSIGN_PREFIXES = ("FORTE", "GHAWK", "REAP", "BAMS", "UAV", "UAS")
_UAV_MODEL_KEYWORDS = ("RQ-", "MQ-", "RQ4", "MQ9", "MQ4", "MQ1", "REAPER", "GLOBALHAWK", "TRITON", "PREDATOR", "HERMES", "HERON", "BAYRAKTAR")
_UAV_MODEL_KEYWORDS = (
"RQ-",
"MQ-",
"RQ4",
"MQ9",
"MQ4",
"MQ1",
"REAPER",
"GLOBALHAWK",
"TRITON",
"PREDATOR",
"HERMES",
"HERON",
"BAYRAKTAR",
)
_UAV_WIKI = {
"RQ4": "https://en.wikipedia.org/wiki/Northrop_Grumman_RQ-4_Global_Hawk",
"RQ-4": "https://en.wikipedia.org/wiki/Northrop_Grumman_RQ-4_Global_Hawk",
@@ -137,13 +152,41 @@ def _classify_uav(model: str, callsign: str):
def fetch_military_flights():
from services.fetchers._store import is_any_active
if not is_any_active("military"):
return
military_flights = []
detected_uavs = []
# Fetch from primary + supplemental military endpoints
all_mil_ac = []
seen_hex = set()
try:
url = "https://api.adsb.lol/v2/mil"
response = fetch_with_curl(url, timeout=10)
if response.status_code == 200:
ac = response.json().get('ac', [])
for a in response.json().get("ac", []):
h = a.get("hex", "").lower()
if h and h not in seen_hex:
seen_hex.add(h)
all_mil_ac.append(a)
except Exception as e:
logger.warning(f"adsb.lol mil fetch failed: {e}")
# Supplemental: airplanes.live military endpoint
try:
resp2 = fetch_with_curl("https://api.airplanes.live/v2/mil", timeout=10)
if resp2.status_code == 200:
for a in resp2.json().get("ac", []):
h = a.get("hex", "").lower()
if h and h not in seen_hex:
seen_hex.add(h)
all_mil_ac.append(a)
logger.info(f"airplanes.live mil: +{len(resp2.json().get('ac', []))} raw, {len(all_mil_ac)} total unique")
except Exception as e:
logger.debug(f"airplanes.live mil supplemental failed: {e}")
try:
if all_mil_ac:
ac = all_mil_ac
for f in ac:
try:
lat = f.get("lat")
@@ -218,18 +261,25 @@ def fetch_military_flights():
except Exception as loop_e:
logger.error(f"Mil flight interpolation error: {loop_e}")
continue
except Exception as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
OSError,
ValueError,
KeyError,
) as e:
logger.error(f"Error fetching military flights: {e}")
if not military_flights and not detected_uavs:
logger.warning("No military flights retrieved — keeping previous data if available")
with _data_lock:
if latest_data.get('military_flights'):
if latest_data.get("military_flights"):
return
with _data_lock:
latest_data['military_flights'] = military_flights
latest_data['uavs'] = detected_uavs
latest_data["military_flights"] = military_flights
latest_data["uavs"] = detected_uavs
_mark_fresh("military_flights", "uavs")
logger.info(f"UAVs: {len(detected_uavs)} real drones detected via ADS-B")
@@ -238,30 +288,30 @@ def fetch_military_flights():
remaining_mil = []
for mf in military_flights:
enrich_with_plane_alert(mf)
if mf.get('alert_category'):
mf['type'] = 'tracked_flight'
if mf.get("alert_category"):
mf["type"] = "tracked_flight"
tracked_mil.append(mf)
else:
remaining_mil.append(mf)
with _data_lock:
latest_data['military_flights'] = remaining_mil
latest_data["military_flights"] = remaining_mil
# Store tracked military flights — update positions for existing entries
with _data_lock:
existing_tracked = list(latest_data.get('tracked_flights', []))
existing_tracked = list(latest_data.get("tracked_flights", []))
fresh_mil_map = {}
for t in tracked_mil:
icao = t.get('icao24', '').upper()
icao = t.get("icao24", "").upper()
if icao:
fresh_mil_map[icao] = t
updated_tracked = []
seen_icaos = set()
for old_t in existing_tracked:
icao = old_t.get('icao24', '').upper()
icao = old_t.get("icao24", "").upper()
if icao in fresh_mil_map:
fresh = fresh_mil_map[icao]
for key in ('alert_category', 'alert_operator', 'alert_special', 'alert_flag'):
for key in ("alert_category", "alert_operator", "alert_special", "alert_flag"):
if key in old_t and key not in fresh:
fresh[key] = old_t[key]
updated_tracked.append(fresh)
@@ -273,5 +323,5 @@ def fetch_military_flights():
if icao not in seen_icaos:
updated_tracked.append(t)
with _data_lock:
latest_data['tracked_flights'] = updated_tracked
latest_data["tracked_flights"] = updated_tracked
logger.info(f"Tracked flights: {len(updated_tracked)} total ({len(tracked_mil)} from military)")
backend/services/fetchers/news.py
+40 -2
View File
@@ -7,6 +7,7 @@ import feedparser
from services.network_utils import fetch_with_curl
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
from services.fetchers.retry import with_retry
from services.oracle_service import enrich_news_items, compute_global_threat_level, detect_breaking_events
logger = logging.getLogger("services.data_fetcher")
@@ -170,7 +171,7 @@ def fetch_news():
logger.warning(f"Feed {source_name} failed: {e}")
return source_name, None
with concurrent.futures.ThreadPoolExecutor(max_workers=len(feeds)) as pool:
with concurrent.futures.ThreadPoolExecutor(max_workers=min(len(feeds), 6)) as pool:
feed_results = list(pool.map(_fetch_feed, feeds.items()))
for source_name, feed in feed_results:
@@ -191,7 +192,14 @@ def fetch_news():
elif alert_level == "Orange": risk_score = 7
else: risk_score = 4
else:
risk_keywords = ['war', 'missile', 'strike', 'attack', 'crisis', 'tension', 'military', 'conflict', 'defense', 'clash', 'nuclear']
risk_keywords = [
'war', 'missile', 'strike', 'attack', 'crisis', 'tension',
'military', 'conflict', 'defense', 'clash', 'nuclear',
'sanctions', 'ceasefire', 'invasion', 'drone', 'artillery',
'blockade', 'escalation', 'casualties', 'airspace',
'mobilization', 'proxy', 'insurgent', 'coup',
'assassination', 'bioweapon', 'chemical',
]
text = (title + " " + summary).lower()
risk_score = 1
@@ -268,6 +276,36 @@ def fetch_news():
})
news_items.sort(key=lambda x: x['risk_score'], reverse=True)
# Oracle enrichment: sentiment, oracle scores, prediction market odds
try:
with _data_lock:
markets = list(latest_data.get("prediction_markets", []))
enrich_news_items(news_items, source_weights, markets)
detect_breaking_events(news_items)
except Exception as e:
logger.warning(f"Oracle enrichment failed (news still usable): {e}")
# Global threat level computation (fuses news + markets + military + jamming)
try:
with _data_lock:
markets = list(latest_data.get("prediction_markets", []))
mil_flights = list(latest_data.get("military_flights", []))
jam_zones = list(latest_data.get("gps_jamming", []))
ships = list(latest_data.get("ships", []))
corr_alerts = list(latest_data.get("correlations", []))
threat_level = compute_global_threat_level(
news_items, markets,
military_flights=mil_flights,
gps_jamming=jam_zones,
ships=ships,
correlations=corr_alerts,
)
except Exception as e:
logger.warning(f"Threat level computation failed: {e}")
threat_level = {"score": 0, "level": "GREEN", "color": "#22c55e", "drivers": []}
with _data_lock:
latest_data['news'] = news_items
latest_data['threat_level'] = threat_level
_mark_fresh("news")
backend/services/fetchers/plane_alert.py
+159 -20
View File
@@ -1,4 +1,5 @@
"""Plane-Alert DB — load and enrich aircraft with tracked metadata."""
import os
import json
import logging
@@ -71,38 +72,126 @@ _CATEGORY_COLOR: dict[str, str] = {
"Radiohead": "purple",
}
def _category_to_color(cat: str) -> str:
"""O(1) exact lookup. Unknown categories default to purple."""
return _CATEGORY_COLOR.get(cat, "purple")
_PLANE_ALERT_DB: dict = {}
# ---------------------------------------------------------------------------
# POTUS Fleet — override colors and operator names for presidential aircraft.
# ---------------------------------------------------------------------------
_POTUS_FLEET: dict[str, dict] = {
"ADFDF8": {"color": "#ff1493", "operator": "Air Force One (82-8000)", "category": "Head of State", "wiki": "Air_Force_One", "fleet": "AF1"},
"ADFDF9": {"color": "#ff1493", "operator": "Air Force One (92-9000)", "category": "Head of State", "wiki": "Air_Force_One", "fleet": "AF1"},
"ADFEB7": {"color": "blue", "operator": "Air Force Two (98-0001)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"ADFEB8": {"color": "blue", "operator": "Air Force Two (98-0002)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"ADFEB9": {"color": "blue", "operator": "Air Force Two (99-0003)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"ADFEBA": {"color": "blue", "operator": "Air Force Two (99-0004)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"AE4AE6": {"color": "blue", "operator": "Air Force Two (09-0015)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"AE4AE8": {"color": "blue", "operator": "Air Force Two (09-0016)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"AE4AEA": {"color": "blue", "operator": "Air Force Two (09-0017)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"AE4AEC": {"color": "blue", "operator": "Air Force Two (19-0018)", "category": "Governments", "wiki": "Air_Force_Two", "fleet": "AF2"},
"AE0865": {"color": "#ff1493", "operator": "Marine One (VH-3D)", "category": "Head of State", "wiki": "Marine_One", "fleet": "M1"},
"AE5E76": {"color": "#ff1493", "operator": "Marine One (VH-92A)", "category": "Head of State", "wiki": "Marine_One", "fleet": "M1"},
"AE5E77": {"color": "#ff1493", "operator": "Marine One (VH-92A)", "category": "Head of State", "wiki": "Marine_One", "fleet": "M1"},
"AE5E79": {"color": "#ff1493", "operator": "Marine One (VH-92A)", "category": "Head of State", "wiki": "Marine_One", "fleet": "M1"},
"ADFDF8": {
"color": "#ff1493",
"operator": "Air Force One (82-8000)",
"category": "Head of State",
"wiki": "Air_Force_One",
"fleet": "AF1",
},
"ADFDF9": {
"color": "#ff1493",
"operator": "Air Force One (92-9000)",
"category": "Head of State",
"wiki": "Air_Force_One",
"fleet": "AF1",
},
"ADFEB7": {
"color": "blue",
"operator": "Air Force Two (98-0001)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"ADFEB8": {
"color": "blue",
"operator": "Air Force Two (98-0002)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"ADFEB9": {
"color": "blue",
"operator": "Air Force Two (99-0003)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"ADFEBA": {
"color": "blue",
"operator": "Air Force Two (99-0004)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"AE4AE6": {
"color": "blue",
"operator": "Air Force Two (09-0015)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"AE4AE8": {
"color": "blue",
"operator": "Air Force Two (09-0016)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"AE4AEA": {
"color": "blue",
"operator": "Air Force Two (09-0017)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"AE4AEC": {
"color": "blue",
"operator": "Air Force Two (19-0018)",
"category": "Governments",
"wiki": "Air_Force_Two",
"fleet": "AF2",
},
"AE0865": {
"color": "#ff1493",
"operator": "Marine One (VH-3D)",
"category": "Head of State",
"wiki": "Marine_One",
"fleet": "M1",
},
"AE5E76": {
"color": "#ff1493",
"operator": "Marine One (VH-92A)",
"category": "Head of State",
"wiki": "Marine_One",
"fleet": "M1",
},
"AE5E77": {
"color": "#ff1493",
"operator": "Marine One (VH-92A)",
"category": "Head of State",
"wiki": "Marine_One",
"fleet": "M1",
},
"AE5E79": {
"color": "#ff1493",
"operator": "Marine One (VH-92A)",
"category": "Head of State",
"wiki": "Marine_One",
"fleet": "M1",
},
}
def _load_plane_alert_db():
"""Load plane_alert_db.json (exported from SQLite) into memory."""
global _PLANE_ALERT_DB
json_path = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))),
"data", "plane_alert_db.json"
"data",
"plane_alert_db.json",
)
if not os.path.exists(json_path):
logger.warning(f"Plane-Alert DB not found at {json_path}")
@@ -124,8 +213,10 @@ def _load_plane_alert_db():
except (IOError, OSError, json.JSONDecodeError, ValueError, KeyError) as e:
logger.error(f"Failed to load Plane-Alert DB: {e}")
_load_plane_alert_db()
def enrich_with_plane_alert(flight: dict) -> dict:
"""If flight's icao24 is in the Plane-Alert DB, add alert metadata."""
icao = flight.get("icao24", "").strip().upper()
@@ -145,13 +236,16 @@ def enrich_with_plane_alert(flight: dict) -> dict:
flight["registration"] = info["registration"]
return flight
_TRACKED_NAMES_DB: dict = {}
def _load_tracked_names():
global _TRACKED_NAMES_DB
json_path = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))),
"data", "tracked_names.json"
"data",
"tracked_names.json",
)
if not os.path.exists(json_path):
return
@@ -160,16 +254,22 @@ def _load_tracked_names():
data = json.load(f)
for name, info in data.get("details", {}).items():
cat = info.get("category", "Other")
socials = info.get("socials")
for reg in info.get("registrations", []):
reg_clean = reg.strip().upper()
if reg_clean:
_TRACKED_NAMES_DB[reg_clean] = {"name": name, "category": cat}
entry = {"name": name, "category": cat}
if socials:
entry["socials"] = socials
_TRACKED_NAMES_DB[reg_clean] = entry
logger.info(f"Tracked Names DB loaded: {len(_TRACKED_NAMES_DB)} registrations")
except (IOError, OSError, json.JSONDecodeError, ValueError, KeyError) as e:
logger.error(f"Failed to load Tracked Names DB: {e}")
_load_tracked_names()
def enrich_with_tracked_names(flight: dict) -> dict:
"""If flight's registration matches our Excel extraction, tag it as tracked."""
icao = flight.get("icao24", "").strip().upper()
@@ -189,11 +289,50 @@ def enrich_with_tracked_names(flight: dict) -> dict:
name = match["name"]
flight["alert_operator"] = name
flight["alert_category"] = match["category"]
if match.get("socials"):
flight["alert_socials"] = match["socials"]
name_lower = name.lower()
is_gov = any(w in name_lower for w in ['state of ', 'government', 'republic', 'ministry', 'department', 'federal', 'cia'])
is_law = any(w in name_lower for w in ['police', 'marshal', 'sheriff', 'douane', 'customs', 'patrol', 'gendarmerie', 'guardia', 'law enforcement'])
is_med = any(w in name_lower for w in ['fire', 'bomberos', 'ambulance', 'paramedic', 'medevac', 'rescue', 'hospital', 'medical', 'lifeflight'])
is_gov = any(
w in name_lower
for w in [
"state of ",
"government",
"republic",
"ministry",
"department",
"federal",
"cia",
]
)
is_law = any(
w in name_lower
for w in [
"police",
"marshal",
"sheriff",
"douane",
"customs",
"patrol",
"gendarmerie",
"guardia",
"law enforcement",
]
)
is_med = any(
w in name_lower
for w in [
"fire",
"bomberos",
"ambulance",
"paramedic",
"medevac",
"rescue",
"hospital",
"medical",
"lifeflight",
]
)
if is_gov or is_law:
flight["alert_color"] = "blue"
backend/services/fetchers/prediction_markets.py
+647
View File
@@ -0,0 +1,647 @@
"""Prediction market fetcher — Polymarket (Gamma API) + Kalshi.
Fetches active prediction market events from both platforms, merges them by
topic similarity, classifies into categories, and stores merged odds with
full metadata (volume, end dates, descriptions, source badges).
"""
import json
import logging
import math
from cachetools import TTLCache, cached
logger = logging.getLogger("services.data_fetcher")
_market_cache = TTLCache(maxsize=1, ttl=60) # 60-second TTL — markets change fast
# Delta tracking: {market_title: previous_consensus_pct}
_prev_probabilities: dict[str, float] = {}
def _finite_or_none(value):
try:
n = float(value)
except (TypeError, ValueError):
return None
return n if math.isfinite(n) else None
# ---------------------------------------------------------------------------
# Category classification
# ---------------------------------------------------------------------------
CATEGORIES = ["POLITICS", "CONFLICT", "NEWS", "FINANCE", "CRYPTO"]
_KALSHI_CATEGORY_MAP = {
"Politics": "POLITICS",
"World": "NEWS",
"Economics": "FINANCE",
"Financials": "FINANCE",
"Tech": "FINANCE",
"Science": "NEWS",
"Climate and Weather": "NEWS",
"Sports": "NEWS",
"Culture": "NEWS",
}
_TAG_CATEGORY_MAP = {
"Politics": "POLITICS",
"Elections": "POLITICS",
"US Politics": "POLITICS",
"Trump": "POLITICS",
"Congress": "POLITICS",
"Supreme Court": "POLITICS",
"Geopolitics": "CONFLICT",
"War": "CONFLICT",
"Military": "CONFLICT",
"Finance": "FINANCE",
"Stocks": "FINANCE",
"Economy": "FINANCE",
"Business": "FINANCE",
"IPOs": "FINANCE",
"Crypto": "CRYPTO",
"Bitcoin": "CRYPTO",
"Ethereum": "CRYPTO",
"AI": "NEWS",
"Science": "NEWS",
"Sports": "NEWS",
"Culture": "NEWS",
"Entertainment": "NEWS",
"Tech": "FINANCE",
}
_KEYWORD_CATEGORIES = {
"CONFLICT": [
"war",
"military",
"attack",
"missile",
"invasion",
"ukraine",
"russia",
"gaza",
"israel",
"nato",
"troops",
"bombing",
"nuclear",
"sanctions",
"ceasefire",
"houthi",
"iran",
"china taiwan",
"clash",
"conflict",
"strike",
"weapon",
],
"POLITICS": [
"trump",
"biden",
"election",
"congress",
"senate",
"governor",
"president",
"democrat",
"republican",
"vote",
"party",
"cabinet",
"impeach",
"legislation",
"scotus",
"poll",
"vance",
"speaker",
"parliament",
"prime minister",
"macron",
"starmer",
],
"CRYPTO": [
"bitcoin",
"btc",
"ethereum",
"eth",
"crypto",
"blockchain",
"solana",
"defi",
"nft",
"binance",
"coinbase",
"token",
"microstrategy",
"stablecoin",
],
"FINANCE": [
"stock",
"fed",
"interest rate",
"inflation",
"gdp",
"recession",
"s&p",
"nasdaq",
"dow",
"oil",
"gold",
"treasury",
"tariff",
"ipo",
"earnings",
"market cap",
"revenue",
],
}
def _classify_category(title: str, poly_tags: list[str], kalshi_category: str) -> str:
"""Classify a market into one of the 5 categories."""
# 1. Kalshi native category
if kalshi_category:
mapped = _KALSHI_CATEGORY_MAP.get(kalshi_category)
if mapped:
return mapped
# 2. Polymarket tag labels
for tag in poly_tags:
mapped = _TAG_CATEGORY_MAP.get(tag)
if mapped:
return mapped
# 3. Keyword matching
title_lower = title.lower()
for cat, keywords in _KEYWORD_CATEGORIES.items():
for kw in keywords:
if kw in title_lower:
return cat
# 4. Default
return "NEWS"
# ---------------------------------------------------------------------------
# Polymarket
# ---------------------------------------------------------------------------
def _fetch_polymarket_events() -> list[dict]:
"""Fetch active events from Polymarket Gamma API (no auth required).
Fetches up to 500 events (multiple pages) for better search coverage.
"""
from services.network_utils import fetch_with_curl
all_events = []
for offset in range(0, 500, 100):
try:
resp = fetch_with_curl(
f"https://gamma-api.polymarket.com/events?active=true&closed=false&limit=100&offset={offset}",
timeout=15,
)
if not resp or resp.status_code != 200:
break
page = resp.json()
if not isinstance(page, list) or not page:
break
all_events.extend(page)
except Exception as e:
logger.warning(f"Polymarket page offset={offset} error: {e}")
break
if not all_events:
return []
try:
results = []
for ev in all_events:
title = ev.get("title", "")
if not title:
continue
# Extract best probability + outcomes from markets
markets = ev.get("markets", [])
best_pct = None
total_volume = 0
outcomes = []
for m in markets:
# Use outcomePrices[0] (Yes price) when available — lastTradePrice
# can be for either Yes or No side, causing "99%" for unlikely events
raw_op = m.get("outcomePrices")
price = None
try:
op = json.loads(raw_op) if isinstance(raw_op, str) else raw_op
if isinstance(op, list) and len(op) >= 1:
price = _finite_or_none(op[0])
except (json.JSONDecodeError, ValueError, TypeError):
pass
if price is None:
price = _finite_or_none(m.get("lastTradePrice") or m.get("bestBid"))
pct = None
if price is not None:
try:
pct = round(price * 100, 1)
if best_pct is None or pct > best_pct:
best_pct = pct
except (ValueError, TypeError):
pass
try:
volume = _finite_or_none(m.get("volume", 0) or 0)
if volume is not None:
total_volume += volume
except (ValueError, TypeError):
pass
# Collect named outcomes for multi-outcome events
oname = m.get("groupItemTitle") or ""
if oname and pct is not None:
outcomes.append({"name": oname, "pct": pct})
# Only keep outcomes for multi-outcome markets (3+ named outcomes)
if len(outcomes) > 2:
outcomes.sort(key=lambda x: x["pct"], reverse=True)
else:
outcomes = []
# Extract tag labels
tag_labels = [t.get("label", "") for t in ev.get("tags", []) if t.get("label")]
results.append(
{
"title": title,
"source": "polymarket",
"pct": best_pct,
"slug": ev.get("slug", ""),
"description": ev.get("description") or "",
"end_date": ev.get("endDate"),
"volume": round(total_volume, 2),
"volume_24h": round(_finite_or_none(ev.get("volume24hr", 0) or 0) or 0, 2),
"tags": tag_labels,
"outcomes": outcomes,
}
)
logger.info(f"Polymarket: fetched {len(results)} active events")
return results
except Exception as e:
logger.error(f"Polymarket fetch error: {e}")
return []
# ---------------------------------------------------------------------------
# Kalshi
# ---------------------------------------------------------------------------
def _fetch_kalshi_events() -> list[dict]:
"""Fetch active events from Kalshi public API (no auth required)."""
from services.network_utils import fetch_with_curl
try:
resp = fetch_with_curl(
"https://api.elections.kalshi.com/v1/events?status=open&limit=100",
timeout=15,
)
if not resp or resp.status_code != 200:
logger.warning(f"Kalshi API returned {getattr(resp, 'status_code', 'N/A')}")
return []
data = resp.json()
events = data.get("events", []) if isinstance(data, dict) else []
results = []
for ev in events:
title = ev.get("title", "")
if not title:
continue
markets = ev.get("markets", [])
best_pct = None
total_volume = 0
close_dates = []
outcomes = []
for m in markets:
price = m.get("yes_price") or m.get("last_price")
pct = None
if price is not None:
try:
price = _finite_or_none(price)
if price is None:
raise ValueError("non-finite")
pct = round(price, 1)
if pct <= 1:
pct = round(pct * 100, 1)
if best_pct is None or pct > best_pct:
best_pct = pct
except (ValueError, TypeError):
pass
try:
volume = _finite_or_none(
m.get("dollar_volume", 0) or m.get("volume", 0) or 0
)
if volume is not None:
total_volume += int(volume)
except (ValueError, TypeError):
pass
cd = m.get("close_date")
if cd:
close_dates.append(cd)
# Collect named outcomes for multi-outcome events
oname = m.get("title") or m.get("subtitle", "")
if oname and pct is not None:
outcomes.append({"name": oname, "pct": pct})
# Only keep outcomes for multi-outcome markets (3+ named outcomes)
if len(outcomes) > 2:
outcomes.sort(key=lambda x: x["pct"], reverse=True)
else:
outcomes = []
# Description: settle_details or underlying
desc = (ev.get("settle_details") or ev.get("underlying") or "").strip()
sub = ev.get("sub_title", "")
results.append(
{
"title": title,
"source": "kalshi",
"pct": best_pct,
"ticker": ev.get("ticker", ""),
"description": desc,
"sub_title": sub,
"end_date": max(close_dates) if close_dates else None,
"volume": total_volume,
"category": ev.get("category", ""),
"outcomes": outcomes,
}
)
logger.info(f"Kalshi: fetched {len(results)} active events")
return results
except Exception as e:
logger.error(f"Kalshi fetch error: {e}")
return []
# ---------------------------------------------------------------------------
# Merge + classify
# ---------------------------------------------------------------------------
def _jaccard(a: str, b: str) -> float:
"""Word-level Jaccard similarity between two strings."""
wa = set(a.lower().split())
wb = set(b.lower().split())
if not wa or not wb:
return 0.0
return len(wa & wb) / len(wa | wb)
def _merge_markets(poly_events: list[dict], kalshi_events: list[dict]) -> list[dict]:
"""Merge Polymarket and Kalshi events by title similarity.
Returns a unified list with full metadata, categorized.
"""
merged = []
used_kalshi = set()
for pe in poly_events:
best_match = None
best_score = 0.0
for i, ke in enumerate(kalshi_events):
if i in used_kalshi:
continue
score = _jaccard(pe["title"], ke["title"])
if score > best_score and score >= 0.25:
best_score = score
best_match = (i, ke)
poly_pct = _finite_or_none(pe.get("pct"))
kalshi_pct = None
kalshi_vol = 0
kalshi_cat = ""
kalshi_end = None
kalshi_desc = ""
kalshi_ticker = ""
if best_match:
used_kalshi.add(best_match[0])
ke = best_match[1]
kalshi_pct = _finite_or_none(ke.get("pct"))
kalshi_vol = _finite_or_none(ke.get("volume", 0)) or 0
kalshi_cat = ke.get("category", "")
kalshi_end = ke.get("end_date")
kalshi_desc = ke.get("description", "")
kalshi_ticker = ke.get("ticker", "")
pcts = [p for p in [poly_pct, kalshi_pct] if p is not None]
consensus = round(sum(pcts) / len(pcts), 1) if pcts else None
# Build sources list
sources = []
if poly_pct is not None:
sources.append({"name": "POLY", "pct": poly_pct})
if kalshi_pct is not None:
sources.append({"name": "KALSHI", "pct": kalshi_pct})
category = _classify_category(pe["title"], pe.get("tags", []), kalshi_cat)
# Use best available description
desc = pe.get("description", "") or kalshi_desc
end_date = pe.get("end_date") or kalshi_end
# Use whichever source has more outcomes
poly_outcomes = pe.get("outcomes", [])
kalshi_outcomes = best_match[1].get("outcomes", []) if best_match else []
outcomes = poly_outcomes if len(poly_outcomes) >= len(kalshi_outcomes) else kalshi_outcomes
merged.append(
{
"title": pe["title"],
"polymarket_pct": poly_pct,
"kalshi_pct": kalshi_pct,
"consensus_pct": consensus,
"description": desc,
"end_date": end_date,
"volume": _finite_or_none(pe.get("volume", 0)) or 0,
"volume_24h": _finite_or_none(pe.get("volume_24h", 0)) or 0,
"kalshi_volume": kalshi_vol,
"category": category,
"sources": sources,
"slug": pe.get("slug", ""),
"kalshi_ticker": kalshi_ticker,
"outcomes": outcomes,
}
)
# Unmatched Kalshi events
for i, ke in enumerate(kalshi_events):
if i in used_kalshi:
continue
pct = _finite_or_none(ke.get("pct"))
sources = []
if pct is not None:
sources.append({"name": "KALSHI", "pct": pct})
category = _classify_category(ke["title"], [], ke.get("category", ""))
merged.append(
{
"title": ke["title"],
"polymarket_pct": None,
"kalshi_pct": pct,
"consensus_pct": pct,
"description": ke.get("description", ""),
"end_date": ke.get("end_date"),
"volume": 0,
"volume_24h": 0,
"kalshi_volume": _finite_or_none(ke.get("volume", 0)) or 0,
"category": category,
"sources": sources,
"slug": "",
"kalshi_ticker": ke.get("ticker", ""),
"outcomes": ke.get("outcomes", []),
}
)
return merged
@cached(_market_cache)
def fetch_prediction_markets_raw() -> list[dict]:
"""Fetch and merge prediction markets from both sources. Cached 5 min."""
poly = _fetch_polymarket_events()
kalshi = _fetch_kalshi_events()
merged = _merge_markets(poly, kalshi)
logger.info(
f"Prediction markets: {len(merged)} merged events "
f"({len(poly)} Polymarket, {len(kalshi)} Kalshi)"
)
return merged
def fetch_prediction_markets():
"""Fetcher entry point — writes merged markets to latest_data."""
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
global _prev_probabilities
markets = fetch_prediction_markets_raw()
# Compute probability deltas vs previous fetch
new_probs: dict[str, float] = {}
for m in markets:
title = m.get("title", "")
pct = m.get("consensus_pct")
if title and pct is not None:
prev = _prev_probabilities.get(title)
if prev is not None:
m["delta_pct"] = round(pct - prev, 1)
else:
m["delta_pct"] = None
new_probs[title] = pct
else:
m["delta_pct"] = None
_prev_probabilities = new_probs
# Build trending list (top 10 by absolute delta)
trending = sorted(
[m for m in markets if m.get("delta_pct") is not None and m["delta_pct"] != 0],
key=lambda x: abs(x["delta_pct"]),
reverse=True,
)[:10]
with _data_lock:
latest_data["prediction_markets"] = markets
latest_data["trending_markets"] = trending
_mark_fresh("prediction_markets")
# ---------------------------------------------------------------------------
# Direct API search (not limited to cached data)
# ---------------------------------------------------------------------------
def search_polymarket_direct(query: str, limit: int = 20) -> list[dict]:
"""Search Polymarket by scanning API pages for title matches.
The Gamma API has no text search parameter, so we scan cached events
plus additional pages until we find enough matches or exhaust the scan.
"""
from services.network_utils import fetch_with_curl
q_lower = query.lower()
q_words = set(q_lower.split())
results = []
# Scan up to 2000 events (10 pages of 200) looking for title matches
for offset in range(0, 2000, 200):
try:
resp = fetch_with_curl(
f"https://gamma-api.polymarket.com/events?active=true&closed=false&limit=200&offset={offset}",
timeout=15,
)
if not resp or resp.status_code != 200:
break
events = resp.json()
if not isinstance(events, list) or not events:
break
for ev in events:
title = ev.get("title", "")
if not title:
continue
title_lower = title.lower()
# Check if query appears in title or word overlap
if q_lower not in title_lower and not any(w in title_lower for w in q_words):
continue
# Extract same fields as regular fetch
markets = ev.get("markets", [])
best_pct = None
total_volume = 0
outcomes = []
for m in markets:
# Use outcomePrices[0] (Yes price) when available
raw_op = m.get("outcomePrices")
price = None
try:
op = json.loads(raw_op) if isinstance(raw_op, str) else raw_op
if isinstance(op, list) and len(op) >= 1:
price = _finite_or_none(op[0])
except (json.JSONDecodeError, ValueError, TypeError):
pass
if price is None:
price = _finite_or_none(m.get("lastTradePrice") or m.get("bestBid"))
pct = None
if price is not None:
try:
pct = round(price * 100, 1)
if best_pct is None or pct > best_pct:
best_pct = pct
except (ValueError, TypeError):
pass
try:
volume = _finite_or_none(m.get("volume", 0) or 0)
if volume is not None:
total_volume += volume
except (ValueError, TypeError):
pass
oname = m.get("groupItemTitle") or ""
if oname and pct is not None:
outcomes.append({"name": oname, "pct": pct})
if len(outcomes) > 2:
outcomes.sort(key=lambda x: x["pct"], reverse=True)
else:
outcomes = []
tag_labels = [t.get("label", "") for t in ev.get("tags", []) if t.get("label")]
category = _classify_category(title, tag_labels, "")
sources = []
if best_pct is not None:
sources.append({"name": "POLY", "pct": best_pct})
results.append(
{
"title": title,
"polymarket_pct": best_pct,
"kalshi_pct": None,
"consensus_pct": best_pct,
"description": ev.get("description") or "",
"end_date": ev.get("endDate"),
"volume": round(total_volume, 2),
"volume_24h": round(_finite_or_none(ev.get("volume24hr", 0) or 0) or 0, 2),
"kalshi_volume": 0,
"category": category,
"sources": sources,
"slug": ev.get("slug", ""),
"outcomes": outcomes,
}
)
# Stop scanning if we have enough results
if len(results) >= limit:
break
except Exception as e:
logger.warning(f"Polymarket search scan offset={offset} error: {e}")
break
logger.info(f"Polymarket search '{query}': {len(results)} results (scanned API)")
return results[:limit]
backend/services/fetchers/retry.py
+28 -5
View File
@@ -5,22 +5,37 @@ Usage:
def fetch_something():
...
"""
import time
import random
import logging
import functools
import requests
logger = logging.getLogger(__name__)
# Only retry on transient network/OS errors — not on parse errors, key errors, etc.
TRANSIENT_ERRORS = (
TimeoutError,
ConnectionError,
OSError,
requests.RequestException,
)
def with_retry(max_retries: int = 3, base_delay: float = 2.0, max_delay: float = 30.0):
"""Decorator: retries the wrapped function on any exception with exponential backoff + jitter.
"""Decorator: retries the wrapped function on transient errors with exponential backoff + jitter.
Only retries on network/OS errors (TimeoutError, ConnectionError, OSError,
requests.RequestException). Non-transient errors (ValueError, KeyError, etc.)
propagate immediately.
Args:
max_retries: Number of retry attempts after the initial failure.
base_delay: Base delay (seconds) for exponential backoff (2 → 4 → 8 …).
max_delay: Cap on the delay between retries.
"""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
@@ -28,22 +43,30 @@ def with_retry(max_retries: int = 3, base_delay: float = 2.0, max_delay: float =
for attempt in range(1 + max_retries):
try:
return func(*args, **kwargs)
except Exception as exc:
except TRANSIENT_ERRORS as exc:
last_exc = exc
if attempt < max_retries:
delay = min(base_delay * (2 ** attempt), max_delay)
delay = min(base_delay * (2**attempt), max_delay)
jitter = random.uniform(0, delay * 0.25)
total = delay + jitter
logger.warning(
"%s failed (attempt %d/%d): %s — retrying in %.1fs",
func.__name__, attempt + 1, max_retries + 1, exc, total,
func.__name__,
attempt + 1,
max_retries + 1,
exc,
total,
)
time.sleep(total)
else:
logger.error(
"%s failed after %d attempts: %s",
func.__name__, max_retries + 1, exc,
func.__name__,
max_retries + 1,
exc,
)
raise last_exc # type: ignore[misc]
return wrapper
return decorator
backend/services/fetchers/satellites.py
+498 -93
View File
@@ -6,6 +6,7 @@ CelesTrak Fair Use Policy (https://celestrak.org/NORAD/elements/):
- No parallel/concurrent connections — one request at a time
- Set a descriptive User-Agent
"""
import math
import time
import json
@@ -24,7 +25,9 @@ logger = logging.getLogger("services.data_fetcher")
def _gmst(jd_ut1):
"""Greenwich Mean Sidereal Time in radians from Julian Date."""
t = (jd_ut1 - 2451545.0) / 36525.0
gmst_sec = 67310.54841 + (876600.0 * 3600 + 8640184.812866) * t + 0.093104 * t * t - 6.2e-6 * t * t * t
gmst_sec = (
67310.54841 + (876600.0 * 3600 + 8640184.812866) * t + 0.093104 * t * t - 6.2e-6 * t * t * t
)
gmst_rad = (gmst_sec % 86400) / 86400.0 * 2 * math.pi
return gmst_rad
@@ -38,17 +41,21 @@ _sat_classified_cache = {"data": None, "gp_fetch_ts": 0}
_SAT_CACHE_PATH = Path(__file__).parent.parent.parent / "data" / "sat_gp_cache.json"
_SAT_CACHE_META_PATH = Path(__file__).parent.parent.parent / "data" / "sat_gp_cache_meta.json"
def _load_sat_cache():
"""Load satellite GP data from local disk cache."""
try:
if _SAT_CACHE_PATH.exists():
import os
age_hours = (time.time() - os.path.getmtime(str(_SAT_CACHE_PATH))) / 3600
if age_hours < 48:
with open(_SAT_CACHE_PATH, "r") as f:
data = json.load(f)
if isinstance(data, list) and len(data) > 10:
logger.info(f"Satellites: Loaded {len(data)} records from disk cache ({age_hours:.1f}h old)")
logger.info(
f"Satellites: Loaded {len(data)} records from disk cache ({age_hours:.1f}h old)"
)
# Restore last_modified from metadata
_load_cache_meta()
return data
@@ -58,6 +65,7 @@ def _load_sat_cache():
logger.warning(f"Satellites: Failed to load disk cache: {e}")
return None
def _save_sat_cache(data):
"""Save satellite GP data to local disk cache."""
try:
@@ -69,6 +77,7 @@ def _save_sat_cache(data):
except (IOError, OSError) as e:
logger.warning(f"Satellites: Failed to save disk cache: {e}")
def _load_cache_meta():
"""Load cache metadata (Last-Modified timestamp) from disk."""
try:
@@ -79,6 +88,7 @@ def _load_cache_meta():
except (IOError, OSError, json.JSONDecodeError, ValueError, KeyError):
pass
def _save_cache_meta():
"""Save cache metadata to disk."""
try:
@@ -90,54 +100,357 @@ def _save_cache_meta():
# Satellite intelligence classification database
_SAT_INTEL_DB = [
("USA 224", {"country": "USA", "mission": "military_recon", "sat_type": "KH-11 Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN"}),
("USA 245", {"country": "USA", "mission": "military_recon", "sat_type": "KH-11 Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN"}),
("USA 290", {"country": "USA", "mission": "military_recon", "sat_type": "KH-11 Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN"}),
("USA 314", {"country": "USA", "mission": "military_recon", "sat_type": "KH-11 Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN"}),
("USA 338", {"country": "USA", "mission": "military_recon", "sat_type": "Keyhole Successor", "wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN"}),
("TOPAZ", {"country": "Russia", "mission": "military_recon", "sat_type": "Optical Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/Persona_(satellite)"}),
("PERSONA", {"country": "Russia", "mission": "military_recon", "sat_type": "Optical Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/Persona_(satellite)"}),
("KONDOR", {"country": "Russia", "mission": "military_sar", "sat_type": "SAR Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/Kondor_(satellite)"}),
("BARS-M", {"country": "Russia", "mission": "military_recon", "sat_type": "Mapping Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/Bars-M"}),
("YAOGAN", {"country": "China", "mission": "military_recon", "sat_type": "Remote Sensing / ELINT", "wiki": "https://en.wikipedia.org/wiki/Yaogan"}),
("GAOFEN", {"country": "China", "mission": "military_recon", "sat_type": "High-Res Imaging", "wiki": "https://en.wikipedia.org/wiki/Gaofen"}),
("JILIN", {"country": "China", "mission": "commercial_imaging", "sat_type": "Video / Imaging", "wiki": "https://en.wikipedia.org/wiki/Jilin-1"}),
("OFEK", {"country": "Israel", "mission": "military_recon", "sat_type": "Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/Ofeq"}),
("CSO", {"country": "France", "mission": "military_recon", "sat_type": "Optical Reconnaissance", "wiki": "https://en.wikipedia.org/wiki/CSO_(satellite)"}),
("IGS", {"country": "Japan", "mission": "military_recon", "sat_type": "Intelligence Gathering", "wiki": "https://en.wikipedia.org/wiki/Information_Gathering_Satellite"}),
("CAPELLA", {"country": "USA", "mission": "sar", "sat_type": "SAR Imaging", "wiki": "https://en.wikipedia.org/wiki/Capella_Space"}),
("ICEYE", {"country": "Finland", "mission": "sar", "sat_type": "SAR Microsatellite", "wiki": "https://en.wikipedia.org/wiki/ICEYE"}),
("COSMO", {"country": "Italy", "mission": "sar", "sat_type": "SAR Constellation", "wiki": "https://en.wikipedia.org/wiki/COSMO-SkyMed"}),
("TANDEM", {"country": "Germany", "mission": "sar", "sat_type": "SAR Interferometry", "wiki": "https://en.wikipedia.org/wiki/TanDEM-X"}),
("PAZ", {"country": "Spain", "mission": "sar", "sat_type": "SAR Imaging", "wiki": "https://en.wikipedia.org/wiki/PAZ_(satellite)"}),
("WORLDVIEW", {"country": "USA", "mission": "commercial_imaging", "sat_type": "Maxar High-Res", "wiki": "https://en.wikipedia.org/wiki/WorldView-3"}),
("GEOEYE", {"country": "USA", "mission": "commercial_imaging", "sat_type": "Maxar Imaging", "wiki": "https://en.wikipedia.org/wiki/GeoEye-1"}),
("PLEIADES", {"country": "France", "mission": "commercial_imaging", "sat_type": "Airbus Imaging", "wiki": "https://en.wikipedia.org/wiki/Pl%C3%A9iades_(satellite)"}),
("SPOT", {"country": "France", "mission": "commercial_imaging", "sat_type": "Airbus Medium-Res", "wiki": "https://en.wikipedia.org/wiki/SPOT_(satellite)"}),
("PLANET", {"country": "USA", "mission": "commercial_imaging", "sat_type": "PlanetScope", "wiki": "https://en.wikipedia.org/wiki/Planet_Labs"}),
("SKYSAT", {"country": "USA", "mission": "commercial_imaging", "sat_type": "Planet Video", "wiki": "https://en.wikipedia.org/wiki/SkySat"}),
("BLACKSKY", {"country": "USA", "mission": "commercial_imaging", "sat_type": "BlackSky Imaging", "wiki": "https://en.wikipedia.org/wiki/BlackSky"}),
("NROL", {"country": "USA", "mission": "sigint", "sat_type": "Classified NRO", "wiki": "https://en.wikipedia.org/wiki/National_Reconnaissance_Office"}),
("MENTOR", {"country": "USA", "mission": "sigint", "sat_type": "SIGINT / ELINT", "wiki": "https://en.wikipedia.org/wiki/Mentor_(satellite)"}),
("LUCH", {"country": "Russia", "mission": "sigint", "sat_type": "Relay / SIGINT", "wiki": "https://en.wikipedia.org/wiki/Luch_(satellite)"}),
("SHIJIAN", {"country": "China", "mission": "sigint", "sat_type": "ELINT / Tech Demo", "wiki": "https://en.wikipedia.org/wiki/Shijian"}),
("NAVSTAR", {"country": "USA", "mission": "navigation", "sat_type": "GPS", "wiki": "https://en.wikipedia.org/wiki/GPS_satellite_blocks"}),
("GLONASS", {"country": "Russia", "mission": "navigation", "sat_type": "GLONASS", "wiki": "https://en.wikipedia.org/wiki/GLONASS"}),
("BEIDOU", {"country": "China", "mission": "navigation", "sat_type": "BeiDou", "wiki": "https://en.wikipedia.org/wiki/BeiDou"}),
("GALILEO", {"country": "EU", "mission": "navigation", "sat_type": "Galileo", "wiki": "https://en.wikipedia.org/wiki/Galileo_(satellite_navigation)"}),
("SBIRS", {"country": "USA", "mission": "early_warning", "sat_type": "Missile Warning", "wiki": "https://en.wikipedia.org/wiki/Space-Based_Infrared_System"}),
("TUNDRA", {"country": "Russia", "mission": "early_warning", "sat_type": "Missile Warning", "wiki": "https://en.wikipedia.org/wiki/Tundra_(satellite)"}),
("ISS", {"country": "Intl", "mission": "space_station", "sat_type": "Space Station", "wiki": "https://en.wikipedia.org/wiki/International_Space_Station"}),
("TIANGONG", {"country": "China", "mission": "space_station", "sat_type": "Space Station", "wiki": "https://en.wikipedia.org/wiki/Tiangong_space_station"}),
("CSS", {"country": "China", "mission": "space_station", "sat_type": "Chinese Space Station", "wiki": "https://en.wikipedia.org/wiki/Tiangong_space_station"}),
# Russian military — COSMOS covers the bulk of active Russian military/SIGINT satellites
("COSMOS", {"country": "Russia", "mission": "military_recon", "sat_type": "Russian Military / COSMOS", "wiki": "https://en.wikipedia.org/wiki/Kosmos_(satellite)"}),
# US military communications
("WGS", {"country": "USA", "mission": "sigint", "sat_type": "Wideband Global SATCOM", "wiki": "https://en.wikipedia.org/wiki/Wideband_Global_SATCOM"}),
("AEHF", {"country": "USA", "mission": "sigint", "sat_type": "Advanced EHF MILSATCOM", "wiki": "https://en.wikipedia.org/wiki/Advanced_Extremely_High_Frequency"}),
("MUOS", {"country": "USA", "mission": "sigint", "sat_type": "Mobile User Objective System", "wiki": "https://en.wikipedia.org/wiki/Mobile_User_Objective_System"}),
# EU Earth observation
("SENTINEL", {"country": "EU", "mission": "commercial_imaging", "sat_type": "ESA Copernicus", "wiki": "https://en.wikipedia.org/wiki/Sentinel_(satellite)"}),
(
"USA 224",
{
"country": "USA",
"mission": "military_recon",
"sat_type": "KH-11 Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN",
},
),
(
"USA 245",
{
"country": "USA",
"mission": "military_recon",
"sat_type": "KH-11 Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN",
},
),
(
"USA 290",
{
"country": "USA",
"mission": "military_recon",
"sat_type": "KH-11 Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN",
},
),
(
"USA 314",
{
"country": "USA",
"mission": "military_recon",
"sat_type": "KH-11 Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN",
},
),
(
"USA 338",
{
"country": "USA",
"mission": "military_recon",
"sat_type": "Keyhole Successor",
"wiki": "https://en.wikipedia.org/wiki/KH-11_KENNEN",
},
),
(
"TOPAZ",
{
"country": "Russia",
"mission": "military_recon",
"sat_type": "Optical Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/Persona_(satellite)",
},
),
(
"PERSONA",
{
"country": "Russia",
"mission": "military_recon",
"sat_type": "Optical Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/Persona_(satellite)",
},
),
(
"KONDOR",
{
"country": "Russia",
"mission": "military_sar",
"sat_type": "SAR Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/Kondor_(satellite)",
},
),
(
"BARS-M",
{
"country": "Russia",
"mission": "military_recon",
"sat_type": "Mapping Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/Bars-M",
},
),
(
"YAOGAN",
{
"country": "China",
"mission": "military_recon",
"sat_type": "Remote Sensing / ELINT",
"wiki": "https://en.wikipedia.org/wiki/Yaogan",
},
),
(
"GAOFEN",
{
"country": "China",
"mission": "military_recon",
"sat_type": "High-Res Imaging",
"wiki": "https://en.wikipedia.org/wiki/Gaofen",
},
),
(
"JILIN",
{
"country": "China",
"mission": "commercial_imaging",
"sat_type": "Video / Imaging",
"wiki": "https://en.wikipedia.org/wiki/Jilin-1",
},
),
(
"OFEK",
{
"country": "Israel",
"mission": "military_recon",
"sat_type": "Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/Ofeq",
},
),
(
"CSO",
{
"country": "France",
"mission": "military_recon",
"sat_type": "Optical Reconnaissance",
"wiki": "https://en.wikipedia.org/wiki/CSO_(satellite)",
},
),
(
"IGS",
{
"country": "Japan",
"mission": "military_recon",
"sat_type": "Intelligence Gathering",
"wiki": "https://en.wikipedia.org/wiki/Information_Gathering_Satellite",
},
),
(
"CAPELLA",
{
"country": "USA",
"mission": "sar",
"sat_type": "SAR Imaging",
"wiki": "https://en.wikipedia.org/wiki/Capella_Space",
},
),
(
"ICEYE",
{
"country": "Finland",
"mission": "sar",
"sat_type": "SAR Microsatellite",
"wiki": "https://en.wikipedia.org/wiki/ICEYE",
},
),
(
"COSMO-SKYMED",
{
"country": "Italy",
"mission": "sar",
"sat_type": "SAR Constellation",
"wiki": "https://en.wikipedia.org/wiki/COSMO-SkyMed",
},
),
(
"TANDEM",
{
"country": "Germany",
"mission": "sar",
"sat_type": "SAR Interferometry",
"wiki": "https://en.wikipedia.org/wiki/TanDEM-X",
},
),
(
"PAZ",
{
"country": "Spain",
"mission": "sar",
"sat_type": "SAR Imaging",
"wiki": "https://en.wikipedia.org/wiki/PAZ_(satellite)",
},
),
(
"WORLDVIEW",
{
"country": "USA",
"mission": "commercial_imaging",
"sat_type": "Maxar High-Res",
"wiki": "https://en.wikipedia.org/wiki/WorldView-3",
},
),
(
"GEOEYE",
{
"country": "USA",
"mission": "commercial_imaging",
"sat_type": "Maxar Imaging",
"wiki": "https://en.wikipedia.org/wiki/GeoEye-1",
},
),
(
"PLEIADES",
{
"country": "France",
"mission": "commercial_imaging",
"sat_type": "Airbus Imaging",
"wiki": "https://en.wikipedia.org/wiki/Pl%C3%A9iades_(satellite)",
},
),
(
"SPOT",
{
"country": "France",
"mission": "commercial_imaging",
"sat_type": "Airbus Medium-Res",
"wiki": "https://en.wikipedia.org/wiki/SPOT_(satellite)",
},
),
(
"PLANET",
{
"country": "USA",
"mission": "commercial_imaging",
"sat_type": "PlanetScope",
"wiki": "https://en.wikipedia.org/wiki/Planet_Labs",
},
),
(
"SKYSAT",
{
"country": "USA",
"mission": "commercial_imaging",
"sat_type": "Planet Video",
"wiki": "https://en.wikipedia.org/wiki/SkySat",
},
),
(
"BLACKSKY",
{
"country": "USA",
"mission": "commercial_imaging",
"sat_type": "BlackSky Imaging",
"wiki": "https://en.wikipedia.org/wiki/BlackSky",
},
),
(
"NROL",
{
"country": "USA",
"mission": "sigint",
"sat_type": "Classified NRO",
"wiki": "https://en.wikipedia.org/wiki/National_Reconnaissance_Office",
},
),
(
"MENTOR",
{
"country": "USA",
"mission": "sigint",
"sat_type": "SIGINT / ELINT",
"wiki": "https://en.wikipedia.org/wiki/Mentor_(satellite)",
},
),
(
"LUCH",
{
"country": "Russia",
"mission": "sigint",
"sat_type": "Relay / SIGINT",
"wiki": "https://en.wikipedia.org/wiki/Luch_(satellite)",
},
),
(
"SHIJIAN",
{
"country": "China",
"mission": "sigint",
"sat_type": "ELINT / Tech Demo",
"wiki": "https://en.wikipedia.org/wiki/Shijian",
},
),
(
"NAVSTAR",
{
"country": "USA",
"mission": "navigation",
"sat_type": "GPS",
"wiki": "https://en.wikipedia.org/wiki/GPS_satellite_blocks",
},
),
(
"GLONASS",
{
"country": "Russia",
"mission": "navigation",
"sat_type": "GLONASS",
"wiki": "https://en.wikipedia.org/wiki/GLONASS",
},
),
(
"BEIDOU",
{
"country": "China",
"mission": "navigation",
"sat_type": "BeiDou",
"wiki": "https://en.wikipedia.org/wiki/BeiDou",
},
),
(
"GALILEO",
{
"country": "EU",
"mission": "navigation",
"sat_type": "Galileo",
"wiki": "https://en.wikipedia.org/wiki/Galileo_(satellite_navigation)",
},
),
(
"SBIRS",
{
"country": "USA",
"mission": "early_warning",
"sat_type": "Missile Warning",
"wiki": "https://en.wikipedia.org/wiki/Space-Based_Infrared_System",
},
),
(
"TUNDRA",
{
"country": "Russia",
"mission": "early_warning",
"sat_type": "Missile Warning",
"wiki": "https://en.wikipedia.org/wiki/Tundra_(satellite)",
},
),
(
"ISS",
{
"country": "Intl",
"mission": "space_station",
"sat_type": "Space Station",
"wiki": "https://en.wikipedia.org/wiki/International_Space_Station",
},
),
(
"TIANGONG",
{
"country": "China",
"mission": "space_station",
"sat_type": "Space Station",
"wiki": "https://en.wikipedia.org/wiki/Tiangong_space_station",
},
),
]
@@ -154,7 +467,7 @@ def _parse_tle_to_gp(name, norad_id, line1, line2):
if bstar_str:
mantissa = float(bstar_str[:-2]) / 1e5
exponent = int(bstar_str[-2:])
bstar = mantissa * (10 ** exponent)
bstar = mantissa * (10**exponent)
else:
bstar = 0.0
epoch_yr = int(line1[18:20])
@@ -206,17 +519,50 @@ def _fetch_satellites_from_tle_api():
seen_ids.add(sat_id)
all_results.append(gp)
time.sleep(1) # Polite delay between requests
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as e:
logger.debug(f"TLE fallback search '{term}' failed: {e}")
return all_results
def fetch_satellites():
from services.fetchers._store import is_any_active
if not is_any_active("satellites"):
return
sats = []
try:
now_ts = time.time()
if _sat_gp_cache["data"] is None or (now_ts - _sat_gp_cache["last_fetch"]) > _CELESTRAK_FETCH_INTERVAL:
# On first call, try disk cache before hitting CelesTrak
if _sat_gp_cache["data"] is None:
disk_data = _load_sat_cache()
if disk_data:
import os
cache_mtime = (
os.path.getmtime(str(_SAT_CACHE_PATH)) if _SAT_CACHE_PATH.exists() else 0
)
_sat_gp_cache["data"] = disk_data
_sat_gp_cache["last_fetch"] = cache_mtime # real fetch time so 24h check works
_sat_gp_cache["source"] = "disk_cache"
logger.info(
f"Satellites: Bootstrapped from disk cache ({len(disk_data)} records, "
f"{(now_ts - cache_mtime) / 3600:.1f}h old)"
)
if (
_sat_gp_cache["data"] is None
or (now_ts - _sat_gp_cache["last_fetch"]) > _CELESTRAK_FETCH_INTERVAL
):
gp_urls = [
"https://celestrak.org/NORAD/elements/gp.php?GROUP=active&FORMAT=json",
"https://celestrak.com/NORAD/elements/gp.php?GROUP=active&FORMAT=json",
@@ -232,7 +578,9 @@ def fetch_satellites():
if response.status_code == 304:
# Data unchanged — reset timer without re-downloading
_sat_gp_cache["last_fetch"] = now_ts
logger.info(f"Satellites: CelesTrak returned 304 Not Modified (data unchanged)")
logger.info(
f"Satellites: CelesTrak returned 304 Not Modified (data unchanged)"
)
break
if response.status_code == 200:
gp_data = response.json()
@@ -241,14 +589,24 @@ def fetch_satellites():
_sat_gp_cache["last_fetch"] = now_ts
_sat_gp_cache["source"] = "celestrak"
# Store Last-Modified header for future conditional requests
if hasattr(response, 'headers'):
if hasattr(response, "headers"):
lm = response.headers.get("Last-Modified")
if lm:
_sat_gp_cache["last_modified"] = lm
_save_sat_cache(gp_data)
logger.info(f"Satellites: Downloaded {len(gp_data)} GP records from CelesTrak")
logger.info(
f"Satellites: Downloaded {len(gp_data)} GP records from CelesTrak"
)
break
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as e:
logger.warning(f"Satellites: Failed to fetch from {url}: {e}")
continue
@@ -261,8 +619,17 @@ def fetch_satellites():
_sat_gp_cache["last_fetch"] = now_ts
_sat_gp_cache["source"] = "tle_api"
_save_sat_cache(fallback_data)
logger.info(f"Satellites: Got {len(fallback_data)} records from TLE fallback API")
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, OSError) as e:
logger.info(
f"Satellites: Got {len(fallback_data)} records from TLE fallback API"
)
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
OSError,
) as e:
logger.error(f"Satellites: TLE fallback also failed: {e}")
if _sat_gp_cache["data"] is None:
@@ -279,9 +646,14 @@ def fetch_satellites():
latest_data["satellites"] = sats
return
if _sat_classified_cache["gp_fetch_ts"] == _sat_gp_cache["last_fetch"] and _sat_classified_cache["data"]:
if (
_sat_classified_cache["gp_fetch_ts"] == _sat_gp_cache["last_fetch"]
and _sat_classified_cache["data"]
):
classified = _sat_classified_cache["data"]
logger.info(f"Satellites: Using cached classification ({len(classified)} sats, TLEs unchanged)")
logger.info(
f"Satellites: Using cached classification ({len(classified)} sats, TLEs unchanged)"
)
else:
classified = []
for sat in data:
@@ -309,41 +681,57 @@ def fetch_satellites():
classified.append(entry)
_sat_classified_cache["data"] = classified
_sat_classified_cache["gp_fetch_ts"] = _sat_gp_cache["last_fetch"]
logger.info(f"Satellites: {len(classified)} intel-classified out of {len(data)} total in catalog")
logger.info(
f"Satellites: {len(classified)} intel-classified out of {len(data)} total in catalog"
)
all_sats = classified
now = datetime.utcnow()
jd, fr = jday(now.year, now.month, now.day, now.hour, now.minute, now.second + now.microsecond / 1e6)
jd, fr = jday(
now.year, now.month, now.day, now.hour, now.minute, now.second + now.microsecond / 1e6
)
for s in all_sats:
try:
mean_motion = s.get('MEAN_MOTION')
ecc = s.get('ECCENTRICITY')
incl = s.get('INCLINATION')
raan = s.get('RA_OF_ASC_NODE')
argp = s.get('ARG_OF_PERICENTER')
ma = s.get('MEAN_ANOMALY')
bstar = s.get('BSTAR', 0)
epoch_str = s.get('EPOCH')
norad_id = s.get('id', 0)
mean_motion = s.get("MEAN_MOTION")
ecc = s.get("ECCENTRICITY")
incl = s.get("INCLINATION")
raan = s.get("RA_OF_ASC_NODE")
argp = s.get("ARG_OF_PERICENTER")
ma = s.get("MEAN_ANOMALY")
bstar = s.get("BSTAR", 0)
epoch_str = s.get("EPOCH")
norad_id = s.get("id", 0)
if mean_motion is None or ecc is None or incl is None:
continue
epoch_dt = datetime.strptime(epoch_str[:19], '%Y-%m-%dT%H:%M:%S')
epoch_jd, epoch_fr = jday(epoch_dt.year, epoch_dt.month, epoch_dt.day,
epoch_dt.hour, epoch_dt.minute, epoch_dt.second)
epoch_dt = datetime.strptime(epoch_str[:19], "%Y-%m-%dT%H:%M:%S")
epoch_jd, epoch_fr = jday(
epoch_dt.year,
epoch_dt.month,
epoch_dt.day,
epoch_dt.hour,
epoch_dt.minute,
epoch_dt.second,
)
sat_obj = Satrec()
sat_obj.sgp4init(
WGS72, 'i', norad_id,
WGS72,
"i",
norad_id,
(epoch_jd + epoch_fr) - 2433281.5,
bstar, 0.0, 0.0, ecc,
math.radians(argp), math.radians(incl),
bstar,
0.0,
0.0,
ecc,
math.radians(argp),
math.radians(incl),
math.radians(ma),
mean_motion * 2 * math.pi / 1440.0,
math.radians(raan)
math.radians(raan),
)
e, r, v = sat_obj.sgp4(jd, fr)
@@ -353,13 +741,13 @@ def fetch_satellites():
x, y, z = r
gmst = _gmst(jd + fr)
lng_rad = math.atan2(y, x) - gmst
lat_rad = math.atan2(z, math.sqrt(x*x + y*y))
alt_km = math.sqrt(x*x + y*y + z*z) - 6371.0
lat_rad = math.atan2(z, math.sqrt(x * x + y * y))
alt_km = math.sqrt(x * x + y * y + z * z) - 6371.0
s['lat'] = round(math.degrees(lat_rad), 4)
s["lat"] = round(math.degrees(lat_rad), 4)
lng_deg = math.degrees(lng_rad) % 360
s['lng'] = round(lng_deg - 360 if lng_deg > 180 else lng_deg, 4)
s['alt_km'] = round(alt_km, 1)
s["lng"] = round(lng_deg - 360 if lng_deg > 180 else lng_deg, 4)
s["alt_km"] = round(alt_km, 1)
vx, vy, vz = v
omega_e = 7.2921159e-5
@@ -373,23 +761,40 @@ def fetch_satellites():
v_east = -sin_lng * vx_g + cos_lng * vy_g
v_north = -sin_lat * cos_lng * vx_g - sin_lat * sin_lng * vy_g + cos_lat * vz_g
ground_speed_kms = math.sqrt(v_east**2 + v_north**2)
s['speed_knots'] = round(ground_speed_kms * 1943.84, 1)
s["speed_knots"] = round(ground_speed_kms * 1943.84, 1)
heading_rad = math.atan2(v_east, v_north)
s['heading'] = round(math.degrees(heading_rad) % 360, 1)
sat_name = s.get('name', '')
usa_match = re.search(r'USA[\s\-]*(\d+)', sat_name)
s["heading"] = round(math.degrees(heading_rad) % 360, 1)
sat_name = s.get("name", "")
usa_match = re.search(r"USA[\s\-]*(\d+)", sat_name)
if usa_match:
s['wiki'] = f"https://en.wikipedia.org/wiki/USA-{usa_match.group(1)}"
for k in ('MEAN_MOTION', 'ECCENTRICITY', 'INCLINATION',
'RA_OF_ASC_NODE', 'ARG_OF_PERICENTER', 'MEAN_ANOMALY',
'BSTAR', 'EPOCH', 'tle1', 'tle2'):
s["wiki"] = f"https://en.wikipedia.org/wiki/USA-{usa_match.group(1)}"
for k in (
"MEAN_MOTION",
"ECCENTRICITY",
"INCLINATION",
"RA_OF_ASC_NODE",
"ARG_OF_PERICENTER",
"MEAN_ANOMALY",
"BSTAR",
"EPOCH",
"tle1",
"tle2",
):
s.pop(k, None)
sats.append(s)
except (ValueError, TypeError, KeyError, AttributeError, ZeroDivisionError):
continue
logger.info(f"Satellites: {len(classified)} classified, {len(sats)} positioned")
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, json.JSONDecodeError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
json.JSONDecodeError,
OSError,
) as e:
logger.error(f"Error fetching satellites: {e}")
if sats:
with _data_lock:
backend/services/fetchers/sigint.py
+102
View File
@@ -0,0 +1,102 @@
"""SIGINT fetcher — pulls latest signals from the SIGINT Grid into latest_data.
Merges live MQTT signals with cached Meshtastic map API nodes.
Live MQTT signals always take priority (fresher) — API nodes fill in the gaps
for the thousands of nodes our MQTT listener hasn't heard yet.
"""
import logging
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
logger = logging.getLogger("services.data_fetcher")
def _merge_sigint_snapshot(
live_signals: list[dict],
api_nodes: list[dict],
) -> list[dict]:
"""Merge live bridge signals with cached Meshtastic map nodes.
Live Meshtastic observations always win over map/API nodes for the same callsign
because they include fresher region/channel metadata.
"""
merged = list(live_signals)
live_callsigns = {s["callsign"] for s in merged if s.get("source") == "meshtastic"}
for node in api_nodes:
if node.get("callsign") in live_callsigns:
continue
merged.append(node)
merged.sort(key=lambda item: str(item.get("timestamp", "") or ""), reverse=True)
return merged
def _sigint_totals(signals: list[dict]) -> dict[str, int]:
totals = {
"total": len(signals),
"meshtastic": 0,
"meshtastic_live": 0,
"meshtastic_map": 0,
"aprs": 0,
"js8call": 0,
}
for sig in signals:
source = str(sig.get("source", "") or "").lower()
if source == "meshtastic":
totals["meshtastic"] += 1
if bool(sig.get("from_api")):
totals["meshtastic_map"] += 1
else:
totals["meshtastic_live"] += 1
elif source == "aprs":
totals["aprs"] += 1
elif source == "js8call":
totals["js8call"] += 1
return totals
def build_sigint_snapshot() -> tuple[list[dict], dict[str, object], dict[str, int]]:
"""Build the current merged SIGINT snapshot without hitting the network."""
from services.sigint_bridge import sigint_grid
live_signals = sigint_grid.get_all_signals()
with _data_lock:
api_nodes = list(latest_data.get("meshtastic_map_nodes", []))
merged = _merge_sigint_snapshot(live_signals, api_nodes)
channel_stats = sigint_grid.get_mesh_channel_stats(api_nodes or None)
totals = _sigint_totals(merged)
return merged, channel_stats, totals
def refresh_sigint_snapshot() -> tuple[list[dict], dict[str, object], dict[str, int]]:
"""Refresh latest_data SIGINT state from current bridge + cache state."""
signals, channel_stats, totals = build_sigint_snapshot()
with _data_lock:
latest_data["sigint"] = signals
latest_data["mesh_channel_stats"] = channel_stats
latest_data["sigint_totals"] = totals
_mark_fresh("sigint")
return signals, channel_stats, totals
def fetch_sigint():
"""Fetch all signals from the SIGINT Grid, merge with Meshtastic map nodes."""
from services.fetchers._store import is_any_active
if not is_any_active("sigint_meshtastic", "sigint_aprs"):
return
from services.sigint_bridge import sigint_grid
# Start bridges on first call (idempotent)
sigint_grid.start()
signals, channel_stats, totals = refresh_sigint_snapshot()
status = sigint_grid.status
logger.info(
f"SIGINT: {len(signals)} signals "
f"(APRS:{status['aprs']} MESH:{status['meshtastic']} "
f"JS8:{status['js8call']} MAP:{totals['meshtastic_map']})"
)
backend/services/fetchers/trains.py
+457
View File
@@ -0,0 +1,457 @@
"""Train tracking fetchers with normalized metadata and non-redundant merging."""
from __future__ import annotations
import logging
import math
from collections.abc import Callable
from datetime import datetime, timezone
from services.fetchers._store import _data_lock, _mark_fresh, latest_data
from services.network_utils import fetch_with_curl
logger = logging.getLogger(__name__)
_EARTH_RADIUS_KM = 6371.0
_MERGE_DISTANCE_KM = 5.0
_MAX_INFERRED_SPEED_KMH = 350.0
_TRACK_CACHE_TTL_S = 6 * 60 * 60
_SOURCE_METADATA: dict[str, dict[str, object]] = {
"amtrak": {
"source_label": "Amtraker",
"operator": "Amtrak",
"country": "US",
"telemetry_quality": "aggregated",
"priority": 70,
},
"digitraffic": {
"source_label": "Digitraffic Finland",
"operator": "Finnish Rail",
"country": "FI",
"telemetry_quality": "official",
"priority": 100,
},
# Future slots so better official feeds can be merged without changing the
# rest of the train pipeline or duplicating map entities.
"networkrail": {
"source_label": "Network Rail Open Data",
"operator": "Network Rail",
"country": "GB",
"telemetry_quality": "official",
"priority": 98,
},
"dbcargo": {
"source_label": "DB Cargo link2rail",
"operator": "DB Cargo",
"country": "DE",
"telemetry_quality": "commercial",
"priority": 96,
},
"railinc": {
"source_label": "Railinc RailSight",
"operator": "Railinc",
"country": "US",
"telemetry_quality": "commercial",
"priority": 97,
},
"sncf": {
"source_label": "SNCF Open Data",
"operator": "SNCF",
"country": "FR",
"telemetry_quality": "official",
"priority": 94,
},
}
_TRAIN_TRACK_CACHE: dict[str, dict[str, float]] = {}
def _safe_float(value) -> float | None:
try:
if value is None or value == "":
return None
return float(value)
except (TypeError, ValueError):
return None
def _parse_observed_at(value) -> float | None:
if value is None or value == "":
return None
if isinstance(value, (int, float)):
raw = float(value)
return raw / 1000.0 if raw > 1_000_000_000_000 else raw
if not isinstance(value, str):
return None
text = value.strip()
if not text:
return None
if text.endswith("Z"):
text = f"{text[:-1]}+00:00"
try:
return datetime.fromisoformat(text).timestamp()
except ValueError:
return None
def _haversine_km(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
lat1_rad, lon1_rad = math.radians(lat1), math.radians(lon1)
lat2_rad, lon2_rad = math.radians(lat2), math.radians(lon2)
dlat = lat2_rad - lat1_rad
dlon = lon2_rad - lon1_rad
a = (
math.sin(dlat / 2.0) ** 2
+ math.cos(lat1_rad) * math.cos(lat2_rad) * math.sin(dlon / 2.0) ** 2
)
return 2.0 * _EARTH_RADIUS_KM * math.asin(math.sqrt(a))
def _bearing_degrees(lat1: float, lon1: float, lat2: float, lon2: float) -> float | None:
if lat1 == lat2 and lon1 == lon2:
return None
lat1_rad, lat2_rad = math.radians(lat1), math.radians(lat2)
dlon_rad = math.radians(lon2 - lon1)
y = math.sin(dlon_rad) * math.cos(lat2_rad)
x = (
math.cos(lat1_rad) * math.sin(lat2_rad)
- math.sin(lat1_rad) * math.cos(lat2_rad) * math.cos(dlon_rad)
)
return (math.degrees(math.atan2(y, x)) + 360.0) % 360.0
def _source_meta(source: str) -> dict[str, object]:
return dict(_SOURCE_METADATA.get(source, {}))
def _normalize_train(
*,
source: str,
raw_id: str,
number: str,
lat,
lng,
name: str = "",
status: str = "Active",
route: str = "",
speed_kmh=None,
heading=None,
operator: str | None = None,
country: str | None = None,
source_label: str | None = None,
telemetry_quality: str | None = None,
observed_at=None,
) -> dict | None:
lat_f = _safe_float(lat)
lng_f = _safe_float(lng)
if lat_f is None or lng_f is None:
return None
if not (-90.0 <= lat_f <= 90.0 and -180.0 <= lng_f <= 180.0):
return None
number_text = str(number or "").strip()
meta = _source_meta(source)
observed_ts = _parse_observed_at(observed_at) or datetime.now(timezone.utc).timestamp()
speed_f = _safe_float(speed_kmh)
heading_f = _safe_float(heading)
normalized = {
"id": str(raw_id or f"{source}-{number_text or 'unknown'}"),
"name": str(name or f"Train {number_text or '?'}").strip(),
"number": number_text,
"source": source,
"source_label": str(source_label or meta.get("source_label") or source.upper()),
"operator": str(operator or meta.get("operator") or "").strip(),
"country": str(country or meta.get("country") or "").strip(),
"telemetry_quality": str(
telemetry_quality or meta.get("telemetry_quality") or "unknown"
).strip(),
"lat": lat_f,
"lng": lng_f,
"speed_kmh": speed_f,
"heading": heading_f,
"status": str(status or "Active").strip(),
"route": str(route or "").strip(),
"_source_priority": int(meta.get("priority") or 0),
"_observed_ts": observed_ts,
}
_apply_motion_estimates(normalized)
return normalized
def _prune_track_cache(now_ts: float) -> None:
stale_before = now_ts - _TRACK_CACHE_TTL_S
stale_ids = [train_id for train_id, entry in _TRAIN_TRACK_CACHE.items() if entry["ts"] < stale_before]
for train_id in stale_ids:
_TRAIN_TRACK_CACHE.pop(train_id, None)
def _apply_motion_estimates(train: dict) -> None:
train_id = str(train.get("id") or "")
if not train_id:
return
now_ts = float(train.get("_observed_ts") or datetime.now(timezone.utc).timestamp())
_prune_track_cache(now_ts)
previous = _TRAIN_TRACK_CACHE.get(train_id)
if previous:
dt_s = now_ts - previous["ts"]
if 5.0 <= dt_s <= 15.0 * 60.0:
distance_km = _haversine_km(
float(previous["lat"]),
float(previous["lng"]),
float(train["lat"]),
float(train["lng"]),
)
if 0.02 <= distance_km <= (_MAX_INFERRED_SPEED_KMH * (dt_s / 3600.0)):
if train.get("speed_kmh") is None:
inferred_speed = distance_km / (dt_s / 3600.0)
train["speed_kmh"] = round(min(inferred_speed, _MAX_INFERRED_SPEED_KMH), 1)
if train.get("heading") is None:
inferred_heading = _bearing_degrees(
float(previous["lat"]),
float(previous["lng"]),
float(train["lat"]),
float(train["lng"]),
)
if inferred_heading is not None:
train["heading"] = round(inferred_heading, 1)
_TRAIN_TRACK_CACHE[train_id] = {
"lat": float(train["lat"]),
"lng": float(train["lng"]),
"ts": now_ts,
}
def _train_merge_key(train: dict) -> str:
operator = str(train.get("operator") or "").strip().lower()
country = str(train.get("country") or "").strip().lower()
number = str(train.get("number") or "").strip().lower()
if operator and number:
return f"{country}|{operator}|{number}"
return f"{str(train.get('source') or '').lower()}|{str(train.get('id') or '').lower()}"
def _train_completeness(train: dict) -> tuple[int, int, int]:
return (
1 if train.get("speed_kmh") is not None else 0,
1 if train.get("heading") is not None else 0,
1 if train.get("route") else 0,
)
def _should_merge(existing: dict, candidate: dict) -> bool:
if _train_merge_key(existing) != _train_merge_key(candidate):
return False
return _haversine_km(
float(existing["lat"]),
float(existing["lng"]),
float(candidate["lat"]),
float(candidate["lng"]),
) <= _MERGE_DISTANCE_KM
def _merge_train_pair(existing: dict, candidate: dict) -> dict:
existing_priority = int(existing.get("_source_priority") or 0)
candidate_priority = int(candidate.get("_source_priority") or 0)
existing_score = (existing_priority, _train_completeness(existing))
candidate_score = (candidate_priority, _train_completeness(candidate))
primary = candidate if candidate_score > existing_score else existing
secondary = existing if primary is candidate else candidate
merged = dict(primary)
for field in (
"speed_kmh",
"heading",
"route",
"status",
"operator",
"country",
"source_label",
"telemetry_quality",
):
if merged.get(field) in (None, "", "Active"):
replacement = secondary.get(field)
if replacement not in (None, ""):
merged[field] = replacement
if primary is not candidate and float(candidate.get("_observed_ts") or 0) > float(
primary.get("_observed_ts") or 0
):
merged["lat"] = candidate["lat"]
merged["lng"] = candidate["lng"]
merged["_observed_ts"] = candidate["_observed_ts"]
return merged
def _merge_nonredundant_trains(*sources: list[dict]) -> list[dict]:
merged: list[dict] = []
for source_trains in sources:
for train in source_trains:
exact_match = next(
(
idx
for idx, existing in enumerate(merged)
if existing.get("source") == train.get("source")
and existing.get("id") == train.get("id")
),
None,
)
if exact_match is not None:
merged[exact_match] = _merge_train_pair(merged[exact_match], train)
continue
merged_idx = next(
(idx for idx, existing in enumerate(merged) if _should_merge(existing, train)),
None,
)
if merged_idx is not None:
merged[merged_idx] = _merge_train_pair(merged[merged_idx], train)
continue
merged.append(train)
merged.sort(
key=lambda train: (
str(train.get("country") or ""),
str(train.get("operator") or ""),
str(train.get("number") or ""),
str(train.get("id") or ""),
)
)
for train in merged:
train.pop("_source_priority", None)
train.pop("_observed_ts", None)
return merged
def _fetch_amtraker() -> list[dict]:
"""Fetch all active Amtrak trains from the Amtraker API."""
try:
resp = fetch_with_curl(
"https://api.amtraker.com/v3/trains",
timeout=20,
headers={
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/136.0.0.0 Safari/537.36"
),
"Accept": "application/json,text/plain,*/*",
"Referer": "https://www.amtraker.com/",
},
)
if resp.status_code != 200:
logger.warning("Amtraker returned %s", resp.status_code)
return []
raw = resp.json()
trains: list[dict] = []
for train_num, variants in raw.items():
if not isinstance(variants, list):
continue
for item in variants:
normalized = _normalize_train(
source="amtrak",
raw_id=f"AMTK-{item.get('trainID', train_num)}",
name=item.get("routeName", f"Train {train_num}"),
number=str(item.get("trainNum", train_num) or train_num),
lat=item.get("lat"),
lng=item.get("lon"),
speed_kmh=item.get("velocity") or item.get("speed"),
heading=item.get("heading") or item.get("bearing"),
status=item.get("trainTimely") or "On Time",
route=item.get("routeName", ""),
observed_at=item.get("updatedAt")
or item.get("lastValTS")
or item.get("eventDT"),
)
if normalized:
trains.append(normalized)
return trains
except Exception as exc:
logger.warning("Amtraker fetch error: %s", exc)
return []
def _fetch_digitraffic() -> list[dict]:
"""Fetch live train positions from Finnish DigiTraffic API."""
try:
resp = fetch_with_curl(
"https://rata.digitraffic.fi/api/v1/train-locations/latest",
timeout=15,
headers={
"Accept-Encoding": "gzip",
"User-Agent": "ShadowBroker-OSINT/1.0",
},
)
if resp.status_code != 200:
logger.warning("DigiTraffic returned %s", resp.status_code)
return []
raw = resp.json()
trains: list[dict] = []
for item in raw:
location = item.get("location", {})
coords = location.get("coordinates")
if not coords or len(coords) < 2:
continue
lon, lat = coords[0], coords[1]
train_number = str(item.get("trainNumber", "") or "").strip()
route_bits = [
str(item.get("departureStationShortCode") or "").strip(),
str(item.get("stationShortCode") or "").strip(),
]
route = " -> ".join([bit for bit in route_bits if bit])
train_type = str(item.get("trainType") or "").strip()
normalized = _normalize_train(
source="digitraffic",
raw_id=f"FIN-{train_number or len(trains)}",
name=f"{train_type} {train_number}".strip() or f"Train {train_number or '?'}",
number=train_number,
lat=lat,
lng=lon,
speed_kmh=item.get("speed"),
heading=item.get("heading"),
status="Active",
route=route,
observed_at=item.get("timestamp"),
)
if normalized:
trains.append(normalized)
return trains
except Exception as exc:
logger.warning("DigiTraffic fetch error: %s", exc)
return []
_TRAIN_FETCHERS: tuple[tuple[str, Callable[[], list[dict]]], ...] = (
("amtrak", _fetch_amtraker),
("digitraffic", _fetch_digitraffic),
)
def fetch_trains():
"""Fetch trains from all configured sources and merge without duplicates."""
with _data_lock:
existing_trains = list(latest_data.get("trains") or [])
source_batches: list[list[dict]] = []
source_counts: list[str] = []
for source_name, fetcher in _TRAIN_FETCHERS:
batch = fetcher()
source_batches.append(batch)
if batch:
source_counts.append(f"{source_name}:{len(batch)}")
trains = _merge_nonredundant_trains(*source_batches)
if not trains and existing_trains:
logger.warning(
"Train refresh returned 0 records — preserving %s cached trains until the next successful poll",
len(existing_trains),
)
trains = existing_trains
with _data_lock:
latest_data["trains"] = trains
_mark_fresh("trains")
logger.info(
"Trains: %s total%s",
len(trains),
f" ({', '.join(source_counts)})" if source_counts else "",
)
backend/services/fetchers/ukraine_alerts.py
+139
View File
@@ -0,0 +1,139 @@
"""Ukraine air raid alerts via alerts.in.ua API.
Polls active alerts every 2 minutes, matches to oblast boundary polygons,
and produces GeoJSON-style records for map rendering.
Requires ALERTS_IN_UA_TOKEN env var (free registration at alerts.in.ua).
Gracefully skips if token is not set.
"""
import json
import logging
import os
from pathlib import Path
from services.network_utils import fetch_with_curl
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
from services.fetchers.retry import with_retry
logger = logging.getLogger(__name__)
# ─── Alert type → color mapping ──────────────────────────────────────────────
ALERT_COLORS = {
"air_raid": "#ef4444", # red
"artillery_shelling": "#f97316", # orange
"urban_fights": "#eab308", # yellow
"chemical": "#a855f7", # purple
"nuclear": "#dc2626", # dark red
}
# ─── Load oblast boundary polygons (once) ────────────────────────────────────
_oblast_geojson = None
def _load_oblasts():
global _oblast_geojson
if _oblast_geojson is not None:
return _oblast_geojson
data_path = Path(__file__).resolve().parent.parent.parent / "data" / "ukraine_oblasts.geojson"
if not data_path.exists():
logger.error(f"Ukraine oblasts GeoJSON not found at {data_path}")
_oblast_geojson = {}
return _oblast_geojson
with open(data_path, "r", encoding="utf-8") as f:
_oblast_geojson = json.load(f)
logger.info(f"Loaded {len(_oblast_geojson.get('features', []))} Ukraine oblast boundaries")
return _oblast_geojson
def _find_oblast_geometry(location_title: str):
"""Find the polygon geometry for an oblast by matching Ukrainian name."""
oblasts = _load_oblasts()
features = oblasts.get("features", [])
for feat in features:
props = feat.get("properties", {})
name = props.get("name", "")
# Exact match on Ukrainian name (e.g. "Луганська область")
if name == location_title:
return feat.get("geometry"), props.get("name_en", "")
# Fuzzy: try partial match (alert may say "Київська область" but GeoJSON says "Київ")
for feat in features:
props = feat.get("properties", {})
name = props.get("name", "")
if location_title in name or name in location_title:
return feat.get("geometry"), props.get("name_en", "")
return None, ""
# ─── Fetcher ─────────────────────────────────────────────────────────────────
@with_retry(max_retries=1, base_delay=2)
def fetch_ukraine_air_raid_alerts():
"""Fetch active Ukraine air raid alerts from alerts.in.ua."""
from services.fetchers._store import is_any_active
if not is_any_active("ukraine_alerts"):
return
token = os.environ.get("ALERTS_IN_UA_TOKEN", "")
if not token:
logger.debug("ALERTS_IN_UA_TOKEN not set, skipping Ukraine air raid alerts")
return
alerts_out = []
try:
url = f"https://api.alerts.in.ua/v1/alerts/active.json?token={token}"
headers = {
"Authorization": f"Bearer {token}",
"Accept": "application/json",
}
response = fetch_with_curl(url, timeout=10, headers=headers)
if response.status_code == 200:
data = response.json()
raw_alerts = data.get("alerts", [])
for alert in raw_alerts:
loc_type = alert.get("location_type", "")
# Only render oblast-level alerts (not raion/city/hromada)
if loc_type != "oblast":
continue
location_title = alert.get("location_title", "")
alert_type = alert.get("alert_type", "air_raid")
geometry, name_en = _find_oblast_geometry(location_title)
if not geometry:
logger.debug(f"No geometry for oblast: {location_title}")
continue
alerts_out.append({
"id": alert.get("id", 0),
"alert_type": alert_type,
"location_title": location_title,
"location_uid": alert.get("location_uid", ""),
"name_en": name_en,
"started_at": alert.get("started_at", ""),
"color": ALERT_COLORS.get(alert_type, "#ef4444"),
"geometry": geometry,
})
logger.info(f"Ukraine alerts: {len(alerts_out)} active oblast-level alerts "
f"(from {len(raw_alerts)} total)")
elif response.status_code == 401:
logger.warning("alerts.in.ua returned 401 — check ALERTS_IN_UA_TOKEN")
elif response.status_code == 429:
logger.warning("alerts.in.ua rate-limited (429)")
else:
logger.warning(f"alerts.in.ua returned HTTP {response.status_code}")
except (ConnectionError, TimeoutError, OSError, ValueError, KeyError, TypeError) as e:
logger.error(f"Error fetching Ukraine alerts: {e}")
with _data_lock:
latest_data["ukraine_alerts"] = alerts_out
if alerts_out:
_mark_fresh("ukraine_alerts")
backend/services/fetchers/unusual_whales.py
+76
View File
@@ -0,0 +1,76 @@
"""Finnhub scheduled fetcher — congress trades, insider transactions, defense quotes.
Runs on a 15-minute schedule and stores results in latest_data["unusual_whales"].
Also updates latest_data["stocks"] with Finnhub quotes (replaces yfinance for defense tickers).
Falls back gracefully if no API key is configured.
"""
import logging
from services.fetchers._store import latest_data, _data_lock, _mark_fresh
from services.fetchers.retry import with_retry
logger = logging.getLogger(__name__)
@with_retry(max_retries=1, base_delay=2)
def fetch_unusual_whales():
"""Fetch congress trades, insider txns, and defense quotes from Finnhub."""
import os
if not os.environ.get("FINNHUB_API_KEY", "").strip():
logger.debug("FINNHUB_API_KEY not set — skipping scheduled fetch.")
return
from services.unusual_whales_connector import (
fetch_congress_trades,
fetch_insider_transactions,
fetch_defense_quotes,
FinnhubConnectorError,
)
result: dict = {}
# Defense stock quotes (also populates latest_data["stocks"])
try:
quotes = fetch_defense_quotes()
if quotes:
result["quotes"] = quotes
# Mirror into stocks for backward compat with existing MarketsPanel fallback
with _data_lock:
latest_data["stocks"] = quotes
_mark_fresh("stocks")
except FinnhubConnectorError as e:
logger.warning(f"Finnhub quotes fetch failed: {e.detail}")
except Exception as e:
logger.warning(f"Finnhub quotes fetch error: {e}")
# Congress trades
try:
congress = fetch_congress_trades()
result["congress_trades"] = congress.get("trades", [])
except FinnhubConnectorError as e:
logger.warning(f"Finnhub congress trades fetch failed: {e.detail}")
except Exception as e:
logger.warning(f"Finnhub congress trades fetch error: {e}")
# Insider transactions
try:
insiders = fetch_insider_transactions()
result["insider_transactions"] = insiders.get("transactions", [])
except FinnhubConnectorError as e:
logger.warning(f"Finnhub insider fetch failed: {e.detail}")
except Exception as e:
logger.warning(f"Finnhub insider fetch error: {e}")
if not result:
logger.warning("Finnhub update produced no data; keeping previous cache.")
return
with _data_lock:
latest_data["unusual_whales"] = result
_mark_fresh("unusual_whales")
logger.info(
f"Finnhub updated: {len(result.get('congress_trades', []))} congress, "
f"{len(result.get('insider_transactions', []))} insider, "
f"{len(result.get('quotes', {}))} quotes"
)
backend/services/fetchers/yacht_alert.py
+3 -1
View File
@@ -1,4 +1,5 @@
"""Yacht-Alert DB — load and enrich AIS vessels with tracked yacht metadata."""
import os
import json
import logging
@@ -26,7 +27,8 @@ def _load_yacht_alert_db():
global _YACHT_ALERT_DB
json_path = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))),
"data", "yacht_alert_db.json"
"data",
"yacht_alert_db.json",
)
if not os.path.exists(json_path):
logger.warning(f"Yacht-Alert DB not found at {json_path}")
backend/services/geocode.py
+255
View File
@@ -0,0 +1,255 @@
"""Geocoding proxy for Nominatim with caching and proper headers."""
from __future__ import annotations
import json
import os
import time
import threading
from typing import Any, Dict, List
from pathlib import Path
from urllib.parse import urlencode
from services.network_utils import fetch_with_curl
from services.fetchers.geo import cached_airports
_CACHE_TTL_S = 900
_CACHE_MAX = 1000
_cache: Dict[str, Dict[str, Any]] = {}
_cache_lock = threading.Lock()
_local_search_cache: List[Dict[str, Any]] | None = None
_local_search_lock = threading.Lock()
_USER_AGENT = os.environ.get(
"NOMINATIM_USER_AGENT", "ShadowBroker/1.0 (https://github.com/BigBodyCobain/Shadowbroker)"
)
def _get_cache(key: str):
now = time.time()
with _cache_lock:
entry = _cache.get(key)
if not entry:
return None
if now - entry["ts"] > _CACHE_TTL_S:
_cache.pop(key, None)
return None
return entry["value"]
def _set_cache(key: str, value):
with _cache_lock:
if len(_cache) >= _CACHE_MAX:
# Simple eviction: drop ~10% oldest keys
keys = list(_cache.keys())[: max(1, _CACHE_MAX // 10)]
for k in keys:
_cache.pop(k, None)
_cache[key] = {"ts": time.time(), "value": value}
def _load_local_search_cache() -> List[Dict[str, Any]]:
global _local_search_cache
with _local_search_lock:
if _local_search_cache is not None:
return _local_search_cache
results: List[Dict[str, Any]] = []
cache_path = Path(__file__).resolve().parents[1] / "data" / "geocode_cache.json"
try:
if cache_path.exists():
raw = json.loads(cache_path.read_text(encoding="utf-8"))
if isinstance(raw, dict):
for label, coords in raw.items():
if (
isinstance(label, str)
and isinstance(coords, list)
and len(coords) == 2
and all(isinstance(v, (int, float)) for v in coords)
):
results.append(
{
"label": label,
"lat": float(coords[0]),
"lng": float(coords[1]),
}
)
except Exception:
results = []
_local_search_cache = results
return _local_search_cache
def _search_local_fallback(query: str, limit: int) -> List[Dict[str, Any]]:
q = query.strip().lower()
if not q:
return []
matches: List[Dict[str, Any]] = []
seen: set[tuple[float, float, str]] = set()
for item in cached_airports:
haystacks = [
str(item.get("name", "")).lower(),
str(item.get("iata", "")).lower(),
str(item.get("id", "")).lower(),
]
if any(q in h for h in haystacks):
label = f'{item.get("name", "Airport")} ({item.get("iata", "")})'
key = (float(item["lat"]), float(item["lng"]), label)
if key not in seen:
seen.add(key)
matches.append(
{
"label": label,
"lat": float(item["lat"]),
"lng": float(item["lng"]),
}
)
if len(matches) >= limit:
return matches
for item in _load_local_search_cache():
label = str(item.get("label", ""))
if q in label.lower():
key = (float(item["lat"]), float(item["lng"]), label)
if key not in seen:
seen.add(key)
matches.append(item)
if len(matches) >= limit:
break
return matches
def _reverse_geocode_offline(lat: float, lng: float) -> Dict[str, Any]:
try:
import reverse_geocoder as rg
hit = rg.search((lat, lng), mode=1)[0]
city = hit.get("name") or ""
state = hit.get("admin1") or ""
country = hit.get("cc") or ""
parts = [city, state, country]
label = ", ".join([p for p in parts if p]) or "Unknown"
return {"label": label}
except Exception:
return {"label": "Unknown"}
def search_geocode(query: str, limit: int = 5, local_only: bool = False) -> List[Dict[str, Any]]:
q = query.strip()
if not q:
return []
limit = max(1, min(int(limit or 5), 10))
key = f"search:{q.lower()}:{limit}:{int(local_only)}"
cached = _get_cache(key)
if cached is not None:
return cached
if local_only:
results = _search_local_fallback(q, limit)
_set_cache(key, results)
return results
params = urlencode({"q": q, "format": "json", "limit": str(limit)})
url = f"https://nominatim.openstreetmap.org/search?{params}"
try:
res = fetch_with_curl(
url,
headers={
"User-Agent": _USER_AGENT,
"Accept-Language": "en",
},
timeout=6,
)
except Exception:
results = _search_local_fallback(q, limit)
_set_cache(key, results)
return results
results: List[Dict[str, Any]] = []
if res and res.status_code == 200:
try:
data = res.json() or []
for item in data:
try:
results.append(
{
"label": item.get("display_name"),
"lat": float(item.get("lat")),
"lng": float(item.get("lon")),
}
)
except (TypeError, ValueError):
continue
except Exception:
results = []
if not results:
results = _search_local_fallback(q, limit)
_set_cache(key, results)
return results
def reverse_geocode(lat: float, lng: float, local_only: bool = False) -> Dict[str, Any]:
key = f"reverse:{lat:.4f},{lng:.4f}:{int(local_only)}"
cached = _get_cache(key)
if cached is not None:
return cached
if local_only:
payload = _reverse_geocode_offline(lat, lng)
_set_cache(key, payload)
return payload
params = urlencode(
{
"lat": f"{lat}",
"lon": f"{lng}",
"format": "json",
"zoom": "10",
"addressdetails": "1",
}
)
url = f"https://nominatim.openstreetmap.org/reverse?{params}"
try:
res = fetch_with_curl(
url,
headers={
"User-Agent": _USER_AGENT,
"Accept-Language": "en",
},
timeout=6,
)
except Exception:
payload = _reverse_geocode_offline(lat, lng)
_set_cache(key, payload)
return payload
label = "Unknown"
if res and res.status_code == 200:
try:
data = res.json() or {}
addr = data.get("address") or {}
city = (
addr.get("city")
or addr.get("town")
or addr.get("village")
or addr.get("county")
or ""
)
state = addr.get("state") or addr.get("region") or ""
country = addr.get("country") or ""
parts = [city, state, country]
label = ", ".join([p for p in parts if p]) or (
data.get("display_name", "") or "Unknown"
)
except Exception:
label = "Unknown"
if label == "Unknown":
payload = _reverse_geocode_offline(lat, lng)
_set_cache(key, payload)
return payload
payload = {"label": label}
_set_cache(key, payload)
return payload
backend/services/geopolitics.py
+248 -92
View File
@@ -1,8 +1,11 @@
import requests
import logging
import zipfile
import socket
import ipaddress
from cachetools import cached, TTLCache
from datetime import datetime
from urllib.parse import urljoin, urlparse
from services.network_utils import fetch_with_curl
logger = logging.getLogger(__name__)
@@ -10,6 +13,7 @@ logger = logging.getLogger(__name__)
# Cache Frontline data for 30 minutes, it doesn't move that fast
frontline_cache = TTLCache(maxsize=1, ttl=1800)
@cached(frontline_cache)
def fetch_ukraine_frontlines():
"""
@@ -18,27 +22,34 @@ def fetch_ukraine_frontlines():
"""
try:
logger.info("Fetching DeepStateMap from GitHub mirror...")
# First, query the repo tree to find the latest file name
tree_url = "https://api.github.com/repos/cyterat/deepstate-map-data/git/trees/main?recursive=1"
tree_url = (
"https://api.github.com/repos/cyterat/deepstate-map-data/git/trees/main?recursive=1"
)
res_tree = requests.get(tree_url, timeout=10)
if res_tree.status_code == 200:
tree_data = res_tree.json().get("tree", [])
# Filter for geojson files in data folder
geo_files = [item["path"] for item in tree_data if item["path"].startswith("data/deepstatemap_data_") and item["path"].endswith(".geojson")]
geo_files = [
item["path"]
for item in tree_data
if item["path"].startswith("data/deepstatemap_data_")
and item["path"].endswith(".geojson")
]
if geo_files:
# Get the alphabetically latest file (since it's named with YYYYMMDD)
latest_file = sorted(geo_files)[-1]
raw_url = f"https://raw.githubusercontent.com/cyterat/deepstate-map-data/main/{latest_file}"
logger.info(f"Downloading latest DeepStateMap: {raw_url}")
res_geo = requests.get(raw_url, timeout=20)
if res_geo.status_code == 200:
data = res_geo.json()
# The Cyterat GitHub mirror strips all properties and just provides a raw array of Feature polygons.
# Based on DeepStateMap's frontend mapping, the array index corresponds to the zone type:
# 0: Russian-occupied areas
@@ -49,68 +60,78 @@ def fetch_ukraine_frontlines():
0: "Russian-occupied areas",
1: "Russian advance",
2: "Liberated area",
3: "Russian-occupied areas", # Crimea / LPR / DPR
4: "Directions of UA attacks"
3: "Russian-occupied areas", # Crimea / LPR / DPR
4: "Directions of UA attacks",
}
if "features" in data:
for idx, feature in enumerate(data["features"]):
if "properties" not in feature or feature["properties"] is None:
feature["properties"] = {}
feature["properties"]["name"] = name_map.get(idx, "Russian-occupied areas")
feature["properties"]["name"] = name_map.get(
idx, "Russian-occupied areas"
)
feature["properties"]["zone_id"] = idx
return data
else:
logger.error(f"Failed to fetch parsed Github Raw GeoJSON: {res_geo.status_code}")
logger.error(
f"Failed to fetch parsed Github Raw GeoJSON: {res_geo.status_code}"
)
else:
logger.error(f"Failed to fetch Github Tree for Deepstatemap: {res_tree.status_code}")
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
logger.error(f"Error fetching DeepStateMap: {e}")
return None
# Cache GDELT data for 6 hours - heavy aggregation, data doesn't change rapidly
gdelt_cache = TTLCache(maxsize=1, ttl=21600)
def _extract_domain(url):
"""Extract a clean source name from a URL, e.g. 'nytimes.com' from 'https://www.nytimes.com/...'"""
try:
from urllib.parse import urlparse
host = urlparse(url).hostname or ''
host = urlparse(url).hostname or ""
# Strip www. prefix
if host.startswith('www.'):
if host.startswith("www."):
host = host[4:]
return host
except (ValueError, AttributeError, KeyError): # non-critical
return url[:40]
def _url_to_headline(url):
"""Extract a human-readable headline from a URL path.
e.g. 'https://nytimes.com/2026/03/us-strikes-iran-nuclear-sites.html' -> 'Us Strikes Iran Nuclear Sites'
Falls back to domain name if the URL slug is gibberish (hex IDs, UUIDs, etc.).
"""
import re
try:
from urllib.parse import urlparse, unquote
parsed = urlparse(url)
domain = parsed.hostname or ''
if domain.startswith('www.'):
domain = parsed.hostname or ""
if domain.startswith("www."):
domain = domain[4:]
# Get last meaningful path segment
path = unquote(parsed.path).strip('/')
path = unquote(parsed.path).strip("/")
if not path:
return domain
# Try the last path segment first, then walk backwards
segments = [s for s in path.split('/') if s]
slug = ''
segments = [s for s in path.split("/") if s]
slug = ""
for seg in reversed(segments):
# Remove file extensions
for ext in ['.html', '.htm', '.php', '.asp', '.aspx', '.shtml']:
for ext in [".html", ".htm", ".php", ".asp", ".aspx", ".shtml"]:
if seg.lower().endswith(ext):
seg = seg[:-len(ext)]
seg = seg[: -len(ext)]
# Skip segments that are clearly not headlines
if _is_gibberish(seg):
continue
@@ -121,22 +142,22 @@ def _url_to_headline(url):
return domain
# Remove common ID patterns at start/end
slug = re.sub(r'^[\d]+-', '', slug) # leading "13847569-"
slug = re.sub(r'-[\da-f]{6,}$', '', slug) # trailing hex IDs
slug = re.sub(r'[-_]c-\d+$', '', slug) # trailing "-c-21803431"
slug = re.sub(r'^p=\d+$', '', slug) # WordPress ?p=1234
slug = re.sub(r"^[\d]+-", "", slug) # leading "13847569-"
slug = re.sub(r"-[\da-f]{6,}$", "", slug) # trailing hex IDs
slug = re.sub(r"[-_]c-\d+$", "", slug) # trailing "-c-21803431"
slug = re.sub(r"^p=\d+$", "", slug) # WordPress ?p=1234
# Convert slug separators to spaces
slug = slug.replace('-', ' ').replace('_', ' ')
slug = re.sub(r'\s+', ' ', slug).strip()
slug = slug.replace("-", " ").replace("_", " ")
slug = re.sub(r"\s+", " ", slug).strip()
# Final gibberish check after cleanup
if len(slug) < 8 or _is_gibberish(slug.replace(' ', '-')):
if len(slug) < 8 or _is_gibberish(slug.replace(" ", "-")):
return domain
# Title case and truncate
headline = slug.title()
if len(headline) > 90:
headline = headline[:87] + '...'
headline = headline[:87] + "..."
return headline
except (ValueError, AttributeError, KeyError): # non-critical
return url[:60]
@@ -146,19 +167,22 @@ def _is_gibberish(text):
"""Detect if a URL segment is gibberish (hex IDs, UUIDs, numeric IDs, etc.)
rather than a real human-readable slug like 'us-strikes-iran'."""
import re
t = text.strip()
if not t:
return True
# Pure numbers
if re.match(r'^\d+$', t):
if re.match(r"^\d+$", t):
return True
# UUID pattern (with or without dashes)
if re.match(r'^[0-9a-f]{8}[_-]?[0-9a-f]{4}[_-]?[0-9a-f]{4}[_-]?[0-9a-f]{4}[_-]?[0-9a-f]{12}$', t, re.I):
if re.match(
r"^[0-9a-f]{8}[_-]?[0-9a-f]{4}[_-]?[0-9a-f]{4}[_-]?[0-9a-f]{4}[_-]?[0-9a-f]{12}$", t, re.I
):
return True
# Hex-heavy string: more than 40% hex digits among alphanumeric chars
alnum = re.sub(r'[^a-zA-Z0-9]', '', t)
alnum = re.sub(r"[^a-zA-Z0-9]", "", t)
if alnum:
hex_chars = sum(1 for c in alnum if c in '0123456789abcdefABCDEF')
hex_chars = sum(1 for c in alnum if c in "0123456789abcdefABCDEF")
if hex_chars / len(alnum) > 0.4 and len(alnum) > 6:
return True
# Mostly digits with a few alpha (like "article8efa6c53")
@@ -169,13 +193,81 @@ def _is_gibberish(text):
if len(t) < 5:
return True
# Query-param style segments
if '=' in t:
if "=" in t:
return True
return False
# Persistent cache for article titles — survives across GDELT cache refreshes
_article_title_cache = {}
# Bounded to 5000 entries with 24hr TTL to prevent unbounded memory growth
_article_title_cache = TTLCache(maxsize=5000, ttl=86400)
_article_url_safety_cache = TTLCache(maxsize=5000, ttl=3600)
_TITLE_FETCH_MAX_REDIRECTS = 3
_TITLE_FETCH_READ_BYTES = 32768
_ALLOWED_ARTICLE_PORTS = {80, 443, 8080, 8443}
def _hostname_resolves_public(hostname: str, port: int) -> bool:
try:
infos = socket.getaddrinfo(hostname, port, type=socket.SOCK_STREAM)
except (socket.gaierror, OSError):
return False
addresses = set()
for info in infos:
sockaddr = info[4] if len(info) > 4 else None
if not sockaddr:
continue
raw_addr = str(sockaddr[0] or "").split("%", 1)[0]
if not raw_addr:
continue
try:
addresses.add(ipaddress.ip_address(raw_addr))
except ValueError:
continue
return bool(addresses) and all(addr.is_global for addr in addresses)
def _is_safe_public_article_url(url: str) -> tuple[bool, str]:
cached = _article_url_safety_cache.get(url)
if cached is not None:
return cached
try:
parsed = urlparse(str(url or "").strip())
except ValueError:
result = (False, "parse_error")
_article_url_safety_cache[url] = result
return result
scheme = str(parsed.scheme or "").lower()
host = str(parsed.hostname or "").strip().lower()
if scheme not in {"http", "https"}:
result = (False, "scheme")
elif not host:
result = (False, "host")
elif parsed.username or parsed.password:
result = (False, "userinfo")
elif host in {"localhost", "localhost.localdomain"}:
result = (False, "localhost")
else:
port = parsed.port or (443 if scheme == "https" else 80)
if port not in _ALLOWED_ARTICLE_PORTS:
result = (False, "port")
else:
try:
target_ip = ipaddress.ip_address(host.split("%", 1)[0])
except ValueError:
target_ip = None
if target_ip is not None:
result = (True, "") if target_ip.is_global else (False, "private_ip")
else:
result = (True, "") if _hostname_resolves_public(host, port) else (False, "private_dns")
_article_url_safety_cache[url] = result
return result
def _fetch_article_title(url):
"""Fetch the real headline from an article's HTML <title> or og:title tag.
@@ -183,51 +275,85 @@ def _fetch_article_title(url):
Uses a persistent cache to avoid refetching."""
if url in _article_title_cache:
return _article_title_cache[url]
import re
try:
# Only read the first 32KB — the <title> is always in <head>
resp = requests.get(url, timeout=4, headers={
'User-Agent': 'Mozilla/5.0 (compatible; OSINT Dashboard/1.0)'
}, stream=True)
if resp.status_code != 200:
current_url = str(url or "").strip()
chunk = ""
for _ in range(_TITLE_FETCH_MAX_REDIRECTS + 1):
allowed, _reason = _is_safe_public_article_url(current_url)
if not allowed:
_article_title_cache[url] = None
return None
resp = requests.get(
current_url,
timeout=4,
headers={"User-Agent": "Mozilla/5.0 (compatible; OSINT Dashboard/1.0)"},
stream=True,
allow_redirects=False,
)
try:
location = str(resp.headers.get("Location") or "").strip()
if 300 <= resp.status_code < 400 and location:
current_url = urljoin(current_url, location)
continue
if resp.status_code != 200:
_article_title_cache[url] = None
return None
chunk = resp.raw.read(_TITLE_FETCH_READ_BYTES).decode("utf-8", errors="replace")
break
finally:
resp.close()
else:
_article_title_cache[url] = None
return None
chunk = resp.raw.read(32768).decode('utf-8', errors='replace')
resp.close()
title = None
# Try og:title first (usually the cleanest)
og_match = re.search(r'<meta[^>]+property=["\']og:title["\'][^>]+content=["\']([^"\'>]+)["\']', chunk, re.I)
og_match = re.search(
r'<meta[^>]+property=["\']og:title["\'][^>]+content=["\']([^"\'>]+)["\']', chunk, re.I
)
if not og_match:
og_match = re.search(r'<meta[^>]+content=["\']([^"\'>]+)["\'][^>]+property=["\']og:title["\']', chunk, re.I)
og_match = re.search(
r'<meta[^>]+content=["\']([^"\'>]+)["\'][^>]+property=["\']og:title["\']',
chunk,
re.I,
)
if og_match:
title = og_match.group(1).strip()
# Fall back to <title> tag
if not title:
title_match = re.search(r'<title[^>]*>([^<]+)</title>', chunk, re.I)
title_match = re.search(r"<title[^>]*>([^<]+)</title>", chunk, re.I)
if title_match:
title = title_match.group(1).strip()
if title:
# Clean up HTML entities
import html as html_mod
title = html_mod.unescape(title)
# Remove site name suffixes like " | CNN" or " - BBC News"
title = re.sub(r'\s*[|\-–—]\s*[^|\-–—]{2,30}$', '', title).strip()
title = re.sub(r"\s*[|\-–—]\s*[^|\-–—]{2,30}$", "", title).strip()
# Truncate very long titles
if len(title) > 120:
title = title[:117] + '...'
title = title[:117] + "..."
if len(title) > 10:
_article_title_cache[url] = title
return title
_article_title_cache[url] = None
return None
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, AttributeError): # non-critical
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
AttributeError,
): # non-critical
_article_title_cache[url] = None
return None
@@ -236,6 +362,7 @@ def _batch_fetch_titles(urls):
"""Fetch real article titles for a list of URLs in parallel.
Returns a dict of url -> title (or None if fetch failed)."""
from concurrent.futures import ThreadPoolExecutor
results = {}
with ThreadPoolExecutor(max_workers=16) as executor:
futures = {executor.submit(_fetch_article_title, u): u for u in urls}
@@ -253,16 +380,19 @@ def _parse_gdelt_export_zip(zip_bytes, conflict_codes, seen_locs, features, loc_
loc_index maps loc_key -> index in features list for fast duplicate merging.
"""
import csv, io, zipfile
try:
zf = zipfile.ZipFile(io.BytesIO(zip_bytes))
csv_name = zf.namelist()[0]
with zf.open(csv_name) as cf:
reader = csv.reader(io.TextIOWrapper(cf, encoding='utf-8', errors='replace'), delimiter='\t')
reader = csv.reader(
io.TextIOWrapper(cf, encoding="utf-8", errors="replace"), delimiter="\t"
)
for row in reader:
try:
if len(row) < 61:
continue
event_code = row[26][:2] if len(row[26]) >= 2 else ''
event_code = row[26][:2] if len(row[26]) >= 2 else ""
if event_code not in conflict_codes:
continue
lat = float(row[56]) if row[56] else None
@@ -270,10 +400,10 @@ def _parse_gdelt_export_zip(zip_bytes, conflict_codes, seen_locs, features, loc_
if lat is None or lng is None or (lat == 0 and lng == 0):
continue
source_url = row[60].strip() if len(row) > 60 else ''
location = row[52].strip() if len(row) > 52 else 'Unknown'
actor1 = row[6].strip() if len(row) > 6 else ''
actor2 = row[16].strip() if len(row) > 16 else ''
source_url = row[60].strip() if len(row) > 60 else ""
location = row[52].strip() if len(row) > 52 else "Unknown"
actor1 = row[6].strip() if len(row) > 6 else ""
actor2 = row[16].strip() if len(row) > 16 else ""
loc_key = f"{round(lat, 1)}_{round(lng, 1)}"
if loc_key in seen_locs:
@@ -293,25 +423,32 @@ def _parse_gdelt_export_zip(zip_bytes, conflict_codes, seen_locs, features, loc_
continue
seen_locs.add(loc_key)
name = location or (f"{actor1} vs {actor2}" if actor1 and actor2 else actor1) or "Unknown Incident"
domain = _extract_domain(source_url) if source_url else ''
name = (
location
or (f"{actor1} vs {actor2}" if actor1 and actor2 else actor1)
or "Unknown Incident"
)
domain = _extract_domain(source_url) if source_url else ""
loc_index[loc_key] = len(features)
features.append({
"type": "Feature",
"properties": {
"name": name,
"count": 1,
"_urls": [source_url] if source_url else [],
"_domains": {domain} if domain else set(),
},
"geometry": {"type": "Point", "coordinates": [lng, lat]},
"_loc_key": loc_key
})
features.append(
{
"type": "Feature",
"properties": {
"name": name,
"count": 1,
"_urls": [source_url] if source_url else [],
"_domains": {domain} if domain else set(),
},
"geometry": {"type": "Point", "coordinates": [lng, lat]},
"_loc_key": loc_key,
}
)
except (ValueError, IndexError):
continue
except (IOError, OSError, ValueError, KeyError, zipfile.BadZipFile) as e:
logger.warning(f"Failed to parse GDELT export zip: {e}")
def _download_gdelt_export(url):
"""Download a single GDELT export file, return bytes or None."""
try:
@@ -322,10 +459,12 @@ def _download_gdelt_export(url):
pass
return None
def _build_feature_html(features, fetched_titles=None):
"""Build URL + headline arrays for frontend rendering.
Uses fetched_titles (real article titles) when available, falls back to URL slug parsing."""
import html as html_mod
for f in features:
urls = f["properties"].pop("_urls", [])
f["properties"].pop("_domains", None)
@@ -338,10 +477,12 @@ def _build_feature_html(features, fetched_titles=None):
if urls:
links = []
for u, h in zip(urls, headlines):
safe_url = u if u.startswith(('http://', 'https://')) else 'about:blank'
safe_url = u if u.startswith(("http://", "https://")) else "about:blank"
safe_h = html_mod.escape(h)
links.append(f'<div style="margin-bottom:6px;"><a href="{safe_url}" target="_blank" rel="noopener noreferrer">{safe_h}</a></div>')
f["properties"]["html"] = ''.join(links)
links.append(
f'<div style="margin-bottom:6px;"><a href="{safe_url}" target="_blank" rel="noopener noreferrer">{safe_h}</a></div>'
)
f["properties"]["html"] = "".join(links)
else:
f["properties"]["html"] = html_mod.escape(f["properties"]["name"])
f.pop("_loc_key", None)
@@ -350,6 +491,7 @@ def _build_feature_html(features, fetched_titles=None):
def _enrich_gdelt_titles_background(features, all_article_urls):
"""Background thread: fetch real article titles then update features in-place."""
import html as html_mod
try:
logger.info(f"[BG] Fetching real article titles for {len(all_article_urls)} URLs...")
fetched_titles = _batch_fetch_titles(all_article_urls)
@@ -368,10 +510,12 @@ def _enrich_gdelt_titles_background(features, all_article_urls):
f["properties"]["_headlines_list"] = headlines
links = []
for u, h in zip(urls, headlines):
safe_url = u if u.startswith(('http://', 'https://')) else 'about:blank'
safe_url = u if u.startswith(("http://", "https://")) else "about:blank"
safe_h = html_mod.escape(h)
links.append(f'<div style="margin-bottom:6px;"><a href="{safe_url}" target="_blank" rel="noopener noreferrer">{safe_h}</a></div>')
f["properties"]["html"] = ''.join(links)
links.append(
f'<div style="margin-bottom:6px;"><a href="{safe_url}" target="_blank" rel="noopener noreferrer">{safe_h}</a></div>'
)
f["properties"]["html"] = "".join(links)
logger.info(f"[BG] GDELT title enrichment complete")
except Exception as e:
logger.error(f"[BG] GDELT title enrichment failed: {e}")
@@ -391,16 +535,18 @@ def fetch_global_military_incidents():
logger.info("Fetching GDELT events via export CDN (multi-file)...")
# Get the latest export URL to determine current timestamp
index_res = fetch_with_curl("http://data.gdeltproject.org/gdeltv2/lastupdate.txt", timeout=10)
index_res = fetch_with_curl(
"http://data.gdeltproject.org/gdeltv2/lastupdate.txt", timeout=10
)
if index_res.status_code != 200:
logger.error(f"GDELT lastupdate failed: {index_res.status_code}")
return []
# Extract latest export URL and its timestamp
latest_url = None
for line in index_res.text.strip().split('\n'):
for line in index_res.text.strip().split("\n"):
parts = line.strip().split()
if len(parts) >= 3 and parts[2].endswith('.export.CSV.zip'):
if len(parts) >= 3 and parts[2].endswith(".export.CSV.zip"):
latest_url = parts[2]
break
@@ -410,19 +556,20 @@ def fetch_global_military_incidents():
# Extract timestamp from URL like: http://data.gdeltproject.org/gdeltv2/20260301120000.export.CSV.zip
import re
ts_match = re.search(r'(\d{14})\.export\.CSV\.zip', latest_url)
ts_match = re.search(r"(\d{14})\.export\.CSV\.zip", latest_url)
if not ts_match:
logger.error("Could not parse GDELT export timestamp")
return []
latest_ts = datetime.strptime(ts_match.group(1), '%Y%m%d%H%M%S')
latest_ts = datetime.strptime(ts_match.group(1), "%Y%m%d%H%M%S")
# Generate URLs for the last 8 hours (32 files at 15-min intervals)
NUM_FILES = 32
urls = []
for i in range(NUM_FILES):
ts = latest_ts - timedelta(minutes=15 * i)
fname = ts.strftime('%Y%m%d%H%M%S') + '.export.CSV.zip'
fname = ts.strftime("%Y%m%d%H%M%S") + ".export.CSV.zip"
url = f"http://data.gdeltproject.org/gdeltv2/{fname}"
urls.append(url)
@@ -436,7 +583,7 @@ def fetch_global_military_incidents():
logger.info(f"Downloaded {successful}/{len(urls)} GDELT exports")
# Parse all downloaded files
CONFLICT_CODES = {'14', '17', '18', '19', '20'}
CONFLICT_CODES = {"14", "17", "18", "19", "20"}
features = []
seen_locs = set()
loc_index = {} # loc_key -> index in features
@@ -455,7 +602,9 @@ def fetch_global_military_incidents():
# Build HTML immediately with URL-slug headlines (instant, no network)
_build_feature_html(features)
logger.info(f"GDELT parsed: {len(features)} conflict locations from {successful} files (titles enriching in background)")
logger.info(
f"GDELT parsed: {len(features)} conflict locations from {successful} files (titles enriching in background)"
)
# Kick off background thread to enrich with real article titles
# Features list is shared — background thread updates in-place
@@ -468,6 +617,13 @@ def fetch_global_military_incidents():
return features
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError, OSError) as e:
except (
requests.RequestException,
ConnectionError,
TimeoutError,
ValueError,
KeyError,
OSError,
) as e:
logger.error(f"Error fetching GDELT data: {e}")
return []
backend/services/kiwisdr_fetcher.py
+15 -13
View File
@@ -16,13 +16,13 @@ kiwisdr_cache = TTLCache(maxsize=1, ttl=600) # 10-minute cache
def _parse_comment(html: str, field: str) -> str:
"""Extract a field value from HTML comment like <!-- field=value -->"""
m = re.search(rf'<!--\s*{field}=(.*?)\s*-->', html)
m = re.search(rf"<!--\s*{field}=(.*?)\s*-->", html)
return m.group(1).strip() if m else ""
def _parse_gps(html: str):
"""Extract lat/lon from <!-- gps=(lat, lon) --> comment."""
m = re.search(r'<!--\s*gps=\(([^,]+),\s*([^)]+)\)\s*-->', html)
m = re.search(r"<!--\s*gps=\(([^,]+),\s*([^)]+)\)\s*-->", html)
if m:
try:
return float(m.group(1)), float(m.group(2))
@@ -78,17 +78,19 @@ def fetch_kiwisdr_nodes() -> list[dict]:
except ValueError:
users_max = 0
nodes.append({
"name": name[:120], # Truncate long names
"lat": round(lat, 5),
"lon": round(lon, 5),
"url": url,
"users": users,
"users_max": users_max,
"bands": bands,
"antenna": antenna[:200] if antenna else "",
"location": location[:100] if location else "",
})
nodes.append(
{
"name": name[:120], # Truncate long names
"lat": round(lat, 5),
"lon": round(lon, 5),
"url": url,
"users": users,
"users_max": users_max,
"bands": bands,
"antenna": antenna[:200] if antenna else "",
"location": location[:100] if location else "",
}
)
logger.info(f"KiwiSDR: parsed {len(nodes)} online receivers")
return nodes
backend/services/liveuamap_scraper.py
+40 -29
View File
@@ -8,90 +8,101 @@ from playwright_stealth import stealth_sync
logger = logging.getLogger(__name__)
def fetch_liveuamap():
logger.info("Starting Liveuamap scraper with Playwright Stealth...")
regions = [
{"name": "Ukraine", "url": "https://liveuamap.com"},
{"name": "Middle East", "url": "https://mideast.liveuamap.com"},
{"name": "Israel-Palestine", "url": "https://israelpalestine.liveuamap.com"},
{"name": "Syria", "url": "https://syria.liveuamap.com"}
{"name": "Syria", "url": "https://syria.liveuamap.com"},
]
all_markers = []
seen_ids = set()
with sync_playwright() as p:
# Launching with a real user agent to bypass Turnstile
browser = p.chromium.launch(headless=True, args=["--disable-blink-features=AutomationControlled"])
browser = p.chromium.launch(
headless=True, args=["--disable-blink-features=AutomationControlled"]
)
context = browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
viewport={"width": 1920, "height": 1080},
color_scheme="dark"
color_scheme="dark",
)
page = context.new_page()
stealth_sync(page)
for region in regions:
try:
logger.info(f"Scraping Liveuamap region: {region['name']}")
page.goto(region["url"], timeout=60000, wait_until="domcontentloaded")
# Wait for the map canvas or markers script to load, max 10s wait
try:
page.wait_for_timeout(5000)
except (TimeoutError, OSError): # non-critical: page load delay
pass
html = page.content()
m = re.search(r"var\s+ovens\s*=\s*(.*?);(?!function)", html, re.DOTALL)
if not m:
logger.warning(f"Could not find 'ovens' data for {region['name']} in raw HTML")
# Let's try grabbing the evaluated JavaScript variable if it's there
try:
ovens_json = page.evaluate("() => typeof ovens !== 'undefined' ? JSON.stringify(ovens) : null")
ovens_json = page.evaluate(
"() => typeof ovens !== 'undefined' ? JSON.stringify(ovens) : null"
)
if ovens_json:
markers = json.loads(ovens_json)
# process below
html = f"var ovens={ovens_json};"
m = re.search(r"var\s+ovens=(.*?);", html, re.DOTALL)
except (ValueError, KeyError, OSError) as e: # non-critical: JS eval fallback
logger.debug(f"Could not evaluate ovens JS variable for {region['name']}: {e}")
logger.debug(
f"Could not evaluate ovens JS variable for {region['name']}: {e}"
)
if m:
json_str = m.group(1).strip()
if json_str.startswith("'") or json_str.startswith('"'):
json_str = json_str.strip('"\'')
json_str = base64.b64decode(urllib.parse.unquote(json_str)).decode('utf-8')
json_str = json_str.strip("\"'")
json_str = base64.b64decode(urllib.parse.unquote(json_str)).decode("utf-8")
try:
markers = json.loads(json_str)
for marker in markers:
mid = marker.get("id")
if mid and mid not in seen_ids:
seen_ids.add(mid)
all_markers.append({
"id": mid,
"type": "liveuamap",
"title": marker.get("s", "Unknown Event") or marker.get("title", ""),
"lat": marker.get("lat"),
"lng": marker.get("lng"),
"timestamp": marker.get("time", ""),
"link": marker.get("link", region["url"]),
"region": region["name"]
})
all_markers.append(
{
"id": mid,
"type": "liveuamap",
"title": marker.get("s", "Unknown Event")
or marker.get("title", ""),
"lat": marker.get("lat"),
"lng": marker.get("lng"),
"timestamp": marker.get("time", ""),
"link": marker.get("link", region["url"]),
"region": region["name"],
}
)
except (json.JSONDecodeError, ValueError, KeyError) as e:
logger.error(f"Error parsing JSON for {region['name']}: {e}")
except Exception as e:
logger.error(f"Error scraping Liveuamap {region['name']}: {e}")
browser.close()
logger.info(f"Liveuamap scraper finished, extracted {len(all_markers)} unique markers.")
return all_markers
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
res = fetch_liveuamap()
backend/services/logging_setup.py
+30
View File
@@ -0,0 +1,30 @@
"""Structured logging setup for backend services."""
import json
import logging
from datetime import datetime
from typing import Any, Dict
class JsonFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
payload: Dict[str, Any] = {
"ts": datetime.utcnow().isoformat(),
"level": record.levelname,
"logger": record.name,
"msg": record.getMessage(),
}
if record.exc_info:
payload["exc"] = self.formatException(record.exc_info)
return json.dumps(payload, ensure_ascii=False)
def setup_logging(level: str = "INFO"):
"""Configure root logger with JSON formatting."""
root = logging.getLogger()
if root.handlers:
return # Respect existing config
root.setLevel(level.upper())
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
root.addHandler(handler)
backend/services/mesh/__init__.py
+1
View File
@@ -0,0 +1 @@
# Mesh protocol services package
backend/services/mesh/mesh_bootstrap_manifest.py
+340
View File
@@ -0,0 +1,340 @@
from __future__ import annotations
import base64
import json
import time
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
from cryptography.exceptions import InvalidSignature
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
from services.config import get_settings
from services.mesh.mesh_crypto import canonical_json, normalize_peer_url
BACKEND_DIR = Path(__file__).resolve().parents[2]
DATA_DIR = BACKEND_DIR / "data"
DEFAULT_BOOTSTRAP_MANIFEST_PATH = DATA_DIR / "bootstrap_peers.json"
BOOTSTRAP_MANIFEST_VERSION = 1
ALLOWED_BOOTSTRAP_TRANSPORTS = {"clearnet", "onion"}
ALLOWED_BOOTSTRAP_ROLES = {"participant", "relay", "seed"}
class BootstrapManifestError(ValueError):
pass
@dataclass(frozen=True)
class BootstrapPeer:
peer_url: str
transport: str
role: str
label: str = ""
def to_dict(self) -> dict[str, Any]:
return asdict(self)
@dataclass(frozen=True)
class BootstrapManifest:
version: int
issued_at: int
valid_until: int
signer_id: str
peers: tuple[BootstrapPeer, ...]
signature: str
def payload_dict(self) -> dict[str, Any]:
return {
"version": int(self.version),
"issued_at": int(self.issued_at),
"valid_until": int(self.valid_until),
"signer_id": str(self.signer_id or ""),
"peers": [peer.to_dict() for peer in self.peers],
}
def to_dict(self) -> dict[str, Any]:
payload = self.payload_dict()
payload["signature"] = str(self.signature or "")
return payload
def _resolve_manifest_path(raw_path: str) -> Path:
raw = str(raw_path or "").strip()
if not raw:
return DEFAULT_BOOTSTRAP_MANIFEST_PATH
candidate = Path(raw)
if candidate.is_absolute():
return candidate
return BACKEND_DIR / candidate
def _canonical_manifest_payload(payload: dict[str, Any]) -> str:
return canonical_json(payload)
def _load_signer_private_key(private_key_b64: str) -> ed25519.Ed25519PrivateKey:
try:
signer_private_key = base64.b64decode(
str(private_key_b64 or "").encode("utf-8"),
validate=True,
)
return ed25519.Ed25519PrivateKey.from_private_bytes(signer_private_key)
except Exception as exc:
raise BootstrapManifestError("bootstrap signer private key must be raw Ed25519 base64") from exc
def bootstrap_signer_public_key_b64(private_key_b64: str) -> str:
signer = _load_signer_private_key(private_key_b64)
public_key = signer.public_key().public_bytes(
serialization.Encoding.Raw,
serialization.PublicFormat.Raw,
)
return base64.b64encode(public_key).decode("utf-8")
def generate_bootstrap_signer() -> dict[str, str]:
signer = ed25519.Ed25519PrivateKey.generate()
private_key = signer.private_bytes(
serialization.Encoding.Raw,
serialization.PrivateFormat.Raw,
serialization.NoEncryption(),
)
public_key = signer.public_key().public_bytes(
serialization.Encoding.Raw,
serialization.PublicFormat.Raw,
)
return {
"private_key_b64": base64.b64encode(private_key).decode("utf-8"),
"public_key_b64": base64.b64encode(public_key).decode("utf-8"),
}
def _verify_manifest_signature(
payload: dict[str, Any],
*,
signature_b64: str,
signer_public_key_b64: str,
) -> None:
try:
signature = base64.b64decode(str(signature_b64 or "").encode("utf-8"), validate=True)
except Exception as exc:
raise BootstrapManifestError("bootstrap manifest signature must be base64") from exc
try:
signer_public_key = base64.b64decode(
str(signer_public_key_b64 or "").encode("utf-8"),
validate=True,
)
verifier = ed25519.Ed25519PublicKey.from_public_bytes(signer_public_key)
except Exception as exc:
raise BootstrapManifestError("bootstrap signer public key must be raw Ed25519 base64") from exc
serialized = _canonical_manifest_payload(payload).encode("utf-8")
try:
verifier.verify(signature, serialized)
except InvalidSignature as exc:
raise BootstrapManifestError("bootstrap manifest signature invalid") from exc
def _validate_bootstrap_peer(peer_data: dict[str, Any]) -> BootstrapPeer:
peer_url = str(peer_data.get("peer_url", "") or "").strip()
transport = str(peer_data.get("transport", "") or "").strip().lower()
role = str(peer_data.get("role", "") or "").strip().lower()
label = str(peer_data.get("label", "") or "").strip()
if transport not in ALLOWED_BOOTSTRAP_TRANSPORTS:
raise BootstrapManifestError(f"unsupported bootstrap transport: {transport or 'missing'}")
if role not in ALLOWED_BOOTSTRAP_ROLES:
raise BootstrapManifestError(f"unsupported bootstrap role: {role or 'missing'}")
normalized = normalize_peer_url(peer_url)
if not normalized or normalized != peer_url:
raise BootstrapManifestError("bootstrap peer_url must be normalized")
parsed = urlparse(normalized)
hostname = str(parsed.hostname or "").strip().lower()
if transport == "clearnet":
if parsed.scheme != "https" or hostname.endswith(".onion"):
raise BootstrapManifestError("clearnet bootstrap peers must use https://")
elif transport == "onion":
if parsed.scheme != "http" or not hostname.endswith(".onion"):
raise BootstrapManifestError("onion bootstrap peers must use http://*.onion")
return BootstrapPeer(
peer_url=normalized,
transport=transport,
role=role,
label=label,
)
def _validate_bootstrap_manifest_payload(
payload: dict[str, Any],
*,
now: float | None = None,
) -> BootstrapManifest:
version = int(payload.get("version", 0) or 0)
issued_at = int(payload.get("issued_at", 0) or 0)
valid_until = int(payload.get("valid_until", 0) or 0)
signer_id = str(payload.get("signer_id", "") or "").strip()
peers_raw = payload.get("peers", [])
current_time = int(now if now is not None else time.time())
if version != BOOTSTRAP_MANIFEST_VERSION:
raise BootstrapManifestError(f"unsupported bootstrap manifest version: {version}")
if not signer_id:
raise BootstrapManifestError("bootstrap manifest signer_id is required")
if issued_at <= 0 or valid_until <= 0 or valid_until <= issued_at:
raise BootstrapManifestError("bootstrap manifest validity window is invalid")
if current_time > valid_until:
raise BootstrapManifestError("bootstrap manifest expired")
if not isinstance(peers_raw, list):
raise BootstrapManifestError("bootstrap manifest peers must be a list")
peers: list[BootstrapPeer] = []
seen: set[tuple[str, str]] = set()
for entry in peers_raw:
if not isinstance(entry, dict):
raise BootstrapManifestError("bootstrap manifest peers must be objects")
peer = _validate_bootstrap_peer(entry)
key = (peer.transport, peer.peer_url)
if key in seen:
raise BootstrapManifestError("bootstrap manifest peers must be unique")
seen.add(key)
peers.append(peer)
if not peers:
raise BootstrapManifestError("bootstrap manifest must contain at least one peer")
return BootstrapManifest(
version=version,
issued_at=issued_at,
valid_until=valid_until,
signer_id=signer_id,
peers=tuple(peers),
signature="",
)
def build_bootstrap_manifest_payload(
*,
signer_id: str,
peers: list[dict[str, Any]] | tuple[dict[str, Any], ...],
issued_at: int | None = None,
valid_until: int | None = None,
valid_for_hours: int = 168,
) -> dict[str, Any]:
timestamp = int(issued_at if issued_at is not None else time.time())
expiry = int(valid_until if valid_until is not None else timestamp + max(1, int(valid_for_hours or 0)) * 3600)
payload = {
"version": BOOTSTRAP_MANIFEST_VERSION,
"issued_at": timestamp,
"valid_until": expiry,
"signer_id": str(signer_id or "").strip(),
"peers": list(peers),
}
manifest = _validate_bootstrap_manifest_payload(payload, now=timestamp)
return manifest.payload_dict()
def sign_bootstrap_manifest_payload(
payload: dict[str, Any],
*,
signer_private_key_b64: str,
) -> str:
signer = _load_signer_private_key(signer_private_key_b64)
serialized = _canonical_manifest_payload(payload).encode("utf-8")
signature = signer.sign(serialized)
return base64.b64encode(signature).decode("utf-8")
def write_signed_bootstrap_manifest(
path: str | Path,
*,
signer_id: str,
signer_private_key_b64: str,
peers: list[dict[str, Any]] | tuple[dict[str, Any], ...],
issued_at: int | None = None,
valid_until: int | None = None,
valid_for_hours: int = 168,
) -> BootstrapManifest:
manifest_path = _resolve_manifest_path(str(path))
payload = build_bootstrap_manifest_payload(
signer_id=signer_id,
peers=list(peers),
issued_at=issued_at,
valid_until=valid_until,
valid_for_hours=valid_for_hours,
)
signature = sign_bootstrap_manifest_payload(
payload,
signer_private_key_b64=signer_private_key_b64,
)
manifest = BootstrapManifest(
version=int(payload["version"]),
issued_at=int(payload["issued_at"]),
valid_until=int(payload["valid_until"]),
signer_id=str(payload["signer_id"]),
peers=tuple(_validate_bootstrap_peer(dict(peer)) for peer in payload["peers"]),
signature=signature,
)
manifest_path.parent.mkdir(parents=True, exist_ok=True)
manifest_path.write_text(json.dumps(manifest.to_dict(), indent=2) + "\n", encoding="utf-8")
return manifest
def load_bootstrap_manifest(
path: str | Path,
*,
signer_public_key_b64: str,
now: float | None = None,
) -> BootstrapManifest:
manifest_path = _resolve_manifest_path(str(path))
try:
raw = json.loads(manifest_path.read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise BootstrapManifestError(f"bootstrap manifest not found: {manifest_path}") from exc
except json.JSONDecodeError as exc:
raise BootstrapManifestError("bootstrap manifest is not valid JSON") from exc
if not isinstance(raw, dict):
raise BootstrapManifestError("bootstrap manifest root must be an object")
signature = str(raw.get("signature", "") or "").strip()
payload = {key: value for key, value in raw.items() if key != "signature"}
if not signature:
raise BootstrapManifestError("bootstrap manifest signature is required")
_verify_manifest_signature(
payload,
signature_b64=signature,
signer_public_key_b64=signer_public_key_b64,
)
manifest = _validate_bootstrap_manifest_payload(payload, now=now)
return BootstrapManifest(
version=manifest.version,
issued_at=manifest.issued_at,
valid_until=manifest.valid_until,
signer_id=manifest.signer_id,
peers=manifest.peers,
signature=signature,
)
def load_bootstrap_manifest_from_settings(*, now: float | None = None) -> BootstrapManifest | None:
settings = get_settings()
if bool(getattr(settings, "MESH_BOOTSTRAP_DISABLED", False)):
return None
signer_public_key_b64 = str(getattr(settings, "MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "") or "").strip()
if not signer_public_key_b64:
return None
manifest_path = _resolve_manifest_path(str(getattr(settings, "MESH_BOOTSTRAP_MANIFEST_PATH", "") or ""))
return load_bootstrap_manifest(
manifest_path,
signer_public_key_b64=signer_public_key_b64,
now=now,
)
backend/services/mesh/mesh_crypto.py
+142
View File
@@ -0,0 +1,142 @@
"""Cryptographic helpers for Mesh protocol verification."""
from __future__ import annotations
import base64
import hashlib
import hmac
import json
from typing import Any
from urllib.parse import urlparse
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec, ed25519
from cryptography.exceptions import InvalidSignature
from services.mesh.mesh_protocol import PROTOCOL_VERSION, NETWORK_ID, normalize_payload
NODE_ID_PREFIX = "!sb_"
NODE_ID_HEX_LEN = 16
def canonical_json(obj: dict[str, Any]) -> str:
return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
def normalize_peer_url(peer_url: str) -> str:
raw = str(peer_url or "").strip()
if not raw:
return ""
parsed = urlparse(raw)
scheme = str(parsed.scheme or "").strip().lower()
hostname = str(parsed.hostname or "").strip().lower()
if not scheme or not hostname:
return ""
port = parsed.port
default_port = 443 if scheme == "https" else 80 if scheme == "http" else None
netloc = hostname
if port and port != default_port:
netloc = f"{hostname}:{port}"
path = str(parsed.path or "").rstrip("/")
return f"{scheme}://{netloc}{path}"
def _derive_peer_key(shared_secret: str, peer_url: str) -> bytes:
normalized_url = normalize_peer_url(peer_url)
if not shared_secret or not normalized_url:
return b""
# HKDF-Extract per RFC 5869 §2.2: PRK = HMAC-Hash(salt, IKM).
# Python's hmac.new(key=salt, msg=IKM) maps directly to that definition.
prk = hmac.new(
b"sb-peer-auth-v1",
shared_secret.encode("utf-8"),
hashlib.sha256,
).digest()
return hmac.new(
prk,
normalized_url.encode("utf-8") + b"\x01",
hashlib.sha256,
).digest()
def _node_digest(public_key_b64: str) -> str:
raw = base64.b64decode(public_key_b64)
return hashlib.sha256(raw).hexdigest()
def derive_node_id(public_key_b64: str, *, legacy: bool = False) -> str:
digest = _node_digest(public_key_b64)
length = NODE_ID_HEX_LEN
return NODE_ID_PREFIX + digest[:length]
def derive_node_id_candidates(public_key_b64: str) -> tuple[str, ...]:
current = derive_node_id(public_key_b64, legacy=False)
return (current,)
def build_signature_payload(
*,
event_type: str,
node_id: str,
sequence: int,
payload: dict[str, Any],
) -> str:
normalized = normalize_payload(event_type, payload)
payload_json = canonical_json(normalized)
return "|".join(
[PROTOCOL_VERSION, NETWORK_ID, event_type, node_id, str(sequence), payload_json]
)
def parse_public_key_algo(value: str) -> str:
val = (value or "").strip().upper()
if val in ("ED25519", "EDDSA"):
return "Ed25519"
if val in ("ECDSA", "ECDSA_P256", "P-256", "P256"):
return "ECDSA_P256"
return ""
def verify_signature(
*,
public_key_b64: str,
public_key_algo: str,
signature_hex: str,
payload: str,
) -> bool:
try:
sig_bytes = bytes.fromhex(signature_hex)
except Exception:
return False
try:
pub_raw = base64.b64decode(public_key_b64)
except Exception:
return False
algo = parse_public_key_algo(public_key_algo)
data = payload.encode("utf-8")
try:
if algo == "Ed25519":
pub = ed25519.Ed25519PublicKey.from_public_bytes(pub_raw)
pub.verify(sig_bytes, data)
return True
if algo == "ECDSA_P256":
pub = ec.EllipticCurvePublicKey.from_encoded_point(ec.SECP256R1(), pub_raw)
pub.verify(sig_bytes, data, ec.ECDSA(hashes.SHA256()))
return True
except InvalidSignature:
return False
except Exception:
return False
return False
def verify_node_binding(node_id: str, public_key_b64: str) -> bool:
try:
return str(node_id or "") in derive_node_id_candidates(public_key_b64)
except Exception:
return False
backend/services/mesh/mesh_dm_mls.py
+669
View File
@@ -0,0 +1,669 @@
"""MLS-backed DM session manager.
This module keeps DM session orchestration in Python while privacy-core owns
the MLS session state. Python-side metadata survives via domain storage, but
Rust session state remains in-memory only. Process restart still requires
session re-establishment until Rust FFI state export is available.
"""
from __future__ import annotations
import base64
import logging
import secrets
import threading
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import x25519
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from services.mesh.mesh_secure_storage import (
read_domain_json,
read_secure_json,
write_domain_json,
)
from services.mesh.mesh_privacy_logging import privacy_log_label
from services.mesh.mesh_wormhole_persona import sign_dm_alias_blob, verify_dm_alias_blob
from services.privacy_core_client import PrivacyCoreClient, PrivacyCoreError
from services.wormhole_supervisor import get_wormhole_state, transport_tier_from_state
logger = logging.getLogger(__name__)
DATA_DIR = Path(__file__).resolve().parents[2] / "data"
STATE_FILE = DATA_DIR / "wormhole_dm_mls.json"
STATE_FILENAME = "wormhole_dm_mls.json"
STATE_DOMAIN = "dm_alias"
_STATE_LOCK = threading.RLock()
_PRIVACY_CLIENT: PrivacyCoreClient | None = None
_STATE_LOADED = False
_TRANSPORT_TIER_ORDER = {
"public_degraded": 0,
"private_transitional": 1,
"private_strong": 2,
}
MLS_DM_FORMAT = "mls1"
MAX_DM_PLAINTEXT_SIZE = 65_536
try:
from nacl.public import PrivateKey as _NaclPrivateKey
from nacl.public import PublicKey as _NaclPublicKey
from nacl.public import SealedBox as _NaclSealedBox
except ImportError:
_NaclPrivateKey = None
_NaclPublicKey = None
_NaclSealedBox = None
def _b64(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def _unb64(data: str | bytes | None) -> bytes:
if not data:
return b""
if isinstance(data, bytes):
return base64.b64decode(data)
return base64.b64decode(data.encode("ascii"))
def _decode_key_text(data: str | bytes | None) -> bytes:
raw = str(data or "").strip()
if not raw:
return b""
try:
return bytes.fromhex(raw)
except ValueError:
return _unb64(raw)
def _normalize_alias(alias: str) -> str:
return str(alias or "").strip().lower()
def _session_id(local_alias: str, remote_alias: str) -> str:
return f"{_normalize_alias(local_alias)}::{_normalize_alias(remote_alias)}"
def _seal_keypair() -> dict[str, str]:
private_key = x25519.X25519PrivateKey.generate()
return {
"public_key": private_key.public_key().public_bytes_raw().hex(),
"private_key": private_key.private_bytes_raw().hex(),
}
def _seal_welcome_for_public_key(payload: bytes, public_key_text: str) -> bytes:
public_key_bytes = _decode_key_text(public_key_text)
if not public_key_bytes:
raise PrivacyCoreError("responder_dh_pub is required for sealed welcome")
if _NaclPublicKey is not None and _NaclSealedBox is not None:
return _NaclSealedBox(_NaclPublicKey(public_key_bytes)).encrypt(payload)
ephemeral_private = x25519.X25519PrivateKey.generate()
ephemeral_public = ephemeral_private.public_key().public_bytes_raw()
recipient_public = x25519.X25519PublicKey.from_public_bytes(public_key_bytes)
shared_secret = ephemeral_private.exchange(recipient_public)
key = HKDF(
algorithm=hashes.SHA256(),
length=32,
salt=None,
info=b"shadowbroker|dm-mls-welcome|v1",
).derive(shared_secret)
nonce = secrets.token_bytes(12)
ciphertext = AESGCM(key).encrypt(
nonce,
payload,
b"shadowbroker|dm-mls-welcome|v1",
)
return ephemeral_public + nonce + ciphertext
def _unseal_welcome_for_private_key(payload: bytes, private_key_text: str) -> bytes:
private_key_bytes = _decode_key_text(private_key_text)
if not private_key_bytes:
raise PrivacyCoreError("local DH secret unavailable for DM session acceptance")
if _NaclPrivateKey is not None and _NaclSealedBox is not None:
return _NaclSealedBox(_NaclPrivateKey(private_key_bytes)).decrypt(payload)
if len(payload) < 44:
raise PrivacyCoreError("sealed DM welcome is truncated")
ephemeral_public = x25519.X25519PublicKey.from_public_bytes(payload[:32])
nonce = payload[32:44]
ciphertext = payload[44:]
private_key = x25519.X25519PrivateKey.from_private_bytes(private_key_bytes)
shared_secret = private_key.exchange(ephemeral_public)
key = HKDF(
algorithm=hashes.SHA256(),
length=32,
salt=None,
info=b"shadowbroker|dm-mls-welcome|v1",
).derive(shared_secret)
try:
return AESGCM(key).decrypt(
nonce,
ciphertext,
b"shadowbroker|dm-mls-welcome|v1",
)
except Exception as exc:
raise PrivacyCoreError("sealed DM welcome decrypt failed") from exc
@dataclass
class _SessionBinding:
session_id: str
local_alias: str
remote_alias: str
role: str
session_handle: int
created_at: int
_ALIAS_IDENTITIES: dict[str, int] = {}
_ALIAS_BINDINGS: dict[str, dict[str, str]] = {}
_ALIAS_SEAL_KEYS: dict[str, dict[str, str]] = {}
_SESSIONS: dict[str, _SessionBinding] = {}
_DM_FORMAT_LOCKS: dict[str, str] = {}
def _default_state() -> dict[str, Any]:
return {
"version": 2,
"updated_at": 0,
"aliases": {},
"alias_seal_keys": {},
"sessions": {},
"dm_format_locks": {},
}
def _privacy_client() -> PrivacyCoreClient:
global _PRIVACY_CLIENT
if _PRIVACY_CLIENT is None:
_PRIVACY_CLIENT = PrivacyCoreClient.load()
return _PRIVACY_CLIENT
def _current_transport_tier() -> str:
return transport_tier_from_state(get_wormhole_state())
def _require_private_transport() -> tuple[bool, str]:
current = _current_transport_tier()
if _TRANSPORT_TIER_ORDER.get(current, 0) < _TRANSPORT_TIER_ORDER["private_transitional"]:
return False, "DM MLS requires PRIVATE transport tier"
return True, current
def _serialize_session(binding: _SessionBinding) -> dict[str, Any]:
return {
"session_id": binding.session_id,
"local_alias": binding.local_alias,
"remote_alias": binding.remote_alias,
"role": binding.role,
"session_handle": int(binding.session_handle),
"created_at": int(binding.created_at),
}
def _binding_record(handle: int, public_bundle: bytes, binding_proof: str) -> dict[str, Any]:
return {
"handle": int(handle),
"public_bundle": _b64(public_bundle),
"binding_proof": str(binding_proof or ""),
}
def _load_state() -> None:
global _STATE_LOADED
with _STATE_LOCK:
if _STATE_LOADED:
return
# KNOWN LIMITATION: Persisted handles only survive when the privacy-core
# library instance is still alive in the same process. Full Rust-state
# export/import is deferred to a later sprint.
domain_path = DATA_DIR / STATE_DOMAIN / STATE_FILENAME
if not domain_path.exists() and STATE_FILE.exists():
try:
legacy = read_secure_json(STATE_FILE, _default_state)
write_domain_json(STATE_DOMAIN, STATE_FILENAME, legacy)
STATE_FILE.unlink(missing_ok=True)
except Exception:
logger.warning(
"Legacy DM MLS state could not be decrypted — "
"discarding stale file and starting fresh"
)
STATE_FILE.unlink(missing_ok=True)
raw = read_domain_json(STATE_DOMAIN, STATE_FILENAME, _default_state)
state = _default_state()
if isinstance(raw, dict):
state.update(raw)
_ALIAS_IDENTITIES.clear()
_ALIAS_BINDINGS.clear()
for alias, payload in dict(state.get("aliases") or {}).items():
alias_key = _normalize_alias(alias)
if not alias_key:
continue
if isinstance(payload, dict):
handle = int(payload.get("handle", 0) or 0)
public_bundle_b64 = str(payload.get("public_bundle", "") or "")
binding_proof = str(payload.get("binding_proof", "") or "")
else:
handle = int(payload or 0)
public_bundle_b64 = ""
binding_proof = ""
if handle <= 0 or not public_bundle_b64 or not binding_proof:
logger.warning("DM MLS alias binding missing proof; identity will be re-created")
continue
try:
public_bundle = _unb64(public_bundle_b64)
except Exception as exc:
logger.warning("DM MLS alias binding decode failed: %s", type(exc).__name__)
continue
ok, reason = verify_dm_alias_blob(alias_key, public_bundle, binding_proof)
if not ok:
logger.warning("DM MLS alias binding invalid: %s", reason)
continue
_ALIAS_IDENTITIES[alias_key] = handle
_ALIAS_BINDINGS[alias_key] = _binding_record(handle, public_bundle, binding_proof)
_ALIAS_SEAL_KEYS.clear()
for alias, keypair in dict(state.get("alias_seal_keys") or {}).items():
alias_key = _normalize_alias(alias)
pair = dict(keypair or {})
public_key = str(pair.get("public_key", "") or "").strip().lower()
private_key = str(pair.get("private_key", "") or "").strip().lower()
if alias_key and public_key and private_key:
_ALIAS_SEAL_KEYS[alias_key] = {
"public_key": public_key,
"private_key": private_key,
}
_SESSIONS.clear()
for session_id, payload in dict(state.get("sessions") or {}).items():
if not isinstance(payload, dict):
continue
binding = _SessionBinding(
session_id=str(payload.get("session_id", session_id) or session_id),
local_alias=_normalize_alias(str(payload.get("local_alias", "") or "")),
remote_alias=_normalize_alias(str(payload.get("remote_alias", "") or "")),
role=str(payload.get("role", "initiator") or "initiator"),
session_handle=int(payload.get("session_handle", 0) or 0),
created_at=int(payload.get("created_at", 0) or 0),
)
if (
binding.session_id
and binding.session_handle > 0
and binding.local_alias in _ALIAS_IDENTITIES
):
_SESSIONS[binding.session_id] = binding
_DM_FORMAT_LOCKS.clear()
for session_id, payload_format in dict(state.get("dm_format_locks") or {}).items():
normalized = str(payload_format or "").strip().lower()
if normalized:
_DM_FORMAT_LOCKS[str(session_id or "")] = normalized
_STATE_LOADED = True
def _save_state() -> None:
with _STATE_LOCK:
write_domain_json(
STATE_DOMAIN,
STATE_FILENAME,
{
"version": 2,
"updated_at": int(time.time()),
"aliases": {
alias: dict(_ALIAS_BINDINGS.get(alias) or {})
for alias, handle in _ALIAS_IDENTITIES.items()
if _ALIAS_BINDINGS.get(alias)
},
"alias_seal_keys": {
alias: dict(keypair or {})
for alias, keypair in _ALIAS_SEAL_KEYS.items()
},
"sessions": {
session_id: _serialize_session(binding)
for session_id, binding in _SESSIONS.items()
},
"dm_format_locks": dict(_DM_FORMAT_LOCKS),
},
)
STATE_FILE.unlink(missing_ok=True)
def reset_dm_mls_state(*, clear_privacy_core: bool = False, clear_persistence: bool = True) -> None:
global _PRIVACY_CLIENT, _STATE_LOADED
with _STATE_LOCK:
if clear_privacy_core and _PRIVACY_CLIENT is not None:
try:
_PRIVACY_CLIENT.reset_all_state()
except Exception:
logger.exception("privacy-core reset failed while clearing DM MLS state")
_ALIAS_IDENTITIES.clear()
_ALIAS_BINDINGS.clear()
_ALIAS_SEAL_KEYS.clear()
_SESSIONS.clear()
_DM_FORMAT_LOCKS.clear()
_STATE_LOADED = False
if clear_persistence and STATE_FILE.exists():
STATE_FILE.unlink()
def _identity_handle_for_alias(alias: str) -> int:
alias_key = _normalize_alias(alias)
if not alias_key:
raise PrivacyCoreError("dm alias is required")
_load_state()
with _STATE_LOCK:
handle = _ALIAS_IDENTITIES.get(alias_key)
if handle:
return handle
handle = _privacy_client().create_identity()
public_bundle = _privacy_client().export_public_bundle(handle)
signed = sign_dm_alias_blob(alias_key, public_bundle)
if not signed.get("ok"):
try:
_privacy_client().release_identity(handle)
except Exception:
pass
raise PrivacyCoreError(str(signed.get("detail") or "dm_mls_identity_binding_failed"))
_ALIAS_IDENTITIES[alias_key] = handle
_ALIAS_BINDINGS[alias_key] = _binding_record(
handle,
public_bundle,
str(signed.get("signature", "") or ""),
)
_save_state()
return handle
def _seal_keypair_for_alias(alias: str) -> dict[str, str]:
alias_key = _normalize_alias(alias)
if not alias_key:
raise PrivacyCoreError("dm alias is required")
_load_state()
with _STATE_LOCK:
existing = _ALIAS_SEAL_KEYS.get(alias_key)
if existing and existing.get("public_key") and existing.get("private_key"):
return dict(existing)
created = _seal_keypair()
_ALIAS_SEAL_KEYS[alias_key] = created
_save_state()
return dict(created)
def export_dm_key_package_for_alias(alias: str) -> dict[str, Any]:
alias_key = _normalize_alias(alias)
if not alias_key:
return {"ok": False, "detail": "alias is required"}
try:
identity_handle = _identity_handle_for_alias(alias_key)
key_package = _privacy_client().export_key_package(identity_handle)
seal_keypair = _seal_keypair_for_alias(alias_key)
return {
"ok": True,
"alias": alias_key,
"mls_key_package": _b64(key_package),
"welcome_dh_pub": str(seal_keypair.get("public_key", "") or ""),
}
except Exception:
logger.exception(
"dm mls key package export failed for %s",
privacy_log_label(alias_key, label="alias"),
)
return {"ok": False, "detail": "dm_mls_key_package_failed"}
def _remember_session(local_alias: str, remote_alias: str, *, role: str, session_handle: int) -> _SessionBinding:
binding = _SessionBinding(
session_id=_session_id(local_alias, remote_alias),
local_alias=_normalize_alias(local_alias),
remote_alias=_normalize_alias(remote_alias),
role=str(role or "initiator"),
session_handle=int(session_handle),
created_at=int(time.time()),
)
with _STATE_LOCK:
existing = _SESSIONS.get(binding.session_id)
if existing is not None:
try:
_privacy_client().release_dm_session(session_handle)
except Exception:
pass
return existing
_SESSIONS[binding.session_id] = binding
_save_state()
return binding
def _forget_session(local_alias: str, remote_alias: str) -> _SessionBinding | None:
_load_state()
with _STATE_LOCK:
binding = _SESSIONS.pop(_session_id(local_alias, remote_alias), None)
_save_state()
return binding
def _lock_dm_format(local_alias: str, remote_alias: str, format_str: str) -> None:
_load_state()
with _STATE_LOCK:
_DM_FORMAT_LOCKS[_session_id(local_alias, remote_alias)] = str(format_str or "").strip().lower()
_save_state()
def is_dm_locked_to_mls(local_alias: str, remote_alias: str) -> bool:
_load_state()
return (
str(_DM_FORMAT_LOCKS.get(_session_id(local_alias, remote_alias), "") or "").strip().lower()
== MLS_DM_FORMAT
)
def _session_binding(local_alias: str, remote_alias: str) -> _SessionBinding:
_load_state()
session_id = _session_id(local_alias, remote_alias)
binding = _SESSIONS.get(session_id)
if binding is None:
raise PrivacyCoreError(f"dm session not found for {session_id}")
return binding
def initiate_dm_session(
local_alias: str,
remote_alias: str,
remote_prekey_bundle: dict,
responder_dh_pub: str = "",
) -> dict[str, Any]:
ok, detail = _require_private_transport()
if not ok:
return {"ok": False, "detail": detail}
local_key = _normalize_alias(local_alias)
remote_key = _normalize_alias(remote_alias)
remote_key_package_b64 = str(
(remote_prekey_bundle or {}).get("mls_key_package")
or (remote_prekey_bundle or {}).get("key_package")
or ""
).strip()
if not local_key or not remote_key or not remote_key_package_b64:
return {"ok": False, "detail": "local_alias, remote_alias, and mls_key_package are required"}
resolved_responder_dh_pub = str(
responder_dh_pub
or (remote_prekey_bundle or {}).get("welcome_dh_pub")
or (remote_prekey_bundle or {}).get("identity_dh_pub_key")
or ""
).strip()
key_package_handle = 0
session_handle = 0
remembered = False
try:
identity_handle = _identity_handle_for_alias(local_key)
key_package_handle = _privacy_client().import_key_package(_unb64(remote_key_package_b64))
session_handle = _privacy_client().create_dm_session(identity_handle, key_package_handle)
welcome = _privacy_client().dm_session_welcome(session_handle)
sealed_welcome = _seal_welcome_for_public_key(welcome, resolved_responder_dh_pub)
binding = _remember_session(local_key, remote_key, role="initiator", session_handle=session_handle)
remembered = True
return {"ok": True, "welcome": _b64(sealed_welcome), "session_id": binding.session_id}
except Exception:
logger.exception(
"dm mls initiate failed for %s -> %s",
privacy_log_label(local_key, label="alias"),
privacy_log_label(remote_key, label="alias"),
)
return {"ok": False, "detail": "dm_mls_initiate_failed"}
finally:
if key_package_handle:
try:
_privacy_client().release_key_package(key_package_handle)
except Exception:
pass
if session_handle and not remembered:
try:
_privacy_client().release_dm_session(session_handle)
except Exception:
pass
def accept_dm_session(
local_alias: str,
remote_alias: str,
welcome_b64: str,
local_dh_secret: str = "",
) -> dict[str, Any]:
ok, detail = _require_private_transport()
if not ok:
return {"ok": False, "detail": detail}
local_key = _normalize_alias(local_alias)
remote_key = _normalize_alias(remote_alias)
if not local_key or not remote_key or not str(welcome_b64 or "").strip():
return {"ok": False, "detail": "local_alias, remote_alias, and welcome are required"}
session_handle = 0
remembered = False
try:
identity_handle = _identity_handle_for_alias(local_key)
seal_keypair = _seal_keypair_for_alias(local_key)
welcome = _unseal_welcome_for_private_key(
_unb64(welcome_b64),
str(local_dh_secret or seal_keypair.get("private_key") or ""),
)
session_handle = _privacy_client().join_dm_session(identity_handle, welcome)
binding = _remember_session(local_key, remote_key, role="responder", session_handle=session_handle)
remembered = True
return {"ok": True, "session_id": binding.session_id}
except Exception:
logger.exception(
"dm mls accept failed for %s <- %s",
privacy_log_label(local_key, label="alias"),
privacy_log_label(remote_key, label="alias"),
)
return {"ok": False, "detail": "dm_mls_accept_failed"}
finally:
if session_handle and not remembered:
try:
_privacy_client().release_dm_session(session_handle)
except Exception:
pass
def has_dm_session(local_alias: str, remote_alias: str) -> dict[str, Any]:
ok, detail = _require_private_transport()
if not ok:
return {"ok": False, "detail": detail}
try:
binding = _session_binding(local_alias, remote_alias)
return {"ok": True, "exists": True, "session_id": binding.session_id}
except Exception:
return {"ok": True, "exists": False, "session_id": _session_id(local_alias, remote_alias)}
def ensure_dm_session(local_alias: str, remote_alias: str, welcome_b64: str) -> dict[str, Any]:
ok, detail = _require_private_transport()
if not ok:
return {"ok": False, "detail": detail}
has_session = has_dm_session(local_alias, remote_alias)
if not has_session.get("ok"):
return has_session
if has_session.get("exists"):
return {"ok": True, "session_id": _session_id(local_alias, remote_alias)}
return accept_dm_session(local_alias, remote_alias, welcome_b64)
def _session_expired_result(local_alias: str, remote_alias: str) -> dict[str, Any]:
binding = _forget_session(local_alias, remote_alias)
session_id = binding.session_id if binding is not None else _session_id(local_alias, remote_alias)
return {"ok": False, "detail": "session_expired", "session_id": session_id}
def encrypt_dm(local_alias: str, remote_alias: str, plaintext: str) -> dict[str, Any]:
ok, detail = _require_private_transport()
if not ok:
return {"ok": False, "detail": detail}
plaintext_bytes = str(plaintext or "").encode("utf-8")
if len(plaintext_bytes) > MAX_DM_PLAINTEXT_SIZE:
return {"ok": False, "detail": "plaintext exceeds maximum size"}
try:
binding = _session_binding(local_alias, remote_alias)
ciphertext = _privacy_client().dm_encrypt(binding.session_handle, plaintext_bytes)
_lock_dm_format(local_alias, remote_alias, MLS_DM_FORMAT)
return {
"ok": True,
"ciphertext": _b64(ciphertext),
# NOTE: nonce is generated for DM envelope compatibility with dm1 format.
# MLS handles its own nonce/IV internally — this field is not consumed by MLS.
"nonce": _b64(secrets.token_bytes(12)),
"session_id": binding.session_id,
}
except PrivacyCoreError as exc:
if "unknown dm session handle" in str(exc).lower():
return _session_expired_result(local_alias, remote_alias)
logger.exception(
"dm mls encrypt failed for %s -> %s",
privacy_log_label(local_alias, label="alias"),
privacy_log_label(remote_alias, label="alias"),
)
return {"ok": False, "detail": "dm_mls_encrypt_failed"}
except Exception:
logger.exception(
"dm mls encrypt failed for %s -> %s",
privacy_log_label(local_alias, label="alias"),
privacy_log_label(remote_alias, label="alias"),
)
return {"ok": False, "detail": "dm_mls_encrypt_failed"}
def decrypt_dm(local_alias: str, remote_alias: str, ciphertext_b64: str, nonce_b64: str) -> dict[str, Any]:
ok, detail = _require_private_transport()
if not ok:
return {"ok": False, "detail": detail}
try:
binding = _session_binding(local_alias, remote_alias)
plaintext = _privacy_client().dm_decrypt(binding.session_handle, _unb64(ciphertext_b64))
_lock_dm_format(local_alias, remote_alias, MLS_DM_FORMAT)
return {
"ok": True,
"plaintext": plaintext.decode("utf-8"),
"session_id": binding.session_id,
"nonce": str(nonce_b64 or ""),
}
except PrivacyCoreError as exc:
if "unknown dm session handle" in str(exc).lower():
return _session_expired_result(local_alias, remote_alias)
logger.exception(
"dm mls decrypt failed for %s <- %s",
privacy_log_label(local_alias, label="alias"),
privacy_log_label(remote_alias, label="alias"),
)
return {"ok": False, "detail": "dm_mls_decrypt_failed"}
except Exception:
logger.exception(
"dm mls decrypt failed for %s <- %s",
privacy_log_label(local_alias, label="alias"),
privacy_log_label(remote_alias, label="alias"),
)
return {"ok": False, "detail": "dm_mls_decrypt_failed"}
backend/services/mesh/mesh_dm_relay.py
+824
View File
@@ -0,0 +1,824 @@
"""Metadata-minimized DM relay for request and shared mailboxes.
This relay never decrypts application payloads. In secure mode it keeps
pending ciphertext in memory only and persists just the minimum metadata
needed for continuity: accepted DH bundles, block lists, witness data,
and nonce replay windows.
"""
from __future__ import annotations
import atexit
import hashlib
import json
import logging
import os
import secrets
import threading
import time
from collections import OrderedDict, defaultdict
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from services.config import get_settings
from services.mesh.mesh_metrics import increment as metrics_inc
from services.mesh.mesh_wormhole_prekey import _validate_bundle_record
from services.mesh.mesh_secure_storage import read_secure_json, write_secure_json
TTL_SECONDS = 3600
EPOCH_SECONDS = 6 * 60 * 60
DATA_DIR = Path(__file__).resolve().parents[2] / "data"
RELAY_FILE = DATA_DIR / "dm_relay.json"
logger = logging.getLogger(__name__)
def _get_token_pepper() -> str:
"""Read token pepper lazily so auto-generated values from startup audit take effect."""
pepper = os.environ.get("MESH_DM_TOKEN_PEPPER", "").strip()
if not pepper:
try:
from services.config import get_settings
from services.env_check import _ensure_dm_token_pepper
pepper = _ensure_dm_token_pepper(get_settings())
except Exception:
pepper = os.environ.get("MESH_DM_TOKEN_PEPPER", "").strip()
if not pepper:
raise RuntimeError("MESH_DM_TOKEN_PEPPER is unavailable at runtime")
return pepper
@dataclass
class DMMessage:
sender_id: str
ciphertext: str
timestamp: float
msg_id: str
delivery_class: str
sender_seal: str = ""
relay_salt: str = ""
sender_block_ref: str = ""
payload_format: str = "dm1"
session_welcome: str = ""
class DMRelay:
"""Relay for encrypted request/shared mailboxes."""
def __init__(self) -> None:
self._lock = threading.RLock()
self._mailboxes: dict[str, list[DMMessage]] = defaultdict(list)
self._dh_keys: dict[str, dict[str, Any]] = {}
self._prekey_bundles: dict[str, dict[str, Any]] = {}
self._mailbox_bindings: dict[str, dict[str, Any]] = defaultdict(dict)
self._witnesses: dict[str, list[dict[str, Any]]] = defaultdict(list)
self._blocks: dict[str, set[str]] = defaultdict(set)
self._nonce_cache: OrderedDict[str, float] = OrderedDict()
self._stats: dict[str, int] = {"messages_in_memory": 0}
self._dirty = False
self._save_timer: threading.Timer | None = None
self._SAVE_INTERVAL = 5.0
atexit.register(self._flush)
self._load()
def _settings(self):
return get_settings()
def _persist_spool_enabled(self) -> bool:
return bool(self._settings().MESH_DM_PERSIST_SPOOL)
def _request_mailbox_limit(self) -> int:
return max(1, int(self._settings().MESH_DM_REQUEST_MAILBOX_LIMIT))
def _shared_mailbox_limit(self) -> int:
return max(1, int(self._settings().MESH_DM_SHARED_MAILBOX_LIMIT))
def _self_mailbox_limit(self) -> int:
return max(1, int(self._settings().MESH_DM_SELF_MAILBOX_LIMIT))
def _nonce_ttl_seconds(self) -> int:
return max(30, int(self._settings().MESH_DM_NONCE_TTL_S))
def _nonce_cache_max_entries(self) -> int:
return max(1, int(getattr(self._settings(), "MESH_DM_NONCE_CACHE_MAX", 4096)))
def _pepper_token(self, token: str) -> str:
material = token
pepper = _get_token_pepper()
if pepper:
material = f"{pepper}|{token}"
return hashlib.sha256(material.encode("utf-8")).hexdigest()
def _sender_block_ref(self, sender_id: str) -> str:
sender = str(sender_id or "").strip()
if not sender:
return ""
return "ref:" + self._pepper_token(f"block|{sender}")
def _canonical_blocked_id(self, blocked_id: str) -> str:
blocked = str(blocked_id or "").strip()
if not blocked:
return ""
if blocked.startswith("ref:"):
return blocked
return self._sender_block_ref(blocked)
def _message_block_ref(self, message: DMMessage) -> str:
block_ref = str(getattr(message, "sender_block_ref", "") or "").strip()
if block_ref:
return block_ref
sender_id = str(message.sender_id or "").strip()
if not sender_id or sender_id.startswith("sealed:") or sender_id.startswith("sender_token:"):
return ""
return self._sender_block_ref(sender_id)
def _mailbox_key(self, mailbox_type: str, mailbox_value: str, epoch: int | None = None) -> str:
if mailbox_type in {"self", "requests"}:
bucket = self._epoch_bucket() if epoch is None else int(epoch)
identifier = f"{mailbox_type}|{bucket}|{mailbox_value}"
else:
identifier = f"{mailbox_type}|{mailbox_value}"
return self._pepper_token(identifier)
def _hashed_mailbox_token(self, token: str) -> str:
return hashlib.sha256(str(token or "").encode("utf-8")).hexdigest()
def _remember_mailbox_binding(self, agent_id: str, mailbox_type: str, token: str) -> str:
token_hash = self._hashed_mailbox_token(token)
self._mailbox_bindings[str(agent_id or "").strip()][str(mailbox_type or "").strip().lower()] = {
"token_hash": token_hash,
"last_used": time.time(),
}
self._save()
return token_hash
def _bound_mailbox_key(self, agent_id: str, mailbox_type: str) -> str:
entry = self._mailbox_bindings.get(str(agent_id or "").strip(), {}).get(
str(mailbox_type or "").strip().lower(),
"",
)
if isinstance(entry, dict):
return str(entry.get("token_hash", "") or "")
return str(entry or "")
def _mailbox_keys_for_claim(self, agent_id: str, claim: dict[str, Any]) -> list[str]:
claim_type = str(claim.get("type", "")).strip().lower()
if claim_type == "shared":
token = str(claim.get("token", "")).strip()
if not token:
metrics_inc("dm_claim_invalid")
return []
return [self._hashed_mailbox_token(token)]
if claim_type == "requests":
token = str(claim.get("token", "")).strip()
if token:
bound_key = self._remember_mailbox_binding(agent_id, "requests", token)
epoch = self._epoch_bucket()
return [
bound_key,
self._mailbox_key("requests", agent_id, epoch),
self._mailbox_key("requests", agent_id, epoch - 1),
]
metrics_inc("dm_claim_invalid")
return []
if claim_type == "self":
token = str(claim.get("token", "")).strip()
if token:
bound_key = self._remember_mailbox_binding(agent_id, "self", token)
epoch = self._epoch_bucket()
return [
bound_key,
self._mailbox_key("self", agent_id, epoch),
self._mailbox_key("self", agent_id, epoch - 1),
]
metrics_inc("dm_claim_invalid")
return []
metrics_inc("dm_claim_invalid")
return []
def mailbox_key_for_delivery(
self,
*,
recipient_id: str,
delivery_class: str,
recipient_token: str | None = None,
) -> str:
delivery_class = str(delivery_class or "").strip().lower()
if delivery_class == "request":
bound_key = self._bound_mailbox_key(recipient_id, "requests")
if bound_key:
return bound_key
return self._mailbox_key("requests", str(recipient_id or "").strip())
if delivery_class == "shared":
token = str(recipient_token or "").strip()
if not token:
raise ValueError("recipient_token required for shared delivery")
return self._hashed_mailbox_token(token)
raise ValueError("Unsupported delivery_class")
def claim_mailbox_keys(self, agent_id: str, claims: list[dict[str, Any]]) -> list[str]:
keys: list[str] = []
for claim in claims[:32]:
keys.extend(self._mailbox_keys_for_claim(agent_id, claim))
return list(dict.fromkeys(keys))
def _legacy_mailbox_token(self, agent_id: str, epoch: int) -> str:
raw = f"sb_dm|{epoch}|{agent_id}".encode("utf-8")
return hashlib.sha256(raw).hexdigest()
def _legacy_token_candidates(self, agent_id: str) -> list[str]:
epoch = self._epoch_bucket()
raw = [self._legacy_mailbox_token(agent_id, epoch), self._legacy_mailbox_token(agent_id, epoch - 1)]
peppered = [self._pepper_token(token) for token in raw]
return list(dict.fromkeys(peppered + raw))
def _save(self) -> None:
"""Mark dirty and schedule a coalesced disk write."""
self._dirty = True
if not RELAY_FILE.exists():
self._flush()
return
with self._lock:
if self._save_timer is None or not self._save_timer.is_alive():
self._save_timer = threading.Timer(self._SAVE_INTERVAL, self._flush)
self._save_timer.daemon = True
self._save_timer.start()
def _prune_stale_metadata(self) -> None:
"""Remove expired DH keys, prekey bundles, and mailbox bindings."""
now = time.time()
settings = self._settings()
key_ttl = max(1, int(getattr(settings, "MESH_DM_KEY_TTL_DAYS", 30) or 30)) * 86400
binding_ttl = max(1, int(getattr(settings, "MESH_DM_BINDING_TTL_DAYS", 7) or 7)) * 86400
stale_keys = [
aid for aid, entry in self._dh_keys.items()
if (now - float(entry.get("timestamp", 0) or 0)) > key_ttl
]
for aid in stale_keys:
del self._dh_keys[aid]
stale_bundles = [
aid for aid, entry in self._prekey_bundles.items()
if (now - float(entry.get("updated_at", entry.get("timestamp", 0)) or 0)) > key_ttl
]
for aid in stale_bundles:
del self._prekey_bundles[aid]
stale_agents: list[str] = []
for agent_id, kinds in self._mailbox_bindings.items():
expired_kinds = [
k for k, v in kinds.items()
if isinstance(v, dict) and (now - float(v.get("last_used", 0) or 0)) > binding_ttl
]
for k in expired_kinds:
del kinds[k]
if not kinds:
stale_agents.append(agent_id)
for agent_id in stale_agents:
del self._mailbox_bindings[agent_id]
def _metadata_persist_enabled(self) -> bool:
return bool(getattr(self._settings(), "MESH_DM_METADATA_PERSIST", True))
def _flush(self) -> None:
"""Actually write to disk (called by timer or atexit)."""
if not self._dirty:
return
try:
self._prune_stale_metadata()
DATA_DIR.mkdir(parents=True, exist_ok=True)
payload: dict[str, Any] = {
"saved_at": int(time.time()),
"dh_keys": self._dh_keys,
"prekey_bundles": self._prekey_bundles,
"witnesses": self._witnesses,
"blocks": {k: sorted(v) for k, v in self._blocks.items()},
"nonce_cache": dict(self._nonce_cache),
"stats": self._stats,
}
if self._metadata_persist_enabled():
payload["mailbox_bindings"] = self._mailbox_bindings
if self._persist_spool_enabled():
payload["mailboxes"] = {
key: [m.__dict__ for m in msgs] for key, msgs in self._mailboxes.items()
}
write_secure_json(RELAY_FILE, payload)
self._dirty = False
except Exception:
pass
def _load(self) -> None:
if not RELAY_FILE.exists():
return
try:
data = read_secure_json(RELAY_FILE, lambda: {})
except Exception:
return
if self._persist_spool_enabled():
mailboxes = data.get("mailboxes", {})
if isinstance(mailboxes, dict):
for key, items in mailboxes.items():
if not isinstance(items, list):
continue
restored: list[DMMessage] = []
for item in items:
try:
restored.append(
DMMessage(
sender_id=str(item.get("sender_id", "")),
ciphertext=str(item.get("ciphertext", "")),
timestamp=float(item.get("timestamp", 0)),
msg_id=str(item.get("msg_id", "")),
delivery_class=str(item.get("delivery_class", "shared")),
sender_seal=str(item.get("sender_seal", "")),
relay_salt=str(item.get("relay_salt", "") or ""),
sender_block_ref=str(item.get("sender_block_ref", "") or ""),
payload_format=str(item.get("payload_format", item.get("format", "dm1")) or "dm1"),
session_welcome=str(item.get("session_welcome", "") or ""),
)
)
except Exception:
continue
for message in restored:
if not message.sender_block_ref:
message.sender_block_ref = self._message_block_ref(message)
if restored:
self._mailboxes[key] = restored
dh_keys = data.get("dh_keys", {})
if isinstance(dh_keys, dict):
self._dh_keys = {str(k): dict(v) for k, v in dh_keys.items() if isinstance(v, dict)}
prekey_bundles = data.get("prekey_bundles", {})
if isinstance(prekey_bundles, dict):
self._prekey_bundles = {
str(k): dict(v) for k, v in prekey_bundles.items() if isinstance(v, dict)
}
mailbox_bindings = data.get("mailbox_bindings", {})
if isinstance(mailbox_bindings, dict):
self._mailbox_bindings = defaultdict(
dict,
{
str(agent_id): {
str(kind): str(token_hash)
for kind, token_hash in dict(bindings or {}).items()
if str(token_hash or "").strip()
}
for agent_id, bindings in mailbox_bindings.items()
if isinstance(bindings, dict)
},
)
witnesses = data.get("witnesses", {})
if isinstance(witnesses, dict):
self._witnesses = defaultdict(
list,
{
str(k): list(v)
for k, v in witnesses.items()
if isinstance(v, list)
},
)
blocks = data.get("blocks", {})
if isinstance(blocks, dict):
for key, values in blocks.items():
if isinstance(values, list):
self._blocks[str(key)] = {
self._canonical_blocked_id(str(v))
for v in values
if str(v or "").strip()
}
nonce_cache = data.get("nonce_cache", {})
if isinstance(nonce_cache, dict):
now = time.time()
restored = sorted(
(
(str(k), float(v))
for k, v in nonce_cache.items()
if float(v) > now
),
key=lambda item: item[1],
)
self._nonce_cache = OrderedDict(restored)
stats = data.get("stats", {})
if isinstance(stats, dict):
self._stats = {str(k): int(v) for k, v in stats.items() if isinstance(v, (int, float))}
self._stats["messages_in_memory"] = sum(len(v) for v in self._mailboxes.values())
def _bundle_fingerprint(
self,
*,
dh_pub_key: str,
dh_algo: str,
public_key: str,
public_key_algo: str,
protocol_version: str,
) -> str:
material = "|".join(
[
dh_pub_key,
dh_algo,
public_key,
public_key_algo,
protocol_version,
]
)
return hashlib.sha256(material.encode("utf-8")).hexdigest()
def register_dh_key(
self,
agent_id: str,
dh_pub_key: str,
dh_algo: str,
timestamp: int,
signature: str,
public_key: str,
public_key_algo: str,
protocol_version: str,
sequence: int,
) -> tuple[bool, str, dict[str, Any] | None]:
"""Register/update an agent's DH public key bundle with replay protection."""
fingerprint = self._bundle_fingerprint(
dh_pub_key=dh_pub_key,
dh_algo=dh_algo,
public_key=public_key,
public_key_algo=public_key_algo,
protocol_version=protocol_version,
)
with self._lock:
existing = self._dh_keys.get(agent_id)
if existing:
existing_seq = int(existing.get("sequence", 0) or 0)
existing_ts = int(existing.get("timestamp", 0) or 0)
if sequence <= existing_seq:
metrics_inc("dm_key_replay")
return False, "DM key replay or rollback rejected", None
if timestamp < existing_ts:
metrics_inc("dm_key_stale")
return False, "DM key timestamp is older than the current bundle", None
self._dh_keys[agent_id] = {
"dh_pub_key": dh_pub_key,
"dh_algo": dh_algo,
"timestamp": timestamp,
"signature": signature,
"public_key": public_key,
"public_key_algo": public_key_algo,
"protocol_version": protocol_version,
"sequence": sequence,
"bundle_fingerprint": fingerprint,
}
self._save()
return True, "ok", {
"accepted_sequence": sequence,
"bundle_fingerprint": fingerprint,
}
def get_dh_key(self, agent_id: str) -> dict[str, Any] | None:
return self._dh_keys.get(agent_id)
def register_prekey_bundle(
self,
agent_id: str,
bundle: dict[str, Any],
signature: str,
public_key: str,
public_key_algo: str,
protocol_version: str,
sequence: int,
) -> tuple[bool, str, dict[str, Any] | None]:
ok, reason = _validate_bundle_record(
{
"bundle": bundle,
"public_key": public_key,
"agent_id": agent_id,
}
)
if not ok:
return False, reason, None
with self._lock:
existing = self._prekey_bundles.get(agent_id)
if existing:
existing_seq = int(existing.get("sequence", 0) or 0)
if sequence <= existing_seq:
return False, "Prekey bundle replay or rollback rejected", None
stored = {
"bundle": dict(bundle or {}),
"signature": signature,
"public_key": public_key,
"public_key_algo": public_key_algo,
"protocol_version": protocol_version,
"sequence": int(sequence),
"updated_at": int(time.time()),
}
self._prekey_bundles[agent_id] = stored
self._save()
return True, "ok", {"accepted_sequence": int(sequence)}
def get_prekey_bundle(self, agent_id: str) -> dict[str, Any] | None:
stored = self._prekey_bundles.get(agent_id)
if not stored:
return None
return dict(stored)
def consume_one_time_prekey(self, agent_id: str) -> dict[str, Any] | None:
"""Atomically claim the next published one-time prekey for a peer bundle."""
claimed: dict[str, Any] | None = None
with self._lock:
stored = self._prekey_bundles.get(agent_id)
if not stored:
return None
bundle = dict(stored.get("bundle") or {})
otks = list(bundle.get("one_time_prekeys") or [])
if not otks:
return dict(stored)
claimed = dict(otks.pop(0) or {})
bundle["one_time_prekeys"] = otks
bundle["one_time_prekey_count"] = len(otks)
stored = dict(stored)
stored["bundle"] = bundle
stored["updated_at"] = int(time.time())
self._prekey_bundles[agent_id] = stored
self._save()
result = dict(stored)
result["claimed_one_time_prekey"] = claimed
return result
def _prune_witnesses(self, target_id: str, ttl_days: int = 30) -> None:
cutoff = time.time() - (ttl_days * 86400)
self._witnesses[target_id] = [
w for w in self._witnesses.get(target_id, []) if float(w.get("timestamp", 0)) >= cutoff
]
if not self._witnesses[target_id]:
del self._witnesses[target_id]
def record_witness(
self,
witness_id: str,
target_id: str,
dh_pub_key: str,
timestamp: int,
) -> tuple[bool, str]:
if not witness_id or not target_id or not dh_pub_key:
return False, "Missing witness_id, target_id, or dh_pub_key"
if witness_id == target_id:
return False, "Cannot witness yourself"
with self._lock:
self._prune_witnesses(target_id)
entries = self._witnesses.get(target_id, [])
for entry in entries:
if entry.get("witness_id") == witness_id and entry.get("dh_pub_key") == dh_pub_key:
return False, "Duplicate witness"
entries.append(
{
"witness_id": witness_id,
"dh_pub_key": dh_pub_key,
"timestamp": int(timestamp),
}
)
self._witnesses[target_id] = entries[-50:]
self._save()
return True, "ok"
def get_witnesses(self, target_id: str, dh_pub_key: str | None = None, limit: int = 5) -> list[dict]:
with self._lock:
self._prune_witnesses(target_id)
entries = list(self._witnesses.get(target_id, []))
if dh_pub_key:
entries = [e for e in entries if e.get("dh_pub_key") == dh_pub_key]
entries = sorted(entries, key=lambda e: e.get("timestamp", 0), reverse=True)
return entries[: max(1, limit)]
def _epoch_bucket(self, ts: float | None = None) -> int:
now = ts if ts is not None else time.time()
return int(now // EPOCH_SECONDS)
def _mailbox_limit_for_class(self, delivery_class: str) -> int:
if delivery_class == "request":
return self._request_mailbox_limit()
if delivery_class == "shared":
return self._shared_mailbox_limit()
return self._self_mailbox_limit()
def _cleanup_expired(self) -> bool:
now = time.time()
changed = False
for mailbox_id in list(self._mailboxes):
fresh = [m for m in self._mailboxes[mailbox_id] if now - m.timestamp < TTL_SECONDS]
if len(fresh) != len(self._mailboxes[mailbox_id]):
changed = True
self._mailboxes[mailbox_id] = fresh
if not self._mailboxes[mailbox_id]:
del self._mailboxes[mailbox_id]
changed = True
self._stats["messages_in_memory"] = sum(len(v) for v in self._mailboxes.values())
return changed
def consume_nonce(self, agent_id: str, nonce: str, timestamp: int) -> tuple[bool, str]:
nonce = str(nonce or "").strip()
if not nonce:
return False, "Missing nonce"
now = time.time()
with self._lock:
self._nonce_cache = OrderedDict(
(key, expiry)
for key, expiry in self._nonce_cache.items()
if float(expiry) > now
)
key = f"{agent_id}:{nonce}"
if key in self._nonce_cache:
metrics_inc("dm_nonce_replay")
return False, "nonce replay detected"
if len(self._nonce_cache) >= self._nonce_cache_max_entries():
metrics_inc("dm_nonce_cache_full")
return False, "nonce cache at capacity"
expiry = max(now + self._nonce_ttl_seconds(), float(timestamp) + self._nonce_ttl_seconds())
self._nonce_cache[key] = expiry
self._nonce_cache.move_to_end(key)
self._save()
return True, "ok"
def deposit(
self,
*,
sender_id: str,
raw_sender_id: str = "",
recipient_id: str = "",
ciphertext: str,
msg_id: str = "",
delivery_class: str,
recipient_token: str | None = None,
sender_seal: str = "",
relay_salt: str = "",
sender_token_hash: str = "",
payload_format: str = "dm1",
session_welcome: str = "",
) -> dict[str, Any]:
with self._lock:
authority_sender = str(raw_sender_id or sender_id or "").strip()
sender_block_ref = self._sender_block_ref(authority_sender)
if recipient_id and sender_block_ref in self._blocks.get(recipient_id, set()):
metrics_inc("dm_drop_blocked")
return {"ok": False, "detail": "Recipient is not accepting your messages"}
if len(ciphertext) > int(self._settings().MESH_DM_MAX_MSG_BYTES):
metrics_inc("dm_drop_oversize")
return {
"ok": False,
"detail": f"Message too large ({len(ciphertext)} > {int(self._settings().MESH_DM_MAX_MSG_BYTES)})",
}
self._cleanup_expired()
if delivery_class == "request":
mailbox_key = self._mailbox_key("requests", recipient_id)
elif delivery_class == "shared":
if not recipient_token:
metrics_inc("dm_claim_invalid")
return {"ok": False, "detail": "recipient_token required for shared delivery"}
mailbox_key = self._hashed_mailbox_token(recipient_token)
else:
return {"ok": False, "detail": "Unsupported delivery_class"}
if len(self._mailboxes[mailbox_key]) >= self._mailbox_limit_for_class(delivery_class):
metrics_inc("dm_drop_full")
return {"ok": False, "detail": "Recipient mailbox full"}
if not msg_id:
msg_id = f"dm_{int(time.time() * 1000)}_{secrets.token_hex(6)}"
elif any(m.msg_id == msg_id for m in self._mailboxes[mailbox_key]):
return {"ok": True, "msg_id": msg_id}
relay_sender_id = (
f"sender_token:{sender_token_hash}"
if sender_token_hash and delivery_class == "shared"
else sender_id
)
self._mailboxes[mailbox_key].append(
DMMessage(
sender_id=relay_sender_id,
ciphertext=ciphertext,
timestamp=time.time(),
msg_id=msg_id,
delivery_class=delivery_class,
sender_seal=sender_seal,
sender_block_ref=sender_block_ref,
payload_format=str(payload_format or "dm1"),
session_welcome=str(session_welcome or ""),
)
)
self._stats["messages_in_memory"] = sum(len(v) for v in self._mailboxes.values())
self._save()
return {"ok": True, "msg_id": msg_id}
def is_blocked(self, recipient_id: str, sender_id: str) -> bool:
with self._lock:
blocked_ref = self._sender_block_ref(sender_id)
if not recipient_id or not blocked_ref:
return False
return blocked_ref in self._blocks.get(recipient_id, set())
def _collect_from_keys(self, keys: list[str], *, destructive: bool) -> list[dict[str, Any]]:
messages: list[DMMessage] = []
seen: set[str] = set()
for key in keys:
mailbox = self._mailboxes.pop(key, []) if destructive else list(self._mailboxes.get(key, []))
for message in mailbox:
if message.msg_id in seen:
continue
seen.add(message.msg_id)
messages.append(message)
if destructive:
self._stats["messages_in_memory"] = sum(len(v) for v in self._mailboxes.values())
self._save()
return [
{
"sender_id": message.sender_id,
"ciphertext": message.ciphertext,
"timestamp": message.timestamp,
"msg_id": message.msg_id,
"delivery_class": message.delivery_class,
"sender_seal": message.sender_seal,
"format": message.payload_format,
"session_welcome": message.session_welcome,
}
for message in sorted(messages, key=lambda item: item.timestamp)
]
def collect_claims(self, agent_id: str, claims: list[dict[str, Any]]) -> list[dict[str, Any]]:
with self._lock:
self._cleanup_expired()
keys: list[str] = []
for claim in claims[:32]:
keys.extend(self._mailbox_keys_for_claim(agent_id, claim))
return self._collect_from_keys(list(dict.fromkeys(keys)), destructive=True)
def count_claims(self, agent_id: str, claims: list[dict[str, Any]]) -> int:
with self._lock:
self._cleanup_expired()
keys: list[str] = []
for claim in claims[:32]:
keys.extend(self._mailbox_keys_for_claim(agent_id, claim))
messages = self._collect_from_keys(list(dict.fromkeys(keys)), destructive=False)
return len(messages)
def claim_message_ids(self, agent_id: str, claims: list[dict[str, Any]]) -> set[str]:
with self._lock:
self._cleanup_expired()
keys: list[str] = []
for claim in claims[:32]:
keys.extend(self._mailbox_keys_for_claim(agent_id, claim))
messages = self._collect_from_keys(list(dict.fromkeys(keys)), destructive=False)
return {
str(message.get("msg_id", "") or "")
for message in messages
if str(message.get("msg_id", "") or "")
}
def collect_legacy(self, agent_id: str | None = None, agent_token: str | None = None) -> list[dict[str, Any]]:
with self._lock:
self._cleanup_expired()
if not agent_token:
return []
keys = [self._pepper_token(agent_token), agent_token]
return self._collect_from_keys(list(dict.fromkeys(keys)), destructive=True)
def count_legacy(self, agent_id: str | None = None, agent_token: str | None = None) -> int:
with self._lock:
self._cleanup_expired()
if not agent_token:
return 0
keys = [self._pepper_token(agent_token), agent_token]
return len(self._collect_from_keys(list(dict.fromkeys(keys)), destructive=False))
def block(self, agent_id: str, blocked_id: str) -> None:
with self._lock:
blocked_ref = self._canonical_blocked_id(blocked_id)
if not blocked_ref:
return
self._blocks[agent_id].add(blocked_ref)
purge_keys = self._legacy_token_candidates(agent_id)
bound_request = self._bound_mailbox_key(agent_id, "requests")
bound_self = self._bound_mailbox_key(agent_id, "self")
if bound_request:
purge_keys.append(bound_request)
if bound_self:
purge_keys.append(bound_self)
purge_keys.extend(
[
self._mailbox_key("self", agent_id),
self._mailbox_key("requests", agent_id),
self._mailbox_key("self", agent_id, self._epoch_bucket() - 1),
self._mailbox_key("requests", agent_id, self._epoch_bucket() - 1),
]
)
for key in set(purge_keys):
if key in self._mailboxes:
self._mailboxes[key] = [
m for m in self._mailboxes[key] if self._message_block_ref(m) != blocked_ref
]
self._stats["messages_in_memory"] = sum(len(v) for v in self._mailboxes.values())
self._save()
def unblock(self, agent_id: str, blocked_id: str) -> None:
with self._lock:
blocked_ref = self._canonical_blocked_id(blocked_id)
if not blocked_ref:
return
self._blocks[agent_id].discard(blocked_ref)
self._save()
dm_relay = DMRelay()
backend/services/mesh/mesh_gate_mls.py
+1352
View File
File diff suppressed because it is too large Load Diff
backend/services/mesh/mesh_hashchain.py
+2034
View File
File diff suppressed because it is too large Load Diff
backend/services/mesh/mesh_ibf.py
+186
View File
@@ -0,0 +1,186 @@
from __future__ import annotations
import base64
import hashlib
from dataclasses import dataclass
from typing import Iterable, List, Tuple
KEY_SIZE = 32
DEFAULT_SEEDS = [0x243F6A8885A308D3, 0x13198A2E03707344, 0xA4093822299F31D0]
FINGERPRINT_SEED = 0xC0FFEE1234567890
def _safe_int(val, default=0) -> int:
try:
return int(val)
except (TypeError, ValueError):
return default
def _hash64(data: bytes, seed: int) -> int:
key = seed.to_bytes(8, "little", signed=False)
digest = hashlib.blake2b(data, digest_size=8, key=key).digest()
return int.from_bytes(digest, "little", signed=False)
def _fingerprint(data: bytes) -> int:
key = FINGERPRINT_SEED.to_bytes(8, "little", signed=False)
digest = hashlib.blake2b(data, digest_size=8, key=key).digest()
return int.from_bytes(digest, "little", signed=False)
def _xor_bytes(a: bytes, b: bytes) -> bytes:
return bytes(x ^ y for x, y in zip(a, b))
def _ensure_key(key: bytes) -> bytes:
if len(key) != KEY_SIZE:
raise ValueError(f"IBF key must be {KEY_SIZE} bytes")
return key
def _b64_encode(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def _b64_decode(data: str) -> bytes:
return base64.b64decode(data.encode("ascii"))
@dataclass
class IBLTCell:
count: int = 0
key_xor: bytes = b"\x00" * KEY_SIZE
hash_xor: int = 0
def add(self, key: bytes, sign: int) -> None:
self.count += sign
self.key_xor = _xor_bytes(self.key_xor, key)
self.hash_xor ^= _fingerprint(key)
class IBLT:
def __init__(self, size: int, seeds: List[int] | None = None) -> None:
if size <= 0:
raise ValueError("IBLT size must be positive")
self.size = size
self.seeds = seeds or list(DEFAULT_SEEDS)
self.cells: List[IBLTCell] = [IBLTCell() for _ in range(size)]
def _indexes(self, key: bytes) -> List[int]:
key = _ensure_key(key)
return [(_hash64(key, seed) % self.size) for seed in self.seeds]
def insert(self, key: bytes) -> None:
key = _ensure_key(key)
for idx in self._indexes(key):
self.cells[idx].add(key, 1)
def delete(self, key: bytes) -> None:
key = _ensure_key(key)
for idx in self._indexes(key):
self.cells[idx].add(key, -1)
def subtract(self, other: "IBLT") -> "IBLT":
if self.size != other.size or self.seeds != other.seeds:
raise ValueError("IBLT mismatch; size or seeds differ")
out = IBLT(self.size, self.seeds)
for i, cell in enumerate(self.cells):
other_cell = other.cells[i]
out.cells[i] = IBLTCell(
count=cell.count - other_cell.count,
key_xor=_xor_bytes(cell.key_xor, other_cell.key_xor),
hash_xor=cell.hash_xor ^ other_cell.hash_xor,
)
return out
def decode(self) -> Tuple[bool, List[bytes], List[bytes]]:
plus: List[bytes] = []
minus: List[bytes] = []
stack = [i for i, c in enumerate(self.cells) if abs(c.count) == 1]
while stack:
idx = stack.pop()
cell = self.cells[idx]
if abs(cell.count) != 1:
continue
key = cell.key_xor
if _fingerprint(key) != cell.hash_xor:
continue
sign = 1 if cell.count == 1 else -1
if sign == 1:
plus.append(key)
else:
minus.append(key)
for j in self._indexes(key):
if j == idx:
continue
self.cells[j].add(key, -sign)
if abs(self.cells[j].count) == 1:
stack.append(j)
self.cells[idx] = IBLTCell()
success = all(
c.count == 0 and c.hash_xor == 0 and c.key_xor == b"\x00" * KEY_SIZE
for c in self.cells
)
return success, plus, minus
def to_compact_dict(self) -> dict:
return {
"m": self.size,
"s": self.seeds,
"c": [[cell.count, _b64_encode(cell.key_xor), cell.hash_xor] for cell in self.cells],
}
@classmethod
def from_compact_dict(cls, data: dict) -> "IBLT":
size = _safe_int(data.get("m", 0) or 0)
seeds = data.get("s") or list(DEFAULT_SEEDS)
cells = data.get("c") or []
iblt = cls(size, list(seeds))
if len(cells) != size:
raise ValueError("IBLT cell count mismatch")
for i, raw in enumerate(cells):
count, key_b64, hash_xor = raw
iblt.cells[i] = IBLTCell(
count=_safe_int(count, 0),
key_xor=_b64_decode(str(key_b64)),
hash_xor=_safe_int(hash_xor, 0),
)
return iblt
def build_iblt(keys: Iterable[bytes], size: int) -> IBLT:
iblt = IBLT(size)
for key in keys:
iblt.insert(key)
return iblt
def minhash_sketch(keys: Iterable[bytes], k: int) -> List[int]:
if k <= 0:
return []
mins: List[int] = []
for key in keys:
h = _hash64(key, 0x9E3779B97F4A7C15)
if len(mins) < k:
mins.append(h)
mins.sort()
elif h < mins[-1]:
mins[-1] = h
mins.sort()
return mins
def minhash_similarity(a: Iterable[int], b: Iterable[int]) -> float:
a_list = list(a)
b_list = list(b)
if not a_list or not b_list:
return 0.0
k = min(len(a_list), len(b_list))
if k <= 0:
return 0.0
a_set = set(a_list[:k])
b_set = set(b_list[:k])
return len(a_set & b_set) / float(k)
backend/services/mesh/mesh_infonet_sync_support.py
+115
View File
@@ -0,0 +1,115 @@
from __future__ import annotations
import time
from dataclasses import asdict, dataclass
from services.mesh.mesh_peer_store import PeerRecord
@dataclass(frozen=True)
class SyncWorkerState:
last_sync_started_at: int = 0
last_sync_finished_at: int = 0
last_sync_ok_at: int = 0
next_sync_due_at: int = 0
last_peer_url: str = ""
last_error: str = ""
last_outcome: str = "idle"
current_head: str = ""
fork_detected: bool = False
consecutive_failures: int = 0
def to_dict(self) -> dict[str, object]:
return asdict(self)
def eligible_sync_peers(records: list[PeerRecord], *, now: float | None = None) -> list[PeerRecord]:
current_time = int(now if now is not None else time.time())
candidates = [
record
for record in records
if record.bucket == "sync" and record.enabled and int(record.cooldown_until or 0) <= current_time
]
return sorted(
candidates,
key=lambda record: (
-int(record.last_sync_ok_at or 0),
int(record.failure_count or 0),
int(record.added_at or 0),
record.peer_url,
),
)
def begin_sync(
state: SyncWorkerState,
*,
peer_url: str = "",
current_head: str = "",
now: float | None = None,
) -> SyncWorkerState:
timestamp = int(now if now is not None else time.time())
return SyncWorkerState(
last_sync_started_at=timestamp,
last_sync_finished_at=state.last_sync_finished_at,
last_sync_ok_at=state.last_sync_ok_at,
next_sync_due_at=state.next_sync_due_at,
last_peer_url=peer_url or state.last_peer_url,
last_error="",
last_outcome="running",
current_head=current_head or state.current_head,
fork_detected=False,
consecutive_failures=state.consecutive_failures,
)
def finish_sync(
state: SyncWorkerState,
*,
ok: bool,
peer_url: str = "",
current_head: str = "",
error: str = "",
fork_detected: bool = False,
now: float | None = None,
interval_s: int = 300,
failure_backoff_s: int = 60,
) -> SyncWorkerState:
timestamp = int(now if now is not None else time.time())
if ok:
return SyncWorkerState(
last_sync_started_at=state.last_sync_started_at,
last_sync_finished_at=timestamp,
last_sync_ok_at=timestamp,
next_sync_due_at=timestamp + max(0, int(interval_s or 0)),
last_peer_url=peer_url or state.last_peer_url,
last_error="",
last_outcome="ok",
current_head=current_head or state.current_head,
fork_detected=bool(fork_detected),
consecutive_failures=0,
)
return SyncWorkerState(
last_sync_started_at=state.last_sync_started_at,
last_sync_finished_at=timestamp,
last_sync_ok_at=state.last_sync_ok_at,
next_sync_due_at=timestamp + max(0, int(failure_backoff_s or 0)),
last_peer_url=peer_url or state.last_peer_url,
last_error=str(error or "").strip(),
last_outcome="fork" if fork_detected else "error",
current_head=current_head or state.current_head,
fork_detected=bool(fork_detected),
consecutive_failures=state.consecutive_failures + 1,
)
def should_run_sync(
state: SyncWorkerState,
*,
now: float | None = None,
) -> bool:
current_time = int(now if now is not None else time.time())
if state.last_outcome == "running":
return False
return int(state.next_sync_due_at or 0) <= current_time
backend/services/mesh/mesh_merkle.py
+74
View File
@@ -0,0 +1,74 @@
from __future__ import annotations
import hashlib
from typing import Any
def _hash_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def hash_leaf(value: str) -> str:
return _hash_bytes(value.encode("utf-8"))
def hash_pair(left: str, right: str) -> str:
return _hash_bytes(f"{left}{right}".encode("utf-8"))
def build_merkle_levels(leaves: list[str]) -> list[list[str]]:
if not leaves:
return []
level = [hash_leaf(leaf) for leaf in leaves]
levels = [level]
while len(level) > 1:
next_level: list[str] = []
for idx in range(0, len(level), 2):
left = level[idx]
right = level[idx + 1] if idx + 1 < len(level) else left
next_level.append(hash_pair(left, right))
level = next_level
levels.append(level)
return levels
def merkle_root(leaves: list[str]) -> str:
levels = build_merkle_levels(leaves)
if not levels:
return ""
return levels[-1][0]
def merkle_proof_from_levels(levels: list[list[str]], index: int) -> list[dict[str, Any]]:
if not levels:
return []
if index < 0 or index >= len(levels[0]):
return []
proof: list[dict[str, Any]] = []
idx = index
for level in levels[:-1]:
is_right = idx % 2 == 1
sibling_idx = idx - 1 if is_right else idx + 1
if sibling_idx >= len(level):
sibling_hash = level[idx]
else:
sibling_hash = level[sibling_idx]
proof.append({"hash": sibling_hash, "side": "left" if is_right else "right"})
idx //= 2
return proof
def verify_merkle_proof(
leaf_value: str, index: int, proof: list[dict[str, Any]], root: str
) -> bool:
current = hash_leaf(leaf_value)
idx = index
for step in proof:
sibling = str(step.get("hash", ""))
side = str(step.get("side", "right")).lower()
if side == "left":
current = hash_pair(sibling, current)
else:
current = hash_pair(current, sibling)
idx //= 2
return current == root
backend/services/mesh/mesh_metrics.py
+25
View File
@@ -0,0 +1,25 @@
"""Lightweight metrics for mesh protocol health signals."""
from __future__ import annotations
import threading
import time
_lock = threading.Lock()
_metrics: dict[str, int] = {}
_last_updated: float = 0.0
def increment(name: str, count: int = 1) -> None:
global _last_updated
with _lock:
_metrics[name] = _metrics.get(name, 0) + count
_last_updated = time.time()
def snapshot() -> dict:
with _lock:
return {
"updated_at": _last_updated,
"counters": dict(_metrics),
}
backend/services/mesh/mesh_oracle.py
+899
View File
@@ -0,0 +1,899 @@
"""Oracle System — prediction-backed truth arbitration for the mesh.
Oracle Rep is a separate reputation tier earned ONLY by:
1. Correctly predicting outcomes on Kalshi/Polymarket-sourced markets
2. Winning truth stakes on posts/comments
Oracle Rep can be staked on posts to protect them from mob downvoting.
Other oracles can counter-stake. After the stake period (1-7 days),
whichever side has more oracle rep staked wins. Losers' rep is divided
proportionally among winners.
Scoring formula for predictions:
oracle_rep_earned = 1.0 - probability_of_chosen_outcome / 100
- Bet YES at 99% → earn 0.01 (trivial, everyone knew)
- Bet YES at 50% → earn 0.50 (genuine uncertainty, real insight)
- Bet YES at 10% → earn 0.90 (contrarian genius if correct)
Designed for AI game theory: this mechanism works identically
whether participants are humans, AI agents, or a mix.
Persistence: JSON files in backend/data/ (auto-saved on change).
"""
import json
import time
import logging
import secrets
import threading
import atexit
from pathlib import Path
from typing import Optional
logger = logging.getLogger("services.mesh_oracle")
DATA_DIR = Path(__file__).resolve().parents[2] / "data"
ORACLE_FILE = DATA_DIR / "oracle_ledger.json"
# ─── Constants ────────────────────────────────────────────────────────────
MIN_STAKE_DAYS = 1 # Minimum stake duration
MAX_STAKE_DAYS = 7 # Maximum stake duration
GRACE_PERIOD_HOURS = 24 # Counter-stakers get 24h after any new stake
ORACLE_DECAY_DAYS = 90 # Oracle rep decays over 90 days like regular rep
class OracleLedger:
"""Oracle reputation ledger — predictions, stakes, and truth arbitration.
Storage:
oracle_rep: {node_id: float} — current oracle rep balances
predictions: [{node_id, market_title, side, probability_at_bet, timestamp, resolved, correct, rep_earned}]
stakes: [{stake_id, message_id, poster_id, staker_id, side ("truth"|"false"),
amount, duration_days, created_at, expires_at, resolved}]
prediction_log: [{node_id, market_title, side, probability_at_bet, rep_earned, timestamp}]
"""
def __init__(self):
self.oracle_rep: dict[str, float] = {}
self.predictions: list[dict] = []
self.market_stakes: list[dict] = [] # Rep staked on prediction markets
self.stakes: list[dict] = [] # Truth stakes on posts (separate system)
self.prediction_log: list[dict] = [] # Public log of all predictions
self._dirty = False
self._save_lock = threading.Lock()
self._save_timer: threading.Timer | None = None
self._SAVE_INTERVAL = 5.0
atexit.register(self._flush)
self._load()
# ─── Persistence ──────────────────────────────────────────────────
def _load(self):
if ORACLE_FILE.exists():
try:
data = json.loads(ORACLE_FILE.read_text(encoding="utf-8"))
self.oracle_rep = data.get("oracle_rep", {})
self.predictions = data.get("predictions", [])
self.market_stakes = data.get("market_stakes", [])
self.stakes = data.get("stakes", [])
self.prediction_log = data.get("prediction_log", [])
logger.info(
f"Loaded oracle ledger: {len(self.oracle_rep)} oracles, "
f"{len(self.predictions)} predictions, "
f"{len(self.market_stakes)} market stakes, {len(self.stakes)} truth stakes"
)
except Exception as e:
logger.error(f"Failed to load oracle ledger: {e}")
def _save(self):
"""Mark dirty and schedule a coalesced disk write."""
self._dirty = True
with self._save_lock:
if self._save_timer is None or not self._save_timer.is_alive():
self._save_timer = threading.Timer(self._SAVE_INTERVAL, self._flush)
self._save_timer.daemon = True
self._save_timer.start()
def _flush(self):
"""Actually write to disk (called by timer or atexit)."""
if not self._dirty:
return
try:
DATA_DIR.mkdir(parents=True, exist_ok=True)
data = {
"oracle_rep": self.oracle_rep,
"predictions": self.predictions,
"market_stakes": self.market_stakes,
"stakes": self.stakes,
"prediction_log": self.prediction_log,
}
ORACLE_FILE.write_text(json.dumps(data, indent=2), encoding="utf-8")
self._dirty = False
except Exception as e:
logger.error(f"Failed to save oracle ledger: {e}")
# ─── Oracle Rep ───────────────────────────────────────────────────
def get_oracle_rep(self, node_id: str) -> float:
"""Get current oracle rep for a node (excludes locked/staked amount)."""
total = self.oracle_rep.get(node_id, 0.0)
# Subtract locked truth stakes on posts
locked = sum(
s["amount"]
for s in self.stakes
if s["staker_id"] == node_id and not s.get("resolved", False)
)
# Subtract locked market stakes
locked += sum(
s["amount"]
for s in self.market_stakes
if s["node_id"] == node_id and not s.get("resolved", False)
)
return round(max(0, total - locked), 3)
def get_total_oracle_rep(self, node_id: str) -> float:
"""Get total oracle rep including locked stakes."""
return round(self.oracle_rep.get(node_id, 0.0), 3)
def _add_oracle_rep(self, node_id: str, amount: float):
"""Add oracle rep to a node."""
self.oracle_rep[node_id] = self.oracle_rep.get(node_id, 0.0) + amount
def _remove_oracle_rep(self, node_id: str, amount: float):
"""Remove oracle rep from a node (floor at 0)."""
self.oracle_rep[node_id] = max(0, self.oracle_rep.get(node_id, 0.0) - amount)
# ─── Predictions ──────────────────────────────────────────────────
def place_prediction(
self, node_id: str, market_title: str, side: str, probability_at_bet: float
) -> tuple[bool, str]:
"""Place a FREE prediction on a market outcome (no rep risked).
Args:
node_id: Predictor's node ID
market_title: Title of the prediction market
side: "yes", "no", or any outcome name for multi-outcome markets
probability_at_bet: Current probability (0-100) of the chosen side
Returns (success, detail)
"""
if not side or not side.strip():
return False, "Side is required"
if not (0 <= probability_at_bet <= 100):
return False, "Probability must be 0-100"
# Check for duplicate predictions on same market
existing = [
p
for p in self.predictions
if p["node_id"] == node_id
and p["market_title"] == market_title
and not p.get("resolved", False)
]
if existing:
return (
False,
f"You already have an active prediction on '{market_title}'. Your decision was FINAL.",
)
# Also check market stakes — can't free-pick AND stake on same market
existing_stake = [
s
for s in self.market_stakes
if s["node_id"] == node_id
and s["market_title"] == market_title
and not s.get("resolved", False)
]
if existing_stake:
return (
False,
f"You already have a STAKED prediction on '{market_title}'. Your decision was FINAL.",
)
self.predictions.append(
{
"prediction_id": secrets.token_hex(6),
"node_id": node_id,
"market_title": market_title,
"side": side,
"probability_at_bet": probability_at_bet,
"timestamp": time.time(),
"resolved": False,
"correct": None,
"rep_earned": 0.0,
}
)
self._save()
# Potential rep = contrarianism score
potential = round(1.0 - probability_at_bet / 100, 3)
logger.info(
f"FREE prediction: {node_id} picks '{side}' on '{market_title}' "
f"at {probability_at_bet}% (potential: {potential} oracle rep)"
)
return True, (
f"FREE PICK placed: {side.upper()} on '{market_title}' "
f"at {probability_at_bet}%. Potential oracle rep: {potential}. "
f"This decision is FINAL."
)
def resolve_market(self, market_title: str, outcome: str) -> tuple[int, int]:
"""Resolve all FREE predictions on a market.
Args:
market_title: Title of the market
outcome: "yes", "no", or any outcome name for multi-outcome markets
Returns (winners, losers) counts
"""
if not outcome:
return 0, 0
outcome_lower = outcome.lower()
winners, losers = 0, 0
now = time.time()
for p in self.predictions:
if p["market_title"] != market_title or p.get("resolved", False):
continue
p["resolved"] = True
correct = p["side"].lower() == outcome_lower
p["correct"] = correct
if correct:
# Rep earned = contrarianism score
rep = round(1.0 - p["probability_at_bet"] / 100, 3)
rep = max(0.01, rep) # Minimum 0.01 even for easy bets
p["rep_earned"] = rep
self._add_oracle_rep(p["node_id"], rep)
winners += 1
self.prediction_log.append(
{
"node_id": p["node_id"],
"market_title": market_title,
"side": p["side"],
"outcome": outcome,
"probability_at_bet": p["probability_at_bet"],
"rep_earned": rep,
"timestamp": p["timestamp"],
"resolved_at": now,
}
)
logger.info(
f"Oracle win: {p['node_id']} earned {rep} oracle rep "
f"on '{market_title}' ({p['side']} at {p['probability_at_bet']}%)"
)
else:
p["rep_earned"] = 0.0
losers += 1
self.prediction_log.append(
{
"node_id": p["node_id"],
"market_title": market_title,
"side": p["side"],
"outcome": outcome,
"probability_at_bet": p["probability_at_bet"],
"rep_earned": 0.0,
"timestamp": p["timestamp"],
"resolved_at": now,
}
)
self._save()
return winners, losers
def get_active_markets(self) -> list[str]:
"""Get list of market titles with unresolved predictions or stakes."""
titles = set()
for p in self.predictions:
if not p.get("resolved", False):
titles.add(p["market_title"])
for s in self.market_stakes:
if not s.get("resolved", False):
titles.add(s["market_title"])
return list(titles)
# ─── Market Stakes (prediction markets) ────────────────────────────
def place_market_stake(
self, node_id: str, market_title: str, side: str, amount: float, probability_at_bet: float
) -> tuple[bool, str]:
"""Stake oracle rep on a prediction market outcome. FINAL decision.
Args:
node_id: Staker's node ID
market_title: Title of the prediction market
side: "yes", "no", or outcome name for multi-outcome markets
amount: How much oracle rep to risk
probability_at_bet: Current probability (0-100) of the chosen side
Returns (success, detail)
"""
if not side or not side.strip():
return False, "Side is required"
if amount <= 0:
return False, "Stake amount must be positive"
if not (0 <= probability_at_bet <= 100):
return False, "Probability must be 0-100"
available = self.get_oracle_rep(node_id)
if available < amount:
return False, f"Insufficient oracle rep (have {available:.2f}, need {amount:.2f})"
# Can't have both a free pick AND a stake on the same market
existing_free = [
p
for p in self.predictions
if p["node_id"] == node_id
and p["market_title"] == market_title
and not p.get("resolved", False)
]
if existing_free:
return (
False,
f"You already have a FREE prediction on '{market_title}'. Your decision was FINAL.",
)
# Can't stake twice on the same market
existing_stake = [
s
for s in self.market_stakes
if s["node_id"] == node_id
and s["market_title"] == market_title
and not s.get("resolved", False)
]
if existing_stake:
return (
False,
f"You already have a STAKED prediction on '{market_title}'. Your decision was FINAL.",
)
self.market_stakes.append(
{
"stake_id": secrets.token_hex(6),
"node_id": node_id,
"market_title": market_title,
"side": side,
"amount": amount,
"probability_at_bet": probability_at_bet,
"timestamp": time.time(),
"resolved": False,
"correct": None,
"rep_earned": 0.0,
}
)
self._save()
logger.info(
f"MARKET STAKE: {node_id} stakes {amount:.2f} rep on '{side}' "
f"for '{market_title}' at {probability_at_bet}%"
)
return True, (
f"STAKED {amount:.2f} oracle rep on {side.upper()} for '{market_title}' "
f"at {probability_at_bet}%. This decision is FINAL. "
f"If correct, you split the loser pool proportionally."
)
def resolve_market_stakes(self, market_title: str, outcome: str) -> dict:
"""Resolve all market stakes for a concluded market.
Winners split the loser pool proportionally to their stake.
If everyone picked the same side, stakes are returned (no profit, no loss).
Returns summary dict.
"""
if not outcome:
return {"resolved": 0}
outcome_lower = outcome.lower()
active = [
s
for s in self.market_stakes
if s["market_title"] == market_title and not s.get("resolved", False)
]
if not active:
return {"resolved": 0}
winners = [s for s in active if s["side"].lower() == outcome_lower]
losers = [s for s in active if s["side"].lower() != outcome_lower]
winner_pool = sum(s["amount"] for s in winners)
loser_pool = sum(s["amount"] for s in losers)
now = time.time()
if not losers:
# Everyone picked the same side — return stakes, no profit
for s in active:
s["resolved"] = True
s["correct"] = True
s["rep_earned"] = 0.0 # No profit when no opposition
self._save()
logger.info(
f"Market stake resolution [{market_title}]: unanimous '{outcome}', "
f"{len(winners)} stakers get rep back (no loser pool)"
)
return {
"resolved": len(active),
"winners": len(winners),
"losers": 0,
"winner_pool": winner_pool,
"loser_pool": 0,
"unanimous": True,
}
# Losers lose their staked rep
for s in losers:
self._remove_oracle_rep(s["node_id"], s["amount"])
s["resolved"] = True
s["correct"] = False
s["rep_earned"] = 0.0
self.prediction_log.append(
{
"node_id": s["node_id"],
"market_title": market_title,
"side": s["side"],
"outcome": outcome,
"probability_at_bet": s["probability_at_bet"],
"rep_earned": 0.0,
"staked": s["amount"],
"timestamp": s["timestamp"],
"resolved_at": now,
}
)
# Winners split loser pool proportionally + keep their own stake
for s in winners:
proportion = s["amount"] / winner_pool if winner_pool > 0 else 0
winnings = round(loser_pool * proportion, 3)
s["resolved"] = True
s["correct"] = True
s["rep_earned"] = winnings
self._add_oracle_rep(s["node_id"], winnings)
self.prediction_log.append(
{
"node_id": s["node_id"],
"market_title": market_title,
"side": s["side"],
"outcome": outcome,
"probability_at_bet": s["probability_at_bet"],
"rep_earned": winnings,
"staked": s["amount"],
"timestamp": s["timestamp"],
"resolved_at": now,
}
)
self._save()
logger.info(
f"Market stake resolution [{market_title}]: '{outcome}' wins. "
f"{len(winners)} winners split {loser_pool:.2f} rep from {len(losers)} losers"
)
return {
"resolved": len(active),
"winners": len(winners),
"losers": len(losers),
"winner_pool": round(winner_pool, 3),
"loser_pool": round(loser_pool, 3),
}
def get_market_consensus(self, market_title: str) -> dict:
"""Get network consensus for a single market — picks + stakes per side."""
sides: dict[str, dict] = {}
# Count free predictions
for p in self.predictions:
if p["market_title"] != market_title or p.get("resolved", False):
continue
s = p["side"]
if s not in sides:
sides[s] = {"picks": 0, "staked": 0.0}
sides[s]["picks"] += 1
# Count market stakes
for st in self.market_stakes:
if st["market_title"] != market_title or st.get("resolved", False):
continue
s = st["side"]
if s not in sides:
sides[s] = {"picks": 0, "staked": 0.0}
sides[s]["picks"] += 1
sides[s]["staked"] = round(sides[s]["staked"] + st["amount"], 3)
total_picks = sum(v["picks"] for v in sides.values())
total_staked = round(sum(v["staked"] for v in sides.values()), 3)
return {
"market_title": market_title,
"total_picks": total_picks,
"total_staked": total_staked,
"sides": sides,
}
def get_all_market_consensus(self) -> dict[str, dict]:
"""Bulk consensus for all active markets. Returns {market_title: consensus_summary}."""
titles = set()
for p in self.predictions:
if not p.get("resolved", False):
titles.add(p["market_title"])
for s in self.market_stakes:
if not s.get("resolved", False):
titles.add(s["market_title"])
result = {}
for title in titles:
c = self.get_market_consensus(title)
result[title] = {
"total_picks": c["total_picks"],
"total_staked": c["total_staked"],
"sides": c["sides"],
}
return result
# ─── Truth Stakes (posts/comments — separate system) ──────────────
def place_stake(
self,
staker_id: str,
message_id: str,
poster_id: str,
side: str,
amount: float,
duration_days: int,
) -> tuple[bool, str]:
"""Stake oracle rep on a post's truthfulness.
Args:
staker_id: Oracle staking their rep
message_id: The post/message being evaluated
poster_id: Who posted the original message
side: "truth" or "false"
amount: How much oracle rep to stake
duration_days: 1-7 days before resolution
Returns (success, detail)
"""
if side not in ("truth", "false"):
return False, "Side must be 'truth' or 'false'"
if not (MIN_STAKE_DAYS <= duration_days <= MAX_STAKE_DAYS):
return False, f"Duration must be {MIN_STAKE_DAYS}-{MAX_STAKE_DAYS} days"
if amount <= 0:
return False, "Stake amount must be positive"
available = self.get_oracle_rep(staker_id)
if available < amount:
return False, f"Insufficient oracle rep (have {available}, need {amount})"
# Check if this staker already has an active stake on this message
existing = [
s
for s in self.stakes
if s["staker_id"] == staker_id
and s["message_id"] == message_id
and not s.get("resolved", False)
]
if existing:
return False, "You already have an active stake on this message"
now = time.time()
expires = now + (duration_days * 86400)
# Check if there are existing stakes — extend grace period
active_stakes = [
s for s in self.stakes if s["message_id"] == message_id and not s.get("resolved", False)
]
# If this is a counter-stake, ensure the expiry is at least GRACE_PERIOD_HOURS
# after the latest stake on the other side
for s in active_stakes:
if s["side"] != side:
min_expires = s.get("last_counter_at", s["created_at"]) + (
GRACE_PERIOD_HOURS * 3600
)
if expires < min_expires:
expires = min_expires
stake = {
"stake_id": secrets.token_hex(6),
"message_id": message_id,
"poster_id": poster_id,
"staker_id": staker_id,
"side": side,
"amount": amount,
"duration_days": duration_days,
"created_at": now,
"expires_at": expires,
"resolved": False,
"last_counter_at": now,
}
self.stakes.append(stake)
# Update last_counter_at on opposing stakes (extends their grace period)
for s in active_stakes:
if s["side"] != side:
s["last_counter_at"] = now
self._save()
days_str = f"{duration_days} day{'s' if duration_days > 1 else ''}"
logger.info(
f"Oracle stake: {staker_id} stakes {amount} oracle rep "
f"as '{side}' on message {message_id} for {days_str}"
)
return True, (
f"Staked {amount} oracle rep as '{side.upper()}' on message "
f"{message_id} for {days_str}. Expires {time.strftime('%Y-%m-%d %H:%M', time.localtime(expires))}"
)
def resolve_expired_stakes(self) -> list[dict]:
"""Resolve all expired stake contests. Called periodically.
Returns list of resolution summaries.
"""
now = time.time()
resolutions = []
# Group active stakes by message_id
active_by_msg: dict[str, list[dict]] = {}
for s in self.stakes:
if not s.get("resolved", False):
active_by_msg.setdefault(s["message_id"], []).append(s)
for msg_id, stakes in active_by_msg.items():
# Check if ALL stakes for this message have expired
if not all(s["expires_at"] <= now for s in stakes):
continue # Some stakes haven't expired yet
# Tally sides
truth_total = sum(s["amount"] for s in stakes if s["side"] == "truth")
false_total = sum(s["amount"] for s in stakes if s["side"] == "false")
if truth_total == false_total:
# Tie — everyone gets their rep back, no resolution
for s in stakes:
s["resolved"] = True
resolutions.append(
{
"message_id": msg_id,
"outcome": "tie",
"truth_total": truth_total,
"false_total": false_total,
}
)
continue
winning_side = "truth" if truth_total > false_total else "false"
losing_total = false_total if winning_side == "truth" else truth_total
winning_total = truth_total if winning_side == "truth" else false_total
winners = [s for s in stakes if s["side"] == winning_side]
losers = [s for s in stakes if s["side"] != winning_side]
# Losers lose their staked rep
for s in losers:
self._remove_oracle_rep(s["staker_id"], s["amount"])
s["resolved"] = True
# Winners divide losers' rep proportionally
for s in winners:
proportion = s["amount"] / winning_total if winning_total > 0 else 0
winnings = round(losing_total * proportion, 3)
self._add_oracle_rep(s["staker_id"], winnings)
s["resolved"] = True
# Duration weight for the poster's reputation effect
max_duration = max(s["duration_days"] for s in stakes)
duration_label = (
"resounding" if max_duration >= 7 else "contested" if max_duration >= 3 else "brief"
)
resolution = {
"message_id": msg_id,
"poster_id": stakes[0].get("poster_id", ""),
"outcome": winning_side,
"truth_total": round(truth_total, 3),
"false_total": round(false_total, 3),
"duration_label": duration_label,
"max_duration_days": max_duration,
"winners": [
{
"node_id": s["staker_id"],
"staked": s["amount"],
"won": (
round(losing_total * (s["amount"] / winning_total), 3)
if winning_total > 0
else 0
),
}
for s in winners
],
"losers": [
{
"node_id": s["staker_id"],
"lost": s["amount"],
}
for s in losers
],
}
resolutions.append(resolution)
logger.info(
f"Oracle resolution [{msg_id}]: {winning_side.upper()} wins "
f"({truth_total} vs {false_total}), {duration_label} verdict"
)
if resolutions:
self._save()
return resolutions
def get_stakes_for_message(self, message_id: str) -> dict:
"""Get all stakes on a message with totals."""
active = [
s for s in self.stakes if s["message_id"] == message_id and not s.get("resolved", False)
]
truth_stakes = [s for s in active if s["side"] == "truth"]
false_stakes = [s for s in active if s["side"] == "false"]
return {
"message_id": message_id,
"truth_total": round(sum(s["amount"] for s in truth_stakes), 3),
"false_total": round(sum(s["amount"] for s in false_stakes), 3),
"truth_stakers": [
{"node_id": s["staker_id"], "amount": s["amount"], "expires": s["expires_at"]}
for s in truth_stakes
],
"false_stakers": [
{"node_id": s["staker_id"], "amount": s["amount"], "expires": s["expires_at"]}
for s in false_stakes
],
"earliest_expiry": min((s["expires_at"] for s in active), default=0),
}
# ─── Oracle Profile ───────────────────────────────────────────────
def get_oracle_profile(self, node_id: str) -> dict:
"""Full oracle profile — rep, prediction history, active stakes."""
total_rep = self.get_total_oracle_rep(node_id)
available_rep = self.get_oracle_rep(node_id)
# Prediction stats
my_predictions = [p for p in self.prediction_log if p["node_id"] == node_id]
wins = [p for p in my_predictions if p["rep_earned"] > 0]
losses = [p for p in my_predictions if p["rep_earned"] == 0]
# Active stakes
active_stakes = [
{
"message_id": s["message_id"],
"side": s["side"],
"amount": s["amount"],
"expires": s["expires_at"],
}
for s in self.stakes
if s["staker_id"] == node_id and not s.get("resolved", False)
]
# Recent prediction log (last 20)
recent = sorted(my_predictions, key=lambda x: x.get("resolved_at", 0), reverse=True)[:20]
prediction_history = [
{
"market": p["market_title"][:50],
"side": p["side"],
"probability": p["probability_at_bet"],
"outcome": p.get("outcome", "?"),
"rep_earned": p["rep_earned"],
"correct": p["rep_earned"] > 0,
"age": f"{int((time.time() - p.get('resolved_at', p['timestamp'])) / 86400)}d ago",
}
for p in recent
]
# Farming score — what % of bets were on >80% probability outcomes
if my_predictions:
easy_bets = sum(
1
for p in my_predictions
if (p["side"] == "yes" and p["probability_at_bet"] > 80)
or (p["side"] == "no" and p["probability_at_bet"] < 20)
)
farming_pct = round(easy_bets / len(my_predictions) * 100)
else:
farming_pct = 0
return {
"node_id": node_id,
"oracle_rep": available_rep,
"oracle_rep_total": total_rep,
"oracle_rep_locked": round(total_rep - available_rep, 3),
"predictions_won": len(wins),
"predictions_lost": len(losses),
"win_rate": round(len(wins) / max(1, len(wins) + len(losses)) * 100),
"farming_pct": farming_pct,
"active_stakes": active_stakes,
"prediction_history": prediction_history,
}
def get_active_predictions(self, node_id: str) -> list[dict]:
"""Get a node's unresolved predictions (free picks + staked)."""
results = []
now = time.time()
# Free picks
for p in self.predictions:
if p["node_id"] != node_id or p.get("resolved", False):
continue
potential = round(1.0 - p["probability_at_bet"] / 100, 3)
days = int((now - p["timestamp"]) / 86400)
results.append(
{
"prediction_id": p["prediction_id"],
"market_title": p["market_title"],
"side": p["side"],
"probability_at_bet": p["probability_at_bet"],
"potential_rep": potential,
"staked": 0,
"mode": "free",
"placed": f"{days}d ago",
}
)
# Market stakes
for s in self.market_stakes:
if s["node_id"] != node_id or s.get("resolved", False):
continue
days = int((now - s["timestamp"]) / 86400)
results.append(
{
"prediction_id": s["stake_id"],
"market_title": s["market_title"],
"side": s["side"],
"probability_at_bet": s["probability_at_bet"],
"potential_rep": 0, # Depends on loser pool — unknown until resolution
"staked": s["amount"],
"mode": "staked",
"placed": f"{days}d ago",
}
)
return results
# ─── Cleanup ──────────────────────────────────────────────────────
def cleanup_old_data(self):
"""Remove resolved predictions and market stakes older than decay window."""
cutoff = time.time() - (ORACLE_DECAY_DAYS * 86400)
before_pred = len(self.predictions)
before_stakes = len(self.market_stakes)
self.predictions = [
p for p in self.predictions if not p.get("resolved", False) or p["timestamp"] >= cutoff
]
self.market_stakes = [
s
for s in self.market_stakes
if not s.get("resolved", False) or s["timestamp"] >= cutoff
]
# Trim prediction log
self.prediction_log = [
p for p in self.prediction_log if p.get("resolved_at", p["timestamp"]) >= cutoff
]
removed = (before_pred - len(self.predictions)) + (before_stakes - len(self.market_stakes))
if removed:
self._save()
logger.info(f"Cleaned up {removed} old predictions/stakes")
# ─── Module-level singleton ──────────────────────────────────────────────
oracle_ledger = OracleLedger()
backend/services/mesh/mesh_peer_store.py
+356
View File
@@ -0,0 +1,356 @@
from __future__ import annotations
import json
import os
import tempfile
import time
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
from services.mesh.mesh_crypto import normalize_peer_url
BACKEND_DIR = Path(__file__).resolve().parents[2]
DATA_DIR = BACKEND_DIR / "data"
DEFAULT_PEER_STORE_PATH = DATA_DIR / "peer_store.json"
PEER_STORE_VERSION = 1
ALLOWED_PEER_BUCKETS = {"bootstrap", "sync", "push"}
ALLOWED_PEER_SOURCES = {"bundle", "operator", "bootstrap_promoted", "runtime"}
ALLOWED_PEER_TRANSPORTS = {"clearnet", "onion"}
ALLOWED_PEER_ROLES = {"participant", "relay", "seed"}
class PeerStoreError(ValueError):
pass
def _atomic_write_text(target: Path, content: str) -> None:
target.parent.mkdir(parents=True, exist_ok=True)
fd, tmp_path = tempfile.mkstemp(dir=str(target.parent), suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding="utf-8") as handle:
handle.write(content)
handle.flush()
os.fsync(handle.fileno())
os.replace(tmp_path, str(target))
except BaseException:
try:
os.unlink(tmp_path)
except OSError:
pass
raise
@dataclass(frozen=True)
class PeerRecord:
bucket: str
source: str
peer_url: str
transport: str
role: str
label: str = ""
signer_id: str = ""
enabled: bool = True
added_at: int = 0
updated_at: int = 0
last_seen_at: int = 0
last_sync_ok_at: int = 0
last_push_ok_at: int = 0
last_error: str = ""
failure_count: int = 0
cooldown_until: int = 0
metadata: dict[str, Any] = field(default_factory=dict)
def record_key(self) -> str:
return f"{self.bucket}:{self.peer_url}"
def to_dict(self) -> dict[str, Any]:
return asdict(self)
def _normalize_peer_record(data: dict[str, Any]) -> PeerRecord:
bucket = str(data.get("bucket", "") or "").strip().lower()
source = str(data.get("source", "") or "").strip().lower()
peer_url = str(data.get("peer_url", "") or "").strip()
transport = str(data.get("transport", "") or "").strip().lower()
role = str(data.get("role", "") or "").strip().lower()
label = str(data.get("label", "") or "").strip()
signer_id = str(data.get("signer_id", "") or "").strip()
enabled = bool(data.get("enabled", True))
metadata = data.get("metadata", {})
if bucket not in ALLOWED_PEER_BUCKETS:
raise PeerStoreError(f"unsupported peer bucket: {bucket or 'missing'}")
if source not in ALLOWED_PEER_SOURCES:
raise PeerStoreError(f"unsupported peer source: {source or 'missing'}")
if transport not in ALLOWED_PEER_TRANSPORTS:
raise PeerStoreError(f"unsupported peer transport: {transport or 'missing'}")
if role not in ALLOWED_PEER_ROLES:
raise PeerStoreError(f"unsupported peer role: {role or 'missing'}")
normalized = normalize_peer_url(peer_url)
if not normalized or normalized != peer_url:
raise PeerStoreError("peer_url must be normalized")
parsed = urlparse(normalized)
hostname = str(parsed.hostname or "").strip().lower()
if transport == "clearnet":
if parsed.scheme not in ("https", "http") or hostname.endswith(".onion"):
raise PeerStoreError("clearnet peers must use https:// (or http:// for LAN/testnet)")
elif transport == "onion":
if parsed.scheme != "http" or not hostname.endswith(".onion"):
raise PeerStoreError("onion peers must use http://*.onion")
if not isinstance(metadata, dict):
raise PeerStoreError("peer metadata must be an object")
return PeerRecord(
bucket=bucket,
source=source,
peer_url=normalized,
transport=transport,
role=role,
label=label,
signer_id=signer_id,
enabled=enabled,
added_at=int(data.get("added_at", 0) or 0),
updated_at=int(data.get("updated_at", 0) or 0),
last_seen_at=int(data.get("last_seen_at", 0) or 0),
last_sync_ok_at=int(data.get("last_sync_ok_at", 0) or 0),
last_push_ok_at=int(data.get("last_push_ok_at", 0) or 0),
last_error=str(data.get("last_error", "") or ""),
failure_count=int(data.get("failure_count", 0) or 0),
cooldown_until=int(data.get("cooldown_until", 0) or 0),
metadata=dict(metadata),
)
def make_bootstrap_peer_record(
*,
peer_url: str,
transport: str,
role: str,
signer_id: str,
label: str = "",
now: float | None = None,
) -> PeerRecord:
timestamp = int(now if now is not None else time.time())
return _normalize_peer_record(
{
"bucket": "bootstrap",
"source": "bundle",
"peer_url": peer_url,
"transport": transport,
"role": role,
"label": label,
"signer_id": signer_id,
"enabled": True,
"added_at": timestamp,
"updated_at": timestamp,
}
)
def make_sync_peer_record(
*,
peer_url: str,
transport: str,
role: str = "participant",
source: str = "operator",
label: str = "",
signer_id: str = "",
now: float | None = None,
) -> PeerRecord:
timestamp = int(now if now is not None else time.time())
return _normalize_peer_record(
{
"bucket": "sync",
"source": source,
"peer_url": peer_url,
"transport": transport,
"role": role,
"label": label,
"signer_id": signer_id,
"enabled": True,
"added_at": timestamp,
"updated_at": timestamp,
}
)
def make_push_peer_record(
*,
peer_url: str,
transport: str,
role: str = "relay",
source: str = "operator",
label: str = "",
now: float | None = None,
) -> PeerRecord:
timestamp = int(now if now is not None else time.time())
return _normalize_peer_record(
{
"bucket": "push",
"source": source,
"peer_url": peer_url,
"transport": transport,
"role": role,
"label": label,
"enabled": True,
"added_at": timestamp,
"updated_at": timestamp,
}
)
class PeerStore:
def __init__(self, path: str | Path = DEFAULT_PEER_STORE_PATH):
self.path = Path(path)
self._records: dict[str, PeerRecord] = {}
def load(self) -> list[PeerRecord]:
if not self.path.exists():
self._records = {}
return []
try:
raw = json.loads(self.path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise PeerStoreError("peer store is not valid JSON") from exc
if not isinstance(raw, dict):
raise PeerStoreError("peer store root must be an object")
version = int(raw.get("version", 0) or 0)
if version != PEER_STORE_VERSION:
raise PeerStoreError(f"unsupported peer store version: {version}")
records_raw = raw.get("records", [])
if not isinstance(records_raw, list):
raise PeerStoreError("peer store records must be a list")
records: dict[str, PeerRecord] = {}
for entry in records_raw:
if not isinstance(entry, dict):
raise PeerStoreError("peer store records must be objects")
record = _normalize_peer_record(entry)
records[record.record_key()] = record
self._records = records
return self.records()
def save(self) -> None:
payload = {
"version": PEER_STORE_VERSION,
"records": [record.to_dict() for record in self.records()],
}
_atomic_write_text(
self.path,
json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False),
)
def records(self) -> list[PeerRecord]:
return sorted(self._records.values(), key=lambda item: (item.bucket, item.peer_url))
def records_for_bucket(self, bucket: str) -> list[PeerRecord]:
normalized_bucket = str(bucket or "").strip().lower()
return [record for record in self.records() if record.bucket == normalized_bucket]
def upsert(self, record: PeerRecord) -> PeerRecord:
existing = self._records.get(record.record_key())
if existing is None:
self._records[record.record_key()] = record
return record
merged = PeerRecord(
bucket=record.bucket,
source=record.source,
peer_url=record.peer_url,
transport=record.transport,
role=record.role,
label=record.label or existing.label,
signer_id=record.signer_id or existing.signer_id,
enabled=record.enabled,
added_at=existing.added_at or record.added_at,
updated_at=max(existing.updated_at, record.updated_at),
last_seen_at=max(existing.last_seen_at, record.last_seen_at),
last_sync_ok_at=max(existing.last_sync_ok_at, record.last_sync_ok_at),
last_push_ok_at=max(existing.last_push_ok_at, record.last_push_ok_at),
last_error=record.last_error or existing.last_error,
failure_count=max(existing.failure_count, record.failure_count),
cooldown_until=max(existing.cooldown_until, record.cooldown_until),
metadata={**existing.metadata, **record.metadata},
)
self._records[record.record_key()] = merged
return merged
def mark_seen(self, peer_url: str, bucket: str, *, now: float | None = None) -> PeerRecord:
record = self._require_record(peer_url, bucket)
timestamp = int(now if now is not None else time.time())
updated = PeerRecord(
**{
**record.to_dict(),
"last_seen_at": timestamp,
"updated_at": timestamp,
}
)
self._records[updated.record_key()] = updated
return updated
def mark_sync_success(self, peer_url: str, bucket: str = "sync", *, now: float | None = None) -> PeerRecord:
record = self._require_record(peer_url, bucket)
timestamp = int(now if now is not None else time.time())
updated = PeerRecord(
**{
**record.to_dict(),
"last_sync_ok_at": timestamp,
"last_error": "",
"failure_count": 0,
"cooldown_until": 0,
"updated_at": timestamp,
}
)
self._records[updated.record_key()] = updated
return updated
def mark_push_success(self, peer_url: str, bucket: str = "push", *, now: float | None = None) -> PeerRecord:
record = self._require_record(peer_url, bucket)
timestamp = int(now if now is not None else time.time())
updated = PeerRecord(
**{
**record.to_dict(),
"last_push_ok_at": timestamp,
"last_error": "",
"failure_count": 0,
"cooldown_until": 0,
"updated_at": timestamp,
}
)
self._records[updated.record_key()] = updated
return updated
def mark_failure(
self,
peer_url: str,
bucket: str,
*,
error: str,
cooldown_s: int = 0,
now: float | None = None,
) -> PeerRecord:
record = self._require_record(peer_url, bucket)
timestamp = int(now if now is not None else time.time())
updated = PeerRecord(
**{
**record.to_dict(),
"last_error": str(error or "").strip(),
"failure_count": int(record.failure_count) + 1,
"cooldown_until": timestamp + max(0, int(cooldown_s or 0)),
"updated_at": timestamp,
}
)
self._records[updated.record_key()] = updated
return updated
def _require_record(self, peer_url: str, bucket: str) -> PeerRecord:
normalized_url = normalize_peer_url(peer_url)
key = f"{str(bucket or '').strip().lower()}:{normalized_url}"
if key not in self._records:
raise PeerStoreError(f"peer record not found: {key}")
return self._records[key]
backend/services/mesh/mesh_privacy_logging.py
+19
View File
@@ -0,0 +1,19 @@
"""Helpers for privacy-aware operational logging.
These helpers keep logs useful for debugging classes of failures without
recording stable private-plane identifiers verbatim.
"""
from __future__ import annotations
import hashlib
def privacy_log_label(value: str, *, label: str = "") -> str:
raw = str(value or "").strip()
if not raw:
return f"{label}#none" if label else ""
digest = hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]
if not label:
return digest
return f"{label}#{digest}"
backend/services/mesh/mesh_protocol.py
+250
View File
@@ -0,0 +1,250 @@
"""Mesh protocol helpers for canonical payloads and versioning."""
from __future__ import annotations
from typing import Any
PROTOCOL_VERSION = "infonet/2"
NETWORK_ID = "sb-testnet-0"
def _safe_int(val, default=0):
try:
return int(val)
except (TypeError, ValueError):
return default
def _normalize_number(value: Any) -> int | float:
try:
num = float(value)
except Exception:
return 0
if num.is_integer():
return int(num)
return num
def normalize_message_payload(payload: dict[str, Any]) -> dict[str, Any]:
normalized = {
"message": str(payload.get("message", "")),
"destination": str(payload.get("destination", "")),
"channel": str(payload.get("channel", "LongFast")),
"priority": str(payload.get("priority", "normal")),
"ephemeral": bool(payload.get("ephemeral", False)),
}
transport_lock = str(payload.get("transport_lock", "") or "").strip().lower()
if transport_lock:
normalized["transport_lock"] = transport_lock
return normalized
def normalize_gate_message_payload(payload: dict[str, Any]) -> dict[str, Any]:
normalized = {
"gate": str(payload.get("gate", "")).strip().lower(),
"ciphertext": str(payload.get("ciphertext", "")),
"nonce": str(payload.get("nonce", payload.get("iv", ""))),
"sender_ref": str(payload.get("sender_ref", "")),
"format": str(payload.get("format", "mls1") or "mls1").strip().lower(),
}
epoch = _safe_int(payload.get("epoch", 0), 0)
if epoch > 0:
normalized["epoch"] = epoch
return normalized
def normalize_vote_payload(payload: dict[str, Any]) -> dict[str, Any]:
vote_val = _safe_int(payload.get("vote", 0), 0)
return {
"target_id": str(payload.get("target_id", "")),
"vote": vote_val,
"gate": str(payload.get("gate", "")),
}
def normalize_gate_create_payload(payload: dict[str, Any]) -> dict[str, Any]:
rules = payload.get("rules", {})
if not isinstance(rules, dict):
rules = {}
return {
"gate_id": str(payload.get("gate_id", "")).lower(),
"display_name": str(payload.get("display_name", ""))[:64],
"rules": rules,
}
def normalize_prediction_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"market_title": str(payload.get("market_title", "")),
"side": str(payload.get("side", "")),
"stake_amount": _normalize_number(payload.get("stake_amount", 0.0)),
}
def normalize_stake_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"message_id": str(payload.get("message_id", "")),
"poster_id": str(payload.get("poster_id", "")),
"side": str(payload.get("side", "")),
"amount": _normalize_number(payload.get("amount", 0.0)),
"duration_days": _safe_int(payload.get("duration_days", 0), 0),
}
def normalize_dm_key_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"dh_pub_key": str(payload.get("dh_pub_key", "")),
"dh_algo": str(payload.get("dh_algo", "")),
"timestamp": _safe_int(payload.get("timestamp", 0), 0),
}
def normalize_dm_message_payload(payload: dict[str, Any]) -> dict[str, Any]:
normalized = {
"recipient_id": str(payload.get("recipient_id", "")),
"delivery_class": str(payload.get("delivery_class", "")).lower(),
"recipient_token": str(payload.get("recipient_token", "")),
"ciphertext": str(payload.get("ciphertext", "")),
"msg_id": str(payload.get("msg_id", "")),
"timestamp": _safe_int(payload.get("timestamp", 0), 0),
"format": str(payload.get("format", "dm1") or "dm1").strip().lower(),
}
session_welcome = payload.get("session_welcome")
if session_welcome:
normalized["session_welcome"] = str(session_welcome)
sender_seal = str(payload.get("sender_seal", "") or "")
if sender_seal:
normalized["sender_seal"] = sender_seal
relay_salt = str(payload.get("relay_salt", "") or "").strip().lower()
if relay_salt:
normalized["relay_salt"] = relay_salt
return normalized
def normalize_dm_message_payload_legacy(payload: dict[str, Any]) -> dict[str, Any]:
return {
"recipient_id": str(payload.get("recipient_id", "")),
"delivery_class": str(payload.get("delivery_class", "")).lower(),
"recipient_token": str(payload.get("recipient_token", "")),
"ciphertext": str(payload.get("ciphertext", "")),
"msg_id": str(payload.get("msg_id", "")),
"timestamp": _safe_int(payload.get("timestamp", 0), 0),
}
def normalize_mailbox_claims(payload: dict[str, Any]) -> list[dict[str, str]]:
claims = payload.get("mailbox_claims", [])
if not isinstance(claims, list):
return []
normalized: list[dict[str, str]] = []
for claim in claims:
if not isinstance(claim, dict):
continue
normalized.append(
{
"type": str(claim.get("type", "")).lower(),
"token": str(claim.get("token", "")),
}
)
return normalized
def normalize_dm_poll_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"mailbox_claims": normalize_mailbox_claims(payload),
"timestamp": _safe_int(payload.get("timestamp", 0), 0),
"nonce": str(payload.get("nonce", "")),
}
def normalize_dm_count_payload(payload: dict[str, Any]) -> dict[str, Any]:
return normalize_dm_poll_payload(payload)
def normalize_dm_block_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"blocked_id": str(payload.get("blocked_id", "")),
"action": str(payload.get("action", "block")).lower(),
}
def normalize_dm_key_witness_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"target_id": str(payload.get("target_id", "")),
"dh_pub_key": str(payload.get("dh_pub_key", "")),
"timestamp": _safe_int(payload.get("timestamp", 0), 0),
}
def normalize_trust_vouch_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"target_id": str(payload.get("target_id", "")),
"note": str(payload.get("note", ""))[:140],
"timestamp": _safe_int(payload.get("timestamp", 0), 0),
}
def normalize_key_rotate_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"old_node_id": str(payload.get("old_node_id", "")),
"old_public_key": str(payload.get("old_public_key", "")),
"old_public_key_algo": str(payload.get("old_public_key_algo", "")),
"new_public_key": str(payload.get("new_public_key", "")),
"new_public_key_algo": str(payload.get("new_public_key_algo", "")),
"timestamp": _safe_int(payload.get("timestamp", 0), 0),
"old_signature": str(payload.get("old_signature", "")),
}
def normalize_key_revoke_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"revoked_public_key": str(payload.get("revoked_public_key", "")),
"revoked_public_key_algo": str(payload.get("revoked_public_key_algo", "")),
"revoked_at": _safe_int(payload.get("revoked_at", 0), 0),
"grace_until": _safe_int(payload.get("grace_until", 0), 0),
"reason": str(payload.get("reason", ""))[:140],
}
def normalize_abuse_report_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"target_id": str(payload.get("target_id", "")),
"reason": str(payload.get("reason", ""))[:280],
"gate": str(payload.get("gate", "")),
"evidence": str(payload.get("evidence", ""))[:256],
}
def normalize_payload(event_type: str, payload: dict[str, Any]) -> dict[str, Any]:
if event_type == "message":
return normalize_message_payload(payload)
if event_type == "gate_message":
return normalize_gate_message_payload(payload)
if event_type == "vote":
return normalize_vote_payload(payload)
if event_type == "gate_create":
return normalize_gate_create_payload(payload)
if event_type == "prediction":
return normalize_prediction_payload(payload)
if event_type == "stake":
return normalize_stake_payload(payload)
if event_type == "dm_key":
return normalize_dm_key_payload(payload)
if event_type == "dm_message":
return normalize_dm_message_payload(payload)
if event_type == "dm_poll":
return normalize_dm_poll_payload(payload)
if event_type == "dm_count":
return normalize_dm_count_payload(payload)
if event_type == "dm_block":
return normalize_dm_block_payload(payload)
if event_type == "dm_key_witness":
return normalize_dm_key_witness_payload(payload)
if event_type == "trust_vouch":
return normalize_trust_vouch_payload(payload)
if event_type == "key_rotate":
return normalize_key_rotate_payload(payload)
if event_type == "key_revoke":
return normalize_key_revoke_payload(payload)
if event_type == "abuse_report":
return normalize_abuse_report_payload(payload)
return payload
backend/services/mesh/mesh_reputation.py
+985
View File
@@ -0,0 +1,985 @@
"""Mesh Reputation Ledger — decentralized node trust scoring with gates.
Every node maintains a local reputation ledger. Votes are weighted by voter
reputation and tenure (anti-Sybil). Scores decay linearly over a 2-year window.
Gates are reputation-scoped communities. Entry requires meeting rep thresholds.
Getting downvoted below the threshold bars you automatically — no moderator needed.
Persistence: JSON files in backend/data/ (auto-saved on change, loaded on start).
"""
import math
import time
import logging
import os
import threading
import atexit
import hmac
import hashlib
from pathlib import Path
from typing import Optional
from services.mesh.mesh_privacy_logging import privacy_log_label
from services.mesh.mesh_secure_storage import (
read_domain_json,
read_secure_json,
write_domain_json,
)
logger = logging.getLogger("services.mesh_reputation")
DATA_DIR = Path(__file__).resolve().parents[2] / "data"
LEDGER_FILE = DATA_DIR / "reputation_ledger.json"
GATES_FILE = DATA_DIR / "gates.json"
LEDGER_DOMAIN = "reputation"
GATES_DOMAIN = "gates"
# ─── Constants ────────────────────────────────────────────────────────────
VOTE_DECAY_MONTHS = 24 # Votes decay over 24 months. Recent votes weigh heaviest.
VOTE_DECAY_DAYS = VOTE_DECAY_MONTHS * 30 # ~720 days total window
MIN_REP_TO_VOTE = 3 # Minimum reputation to cast any vote (only enforced after bootstrap)
BOOTSTRAP_THRESHOLD = 1000 # Rep-to-vote rule kicks in after this many nodes join
MIN_REP_TO_CREATE_GATE = 10 # Minimum overall rep to create a gate
GATE_RATIFICATION_REP = (
50 # Cumulative member rep needed for a gate to be ratified (after bootstrap)
)
ALLOW_DYNAMIC_GATES = False
_VOTE_STORAGE_SALT_CACHE: bytes | None = None
_VOTE_STORAGE_SALT_WARNING_EMITTED = False
DEFAULT_PRIVATE_GATES: dict[str, dict] = {
"infonet": {
"display_name": "Main Infonet",
"description": "Private network operations floor. Core testnet traffic, protocol notes, and live coordination stay here.",
"welcome": "WELCOME TO MAIN INFONET. Treat this as the protocol floor, not a public lobby.",
"sort_order": 10,
},
"general-talk": {
"display_name": "General Talk",
"description": "Lower-friction private lounge for day-to-day chatter, intros, and community pulse checks.",
"welcome": "WELCOME TO GENERAL TALK. Keep it human, but remember the lane is still private and reputation-backed.",
"sort_order": 20,
},
"gathered-intel": {
"display_name": "Gathered Intel",
"description": "Drop sourced observations, OSINT fragments, and operator notes worth preserving for later review.",
"welcome": "WELCOME TO GATHERED INTEL. Bring sources, timestamps, and enough context for someone else to verify you.",
"sort_order": 30,
},
"tracked-planes": {
"display_name": "Tracked Planes",
"description": "Aviation watchers, route anomalies, military traffic, and callout chatter for flights worth tracking.",
"welcome": "WELCOME TO TRACKED PLANES. Call out the flight, route, why it matters, and what pattern you think you see.",
"sort_order": 40,
},
"ukraine-front": {
"display_name": "Ukraine Front",
"description": "Focused room for Ukraine war developments, map observations, and source cross-checking.",
"welcome": "WELCOME TO UKRAINE FRONT. Keep reporting tight, sourced, and separated from wishcasting.",
"sort_order": 50,
},
"iran-front": {
"display_name": "Iran Front",
"description": "Iran flashpoint monitoring, regional spillover, and escalation watch from a private-lane perspective.",
"welcome": "WELCOME TO IRAN FRONT. Track escalation, proxies, logistics, and what changes the risk picture.",
"sort_order": 60,
},
"world-news": {
"display_name": "World News",
"description": "Big-picture geopolitical developments, breaking stories, and broader context outside the narrow fronts.",
"welcome": "WELCOME TO WORLD NEWS. Use this room when the story matters but does not fit a narrower gate.",
"sort_order": 70,
},
"prediction-markets": {
"display_name": "Prediction Markets",
"description": "Discuss market signals, event contracts, and whether crowd pricing is tracking reality or pure narrative.",
"welcome": "WELCOME TO PREDICTION MARKETS. Bring the market angle and the narrative angle, then compare them honestly.",
"sort_order": 80,
},
"finance": {
"display_name": "Finance",
"description": "Macro moves, defense names, rates, liquidity stress, and the parts of finance that steer the rest of the board.",
"welcome": "WELCOME TO FINANCE. Macro, defense names, liquidity stress, and market structure all belong here.",
"sort_order": 90,
},
"cryptography": {
"display_name": "Cryptography",
"description": "Protocol design, primitives, breakage reports, and the sharper math behind the network.",
"welcome": "WELCOME TO CRYPTOGRAPHY. If you think something can be broken, this is where you try to prove it.",
"sort_order": 100,
},
"cryptocurrencies": {
"display_name": "Cryptocurrencies",
"description": "Chain activity, privacy coin chatter, market structure, and crypto-adjacent threat intel.",
"welcome": "WELCOME TO CRYPTOCURRENCIES. Chain behavior, privacy tooling, and market weirdness all go on the table.",
"sort_order": 110,
},
"meet-chat": {
"display_name": "Meet Chat",
"description": "Casual private hangout for getting to know the other operators behind the personas.",
"welcome": "WELCOME TO MEET CHAT. Lighten up a little and let the community feel like it has actual people in it.",
"sort_order": 120,
},
"opsec-lab": {
"display_name": "OPSEC Lab",
"description": "Stress-test assumptions, try to break rep or persona boundaries, and document privacy failures without mercy.",
"welcome": "WELCOME TO OPSEC LAB. Be ruthless, document the leak, and assume everyone is smarter than the last audit.",
"sort_order": 130,
},
}
def _blind_voter(voter_id: str, salt: bytes) -> str:
if not voter_id:
return ""
digest = hmac.new(salt, voter_id.encode("utf-8"), hashlib.sha256).hexdigest()
return f"{digest[:8]}…"
def _vote_storage_salt() -> bytes:
global _VOTE_STORAGE_SALT_CACHE, _VOTE_STORAGE_SALT_WARNING_EMITTED
if _VOTE_STORAGE_SALT_CACHE is not None:
return _VOTE_STORAGE_SALT_CACHE
try:
from services.config import get_settings
secret = str(get_settings().MESH_PEER_PUSH_SECRET or "").strip()
except Exception:
secret = ""
if not secret and not _VOTE_STORAGE_SALT_WARNING_EMITTED:
logger.warning("MESH_PEER_PUSH_SECRET missing; falling back to local voter blinding salt")
_VOTE_STORAGE_SALT_WARNING_EMITTED = True
if secret:
_VOTE_STORAGE_SALT_CACHE = hmac.new(
secret.encode("utf-8"),
b"shadowbroker|reputation|voter-blind|v1",
hashlib.sha256,
).digest()
else:
# Persist a stable salt to disk so blinded voter IDs survive restarts.
# Without this, duplicate-vote detection breaks on every restart
# because the blinded ID changes with a new random salt.
salt_path = DATA_DIR / "voter_blind_salt.bin"
try:
if salt_path.exists() and salt_path.stat().st_size == 32:
_VOTE_STORAGE_SALT_CACHE = salt_path.read_bytes()
else:
DATA_DIR.mkdir(parents=True, exist_ok=True)
new_salt = os.urandom(32)
salt_path.write_bytes(new_salt)
_VOTE_STORAGE_SALT_CACHE = new_salt
logger.info("Generated new persistent voter blinding salt")
except Exception as e:
logger.error(f"Failed to persist voter salt, falling back to random: {e}")
_VOTE_STORAGE_SALT_CACHE = os.urandom(32)
return _VOTE_STORAGE_SALT_CACHE
def _stored_voter_id(vote: dict) -> str:
blinded = str(vote.get("blinded_voter_id", "") or "").strip()
if blinded:
return blinded
raw = str(vote.get("voter_id", "") or "").strip()
if not raw:
return ""
return _blind_voter(raw, _vote_storage_salt())
def _serialize_vote_record(vote: dict) -> dict:
blinded = _stored_voter_id(vote)
payload = dict(vote or {})
payload.pop("voter_id", None)
if blinded:
payload["blinded_voter_id"] = blinded
return payload
# ─── Vote Record ──────────────────────────────────────────────────────────
class ReputationLedger:
"""Local reputation ledger — each node maintains its own view.
Storage format:
nodes: {node_id: {first_seen, public_key, agent}}
votes: [{voter_id, target_id, vote (+1/-1), gate (optional), timestamp}]
scores_cache: {node_id: {overall: int, gates: {gate_id: int}}}
"""
def __init__(self):
self.nodes: dict[str, dict] = {} # {node_id: {first_seen, public_key, agent}}
self.votes: list[dict] = [] # [{voter_id, target_id, vote, gate, timestamp}]
self.vouches: list[dict] = [] # [{voucher_id, target_id, note, timestamp}]
self.aliases: dict[str, str] = {} # {new_node_id: old_node_id}
self._scores_dirty = True
self._scores_cache: dict[str, dict] = {}
self._dirty = False
self._save_lock = threading.Lock()
self._save_timer: threading.Timer | None = None
self._SAVE_INTERVAL = 5.0
atexit.register(self._flush)
self._load()
# ─── Persistence ──────────────────────────────────────────────────
def _load(self):
"""Load ledger from disk."""
domain_path = DATA_DIR / LEDGER_DOMAIN / LEDGER_FILE.name
if not domain_path.exists() and LEDGER_FILE.exists():
try:
legacy = read_secure_json(
LEDGER_FILE,
lambda: {"nodes": {}, "votes": [], "vouches": [], "aliases": {}},
)
write_domain_json(LEDGER_DOMAIN, LEDGER_FILE.name, legacy)
LEDGER_FILE.unlink(missing_ok=True)
except Exception as e:
logger.error(f"Failed to migrate reputation ledger: {e}")
try:
data = read_domain_json(
LEDGER_DOMAIN,
LEDGER_FILE.name,
lambda: {"nodes": {}, "votes": [], "vouches": [], "aliases": {}},
)
self.nodes = data.get("nodes", {})
raw_votes = data.get("votes", [])
# Purge legacy __system__ cost votes — they stored raw voter
# identities as target_id, which is a privacy leak.
before = len(raw_votes)
self.votes = [v for v in raw_votes if not v.get("system_cost")]
purged = before - len(self.votes)
if purged:
self._dirty = True # re-save without the leaked records
logger.info(f"Purged {purged} legacy system_cost vote(s) with raw identity leak")
self.vouches = data.get("vouches", [])
self.aliases = data.get("aliases", {})
self._scores_dirty = True
logger.info(
f"Loaded reputation ledger: {len(self.nodes)} nodes, {len(self.votes)} votes"
)
except Exception as e:
logger.error(f"Failed to load reputation ledger: {e}")
def _save(self):
"""Mark dirty and schedule a coalesced disk write."""
self._dirty = True
with self._save_lock:
if self._save_timer is None or not self._save_timer.is_alive():
self._save_timer = threading.Timer(self._SAVE_INTERVAL, self._flush)
self._save_timer.daemon = True
self._save_timer.start()
def _flush(self):
"""Actually write to disk (called by timer or atexit)."""
if not self._dirty:
return
try:
DATA_DIR.mkdir(parents=True, exist_ok=True)
data = {
"nodes": self.nodes,
"votes": [_serialize_vote_record(vote) for vote in self.votes],
"vouches": self.vouches,
"aliases": self.aliases,
}
write_domain_json(LEDGER_DOMAIN, LEDGER_FILE.name, data)
LEDGER_FILE.unlink(missing_ok=True)
self._dirty = False
except Exception as e:
logger.error(f"Failed to save reputation ledger: {e}")
# ─── Node Registration ────────────────────────────────────────────
def register_node(
self, node_id: str, public_key: str = "", public_key_algo: str = "", agent: bool = False
):
"""Register a node if not already known. Updates public_key if provided."""
if node_id not in self.nodes:
self.nodes[node_id] = {
"first_seen": time.time(),
"public_key": public_key,
"public_key_algo": public_key_algo,
"agent": agent,
}
self._save()
logger.info(
"Registered new node: %s",
privacy_log_label(node_id, label="node"),
)
elif public_key and not self.nodes[node_id].get("public_key"):
self.nodes[node_id]["public_key"] = public_key
if public_key_algo:
self.nodes[node_id]["public_key_algo"] = public_key_algo
self._save()
def link_identities(self, old_id: str, new_id: str) -> tuple[bool, str]:
"""Link a new node_id to an old one for reputation continuity."""
if not old_id or not new_id:
return False, "Missing old_id or new_id"
if old_id == new_id:
return False, "Old and new IDs must differ"
if new_id in self.aliases:
return False, f"{new_id} is already linked"
if old_id in self.aliases:
return False, f"{old_id} is already linked to {self.aliases[old_id]}"
if old_id in self.aliases.values():
return False, f"{old_id} is already the source of a link"
self.aliases[new_id] = old_id
self._scores_dirty = True
self._save()
logger.info(
"Linked identity: %s -> %s",
privacy_log_label(old_id, label="node"),
privacy_log_label(new_id, label="node"),
)
return True, "linked"
def get_node_age_days(self, node_id: str) -> float:
"""Get node age in days."""
node = self.nodes.get(node_id)
if not node:
return 0
return (time.time() - node.get("first_seen", time.time())) / 86400
def is_agent(self, node_id: str) -> bool:
"""Check if node is registered as an Agent (bot/AI)."""
return self.nodes.get(node_id, {}).get("agent", False)
# ─── Voting ───────────────────────────────────────────────────────
def _compute_vote_weight(self, voter_id: str) -> float:
"""Rep-weighted voting — your vote's power scales with reputation and tenure.
Two factors combine:
rep_factor: log10(1 + |rep|) / log10(101) → 0 rep ≈ 0.0, 100 rep = 1.0
tenure_factor: age_days / 720 → new = 0.0, 2yr+ = 1.0
Combined weight = max(0.1, rep_factor × tenure_factor), capped at 1.0.
Floor of 0.1 ensures every user's vote still counts for something.
| Rep | Day 1 | 6 months | 1 year | 2 years |
|------|-------|----------|--------|---------|
| 0 | 0.10 | 0.10 | 0.10 | 0.10 |
| 5 | 0.10 | 0.10 | 0.20 | 0.39 |
| 25 | 0.10 | 0.18 | 0.35 | 0.70 |
| 50 | 0.10 | 0.21 | 0.42 | 0.85 |
| 100+ | 0.10 | 0.25 | 0.50 | 1.00 |
"""
# ── Rep factor: logarithmic scale, 100 rep = full power ──
rep = self.get_reputation(voter_id).get("overall", 0)
rep_factor = math.log10(1 + abs(rep)) / math.log10(101) # 0→0.0, 100→1.0
# ── Tenure factor: linear over the 2-year decay window ──
age_days = self.get_node_age_days(voter_id)
decay_window = float(VOTE_DECAY_DAYS) if VOTE_DECAY_DAYS > 0 else 720.0
tenure_factor = min(1.0, age_days / decay_window)
weight = rep_factor * tenure_factor
return max(0.1, min(1.0, weight))
def cast_vote(
self, voter_id: str, target_id: str, vote: int, gate: str = ""
) -> tuple[bool, str, float]:
"""Cast a vote. Returns (success, reason, weight).
Rules:
- Self-votes allowed (costs -1 rep like any vote — net negative for voter)
- Must have rep >= MIN_REP_TO_VOTE (except first few bootstrap votes)
- One vote per voter per target per gate (can change direction)
- Vote value is weighted by voter's reputation and tenure
"""
if vote not in (1, -1):
return False, "Vote must be +1 or -1", 0.0
# Reputation burn: minimum rep to vote — only enforced after network bootstraps
network_size = len(self.nodes)
if network_size >= BOOTSTRAP_THRESHOLD:
voter_rep = self.get_reputation(voter_id).get("overall", 0)
if voter_rep < MIN_REP_TO_VOTE:
return (
False,
f"Need {MIN_REP_TO_VOTE} reputation to vote (you have {voter_rep}). Network has {network_size} nodes — rep-to-vote is active.",
0.0,
)
blinded_voter_id = _blind_voter(voter_id, _vote_storage_salt())
existing_vote = next(
(
v
for v in self.votes
if _stored_voter_id(v) == blinded_voter_id
and v["target_id"] == target_id
and v.get("gate", "") == gate
),
None,
)
existing_vote_value = None
if existing_vote is not None:
try:
existing_vote_value = int(existing_vote.get("vote", 0))
except (TypeError, ValueError):
existing_vote_value = None
if existing_vote and existing_vote_value == vote:
direction = "up" if vote == 1 else "down"
gate_str = f" in gate '{gate}'" if gate else ""
return False, f"Vote already set to {direction} on {target_id}{gate_str}", 0.0
# Remove existing vote from this voter for this target in this gate
self.votes = [
v
for v in self.votes
if not (
_stored_voter_id(v) == blinded_voter_id
and v["target_id"] == target_id
and v.get("gate", "") == gate
)
]
# Record the new vote
now = time.time()
weight = self._compute_vote_weight(voter_id)
is_direction_change = existing_vote is not None
self.votes.append(
{
"voter_id": voter_id,
"target_id": target_id,
"vote": vote,
"gate": gate,
"timestamp": now,
"weight": weight,
"agent_verify": self.is_agent(voter_id),
}
)
# Vote cost: costs the same as the vote's weight (if your vote only
# moves the score by 0.1, you only pay 0.1 — not a flat 1.0).
# Only on first vote, not direction changes. The target_id is the
# *blinded* voter ID so the root identity never touches disk.
if not is_direction_change:
self.votes.append(
{
"voter_id": "__system__",
"target_id": blinded_voter_id,
"vote": -1,
"gate": "",
"timestamp": now,
"weight": weight,
"agent_verify": False,
"vote_cost": True,
}
)
self._scores_dirty = True
self._save()
direction = "up" if vote == 1 else "down"
gate_str = f" in gate '{gate}'" if gate else ""
logger.info(
"Vote: %s voted %s on %s%s",
privacy_log_label(voter_id, label="node"),
direction,
privacy_log_label(target_id, label="node"),
f" in {privacy_log_label(gate, label='gate')}" if gate else "",
)
return True, f"Voted {direction} on {target_id}{gate_str}", weight
# ─── Trust Vouches ────────────────────────────────────────────────
def add_vouch(
self, voucher_id: str, target_id: str, note: str = "", timestamp: float | None = None
) -> tuple[bool, str]:
if not voucher_id or not target_id:
return False, "Missing voucher_id or target_id"
if voucher_id == target_id:
return False, "Cannot vouch for yourself"
ts = timestamp if timestamp is not None else time.time()
# Deduplicate vouches from same voucher to same target within 30 days
cutoff = ts - (30 * 86400)
for v in self.vouches:
if (
v.get("voucher_id") == voucher_id
and v.get("target_id") == target_id
and float(v.get("timestamp", 0)) >= cutoff
):
return False, "Duplicate vouch"
self.vouches.append(
{
"voucher_id": voucher_id,
"target_id": target_id,
"note": str(note)[:140],
"timestamp": float(ts),
}
)
self._save()
return True, "vouched"
def get_vouches(self, target_id: str, limit: int = 50) -> list[dict]:
if not target_id:
return []
entries = [v for v in self.vouches if v.get("target_id") == target_id]
entries = sorted(entries, key=lambda v: v.get("timestamp", 0), reverse=True)
return entries[: max(1, limit)]
# ─── Score Computation ────────────────────────────────────────────
def _recompute_scores(self):
"""Recompute all scores with time-weighted decay.
Votes decay linearly over VOTE_DECAY_MONTHS (24 months).
A vote cast today has full weight (1.0). A vote cast 12 months ago
has ~0.5 weight. A vote older than 24 months has 0 weight and is
skipped entirely. This means recent activity always matters more
than ancient history, but nothing ever fully disappears until 2 years
have passed.
"""
if not self._scores_dirty:
return
now = time.time()
decay_seconds = VOTE_DECAY_DAYS * 86400 if VOTE_DECAY_DAYS > 0 else 0
scores: dict[str, dict] = {}
for v in self.votes:
age = now - v["timestamp"]
# If decay is enabled, skip votes older than the window
if decay_seconds and age >= decay_seconds:
continue
# Time-decay multiplier: 1.0 for brand new, 0.0 at the boundary.
# Recent months weigh heaviest.
if decay_seconds:
decay_factor = 1.0 - (age / decay_seconds)
else:
decay_factor = 1.0
target = v["target_id"]
if target not in scores:
scores[target] = {"overall": 0.0, "gates": {}, "upvotes": 0, "downvotes": 0}
weighted = v["vote"] * v.get("weight", 1.0) * decay_factor
scores[target]["overall"] += weighted
if v["vote"] > 0:
scores[target]["upvotes"] += 1
else:
scores[target]["downvotes"] += 1
gate = v.get("gate", "")
if gate:
scores[target]["gates"].setdefault(gate, 0.0)
scores[target]["gates"][gate] += weighted
# Round to 1 decimal place — weighted votes produce fractional scores
for nid in scores:
scores[nid]["overall"] = round(scores[nid]["overall"], 1)
for gid in scores[nid]["gates"]:
scores[nid]["gates"][gid] = round(scores[nid]["gates"][gid], 1)
self._scores_cache = scores
self._scores_dirty = False
def get_reputation(self, node_id: str) -> dict:
"""Get reputation for a single node.
Returns {overall: int, gates: {gate_id: int}, upvotes: int, downvotes: int}
Scores are merged from three possible sources:
1. Direct scores on ``node_id`` (votes targeting posts by this identity).
2. Alias scores (if ``node_id`` is linked to an older identity).
3. Blinded-wallet scores — vote costs are stored under the deterministic
HMAC-blinded form of the voter's root identity so the raw private key
never touches disk. When the caller supplies the raw node_id we can
recompute the blind and merge those costs in.
"""
self._recompute_scores()
_zero = lambda: {"overall": 0, "gates": {}, "upvotes": 0, "downvotes": 0}
base = self._scores_cache.get(node_id, _zero())
# Merge alias (old identity linked to this one)
alias = self.aliases.get(node_id)
if alias:
old = self._scores_cache.get(alias, _zero())
base = self._merge_scores(base, old)
# Merge blinded-wallet costs (vote-cost records target the blinded ID)
blinded = _blind_voter(node_id, _vote_storage_salt())
if blinded and blinded != node_id:
wallet = self._scores_cache.get(blinded, _zero())
if wallet["overall"] != 0 or wallet["upvotes"] != 0 or wallet["downvotes"] != 0:
base = self._merge_scores(base, wallet)
return base
@staticmethod
def _merge_scores(a: dict, b: dict) -> dict:
merged = {
"overall": a["overall"] + b["overall"],
"gates": {},
"upvotes": a["upvotes"] + b["upvotes"],
"downvotes": a["downvotes"] + b["downvotes"],
}
gates = set(a.get("gates", {}).keys()) | set(b.get("gates", {}).keys())
for g in gates:
merged["gates"][g] = a.get("gates", {}).get(g, 0) + b.get("gates", {}).get(g, 0)
return merged
def get_all_reputations(self) -> dict[str, int]:
"""Get overall reputation for all known nodes."""
self._recompute_scores()
return {nid: s["overall"] for nid, s in self._scores_cache.items()}
def get_reputation_log(self, node_id: str, *, detailed: bool = False) -> dict:
"""Return reputation data for a node.
Public callers receive a summary-only view. Rich breakdowns remain
available to authenticated audit tooling.
"""
cutoff = time.time() - (VOTE_DECAY_DAYS * 86400)
rep = self.get_reputation(node_id)
result = {
"node_id": node_id,
"overall": rep.get("overall", 0),
"upvotes": rep.get("upvotes", 0),
"downvotes": rep.get("downvotes", 0),
}
if not detailed:
return result
alias = self.aliases.get(node_id)
target_ids = {node_id}
if alias:
target_ids.add(alias)
query_salt = os.urandom(8)
recent = [
{
"voter": _blind_voter(_stored_voter_id(v), query_salt),
"vote": v["vote"],
"gate": "",
"weight": v.get("weight", 1.0),
"agent_verify": v.get("agent_verify", False),
"age": f"{int((time.time() - v['timestamp']) / 86400)}d ago",
}
for v in sorted(self.votes, key=lambda x: x["timestamp"], reverse=True)
if v["target_id"] in target_ids and v["timestamp"] >= cutoff
][:20]
result.update(
{
"gates": {},
"recent_votes": recent,
"node_age_days": round(self.get_node_age_days(node_id), 1),
"is_agent": self.is_agent(node_id),
}
)
return result
# ─── DM Threshold ────────────────────────────────────────────────
def should_accept_message(self, sender_id: str, recipient_threshold: int) -> bool:
"""Check if sender meets recipient's reputation threshold for DMs."""
if recipient_threshold <= 0:
return True
sender_rep = self.get_reputation(sender_id).get("overall", 0)
return sender_rep >= recipient_threshold
# ─── Cleanup ──────────────────────────────────────────────────────
def cleanup_expired(self):
"""Remove votes older than the 2-year decay window."""
if VOTE_DECAY_DAYS <= 0:
return
cutoff = time.time() - (VOTE_DECAY_DAYS * 86400)
before = len(self.votes)
self.votes = [v for v in self.votes if v["timestamp"] >= cutoff]
after = len(self.votes)
if before != after:
self._scores_dirty = True
self._save()
logger.info(f"Cleaned up {before - after} expired votes")
# ─── Gate System ──────────────────────────────────────────────────────────
class GateManager:
"""Self-governing reputation-gated communities.
Anyone with rep >= 10 can create a gate. Entry requires meeting rep thresholds.
Getting downvoted below threshold bars you automatically.
"""
def __init__(self, ledger: ReputationLedger):
self.ledger = ledger
self.gates: dict[str, dict] = {}
self._dirty = False
self._save_lock = threading.Lock()
self._save_timer: threading.Timer | None = None
self._SAVE_INTERVAL = 5.0
atexit.register(self._flush)
self._load()
def _load(self):
domain_path = DATA_DIR / GATES_DOMAIN / GATES_FILE.name
if not domain_path.exists() and GATES_FILE.exists():
try:
legacy = read_secure_json(GATES_FILE, lambda: {})
write_domain_json(GATES_DOMAIN, GATES_FILE.name, legacy)
GATES_FILE.unlink(missing_ok=True)
except Exception as e:
logger.error(f"Failed to migrate gates: {e}")
try:
self.gates = read_domain_json(GATES_DOMAIN, GATES_FILE.name, lambda: {})
logger.info(f"Loaded {len(self.gates)} gates")
except Exception as e:
logger.error(f"Failed to load gates: {e}")
if self._apply_gate_catalog():
self._save()
def _apply_gate_catalog(self) -> bool:
"""Seed fixed private launch gates and retire obsolete defaults."""
changed = False
legacy_public_square = self.gates.get("public-square")
if isinstance(legacy_public_square, dict) and not legacy_public_square.get("fixed"):
self.gates.pop("public-square", None)
changed = True
for gate_id, seed in DEFAULT_PRIVATE_GATES.items():
gate = self.gates.get(gate_id)
if not isinstance(gate, dict):
self.gates[gate_id] = {
"creator_node_id": "!sb_seed",
"display_name": seed["display_name"],
"description": seed["description"],
"welcome": seed["welcome"],
"rules": {
"min_overall_rep": 0,
"min_gate_rep": {},
},
"created_at": time.time(),
"message_count": 0,
"fixed": True,
"sort_order": seed["sort_order"],
}
changed = True
continue
for key in ("display_name", "description", "welcome", "sort_order"):
if gate.get(key) != seed[key]:
gate[key] = seed[key]
changed = True
if gate.get("fixed") is not True:
gate["fixed"] = True
changed = True
if "rules" not in gate or not isinstance(gate["rules"], dict):
gate["rules"] = {"min_overall_rep": 0, "min_gate_rep": {}}
changed = True
gate["rules"].setdefault("min_overall_rep", 0)
gate["rules"].setdefault("min_gate_rep", {})
gate.setdefault("message_count", 0)
gate.setdefault("created_at", time.time())
return changed
def _save(self):
"""Mark dirty and schedule a coalesced disk write."""
self._dirty = True
with self._save_lock:
if self._save_timer is None or not self._save_timer.is_alive():
self._save_timer = threading.Timer(self._SAVE_INTERVAL, self._flush)
self._save_timer.daemon = True
self._save_timer.start()
def _flush(self):
"""Actually write to disk (called by timer or atexit)."""
if not self._dirty:
return
try:
DATA_DIR.mkdir(parents=True, exist_ok=True)
write_domain_json(GATES_DOMAIN, GATES_FILE.name, self.gates)
GATES_FILE.unlink(missing_ok=True)
self._dirty = False
except Exception as e:
logger.error(f"Failed to save gates: {e}")
def create_gate(
self,
creator_id: str,
gate_id: str,
display_name: str,
min_overall_rep: int = 0,
min_gate_rep: Optional[dict] = None,
description: str = "",
) -> tuple[bool, str]:
"""Create a new gate. Rep gate disabled until network reaches critical mass."""
if not ALLOW_DYNAMIC_GATES:
return False, "Gate creation is disabled for the fixed private launch catalog"
gate_id = gate_id.lower().strip()
if not gate_id or not gate_id.isalnum() and "-" not in gate_id:
return False, "Gate ID must be alphanumeric (hyphens allowed)"
if len(gate_id) > 32:
return False, "Gate ID too long (max 32 chars)"
if gate_id in self.gates:
return False, f"Gate '{gate_id}' already exists"
self.gates[gate_id] = {
"creator_node_id": creator_id,
"display_name": display_name[:64],
"description": description[:240],
"rules": {
"min_overall_rep": min_overall_rep,
"min_gate_rep": min_gate_rep or {},
},
"created_at": time.time(),
"message_count": 0,
"fixed": False,
"sort_order": 1000,
}
self._save()
logger.info(
"Gate created: %s by %s",
privacy_log_label(gate_id, label="gate"),
privacy_log_label(creator_id, label="node"),
)
return True, f"Gate '{gate_id}' created"
def can_enter(self, node_id: str, gate_id: str) -> tuple[bool, str]:
"""Check if a node meets the entry rules for a gate."""
gate = self.gates.get(gate_id)
if not gate:
return False, f"Gate '{gate_id}' does not exist"
rules = gate.get("rules", {})
rep = self.ledger.get_reputation(node_id)
# Check overall rep requirement
min_overall = rules.get("min_overall_rep", 0)
if rep.get("overall", 0) < min_overall:
return False, f"Need {min_overall} overall rep (you have {rep.get('overall', 0)})"
# Check gate-specific rep requirements
for req_gate, req_min in rules.get("min_gate_rep", {}).items():
gate_rep = rep.get("gates", {}).get(req_gate, 0)
if gate_rep < req_min:
return False, f"Need {req_min} rep in '{req_gate}' gate (you have {gate_rep})"
return True, "Access granted"
def list_gates(self) -> list[dict]:
"""List all gates with metadata."""
result = []
for gid, gate in self.gates.items():
result.append(
{
"gate_id": gid,
"display_name": gate.get("display_name", gid),
"description": gate.get("description", ""),
"welcome": gate.get("welcome", ""),
"rules": gate.get("rules", {}),
"created_at": gate.get("created_at", 0),
"fixed": bool(gate.get("fixed", False)),
"sort_order": int(gate.get("sort_order", 1000) or 1000),
}
)
return sorted(
result,
key=lambda x: (
0 if x.get("fixed") else 1,
int(x.get("sort_order", 1000) or 1000),
-float(x.get("created_at", 0) or 0),
x.get("gate_id", ""),
),
)
def get_gate(self, gate_id: str) -> Optional[dict]:
"""Get gate details."""
gate = self.gates.get(gate_id)
if not gate:
return None
public_gate = {
key: value
for key, value in gate.items()
if key not in {"creator_node_id", "message_count"}
}
return {
"gate_id": gate_id,
**public_gate,
}
def record_message(self, gate_id: str):
"""Increment message count for a gate."""
if gate_id in self.gates:
self.gates[gate_id]["message_count"] = self.gates[gate_id].get("message_count", 0) + 1
self._save()
def is_ratified(self, gate_id: str) -> bool:
"""Check if a gate is ratified (has permanent chain address).
Before BOOTSTRAP_THRESHOLD nodes: all gates are ratified (early access).
After bootstrap: gates need cumulative member rep >= GATE_RATIFICATION_REP.
"""
if len(self.ledger.nodes) < BOOTSTRAP_THRESHOLD:
return True # Pre-bootstrap: all gates are ratified
gate = self.gates.get(gate_id)
if not gate:
return False
# Sum rep of all nodes that have gate-specific rep in this gate
all_reps = self.ledger.get_all_reputations()
self.ledger._recompute_scores()
cumulative = 0
for nid, score_data in self.ledger._scores_cache.items():
gate_rep = score_data.get("gates", {}).get(gate_id, 0)
if gate_rep > 0:
cumulative += gate_rep
return cumulative >= GATE_RATIFICATION_REP
def get_ratification_status(self, gate_id: str) -> dict:
"""Get gate's ratification progress."""
gate = self.gates.get(gate_id)
if not gate:
return {"ratified": False, "reason": "Gate not found"}
network_size = len(self.ledger.nodes)
if network_size < BOOTSTRAP_THRESHOLD:
return {
"ratified": True,
"reason": f"Pre-bootstrap ({network_size}/{BOOTSTRAP_THRESHOLD} nodes) — all gates ratified",
"cumulative_rep": 0,
"required_rep": GATE_RATIFICATION_REP,
}
# Compute cumulative gate rep
self.ledger._recompute_scores()
cumulative = 0
contributors = 0
for nid, score_data in self.ledger._scores_cache.items():
gate_rep = score_data.get("gates", {}).get(gate_id, 0)
if gate_rep > 0:
cumulative += gate_rep
contributors += 1
ratified = cumulative >= GATE_RATIFICATION_REP
return {
"ratified": ratified,
"cumulative_rep": cumulative,
"required_rep": GATE_RATIFICATION_REP,
"contributors": contributors,
"reason": (
"Ratified — permanent chain address"
if ratified
else f"Need {GATE_RATIFICATION_REP - cumulative} more cumulative rep"
),
}
# ─── Module-level singletons ─────────────────────────────────────────────
reputation_ledger = ReputationLedger()
gate_manager = GateManager(reputation_ledger)
backend/services/mesh/mesh_rns.py
+1622
View File
File diff suppressed because it is too large Load Diff
backend/services/mesh/mesh_router.py
+1087
View File
File diff suppressed because it is too large Load Diff
backend/services/mesh/mesh_schema.py
+399
View File
@@ -0,0 +1,399 @@
"""Central schema registry for mesh protocol events."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Callable
from services.mesh.mesh_protocol import normalize_payload, PROTOCOL_VERSION, NETWORK_ID
def _safe_int(val, default=0):
try:
return int(val)
except (TypeError, ValueError):
return default
@dataclass(frozen=True)
class EventSchema:
event_type: str
required_fields: tuple[str, ...]
optional_fields: tuple[str, ...]
validate: Callable[[dict[str, Any]], tuple[bool, str]]
def validate_payload(self, payload: dict[str, Any]) -> tuple[bool, str]:
return self.validate(payload)
def _require_fields(payload: dict[str, Any], fields: tuple[str, ...]) -> tuple[bool, str]:
for key in fields:
if key not in payload:
return False, f"Missing field: {key}"
return True, "ok"
def _validate_message(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(
payload, ("message", "destination", "channel", "priority", "ephemeral")
)
if not ok:
return ok, reason
if payload.get("priority") not in ("normal", "high", "emergency", "low"):
return False, "Invalid priority"
if not isinstance(payload.get("ephemeral"), bool):
return False, "ephemeral must be boolean"
return True, "ok"
def _validate_gate_message(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(payload, ("gate", "ciphertext", "nonce", "sender_ref"))
if not ok:
return ok, reason
if "message" in payload:
return False, "plaintext gate message field is not allowed"
gate = str(payload.get("gate", "")).strip().lower()
if not gate:
return False, "gate cannot be empty"
if "epoch" in payload:
epoch = _safe_int(payload.get("epoch", 0) or 0, 0)
if epoch <= 0:
return False, "epoch must be a positive integer"
elif (
not str(payload.get("ciphertext", "")).strip()
and not str(payload.get("nonce", "")).strip()
and not str(payload.get("sender_ref", "")).strip()
):
return False, "epoch must be a positive integer"
if not str(payload.get("ciphertext", "")).strip():
return False, "ciphertext cannot be empty"
if not str(payload.get("nonce", "")).strip():
return False, "nonce cannot be empty"
if not str(payload.get("sender_ref", "")).strip():
return False, "sender_ref cannot be empty"
payload_format = str(payload.get("format", "mls1") or "mls1").strip().lower()
if payload_format != "mls1":
return False, "Unsupported gate message format"
return True, "ok"
def _validate_vote(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(payload, ("target_id", "vote", "gate"))
if not ok:
return ok, reason
if payload.get("vote") not in (-1, 1):
return False, "Invalid vote"
return True, "ok"
def _validate_gate_create(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(payload, ("gate_id", "display_name", "rules"))
if not ok:
return ok, reason
if not isinstance(payload.get("rules"), dict):
return False, "rules must be an object"
return True, "ok"
def _validate_prediction(payload: dict[str, Any]) -> tuple[bool, str]:
return _require_fields(payload, ("market_title", "side", "stake_amount"))
def _validate_stake(payload: dict[str, Any]) -> tuple[bool, str]:
return _require_fields(payload, ("message_id", "poster_id", "side", "amount", "duration_days"))
def _validate_dm_block(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(payload, ("blocked_id", "action"))
if not ok:
return ok, reason
if payload.get("action") not in ("block", "unblock"):
return False, "Invalid action"
return True, "ok"
def _validate_dm_key(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(payload, ("dh_pub_key", "dh_algo", "timestamp"))
if not ok:
return ok, reason
if payload.get("dh_algo") not in ("X25519", "ECDH", "ECDH_P256"):
return False, "Invalid dh_algo"
return True, "ok"
def _validate_dm_message(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(
payload, ("recipient_id", "delivery_class", "recipient_token", "ciphertext", "msg_id", "timestamp")
)
if not ok:
return ok, reason
delivery_class = str(payload.get("delivery_class", "")).lower()
if delivery_class not in ("request", "shared"):
return False, "Invalid delivery_class"
if delivery_class == "shared" and not str(payload.get("recipient_token", "")).strip():
return False, "recipient_token required for shared delivery"
dm_format = str(payload.get("format", "mls1") or "mls1").strip().lower()
if dm_format not in ("mls1", "dm1"):
return False, f"Unknown DM format: {dm_format}"
return True, "ok"
def _validate_mailbox_claims(claims: Any) -> tuple[bool, str]:
if not isinstance(claims, list) or not claims:
return False, "mailbox_claims must be a non-empty list"
for claim in claims:
if not isinstance(claim, dict):
return False, "mailbox_claims entries must be objects"
claim_type = str(claim.get("type", "")).lower()
if claim_type not in ("self", "requests", "shared"):
return False, "Invalid mailbox claim type"
if not str(claim.get("token", "")).strip():
return False, f"{claim_type} mailbox claims require token"
return True, "ok"
def _validate_dm_poll(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(payload, ("mailbox_claims", "timestamp", "nonce"))
if not ok:
return ok, reason
return _validate_mailbox_claims(payload.get("mailbox_claims"))
def _validate_dm_count(payload: dict[str, Any]) -> tuple[bool, str]:
return _validate_dm_poll(payload)
def _validate_key_rotate(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(
payload,
(
"old_node_id",
"old_public_key",
"old_public_key_algo",
"new_public_key",
"new_public_key_algo",
"timestamp",
"old_signature",
),
)
if not ok:
return ok, reason
return True, "ok"
def _validate_key_revoke(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(
payload,
(
"revoked_public_key",
"revoked_public_key_algo",
"revoked_at",
"grace_until",
"reason",
),
)
if not ok:
return ok, reason
revoked_at = _safe_int(payload.get("revoked_at", 0) or 0, 0)
grace_until = _safe_int(payload.get("grace_until", 0) or 0, 0)
if revoked_at <= 0:
return False, "revoked_at must be a positive timestamp"
if grace_until < revoked_at:
return False, "grace_until must be >= revoked_at"
return True, "ok"
def _validate_abuse_report(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(payload, ("target_id", "reason"))
if not ok:
return ok, reason
if not str(payload.get("reason", "")).strip():
return False, "reason cannot be empty"
return True, "ok"
SCHEMA_REGISTRY: dict[str, EventSchema] = {
"message": EventSchema(
event_type="message",
required_fields=("message", "destination", "channel", "priority", "ephemeral"),
optional_fields=(),
validate=_validate_message,
),
"gate_message": EventSchema(
event_type="gate_message",
required_fields=("gate", "ciphertext", "nonce", "sender_ref"),
optional_fields=("format",),
validate=_validate_gate_message,
),
"vote": EventSchema(
event_type="vote",
required_fields=("target_id", "vote", "gate"),
optional_fields=(),
validate=_validate_vote,
),
"gate_create": EventSchema(
event_type="gate_create",
required_fields=("gate_id", "display_name", "rules"),
optional_fields=(),
validate=_validate_gate_create,
),
"prediction": EventSchema(
event_type="prediction",
required_fields=("market_title", "side", "stake_amount"),
optional_fields=(),
validate=_validate_prediction,
),
"stake": EventSchema(
event_type="stake",
required_fields=("message_id", "poster_id", "side", "amount", "duration_days"),
optional_fields=(),
validate=_validate_stake,
),
"dm_block": EventSchema(
event_type="dm_block",
required_fields=("blocked_id", "action"),
optional_fields=(),
validate=_validate_dm_block,
),
"dm_key": EventSchema(
event_type="dm_key",
required_fields=("dh_pub_key", "dh_algo", "timestamp"),
optional_fields=(),
validate=_validate_dm_key,
),
"dm_message": EventSchema(
event_type="dm_message",
required_fields=("recipient_id", "delivery_class", "recipient_token", "ciphertext", "msg_id", "timestamp"),
optional_fields=(),
validate=_validate_dm_message,
),
"dm_poll": EventSchema(
event_type="dm_poll",
required_fields=("mailbox_claims", "timestamp", "nonce"),
optional_fields=(),
validate=_validate_dm_poll,
),
"dm_count": EventSchema(
event_type="dm_count",
required_fields=("mailbox_claims", "timestamp", "nonce"),
optional_fields=(),
validate=_validate_dm_count,
),
"key_rotate": EventSchema(
event_type="key_rotate",
required_fields=(
"old_node_id",
"old_public_key",
"old_public_key_algo",
"new_public_key",
"new_public_key_algo",
"timestamp",
"old_signature",
),
optional_fields=(),
validate=_validate_key_rotate,
),
"key_revoke": EventSchema(
event_type="key_revoke",
required_fields=(
"revoked_public_key",
"revoked_public_key_algo",
"revoked_at",
"grace_until",
"reason",
),
optional_fields=(),
validate=_validate_key_revoke,
),
"abuse_report": EventSchema(
event_type="abuse_report",
required_fields=("target_id", "reason"),
optional_fields=("gate", "evidence"),
validate=_validate_abuse_report,
),
}
PUBLIC_LEDGER_EVENT_TYPES: frozenset[str] = frozenset(
{
"message",
"vote",
"gate_create",
"gate_message",
"prediction",
"stake",
"key_rotate",
"key_revoke",
"abuse_report",
}
)
_PUBLIC_LEDGER_FORBIDDEN_FIELDS: frozenset[str] = frozenset(
{
"ip",
"ip_address",
"origin_ip",
"source_ip",
"client_ip",
"host",
"hostname",
"origin",
"originator",
"originator_hint",
"routing_hint",
"route",
"route_hint",
"route_reason",
"routed_via",
"transport",
"transport_handle",
"transport_lock",
"recipient_id",
"recipient_token",
"delivery_class",
"mailbox_claims",
"dh_pub_key",
"sender_token",
}
)
def get_schema(event_type: str) -> EventSchema | None:
return SCHEMA_REGISTRY.get(event_type)
def validate_event_payload(event_type: str, payload: dict[str, Any]) -> tuple[bool, str]:
schema = get_schema(event_type)
if not schema:
return False, "Unknown event_type"
normalized = normalize_payload(event_type, payload)
if normalized != payload:
return False, "Payload is not normalized"
if event_type not in ("message", "gate_message") and "ephemeral" in payload:
return False, "ephemeral not allowed for this event type"
return schema.validate_payload(payload)
def validate_public_ledger_payload(event_type: str, payload: dict[str, Any]) -> tuple[bool, str]:
if event_type not in PUBLIC_LEDGER_EVENT_TYPES:
return False, f"{event_type} is not allowed on the public ledger"
forbidden = sorted(
key
for key in payload.keys()
if str(key or "").strip().lower() in _PUBLIC_LEDGER_FORBIDDEN_FIELDS
)
if forbidden:
return False, f"public ledger payload contains forbidden fields: {', '.join(forbidden)}"
if event_type == "message":
destination = str(payload.get("destination", "") or "").strip().lower()
if destination and destination != "broadcast":
return False, "public ledger message destination must be broadcast"
return True, "ok"
def validate_protocol_fields(protocol_version: str, network_id: str) -> tuple[bool, str]:
if protocol_version != PROTOCOL_VERSION:
return False, "Unsupported protocol_version"
if network_id != NETWORK_ID:
return False, "network_id mismatch"
return True, "ok"
backend/services/mesh/mesh_secure_storage.py
+577
View File
@@ -0,0 +1,577 @@
"""Secure local storage helpers for Wormhole-owned state.
Windows uses DPAPI to protect local key envelopes. Root secure-json payloads
still use a dedicated master key, while domain-scoped payloads now use
independent per-domain keys so compromise of one domain key does not
automatically collapse every other Wormhole compartment. Non-Windows platforms
can fall back to raw local key files only when tests are running or an
explicit development/CI opt-in is set until native keyrings are added in the
desktop phase.
"""
from __future__ import annotations
import base64
import ctypes
import hashlib
import hmac
import json
import os
import re
import tempfile
import time
from pathlib import Path
from typing import Any, Callable, TypeVar
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
DATA_DIR = Path(__file__).resolve().parents[2] / "data"
MASTER_KEY_FILE = DATA_DIR / "wormhole_secure_store.key"
_ENVELOPE_KIND = "sb_secure_json"
_ENVELOPE_VERSION = 1
_MASTER_KIND = "sb_secure_master_key"
_MASTER_VERSION = 1
_DOMAIN_KEY_KIND = "sb_secure_domain_key"
_DOMAIN_KEY_VERSION = 1
_MASTER_KEY_CACHE: tuple[str, bytes] | None = None
_DOMAIN_KEY_CACHE: dict[str, tuple[str, bytes]] = {}
T = TypeVar("T")
class SecureStorageError(RuntimeError):
"""Raised when secure local storage cannot be read or written safely."""
def _atomic_write_text(target: Path, content: str, encoding: str = "utf-8") -> None:
"""Write content atomically via temp file + os.replace()."""
parent = target.parent
parent.mkdir(parents=True, exist_ok=True)
fd, tmp_path = tempfile.mkstemp(dir=str(parent), suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding=encoding) as handle:
handle.write(content)
handle.flush()
os.fsync(handle.fileno())
last_exc: Exception | None = None
for _ in range(5):
try:
os.replace(tmp_path, str(target))
last_exc = None
break
except PermissionError as exc:
last_exc = exc
time.sleep(0.02)
if last_exc is not None:
raise last_exc
except BaseException:
try:
os.unlink(tmp_path)
except OSError:
pass
raise
def _b64(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def _unb64(data: str | bytes | None) -> bytes:
if not data:
return b""
if isinstance(data, bytes):
return base64.b64decode(data)
return base64.b64decode(data.encode("ascii"))
def _stable_json(value: Any) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8")
def _envelope_aad(path: Path) -> bytes:
return f"shadowbroker|secure-json|v{_ENVELOPE_VERSION}|{path.name}".encode("utf-8")
def _master_aad() -> bytes:
return f"shadowbroker|master-key|v{_MASTER_VERSION}".encode("utf-8")
def _domain_key_aad(domain: str) -> bytes:
return f"shadowbroker|domain-key|v{_DOMAIN_KEY_VERSION}|{domain}".encode("utf-8")
def _storage_root(base_dir: str | Path | None = None) -> Path:
return Path(base_dir).resolve() if base_dir is not None else DATA_DIR.resolve()
def _domain_key_dir(base_dir: str | Path | None = None) -> Path:
return _storage_root(base_dir) / "_domain_keys"
def _normalize_domain_name(domain: str) -> str:
domain_name = str(domain or "").strip().lower()
if not domain_name:
raise SecureStorageError("domain name required for domain-scoped storage")
if not re.fullmatch(r"[a-z0-9_]+", domain_name):
raise SecureStorageError(f"invalid domain name: {domain_name!r}")
return domain_name
def _domain_aad(domain: str, filename: str) -> bytes:
return f"shadowbroker|domain-json|v{_ENVELOPE_VERSION}|{domain}|{filename}".encode("utf-8")
def _master_envelope_for_windows(protected_key: bytes, *, provider: str) -> dict[str, Any]:
return {
"kind": _MASTER_KIND,
"version": _MASTER_VERSION,
"provider": provider,
"protected_key": _b64(protected_key),
}
def _master_envelope_for_fallback(raw_key: bytes) -> dict[str, Any]:
return {
"kind": _MASTER_KIND,
"version": _MASTER_VERSION,
"provider": "raw",
"key": _b64(raw_key),
}
def _domain_key_envelope_for_windows(
domain: str,
protected_key: bytes,
*,
provider: str,
) -> dict[str, Any]:
return {
"kind": _DOMAIN_KEY_KIND,
"version": _DOMAIN_KEY_VERSION,
"provider": provider,
"domain": domain,
"protected_key": _b64(protected_key),
}
def _domain_key_envelope_for_fallback(domain: str, raw_key: bytes) -> dict[str, Any]:
return {
"kind": _DOMAIN_KEY_KIND,
"version": _DOMAIN_KEY_VERSION,
"provider": "raw",
"domain": domain,
"key": _b64(raw_key),
}
def _secure_envelope(path: Path, nonce: bytes, ciphertext: bytes) -> dict[str, Any]:
return {
"kind": _ENVELOPE_KIND,
"version": _ENVELOPE_VERSION,
"path": path.name,
"nonce": _b64(nonce),
"ciphertext": _b64(ciphertext),
}
def _is_secure_envelope(value: Any) -> bool:
return (
isinstance(value, dict)
and str(value.get("kind", "") or "") == _ENVELOPE_KIND
and int(value.get("version", 0) or 0) == _ENVELOPE_VERSION
and "nonce" in value
and "ciphertext" in value
)
def _is_windows() -> bool:
return os.name == "nt"
def _raw_fallback_allowed() -> bool:
if _is_windows():
return False
if os.environ.get("PYTEST_CURRENT_TEST"):
return True
try:
from services.config import get_settings
settings = get_settings()
if bool(getattr(settings, "MESH_ALLOW_RAW_SECURE_STORAGE_FALLBACK", False)):
return True
except Exception:
pass
return False
if _is_windows():
from ctypes import wintypes
class _DATA_BLOB(ctypes.Structure):
_fields_ = [("cbData", wintypes.DWORD), ("pbData", ctypes.POINTER(ctypes.c_byte))]
_crypt32 = ctypes.windll.crypt32
_kernel32 = ctypes.windll.kernel32
_CRYPTPROTECT_UI_FORBIDDEN = 0x1
_CRYPTPROTECT_LOCAL_MACHINE = 0x4
_crypt32.CryptProtectData.argtypes = [
ctypes.POINTER(_DATA_BLOB),
wintypes.LPCWSTR,
ctypes.c_void_p,
ctypes.c_void_p,
ctypes.c_void_p,
wintypes.DWORD,
ctypes.POINTER(_DATA_BLOB),
]
_crypt32.CryptProtectData.restype = wintypes.BOOL
_crypt32.CryptUnprotectData.argtypes = [
ctypes.POINTER(_DATA_BLOB),
ctypes.c_void_p,
ctypes.c_void_p,
ctypes.c_void_p,
ctypes.c_void_p,
wintypes.DWORD,
ctypes.POINTER(_DATA_BLOB),
]
_crypt32.CryptUnprotectData.restype = wintypes.BOOL
_kernel32.LocalFree.argtypes = [ctypes.c_void_p]
_kernel32.LocalFree.restype = ctypes.c_void_p
def _blob_from_bytes(data: bytes) -> tuple[_DATA_BLOB, ctypes.Array[ctypes.c_char]]:
buf = ctypes.create_string_buffer(data, len(data))
blob = _DATA_BLOB(len(data), ctypes.cast(buf, ctypes.POINTER(ctypes.c_byte)))
return blob, buf
def _bytes_from_blob(blob: _DATA_BLOB) -> bytes:
return ctypes.string_at(blob.pbData, blob.cbData)
def _dpapi_protect(data: bytes, *, machine_scope: bool) -> bytes:
in_blob, in_buf = _blob_from_bytes(data)
out_blob = _DATA_BLOB()
flags = _CRYPTPROTECT_UI_FORBIDDEN
if machine_scope:
flags |= _CRYPTPROTECT_LOCAL_MACHINE
if not _crypt32.CryptProtectData(
ctypes.byref(in_blob),
"ShadowBroker Wormhole",
None,
None,
None,
flags,
ctypes.byref(out_blob),
):
raise ctypes.WinError()
try:
_ = in_buf # Keep the backing buffer alive for the API call.
return _bytes_from_blob(out_blob)
finally:
if out_blob.pbData:
_kernel32.LocalFree(out_blob.pbData)
def _dpapi_unprotect(data: bytes) -> bytes:
in_blob, in_buf = _blob_from_bytes(data)
out_blob = _DATA_BLOB()
if not _crypt32.CryptUnprotectData(
ctypes.byref(in_blob),
None,
None,
None,
None,
_CRYPTPROTECT_UI_FORBIDDEN,
ctypes.byref(out_blob),
):
raise ctypes.WinError()
try:
_ = in_buf # Keep the backing buffer alive for the API call.
return _bytes_from_blob(out_blob)
finally:
if out_blob.pbData:
_kernel32.LocalFree(out_blob.pbData)
else:
def _dpapi_protect(data: bytes, *, machine_scope: bool) -> bytes:
raise SecureStorageError("DPAPI is only available on Windows")
def _dpapi_unprotect(data: bytes) -> bytes:
raise SecureStorageError("DPAPI is only available on Windows")
def _load_master_key() -> bytes:
global _MASTER_KEY_CACHE
DATA_DIR.mkdir(parents=True, exist_ok=True)
cache_key = str(MASTER_KEY_FILE.resolve())
if _MASTER_KEY_CACHE and _MASTER_KEY_CACHE[0] == cache_key:
return _MASTER_KEY_CACHE[1]
if not MASTER_KEY_FILE.exists():
raw_key = os.urandom(32)
if _is_windows():
envelope = _master_envelope_for_windows(
_dpapi_protect(raw_key, machine_scope=True),
provider="dpapi-machine",
)
else:
if not _raw_fallback_allowed():
raise SecureStorageError(
"Non-Windows secure storage requires a native keyring or explicit raw fallback opt-in"
)
envelope = _master_envelope_for_fallback(raw_key)
_atomic_write_text(MASTER_KEY_FILE, json.dumps(envelope, indent=2), encoding="utf-8")
_MASTER_KEY_CACHE = (cache_key, raw_key)
return raw_key
try:
payload = json.loads(MASTER_KEY_FILE.read_text(encoding="utf-8"))
except Exception as exc:
raise SecureStorageError(f"Failed to load secure storage master key: {exc}") from exc
if not isinstance(payload, dict) or payload.get("kind") != _MASTER_KIND:
raise SecureStorageError("Malformed secure storage master key envelope")
provider = str(payload.get("provider", "") or "").lower()
if provider in {"dpapi", "dpapi-user", "dpapi-machine"}:
try:
raw_key = _dpapi_unprotect(_unb64(payload.get("protected_key")))
_MASTER_KEY_CACHE = (cache_key, raw_key)
return raw_key
except Exception as exc:
raise SecureStorageError(f"Failed to unwrap DPAPI master key: {exc}") from exc
if provider == "raw":
if not _raw_fallback_allowed():
raise SecureStorageError(
"Raw secure-storage envelopes are disabled outside debug/test unless explicitly opted in"
)
raw_key = _unb64(payload.get("key"))
_MASTER_KEY_CACHE = (cache_key, raw_key)
return raw_key
raise SecureStorageError(f"Unsupported secure storage provider: {provider}")
def _domain_key_file(domain: str, *, base_dir: str | Path | None = None) -> Path:
domain_name = _normalize_domain_name(domain)
return (_domain_key_dir(base_dir) / f"{domain_name}.key").resolve()
def _load_domain_key(
domain: str,
*,
create_if_missing: bool = True,
base_dir: str | Path | None = None,
) -> bytes:
domain_name = _normalize_domain_name(domain)
root = _storage_root(base_dir)
root.mkdir(parents=True, exist_ok=True)
key_file = _domain_key_file(domain_name, base_dir=base_dir)
cache_key = str(key_file)
cache_slot = f"{root}::{domain_name}"
cached = _DOMAIN_KEY_CACHE.get(cache_slot)
if cached and cached[0] == cache_key:
return cached[1]
if not key_file.exists():
if not create_if_missing:
raise SecureStorageError(f"Domain key not found for {domain_name}")
raw_key = os.urandom(32)
if _is_windows():
envelope = _domain_key_envelope_for_windows(
domain_name,
_dpapi_protect(raw_key, machine_scope=True),
provider="dpapi-machine",
)
else:
if not _raw_fallback_allowed():
raise SecureStorageError(
"Non-Windows secure storage requires a native keyring or explicit raw fallback opt-in"
)
envelope = _domain_key_envelope_for_fallback(domain_name, raw_key)
_atomic_write_text(key_file, json.dumps(envelope, indent=2), encoding="utf-8")
_DOMAIN_KEY_CACHE[cache_slot] = (cache_key, raw_key)
return raw_key
try:
payload = json.loads(key_file.read_text(encoding="utf-8"))
except Exception as exc:
raise SecureStorageError(f"Failed to load domain key for {domain_name}: {exc}") from exc
if not isinstance(payload, dict) or payload.get("kind") != _DOMAIN_KEY_KIND:
raise SecureStorageError(f"Malformed domain key envelope for {domain_name}")
if str(payload.get("domain", "") or "").strip().lower() != domain_name:
raise SecureStorageError(f"Domain key envelope mismatch for {domain_name}")
provider = str(payload.get("provider", "") or "").lower()
if provider in {"dpapi", "dpapi-user", "dpapi-machine"}:
try:
raw_key = _dpapi_unprotect(_unb64(payload.get("protected_key")))
_DOMAIN_KEY_CACHE[cache_slot] = (cache_key, raw_key)
return raw_key
except Exception as exc:
raise SecureStorageError(f"Failed to unwrap domain key for {domain_name}: {exc}") from exc
if provider == "raw":
if not _raw_fallback_allowed():
raise SecureStorageError(
"Raw secure-storage envelopes are disabled outside debug/test unless explicitly opted in"
)
raw_key = _unb64(payload.get("key"))
_DOMAIN_KEY_CACHE[cache_slot] = (cache_key, raw_key)
return raw_key
raise SecureStorageError(f"Unsupported domain key provider for {domain_name}: {provider}")
def _derive_legacy_domain_key(domain: str) -> bytes:
domain_name = _normalize_domain_name(domain)
return hmac.new(
_load_master_key(),
domain_name.encode("utf-8"),
hashlib.sha256,
).digest()
def _domain_file_path(domain: str, filename: str, *, base_dir: str | Path | None = None) -> Path:
domain_name = _normalize_domain_name(domain)
file_name = str(filename or "").strip()
if not file_name:
raise SecureStorageError("filename required for domain-scoped storage")
if not re.fullmatch(r"[a-z0-9_.]+", file_name):
raise SecureStorageError(f"invalid filename: {file_name!r}")
root = _storage_root(base_dir)
resolved = (root / domain_name / file_name).resolve()
if not str(resolved).startswith(str(root)):
raise SecureStorageError("domain storage path traversal rejected")
return resolved
def write_secure_json(path: str | Path, payload: Any) -> None:
file_path = Path(path)
file_path.parent.mkdir(parents=True, exist_ok=True)
master_key = _load_master_key()
nonce = os.urandom(12)
ciphertext = AESGCM(master_key).encrypt(nonce, _stable_json(payload), _envelope_aad(file_path))
envelope = _secure_envelope(file_path, nonce, ciphertext)
_atomic_write_text(file_path, json.dumps(envelope, indent=2), encoding="utf-8")
def read_secure_json(path: str | Path, default_factory: Callable[[], T]) -> T:
file_path = Path(path)
if not file_path.exists():
return default_factory()
try:
raw = json.loads(file_path.read_text(encoding="utf-8"))
except Exception as exc:
raise SecureStorageError(f"Failed to parse secure JSON {file_path.name}: {exc}") from exc
if _is_secure_envelope(raw):
master_key = _load_master_key()
try:
plaintext = AESGCM(master_key).decrypt(
_unb64(raw.get("nonce")),
_unb64(raw.get("ciphertext")),
_envelope_aad(file_path),
)
except Exception as exc:
raise SecureStorageError(f"Failed to decrypt secure JSON {file_path.name}: {exc}") from exc
try:
return json.loads(plaintext.decode("utf-8"))
except Exception as exc:
raise SecureStorageError(
f"Failed to decode secure JSON payload {file_path.name}: {exc}"
) from exc
# Legacy plaintext JSON: migrate in place on first successful read.
migrated = raw if isinstance(raw, (dict, list)) else default_factory()
write_secure_json(file_path, migrated)
return migrated
def write_domain_json(
domain: str,
filename: str,
payload: Any,
*,
base_dir: str | Path | None = None,
) -> Path:
file_path = _domain_file_path(domain, filename, base_dir=base_dir)
file_path.parent.mkdir(parents=True, exist_ok=True)
nonce = os.urandom(12)
domain_name = _normalize_domain_name(domain)
ciphertext = AESGCM(_load_domain_key(domain_name, base_dir=base_dir)).encrypt(
nonce,
_stable_json(payload),
_domain_aad(domain_name, file_path.name),
)
envelope = _secure_envelope(file_path, nonce, ciphertext)
_atomic_write_text(file_path, json.dumps(envelope, indent=2), encoding="utf-8")
return file_path
def read_domain_json(
domain: str,
filename: str,
default_factory: Callable[[], T],
*,
base_dir: str | Path | None = None,
) -> T:
file_path = _domain_file_path(domain, filename, base_dir=base_dir)
domain_name = _normalize_domain_name(domain)
if not file_path.exists():
return default_factory()
try:
raw = json.loads(file_path.read_text(encoding="utf-8"))
except Exception as exc:
raise SecureStorageError(f"Failed to parse domain JSON {file_path.name}: {exc}") from exc
if _is_secure_envelope(raw):
aad = _domain_aad(domain_name, file_path.name)
plaintext: bytes | None = None
used_legacy_key = False
used_master_key = False
try:
current_key = _load_domain_key(domain_name, create_if_missing=False, base_dir=base_dir)
except SecureStorageError:
current_key = None
if current_key is not None:
try:
plaintext = AESGCM(current_key).decrypt(
_unb64(raw.get("nonce")),
_unb64(raw.get("ciphertext")),
aad,
)
except Exception:
plaintext = None
if plaintext is None:
try:
plaintext = AESGCM(_derive_legacy_domain_key(domain_name)).decrypt(
_unb64(raw.get("nonce")),
_unb64(raw.get("ciphertext")),
aad,
)
used_legacy_key = True
except Exception as exc:
try:
plaintext = AESGCM(_load_master_key()).decrypt(
_unb64(raw.get("nonce")),
_unb64(raw.get("ciphertext")),
_envelope_aad(file_path),
)
used_master_key = True
except Exception:
raise SecureStorageError(
f"Failed to decrypt domain JSON {file_path.name}: {exc}"
) from exc
try:
decoded = json.loads(plaintext.decode("utf-8"))
except Exception as exc:
raise SecureStorageError(
f"Failed to decode domain JSON payload {file_path.name}: {exc}"
) from exc
if used_legacy_key or used_master_key:
write_domain_json(domain_name, file_path.name, decoded, base_dir=base_dir)
return decoded
migrated = raw if isinstance(raw, (dict, list)) else default_factory()
write_domain_json(domain_name, file_path.name, migrated, base_dir=base_dir)
return migrated
backend/services/mesh/mesh_wormhole_contacts.py
+225
View File
@@ -0,0 +1,225 @@
"""Wormhole-owned DM contact and alias graph state."""
from __future__ import annotations
import time
from pathlib import Path
from typing import Any
from services.mesh.mesh_secure_storage import read_secure_json, write_secure_json
DATA_DIR = Path(__file__).resolve().parents[2] / "data"
CONTACTS_FILE = DATA_DIR / "wormhole_dm_contacts.json"
def _default_contact() -> dict[str, Any]:
return {
"alias": "",
"blocked": False,
"dhPubKey": "",
"dhAlgo": "",
"sharedAlias": "",
"previousSharedAliases": [],
"pendingSharedAlias": "",
"sharedAliasGraceUntil": 0,
"sharedAliasRotatedAt": 0,
"verify_inband": False,
"verify_registry": False,
"verified": False,
"verify_mismatch": False,
"verified_at": 0,
"remotePrekeyFingerprint": "",
"remotePrekeyObservedFingerprint": "",
"remotePrekeyPinnedAt": 0,
"remotePrekeyLastSeenAt": 0,
"remotePrekeySequence": 0,
"remotePrekeySignedAt": 0,
"remotePrekeyMismatch": False,
"witness_count": 0,
"witness_checked_at": 0,
"vouch_count": 0,
"vouch_checked_at": 0,
"updated_at": 0,
}
def _normalize_contact(value: dict[str, Any] | None) -> dict[str, Any]:
current = _default_contact()
current.update(value or {})
current["alias"] = str(current.get("alias", "") or "")
current["blocked"] = bool(current.get("blocked"))
current["dhPubKey"] = str(current.get("dhPubKey", "") or "")
current["dhAlgo"] = str(current.get("dhAlgo", "") or "")
current["sharedAlias"] = str(current.get("sharedAlias", "") or "")
current["previousSharedAliases"] = [
str(item or "") for item in list(current.get("previousSharedAliases") or []) if str(item or "").strip()
][-8:]
current["pendingSharedAlias"] = str(current.get("pendingSharedAlias", "") or "")
current["remotePrekeyFingerprint"] = str(current.get("remotePrekeyFingerprint", "") or "")
current["remotePrekeyObservedFingerprint"] = str(current.get("remotePrekeyObservedFingerprint", "") or "")
for key in (
"sharedAliasGraceUntil",
"sharedAliasRotatedAt",
"verified_at",
"remotePrekeyPinnedAt",
"remotePrekeyLastSeenAt",
"remotePrekeySequence",
"remotePrekeySignedAt",
"witness_count",
"witness_checked_at",
"vouch_count",
"vouch_checked_at",
"updated_at",
):
current[key] = int(current.get(key, 0) or 0)
for key in (
"verify_inband",
"verify_registry",
"verified",
"verify_mismatch",
"remotePrekeyMismatch",
):
current[key] = bool(current.get(key))
return current
def _merge_alias_history(*aliases: str, limit: int = 8) -> list[str]:
unique: set[str] = set()
ordered: list[str] = []
for alias in aliases:
value = str(alias or "").strip()
if not value or value in unique:
continue
unique.add(value)
ordered.append(value)
if len(ordered) >= limit:
break
return ordered
def _promote_pending_alias_if_due(contact: dict[str, Any]) -> tuple[dict[str, Any], bool]:
current = _normalize_contact(contact)
pending = str(current.get("pendingSharedAlias", "") or "").strip()
grace_until = int(current.get("sharedAliasGraceUntil", 0) or 0)
if not pending or grace_until <= 0 or grace_until > int(time.time() * 1000):
return current, False
active = str(current.get("sharedAlias", "") or "").strip()
promoted = dict(current)
promoted["sharedAlias"] = pending or active
promoted["pendingSharedAlias"] = ""
promoted["sharedAliasGraceUntil"] = 0
promoted["sharedAliasRotatedAt"] = int(time.time() * 1000)
promoted["previousSharedAliases"] = _merge_alias_history(
active,
*list(current.get("previousSharedAliases") or []),
)
return _normalize_contact(promoted), True
def _read_contacts() -> dict[str, dict[str, Any]]:
try:
raw = read_secure_json(CONTACTS_FILE, lambda: {})
except Exception:
import logging
logging.getLogger(__name__).warning(
"Contacts file could not be decrypted — starting with empty contacts"
)
CONTACTS_FILE.unlink(missing_ok=True)
return {}
if not isinstance(raw, dict):
return {}
contacts: dict[str, dict[str, Any]] = {}
changed = False
for peer_id, value in raw.items():
key = str(peer_id or "").strip()
if not key:
continue
normalized, promoted = _promote_pending_alias_if_due(value if isinstance(value, dict) else {})
contacts[key] = normalized
changed = changed or promoted
if changed:
_write_contacts(contacts)
return contacts
def _write_contacts(contacts: dict[str, dict[str, Any]]) -> None:
DATA_DIR.mkdir(parents=True, exist_ok=True)
payload = {
str(peer_id): _normalize_contact(contact)
for peer_id, contact in contacts.items()
if str(peer_id or "").strip()
}
write_secure_json(CONTACTS_FILE, payload)
def list_wormhole_dm_contacts() -> dict[str, dict[str, Any]]:
return _read_contacts()
def upsert_wormhole_dm_contact(peer_id: str, updates: dict[str, Any]) -> dict[str, Any]:
peer_id = str(peer_id or "").strip()
if not peer_id:
raise ValueError("peer_id required")
contacts = _read_contacts()
merged = _normalize_contact({**contacts.get(peer_id, _default_contact()), **dict(updates or {})})
merged["updated_at"] = int(time.time())
contacts[peer_id] = merged
_write_contacts(contacts)
return merged
def delete_wormhole_dm_contact(peer_id: str) -> bool:
peer_id = str(peer_id or "").strip()
if not peer_id:
return False
contacts = _read_contacts()
if peer_id not in contacts:
return False
del contacts[peer_id]
_write_contacts(contacts)
return True
def observe_remote_prekey_identity(
peer_id: str,
*,
fingerprint: str,
sequence: int = 0,
signed_at: int = 0,
) -> dict[str, Any]:
peer_key = str(peer_id or "").strip()
candidate = str(fingerprint or "").strip().lower()
if not peer_key:
raise ValueError("peer_id required")
if not candidate:
raise ValueError("fingerprint required")
contacts = _read_contacts()
current = _normalize_contact(contacts.get(peer_key))
now = int(time.time())
pinned = str(current.get("remotePrekeyFingerprint", "") or "").strip().lower()
current["remotePrekeyObservedFingerprint"] = candidate
current["remotePrekeyLastSeenAt"] = now
current["remotePrekeySequence"] = int(sequence or 0)
current["remotePrekeySignedAt"] = int(signed_at or 0)
trust_changed = False
if not pinned:
current["remotePrekeyFingerprint"] = candidate
current["remotePrekeyPinnedAt"] = now
current["remotePrekeyMismatch"] = False
pinned = candidate
else:
trust_changed = pinned != candidate
current["remotePrekeyMismatch"] = trust_changed
current["updated_at"] = int(time.time())
contacts[peer_key] = _normalize_contact(current)
_write_contacts(contacts)
return {
"ok": True,
"peer_id": peer_key,
"trust_changed": trust_changed,
"contact": contacts[peer_key],
}
backend/services/mesh/mesh_wormhole_dead_drop.py
+416
View File
@@ -0,0 +1,416 @@
"""Wormhole-owned dead-drop token derivation helpers.
These helpers move mailbox token derivation off the browser when Wormhole is the
secure trust anchor. The browser supplies only peer identifiers and peer DH
public keys; Wormhole derives the shared secret locally and returns mailbox
tokens for the current and previous epochs.
"""
from __future__ import annotations
import base64
import hashlib
import hmac
import json
import secrets
import time
from typing import Any
from cryptography.hazmat.primitives.asymmetric import x25519
from services.mesh.mesh_wormhole_identity import bootstrap_wormhole_identity, read_wormhole_identity
from services.mesh.mesh_wormhole_contacts import list_wormhole_dm_contacts, upsert_wormhole_dm_contact
from services.wormhole_settings import read_wormhole_settings
DEFAULT_DM_EPOCH_SECONDS = 6 * 60 * 60
HIGH_PRIVACY_DM_EPOCH_SECONDS = 2 * 60 * 60
SAS_PREFIXES = [
"amber",
"apex",
"atlas",
"birch",
"cinder",
"cobalt",
"delta",
"ember",
"falcon",
"frost",
"glint",
"harbor",
"juno",
"kepler",
"lumen",
"nova",
]
SAS_SUFFIXES = [
"anchor",
"arrow",
"bloom",
"cabin",
"cedar",
"cipher",
"comet",
"field",
"grove",
"harvest",
"meadow",
"mesa",
"orbit",
"signal",
"summit",
"thunder",
]
SAS_WORDS = [f"{prefix}-{suffix}" for prefix in SAS_PREFIXES for suffix in SAS_SUFFIXES]
DM_CONSENT_PREFIX = "DM_CONSENT:"
PAIRWISE_ALIAS_PREFIX = "dmx_"
def _unb64(data: str | bytes | None) -> bytes:
if not data:
return b""
if isinstance(data, bytes):
return base64.b64decode(data)
return base64.b64decode(data.encode("ascii"))
def build_contact_offer(*, dh_pub_key: str, dh_algo: str, geo_hint: str = "") -> str:
return (
f"{DM_CONSENT_PREFIX}"
+ json.dumps(
{
"kind": "contact_offer",
"dh_pub_key": str(dh_pub_key or ""),
"dh_algo": str(dh_algo or ""),
"geo_hint": str(geo_hint or ""),
},
separators=(",", ":"),
)
)
def build_contact_accept(*, shared_alias: str) -> str:
return (
f"{DM_CONSENT_PREFIX}"
+ json.dumps(
{
"kind": "contact_accept",
"shared_alias": str(shared_alias or ""),
},
separators=(",", ":"),
)
)
def build_contact_deny(*, reason: str = "") -> str:
return (
f"{DM_CONSENT_PREFIX}"
+ json.dumps(
{
"kind": "contact_deny",
"reason": str(reason or ""),
},
separators=(",", ":"),
)
)
def parse_contact_consent(message: str) -> dict[str, Any] | None:
text = str(message or "").strip()
if not text.startswith(DM_CONSENT_PREFIX):
return None
try:
payload = json.loads(text[len(DM_CONSENT_PREFIX) :])
except Exception:
return None
kind = str(payload.get("kind", "") or "").strip().lower()
if kind == "contact_offer":
dh_pub_key = str(payload.get("dh_pub_key", "") or "").strip()
if not dh_pub_key:
return None
return {
"kind": kind,
"dh_pub_key": dh_pub_key,
"dh_algo": str(payload.get("dh_algo", "") or "").strip() or "X25519",
"geo_hint": str(payload.get("geo_hint", "") or "").strip(),
}
if kind == "contact_accept":
shared_alias = str(payload.get("shared_alias", "") or "").strip()
if not shared_alias:
return None
return {"kind": kind, "shared_alias": shared_alias}
if kind == "contact_deny":
return {
"kind": kind,
"reason": str(payload.get("reason", "") or "").strip(),
}
return None
def _new_pairwise_alias() -> str:
return f"{PAIRWISE_ALIAS_PREFIX}{secrets.token_hex(12)}"
def _merge_alias_history(*aliases: str, limit: int = 8) -> list[str]:
unique: set[str] = set()
ordered: list[str] = []
for alias in aliases:
value = str(alias or "").strip()
if not value or value in unique:
continue
unique.add(value)
ordered.append(value)
if len(ordered) >= limit:
break
return ordered
def issue_pairwise_dm_alias(*, peer_id: str, peer_dh_pub: str = "") -> dict[str, Any]:
peer_id = str(peer_id or "").strip()
peer_dh_pub = str(peer_dh_pub or "").strip()
if not peer_id:
return {"ok": False, "detail": "peer_id required"}
from services.mesh.mesh_wormhole_persona import (
bootstrap_wormhole_persona_state,
get_dm_identity,
)
bootstrap_wormhole_persona_state()
dm_identity = get_dm_identity()
current = dict(list_wormhole_dm_contacts().get(peer_id) or {})
previous_alias = str(current.get("sharedAlias", "") or "").strip()
shared_alias = _new_pairwise_alias()
while shared_alias == previous_alias:
shared_alias = _new_pairwise_alias()
rotated_at_ms = int(time.time() * 1000)
contact_updates: dict[str, Any] = {
"sharedAlias": shared_alias,
"pendingSharedAlias": "",
"sharedAliasGraceUntil": 0,
"sharedAliasRotatedAt": rotated_at_ms,
"previousSharedAliases": _merge_alias_history(
previous_alias,
*list(current.get("previousSharedAliases") or []),
),
}
if peer_dh_pub:
contact_updates["dhPubKey"] = peer_dh_pub
elif str(current.get("dhPubKey", "") or "").strip():
contact_updates["dhPubKey"] = str(current.get("dhPubKey", "") or "").strip()
if str(current.get("dhAlgo", "") or "").strip():
contact_updates["dhAlgo"] = str(current.get("dhAlgo", "") or "").strip()
contact = upsert_wormhole_dm_contact(peer_id, contact_updates)
return {
"ok": True,
"peer_id": peer_id,
"shared_alias": shared_alias,
"replaced_alias": previous_alias,
"identity_scope": "dm_alias",
"dm_identity_id": str(dm_identity.get("node_id", "") or ""),
"contact": contact,
}
def rotate_pairwise_dm_alias(
*,
peer_id: str,
peer_dh_pub: str = "",
grace_ms: int = 45_000,
) -> dict[str, Any]:
peer_id = str(peer_id or "").strip()
peer_dh_pub = str(peer_dh_pub or "").strip()
if not peer_id:
return {"ok": False, "detail": "peer_id required"}
from services.mesh.mesh_wormhole_persona import (
bootstrap_wormhole_persona_state,
get_dm_identity,
)
bootstrap_wormhole_persona_state()
dm_identity = get_dm_identity()
current = dict(list_wormhole_dm_contacts().get(peer_id) or {})
active_alias = str(current.get("sharedAlias", "") or "").strip()
if not active_alias:
return issue_pairwise_dm_alias(peer_id=peer_id, peer_dh_pub=peer_dh_pub)
now_ms = int(time.time() * 1000)
pending_alias = str(current.get("pendingSharedAlias", "") or "").strip()
grace_until = int(current.get("sharedAliasGraceUntil", 0) or 0)
if pending_alias and grace_until > now_ms:
return {
"ok": True,
"peer_id": peer_id,
"active_alias": active_alias,
"pending_alias": pending_alias,
"grace_until": grace_until,
"identity_scope": "dm_alias",
"dm_identity_id": str(dm_identity.get("node_id", "") or ""),
"contact": current,
"rotated": False,
}
next_alias = _new_pairwise_alias()
reserved = {
active_alias,
pending_alias,
*[str(item or "").strip() for item in list(current.get("previousSharedAliases") or [])],
}
while next_alias in reserved:
next_alias = _new_pairwise_alias()
clamped_grace_ms = max(5_000, min(int(grace_ms or 45_000), 5 * 60 * 1000))
next_grace_until = now_ms + clamped_grace_ms
contact_updates: dict[str, Any] = {
"pendingSharedAlias": next_alias,
"sharedAliasGraceUntil": next_grace_until,
"sharedAliasRotatedAt": now_ms,
"previousSharedAliases": _merge_alias_history(
active_alias,
pending_alias,
*list(current.get("previousSharedAliases") or []),
),
}
if peer_dh_pub:
contact_updates["dhPubKey"] = peer_dh_pub
elif str(current.get("dhPubKey", "") or "").strip():
contact_updates["dhPubKey"] = str(current.get("dhPubKey", "") or "").strip()
if str(current.get("dhAlgo", "") or "").strip():
contact_updates["dhAlgo"] = str(current.get("dhAlgo", "") or "").strip()
contact = upsert_wormhole_dm_contact(peer_id, contact_updates)
return {
"ok": True,
"peer_id": peer_id,
"active_alias": active_alias,
"pending_alias": next_alias,
"grace_until": next_grace_until,
"identity_scope": "dm_alias",
"dm_identity_id": str(dm_identity.get("node_id", "") or ""),
"contact": contact,
"rotated": True,
}
def mailbox_epoch_seconds() -> int:
try:
settings = read_wormhole_settings()
if str(settings.get("privacy_profile", "default") or "default").lower() == "high":
return HIGH_PRIVACY_DM_EPOCH_SECONDS
except Exception:
pass
return DEFAULT_DM_EPOCH_SECONDS
def current_mailbox_epoch(ts_seconds: int | None = None) -> int:
now = int(ts_seconds) if ts_seconds is not None else int(time.time())
return now // mailbox_epoch_seconds()
def _derive_shared_secret(my_private_b64: str, peer_public_b64: str) -> bytes:
priv = x25519.X25519PrivateKey.from_private_bytes(_unb64(my_private_b64))
pub = x25519.X25519PublicKey.from_public_bytes(_unb64(peer_public_b64))
return priv.exchange(pub)
def _token_for(secret: bytes, peer_id: str, my_node_id: str, epoch: int) -> str:
ids = "|".join(sorted([str(my_node_id or ""), str(peer_id or "")]))
message = f"sb_dd|v1|{int(epoch)}|{ids}".encode("utf-8")
return hmac.new(secret, message, hashlib.sha256).hexdigest()
def _sas_words_from_digest(digest: bytes, count: int) -> list[str]:
out: list[str] = []
acc = 0
acc_bits = 0
for byte in digest:
acc = (acc << 8) | byte
acc_bits += 8
while acc_bits >= 8 and len(out) < count:
idx = (acc >> (acc_bits - 8)) & 0xFF
out.append(SAS_WORDS[idx])
acc_bits -= 8
if len(out) >= count:
break
return out
def derive_dead_drop_token_pair(*, peer_id: str, peer_dh_pub: str) -> dict[str, Any]:
peer_id = str(peer_id or "").strip()
peer_dh_pub = str(peer_dh_pub or "").strip()
if not peer_id or not peer_dh_pub:
return {"ok": False, "detail": "peer_id and peer_dh_pub required"}
identity = read_wormhole_identity()
if not identity.get("bootstrapped"):
bootstrap_wormhole_identity()
identity = read_wormhole_identity()
my_private = str(identity.get("dh_private_key", "") or "")
my_node_id = str(identity.get("node_id", "") or "")
if not my_private or not my_node_id:
return {"ok": False, "detail": "Wormhole DH identity unavailable"}
try:
secret = _derive_shared_secret(my_private, peer_dh_pub)
except Exception as exc:
return {"ok": False, "detail": str(exc) or "dead_drop_secret_failed"}
epoch = current_mailbox_epoch()
return {
"ok": True,
"peer_id": peer_id,
"epoch": epoch,
"current": _token_for(secret, peer_id, my_node_id, epoch),
"previous": _token_for(secret, peer_id, my_node_id, epoch - 1),
}
def derive_dead_drop_tokens_for_contacts(*, contacts: list[dict[str, Any]], limit: int = 24) -> dict[str, Any]:
results: list[dict[str, Any]] = []
for item in contacts[: max(1, min(int(limit or 24), 64))]:
peer_id = str((item or {}).get("peer_id", "") or "").strip()
peer_dh_pub = str((item or {}).get("peer_dh_pub", "") or "").strip()
if not peer_id or not peer_dh_pub:
continue
pair = derive_dead_drop_token_pair(peer_id=peer_id, peer_dh_pub=peer_dh_pub)
if pair.get("ok"):
results.append(
{
"peer_id": peer_id,
"current": str(pair.get("current", "") or ""),
"previous": str(pair.get("previous", "") or ""),
"epoch": int(pair.get("epoch", 0) or 0),
}
)
return {"ok": True, "tokens": results}
def derive_sas_phrase(*, peer_id: str, peer_dh_pub: str, words: int = 8) -> dict[str, Any]:
peer_id = str(peer_id or "").strip()
peer_dh_pub = str(peer_dh_pub or "").strip()
word_count = max(2, min(int(words or 8), 16))
if not peer_id or not peer_dh_pub:
return {"ok": False, "detail": "peer_id and peer_dh_pub required"}
identity = read_wormhole_identity()
if not identity.get("bootstrapped"):
bootstrap_wormhole_identity()
identity = read_wormhole_identity()
my_private = str(identity.get("dh_private_key", "") or "")
my_node_id = str(identity.get("node_id", "") or "")
if not my_private or not my_node_id:
return {"ok": False, "detail": "Wormhole DH identity unavailable"}
try:
secret = _derive_shared_secret(my_private, peer_dh_pub)
except Exception as exc:
return {"ok": False, "detail": str(exc) or "sas_secret_failed"}
ids = "|".join(sorted([my_node_id, peer_id]))
digest = hmac.new(secret, f"sb_sas|v1|{ids}".encode("utf-8"), hashlib.sha256).digest()
phrase = " ".join(_sas_words_from_digest(digest, word_count))
return {"ok": True, "peer_id": peer_id, "phrase": phrase, "words": word_count}
backend/services/mesh/mesh_wormhole_identity.py
+231
View File
@@ -0,0 +1,231 @@
"""Wormhole-managed DM identity wrappers.
This module preserves the legacy DM identity API while sourcing its state from
the Wormhole persona manager. Public transport identity stays separate, and DM
operations now use the dedicated DM alias compartment.
"""
from __future__ import annotations
import base64
import hmac
import hashlib
import time
from typing import Any
from services.mesh.mesh_protocol import PROTOCOL_VERSION
from services.mesh.mesh_wormhole_persona import (
bootstrap_wormhole_persona_state,
ensure_dm_mailbox_client_secret,
get_dm_identity,
read_dm_identity,
read_wormhole_persona_state,
sign_dm_wormhole_event,
sign_dm_wormhole_message,
write_dm_identity,
)
def _safe_int(val, default=0) -> int:
try:
return int(val)
except (TypeError, ValueError):
return default
def _default_identity() -> dict[str, Any]:
return {
"bootstrapped": False,
"bootstrapped_at": 0,
"updated_at": 0,
"scope": "dm_alias",
"label": "dm-alias",
"node_id": "",
"public_key": "",
"public_key_algo": "Ed25519",
"private_key": "",
"sequence": 0,
"dh_pub_key": "",
"dh_algo": "X25519",
"dh_private_key": "",
"last_dh_timestamp": 0,
"bundle_fingerprint": "",
"bundle_sequence": 0,
"bundle_registered_at": 0,
"signed_prekey_id": 0,
"signed_prekey_pub": "",
"signed_prekey_priv": "",
"signed_prekey_signature": "",
"signed_prekey_generated_at": 0,
"signed_prekey_history": [],
"one_time_prekeys": [],
"prekey_bundle_registered_at": 0,
"prekey_republish_threshold": 0,
"prekey_republish_target": 0,
"prekey_next_republish_after": 0,
}
def _public_view(data: dict[str, Any]) -> dict[str, Any]:
return {
"bootstrapped": bool(data.get("bootstrapped")),
"bootstrapped_at": _safe_int(data.get("bootstrapped_at", 0) or 0),
"scope": str(data.get("scope", "dm_alias") or "dm_alias"),
"label": str(data.get("label", "dm-alias") or "dm-alias"),
"node_id": str(data.get("node_id", "") or ""),
"public_key": str(data.get("public_key", "") or ""),
"public_key_algo": str(data.get("public_key_algo", "Ed25519") or "Ed25519"),
"sequence": _safe_int(data.get("sequence", 0) or 0),
"dh_pub_key": str(data.get("dh_pub_key", "") or ""),
"dh_algo": str(data.get("dh_algo", "X25519") or "X25519"),
"last_dh_timestamp": _safe_int(data.get("last_dh_timestamp", 0) or 0),
"bundle_fingerprint": str(data.get("bundle_fingerprint", "") or ""),
"bundle_sequence": _safe_int(data.get("bundle_sequence", 0) or 0),
"bundle_registered_at": _safe_int(data.get("bundle_registered_at", 0) or 0),
"protocol_version": PROTOCOL_VERSION,
}
def read_wormhole_identity() -> dict[str, Any]:
bootstrap_wormhole_persona_state()
persona_state = read_wormhole_persona_state()
data = {**_default_identity(), **read_dm_identity()}
data["bootstrapped"] = True
data["bootstrapped_at"] = _safe_int(persona_state.get("bootstrapped_at", 0) or 0)
return data
def _write_identity(data: dict[str, Any]) -> dict[str, Any]:
current = read_wormhole_identity()
merged = {**current, **dict(data or {})}
merged["scope"] = "dm_alias"
merged["label"] = str(merged.get("label", "dm-alias") or "dm-alias")
merged["updated_at"] = int(time.time())
saved = write_dm_identity(merged)
saved["bootstrapped"] = True
return {**_default_identity(), **saved}
def bootstrap_wormhole_identity(force: bool = False) -> dict[str, Any]:
bootstrap_wormhole_persona_state(force=force)
data = read_wormhole_identity()
if force:
data["bundle_fingerprint"] = ""
data["bundle_sequence"] = 0
data["bundle_registered_at"] = 0
data["signed_prekey_id"] = 0
data["signed_prekey_pub"] = ""
data["signed_prekey_priv"] = ""
data["signed_prekey_signature"] = ""
data["signed_prekey_generated_at"] = 0
data["signed_prekey_history"] = []
data["one_time_prekeys"] = []
data["prekey_bundle_registered_at"] = 0
data["prekey_republish_threshold"] = 0
data["prekey_republish_target"] = 0
data["prekey_next_republish_after"] = 0
data = _write_identity(data)
return _public_view(data)
def get_wormhole_identity() -> dict[str, Any]:
return get_dm_identity()
def sign_wormhole_event(
*,
event_type: str,
payload: dict[str, Any],
sequence: int | None = None,
) -> dict[str, Any]:
return sign_dm_wormhole_event(event_type=event_type, payload=payload, sequence=sequence)
def sign_wormhole_message(message: str) -> dict[str, Any]:
return sign_dm_wormhole_message(message)
def _bundle_fingerprint(data: dict[str, Any]) -> str:
raw = "|".join(
[
str(data.get("dh_pub_key", "")),
str(data.get("dh_algo", "X25519")),
str(data.get("public_key", "")),
str(data.get("public_key_algo", "Ed25519")),
PROTOCOL_VERSION,
]
)
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
def register_wormhole_dm_key(force: bool = False) -> dict[str, Any]:
data = read_wormhole_identity()
timestamp = int(time.time())
fingerprint = _bundle_fingerprint(data)
if not force and fingerprint and fingerprint == data.get("bundle_fingerprint"):
return {
"ok": True,
**_public_view(data),
}
payload = {
"dh_pub_key": str(data.get("dh_pub_key", "")),
"dh_algo": str(data.get("dh_algo", "X25519")),
"timestamp": timestamp,
}
signed = sign_wormhole_event(event_type="dm_key", payload=payload)
from services.mesh.mesh_dm_relay import dm_relay
accepted, detail, metadata = dm_relay.register_dh_key(
signed["node_id"],
payload["dh_pub_key"],
payload["dh_algo"],
payload["timestamp"],
signed["signature"],
signed["public_key"],
signed["public_key_algo"],
signed["protocol_version"],
signed["sequence"],
)
if not accepted:
return {"ok": False, "detail": detail}
data = read_wormhole_identity()
data["bundle_fingerprint"] = metadata.get("bundle_fingerprint", fingerprint) if metadata else fingerprint
data["bundle_sequence"] = _safe_int(
metadata.get("accepted_sequence", signed["sequence"]) if metadata else signed["sequence"],
_safe_int(signed.get("sequence", 0), 0),
)
data["bundle_registered_at"] = timestamp
data["last_dh_timestamp"] = timestamp
saved = _write_identity(data)
return {
"ok": True,
**_public_view(saved),
**(metadata or {}),
}
def get_dm_mailbox_client_secret(*, generate: bool = True) -> str:
return ensure_dm_mailbox_client_secret(generate=generate)
def derive_dm_mailbox_token(
dm_alias_id: str | None = None,
*,
generate_secret: bool = True,
) -> str:
data = read_wormhole_identity()
alias_id = str(dm_alias_id or data.get("node_id", "") or "").strip()
if not alias_id:
return ""
secret_b64 = get_dm_mailbox_client_secret(generate=generate_secret)
if not secret_b64:
return ""
try:
secret = base64.b64decode(secret_b64.encode("ascii"))
except Exception:
return ""
return hmac.new(secret, alias_id.encode("utf-8"), hashlib.sha256).hexdigest()
backend/services/mesh/mesh_wormhole_persona.py
+1038
View File
File diff suppressed because it is too large Load Diff
backend/services/mesh/mesh_wormhole_prekey.py
+538
View File
@@ -0,0 +1,538 @@
"""Wormhole-managed prekey bundles and X3DH-style bootstrap helpers."""
from __future__ import annotations
import base64
import hashlib
import json
import os
import random
import time
from typing import Any
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ed25519, x25519
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives.serialization import Encoding, NoEncryption, PrivateFormat, PublicFormat
from services.mesh.mesh_crypto import derive_node_id
from services.mesh.mesh_wormhole_identity import (
_write_identity,
bootstrap_wormhole_identity,
read_wormhole_identity,
sign_wormhole_event,
sign_wormhole_message,
)
PREKEY_TARGET = 8
PREKEY_MIN_THRESHOLD = 3
PREKEY_MAX_THRESHOLD = 5
PREKEY_MIN_TARGET = 7
PREKEY_MAX_TARGET = 9
PREKEY_MIN_REPUBLISH_DELAY_S = 45
PREKEY_MAX_REPUBLISH_DELAY_S = 120
PREKEY_REPUBLISH_THRESHOLD_RANGE = (PREKEY_MIN_THRESHOLD, PREKEY_MAX_THRESHOLD)
PREKEY_REPUBLISH_TARGET_RANGE = (PREKEY_MIN_TARGET, PREKEY_MAX_TARGET)
PREKEY_REPUBLISH_DELAY_RANGE_S = (PREKEY_MIN_REPUBLISH_DELAY_S, PREKEY_MAX_REPUBLISH_DELAY_S)
SIGNED_PREKEY_ROTATE_AFTER_S = 24 * 60 * 60
SIGNED_PREKEY_GRACE_S = 3 * 24 * 60 * 60
def _safe_int(val, default=0) -> int:
try:
return int(val)
except (TypeError, ValueError):
return default
def _b64(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def _unb64(data: str | bytes | None) -> bytes:
if not data:
return b""
if isinstance(data, bytes):
return base64.b64decode(data)
return base64.b64decode(data.encode("ascii"))
def _stable_json(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"))
def _x25519_pair() -> dict[str, str]:
priv = x25519.X25519PrivateKey.generate()
priv_raw = priv.private_bytes(Encoding.Raw, PrivateFormat.Raw, NoEncryption())
pub_raw = priv.public_key().public_bytes(Encoding.Raw, PublicFormat.Raw)
return {"public_key": _b64(pub_raw), "private_key": _b64(priv_raw)}
def _derive(priv_b64: str, pub_b64: str) -> bytes:
priv = x25519.X25519PrivateKey.from_private_bytes(_unb64(priv_b64))
pub = x25519.X25519PublicKey.from_public_bytes(_unb64(pub_b64))
return priv.exchange(pub)
def _hkdf(ikm: bytes, info: str, length: int = 32) -> bytes:
return HKDF(
algorithm=hashes.SHA256(),
length=length,
salt=b"\xff" * 32,
info=info.encode("utf-8"),
).derive(ikm)
def _bundle_payload(data: dict[str, Any]) -> dict[str, Any]:
one_time_prekeys = [
{
"prekey_id": _safe_int(item.get("prekey_id", 0) or 0),
"public_key": str(item.get("public_key", "") or ""),
}
for item in list(data.get("one_time_prekeys") or [])
if item.get("public_key")
]
return {
"identity_dh_pub_key": str(data.get("dh_pub_key", "") or ""),
"dh_algo": str(data.get("dh_algo", "X25519") or "X25519"),
"signed_prekey_id": _safe_int(data.get("signed_prekey_id", 0) or 0),
"signed_prekey_pub": str(data.get("signed_prekey_pub", "") or ""),
"signed_prekey_signature": str(data.get("signed_prekey_signature", "") or ""),
"signed_prekey_timestamp": _safe_int(data.get("signed_prekey_generated_at", 0) or 0),
"signed_at": _safe_int(data.get("prekey_bundle_signed_at", 0) or 0),
"bundle_signature": str(data.get("prekey_bundle_signature", "") or ""),
"mls_key_package": str(data.get("mls_key_package", "") or ""),
"one_time_prekeys": one_time_prekeys,
"one_time_prekey_count": len(one_time_prekeys),
}
def _bundle_signature_payload(data: dict[str, Any]) -> str:
# OTK binding: One-time key hashes are included in the bundle signature
# as of Sprint 12 (S12-3). Relay substitution of OTKs will now break
# the bundle signature and be rejected by verify_prekey_bundle().
otk_hashes = sorted(
hashlib.sha256(str(item.get("public_key", "")).encode("utf-8")).hexdigest()
for item in (data.get("one_time_prekeys") or [])
)
return _stable_json(
{
"identity_dh_pub_key": str(data.get("identity_dh_pub_key", "") or ""),
"dh_algo": str(data.get("dh_algo", "X25519") or "X25519"),
"signed_prekey_id": _safe_int(data.get("signed_prekey_id", 0) or 0),
"signed_prekey_pub": str(data.get("signed_prekey_pub", "") or ""),
"signed_prekey_signature": str(data.get("signed_prekey_signature", "") or ""),
"signed_at": _safe_int(data.get("signed_at", 0) or 0),
"mls_key_package": str(data.get("mls_key_package", "") or ""),
"one_time_prekey_hashes": otk_hashes,
}
)
def _max_prekey_bundle_age_s() -> int:
return SIGNED_PREKEY_ROTATE_AFTER_S + SIGNED_PREKEY_GRACE_S
def trust_fingerprint_for_bundle_record(record: dict[str, Any]) -> str:
bundle = dict(record.get("bundle") or record or {})
material = {
"agent_id": str(record.get("agent_id", "") or ""),
"identity_dh_pub_key": str(bundle.get("identity_dh_pub_key", "") or ""),
"dh_algo": str(bundle.get("dh_algo", "X25519") or "X25519"),
"public_key": str(record.get("public_key", "") or ""),
"public_key_algo": str(record.get("public_key_algo", "") or ""),
"protocol_version": str(record.get("protocol_version", "") or ""),
}
return hashlib.sha256(_stable_json(material).encode("utf-8")).hexdigest()
def _attach_bundle_signature(bundle: dict[str, Any], *, signed_at: int | None = None) -> dict[str, Any]:
# KNOWN LIMITATION: Bundle signature is self-signed by the identity key it contains.
# This proves possession of the private key and detects post-registration tampering,
# but cannot prevent initial impersonation (no external PKI). Mitigated by reputation
# system in Phase 9 (Oracle Rep). See threat-model.md for full analysis.
payload = dict(bundle or {})
payload["signed_at"] = int(signed_at if signed_at is not None else time.time())
signed = sign_wormhole_message(_bundle_signature_payload(payload))
payload["bundle_signature"] = str(signed.get("signature", "") or "")
return payload
def _verify_bundle_signature(bundle: dict[str, Any], public_key: str) -> tuple[bool, str]:
try:
signing_pub = ed25519.Ed25519PublicKey.from_public_bytes(_unb64(public_key))
signing_pub.verify(
bytes.fromhex(str(bundle.get("bundle_signature", "") or "")),
_bundle_signature_payload(bundle).encode("utf-8"),
)
except Exception:
return False, "Prekey bundle signature invalid"
return True, "ok"
def _validate_bundle_record(record: dict[str, Any]) -> tuple[bool, str]:
bundle = dict(record.get("bundle") or {})
now = time.time()
signed_at = _safe_int(bundle.get("signed_at", 0) or 0)
if signed_at <= 0:
return False, "Prekey bundle missing signed_at"
if signed_at > now + 299:
return False, "Prekey bundle signed_at is in the future"
if not str(bundle.get("bundle_signature", "") or "").strip():
return False, "Prekey bundle missing bundle_signature"
public_key = str(record.get("public_key", "") or "")
if not public_key:
return False, "Prekey bundle missing signing key"
ok, reason = _verify_bundle_signature(bundle, public_key)
if not ok:
return False, reason
if (now - signed_at) > _max_prekey_bundle_age_s():
return False, "Prekey bundle is stale"
if str(record.get("agent_id", "") or "").strip():
derived = derive_node_id(public_key)
if derived != str(record.get("agent_id", "") or "").strip():
return False, "Prekey bundle public key binding mismatch"
return True, "ok"
def _jittered_republish_policy(data: dict[str, Any], *, reset: bool = False) -> tuple[int, int]:
threshold = _safe_int(data.get("prekey_republish_threshold", 0) or 0)
target = _safe_int(data.get("prekey_republish_target", 0) or 0)
min_threshold, max_threshold = PREKEY_REPUBLISH_THRESHOLD_RANGE
min_target, max_target = PREKEY_REPUBLISH_TARGET_RANGE
if reset or threshold < min_threshold or threshold > max_threshold:
threshold = random.randint(min_threshold, max_threshold)
data["prekey_republish_threshold"] = threshold
if reset or target < min_target or target > max_target:
target = random.randint(min_target, max_target)
data["prekey_republish_target"] = target
return threshold, target
def _schedule_next_republish_window(data: dict[str, Any]) -> None:
min_delay_s, max_delay_s = PREKEY_REPUBLISH_DELAY_RANGE_S
data["prekey_next_republish_after"] = int(
time.time() + random.randint(min_delay_s, max_delay_s)
)
def _archive_current_signed_prekey(data: dict[str, Any], retired_at: int) -> None:
current_id = _safe_int(data.get("signed_prekey_id", 0) or 0)
current_pub = str(data.get("signed_prekey_pub", "") or "")
current_priv = str(data.get("signed_prekey_priv", "") or "")
current_sig = str(data.get("signed_prekey_signature", "") or "")
current_generated_at = _safe_int(data.get("signed_prekey_generated_at", 0) or 0)
if not current_id or not current_pub or not current_priv:
return
history = list(data.get("signed_prekey_history") or [])
history.append(
{
"signed_prekey_id": current_id,
"signed_prekey_pub": current_pub,
"signed_prekey_priv": current_priv,
"signed_prekey_signature": current_sig,
"signed_prekey_generated_at": current_generated_at,
"retired_at": retired_at,
}
)
cutoff = retired_at - SIGNED_PREKEY_GRACE_S
data["signed_prekey_history"] = [
item
for item in history[-4:]
if _safe_int(item.get("retired_at", retired_at) or retired_at) >= cutoff
]
def _find_signed_prekey_private(data: dict[str, Any], spk_id: int) -> str:
if _safe_int(data.get("signed_prekey_id", 0) or 0) == spk_id:
return str(data.get("signed_prekey_priv", "") or "")
for item in list(data.get("signed_prekey_history") or []):
if _safe_int(item.get("signed_prekey_id", 0) or 0) == spk_id:
return str(item.get("signed_prekey_priv", "") or "")
return ""
def ensure_wormhole_prekeys(force_signed_prekey: bool = False, replenish_target: int = PREKEY_TARGET) -> dict[str, Any]:
data = read_wormhole_identity()
if not data.get("bootstrapped"):
bootstrap_wormhole_identity()
data = read_wormhole_identity()
changed = False
now = int(time.time())
_, jitter_target = _jittered_republish_policy(data)
replenish_target = max(1, _safe_int(replenish_target or jitter_target, 1))
spk_generated_at = _safe_int(data.get("signed_prekey_generated_at", 0) or 0)
spk_too_old = bool(spk_generated_at and (now - spk_generated_at) >= SIGNED_PREKEY_ROTATE_AFTER_S)
if force_signed_prekey or spk_too_old or not data.get("signed_prekey_pub") or not data.get("signed_prekey_priv"):
_archive_current_signed_prekey(data, now)
pair = _x25519_pair()
spk_id = _safe_int(data.get("signed_prekey_id", 0) or 0) + 1
signed_prekey_payload = {
"signed_prekey_id": spk_id,
"signed_prekey_pub": pair["public_key"],
"signed_prekey_timestamp": now,
}
signed = sign_wormhole_event(
event_type="dm_signed_prekey",
payload=signed_prekey_payload,
)
data["signed_prekey_id"] = spk_id
data["signed_prekey_pub"] = pair["public_key"]
data["signed_prekey_priv"] = pair["private_key"]
data["signed_prekey_signature"] = signed["signature"]
data["signed_prekey_generated_at"] = now
changed = True
existing_otks = list(data.get("one_time_prekeys") or [])
next_id = max([_safe_int(item.get("prekey_id", 0) or 0) for item in existing_otks] + [0])
while len(existing_otks) < max(1, replenish_target):
next_id += 1
pair = _x25519_pair()
existing_otks.append(
{
"prekey_id": next_id,
"public_key": pair["public_key"],
"private_key": pair["private_key"],
"created_at": now,
}
)
changed = True
data["one_time_prekeys"] = existing_otks
_jittered_republish_policy(data)
if changed:
_write_identity(data)
return _bundle_payload(data)
def register_wormhole_prekey_bundle(force_signed_prekey: bool = False) -> dict[str, Any]:
data = read_wormhole_identity()
if not data.get("bootstrapped"):
bootstrap_wormhole_identity()
data = read_wormhole_identity()
_, jitter_target = _jittered_republish_policy(data, reset=force_signed_prekey)
if force_signed_prekey:
_schedule_next_republish_window(data)
_write_identity(data)
data = read_wormhole_identity()
bundle = ensure_wormhole_prekeys(force_signed_prekey=force_signed_prekey, replenish_target=jitter_target)
from services.mesh.mesh_dm_mls import export_dm_key_package_for_alias
mls_key_package = export_dm_key_package_for_alias(str(data.get("node_id", "") or ""))
if not mls_key_package.get("ok"):
return {"ok": False, "detail": str(mls_key_package.get("detail", "") or "mls key package unavailable")}
bundle["mls_key_package"] = str(mls_key_package.get("mls_key_package", "") or "")
bundle = _attach_bundle_signature(bundle)
signed = sign_wormhole_event(
event_type="dm_prekey_bundle",
payload=bundle,
)
from services.mesh.mesh_dm_relay import dm_relay
accepted, detail, metadata = dm_relay.register_prekey_bundle(
signed["node_id"],
bundle,
signed["signature"],
signed["public_key"],
signed["public_key_algo"],
signed["protocol_version"],
signed["sequence"],
)
if not accepted:
return {"ok": False, "detail": detail}
refreshed = read_wormhole_identity()
refreshed["prekey_bundle_registered_at"] = int(time.time())
refreshed["prekey_bundle_signed_at"] = _safe_int(bundle.get("signed_at", 0) or 0)
refreshed["prekey_bundle_signature"] = str(bundle.get("bundle_signature", "") or "")
_schedule_next_republish_window(refreshed)
_jittered_republish_policy(refreshed, reset=True)
_write_identity(refreshed)
return {
"ok": True,
"agent_id": signed["node_id"],
"bundle": bundle,
"signature": signed["signature"],
"public_key": signed["public_key"],
"public_key_algo": signed["public_key_algo"],
"protocol_version": signed["protocol_version"],
"sequence": signed["sequence"],
**(metadata or {}),
}
def fetch_dm_prekey_bundle(agent_id: str) -> dict[str, Any]:
from services.mesh.mesh_dm_relay import dm_relay
stored = dm_relay.get_prekey_bundle(agent_id)
if not stored:
return {"ok": False, "detail": "Prekey bundle not found"}
validated_record = {**dict(stored), "agent_id": str(agent_id or "").strip()}
ok, reason = _validate_bundle_record(validated_record)
if not ok:
return {"ok": False, "detail": reason}
bundle = dict(stored.get("bundle") or {})
bundle["one_time_prekeys"] = []
bundle["one_time_prekey_count"] = _safe_int(bundle.get("one_time_prekey_count", 0) or 0)
return {
"ok": True,
"agent_id": agent_id,
**bundle,
"signature": str(stored.get("signature", "") or ""),
"public_key": str(stored.get("public_key", "") or ""),
"public_key_algo": str(stored.get("public_key_algo", "") or ""),
"protocol_version": str(stored.get("protocol_version", "") or ""),
"sequence": _safe_int(stored.get("sequence", 0) or 0),
"trust_fingerprint": trust_fingerprint_for_bundle_record(validated_record),
}
def _consume_local_one_time_prekey(prekey_id: int) -> int:
if prekey_id <= 0:
data = read_wormhole_identity()
return len(list(data.get("one_time_prekeys") or []))
data = read_wormhole_identity()
existing = list(data.get("one_time_prekeys") or [])
filtered = [
item for item in existing if _safe_int(item.get("prekey_id", 0) or 0) != _safe_int(prekey_id)
]
if len(filtered) == len(existing):
return len(existing)
data["one_time_prekeys"] = filtered
_write_identity(data)
return len(filtered)
def bootstrap_encrypt_for_peer(peer_id: str, plaintext: str) -> dict[str, Any]:
from services.mesh.mesh_dm_relay import dm_relay
stored = dm_relay.get_prekey_bundle(peer_id)
if not stored:
return {"ok": False, "detail": "Peer prekey bundle not found"}
validated_record = {**dict(stored), "agent_id": str(peer_id or "").strip()}
ok, reason = _validate_bundle_record(validated_record)
if not ok:
return {"ok": False, "detail": reason}
peer_bundle_stored = dm_relay.consume_one_time_prekey(peer_id)
if not peer_bundle_stored:
return {"ok": False, "detail": "Peer prekey bundle not found"}
peer_bundle = dict(peer_bundle_stored.get("bundle") or {})
peer_static = str(peer_bundle.get("identity_dh_pub_key", "") or "")
peer_spk = str(peer_bundle.get("signed_prekey_pub", "") or "")
peer_spk_id = _safe_int(peer_bundle.get("signed_prekey_id", 0) or 0)
peer_otk = dict(peer_bundle_stored.get("claimed_one_time_prekey") or {})
data = read_wormhole_identity()
if not data.get("bootstrapped"):
bootstrap_wormhole_identity()
data = read_wormhole_identity()
my_static_priv = str(data.get("dh_private_key", "") or "")
my_static_pub = str(data.get("dh_pub_key", "") or "")
if not my_static_priv or not my_static_pub or not peer_static or not peer_spk:
return {"ok": False, "detail": "Missing static or signed prekey material"}
eph = _x25519_pair()
dh_parts = [
_derive(my_static_priv, peer_spk),
_derive(eph["private_key"], peer_static),
_derive(eph["private_key"], peer_spk),
]
otk_id = 0
if peer_otk and peer_otk.get("public_key"):
dh_parts.append(_derive(eph["private_key"], str(peer_otk.get("public_key"))))
otk_id = _safe_int(peer_otk.get("prekey_id", 0) or 0)
secret = _hkdf(b"".join(dh_parts), "SB-X3DH", 32)
header = {
"v": 1,
"alg": "X25519",
"ik_pub": my_static_pub,
"ek_pub": eph["public_key"],
"spk_id": peer_spk_id,
"otk_id": otk_id,
}
aad = _stable_json(header).encode("utf-8")
iv = os.urandom(12)
ciphertext = AESGCM(secret).encrypt(iv, plaintext.encode("utf-8"), aad)
envelope = {
"h": header,
"ct": _b64(iv + ciphertext),
}
wrapped = _b64(_stable_json(envelope).encode("utf-8"))
return {"ok": True, "result": f"x3dh1:{wrapped}"}
def bootstrap_decrypt_from_sender(sender_id: str, ciphertext: str) -> dict[str, Any]:
if not ciphertext.startswith("x3dh1:"):
return {"ok": False, "detail": "legacy"}
try:
raw = ciphertext[len("x3dh1:") :]
envelope = json.loads(_unb64(raw).decode("utf-8"))
header = dict(envelope.get("h") or {})
combined = _unb64(str(envelope.get("ct") or ""))
my_data = read_wormhole_identity()
if not my_data.get("bootstrapped"):
bootstrap_wormhole_identity()
my_data = read_wormhole_identity()
sender_static_pub = str(header.get("ik_pub", "") or "")
sender_eph_pub = str(header.get("ek_pub", "") or "")
spk_id = _safe_int(header.get("spk_id", 0) or 0)
otk_id = _safe_int(header.get("otk_id", 0) or 0)
if not sender_static_pub or not sender_eph_pub:
return {"ok": False, "detail": "Missing sender bootstrap keys"}
from services.mesh.mesh_dm_relay import dm_relay
sender_dh = dm_relay.get_dh_key(sender_id)
if sender_dh and sender_dh.get("dh_pub_key") and str(sender_dh.get("dh_pub_key")) != sender_static_pub:
return {"ok": False, "detail": "Sender static DH key mismatch"}
signed_prekey_priv = _find_signed_prekey_private(my_data, spk_id)
my_static_priv = str(my_data.get("dh_private_key", "") or "")
if not signed_prekey_priv or not my_static_priv:
return {"ok": False, "detail": "Missing local bootstrap private keys"}
dh_parts = [
_derive(signed_prekey_priv, sender_static_pub),
_derive(my_static_priv, sender_eph_pub),
_derive(signed_prekey_priv, sender_eph_pub),
]
if otk_id:
otk_match = next(
(
item
for item in list(my_data.get("one_time_prekeys") or [])
if _safe_int(item.get("prekey_id", 0) or 0) == otk_id and item.get("private_key")
),
None,
)
if not otk_match:
return {"ok": False, "detail": "One-time prekey mismatch"}
dh_parts.append(_derive(str(otk_match.get("private_key", "")), sender_eph_pub))
secret = _hkdf(b"".join(dh_parts), "SB-X3DH", 32)
aad = _stable_json(header).encode("utf-8")
iv = combined[:12]
ct = combined[12:]
plaintext = AESGCM(secret).decrypt(iv, ct, aad).decode("utf-8")
if otk_id:
remaining_otks = _consume_local_one_time_prekey(otk_id)
my_data = read_wormhole_identity()
threshold, target = _jittered_republish_policy(my_data)
next_republish_after = _safe_int(my_data.get("prekey_next_republish_after", 0) or 0)
now_ts = int(time.time())
should_republish = remaining_otks <= 0
if not should_republish and remaining_otks <= threshold and now_ts >= next_republish_after:
should_republish = True
if should_republish:
register_wormhole_prekey_bundle()
else:
_write_identity(my_data)
return {"ok": True, "result": plaintext}
except Exception as exc:
return {"ok": False, "detail": str(exc) or "bootstrap_decrypt_failed"}
backend/services/mesh/mesh_wormhole_ratchet.py
+361
View File
@@ -0,0 +1,361 @@
"""Wormhole-backed DM ratchet state and crypto.
This is the first DM custody move out of the browser. When Wormhole is active,
the frontend no longer persists ratchet/session state in IndexedDB; instead the
agent owns the session records and performs ratchet encrypt/decrypt operations
locally on behalf of the UI.
"""
from __future__ import annotations
import base64
import json
import os
import threading
import time
from pathlib import Path
from typing import Any
from cryptography.hazmat.primitives import hashes, hmac
from cryptography.hazmat.primitives.asymmetric import x25519
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives.serialization import Encoding, PrivateFormat, PublicFormat, NoEncryption
from services.mesh.mesh_wormhole_identity import bootstrap_wormhole_identity, read_wormhole_identity
from services.mesh.mesh_secure_storage import read_secure_json, write_secure_json
DATA_DIR = Path(__file__).resolve().parents[2] / "data"
STATE_FILE = DATA_DIR / "wormhole_dm_ratchet.json"
STATE_LOCK = threading.RLock()
MAX_SKIP = 32
PAD_BUCKET = 1024
PAD_STEP = 512
PAD_MAX = 4096
PAD_MAGIC = "SBP1"
def _b64(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def _unb64(data: str | bytes | None) -> bytes:
if not data:
return b""
if isinstance(data, bytes):
return base64.b64decode(data)
return base64.b64decode(data.encode("ascii"))
def _zero_bytes(length: int) -> bytes:
return bytes([0] * length)
def _stable_json(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"))
def _header_aad(header: dict[str, Any]) -> bytes:
return _stable_json(header).encode("utf-8")
def _build_padded_payload(plaintext: str) -> bytes:
data = plaintext.encode("utf-8")
length = len(data)
target = PAD_BUCKET
if length + 6 > target:
target = ((length + 6 + PAD_STEP - 1) // PAD_STEP) * PAD_STEP
if target > PAD_MAX:
target = ((length + 6 + PAD_STEP - 1) // PAD_STEP) * PAD_STEP
out = bytearray(target)
out[0:4] = PAD_MAGIC.encode("utf-8")
out[4] = (length >> 8) & 0xFF
out[5] = length & 0xFF
out[6 : 6 + length] = data
if target > length + 6:
out[6 + length :] = os.urandom(target - (6 + length))
return bytes(out)
def _unpad_payload(data: bytes) -> str:
if len(data) < 6:
return data.decode("utf-8", errors="replace")
magic = data[:4].decode("utf-8", errors="ignore")
if magic != PAD_MAGIC:
return data.decode("utf-8", errors="replace")
length = (data[4] << 8) + data[5]
if length <= 0 or 6 + length > len(data):
return data.decode("utf-8", errors="replace")
return data[6 : 6 + length].decode("utf-8", errors="replace")
def _load_all_states() -> dict[str, dict[str, Any]]:
with STATE_LOCK:
try:
raw = read_secure_json(STATE_FILE, lambda: {})
except Exception:
import logging
logging.getLogger(__name__).warning(
"Wormhole ratchet state could not be decrypted — starting fresh"
)
STATE_FILE.unlink(missing_ok=True)
raw = {}
if not isinstance(raw, dict):
return {}
return {str(k): dict(v) for k, v in raw.items() if isinstance(v, dict)}
def _save_all_states(states: dict[str, dict[str, Any]]) -> None:
with STATE_LOCK:
DATA_DIR.mkdir(parents=True, exist_ok=True)
write_secure_json(STATE_FILE, states)
def _get_state(peer_id: str) -> dict[str, Any] | None:
return _load_all_states().get(peer_id)
def _set_state(peer_id: str, state: dict[str, Any]) -> None:
states = _load_all_states()
states[peer_id] = state
_save_all_states(states)
def reset_wormhole_dm_ratchet(peer_id: str | None = None) -> dict[str, Any]:
if peer_id:
states = _load_all_states()
states.pop(peer_id, None)
_save_all_states(states)
else:
_save_all_states({})
return {"ok": True, "peer_id": peer_id or "", "cleared_all": not bool(peer_id)}
def _generate_ratchet_key_pair() -> dict[str, str]:
priv = x25519.X25519PrivateKey.generate()
priv_raw = priv.private_bytes(Encoding.Raw, PrivateFormat.Raw, NoEncryption())
pub_raw = priv.public_key().public_bytes(Encoding.Raw, PublicFormat.Raw)
return {
"pub": _b64(pub_raw),
"priv": _b64(priv_raw),
"algo": "X25519",
}
def _derive_dh_secret(priv_b64: str, their_pub_b64: str) -> bytes:
priv = x25519.X25519PrivateKey.from_private_bytes(_unb64(priv_b64))
pub = x25519.X25519PublicKey.from_public_bytes(_unb64(their_pub_b64))
return priv.exchange(pub)
def _wormhole_long_term_dh_priv_b64() -> str:
data = read_wormhole_identity()
if not data.get("bootstrapped") or not data.get("dh_private_key"):
bootstrap_wormhole_identity()
data = read_wormhole_identity()
return str(data.get("dh_private_key", "") or "")
def _hkdf(ikm: bytes, salt: bytes, info: str, length: int) -> bytes:
derived = HKDF(
algorithm=hashes.SHA256(),
length=length,
salt=salt,
info=info.encode("utf-8"),
).derive(ikm)
return bytes(derived)
def _kdf_rk(rk: bytes, dh_out: bytes) -> tuple[bytes, bytes]:
salt = rk if rk else _zero_bytes(32)
out = _hkdf(dh_out, salt, "SB-DR-RK", 64)
return out[:32], out[32:64]
def _hmac_sha256(key_bytes: bytes, data: bytes) -> bytes:
mac = hmac.HMAC(key_bytes, hashes.SHA256())
mac.update(data)
return bytes(mac.finalize())
def _kdf_ck(ck: bytes) -> tuple[bytes, bytes]:
mk = _hmac_sha256(ck, b"\x01")
next_ck = _hmac_sha256(ck, b"\x02")
return next_ck, mk
def _aes_gcm_encrypt(mk: bytes, plaintext: str, aad: bytes) -> str:
iv = os.urandom(12)
aes = AESGCM(mk)
encoded = _build_padded_payload(plaintext)
ciphertext = aes.encrypt(iv, encoded, aad)
return _b64(iv + ciphertext)
def _aes_gcm_decrypt(mk: bytes, ciphertext_b64: str, aad: bytes) -> str:
combined = _unb64(ciphertext_b64)
iv = combined[:12]
ciphertext = combined[12:]
aes = AESGCM(mk)
plaintext = aes.decrypt(iv, ciphertext, aad)
return _unpad_payload(bytes(plaintext))
def _skip_message_keys(state: dict[str, Any], until: int) -> None:
if not state.get("ckr"):
return
skipped = dict(state.get("skipped") or {})
while int(state.get("nr", 0) or 0) < until:
next_ck, mk = _kdf_ck(_unb64(str(state["ckr"])))
key_id = f"{state.get('dhRemote', '')}:{int(state.get('nr', 0) or 0)}"
if len(skipped) < MAX_SKIP:
skipped[key_id] = _b64(mk)
state["ckr"] = _b64(next_ck)
state["nr"] = int(state.get("nr", 0) or 0) + 1
state["skipped"] = skipped
def _dh_ratchet(state: dict[str, Any], remote_dh: str, pn: int) -> dict[str, Any]:
_skip_message_keys(state, pn)
state["pn"] = int(state.get("ns", 0) or 0)
state["ns"] = 0
state["nr"] = 0
state["dhRemote"] = remote_dh
rk_bytes = _unb64(str(state.get("rk", "") or "")) or _zero_bytes(32)
dh_out_1 = _derive_dh_secret(str(state.get("dhSelfPriv", "")), str(state.get("dhRemote", "")))
rk_1, ck_r = _kdf_rk(rk_bytes, dh_out_1)
state["rk"] = _b64(rk_1)
state["ckr"] = _b64(ck_r)
fresh = _generate_ratchet_key_pair()
state["dhSelfPub"] = fresh["pub"]
state["dhSelfPriv"] = fresh["priv"]
dh_out_2 = _derive_dh_secret(str(state.get("dhSelfPriv", "")), str(state.get("dhRemote", "")))
rk_2, ck_s = _kdf_rk(_unb64(str(state.get("rk", ""))), dh_out_2)
state["rk"] = _b64(rk_2)
state["cks"] = _b64(ck_s)
state["algo"] = "X25519"
state["updated"] = int(time.time() * 1000)
return state
def _init_sender_state(peer_id: str, their_dh_pub: str) -> dict[str, Any]:
fresh = _generate_ratchet_key_pair()
dh_out = _derive_dh_secret(fresh["priv"], their_dh_pub)
rk, ck = _kdf_rk(_zero_bytes(32), dh_out)
return {
"algo": "X25519",
"rk": _b64(rk),
"cks": _b64(ck),
"ckr": "",
"dhSelfPub": fresh["pub"],
"dhSelfPriv": fresh["priv"],
"dhRemote": their_dh_pub,
"ns": 0,
"nr": 0,
"pn": 0,
"skipped": {},
"updated": int(time.time() * 1000),
}
def _init_receiver_state(peer_id: str, sender_dh_pub: str) -> dict[str, Any]:
long_term_priv = _wormhole_long_term_dh_priv_b64()
if not long_term_priv:
raise ValueError("missing_long_term_key")
dh_out = _derive_dh_secret(long_term_priv, sender_dh_pub)
rk, ck = _kdf_rk(_zero_bytes(32), dh_out)
fresh = _generate_ratchet_key_pair()
return {
"algo": "X25519",
"rk": _b64(rk),
"cks": "",
"ckr": _b64(ck),
"dhSelfPub": fresh["pub"],
"dhSelfPriv": fresh["priv"],
"dhRemote": sender_dh_pub,
"ns": 0,
"nr": 0,
"pn": 0,
"skipped": {},
"updated": int(time.time() * 1000),
}
def _ensure_send_chain(state: dict[str, Any]) -> dict[str, Any]:
if state.get("cks"):
return state
rk_bytes = _unb64(str(state.get("rk", "") or "")) or _zero_bytes(32)
dh_out = _derive_dh_secret(str(state.get("dhSelfPriv", "")), str(state.get("dhRemote", "")))
rk, ck = _kdf_rk(rk_bytes, dh_out)
state["rk"] = _b64(rk)
state["cks"] = _b64(ck)
state["updated"] = int(time.time() * 1000)
return state
def encrypt_wormhole_dm(peer_id: str, peer_dh_pub: str, plaintext: str) -> dict[str, Any]:
if not peer_id or not peer_dh_pub:
return {"ok": False, "detail": "peer_id and peer_dh_pub are required"}
state = _get_state(peer_id)
if not state:
state = _init_sender_state(peer_id, peer_dh_pub)
state = _ensure_send_chain(state)
next_ck, mk = _kdf_ck(_unb64(str(state.get("cks", ""))))
n = int(state.get("ns", 0) or 0)
state["ns"] = n + 1
state["cks"] = _b64(next_ck)
header = {
"v": 2,
"dh": str(state.get("dhSelfPub", "")),
"pn": int(state.get("pn", 0) or 0),
"n": n,
"alg": "X25519",
}
ct = _aes_gcm_encrypt(mk, plaintext, _header_aad(header))
wrapped = _b64(_stable_json({"h": header, "ct": ct}).encode("utf-8"))
state["updated"] = int(time.time() * 1000)
_set_state(peer_id, state)
return {"ok": True, "result": f"dr2:{wrapped}"}
def decrypt_wormhole_dm(peer_id: str, ciphertext: str) -> dict[str, Any]:
if not ciphertext.startswith("dr2:"):
return {"ok": False, "detail": "legacy"}
try:
raw = ciphertext[4:]
payload = json.loads(_unb64(raw).decode("utf-8"))
header = dict(payload.get("h") or {})
ct = str(payload.get("ct") or "")
remote_dh = str(header.get("dh") or "")
pn = int(header.get("pn", 0) or 0)
n = int(header.get("n", 0) or 0)
state = _get_state(peer_id)
if not state:
state = _init_receiver_state(peer_id, remote_dh)
if remote_dh and remote_dh != str(state.get("dhRemote", "")):
state = _dh_ratchet(state, remote_dh, pn)
skipped = dict(state.get("skipped") or {})
skip_key = f"{remote_dh}:{n}"
if skip_key in skipped:
mk = _unb64(skipped.pop(skip_key))
state["skipped"] = skipped
_set_state(peer_id, state)
return {"ok": True, "result": _aes_gcm_decrypt(mk, ct, _header_aad(header))}
_skip_message_keys(state, n)
if not state.get("ckr"):
return {"ok": False, "detail": "no_receive_chain"}
next_ck, mk = _kdf_ck(_unb64(str(state.get("ckr", ""))))
state["ckr"] = _b64(next_ck)
state["nr"] = int(state.get("nr", 0) or 0) + 1
state["updated"] = int(time.time() * 1000)
_set_state(peer_id, state)
return {"ok": True, "result": _aes_gcm_decrypt(mk, ct, _header_aad(header))}
except Exception as exc:
return {"ok": False, "detail": str(exc) or "ratchet_decrypt_failed"}
backend/services/mesh/mesh_wormhole_seal.py
+267
View File
@@ -0,0 +1,267 @@
"""Wormhole-owned sender seal helpers."""
from __future__ import annotations
import base64
import json
import os
from typing import Any
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import x25519
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives.serialization import Encoding, NoEncryption, PrivateFormat, PublicFormat
from services.mesh.mesh_crypto import (
parse_public_key_algo,
verify_node_binding,
verify_signature,
)
from services.mesh.mesh_protocol import PROTOCOL_VERSION
from services.mesh.mesh_wormhole_identity import (
bootstrap_wormhole_identity,
read_wormhole_identity,
sign_wormhole_message,
)
from services.wormhole_settings import read_wormhole_settings
def _unb64(data: str | bytes | None) -> bytes:
if not data:
return b""
if isinstance(data, bytes):
return base64.b64decode(data)
return base64.b64decode(data.encode("ascii"))
def _derive_aes_key(my_private_b64: str, peer_public_b64: str) -> bytes:
priv = x25519.X25519PrivateKey.from_private_bytes(_unb64(my_private_b64))
pub = x25519.X25519PublicKey.from_public_bytes(_unb64(peer_public_b64))
secret = priv.exchange(pub)
# For compatibility with the browser path, use the raw 32-byte X25519 secret directly
# as the AES-256-GCM key material.
return secret
def _seal_salt(recipient_id: str, msg_id: str, extra: str = "") -> bytes:
material = f"SB-SEAL-SALT|{recipient_id}|{msg_id}|{PROTOCOL_VERSION}|{extra}".encode("utf-8")
digest = hashes.Hash(hashes.SHA256())
digest.update(material)
return digest.finalize()
def _derive_seal_key_v2(my_private_b64: str, peer_public_b64: str, recipient_id: str, msg_id: str) -> bytes:
priv = x25519.X25519PrivateKey.from_private_bytes(_unb64(my_private_b64))
pub = x25519.X25519PublicKey.from_public_bytes(_unb64(peer_public_b64))
secret = priv.exchange(pub)
return HKDF(
algorithm=hashes.SHA256(),
length=32,
salt=_seal_salt(recipient_id, msg_id),
info=b"SB-SENDER-SEAL-V2",
).derive(secret)
def _x25519_pair() -> tuple[str, str]:
priv = x25519.X25519PrivateKey.generate()
priv_raw = priv.private_bytes(
encoding=Encoding.Raw,
format=PrivateFormat.Raw,
encryption_algorithm=NoEncryption(),
)
pub_raw = priv.public_key().public_bytes(
encoding=Encoding.Raw,
format=PublicFormat.Raw,
)
return _b64(priv_raw), _b64(pub_raw)
def _derive_seal_key_v3(
my_private_b64: str,
peer_public_b64: str,
recipient_id: str,
msg_id: str,
ephemeral_pub_b64: str,
) -> bytes:
priv = x25519.X25519PrivateKey.from_private_bytes(_unb64(my_private_b64))
pub = x25519.X25519PublicKey.from_public_bytes(_unb64(peer_public_b64))
secret = priv.exchange(pub)
return HKDF(
algorithm=hashes.SHA256(),
length=32,
salt=_seal_salt(recipient_id, msg_id, ephemeral_pub_b64),
info=b"SB-SENDER-SEAL-V3",
).derive(secret)
def _b64(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
def _seal_payload_version(sender_seal: str) -> tuple[str, str, str]:
value = str(sender_seal or "").strip()
if value.startswith("v3:"):
_, ephemeral_pub, encoded = value.split(":", 2)
return "v3", ephemeral_pub, encoded
if value.startswith("v2:"):
return "v2", "", value[3:]
return "legacy", "", value
def _legacy_seal_allowed() -> bool:
try:
settings = read_wormhole_settings()
if bool(settings.get("enabled")) or bool(settings.get("anonymous_mode")):
return False
except Exception:
pass
return True
def build_sender_seal(
*,
recipient_id: str,
recipient_dh_pub: str,
msg_id: str,
timestamp: int,
) -> dict[str, Any]:
recipient_id = str(recipient_id or "").strip()
recipient_dh_pub = str(recipient_dh_pub or "").strip()
msg_id = str(msg_id or "").strip()
timestamp = int(timestamp or 0)
if not recipient_id or not recipient_dh_pub or not msg_id or timestamp <= 0:
return {"ok": False, "detail": "recipient_id, recipient_dh_pub, msg_id, and timestamp required"}
identity = read_wormhole_identity()
if not identity.get("bootstrapped"):
bootstrap_wormhole_identity()
identity = read_wormhole_identity()
my_private = str(identity.get("dh_private_key", "") or "")
if not my_private:
return {"ok": False, "detail": "Missing Wormhole DH private key"}
try:
ephemeral_private, ephemeral_public = _x25519_pair()
signed = sign_wormhole_message(
f"seal|v3|{msg_id}|{timestamp}|{recipient_id}|{ephemeral_public}"
)
if not verify_node_binding(
str(signed.get("node_id", "") or ""),
str(signed.get("public_key", "") or ""),
):
return {"ok": False, "detail": "Sender seal node binding failed"}
key = _derive_seal_key_v3(
ephemeral_private,
recipient_dh_pub,
recipient_id,
msg_id,
ephemeral_public,
)
plaintext = json.dumps(
{
"seal_version": "v3",
"ephemeral_pub_key": ephemeral_public,
"sender_id": str(signed.get("node_id", "") or ""),
"public_key": str(signed.get("public_key", "") or ""),
"public_key_algo": str(signed.get("public_key_algo", "") or ""),
"msg_id": msg_id,
"timestamp": timestamp,
"signature": str(signed.get("signature", "") or ""),
"protocol_version": str(signed.get("protocol_version", "") or ""),
},
ensure_ascii=False,
separators=(",", ":"),
).encode("utf-8")
iv = _b64(os.urandom(12))
except Exception as exc:
return {"ok": False, "detail": str(exc) or "sender_seal_build_failed"}
iv_bytes = _unb64(iv)
ciphertext = AESGCM(key).encrypt(iv_bytes, plaintext, None)
combined = iv_bytes + ciphertext
return {
"ok": True,
"sender_seal": f"v3:{ephemeral_public}:{_b64(combined)}",
"sender_id": str(signed.get("node_id", "") or ""),
"public_key": str(signed.get("public_key", "") or ""),
"public_key_algo": str(signed.get("public_key_algo", "") or ""),
"protocol_version": str(signed.get("protocol_version", "") or ""),
}
def open_sender_seal(
*,
sender_seal: str,
candidate_dh_pub: str,
recipient_id: str,
expected_msg_id: str,
) -> dict[str, Any]:
if not sender_seal or not candidate_dh_pub or not recipient_id or not expected_msg_id:
return {"ok": False, "detail": "Missing sender_seal, candidate_dh_pub, recipient_id, or expected_msg_id"}
identity = read_wormhole_identity()
if not identity.get("bootstrapped"):
bootstrap_wormhole_identity()
identity = read_wormhole_identity()
my_private = str(identity.get("dh_private_key", "") or "")
if not my_private:
return {"ok": False, "detail": "Missing Wormhole DH private key"}
try:
seal_version, ephemeral_pub, encoded = _seal_payload_version(sender_seal)
if seal_version == "v3":
key = _derive_seal_key_v3(my_private, ephemeral_pub, recipient_id, expected_msg_id, ephemeral_pub)
elif seal_version == "v2":
key = _derive_seal_key_v2(my_private, candidate_dh_pub, recipient_id, expected_msg_id)
else:
if not _legacy_seal_allowed():
return {"ok": False, "detail": "Legacy sender seals are disabled in hardened modes"}
key = _derive_aes_key(my_private, candidate_dh_pub)
combined = _unb64(encoded)
iv = combined[:12]
ciphertext = combined[12:]
plaintext = AESGCM(key).decrypt(iv, ciphertext, None).decode("utf-8")
seal = json.loads(plaintext)
except Exception as exc:
return {"ok": False, "detail": str(exc) or "sender_seal_decrypt_failed"}
sender_id = str(seal.get("sender_id", "") or "")
public_key = str(seal.get("public_key", "") or "")
public_key_algo = str(seal.get("public_key_algo", "") or "")
msg_id = str(seal.get("msg_id", "") or "")
timestamp = int(seal.get("timestamp", 0) or 0)
signature = str(seal.get("signature", "") or "")
if not sender_id or not public_key or not public_key_algo or not msg_id or not signature:
return {"ok": False, "detail": "Malformed sender seal"}
if msg_id != expected_msg_id:
return {"ok": False, "detail": "Sender seal message mismatch"}
if seal_version == "v3" and str(seal.get("ephemeral_pub_key", "") or "") != ephemeral_pub:
return {"ok": False, "detail": "Sender seal ephemeral key mismatch"}
if not verify_node_binding(sender_id, public_key):
return {"ok": True, "sender_id": sender_id, "seal_verified": False}
algo = parse_public_key_algo(public_key_algo)
if not algo:
return {"ok": True, "sender_id": sender_id, "seal_verified": False}
if seal_version == "v3":
message = f"seal|v3|{msg_id}|{timestamp}|{recipient_id}|{ephemeral_pub}"
else:
message = f"seal|{msg_id}|{timestamp}|{recipient_id}"
verified = verify_signature(
public_key_b64=public_key,
public_key_algo=algo,
signature_hex=signature,
payload=message,
)
return {
"ok": True,
"sender_id": sender_id,
"seal_verified": bool(verified),
"public_key": public_key,
"public_key_algo": public_key_algo,
"timestamp": timestamp,
"msg_id": msg_id,
}
backend/services/mesh/mesh_wormhole_sender_token.py
+134
View File
@@ -0,0 +1,134 @@
"""Short-lived Wormhole sender tokens for DM metadata reduction.
These tokens let the client send a sealed DM without placing the long-term
sender id and public key directly into the DM send request body. The token is
single-use, recipient-bound, and kept in memory only.
"""
from __future__ import annotations
import hashlib
import secrets
import time
from typing import Any
from cachetools import TTLCache
from services.mesh.mesh_wormhole_identity import bootstrap_wormhole_identity, read_wormhole_identity
from services.mesh.mesh_protocol import PROTOCOL_VERSION
_SENDER_TOKEN_TTL_S = 5 * 60
_sender_tokens: TTLCache[str, dict[str, Any]] = TTLCache(maxsize=2048, ttl=_SENDER_TOKEN_TTL_S)
def _sender_token_hash(token: str) -> str:
return hashlib.sha256((token or "").encode("utf-8")).hexdigest()
def _token_binding_hash(recipient_token: str) -> str:
return hashlib.sha256((recipient_token or "").encode("utf-8")).hexdigest()
def issue_wormhole_dm_sender_token(
*,
recipient_id: str,
delivery_class: str,
recipient_token: str = "",
ttl_seconds: int = _SENDER_TOKEN_TTL_S,
) -> dict[str, Any]:
recipient_id = str(recipient_id or "").strip()
delivery_class = str(delivery_class or "").strip().lower()
if delivery_class not in ("request", "shared"):
return {"ok": False, "detail": "Invalid delivery_class"}
if not recipient_id:
return {"ok": False, "detail": "recipient_id required"}
if delivery_class == "shared" and not recipient_token:
return {"ok": False, "detail": "recipient_token required for shared delivery"}
data = read_wormhole_identity()
if not data.get("bootstrapped"):
bootstrap_wormhole_identity()
data = read_wormhole_identity()
if not data.get("node_id") or not data.get("public_key"):
return {"ok": False, "detail": "Wormhole identity unavailable"}
token = secrets.token_urlsafe(32)
now = int(time.time())
expires_at = now + max(30, min(int(ttl_seconds or _SENDER_TOKEN_TTL_S), _SENDER_TOKEN_TTL_S))
_sender_tokens[token] = {
"sender_id": str(data.get("node_id", "")),
"public_key": str(data.get("public_key", "")),
"public_key_algo": str(data.get("public_key_algo", "Ed25519") or "Ed25519"),
"protocol_version": PROTOCOL_VERSION,
"recipient_id": recipient_id,
"delivery_class": delivery_class,
"recipient_token_hash": _token_binding_hash(recipient_token),
"issued_at": now,
"expires_at": expires_at,
}
return {
"ok": True,
"sender_token": token,
"expires_at": expires_at,
"delivery_class": delivery_class,
}
def issue_wormhole_dm_sender_tokens(
*,
recipient_id: str,
delivery_class: str,
recipient_token: str = "",
count: int = 3,
ttl_seconds: int = _SENDER_TOKEN_TTL_S,
) -> dict[str, Any]:
token_count = max(1, min(int(count or 1), 4))
tokens: list[dict[str, Any]] = []
for _ in range(token_count):
issued = issue_wormhole_dm_sender_token(
recipient_id=recipient_id,
delivery_class=delivery_class,
recipient_token=recipient_token,
ttl_seconds=ttl_seconds,
)
if not issued.get("ok"):
return issued
tokens.append(
{
"sender_token": str(issued.get("sender_token", "")),
"expires_at": int(issued.get("expires_at", 0) or 0),
}
)
return {
"ok": True,
"delivery_class": delivery_class,
"tokens": tokens,
}
def consume_wormhole_dm_sender_token(
*,
sender_token: str,
recipient_id: str,
delivery_class: str,
recipient_token: str = "",
) -> dict[str, Any]:
token = str(sender_token or "").strip()
if not token:
return {"ok": False, "detail": "sender_token required"}
token_hash = _sender_token_hash(token)
record = _sender_tokens.pop(token, None)
if not record:
return {"ok": False, "detail": "sender_token invalid or expired"}
bound_recipient_id = str(record.get("recipient_id", "") or "")
normalized_recipient_id = str(recipient_id or "").strip()
if normalized_recipient_id and bound_recipient_id != normalized_recipient_id:
return {"ok": False, "detail": "sender_token recipient mismatch"}
if str(record.get("delivery_class", "")) != str(delivery_class or "").strip().lower():
return {"ok": False, "detail": "sender_token delivery_class mismatch"}
if str(record.get("recipient_token_hash", "")) != _token_binding_hash(recipient_token):
return {"ok": False, "detail": "sender_token mailbox binding mismatch"}
expires_at = int(record.get("expires_at", 0) or 0)
if expires_at and expires_at < int(time.time()):
return {"ok": False, "detail": "sender_token expired"}
return {"ok": True, "sender_token_hash": token_hash, **record}
backend/services/mesh/meshtastic_topics.py
+176
View File
@@ -0,0 +1,176 @@
"""Helpers for Meshtastic MQTT roots, topic parsing, and subscriptions."""
from __future__ import annotations
import re
from typing import Iterable
# Official/default region roots we actively watch on the public broker.
DEFAULT_ROOTS: tuple[str, ...] = (
"US",
"EU_868",
"EU_433",
"CN",
"JP",
"KR",
"TW",
"RU",
"IN",
"ANZ",
"ANZ_433",
"NZ_865",
"TH",
"UA_868",
"UA_433",
"MY_433",
"MY_919",
"SG_923",
"LORA_24",
)
# Legacy/community roots still seen in the wild on public/community brokers.
COMMUNITY_ROOTS: tuple[str, ...] = (
"EU",
"AU",
"UA",
"BR",
"AF",
"ME",
"SEA",
"SA",
"PL",
)
_ROOT_SEGMENT_RE = re.compile(r"^[A-Za-z0-9_+\-]+$")
_TOPIC_SEGMENT_RE = re.compile(r"^[A-Za-z0-9_+\-#]+$")
def _dedupe(values: Iterable[str]) -> list[str]:
out: list[str] = []
seen: set[str] = set()
for value in values:
if value not in seen:
out.append(value)
seen.add(value)
return out
def _split_config_values(raw: str) -> list[str]:
if not raw:
return []
normalized = raw.replace("\n", ",").replace(";", ",")
return [item.strip() for item in normalized.split(",") if item.strip()]
def normalize_root(value: str) -> str | None:
"""Normalize a Meshtastic root like `PL` or `US/rob/snd`."""
raw = str(value or "").strip()
if not raw:
return None
if raw.startswith("msh/"):
raw = raw[4:]
raw = raw.strip("/")
if raw.endswith("/#"):
raw = raw[:-2].rstrip("/")
if not raw:
return None
parts = [part for part in raw.split("/") if part]
if not parts:
return None
if any(part in {"+", "#"} for part in parts):
return None
if any(not _ROOT_SEGMENT_RE.match(part) for part in parts):
return None
return "/".join(parts)
def normalize_topic_filter(value: str) -> str | None:
"""Normalize a full MQTT subscription filter."""
raw = str(value or "").strip()
if not raw:
return None
if not raw.startswith("msh/"):
root = normalize_root(raw)
return f"msh/{root}/#" if root else None
raw = raw.strip("/")
parts = [part for part in raw.split("/") if part]
if not parts or parts[0] != "msh":
return None
if any(part != "+" and not _TOPIC_SEGMENT_RE.match(part) for part in parts[1:]):
return None
return "/".join(parts)
def build_subscription_topics(
extra_roots: str = "",
extra_topics: str = "",
include_defaults: bool = True,
) -> list[str]:
roots: list[str] = []
if include_defaults:
roots.extend(DEFAULT_ROOTS)
roots.extend(COMMUNITY_ROOTS)
roots.extend(root for root in (normalize_root(item) for item in _split_config_values(extra_roots)) if root)
topics = [f"msh/{root}/#" for root in _dedupe(roots)]
topics.extend(
topic
for topic in (
normalize_topic_filter(item) for item in _split_config_values(extra_topics)
)
if topic
)
return _dedupe(topics)
def known_roots(extra_roots: str = "", include_defaults: bool = True) -> list[str]:
topics = build_subscription_topics(extra_roots=extra_roots, include_defaults=include_defaults)
roots: list[str] = []
for topic in topics:
if not topic.startswith("msh/") or not topic.endswith("/#"):
continue
root = normalize_root(topic[4:-2])
if root:
roots.append(root)
return _dedupe(roots)
def parse_topic_metadata(topic: str) -> dict[str, str]:
"""Extract region/root/channel metadata from a Meshtastic MQTT topic."""
parts = [part for part in str(topic or "").strip("/").split("/") if part]
if not parts or parts[0] != "msh":
return {"region": "?", "root": "?", "channel": "LongFast", "mode": "", "version": ""}
mode_idx = -1
for idx in range(1, len(parts)):
if parts[idx] in {"e", "c", "json"}:
mode_idx = idx
break
version = ""
root_parts = parts[1:]
channel = "LongFast"
mode = ""
if mode_idx != -1:
mode = parts[mode_idx]
maybe_version_idx = mode_idx - 1
if maybe_version_idx >= 1 and parts[maybe_version_idx].isdigit():
version = parts[maybe_version_idx]
root_parts = parts[1:maybe_version_idx]
else:
root_parts = parts[1:mode_idx]
if len(parts) > mode_idx + 1:
channel = parts[mode_idx + 1]
root = "/".join(root_parts) if root_parts else "?"
region = root_parts[0] if root_parts else "?"
return {
"region": region,
"root": root,
"channel": channel or "LongFast",
"mode": mode,
"version": version,
}
backend/services/network_utils.py
+13 -3
View File
@@ -34,6 +34,11 @@ _CIRCUIT_BREAKER_TTL = 120 # 2 minutes
# Lock protecting _domain_fail_cache and _circuit_breaker mutations
_cb_lock = threading.Lock()
class UpstreamCircuitBreakerError(OSError):
"""Raised when a domain recently failed hard and is temporarily skipped."""
class _DummyResponse:
"""Minimal response object matching requests.Response interface."""
def __init__(self, status_code, text):
@@ -67,7 +72,9 @@ def fetch_with_curl(url, method="GET", json_data=None, timeout=15, headers=None)
# Circuit breaker: if domain failed completely <2min ago, fail fast
with _cb_lock:
if domain in _circuit_breaker and (time.time() - _circuit_breaker[domain]) < _CIRCUIT_BREAKER_TTL:
raise Exception(f"Circuit breaker open for {domain} (failed <{_CIRCUIT_BREAKER_TTL}s ago)")
raise UpstreamCircuitBreakerError(
f"Circuit breaker open for {domain} (failed <{_CIRCUIT_BREAKER_TTL}s ago)"
)
# Check if this domain recently failed with requests — skip straight to curl
with _cb_lock:
@@ -81,6 +88,9 @@ def fetch_with_curl(url, method="GET", json_data=None, timeout=15, headers=None)
res = _session.post(url, json=json_data, timeout=req_timeout, headers=default_headers)
else:
res = _session.get(url, timeout=req_timeout, headers=default_headers)
if res.status_code == 429:
logger.warning(f"Upstream rate limit hit for {url}; not bypassing with curl.")
return res
res.raise_for_status()
# Clear failure caches on success
with _cb_lock:
@@ -106,9 +116,9 @@ def fetch_with_curl(url, method="GET", json_data=None, timeout=15, headers=None)
stdin_data = json.dumps(json_data) if (method == "POST" and json_data) else None
res = subprocess.run(
cmd, capture_output=True, text=True, timeout=timeout + 5,
input=stdin_data
input=stdin_data, encoding="utf-8", errors="replace"
)
if res.returncode == 0 and res.stdout.strip():
if res.returncode == 0 and (res.stdout or "").strip():
# Parse HTTP status code from -w output (last line)
lines = res.stdout.rstrip().rsplit("\n", 1)
body = lines[0] if len(lines) > 1 else res.stdout
backend/services/news_feed_config.py
+42 -9
View File
@@ -2,6 +2,7 @@
News feed configuration — manages the user-customisable RSS feed list.
Feeds are stored in backend/config/news_feeds.json and persist across restarts.
"""
import json
import logging
from pathlib import Path
@@ -9,7 +10,18 @@ from pathlib import Path
logger = logging.getLogger(__name__)
CONFIG_PATH = Path(__file__).parent.parent / "config" / "news_feeds.json"
MAX_FEEDS = 25
MAX_FEEDS = 50
_FEED_URL_REPLACEMENTS = {
"https://www.channelnewsasia.com/rssfeed/8395986": "https://www.channelnewsasia.com/api/v1/rss-outbound-feed?_format=xml",
}
_DEAD_FEED_URLS = {
"https://www3.nhk.or.jp/nhkworld/rss/world.xml",
"https://focustaiwan.tw/rss",
"https://english.kyodonews.net/rss/news.xml",
"https://www.stripes.com/feeds/pacific.rss",
"https://asia.nikkei.com/rss",
"https://www.taipeitimes.com/xml/pda.rss",
}
DEFAULT_FEEDS = [
{"name": "NPR", "url": "https://feeds.npr.org/1004/rss.xml", "weight": 4},
@@ -17,23 +29,40 @@ DEFAULT_FEEDS = [
{"name": "AlJazeera", "url": "https://www.aljazeera.com/xml/rss/all.xml", "weight": 2},
{"name": "NYT", "url": "https://rss.nytimes.com/services/xml/rss/nyt/World.xml", "weight": 1},
{"name": "GDACS", "url": "https://www.gdacs.org/xml/rss.xml", "weight": 5},
{"name": "NHK", "url": "https://www3.nhk.or.jp/nhkworld/rss/world.xml", "weight": 3},
{"name": "CNA", "url": "https://www.channelnewsasia.com/rssfeed/8395986", "weight": 3},
{"name": "CNA", "url": "https://www.channelnewsasia.com/api/v1/rss-outbound-feed?_format=xml", "weight": 3},
{"name": "Mercopress", "url": "https://en.mercopress.com/rss/", "weight": 3},
{"name": "FocusTaiwan", "url": "https://focustaiwan.tw/rss", "weight": 5},
{"name": "Kyodo", "url": "https://english.kyodonews.net/rss/news.xml", "weight": 4},
{"name": "SCMP", "url": "https://www.scmp.com/rss/91/feed", "weight": 4},
{"name": "The Diplomat", "url": "https://thediplomat.com/feed/", "weight": 4},
{"name": "Stars and Stripes", "url": "https://www.stripes.com/feeds/pacific.rss", "weight": 4},
{"name": "Yonhap", "url": "https://en.yna.co.kr/RSS/news.xml", "weight": 4},
{"name": "Nikkei Asia", "url": "https://asia.nikkei.com/rss", "weight": 3},
{"name": "Taipei Times", "url": "https://www.taipeitimes.com/xml/pda.rss", "weight": 4},
{"name": "Asia Times", "url": "https://asiatimes.com/feed/", "weight": 3},
{"name": "Defense News", "url": "https://www.defensenews.com/arc/outboundfeeds/rss/", "weight": 3},
{"name": "Japan Times", "url": "https://www.japantimes.co.jp/feed/", "weight": 3},
{"name": "CSM", "url": "https://www.csmonitor.com/rss/world", "weight": 4},
{"name": "PBS NewsHour", "url": "https://www.pbs.org/newshour/feeds/rss/world", "weight": 4},
{"name": "France 24", "url": "https://www.france24.com/en/rss", "weight": 4},
{"name": "DW", "url": "https://rss.dw.com/xml/rss-en-world", "weight": 4},
]
def _normalise_feeds(feeds: list[dict]) -> list[dict]:
cleaned: list[dict] = []
for feed in feeds:
if not isinstance(feed, dict):
continue
item = dict(feed)
url = str(item.get("url", "")).strip()
if not url:
continue
if url in _FEED_URL_REPLACEMENTS:
item["url"] = _FEED_URL_REPLACEMENTS[url]
url = item["url"]
if url in _DEAD_FEED_URLS:
logger.warning("Dropping dead RSS feed URL from configuration: %s", url)
continue
cleaned.append(item)
return cleaned
def get_feeds() -> list[dict]:
"""Load feeds from config file, falling back to defaults."""
try:
@@ -41,7 +70,10 @@ def get_feeds() -> list[dict]:
data = json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
feeds = data.get("feeds", []) if isinstance(data, dict) else data
if isinstance(feeds, list) and len(feeds) > 0:
return feeds
normalised = _normalise_feeds(feeds)
if normalised != feeds:
save_feeds(normalised)
return normalised
except (IOError, OSError, json.JSONDecodeError, ValueError) as e:
logger.warning(f"Failed to read news feed config: {e}")
return list(DEFAULT_FEEDS)
@@ -51,6 +83,7 @@ def save_feeds(feeds: list[dict]) -> bool:
"""Validate and save feeds to config file. Returns True on success."""
if not isinstance(feeds, list):
return False
feeds = _normalise_feeds(feeds)
if len(feeds) > MAX_FEEDS:
return False
# Validate each feed entry
backend/services/node_settings.py
+57
View File
@@ -0,0 +1,57 @@
from __future__ import annotations
import json
import time
from pathlib import Path
DATA_DIR = Path(__file__).parent.parent / "data"
NODE_FILE = DATA_DIR / "node.json"
_cache: dict | None = None
_cache_ts: float = 0.0
_CACHE_TTL = 5.0
_DEFAULTS = {
"enabled": False,
}
def _safe_int(value: object, default: int = 0) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def read_node_settings() -> dict:
global _cache, _cache_ts
now = time.monotonic()
if _cache is not None and (now - _cache_ts) < _CACHE_TTL:
return _cache
if not NODE_FILE.exists():
result = {**_DEFAULTS, "updated_at": 0}
else:
try:
data = json.loads(NODE_FILE.read_text(encoding="utf-8"))
except Exception:
result = {**_DEFAULTS, "updated_at": 0}
else:
result = {
"enabled": bool(data.get("enabled", _DEFAULTS["enabled"])),
"updated_at": _safe_int(data.get("updated_at", 0) or 0),
}
_cache = result
_cache_ts = now
return result
def write_node_settings(*, enabled: bool | None = None) -> dict:
DATA_DIR.mkdir(parents=True, exist_ok=True)
existing = read_node_settings()
payload = {
"enabled": bool(existing.get("enabled", _DEFAULTS["enabled"])) if enabled is None else bool(enabled),
"updated_at": int(time.time()),
}
NODE_FILE.write_text(json.dumps(payload, indent=2), encoding="utf-8")
global _cache, _cache_ts
_cache = payload
_cache_ts = time.monotonic()
return payload
backend/services/oracle_service.py
+395
View File
@@ -0,0 +1,395 @@
"""Oracle Service — deterministic intelligence ranking for news items.
Enriches news items with:
- oracle_score: risk_score weighted by source confidence (0–10)
- sentiment: VADER compound score (-1.0 to +1.0)
- prediction_odds: matched prediction market probabilities (or None)
- machine_assessment: structured human-readable analysis string
"""
import logging
logger = logging.getLogger(__name__)
_analyzer = None
def _get_analyzer():
global _analyzer
if _analyzer is None:
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
_analyzer = SentimentIntensityAnalyzer()
return _analyzer
def compute_sentiment(headline: str) -> float:
"""VADER compound sentiment score for a headline. Range: -1.0 to +1.0."""
if not headline:
return 0.0
return _get_analyzer().polarity_scores(headline)["compound"]
def compute_oracle_score(risk_score: int, source_weight: float) -> float:
"""Weighted oracle score: risk_score scaled by source confidence.
source_weight is 1–5 (from feed config). Normalised to 0.2–1.0 multiplier.
Result range: 0.0–10.0.
"""
multiplier = source_weight / 5.0 # 1→0.2, 5→1.0
return round(risk_score * multiplier, 1)
_STOP_WORDS = frozenset({
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
"being", "have", "has", "had", "do", "does", "did", "will", "would",
"could", "should", "may", "might", "shall", "can", "this", "that",
"these", "those", "it", "its", "if", "not", "no", "so", "as", "up",
"out", "about", "into", "over", "after", "before", "between", "under",
"than", "then", "more", "most", "other", "some", "such", "only", "own",
"same", "also", "just", "how", "what", "which", "who", "whom", "when",
"where", "why", "all", "each", "every", "both", "few", "many", "much",
"any", "very", "too", "here", "there", "now", "new", "says", "said",
"-", "--", "—", "vs", "vs.", "&", "he", "she", "they", "we", "you",
"his", "her", "my", "our", "your", "their", "him", "us", "them",
})
def _tokenize(text: str) -> set[str]:
"""Lowercase, strip punctuation, remove stop words."""
import re
words = re.findall(r"[a-z0-9]+(?:'[a-z]+)?", text.lower())
return {w for w in words if w not in _STOP_WORDS and len(w) > 1}
def _match_prediction_markets(title: str, markets: list[dict]) -> dict | None:
"""Find best-matching prediction market for a news headline.
Uses Jaccard similarity on meaningful tokens (stop words removed).
Requires at least 2 meaningful keyword overlaps AND Jaccard >= 0.15.
"""
if not markets or not title:
return None
title_words = _tokenize(title)
if len(title_words) < 2:
return None
best_match = None
best_score = 0.0
for market in markets:
market_title = market.get("title", "")
market_words = _tokenize(market_title)
if len(market_words) < 2:
continue
intersection = title_words & market_words
if len(intersection) < 2:
continue
union = title_words | market_words
jaccard = len(intersection) / len(union) if union else 0.0
if jaccard > best_score and jaccard >= 0.15:
best_score = jaccard
best_match = market
if not best_match:
return None
return {
"title": best_match.get("title", ""),
"polymarket_pct": best_match.get("polymarket_pct"),
"kalshi_pct": best_match.get("kalshi_pct"),
"consensus_pct": best_match.get("consensus_pct"),
"match_score": round(best_score, 2),
}
def _build_assessment(oracle_score: float, sentiment: float, prediction: dict | None) -> str:
"""Build structured machine_assessment string."""
parts = []
# Oracle tier
if oracle_score >= 7:
tier = "CRITICAL"
elif oracle_score >= 4:
tier = "ELEVATED"
else:
tier = "ROUTINE"
parts.append(f"ORACLE: {oracle_score}/10 [{tier}]")
# Sentiment
if sentiment >= 0.05:
sdir = "POSITIVE"
elif sentiment <= -0.05:
sdir = "NEGATIVE"
else:
sdir = "NEUTRAL"
parts.append(f"SENTIMENT: {sentiment:+.2f} [{sdir}]")
# Prediction market
if prediction:
consensus = prediction.get("consensus_pct")
if consensus is not None:
parts.append(f"MKT CONSENSUS: {consensus}%")
poly = prediction.get("polymarket_pct")
kalshi = prediction.get("kalshi_pct")
sources = []
if poly is not None:
sources.append(f"Polymarket {poly}%")
if kalshi is not None:
sources.append(f"Kalshi {kalshi}%")
if sources:
parts.append(f" Sources: {' | '.join(sources)}")
return " // ".join(parts[:3]) + ("\n" + parts[3] if len(parts) > 3 else "")
def enrich_news_items(
news_items: list[dict], source_weights: dict[str, float], markets: list[dict] | None = None
) -> list[dict]:
"""Enrich news items with oracle scores, sentiment, and prediction market odds.
Args:
news_items: list of news item dicts (modified in-place)
source_weights: {source_name: weight} from feed config (1–5 scale)
markets: merged prediction market events list (or None)
Returns:
The same list, enriched with oracle_score, sentiment, prediction_odds, machine_assessment.
"""
if markets is None:
markets = []
for item in news_items:
title = item.get("title", "")
source = item.get("source", "")
risk_score = item.get("risk_score", 1)
weight = source_weights.get(source, 3) # default weight 3 (mid-range)
sentiment = compute_sentiment(title)
oracle_score = compute_oracle_score(risk_score, weight)
prediction = _match_prediction_markets(title, markets)
item["sentiment"] = sentiment
item["oracle_score"] = oracle_score
item["prediction_odds"] = prediction
item["machine_assessment"] = _build_assessment(oracle_score, sentiment, prediction)
return news_items
# ---------------------------------------------------------------------------
# Global threat level
# ---------------------------------------------------------------------------
_THREAT_TIERS = [
(80, "SEVERE", "#ef4444"), # red
(60, "HIGH", "#f97316"), # orange
(40, "ELEVATED", "#eab308"), # yellow
(20, "GUARDED", "#3b82f6"), # blue
(0, "GREEN", "#22c55e"), # green
]
def compute_global_threat_level(
news_items: list[dict],
markets: list[dict] | None = None,
military_flights: list[dict] | None = None,
gps_jamming: list[dict] | None = None,
ships: list[dict] | None = None,
correlations: list[dict] | None = None,
) -> dict:
"""Fuse news sentiment, prediction-market conflict odds, event frequency,
military activity, GPS jamming, and cross-layer correlations into a single
0-100 threat score.
Formula (weights sum to 1.0):
0.25 × negative_sentiment_intensity
0.25 × conflict_market_avg_probability
0.10 × high_risk_event_ratio
0.10 × max_oracle_score (normalised to 0-100)
0.10 × military_activity_anomaly
0.10 × gps_jamming_indicator
0.10 × correlation_alerts
"""
if not news_items:
return {"score": 0, "level": "GREEN", "color": "#22c55e", "drivers": []}
# --- Component 1: negative sentiment intensity (0-100) ---
neg_scores = [abs(it.get("sentiment", 0)) for it in news_items if (it.get("sentiment") or 0) <= -0.05]
neg_intensity = (sum(neg_scores) / len(news_items)) * 100 if news_items else 0
neg_intensity = min(100, neg_intensity * 2.5) # scale up — avg abs sentiment rarely > 0.4
# --- Component 2: conflict market avg probability (0-100) ---
conflict_probs: list[float] = []
for m in (markets or []):
if m.get("category") == "CONFLICT":
pct = m.get("consensus_pct") or m.get("polymarket_pct") or m.get("kalshi_pct")
if pct is not None:
conflict_probs.append(float(pct))
conflict_avg = sum(conflict_probs) / len(conflict_probs) if conflict_probs else 0
# --- Component 3: high-risk event ratio (0-100) ---
high_risk = sum(1 for it in news_items if (it.get("risk_score") or 0) >= 7)
event_ratio = (high_risk / len(news_items)) * 100 if news_items else 0
# --- Component 4: max oracle score (0-100) ---
max_oracle = max((it.get("oracle_score") or 0) for it in news_items)
max_oracle_pct = max_oracle * 10 # 0-10 → 0-100
# --- Component 5: military activity anomaly (0-100) ---
mil_count = len(military_flights or [])
# Baseline: ~20-50 military flights is normal. Spike above 80 is anomalous.
mil_anomaly = min(100, max(0, (mil_count - 30) * 2)) if mil_count > 30 else 0
# --- Component 6: GPS jamming indicator (0-100) ---
jam_zones = gps_jamming or []
high_jam = sum(1 for z in jam_zones if z.get("severity") == "high")
med_jam = sum(1 for z in jam_zones if z.get("severity") == "medium")
jam_score = min(100, high_jam * 25 + med_jam * 10)
# --- Component 7: cross-layer correlation alerts (0-100) ---
corr_list: list[dict] = correlations if correlations else []
corr_points = sum(
15 if a.get("severity") == "high" else 8 if a.get("severity") == "medium" else 3
for a in corr_list
)
corr_score = min(100, corr_points)
# --- Weighted fusion ---
score = (
0.25 * neg_intensity
+ 0.25 * conflict_avg
+ 0.10 * event_ratio
+ 0.10 * max_oracle_pct
+ 0.10 * mil_anomaly
+ 0.10 * jam_score
+ 0.10 * corr_score
)
score = max(0, min(100, round(score)))
# --- Tier ---
level, color = "GREEN", "#22c55e"
for threshold, name, c in _THREAT_TIERS:
if score >= threshold:
level, color = name, c
break
# --- Drivers (top reasons for current level) ---
drivers: list[str] = []
if high_risk:
drivers.append(f"{high_risk} CRITICAL-tier news item{'s' if high_risk != 1 else ''}")
if conflict_avg >= 30:
drivers.append(f"CONFLICT markets avg {conflict_avg:.0f}%")
if neg_intensity >= 40:
drivers.append(f"Negative sentiment intensity {neg_intensity:.0f}/100")
if max_oracle >= 7:
drivers.append(f"Max oracle score {max_oracle}/10")
if mil_anomaly >= 30:
drivers.append(f"Military flight spike: {mil_count} tracked")
if jam_score >= 25:
drivers.append(f"GPS jamming: {high_jam} HIGH + {med_jam} MED zones")
if corr_score >= 15:
corr_high = sum(1 for a in corr_list if a.get("severity") == "high")
corr_med = sum(1 for a in corr_list if a.get("severity") == "medium")
drivers.append(f"Cross-layer correlations: {corr_high} HIGH + {corr_med} MED")
if not drivers:
drivers.append("Baseline — no significant threat indicators")
return {
"score": score,
"level": level,
"color": color,
"drivers": drivers[:4],
}
def detect_breaking_events(news_items: list[dict]) -> None:
"""Mark news items as 'breaking' when multiple credible sources converge.
Criteria: cluster_count >= 3 AND risk_score >= 7.
Modifies items in-place by setting ``breaking = True``.
"""
for item in news_items:
cluster = item.get("cluster_count", 1)
risk = item.get("risk_score", 0)
if cluster >= 3 and risk >= 7:
item["breaking"] = True
# ---------------------------------------------------------------------------
# Region oracle intel (for map entity tooltips)
# ---------------------------------------------------------------------------
_region_cache: dict[str, tuple[float, dict]] = {} # "lat,lng" -> (timestamp, result)
_REGION_CACHE_TTL = 60 # seconds
_REGION_RADIUS_DEG = 5.0 # ~500km at equator
def get_region_oracle_intel(lat: float, lng: float, news_items: list[dict]) -> dict:
"""Get oracle intelligence summary for a geographic region.
Finds news items within ~5 degrees, returns top oracle_score item,
average sentiment, and best market match. Cached on 0.5-degree grid.
"""
import time
# Grid-snap for cache key (0.5 degree grid)
grid_lat = round(lat * 2) / 2
grid_lng = round(lng * 2) / 2
cache_key = f"{grid_lat},{grid_lng}"
now = time.time()
if cache_key in _region_cache:
ts, cached_result = _region_cache[cache_key]
if now - ts < _REGION_CACHE_TTL:
return cached_result
# Find nearby news items
nearby = []
for item in news_items:
coords = item.get("coords")
if not coords or len(coords) < 2:
continue
ilat, ilng = coords[0], coords[1]
if abs(ilat - lat) <= _REGION_RADIUS_DEG and abs(ilng - lng) <= _REGION_RADIUS_DEG:
nearby.append(item)
if not nearby:
result = {"found": False}
_region_cache[cache_key] = (now, result)
return result
# Top oracle score item
top = max(nearby, key=lambda x: x.get("oracle_score", 0))
avg_sentiment = sum(it.get("sentiment", 0) for it in nearby) / len(nearby)
# Best market match from nearby items
best_market = None
for it in nearby:
po = it.get("prediction_odds")
if po and po.get("consensus_pct") is not None:
if best_market is None or (po.get("consensus_pct") or 0) > (best_market.get("consensus_pct") or 0):
best_market = po
# Oracle tier
oracle_score = top.get("oracle_score", 0)
tier = "CRITICAL" if oracle_score >= 7 else "ELEVATED" if oracle_score >= 4 else "ROUTINE"
result = {
"found": True,
"top_headline": top.get("title", ""),
"oracle_score": oracle_score,
"tier": tier,
"avg_sentiment": round(avg_sentiment, 2),
"nearby_count": len(nearby),
"market": {
"title": best_market.get("title", ""),
"consensus_pct": best_market.get("consensus_pct"),
} if best_market else None,
}
_region_cache[cache_key] = (now, result)
return result
backend/services/privacy_core_client.py
+440
View File
@@ -0,0 +1,440 @@
"""ctypes bridge for the Rust privacy-core crate.
This module follows the architecture docs in extra/docs-internal:
- Python orchestrates
- Rust owns private protocol state
- Python sees opaque integer handles and serialized ciphertext only
"""
from __future__ import annotations
import ctypes
import json
import os
from pathlib import Path
from typing import Iterable
class PrivacyCoreError(RuntimeError):
"""Raised when the Rust privacy-core returns an error."""
class PrivacyCoreUnavailable(PrivacyCoreError):
"""Raised when the shared library cannot be found or loaded."""
class _ByteBuffer(ctypes.Structure):
_fields_ = [
("data", ctypes.POINTER(ctypes.c_uint8)),
("len", ctypes.c_size_t),
]
class PrivacyCoreClient:
"""Handle-based interface to the local Rust privacy-core."""
def __init__(self, library: ctypes.CDLL, library_path: Path) -> None:
self._library = library
self.library_path = library_path
self._configure_functions()
@classmethod
def load(cls, library_path: str | os.PathLike[str] | None = None) -> "PrivacyCoreClient":
resolved = cls._resolve_library_path(library_path)
try:
library = ctypes.CDLL(str(resolved))
except OSError as exc:
raise PrivacyCoreUnavailable(f"failed to load privacy-core library: {resolved}") from exc
return cls(library, resolved)
@staticmethod
def _resolve_library_path(library_path: str | os.PathLike[str] | None) -> Path:
if library_path:
resolved = Path(library_path).expanduser().resolve()
if not resolved.exists():
raise PrivacyCoreUnavailable(f"privacy-core library not found: {resolved}")
return resolved
env_override = os.environ.get("PRIVACY_CORE_LIB")
if env_override:
resolved = Path(env_override).expanduser().resolve()
if not resolved.exists():
raise PrivacyCoreUnavailable(f"privacy-core library not found: {resolved}")
return resolved
repo_root = Path(__file__).resolve().parents[2]
candidates = []
for profile in ("debug", "release"):
target_dir = repo_root / "privacy-core" / "target" / profile
candidates.extend(
[
target_dir / "privacy_core.dll",
target_dir / "libprivacy_core.so",
target_dir / "libprivacy_core.dylib",
]
)
for candidate in candidates:
if candidate.exists():
return candidate.resolve()
searched = "\n".join(str(candidate) for candidate in candidates)
raise PrivacyCoreUnavailable(
"privacy-core shared library not found. Looked in:\n" f"{searched}"
)
def _configure_functions(self) -> None:
self._library.privacy_core_version.argtypes = []
self._library.privacy_core_version.restype = _ByteBuffer
self._library.privacy_core_last_error_message.argtypes = []
self._library.privacy_core_last_error_message.restype = _ByteBuffer
self._library.privacy_core_free_buffer.argtypes = [_ByteBuffer]
self._library.privacy_core_free_buffer.restype = None
self._library.privacy_core_create_identity.argtypes = []
self._library.privacy_core_create_identity.restype = ctypes.c_uint64
self._library.privacy_core_export_key_package.argtypes = [ctypes.c_uint64]
self._library.privacy_core_export_key_package.restype = _ByteBuffer
self._library.privacy_core_import_key_package.argtypes = [
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_import_key_package.restype = ctypes.c_uint64
self._library.privacy_core_create_group.argtypes = [ctypes.c_uint64]
self._library.privacy_core_create_group.restype = ctypes.c_uint64
self._library.privacy_core_add_member.argtypes = [ctypes.c_uint64, ctypes.c_uint64]
self._library.privacy_core_add_member.restype = ctypes.c_uint64
self._library.privacy_core_remove_member.argtypes = [ctypes.c_uint64, ctypes.c_uint32]
self._library.privacy_core_remove_member.restype = ctypes.c_uint64
self._library.privacy_core_encrypt_group_message.argtypes = [
ctypes.c_uint64,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_encrypt_group_message.restype = _ByteBuffer
self._library.privacy_core_decrypt_group_message.argtypes = [
ctypes.c_uint64,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_decrypt_group_message.restype = _ByteBuffer
self._library.privacy_core_export_public_bundle.argtypes = [ctypes.c_uint64]
self._library.privacy_core_export_public_bundle.restype = _ByteBuffer
self._library.privacy_core_handle_stats.argtypes = [
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_handle_stats.restype = ctypes.c_int64
self._library.privacy_core_commit_message_bytes.argtypes = [ctypes.c_uint64]
self._library.privacy_core_commit_message_bytes.restype = _ByteBuffer
self._library.privacy_core_commit_welcome_message_bytes.argtypes = [
ctypes.c_uint64,
ctypes.c_size_t,
]
self._library.privacy_core_commit_welcome_message_bytes.restype = _ByteBuffer
self._library.privacy_core_commit_joined_group_handle.argtypes = [
ctypes.c_uint64,
ctypes.c_size_t,
]
self._library.privacy_core_commit_joined_group_handle.restype = ctypes.c_uint64
self._library.privacy_core_create_dm_session.argtypes = [
ctypes.c_uint64,
ctypes.c_uint64,
]
self._library.privacy_core_create_dm_session.restype = ctypes.c_int64
self._library.privacy_core_dm_encrypt.argtypes = [
ctypes.c_uint64,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_dm_encrypt.restype = ctypes.c_int64
self._library.privacy_core_dm_decrypt.argtypes = [
ctypes.c_uint64,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_dm_decrypt.restype = ctypes.c_int64
self._library.privacy_core_dm_session_welcome.argtypes = [
ctypes.c_uint64,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_dm_session_welcome.restype = ctypes.c_int64
self._library.privacy_core_join_dm_session.argtypes = [
ctypes.c_uint64,
ctypes.POINTER(ctypes.c_uint8),
ctypes.c_size_t,
]
self._library.privacy_core_join_dm_session.restype = ctypes.c_int64
self._library.privacy_core_release_dm_session.argtypes = [ctypes.c_uint64]
self._library.privacy_core_release_dm_session.restype = ctypes.c_int32
self._library.privacy_core_release_identity.argtypes = [ctypes.c_uint64]
self._library.privacy_core_release_identity.restype = ctypes.c_bool
self._library.privacy_core_release_key_package.argtypes = [ctypes.c_uint64]
self._library.privacy_core_release_key_package.restype = ctypes.c_bool
self._library.privacy_core_release_group.argtypes = [ctypes.c_uint64]
self._library.privacy_core_release_group.restype = ctypes.c_bool
self._library.privacy_core_release_commit.argtypes = [ctypes.c_uint64]
self._library.privacy_core_release_commit.restype = ctypes.c_bool
self._library.privacy_core_reset_all_state.argtypes = []
self._library.privacy_core_reset_all_state.restype = ctypes.c_bool
def version(self) -> str:
return self._consume_string(self._library.privacy_core_version())
def create_identity(self) -> int:
return self._ensure_handle(self._library.privacy_core_create_identity(), "create_identity")
def export_key_package(self, identity_handle: int) -> bytes:
return self._consume_bytes(
self._library.privacy_core_export_key_package(ctypes.c_uint64(identity_handle)),
"export_key_package",
)
def import_key_package(self, data: bytes) -> int:
buffer = self._as_ubyte_buffer(data)
handle = self._library.privacy_core_import_key_package(buffer, len(data))
return self._ensure_handle(handle, "import_key_package")
def create_group(self, identity_handle: int) -> int:
handle = self._library.privacy_core_create_group(ctypes.c_uint64(identity_handle))
return self._ensure_handle(handle, "create_group")
def add_member(self, group_handle: int, key_package_handle: int) -> int:
handle = self._library.privacy_core_add_member(
ctypes.c_uint64(group_handle),
ctypes.c_uint64(key_package_handle),
)
return self._ensure_handle(handle, "add_member")
def remove_member(self, group_handle: int, member_ref: int) -> int:
handle = self._library.privacy_core_remove_member(
ctypes.c_uint64(group_handle),
ctypes.c_uint32(member_ref),
)
return self._ensure_handle(handle, "remove_member")
def encrypt_group_message(self, group_handle: int, plaintext: bytes) -> bytes:
buffer = self._as_ubyte_buffer(plaintext)
return self._consume_bytes(
self._library.privacy_core_encrypt_group_message(
ctypes.c_uint64(group_handle),
buffer,
len(plaintext),
),
"encrypt_group_message",
)
def decrypt_group_message(self, group_handle: int, ciphertext: bytes) -> bytes:
buffer = self._as_ubyte_buffer(ciphertext)
return self._consume_bytes(
self._library.privacy_core_decrypt_group_message(
ctypes.c_uint64(group_handle),
buffer,
len(ciphertext),
),
"decrypt_group_message",
)
def export_public_bundle(self, identity_handle: int) -> bytes:
return self._consume_bytes(
self._library.privacy_core_export_public_bundle(ctypes.c_uint64(identity_handle)),
"export_public_bundle",
)
def handle_stats(self) -> dict:
payload = self._call_i64_bytes_op(
"handle_stats",
lambda out_buf, out_cap: self._library.privacy_core_handle_stats(out_buf, out_cap),
)
try:
return json.loads(payload.decode("utf-8"))
except Exception as exc:
raise PrivacyCoreError(f"handle_stats failed: invalid JSON: {exc}") from exc
def commit_message_bytes(self, commit_handle: int) -> bytes:
return self._consume_bytes(
self._library.privacy_core_commit_message_bytes(ctypes.c_uint64(commit_handle)),
"commit_message_bytes",
)
def commit_welcome_message_bytes(self, commit_handle: int, index: int = 0) -> bytes:
return self._consume_bytes(
self._library.privacy_core_commit_welcome_message_bytes(
ctypes.c_uint64(commit_handle),
ctypes.c_size_t(index),
),
"commit_welcome_message_bytes",
)
def commit_joined_group_handle(self, commit_handle: int, index: int = 0) -> int:
handle = self._library.privacy_core_commit_joined_group_handle(
ctypes.c_uint64(commit_handle),
ctypes.c_size_t(index),
)
return self._ensure_handle(handle, "commit_joined_group_handle")
def create_dm_session(self, initiator_identity: int, responder_key_package: int) -> int:
handle = self._library.privacy_core_create_dm_session(
ctypes.c_uint64(initiator_identity),
ctypes.c_uint64(responder_key_package),
)
if handle > 0:
return int(handle)
raise self._error_for("create_dm_session")
def dm_encrypt(self, session_handle: int, plaintext: bytes) -> bytes:
buffer = self._as_ubyte_buffer(plaintext)
return self._call_i64_bytes_op(
"dm_encrypt",
lambda out_buf, out_cap: self._library.privacy_core_dm_encrypt(
ctypes.c_uint64(session_handle),
buffer,
len(plaintext),
out_buf,
out_cap,
),
)
def dm_decrypt(self, session_handle: int, ciphertext: bytes) -> bytes:
buffer = self._as_ubyte_buffer(ciphertext)
return self._call_i64_bytes_op(
"dm_decrypt",
lambda out_buf, out_cap: self._library.privacy_core_dm_decrypt(
ctypes.c_uint64(session_handle),
buffer,
len(ciphertext),
out_buf,
out_cap,
),
)
def dm_session_welcome(self, session_handle: int) -> bytes:
return self._call_i64_bytes_op(
"dm_session_welcome",
lambda out_buf, out_cap: self._library.privacy_core_dm_session_welcome(
ctypes.c_uint64(session_handle),
out_buf,
out_cap,
),
)
def join_dm_session(self, responder_identity: int, welcome: bytes) -> int:
buffer = self._as_ubyte_buffer(welcome)
handle = self._library.privacy_core_join_dm_session(
ctypes.c_uint64(responder_identity),
buffer,
len(welcome),
)
if handle > 0:
return int(handle)
raise self._error_for("join_dm_session")
def release_dm_session(self, handle: int) -> bool:
return bool(self._library.privacy_core_release_dm_session(ctypes.c_uint64(handle)))
def release_identity(self, handle: int) -> bool:
return bool(self._library.privacy_core_release_identity(ctypes.c_uint64(handle)))
def release_key_package(self, handle: int) -> bool:
return bool(self._library.privacy_core_release_key_package(ctypes.c_uint64(handle)))
def release_group(self, handle: int) -> bool:
return bool(self._library.privacy_core_release_group(ctypes.c_uint64(handle)))
def release_commit(self, handle: int) -> bool:
return bool(self._library.privacy_core_release_commit(ctypes.c_uint64(handle)))
def reset_all_state(self) -> bool:
return bool(self._library.privacy_core_reset_all_state())
def _consume_string(self, buffer: _ByteBuffer) -> str:
payload = self._consume_buffer(buffer)
return payload.decode("utf-8")
def _consume_bytes(self, buffer: _ByteBuffer, operation: str) -> bytes:
payload = self._consume_buffer(buffer)
if payload:
return payload
raise self._error_for(operation)
def _consume_buffer(self, buffer: _ByteBuffer) -> bytes:
try:
if not buffer.data or buffer.len == 0:
return b""
return bytes(ctypes.string_at(buffer.data, buffer.len))
finally:
self._library.privacy_core_free_buffer(buffer)
def _ensure_handle(self, handle: int, operation: str) -> int:
if handle:
return int(handle)
raise self._error_for(operation)
def _call_i64_bytes_op(self, operation: str, invoker) -> bytes:
required = int(invoker(None, 0))
if required < 0:
raise self._error_for(operation)
if required == 0:
return b""
output = (ctypes.c_uint8 * required)()
written = int(invoker(output, required))
if written < 0:
raise self._error_for(operation)
return bytes(output[:written])
def _error_for(self, operation: str) -> PrivacyCoreError:
message = self._last_error()
if message:
return PrivacyCoreError(f"{operation} failed: {message}")
return PrivacyCoreError(f"{operation} failed without an error message")
def _last_error(self) -> str:
return self._consume_string(self._library.privacy_core_last_error_message())
@staticmethod
def _as_ubyte_buffer(data: bytes | bytearray | memoryview) -> ctypes.Array[ctypes.c_uint8]:
if not isinstance(data, (bytes, bytearray, memoryview)):
raise TypeError("privacy-core byte arguments must be bytes-like")
raw = bytes(data)
return (ctypes.c_uint8 * len(raw)).from_buffer_copy(raw)
def candidate_library_paths() -> Iterable[Path]:
"""Expose the default search order for diagnostics/tests."""
repo_root = Path(__file__).resolve().parents[2]
for profile in ("debug", "release"):
target_dir = repo_root / "privacy-core" / "target" / profile
yield target_dir / "privacy_core.dll"
yield target_dir / "libprivacy_core.so"
yield target_dir / "libprivacy_core.dylib"
backend/services/psk_reporter_fetcher.py
+113
View File
@@ -0,0 +1,113 @@
"""
PSK Reporter fetcher — pulls recent digital mode signal reports (FT8, WSPR, etc.)
from the global PSK Reporter network. No API key required.
Docs: https://pskreporter.info/pskdev.html
"""
import logging
import xml.etree.ElementTree as ET
import requests
from cachetools import TTLCache, cached
logger = logging.getLogger(__name__)
_cache = TTLCache(maxsize=1, ttl=600) # 10-minute cache
_ENDPOINT = "https://retrieve.pskreporter.info/query"
def maidenhead_to_latlon(locator: str) -> tuple[float, float] | None:
"""Convert a 4-or-6 character Maidenhead grid locator to (lat, lon)."""
loc = locator.strip().upper()
if len(loc) < 4:
return None
try:
lon = (ord(loc[0]) - ord("A")) * 20 - 180
lat = (ord(loc[1]) - ord("A")) * 10 - 90
lon += int(loc[2]) * 2
lat += int(loc[3])
if len(loc) >= 6:
lon += (ord(loc[4]) - ord("A")) * (2 / 24)
lat += (ord(loc[5]) - ord("A")) * (1 / 24)
# center of sub-square
lon += 1 / 24
lat += 1 / 48
else:
# center of grid square
lon += 1
lat += 0.5
if abs(lat) > 90 or abs(lon) > 180:
return None
return round(lat, 4), round(lon, 4)
except (IndexError, ValueError):
return None
@cached(_cache)
def fetch_psk_reporter_spots() -> list[dict]:
"""Fetch recent FT8 reception reports from PSK Reporter."""
try:
resp = requests.get(
_ENDPOINT,
params={
"mode": "FT8",
"flowStartSeconds": -900, # last 15 minutes
"rronly": 1, # reception reports only
"noactive": 1, # exclude active monitor noise
"rptlimit": 5000, # cap payload size
},
timeout=30,
headers={"Accept": "application/xml"},
)
resp.raise_for_status()
root = ET.fromstring(resp.content)
# PSK Reporter XML uses namespaces
ns = ""
if root.tag.startswith("{"):
ns = root.tag.split("}")[0] + "}"
spots: list[dict] = []
for rec in root.iter(f"{ns}receptionReport"):
receiver_loc = rec.get("receiverLocator", "")
sender_loc = rec.get("senderLocator", "")
# Prefer receiver location (where the signal was heard)
loc_str = receiver_loc or sender_loc
if not loc_str:
continue
coords = maidenhead_to_latlon(loc_str)
if coords is None:
continue
lat, lon = coords
try:
freq = int(rec.get("frequency", "0"))
except (ValueError, TypeError):
freq = 0
try:
snr = int(rec.get("sNR", "0"))
except (ValueError, TypeError):
snr = 0
spots.append({
"lat": lat,
"lon": lon,
"sender": (rec.get("senderCallsign") or "")[:20],
"receiver": (rec.get("receiverCallsign") or "")[:20],
"frequency": freq,
"mode": (rec.get("mode") or "FT8")[:10],
"snr": snr,
"time": rec.get("flowStartSeconds", ""),
})
logger.info("PSK Reporter: fetched %d spots", len(spots))
return spots
except (requests.RequestException, ET.ParseError, Exception) as e:
logger.error("PSK Reporter fetch error: %s", e)
return []
backend/services/radio_intercept.py
+113 -57
View File
@@ -10,6 +10,7 @@ logger = logging.getLogger(__name__)
# Cache the top feeds for 5 minutes so we don't hammer Broadcastify
radio_cache = TTLCache(maxsize=1, ttl=300)
@cached(radio_cache)
def get_top_broadcastify_feeds():
"""
@@ -18,130 +19,184 @@ def get_top_broadcastify_feeds():
"""
logger.info("Scraping Broadcastify Top Feeds (Cache Miss)")
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.9',
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
try:
res = requests.get("https://www.broadcastify.com/listen/top", headers=headers, timeout=10)
if res.status_code != 200:
logger.error(f"Broadcastify Scrape Failed: HTTP {res.status_code}")
return []
soup = BeautifulSoup(res.text, 'html.parser')
table = soup.find('table', {'class': 'btable'})
soup = BeautifulSoup(res.text, "html.parser")
table = soup.find("table", {"class": "btable"})
if not table:
logger.error("Could not find feeds table on Broadcastify.")
return []
feeds = []
rows = table.find_all('tr')[1:] # Skip header row
rows = table.find_all("tr")[1:] # Skip header row
for row in rows:
cols = row.find_all('td')
cols = row.find_all("td")
if len(cols) >= 5:
# Top layout: [Listeners, Feed ID (hidden), Location, Feed Name, Category, Genre]
listeners_str = cols[0].text.strip().replace(',', '')
listeners_str = cols[0].text.strip().replace(",", "")
listeners = int(listeners_str) if listeners_str.isdigit() else 0
link_tag = cols[2].find('a')
link_tag = cols[2].find("a")
if not link_tag:
continue
href = link_tag.get('href', '')
feed_id = href.split('/')[-1] if '/listen/feed/' in href else None
href = link_tag.get("href", "")
feed_id = href.split("/")[-1] if "/listen/feed/" in href else None
if not feed_id:
continue
location = cols[1].text.strip()
name = cols[2].text.strip()
category = cols[3].text.strip()
feeds.append({
"id": feed_id,
"listeners": listeners,
"location": location,
"name": name,
"category": category,
"stream_url": f"https://broadcastify.cdnstream1.com/{feed_id}"
})
feeds.append(
{
"id": feed_id,
"listeners": listeners,
"location": location,
"name": name,
"category": category,
"stream_url": f"https://broadcastify.cdnstream1.com/{feed_id}",
}
)
logger.info(f"Successfully scraped {len(feeds)} top feeds from Broadcastify.")
return feeds
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
logger.error(f"Broadcastify Scrape Exception: {e}")
return []
# Cache OpenMHZ systems mapping so we don't have to fetch all 450+ every time
openmhz_systems_cache = TTLCache(maxsize=1, ttl=3600)
@cached(openmhz_systems_cache)
def get_openmhz_systems():
"""Fetches the full directory of OpenMHZ systems."""
logger.info("Scraping OpenMHZ Systems (Cache Miss)")
scraper = cloudscraper.create_scraper(browser={'browser': 'chrome', 'platform': 'windows', 'desktop': True})
scraper = cloudscraper.create_scraper(
browser={"browser": "chrome", "platform": "windows", "desktop": True}
)
try:
res = scraper.get("https://api.openmhz.com/systems", timeout=15)
if res.status_code == 200:
data = res.json()
# Return list of systems
return data.get('systems', []) if isinstance(data, dict) else []
return data.get("systems", []) if isinstance(data, dict) else []
return []
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
logger.error(f"OpenMHZ Systems Scrape Exception: {e}")
return []
# Cache specific city calls briefly (15-30s) to limit our polling rate
openmhz_calls_cache = TTLCache(maxsize=100, ttl=20)
@cached(openmhz_calls_cache)
def get_recent_openmhz_calls(sys_name: str):
"""Fetches the actual audio burst .m4a URLs for a specific system (e.g., 'wmata')."""
logger.info(f"Fetching OpenMHZ calls for {sys_name} (Cache Miss)")
scraper = cloudscraper.create_scraper(browser={'browser': 'chrome', 'platform': 'windows', 'desktop': True})
scraper = cloudscraper.create_scraper(
browser={"browser": "chrome", "platform": "windows", "desktop": True}
)
try:
url = f"https://api.openmhz.com/{sys_name}/calls"
res = scraper.get(url, timeout=15)
if res.status_code == 200:
data = res.json()
return data.get('calls', []) if isinstance(data, dict) else []
return data.get("calls", []) if isinstance(data, dict) else []
return []
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
logger.error(f"OpenMHZ Calls Scrape Exception ({sys_name}): {e}")
return []
US_STATES = {
'Alabama': 'AL', 'Alaska': 'AK', 'Arizona': 'AZ', 'Arkansas': 'AR', 'California': 'CA',
'Colorado': 'CO', 'Connecticut': 'CT', 'Delaware': 'DE', 'Florida': 'FL', 'Georgia': 'GA',
'Hawaii': 'HI', 'Idaho': 'ID', 'Illinois': 'IL', 'Indiana': 'IN', 'Iowa': 'IA',
'Kansas': 'KS', 'Kentucky': 'KY', 'Louisiana': 'LA', 'Maine': 'ME', 'Maryland': 'MD',
'Massachusetts': 'MA', 'Michigan': 'MI', 'Minnesota': 'MN', 'Mississippi': 'MS',
'Missouri': 'MO', 'Montana': 'MT', 'Nebraska': 'NE', 'Nevada': 'NV', 'New Hampshire': 'NH',
'New Jersey': 'NJ', 'New Mexico': 'NM', 'New York': 'NY', 'North Carolina': 'NC',
'North Dakota': 'ND', 'Ohio': 'OH', 'Oklahoma': 'OK', 'Oregon': 'OR', 'Pennsylvania': 'PA',
'Rhode Island': 'RI', 'South Carolina': 'SC', 'South Dakota': 'SD', 'Tennessee': 'TN',
'Texas': 'TX', 'Utah': 'UT', 'Vermont': 'VT', 'Virginia': 'VA', 'Washington': 'WA',
'West Virginia': 'WV', 'Wisconsin': 'WI', 'Wyoming': 'WY', 'Washington, D.C.': 'DC', 'District of Columbia': 'DC'
"Alabama": "AL",
"Alaska": "AK",
"Arizona": "AZ",
"Arkansas": "AR",
"California": "CA",
"Colorado": "CO",
"Connecticut": "CT",
"Delaware": "DE",
"Florida": "FL",
"Georgia": "GA",
"Hawaii": "HI",
"Idaho": "ID",
"Illinois": "IL",
"Indiana": "IN",
"Iowa": "IA",
"Kansas": "KS",
"Kentucky": "KY",
"Louisiana": "LA",
"Maine": "ME",
"Maryland": "MD",
"Massachusetts": "MA",
"Michigan": "MI",
"Minnesota": "MN",
"Mississippi": "MS",
"Missouri": "MO",
"Montana": "MT",
"Nebraska": "NE",
"Nevada": "NV",
"New Hampshire": "NH",
"New Jersey": "NJ",
"New Mexico": "NM",
"New York": "NY",
"North Carolina": "NC",
"North Dakota": "ND",
"Ohio": "OH",
"Oklahoma": "OK",
"Oregon": "OR",
"Pennsylvania": "PA",
"Rhode Island": "RI",
"South Carolina": "SC",
"South Dakota": "SD",
"Tennessee": "TN",
"Texas": "TX",
"Utah": "UT",
"Vermont": "VT",
"Virginia": "VA",
"Washington": "WA",
"West Virginia": "WV",
"Wisconsin": "WI",
"Wyoming": "WY",
"Washington, D.C.": "DC",
"District of Columbia": "DC",
}
import math
def haversine_distance(lat1, lon1, lat2, lon2):
R = 3958.8 # Earth radius in miles
R = 3958.8 # Earth radius in miles
dLat = math.radians(lat2 - lat1)
dLon = math.radians(lon2 - lon1)
a = math.sin(dLat/2) * math.sin(dLat/2) + \
math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) * \
math.sin(dLon/2) * math.sin(dLon/2)
c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))
a = math.sin(dLat / 2) * math.sin(dLat / 2) + math.cos(math.radians(lat1)) * math.cos(
math.radians(lat2)
) * math.sin(dLon / 2) * math.sin(dLon / 2)
c = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))
return R * c
def find_nearest_openmhz_systems_list(lat: float, lng: float, limit: int = 5):
"""
Finds the strictly nearest OpenMHZ systems by distance.
@@ -153,20 +208,21 @@ def find_nearest_openmhz_systems_list(lat: float, lng: float, limit: int = 5):
# Calculate distance for all systems that provide coordinates
valid_systems = []
for s in systems:
s_lat = s.get('lat')
s_lng = s.get('lng')
s_lat = s.get("lat")
s_lng = s.get("lng")
if s_lat is not None and s_lng is not None:
dist = haversine_distance(lat, lng, float(s_lat), float(s_lng))
s['distance_miles'] = dist
s["distance_miles"] = dist
valid_systems.append(s)
if not valid_systems:
return []
# Sort strictly by distance
valid_systems.sort(key=lambda x: x['distance_miles'])
valid_systems.sort(key=lambda x: x["distance_miles"])
return valid_systems[:limit]
def find_nearest_openmhz_system(lat: float, lng: float):
"""
Returns the single closest OpenMHZ system by distance.
backend/services/region_dossier.py
+83 -35
View File
@@ -16,13 +16,38 @@ dossier_cache = TTLCache(maxsize=500, ttl=86400)
_nominatim_last_call = 0.0
def _reverse_geocode_offline(lat: float, lng: float) -> dict:
"""Offline fallback via reverse_geocoder when external reverse geocoding is blocked."""
try:
import reverse_geocoder as rg
hit = rg.search((lat, lng), mode=1)[0]
country_code = (hit.get("cc") or "").upper()
city = hit.get("name") or ""
state = hit.get("admin1") or ""
display = ", ".join(part for part in [city, state, country_code] if part)
return {
"city": city,
"state": state,
"country": country_code or "Unknown",
"country_code": country_code,
"display_name": display,
"offline_fallback": True,
}
except Exception as e:
logger.warning(f"Offline reverse geocode failed: {e}")
return {}
def _reverse_geocode(lat: float, lng: float) -> dict:
global _nominatim_last_call
url = (
f"https://nominatim.openstreetmap.org/reverse?"
f"lat={lat}&lon={lng}&format=json&zoom=10&addressdetails=1&accept-language=en"
)
headers = {"User-Agent": "ShadowBroker-OSINT/1.0 (live-risk-dashboard; contact@shadowbroker.app)"}
headers = {
"User-Agent": "ShadowBroker-OSINT/1.0 (live-risk-dashboard; contact@shadowbroker.app)"
}
for attempt in range(2):
# Enforce Nominatim's 1 req/sec policy
@@ -33,26 +58,32 @@ def _reverse_geocode(lat: float, lng: float) -> dict:
try:
# Use requests directly — fetch_with_curl raises on non-200 which breaks 429 handling
res = _requests.get(url, timeout=10, headers=headers)
res = _requests.get(url, timeout=4, headers=headers)
if res.status_code == 200:
data = res.json()
addr = data.get("address", {})
return {
"city": addr.get("city") or addr.get("town") or addr.get("village") or addr.get("county") or "",
"city": addr.get("city")
or addr.get("town")
or addr.get("village")
or addr.get("county")
or "",
"state": addr.get("state") or addr.get("region") or "",
"country": addr.get("country") or "",
"country_code": (addr.get("country_code") or "").upper(),
"display_name": data.get("display_name", ""),
}
elif res.status_code == 429:
logger.warning(f"Nominatim 429 rate-limited, retrying after 2s (attempt {attempt+1})")
time.sleep(2)
logger.warning(
f"Nominatim 429 rate-limited, retrying after 1s (attempt {attempt+1})"
)
time.sleep(1)
continue
else:
logger.warning(f"Nominatim returned {res.status_code}")
except (_requests.RequestException, ConnectionError, TimeoutError, OSError) as e:
logger.warning(f"Reverse geocode failed: {e}")
return {}
return _reverse_geocode_offline(lat, lng)
def _fetch_country_data(country_code: str) -> dict:
@@ -63,9 +94,12 @@ def _fetch_country_data(country_code: str) -> dict:
f"?fields=name,population,capital,languages,region,subregion,area,currencies,borders,flag"
)
try:
res = fetch_with_curl(url, timeout=10)
res = fetch_with_curl(url, timeout=5)
if res.status_code == 200:
return res.json()
data = res.json()
if isinstance(data, list):
return data[0] if data and isinstance(data[0], dict) else {}
return data if isinstance(data, dict) else {}
except (ConnectionError, TimeoutError, ValueError, KeyError, OSError) as e:
logger.warning(f"RestCountries failed for {country_code}: {e}")
return {}
@@ -87,7 +121,7 @@ def _fetch_wikidata_leader(country_name: str) -> dict:
"""
url = f"https://query.wikidata.org/sparql?query={quote(sparql)}&format=json"
try:
res = fetch_with_curl(url, timeout=15)
res = fetch_with_curl(url, timeout=6)
if res.status_code == 200:
results = res.json().get("results", {}).get("bindings", [])
if results:
@@ -113,7 +147,7 @@ def _fetch_local_wiki_summary(place_name: str, country_name: str = "") -> dict:
slug = quote(name.replace(" ", "_"))
url = f"https://en.wikipedia.org/api/rest_v1/page/summary/{slug}"
try:
res = fetch_with_curl(url, timeout=10)
res = fetch_with_curl(url, timeout=5)
if res.status_code == 200:
data = res.json()
if data.get("type") != "disambiguation":
@@ -122,7 +156,13 @@ def _fetch_local_wiki_summary(place_name: str, country_name: str = "") -> dict:
"extract": data.get("extract", ""),
"thumbnail": data.get("thumbnail", {}).get("source", ""),
}
except (ConnectionError, TimeoutError, ValueError, KeyError, OSError): # Intentional: optional enrichment
except (
ConnectionError,
TimeoutError,
ValueError,
KeyError,
OSError,
): # Intentional: optional enrichment
continue
return {}
@@ -148,33 +188,37 @@ def get_region_dossier(lat: float, lng: float) -> dict:
city_name = geo.get("city", "")
state_name = geo.get("state", "")
# Step 2: Parallel fetch with timeouts to prevent hanging
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
# Step 2: Parallel fetch with real timeouts that do not block on executor shutdown
pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)
try:
country_fut = pool.submit(_fetch_country_data, country_code)
leader_fut = pool.submit(_fetch_wikidata_leader, country_name)
local_fut = pool.submit(_fetch_local_wiki_summary, city_name or state_name, country_name)
# Also fetch country-level Wikipedia summary as fallback for local
local_fut = pool.submit(
_fetch_local_wiki_summary, city_name or state_name, country_name
)
country_wiki_fut = pool.submit(_fetch_local_wiki_summary, country_name, "")
try:
country_data = country_fut.result(timeout=12)
except Exception: # Intentional: optional enrichment
logger.warning("Country data fetch timed out or failed")
country_data = {}
try:
leader_data = leader_fut.result(timeout=12)
except Exception: # Intentional: optional enrichment
logger.warning("Leader data fetch timed out or failed")
leader_data = {"leader": "Unknown", "government_type": "Unknown"}
try:
local_data = local_fut.result(timeout=12)
except Exception: # Intentional: optional enrichment
logger.warning("Local wiki fetch timed out or failed")
local_data = {}
try:
country_wiki_data = country_wiki_fut.result(timeout=12)
except Exception: # Intentional: optional enrichment
country_wiki_data = {}
try:
country_data = country_fut.result(timeout=6)
except Exception: # Intentional: optional enrichment
logger.warning("Country data fetch timed out or failed")
country_data = {}
try:
leader_data = leader_fut.result(timeout=6)
except Exception: # Intentional: optional enrichment
logger.warning("Leader data fetch timed out or failed")
leader_data = {"leader": "Unknown", "government_type": "Unknown"}
try:
local_data = local_fut.result(timeout=5)
except Exception: # Intentional: optional enrichment
logger.warning("Local wiki fetch timed out or failed")
local_data = {}
try:
country_wiki_data = country_wiki_fut.result(timeout=5)
except Exception: # Intentional: optional enrichment
country_wiki_data = {}
finally:
pool.shutdown(wait=False, cancel_futures=True)
# If no local data but we have country wiki summary, use that
if not local_data.get("extract") and country_wiki_data.get("extract"):
@@ -203,7 +247,11 @@ def get_region_dossier(lat: float, lng: float) -> dict:
"leader": leader_data.get("leader", "Unknown"),
"government_type": leader_data.get("government_type", "Unknown"),
"population": country_data.get("population", 0),
"capital": (country_data.get("capital") or ["Unknown"])[0] if isinstance(country_data.get("capital"), list) else "Unknown",
"capital": (
(country_data.get("capital") or ["Unknown"])[0]
if isinstance(country_data.get("capital"), list)
else "Unknown"
),
"languages": lang_list,
"currencies": currency_list,
"region": country_data.get("region", ""),
backend/services/satnogs_fetcher.py
+112
View File
@@ -0,0 +1,112 @@
"""
SatNOGS ground station + observation fetcher.
Queries the SatNOGS Network API for online ground stations and recent
satellite observations. No API key required for read-only access.
"""
import logging
import requests
from cachetools import TTLCache, cached
logger = logging.getLogger(__name__)
_station_cache = TTLCache(maxsize=1, ttl=600) # 10-minute cache
_obs_cache = TTLCache(maxsize=1, ttl=300) # 5-minute cache
@cached(_station_cache)
def fetch_satnogs_stations() -> list[dict]:
"""Fetch online SatNOGS ground stations (status=2 = online)."""
try:
resp = requests.get(
"https://network.satnogs.org/api/stations/",
params={"format": "json", "status": 2},
timeout=20,
headers={"Accept": "application/json"},
)
resp.raise_for_status()
stations = []
for s in resp.json():
lat, lng = s.get("lat"), s.get("lng")
if lat is None or lng is None:
continue
try:
lat, lng = float(lat), float(lng)
except (ValueError, TypeError):
continue
if abs(lat) > 90 or abs(lng) > 180:
continue
antennas = s.get("antenna") or []
antenna_str = ", ".join(
a.get("antenna_type", "") for a in antennas if a.get("antenna_type")
)
stations.append(
{
"id": s.get("id"),
"name": (s.get("name") or "Unknown")[:120],
"lat": round(lat, 5),
"lng": round(lng, 5),
"altitude": s.get("altitude"),
"antenna": antenna_str[:200],
"observations": s.get("observations", 0),
"status": s.get("status"),
"last_seen": s.get("last_seen"),
}
)
logger.info(f"SatNOGS: fetched {len(stations)} online stations")
return stations
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
logger.error(f"SatNOGS stations error: {e}")
return []
@cached(_obs_cache)
def fetch_satnogs_observations() -> list[dict]:
"""Fetch recent good observations (first page, ~25 results)."""
try:
resp = requests.get(
"https://network.satnogs.org/api/observations/",
params={"format": "json", "status": "good"},
timeout=20,
headers={"Accept": "application/json"},
)
resp.raise_for_status()
obs = []
for o in resp.json():
lat = o.get("station_lat")
lng = o.get("station_lng")
if lat is None or lng is None:
continue
try:
lat, lng = float(lat), float(lng)
except (ValueError, TypeError):
continue
# Satellite name from TLE line 0, or fall back to NORAD ID
tle0 = (o.get("tle0") or "").strip()
sat_name = tle0 if tle0 else f"NORAD {o.get('norad_cat_id', '?')}"
obs.append(
{
"id": o.get("id"),
"satellite_name": sat_name[:80],
"norad_id": o.get("norad_cat_id"),
"station_name": (o.get("station_name") or "Unknown")[:80],
"lat": round(lat, 5),
"lng": round(lng, 5),
"start": o.get("start"),
"end": o.get("end"),
"frequency": o.get("transmitter_downlink_low"),
"mode": o.get("transmitter_mode"),
"waterfall": o.get("waterfall"),
"audio": o.get("archive_url") or o.get("payload"),
"status": o.get("vetted_status"),
}
)
logger.info(f"SatNOGS: fetched {len(obs)} recent observations")
return obs
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
logger.error(f"SatNOGS observations error: {e}")
return []
backend/services/sentinel_search.py
+62 -39
View File
@@ -1,6 +1,9 @@
"""
Sentinel-2 satellite imagery search via Microsoft Planetary Computer STAC API.
Free, keyless search for metadata + thumbnails. Used in the right-click dossier.
We use the raw STAC HTTP API with explicit timeouts so the right-click dossier
cannot hang behind a slow client library call.
"""
import logging
@@ -14,6 +17,32 @@ logger = logging.getLogger(__name__)
_sentinel_cache = TTLCache(maxsize=200, ttl=3600)
def _esri_imagery_fallback(lat: float, lng: float) -> dict:
lat_span = 0.18
lng_span = 0.24
bbox = f"{lng - lng_span},{lat - lat_span},{lng + lng_span},{lat + lat_span}"
fullres = (
"https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/"
f"export?bbox={bbox}&bboxSR=4326&imageSR=4326&size=1600,900&format=png32&f=image"
)
thumbnail = (
"https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/"
f"export?bbox={bbox}&bboxSR=4326&imageSR=4326&size=640,360&format=png32&f=image"
)
return {
"found": True,
"scene_id": None,
"datetime": None,
"cloud_cover": None,
"thumbnail_url": thumbnail,
"fullres_url": fullres,
"bbox": [lng - lng_span, lat - lat_span, lng + lng_span, lat + lat_span],
"platform": "Esri World Imagery",
"fallback": True,
"message": "Planetary Computer unavailable; using Esri World Imagery fallback",
}
def search_sentinel2_scene(lat: float, lng: float) -> dict:
"""Search for the latest Sentinel-2 L2A scene covering a point."""
cache_key = f"{round(lat, 2)}_{round(lng, 2)}"
@@ -21,62 +50,56 @@ def search_sentinel2_scene(lat: float, lng: float) -> dict:
return _sentinel_cache[cache_key]
try:
from pystac_client import Client
catalog = Client.open("https://planetarycomputer.microsoft.com/api/stac/v1")
end = datetime.utcnow()
start = end - timedelta(days=30)
search = catalog.search(
collections=["sentinel-2-l2a"],
intersects={"type": "Point", "coordinates": [lng, lat]},
datetime=f"{start.isoformat()}Z/{end.isoformat()}Z",
sortby=[{"field": "datetime", "direction": "desc"}],
max_items=3,
query={"eo:cloud_cover": {"lt": 30}},
search_payload = {
"collections": ["sentinel-2-l2a"],
"intersects": {"type": "Point", "coordinates": [lng, lat]},
"datetime": f"{start.isoformat()}Z/{end.isoformat()}Z",
"sortby": [{"field": "datetime", "direction": "desc"}],
"limit": 3,
"query": {"eo:cloud_cover": {"lt": 30}},
}
search_res = requests.post(
"https://planetarycomputer.microsoft.com/api/stac/v1/search",
json=search_payload,
timeout=8,
headers={"User-Agent": "ShadowBroker-OSINT/1.0 (live-risk-dashboard)"},
)
items = list(search.items())
if not items:
result = {"found": False, "message": "No clear scenes in last 30 days"}
search_res.raise_for_status()
data = search_res.json()
features = data.get("features", [])
if not features:
result = _esri_imagery_fallback(lat, lng)
_sentinel_cache[cache_key] = result
return result
item = items[0]
# Try to sign item first for Azure blob URLs
try:
import planetary_computer
item = planetary_computer.sign_item(item)
except ImportError:
pass # planetary_computer not installed, try unsigned URLs
except (ConnectionError, TimeoutError, ValueError) as e:
logger.warning(f"Sentinel-2 signing failed: {e}")
# Get the rendered_preview (full-res PNG) and thumbnail separately
rendered = item.assets.get("rendered_preview")
thumbnail = item.assets.get("thumbnail")
item = features[0]
assets = item.get("assets", {}) or {}
rendered = assets.get("rendered_preview") or {}
thumbnail = assets.get("thumbnail") or {}
# Full-res image URL — what opens when user clicks
fullres_url = rendered.href if rendered else (thumbnail.href if thumbnail else None)
fullres_url = rendered.get("href") or thumbnail.get("href")
# Thumbnail URL — what shows in the popup card
thumb_url = thumbnail.href if thumbnail else (rendered.href if rendered else None)
thumb_url = thumbnail.get("href") or rendered.get("href")
result = {
"found": True,
"scene_id": item.id,
"datetime": item.datetime.isoformat() if item.datetime else None,
"cloud_cover": item.properties.get("eo:cloud_cover"),
"scene_id": item.get("id"),
"datetime": item.get("properties", {}).get("datetime"),
"cloud_cover": item.get("properties", {}).get("eo:cloud_cover"),
"thumbnail_url": thumb_url,
"fullres_url": fullres_url,
"bbox": list(item.bbox) if item.bbox else None,
"platform": item.properties.get("platform", "Sentinel-2"),
"bbox": list(item.get("bbox", [])) if item.get("bbox") else None,
"platform": item.get("properties", {}).get("platform", "Sentinel-2"),
}
_sentinel_cache[cache_key] = result
return result
except ImportError:
logger.warning("pystac-client not installed — Sentinel-2 search unavailable")
return {"found": False, "error": "pystac-client not installed"}
except (requests.RequestException, ConnectionError, TimeoutError, ValueError) as e:
logger.error(f"Sentinel-2 search failed for ({lat}, {lng}): {e}")
return {"found": False, "error": str(e)}
result = _esri_imagery_fallback(lat, lng)
result["error"] = str(e)
_sentinel_cache[cache_key] = result
return result
backend/services/shodan_connector.py
+316
View File
@@ -0,0 +1,316 @@
"""
Local-only Shodan connector.
This module intentionally does NOT merge Shodan results into the dashboard's
canonical live-data store. It exposes manual, operator-triggered lookups that
can be rendered locally in the UI as a temporary investigative overlay.
"""
from __future__ import annotations
import logging
import os
import threading
import time
from typing import Any
import requests
from cachetools import TTLCache
logger = logging.getLogger(__name__)
_SHODAN_BASE = "https://api.shodan.io"
_USER_AGENT = "ShadowBroker/0.9.6 local Shodan connector"
_REQUEST_TIMEOUT = 15
_MIN_INTERVAL_SECONDS = 1.05 # Shodan docs say API plans are rate limited to ~1 req/sec.
_DEFAULT_SEARCH_PAGES = 1
_MAX_SEARCH_PAGES = 2
_search_cache: TTLCache[str, dict[str, Any]] = TTLCache(maxsize=24, ttl=90)
_count_cache: TTLCache[str, dict[str, Any]] = TTLCache(maxsize=24, ttl=120)
_host_cache: TTLCache[str, dict[str, Any]] = TTLCache(maxsize=32, ttl=300)
_request_lock = threading.Lock()
_last_request_at = 0.0
class ShodanConnectorError(Exception):
def __init__(self, detail: str, status_code: int = 400):
super().__init__(detail)
self.detail = detail
self.status_code = status_code
def _get_api_key() -> str:
api_key = os.environ.get("SHODAN_API_KEY", "").strip()
if not api_key:
raise ShodanConnectorError(
"Shodan API key not configured. Add SHODAN_API_KEY in Settings > API Keys.",
status_code=428,
)
return api_key
def _clean_query(value: str | None) -> str:
query = (value or "").strip()
if not query:
raise ShodanConnectorError("Shodan query cannot be empty.", status_code=400)
if "\n" in query or "\r" in query:
raise ShodanConnectorError("Shodan query must be a single line.", status_code=400)
return query
def _cache_key(prefix: str, payload: dict[str, Any]) -> str:
normalized = tuple(sorted((str(k), str(v)) for k, v in payload.items()))
return f"{prefix}:{normalized!r}"
def _normalize_string_list(values: Any, limit: int = 10) -> list[str]:
if not isinstance(values, list):
return []
cleaned: list[str] = []
for item in values:
text = str(item).strip()
if text:
cleaned.append(text)
if len(cleaned) >= limit:
break
return cleaned
def _location_label(location: dict[str, Any]) -> str | None:
parts = [
str(location.get("city") or "").strip(),
str(location.get("region_code") or "").strip(),
str(location.get("country_code") or "").strip(),
]
label = ", ".join([p for p in parts if p])
return label or None
def _normalize_match(match: dict[str, Any]) -> dict[str, Any]:
location = match.get("location") or {}
lat = location.get("latitude")
lng = location.get("longitude")
port = match.get("port")
ip_str = str(match.get("ip_str") or match.get("ip") or "").strip()
host_id = f"shodan-{ip_str or 'unknown'}-{port or 'na'}"
vulns = match.get("vulns") or []
if isinstance(vulns, dict):
vuln_list = _normalize_string_list(list(vulns.keys()), limit=12)
else:
vuln_list = _normalize_string_list(vulns, limit=12)
return {
"id": host_id,
"ip": ip_str or "UNKNOWN",
"port": port,
"transport": match.get("transport"),
"timestamp": match.get("timestamp"),
"lat": lat if isinstance(lat, (int, float)) else None,
"lng": lng if isinstance(lng, (int, float)) else None,
"city": location.get("city"),
"region_code": location.get("region_code"),
"country_code": location.get("country_code"),
"country_name": location.get("country_name"),
"location_label": _location_label(location),
"asn": match.get("asn"),
"org": match.get("org"),
"isp": match.get("isp"),
"product": match.get("product"),
"os": match.get("os"),
"hostnames": _normalize_string_list(match.get("hostnames")),
"domains": _normalize_string_list(match.get("domains")),
"tags": _normalize_string_list(match.get("tags")),
"vulns": vuln_list,
"data_snippet": str(match.get("data") or "").strip()[:280] or None,
"attribution": "Data from Shodan",
}
def _normalize_services(items: Any) -> list[dict[str, Any]]:
if not isinstance(items, list):
return []
services: list[dict[str, Any]] = []
for item in items[:30]:
if not isinstance(item, dict):
continue
services.append(
{
"port": item.get("port"),
"transport": item.get("transport"),
"product": item.get("product"),
"timestamp": item.get("timestamp"),
"tags": _normalize_string_list(item.get("tags"), limit=8),
"banner_excerpt": str(item.get("data") or "").strip()[:320] or None,
}
)
return services
def _normalize_facets(raw_facets: Any) -> dict[str, list[dict[str, Any]]]:
if not isinstance(raw_facets, dict):
return {}
normalized: dict[str, list[dict[str, Any]]] = {}
for key, bucket_list in raw_facets.items():
if not isinstance(bucket_list, list):
continue
normalized[str(key)] = [
{"value": str(bucket.get("value") or ""), "count": int(bucket.get("count") or 0)}
for bucket in bucket_list[:12]
if isinstance(bucket, dict)
]
return normalized
def _request(path: str, *, params: dict[str, Any], cache: TTLCache[str, dict[str, Any]] | None = None) -> dict[str, Any]:
api_key = _get_api_key()
payload = {**params, "key": api_key}
cache_key = _cache_key(path, {k: v for k, v in payload.items() if k != "key"})
if cache is not None and cache_key in cache:
return cache[cache_key]
global _last_request_at
with _request_lock:
elapsed = time.monotonic() - _last_request_at
if elapsed < _MIN_INTERVAL_SECONDS:
time.sleep(_MIN_INTERVAL_SECONDS - elapsed)
try:
response = requests.get(
f"{_SHODAN_BASE}{path}",
params=payload,
timeout=_REQUEST_TIMEOUT,
headers={"User-Agent": _USER_AGENT, "Accept": "application/json"},
)
finally:
_last_request_at = time.monotonic()
if response.status_code == 401:
raise ShodanConnectorError("Shodan rejected the API key. Check SHODAN_API_KEY.", 401)
if response.status_code == 402:
raise ShodanConnectorError(
"Shodan returned payment/plan required. This feature needs a paid Shodan API plan.",
402,
)
if response.status_code == 429:
raise ShodanConnectorError(
"Shodan rate limit reached. Slow down queries and try again shortly.",
429,
)
if response.status_code >= 400:
detail = response.text.strip()[:240] or "Unexpected Shodan API error."
raise ShodanConnectorError(f"Shodan request failed: {detail}", response.status_code)
try:
parsed = response.json()
except ValueError as exc:
raise ShodanConnectorError(f"Shodan returned invalid JSON: {exc}", 502) from exc
if cache is not None:
cache[cache_key] = parsed
return parsed
def get_shodan_connector_status() -> dict[str, Any]:
has_key = bool(os.environ.get("SHODAN_API_KEY", "").strip())
return {
"ok": True,
"configured": has_key,
"source": "Shodan",
"mode": "operator-supplied local overlay",
"paid_api": True,
"manual_only": True,
"background_polling": False,
"local_only": True,
"attribution": "Data from Shodan",
"warning": (
"Shodan is a paid API. Searches use your local SHODAN_API_KEY, results stay local to "
"your ShadowBroker session by default, and any downstream use is your responsibility."
),
"limits": {
"default_pages_per_search": _DEFAULT_SEARCH_PAGES,
"max_pages_per_search": _MAX_SEARCH_PAGES,
"cooldown_seconds": _MIN_INTERVAL_SECONDS,
},
}
def search_shodan(query: str, page: int = 1, facets: list[str] | None = None) -> dict[str, Any]:
cleaned_query = _clean_query(query)
safe_page = max(1, min(int(page or 1), _MAX_SEARCH_PAGES))
facet_list = [str(f).strip() for f in (facets or []) if str(f).strip()][:6]
params: dict[str, Any] = {"query": cleaned_query, "page": safe_page}
if facet_list:
params["facets"] = ",".join(facet_list)
raw = _request("/shodan/host/search", params=params, cache=_search_cache)
matches = [_normalize_match(match) for match in raw.get("matches") or [] if isinstance(match, dict)]
return {
"ok": True,
"source": "Shodan",
"attribution": "Data from Shodan",
"query": cleaned_query,
"page": safe_page,
"total": int(raw.get("total") or 0),
"matches": matches,
"facets": _normalize_facets(raw.get("facets")),
"note": "Operator-triggered Shodan results. Not part of ShadowBroker core feeds.",
}
def count_shodan(query: str, facets: list[str] | None = None) -> dict[str, Any]:
cleaned_query = _clean_query(query)
facet_list = [str(f).strip() for f in (facets or []) if str(f).strip()][:8]
params: dict[str, Any] = {"query": cleaned_query}
if facet_list:
params["facets"] = ",".join(facet_list)
raw = _request("/shodan/host/count", params=params, cache=_count_cache)
return {
"ok": True,
"source": "Shodan",
"attribution": "Data from Shodan",
"query": cleaned_query,
"total": int(raw.get("total") or 0),
"facets": _normalize_facets(raw.get("facets")),
"note": "Count/facets query only. No persistent ShadowBroker storage.",
}
def lookup_shodan_host(ip: str, history: bool = False) -> dict[str, Any]:
clean_ip = str(ip or "").strip()
if not clean_ip:
raise ShodanConnectorError("Host lookup requires an IP address.", 400)
raw = _request(
f"/shodan/host/{clean_ip}",
params={"history": "true" if history else "false"},
cache=_host_cache,
)
location = raw.get("location") or {}
host = {
"id": f"shodan-{clean_ip}-host",
"ip": str(raw.get("ip_str") or clean_ip),
"lat": location.get("latitude") if isinstance(location.get("latitude"), (int, float)) else None,
"lng": location.get("longitude") if isinstance(location.get("longitude"), (int, float)) else None,
"city": location.get("city"),
"region_code": location.get("region_code"),
"country_code": location.get("country_code"),
"country_name": location.get("country_name"),
"location_label": _location_label(location),
"asn": raw.get("asn"),
"org": raw.get("org"),
"isp": raw.get("isp"),
"os": raw.get("os"),
"hostnames": _normalize_string_list(raw.get("hostnames")),
"domains": _normalize_string_list(raw.get("domains")),
"tags": _normalize_string_list(raw.get("tags")),
"ports": [int(p) for p in (raw.get("ports") or []) if isinstance(p, int)],
"services": _normalize_services(raw.get("data")),
"vulns": _normalize_string_list(list((raw.get("vulns") or {}).keys()) if isinstance(raw.get("vulns"), dict) else raw.get("vulns"), limit=20),
"attribution": "Data from Shodan",
}
return {
"ok": True,
"source": "Shodan",
"attribution": "Data from Shodan",
"host": host,
"history": bool(history),
"note": "Operator-triggered Shodan host lookup. Not merged into ShadowBroker datasets.",
}
backend/services/sigint_bridge.py
+1133
View File
File diff suppressed because it is too large Load Diff
backend/services/thermal_sentinel.py
+265
View File
@@ -0,0 +1,265 @@
"""Thermal Sentinel — SWIR spectral anomaly detection via Sentinel-2 L2A.
Queries Microsoft Planetary Computer for Sentinel-2 scenes near a given
coordinate and checks SWIR bands (B11 @ 1610nm, B12 @ 2190nm) for thermal
anomalies that could indicate kinetic events (explosions, fires, strikes).
Thermal index: (B12 - B11) / (B12 + B11) — values > 0.1 suggest heat anomaly.
Falls back to metadata-only analysis (cloud cover, scene age) when rasterio
is not available, and cross-references with FIRMS fire data for corroboration.
"""
import logging
import requests
from datetime import datetime, timedelta
from cachetools import TTLCache
logger = logging.getLogger(__name__)
# Cache by rounded lat/lon (0.05° grid ~= 5km), TTL 30 min
_thermal_cache = TTLCache(maxsize=100, ttl=1800)
def search_thermal_anomaly(
lat: float, lng: float, radius_km: float = 10, days_back: int = 5
) -> dict:
"""Search for thermal anomalies near a coordinate using Sentinel-2 SWIR bands.
Args:
lat, lng: Target coordinates
radius_km: Search radius in km (default 10)
days_back: How many days back to search (default 5)
Returns:
dict with: verified (bool), confidence (float 0-1), scenes_checked (int),
thermal_index (float or None), latest_scene (str), firms_corroboration (bool)
"""
cache_key = f"{round(lat, 2)}_{round(lng, 2)}_{radius_km}_{days_back}"
if cache_key in _thermal_cache:
return _thermal_cache[cache_key]
result = {
"verified": False,
"confidence": 0.0,
"scenes_checked": 0,
"thermal_index": None,
"latest_scene": None,
"latest_scene_date": None,
"cloud_cover": None,
"firms_corroboration": False,
"method": "metadata", # or "swir_analysis" if rasterio available
}
try:
# Step 1: STAC search for Sentinel-2 scenes
scenes = _search_scenes(lat, lng, radius_km, days_back)
result["scenes_checked"] = len(scenes)
if not scenes:
result["confidence"] = 0.0
_thermal_cache[cache_key] = result
return result
best_scene = scenes[0]
result["latest_scene"] = best_scene.get("id")
result["latest_scene_date"] = best_scene.get("datetime")
result["cloud_cover"] = best_scene.get("cloud_cover")
# Step 2: Try SWIR band analysis if rasterio is available
swir_result = _analyze_swir_bands(best_scene, lat, lng)
if swir_result is not None:
result["thermal_index"] = swir_result["thermal_index"]
result["method"] = "swir_analysis"
if swir_result["thermal_index"] > 0.1:
result["verified"] = True
result["confidence"] = min(0.9, swir_result["thermal_index"] * 3)
elif swir_result["thermal_index"] > 0.05:
result["confidence"] = 0.3
else:
# Fallback: metadata-only analysis
# Recent scene + low cloud cover = higher confidence that we CAN verify
scene_age_days = _scene_age_days(best_scene.get("datetime"))
if scene_age_days is not None and scene_age_days <= 2:
result["confidence"] = 0.2 # recent scene, but no SWIR analysis
else:
result["confidence"] = 0.1
# Step 3: Cross-reference with FIRMS fire data
firms_hit = _check_firms_corroboration(lat, lng, radius_km)
if firms_hit:
result["firms_corroboration"] = True
result["confidence"] = min(1.0, result["confidence"] + 0.4)
if not result["verified"]:
result["verified"] = True # FIRMS confirms thermal activity
_thermal_cache[cache_key] = result
return result
except ImportError:
logger.warning("pystac-client not installed — Thermal Sentinel unavailable")
result["confidence"] = 0.0
return result
except Exception as e:
logger.error(f"Thermal Sentinel error for ({lat}, {lng}): {e}")
result["confidence"] = 0.0
return result
def _search_scenes(lat: float, lng: float, radius_km: float, days_back: int) -> list[dict]:
"""Search Planetary Computer STAC for Sentinel-2 scenes."""
from pystac_client import Client
catalog = Client.open("https://planetarycomputer.microsoft.com/api/stac/v1")
end = datetime.utcnow()
start = end - timedelta(days=days_back)
# Convert radius_km to rough bbox
dlat = radius_km / 111.0
dlng = radius_km / (
111.0
* max(0.1, abs(lat) < 89 and __import__("math").cos(__import__("math").radians(lat)) or 0.1)
)
bbox = [lng - dlng, lat - dlat, lng + dlng, lat + dlat]
search = catalog.search(
collections=["sentinel-2-l2a"],
bbox=bbox,
datetime=f"{start.isoformat()}Z/{end.isoformat()}Z",
sortby=[{"field": "datetime", "direction": "desc"}],
max_items=5,
query={"eo:cloud_cover": {"lt": 50}},
)
scenes = []
for item in search.items():
scenes.append(
{
"id": item.id,
"datetime": item.datetime.isoformat() if item.datetime else None,
"cloud_cover": item.properties.get("eo:cloud_cover"),
"b11_href": item.assets.get("B11", {}).href if "B11" in item.assets else None,
"b12_href": item.assets.get("B12", {}).href if "B12" in item.assets else None,
"item": item,
}
)
return scenes
def _analyze_swir_bands(scene: dict, lat: float, lng: float) -> dict | None:
"""Analyze SWIR bands B11 and B12 for thermal anomalies.
Returns dict with thermal_index or None if rasterio unavailable.
"""
try:
import rasterio
from rasterio.windows import from_bounds
except ImportError:
logger.debug("rasterio not installed — falling back to metadata analysis")
return None
b11_href = scene.get("b11_href")
b12_href = scene.get("b12_href")
if not b11_href or not b12_href:
return None
# Sign URLs for Azure blob access
item = scene.get("item")
if item:
try:
import planetary_computer
item = planetary_computer.sign_item(item)
b11_href = item.assets["B11"].href
b12_href = item.assets["B12"].href
except (ImportError, KeyError, Exception) as e:
logger.debug(f"SWIR signing failed: {e}")
return None
try:
# Read a small window around the target coordinate
# Sentinel-2 SWIR bands are 20m resolution
buffer_deg = 0.005 # ~500m window
with rasterio.open(b11_href) as b11_ds:
window = from_bounds(
lng - buffer_deg,
lat - buffer_deg,
lng + buffer_deg,
lat + buffer_deg,
b11_ds.transform,
)
b11_data = b11_ds.read(1, window=window).astype(float)
with rasterio.open(b12_href) as b12_ds:
window = from_bounds(
lng - buffer_deg,
lat - buffer_deg,
lng + buffer_deg,
lat + buffer_deg,
b12_ds.transform,
)
b12_data = b12_ds.read(1, window=window).astype(float)
if b11_data.size == 0 or b12_data.size == 0:
return None
# Compute thermal index: (B12 - B11) / (B12 + B11)
denom = b12_data + b11_data
# Avoid division by zero
valid = denom > 0
if not valid.any():
return None
thermal_index = (b12_data[valid] - b11_data[valid]) / denom[valid]
max_ti = float(thermal_index.max())
return {"thermal_index": round(max_ti, 4)}
except Exception as e:
logger.warning(f"SWIR band read failed: {e}")
return None
def _scene_age_days(dt_str: str | None) -> float | None:
"""Calculate age of a scene in days."""
if not dt_str:
return None
try:
scene_dt = datetime.fromisoformat(dt_str.replace("Z", "+00:00"))
now = datetime.now(scene_dt.tzinfo) if scene_dt.tzinfo else datetime.utcnow()
return (now - scene_dt).total_seconds() / 86400
except (ValueError, TypeError):
return None
def _check_firms_corroboration(lat: float, lng: float, radius_km: float) -> bool:
"""Check if FIRMS fire data corroborates thermal activity near the coordinate."""
from services.fetchers._store import latest_data, _data_lock
with _data_lock:
fires = list(latest_data.get("firms_fires", []))
if not fires:
return False
# Simple distance check (approximate, using equirectangular projection)
import math
threshold_deg = radius_km / 111.0
for fire in fires:
try:
flat = fire.get("lat") or fire.get("latitude")
flng = fire.get("lng") or fire.get("longitude")
if flat is None or flng is None:
continue
dlat = abs(float(flat) - lat)
dlng = abs(float(flng) - lng) * math.cos(math.radians(lat))
if math.sqrt(dlat**2 + dlng**2) <= threshold_deg:
return True
except (ValueError, TypeError):
continue
return False
backend/services/tinygs_fetcher.py
+385
View File
@@ -0,0 +1,385 @@
"""
TinyGS LoRa satellite tracker — SGP4 orbit propagation + TinyGS telemetry.
Primary position source: CelesTrak TLEs propagated via SGP4 (always available).
Secondary validation: TinyGS API confirms satellite is actively transmitting LoRa
and provides modulation/frequency/status metadata.
CelesTrak Fair Use: TLEs fetched at most once per 24 hours, cached to disk.
TinyGS: polled every 5 minutes (their server has limited capacity).
"""
import json
import logging
import math
import time
from datetime import datetime
from pathlib import Path
import requests
from sgp4.api import Satrec, WGS72, jday
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# CelesTrak TLE cache (24-hour refresh, disk-backed)
# ---------------------------------------------------------------------------
_CELESTRAK_FETCH_INTERVAL = 86400 # 24 hours
_TLE_CACHE_PATH = Path(__file__).parent.parent / "data" / "tinygs_tle_cache.json"
_tle_cache: dict = {"data": None, "last_fetch": 0.0}
# TinyGS API telemetry cache
_TINYGS_FETCH_INTERVAL = 300 # 5 minutes
_tinygs_telemetry: dict[str, dict] = {} # name_key → {modulation, frequency, status}
_tinygs_last_fetch: float = 0.0
_tinygs_known_names: set[str] = set() # names seen from TinyGS API
# Final result cache
_last_result: list[dict] = []
# CelesTrak GP groups containing LoRa / amateur cubesats
_CELESTRAK_GROUPS = ["amateur", "cubesat"]
_CELESTRAK_BASE = "https://celestrak.org/NORAD/elements/gp.php"
def _gmst(jd_ut1: float) -> float:
"""Greenwich Mean Sidereal Time in radians from Julian Date."""
t = (jd_ut1 - 2451545.0) / 36525.0
gmst_sec = (
67310.54841
+ (876600.0 * 3600 + 8640184.812866) * t
+ 0.093104 * t * t
- 6.2e-6 * t * t * t
)
return (gmst_sec % 86400) / 86400.0 * 2 * math.pi
# ---------------------------------------------------------------------------
# CelesTrak TLE fetch + disk cache
# ---------------------------------------------------------------------------
def _load_tle_cache() -> list[dict] | None:
"""Load TLE data from disk cache."""
try:
if _TLE_CACHE_PATH.exists():
import os
age = time.time() - os.path.getmtime(str(_TLE_CACHE_PATH))
if age < _CELESTRAK_FETCH_INTERVAL * 2: # accept up to 48h old cache
data = json.loads(_TLE_CACHE_PATH.read_text(encoding="utf-8"))
if isinstance(data, list) and len(data) > 0:
return data
except (IOError, json.JSONDecodeError, ValueError) as e:
logger.warning("TinyGS TLE: disk cache load failed: %s", e)
return None
def _save_tle_cache(data: list[dict]) -> None:
"""Save TLE data to disk cache."""
try:
_TLE_CACHE_PATH.parent.mkdir(parents=True, exist_ok=True)
_TLE_CACHE_PATH.write_text(
json.dumps(data, ensure_ascii=False), encoding="utf-8"
)
except (IOError, OSError) as e:
logger.warning("TinyGS TLE: disk cache save failed: %s", e)
def _fetch_celestrak_tles() -> list[dict]:
"""Fetch GP data from CelesTrak for amateur + cubesat groups."""
global _tle_cache
now = time.time()
# Return memory cache if fresh
if _tle_cache["data"] and now - _tle_cache["last_fetch"] < _CELESTRAK_FETCH_INTERVAL:
return _tle_cache["data"]
# Try disk cache first
if not _tle_cache["data"]:
disk = _load_tle_cache()
if disk:
_tle_cache["data"] = disk
_tle_cache["last_fetch"] = now - _CELESTRAK_FETCH_INTERVAL + 3600 # re-check in 1h
logger.info("TinyGS TLE: loaded %d elements from disk cache", len(disk))
# Fetch fresh from CelesTrak
all_sats: dict[int, dict] = {} # keyed by NORAD_CAT_ID to deduplicate
for group in _CELESTRAK_GROUPS:
try:
resp = requests.get(
_CELESTRAK_BASE,
params={"GROUP": group, "FORMAT": "json"},
timeout=20,
headers={
"User-Agent": "ShadowBroker-OSINT/1.0 (CelesTrak fair-use)",
"Accept": "application/json",
},
)
resp.raise_for_status()
for s in resp.json():
norad_id = s.get("NORAD_CAT_ID")
if norad_id:
all_sats[norad_id] = s
logger.info("TinyGS TLE: fetched %s group (%d sats)", group, len(resp.json()))
except (requests.RequestException, ValueError, KeyError) as e:
logger.warning("TinyGS TLE: CelesTrak %s fetch failed: %s", group, e)
if all_sats:
result = list(all_sats.values())
_tle_cache["data"] = result
_tle_cache["last_fetch"] = now
_save_tle_cache(result)
logger.info("TinyGS TLE: cached %d total orbital elements", len(result))
return result
# Fall back to whatever we have
return _tle_cache["data"] or []
# ---------------------------------------------------------------------------
# SGP4 propagation
# ---------------------------------------------------------------------------
def _propagate_all(gp_data: list[dict]) -> dict[int, dict]:
"""Propagate all satellites to current time via SGP4.
Returns dict keyed by NORAD_CAT_ID with position/velocity data.
"""
now = datetime.utcnow()
jd, fr = jday(
now.year, now.month, now.day,
now.hour, now.minute,
now.second + now.microsecond / 1e6,
)
results: dict[int, dict] = {}
for s in gp_data:
try:
norad_id = s.get("NORAD_CAT_ID", 0)
mean_motion = s.get("MEAN_MOTION")
ecc = s.get("ECCENTRICITY")
incl = s.get("INCLINATION")
raan = s.get("RA_OF_ASC_NODE")
argp = s.get("ARG_OF_PERICENTER")
ma = s.get("MEAN_ANOMALY")
bstar = s.get("BSTAR", 0)
epoch_str = s.get("EPOCH")
obj_name = s.get("OBJECT_NAME", "")
if mean_motion is None or ecc is None or incl is None or not epoch_str:
continue
epoch_dt = datetime.strptime(epoch_str[:19], "%Y-%m-%dT%H:%M:%S")
epoch_jd, epoch_fr = jday(
epoch_dt.year, epoch_dt.month, epoch_dt.day,
epoch_dt.hour, epoch_dt.minute, epoch_dt.second,
)
sat_obj = Satrec()
sat_obj.sgp4init(
WGS72, "i", norad_id,
(epoch_jd + epoch_fr) - 2433281.5,
bstar, 0.0, 0.0,
ecc,
math.radians(argp),
math.radians(incl),
math.radians(ma),
mean_motion * 2 * math.pi / 1440.0,
math.radians(raan),
)
e, r, v = sat_obj.sgp4(jd, fr)
if e != 0:
continue
x, y, z = r
gmst = _gmst(jd + fr)
lng_rad = math.atan2(y, x) - gmst
lat_rad = math.atan2(z, math.sqrt(x * x + y * y))
alt_km = math.sqrt(x * x + y * y + z * z) - 6371.0
lat = math.degrees(lat_rad)
lng_deg = math.degrees(lng_rad) % 360
lng = lng_deg - 360 if lng_deg > 180 else lng_deg
# Ground-relative velocity for heading/speed
vx, vy, vz = v
omega_e = 7.2921159e-5
vx_g = vx + omega_e * y
vy_g = vy - omega_e * x
cos_lat = math.cos(lat_rad)
sin_lat = math.sin(lat_rad)
cos_lng = math.cos(lng_rad + gmst)
sin_lng = math.sin(lng_rad + gmst)
v_east = -sin_lng * vx_g + cos_lng * vy_g
v_north = -sin_lat * cos_lng * vx_g - sin_lat * sin_lng * vy_g + cos_lat * vz
ground_speed_kms = math.sqrt(v_east**2 + v_north**2)
speed_knots = ground_speed_kms * 1943.84
heading = math.degrees(math.atan2(v_east, v_north)) % 360
results[norad_id] = {
"name": obj_name,
"lat": round(lat, 4),
"lng": round(lng, 4),
"alt_km": round(alt_km, 1),
"heading": round(heading, 1),
"speed_knots": round(speed_knots, 0),
"norad_id": norad_id,
}
except (ValueError, TypeError, KeyError, OverflowError):
continue
return results
# ---------------------------------------------------------------------------
# TinyGS API telemetry fetch
# ---------------------------------------------------------------------------
def _name_key(name: str) -> str:
"""Normalise a satellite name for fuzzy matching."""
return name.upper().replace("-", "").replace("_", "").replace(" ", "")
def _fetch_tinygs_telemetry() -> None:
"""Fetch active satellite list from TinyGS for telemetry metadata."""
global _tinygs_last_fetch, _tinygs_telemetry, _tinygs_known_names
now = time.time()
if now - _tinygs_last_fetch < _TINYGS_FETCH_INTERVAL:
return
try:
resp = requests.get(
"https://api.tinygs.com/v1/satellitesWorldmap",
timeout=15,
headers={
"Accept": "application/json",
"User-Agent": "ShadowBroker-OSINT/1.0",
},
)
resp.raise_for_status()
new_telemetry: dict[str, dict] = {}
names: set[str] = set()
for s in resp.json():
display_name = (s.get("displayName") or s.get("name") or "")[:80]
if not display_name:
continue
key = _name_key(display_name)
names.add(key)
tags = s.get("tags") or {}
new_telemetry[key] = {
"display_name": display_name,
"status": s.get("status", ""),
"modulation": ", ".join(tags.get("modulation", [])),
"frequency": ", ".join(str(f) for f in tags.get("frequency", [])),
}
_tinygs_telemetry = new_telemetry
_tinygs_known_names = names
_tinygs_last_fetch = now
logger.info("TinyGS telemetry: fetched %d active satellites", len(new_telemetry))
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
logger.warning("TinyGS telemetry fetch failed (SGP4 still active): %s", e)
# Keep existing telemetry — don't clear on failure
# ---------------------------------------------------------------------------
# Merge SGP4 positions + TinyGS telemetry
# ---------------------------------------------------------------------------
def _match_name(celestrak_name: str) -> dict | None:
"""Try to match a CelesTrak object name to TinyGS telemetry."""
key = _name_key(celestrak_name)
# Exact match
if key in _tinygs_telemetry:
return _tinygs_telemetry[key]
# Substring match — CelesTrak name contains TinyGS name or vice versa
for tgs_key, tgs_data in _tinygs_telemetry.items():
if tgs_key in key or key in tgs_key:
return tgs_data
return None
def fetch_tinygs_satellites() -> list[dict]:
"""Fetch LoRa satellite positions via SGP4 + TinyGS telemetry merge.
1. Propagate cached CelesTrak TLEs via SGP4 (instant, no network needed)
2. Attempt TinyGS API for telemetry (modulation, frequency, status)
3. Merge: SGP4 provides position, TinyGS provides metadata
4. Filter to only satellites known to TinyGS (if we have TinyGS data)
"""
global _last_result
# Step 1: Get TLE data (from cache or CelesTrak)
gp_data = _fetch_celestrak_tles()
if not gp_data:
logger.warning("TinyGS: no TLE data available")
return _last_result or []
# Step 2: Try to fetch TinyGS telemetry (non-blocking, uses cache)
_fetch_tinygs_telemetry()
# Step 3: Propagate all satellites via SGP4
propagated = _propagate_all(gp_data)
if not propagated:
logger.warning("TinyGS: SGP4 propagation returned no results")
return _last_result or []
# Step 4: Merge and filter
sats: list[dict] = []
have_tinygs = bool(_tinygs_known_names)
for norad_id, pos in propagated.items():
celestrak_name = pos["name"]
telemetry = _match_name(celestrak_name)
# If we have TinyGS data, only show satellites that TinyGS knows about
# (filters out non-LoRa amateur/cubesats from the CelesTrak groups)
if have_tinygs and telemetry is None:
continue
entry = {
"name": telemetry["display_name"] if telemetry else celestrak_name,
"lat": pos["lat"],
"lng": pos["lng"],
"heading": pos["heading"],
"speed_knots": pos["speed_knots"],
"alt_km": pos["alt_km"],
"status": telemetry.get("status", "") if telemetry else "",
"modulation": telemetry.get("modulation", "") if telemetry else "",
"frequency": telemetry.get("frequency", "") if telemetry else "",
"sgp4_propagated": True,
"tinygs_confirmed": telemetry is not None,
}
sats.append(entry)
# If we have no TinyGS data at all (API never responded), show all propagated
# sats from the amateur group only (smaller, more relevant set)
if not have_tinygs and not sats:
for norad_id, pos in propagated.items():
sats.append({
"name": pos["name"],
"lat": pos["lat"],
"lng": pos["lng"],
"heading": pos["heading"],
"speed_knots": pos["speed_knots"],
"alt_km": pos["alt_km"],
"status": "",
"modulation": "",
"frequency": "",
"sgp4_propagated": True,
"tinygs_confirmed": False,
})
_last_result = sats
logger.info(
"TinyGS: %d satellites (SGP4 propagated, %d TinyGS confirmed)",
len(sats),
sum(1 for s in sats if s.get("tinygs_confirmed")),
)
return sats
backend/services/unusual_whales_connector.py
+316
View File
@@ -0,0 +1,316 @@
"""
Finnhub API connector.
Provides defense stock quotes, congressional trading data, and insider
transactions — all from Finnhub's free tier (60 calls/min).
File kept at this path for git history; the module is referenced as
services.unusual_whales_connector in main.py imports but the public
surface is entirely Finnhub now.
"""
from __future__ import annotations
import logging
import os
import threading
import time
from typing import Any
import math
import requests
from cachetools import TTLCache
logger = logging.getLogger(__name__)
_FINNHUB_BASE = "https://finnhub.io/api/v1"
_USER_AGENT = "ShadowBroker/0.9.6 Finnhub connector"
_REQUEST_TIMEOUT = 12
_MIN_INTERVAL_SECONDS = 0.35 # Stay well under 60 calls/min
# Tickers we poll for congress trades & insider activity
WATCHED_TICKERS = [
"NVDA", "AAPL", "TSLA", "MSFT", "GOOGL", "AMZN", "META",
"RTX", "LMT", "NOC", "GD", "BA", "PLTR",
]
# Defense + oil tickers for quotes (replaces yfinance)
QUOTE_TICKERS = [
("RTX", "RTX"), ("LMT", "LMT"), ("NOC", "NOC"),
("GD", "GD"), ("BA", "BA"), ("PLTR", "PLTR"),
]
CRYPTO_TICKERS = [
("BTC", "BINANCE:BTCUSDT"),
("ETH", "BINANCE:ETHUSDT"),
]
_quote_cache: TTLCache[str, dict[str, Any]] = TTLCache(maxsize=32, ttl=300)
_congress_cache: TTLCache[str, list[dict[str, Any]]] = TTLCache(maxsize=32, ttl=600)
_insider_cache: TTLCache[str, list[dict[str, Any]]] = TTLCache(maxsize=32, ttl=600)
_request_lock = threading.Lock()
_last_request_at = 0.0
class FinnhubConnectorError(Exception):
def __init__(self, detail: str, status_code: int = 400):
super().__init__(detail)
self.detail = detail
self.status_code = status_code
# Keep old name as alias for main.py imports
UWConnectorError = FinnhubConnectorError
def _get_api_key() -> str:
api_key = os.environ.get("FINNHUB_API_KEY", "").strip()
if not api_key:
raise FinnhubConnectorError(
"Finnhub API key not configured. Add FINNHUB_API_KEY in Settings > API Keys (free at finnhub.io).",
status_code=428,
)
return api_key
def _request(path: str, params: dict[str, Any] | None = None) -> Any:
"""Rate-limited GET to Finnhub. Returns parsed JSON."""
api_key = _get_api_key()
payload = dict(params or {})
payload["token"] = api_key
global _last_request_at
with _request_lock:
elapsed = time.monotonic() - _last_request_at
if elapsed < _MIN_INTERVAL_SECONDS:
time.sleep(_MIN_INTERVAL_SECONDS - elapsed)
try:
response = requests.get(
f"{_FINNHUB_BASE}{path}",
params=payload,
timeout=_REQUEST_TIMEOUT,
headers={"User-Agent": _USER_AGENT, "Accept": "application/json"},
)
finally:
_last_request_at = time.monotonic()
if response.status_code == 401:
raise FinnhubConnectorError("Finnhub rejected the API key. Check FINNHUB_API_KEY.", 401)
if response.status_code == 403:
raise FinnhubConnectorError(
"Finnhub returned 403 — this endpoint may require a premium plan.", 403
)
if response.status_code == 429:
raise FinnhubConnectorError(
"Finnhub rate limit reached (60/min free). Try again shortly.", 429
)
if response.status_code >= 400:
detail = response.text.strip()[:240] or "Unexpected Finnhub API error."
raise FinnhubConnectorError(f"Finnhub request failed: {detail}", response.status_code)
try:
return response.json()
except ValueError as exc:
raise FinnhubConnectorError(f"Finnhub returned invalid JSON: {exc}", 502) from exc
# ---------------------------------------------------------------------------
# Status
# ---------------------------------------------------------------------------
def get_uw_status() -> dict[str, Any]:
"""Status check — kept as get_uw_status for route compatibility."""
has_key = bool(os.environ.get("FINNHUB_API_KEY", "").strip())
return {
"ok": True,
"configured": has_key,
"source": "Finnhub",
"attribution": "Data from Finnhub",
"mode": "free-tier market intelligence",
"local_only": True,
}
# ---------------------------------------------------------------------------
# Stock Quotes
# ---------------------------------------------------------------------------
def fetch_defense_quotes() -> dict[str, dict[str, Any]]:
"""Fetch real-time quotes for defense tickers. Returns {ticker: {...}}."""
results: dict[str, dict[str, Any]] = {}
for label, symbol in QUOTE_TICKERS:
cache_key = f"quote:{symbol}"
if cache_key in _quote_cache:
results[label] = _quote_cache[cache_key]
continue
try:
raw = _request("/quote", {"symbol": symbol})
if raw and raw.get("c"):
entry = {
"price": round(float(raw["c"]), 2),
"change_percent": round(float(raw.get("dp") or 0), 2),
"up": float(raw.get("dp") or 0) >= 0,
}
results[label] = entry
_quote_cache[cache_key] = entry
except FinnhubConnectorError:
logger.warning(f"Finnhub quote failed for {symbol}")
except Exception as e:
logger.warning(f"Finnhub quote error for {symbol}: {e}")
# Crypto quotes via /crypto/candle
for label, symbol in CRYPTO_TICKERS:
cache_key = f"quote:{symbol}"
if cache_key in _quote_cache:
results[label] = _quote_cache[cache_key]
continue
try:
now = int(time.time())
raw = _request("/crypto/candle", {
"symbol": symbol,
"resolution": "D",
"from": now - 172800, # 2 days back
"to": now,
})
closes = raw.get("c") or []
if len(closes) >= 1:
current = float(closes[-1])
prev = float(closes[-2]) if len(closes) >= 2 else current
change_pct = ((current - prev) / prev * 100) if prev else 0
if math.isfinite(current) and math.isfinite(change_pct):
entry = {
"price": round(current, 2),
"change_percent": round(change_pct, 2),
"up": change_pct >= 0,
"crypto": True,
}
results[label] = entry
_quote_cache[cache_key] = entry
except FinnhubConnectorError:
logger.warning(f"Finnhub crypto quote failed for {symbol}")
except Exception as e:
logger.warning(f"Finnhub crypto quote error for {symbol}: {e}")
return results
# ---------------------------------------------------------------------------
# Congressional Trading
# ---------------------------------------------------------------------------
def _normalize_congress_trade(raw: dict[str, Any], symbol: str) -> dict[str, Any]:
amount_from = raw.get("amountFrom") or 0
amount_to = raw.get("amountTo") or 0
if amount_from and amount_to:
amount_range = f"${int(amount_from):,}–${int(amount_to):,}"
elif amount_to:
amount_range = f"Up to ${int(amount_to):,}"
else:
amount_range = ""
return {
"politician_name": str(raw.get("name") or "Unknown"),
"chamber": str(raw.get("position") or "unknown").lower(),
"filing_date": str(raw.get("filingDate") or raw.get("transactionDate") or ""),
"transaction_date": str(raw.get("transactionDate") or ""),
"ticker": symbol,
"asset_name": str(raw.get("assetName") or ""),
"transaction_type": str(raw.get("transactionType") or ""),
"amount_range": amount_range,
"owner_type": str(raw.get("ownerType") or ""),
}
def fetch_congress_trades() -> dict[str, Any]:
"""Fetch congressional trades across watched tickers."""
all_trades: list[dict[str, Any]] = []
for symbol in WATCHED_TICKERS:
cache_key = f"congress:{symbol}"
if cache_key in _congress_cache:
all_trades.extend(_congress_cache[cache_key])
continue
try:
raw = _request("/stock/congressional-trading", {"symbol": symbol})
data = raw.get("data") or []
if isinstance(data, list):
normalized = [_normalize_congress_trade(t, symbol) for t in data[:10] if isinstance(t, dict)]
_congress_cache[cache_key] = normalized
all_trades.extend(normalized)
except FinnhubConnectorError as e:
if e.status_code in (403, 402):
logger.info(f"Congressional trading endpoint not available on free tier for {symbol}")
# Cache empty to avoid re-hitting a premium endpoint
_congress_cache[cache_key] = []
else:
logger.warning(f"Finnhub congress fetch failed for {symbol}: {e.detail}")
except Exception as e:
logger.warning(f"Finnhub congress error for {symbol}: {e}")
# Sort by filing date descending
all_trades.sort(key=lambda t: t.get("filing_date", ""), reverse=True)
return {
"ok": True,
"source": "Finnhub",
"attribution": "Data from Finnhub",
"trades": all_trades[:50],
}
# ---------------------------------------------------------------------------
# Insider Transactions
# ---------------------------------------------------------------------------
def _normalize_insider(raw: dict[str, Any]) -> dict[str, Any]:
return {
"name": str(raw.get("name") or "Unknown"),
"ticker": str(raw.get("symbol") or ""),
"share": int(raw.get("share") or 0),
"change": int(raw.get("change") or 0),
"filing_date": str(raw.get("filingDate") or ""),
"transaction_date": str(raw.get("transactionDate") or ""),
"transaction_code": str(raw.get("transactionCode") or ""),
"transaction_price": float(raw.get("transactionPrice") or 0),
}
def fetch_insider_transactions() -> dict[str, Any]:
"""Fetch insider transactions across watched tickers."""
all_insiders: list[dict[str, Any]] = []
for symbol in WATCHED_TICKERS:
cache_key = f"insider:{symbol}"
if cache_key in _insider_cache:
all_insiders.extend(_insider_cache[cache_key])
continue
try:
raw = _request("/stock/insider-transactions", {"symbol": symbol})
data = raw.get("data") or []
if isinstance(data, list):
normalized = [_normalize_insider(t) for t in data[:8] if isinstance(t, dict)]
_insider_cache[cache_key] = normalized
all_insiders.extend(normalized)
except FinnhubConnectorError as e:
logger.warning(f"Finnhub insider fetch failed for {symbol}: {e.detail}")
except Exception as e:
logger.warning(f"Finnhub insider error for {symbol}: {e}")
all_insiders.sort(key=lambda t: t.get("filing_date", ""), reverse=True)
return {
"ok": True,
"source": "Finnhub",
"attribution": "Data from Finnhub",
"transactions": all_insiders[:50],
}
# ---------------------------------------------------------------------------
# Aliases for backward compatibility with main.py imports
# ---------------------------------------------------------------------------
def fetch_darkpool_recent() -> dict[str, Any]:
"""Replaced — now returns insider transactions instead of dark pool."""
return fetch_insider_transactions()
def fetch_flow_alerts() -> dict[str, Any]:
"""Replaced — now returns defense quotes formatted as alerts."""
quotes = fetch_defense_quotes()
return {
"ok": True,
"source": "Finnhub",
"attribution": "Data from Finnhub",
"alerts": [],
"quotes": quotes,
}
backend/services/updater.py
+83 -5
View File
@@ -5,6 +5,7 @@ Public API:
perform_update(project_root) -> dict (download + backup + extract)
schedule_restart(project_root) (spawn detached start script, then exit)
"""
import os
import sys
import logging
@@ -13,6 +14,8 @@ import subprocess
import tempfile
import time
import zipfile
import hashlib
from urllib.parse import urlparse
from datetime import datetime
from pathlib import Path
@@ -21,17 +24,41 @@ import requests
logger = logging.getLogger(__name__)
GITHUB_RELEASES_URL = "https://api.github.com/repos/BigBodyCobain/Shadowbroker/releases/latest"
GITHUB_RELEASES_PAGE_URL = "https://github.com/BigBodyCobain/Shadowbroker/releases/latest"
_EXPECTED_SHA256 = os.environ.get("MESH_UPDATE_SHA256", "").strip().lower()
_ALLOWED_UPDATE_HOSTS = {
"api.github.com",
"github.com",
"objects.githubusercontent.com",
"release-assets.githubusercontent.com",
"github-releases.githubusercontent.com",
}
# ---------------------------------------------------------------------------
# Protected patterns — files/dirs that must NEVER be overwritten during update
# ---------------------------------------------------------------------------
_PROTECTED_DIRS = {"venv", "node_modules", ".next", "__pycache__", ".git", ".github", ".claude"}
_PROTECTED_EXTENSIONS = {".db", ".sqlite"}
_PROTECTED_DIRS = {
"venv", "node_modules", ".next", "__pycache__", ".git", ".github", ".claude",
"_domain_keys", "node-local", "gate_persona", "gate_session", "dm_alias",
"root", "transport", "reputation",
}
_PROTECTED_EXTENSIONS = {".db", ".sqlite", ".key", ".pem", ".bin"}
_PROTECTED_NAMES = {
".env",
"ais_cache.json",
"carrier_cache.json",
"geocode_cache.json",
"infonet.json",
"infonet.json.bak",
"peer_store.json",
"node.json",
"wormhole.json",
"wormhole_status.json",
"wormhole_secure_store.key",
"dm_token_pepper.key",
"voter_blind_salt.bin",
"reputation_ledger.json",
"gates.json",
}
@@ -55,19 +82,39 @@ def _is_protected(rel_path: str) -> bool:
return False
def _validate_update_url(url: str, *, allow_release_page: bool = False) -> str:
parsed = urlparse(str(url or "").strip())
host = (parsed.hostname or "").strip().lower()
if parsed.scheme != "https":
raise RuntimeError("Updater refused a non-HTTPS release URL")
if parsed.username or parsed.password:
raise RuntimeError("Updater refused a credentialed release URL")
if not host or host not in _ALLOWED_UPDATE_HOSTS:
raise RuntimeError(f"Updater refused an untrusted release host: {host or 'unknown'}")
if parsed.port not in (None, 443):
raise RuntimeError("Updater refused a non-standard release port")
if not allow_release_page and host == "github.com" and "/releases/" not in parsed.path:
raise RuntimeError("Updater refused a non-release GitHub URL")
return parsed.geturl()
# ---------------------------------------------------------------------------
# Download
# ---------------------------------------------------------------------------
def _download_release(temp_dir: str) -> tuple:
"""Fetch latest release info and download the zip asset.
Returns (zip_path, version_tag, download_url).
Returns (zip_path, version_tag, download_url, release_url).
"""
logger.info("Fetching latest release info from GitHub...")
_validate_update_url(GITHUB_RELEASES_URL)
resp = requests.get(GITHUB_RELEASES_URL, timeout=15)
resp.raise_for_status()
_validate_update_url(resp.url)
release = resp.json()
tag = release.get("tag_name", "unknown")
release_url = str(release.get("html_url") or GITHUB_RELEASES_PAGE_URL).strip()
_validate_update_url(release_url, allow_release_page=True)
assets = release.get("assets", [])
# Find the .zip asset
@@ -80,11 +127,13 @@ def _download_release(temp_dir: str) -> tuple:
if not zip_url:
raise RuntimeError("No .zip asset found in the latest release")
_validate_update_url(zip_url)
logger.info(f"Downloading {zip_url} ...")
zip_path = os.path.join(temp_dir, "update.zip")
with requests.get(zip_url, stream=True, timeout=120) as dl:
dl.raise_for_status()
_validate_update_url(dl.url)
with open(zip_path, "wb") as f:
for chunk in dl.iter_content(chunk_size=1024 * 64):
f.write(chunk)
@@ -94,7 +143,19 @@ def _download_release(temp_dir: str) -> tuple:
size_mb = os.path.getsize(zip_path) / (1024 * 1024)
logger.info(f"Downloaded {size_mb:.1f} MB — ZIP validated OK")
return zip_path, tag, zip_url
return zip_path, tag, zip_url, release_url
def _validate_zip_hash(zip_path: str) -> None:
if not _EXPECTED_SHA256:
return
h = hashlib.sha256()
with open(zip_path, "rb") as f:
for chunk in iter(lambda: f.read(1024 * 128), b""):
h.update(chunk)
digest = h.hexdigest().lower()
if digest != _EXPECTED_SHA256:
raise RuntimeError("Update SHA-256 mismatch")
# ---------------------------------------------------------------------------
@@ -142,6 +203,16 @@ def _extract_and_copy(zip_path: str, project_root: str, temp_dir: str) -> int:
extract_dir = os.path.join(temp_dir, "extracted")
logger.info("Extracting update zip...")
with zipfile.ZipFile(zip_path, "r") as zf:
extract_root = Path(extract_dir).resolve()
for member in zf.infolist():
try:
target = (extract_root / member.filename).resolve()
except OSError as exc:
raise RuntimeError(f"Updater refused archive entry {member.filename}: {exc}") from exc
try:
target.relative_to(extract_root)
except ValueError:
raise RuntimeError(f"Updater refused archive path traversal entry: {member.filename}")
zf.extractall(extract_dir)
# Detect wrapper folder: if extracted root has a single directory that
@@ -245,8 +316,11 @@ def perform_update(project_root: str) -> dict:
separately after the HTTP response has been sent.
"""
temp_dir = tempfile.mkdtemp(prefix="sb_update_")
manual_url = GITHUB_RELEASES_PAGE_URL
try:
zip_path, version, url = _download_release(temp_dir)
zip_path, version, url, release_url = _download_release(temp_dir)
manual_url = release_url or manual_url
_validate_zip_hash(zip_path)
backup_path = _backup_current(project_root, temp_dir)
copied = _extract_and_copy(zip_path, project_root, temp_dir)
@@ -255,6 +329,9 @@ def perform_update(project_root: str) -> dict:
"version": version,
"files_updated": copied,
"backup_path": backup_path,
"manual_url": manual_url,
"release_url": release_url,
"download_url": url,
"message": f"Updated to {version} — {copied} files replaced. Restarting...",
}
except Exception as e:
@@ -262,4 +339,5 @@ def perform_update(project_root: str) -> dict:
return {
"status": "error",
"message": str(e),
"manual_url": manual_url,
}
backend/services/wormhole_settings.py
+92
View File
@@ -0,0 +1,92 @@
from __future__ import annotations
import json
import time
from pathlib import Path
DATA_DIR = Path(__file__).parent.parent / "data"
WORMHOLE_FILE = DATA_DIR / "wormhole.json"
_cache: dict | None = None
_cache_ts: float = 0.0
_CACHE_TTL = 5.0 # seconds
_DEFAULTS = {
"enabled": False,
"transport": "direct",
"socks_proxy": "",
"socks_dns": True,
"privacy_profile": "default",
"anonymous_mode": False,
}
def _safe_int(val, default=0) -> int:
try:
return int(val)
except (TypeError, ValueError):
return default
def read_wormhole_settings() -> dict:
global _cache, _cache_ts
now = time.monotonic()
if _cache is not None and (now - _cache_ts) < _CACHE_TTL:
return _cache
if not WORMHOLE_FILE.exists():
result = {**_DEFAULTS, "updated_at": 0}
else:
try:
data = json.loads(WORMHOLE_FILE.read_text(encoding="utf-8"))
except Exception:
result = {**_DEFAULTS, "updated_at": 0}
else:
result = {
"enabled": bool(data.get("enabled", _DEFAULTS["enabled"])),
"transport": str(data.get("transport", _DEFAULTS["transport"]) or _DEFAULTS["transport"]),
"socks_proxy": str(data.get("socks_proxy", _DEFAULTS["socks_proxy"]) or ""),
"socks_dns": bool(data.get("socks_dns", _DEFAULTS["socks_dns"])),
"privacy_profile": str(
data.get("privacy_profile", _DEFAULTS["privacy_profile"]) or _DEFAULTS["privacy_profile"]
),
"anonymous_mode": bool(data.get("anonymous_mode", _DEFAULTS["anonymous_mode"])),
"updated_at": _safe_int(data.get("updated_at", 0) or 0),
}
_cache = result
_cache_ts = now
return result
def write_wormhole_settings(
*,
enabled: bool | None = None,
transport: str | None = None,
socks_proxy: str | None = None,
socks_dns: bool | None = None,
privacy_profile: str | None = None,
anonymous_mode: bool | None = None,
) -> dict:
DATA_DIR.mkdir(parents=True, exist_ok=True)
existing = read_wormhole_settings()
payload = {
"enabled": bool(existing.get("enabled")) if enabled is None else bool(enabled),
"transport": existing.get("transport", _DEFAULTS["transport"])
if transport is None
else str(transport),
"socks_proxy": existing.get("socks_proxy", "")
if socks_proxy is None
else str(socks_proxy),
"socks_dns": bool(existing.get("socks_dns", _DEFAULTS["socks_dns"]))
if socks_dns is None
else bool(socks_dns),
"privacy_profile": existing.get("privacy_profile", _DEFAULTS["privacy_profile"])
if privacy_profile is None
else str(privacy_profile),
"anonymous_mode": bool(existing.get("anonymous_mode", _DEFAULTS["anonymous_mode"]))
if anonymous_mode is None
else bool(anonymous_mode),
"updated_at": int(time.time()),
}
WORMHOLE_FILE.write_text(json.dumps(payload, indent=2), encoding="utf-8")
global _cache, _cache_ts
_cache = payload
_cache_ts = time.monotonic()
return payload

Some files were not shown because too many files have changed in this diff Show More