2 Commits

Author	SHA1	Message	Date
Shadowbroker	76750caa92	Round 7a: per-operator outbound attribution + GDELT GCS-direct fix (#292) ... == Per-install operator handle for every third-party API call == Before this PR, every Shadowbroker install identified itself to Wikipedia, Wikidata, Nominatim, GDELT, OpenMHz, Broadcastify, weather.gov, NUFORC, Sentinel/Planetary Computer, TinyGS / CelesTrak, Shodan, Finnhub, and others with a single project-wide User-Agent ("Shadowbroker/1.0" or "ShadowBroker-OSINT/1.0"). From the upstream's perspective every install in the world looked like one giant scraper. If one install misbehaved, the upstream's only recourse was to block "Shadowbroker" as a whole. PR #284 inadvertently doubled down on this in the frontend by introducing a shared `WIKIMEDIA_API_USER_AGENT` constant. This PR retrofits both backends to per-operator attribution. New setting: OPERATOR_HANDLE (env var / settings UI / auto-gen) New helper: network_utils.outbound_user_agent("purpose") The handle is auto-generated as "operator-XXXXXX" on first call (the "shadow-" prefix from earlier drafts was deliberately dropped — too suspicious-looking for abuse-detection systems). Operators can override via OPERATOR_HANDLE; the value is sanitized to lowercase alphanumeric+dash+underscore and capped at 48 chars. Persisted to backend/data/operator_handle.json so it survives container restarts. Retrofitted call sites (every previously-MONSTER User-Agent): - services/region_dossier.py (Wikipedia + Wikidata + Nominatim) - services/geocode.py (Nominatim) - services/sentinel_search.py (Microsoft Planetary Computer) - services/feed_ingester.py (operator-curated RSS feeds) - services/fetchers/earth_observation.py (weather.gov, NUFORC) - services/fetchers/infrastructure.py - services/fetchers/aircraft_database.py - services/fetchers/route_database.py - services/fetchers/trains.py - services/fetchers/meshtastic_map.py - services/shodan_connector.py - services/unusual_whales_connector.py (Finnhub) - services/tinygs_fetcher.py (CelesTrak + TinyGS) - services/sar/sar_products_client.py - services/geopolitics.py (GDELT) - services/radio_intercept.py (Broadcastify + OpenMHz) - routers/cctv.py + main.py (CCTV proxy) - routers/ai_intel.py - scripts/convert_power_plants.py (release-time data refresh) Spoofed browser UAs removed (issues #289 / #290 / #291 — tg12 audit): - cloudscraper-based Chrome impersonation against api.openmhz.com -> replaced with honest requests + per-install UA - Mozilla/5.0 spoofed UA on Broadcastify scrape -> replaced with honest UA - Mozilla/5.0 + fake first-party Referer on OpenMHz audio relay -> replaced with honest UA - cloudscraper dependency dropped from pyproject.toml + uv.lock Frontend retrofit: - new GET /api/settings/operator-handle endpoint (local-operator gated) returns the install's handle - frontend/src/lib/wikimediaClient.ts fetches the handle once on first use, caches it for page lifetime, embeds it in the Api-User-Agent for every Wikipedia / Wikidata browser-direct call == GDELT GCS-direct fix == GDELT's data.gdeltproject.org is a CNAME to a Google Cloud Storage bucket. GCS responds with the wildcard *.storage.googleapis.com cert which legitimately does NOT cover the GDELT custom domain, so Python's TLS verification correctly refuses the connection. Some networks happen to route through a path where this works; many (notably Docker Desktop's outbound NAT on local installs) do not. Verified on the maintainer's local install: GDELT was unreachable; 1610 geopolitical events / 48 export files were dropping silently. Fix: services/geopolitics._gcs_direct_gdelt_url() rewrites any data.gdeltproject.org URL to its GCS-direct equivalent (storage.googleapis.com/data.gdeltproject.org/...) where the standard GCS cert is genuinely valid. api.gdeltproject.org and every other host are left untouched. Confirmed live: backend log goes from GDELT lastupdate failed: 500 to Downloading 48 GDELT export files... Downloaded 48/48 GDELT exports GDELT parsed: 1610 conflict locations from 48 files == Tests == backend/tests/test_per_operator_outbound_attribution.py (12 tests) backend/tests/test_gdelt_gcs_direct_rewrite.py (6 tests) backend/tests/test_region_dossier_wikimedia_ua.py (updated to pin the helper + per-operator handle, not the old constant) frontend/src/__tests__/utils/wikimediaClient.test.ts (rewritten to mock /api/settings/operator-handle and assert per-operator UA) Local: backend 114/114 security+audit+round7a suite green; frontend 718/718 vitest suite green. Credit: tg12 (external security audit, issues #289/#290/#291 relating to spoofed UAs); BigBodyCobain (operator-prefix call, GDELT cloud-vs-local diagnosis).	2026-05-21 15:11:28 -06:00
Shadowbroker	0fee36e8f7	Fix #218/#219/#220: identify ShadowBroker on Wikipedia + Wikidata calls (#284) ... Wikimedia's User-Agent policy asks API clients to identify themselves with a stable, contactable identifier so their operators can rate-limit or coordinate. Before this change, ShadowBroker was sending: - Backend (region_dossier.py): generic project default UA only; no Api-User-Agent. - Frontend (useRegionDossier.ts, WikiImage.tsx, NewsFeed.tsx): zero identifying header at all; three separate copy-pasted anonymous fetches with their own module-local caches. Three separate components doing the same broken thing meant policy fixes had to happen in three places, with no shared cache or kill switch. Fix (no UX change, zero hostility): == Backend == `backend/services/region_dossier.py` now sets explicit `User-Agent` + `Api-User-Agent` headers on every outbound Wikidata and Wikipedia request via a new `_WIKIMEDIA_REQUEST_HEADERS` constant. The identifier includes a contact path (issues page on the public GitHub repo). == Frontend == New shared helper `frontend/src/lib/wikimediaClient.ts`: - `fetchWikipediaSummary(title)` — single source of truth for Wikipedia REST summary lookups, with one shared LRU cache (in-flight requests deduplicated, 512-entry cap), `Api-User-Agent` on every fetch. - `fetchWikidataSparql(query)` — same shape for Wikidata SPARQL. - `WIKIMEDIA_API_USER_AGENT` — exported constant; one place to update if Wikimedia ever asks us to back off. Refactored three components to use the shared client: - `frontend/src/hooks/useRegionDossier.ts` — fetchLeader() and fetchLocalWikiSummary() now route through the shared helpers. - `frontend/src/components/WikiImage.tsx` — uses fetchWikipediaSummary, proper React state instead of module-mutation + forceUpdate trick. - `frontend/src/components/NewsFeed.tsx` — same shape. UX: byte-for-byte identical. Same thumbnails, same dossier content, same load behavior. The only observable difference is the outgoing request header. Note on #239 (route duplication): an audit-grade inventory shows 166 main.py routes are shadowed by router modules. That cleanup is too large to land safely in this PR; it will be staged as a separate ladder of small PRs grouped by router module. Tests: - `backend/tests/test_region_dossier_wikimedia_ua.py` — 3 tests asserting backend headers are present. - `frontend/src/__tests__/utils/wikimediaClient.test.ts` — 9 tests covering Api-User-Agent presence, shared cache, concurrent deduplication, disambiguation/HTTP-error/network-error fallthroughs, empty-input safety. Local: backend 76/76 security suite green, frontend 716/716 vitest suite green. Credit: tg12 (external security audit).	2026-05-21 10:48:05 -06:00