Pre-fix, adsb.lol records (the primary source for most flights) carried
no source marker. OpenSky records got is_opensky: True and supplementals
got supplemental_source, so any UI inspecting source labels saw
OpenSky/airplanes.live records as explicitly tagged and adsb.lol records
as "unlabeled" — making it look like adsb.lol wasn't being used at all
even though it's the primary source.
Changes:
* _fetch_adsb_lol_regions stamps source="adsb.lol" on each aircraft
before returning, so the tag survives the OpenSky dedupe-by-hex merge.
* OpenSky records get source="OpenSky" (alongside is_opensky=True for
back-compat).
* military fetcher tags source on both adsb.lol and airplanes.live
records before they're merged, and propagates source into the
military_flights and uavs output dicts.
* _classify_and_publish promotes the explicit source field into the
published flight dict. Falls back to legacy supplemental_source if
source is absent. Final fallback "adsb.lol" preserves prior behavior
for any caller synthesizing records without going through a fetcher.
8 new tests cover the published-dict propagation, OpenSky tagging,
supplemental fallback, explicit-wins precedence, default behavior, the
adsb.lol regional fetcher tagging, and the military output dict.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Background
==========
PR #285 set up the seed -> cache -> GDELT model for the carrier tracker
to address audit issues #244/#245/#246. The GDELT half of that pipeline
hits api.gdeltproject.org's doc API for headline-region keyword
matching -- low precision (false centroid positions per #245) AND
unreliable (the host times out from some networks, including Docker
Desktop on Windows).
USNI publishes a weekly Fleet & Marine Tracker with explicit prose like:
"The Gerald R. Ford Carrier Strike Group is operating in the Red Sea"
"Aircraft carrier USS George Washington (CVN-73) is in port in
Yokosuka, Japan"
That is a strictly better source for U.S. Navy carrier positions:
authoritative, deterministically parseable, weekly cadence.
What this PR does
=================
New module: backend/services/fetchers/usni_fleet_tracker.py
- Pulls USNI's WordPress RSS feeds (site-wide + category, unioned).
- Picks the most recent fleet-tracker post by parsed pubDate.
- For each carrier in the registry, scans the article body for
"is operating in / is in port in / returned to / transiting" near
the carrier's name, hull code, or "<name> Carrier Strike Group"
variant. Captures the region/port phrase that follows.
- Maps the region phrase to coordinates via the existing
REGION_COORDS table, with a USNI-phrase alias table for the
specific wording USNI uses ("Yokosuka, Japan", "Norfolk, Va.",
"Naval Station San Diego", "5th Fleet AOR", etc.).
- Returns {hull: position_entry} with position_confidence="recent"
and position_source_at = the article's actual publication
timestamp (not now()).
Politeness
----------
Uses outbound_user_agent("usni-fleet-tracker") so USNI sees a
per-install Shadowbroker identifier (Round 7a / PR #292). The
article body pages return 403 to non-browser UAs; the WordPress RSS
feed serves the full <content:encoded> body and is the supported
aggregator path. No browser UA spoofing.
carrier_tracker.update_carrier_positions() now runs three phases:
1. Bootstrap from cache (or seed on first run).
2. USNI fleet tracker -- PRIMARY high-confidence source.
3. GDELT -- SECONDARY backfill; can NOT demote a "recent" USNI
position to an "approximate" GDELT headline match.
Verified live: 6 of 11 carriers picked up real May 18, 2026 positions
on first refresh (Eisenhower, Ford, Bush, Roosevelt, Lincoln,
Washington). The other 5 weren't mentioned in this week's article
(they're in port at homeports with no deployment changes) and kept
their cache entries -- which is the correct seed/cache contract from
PR #285.
Other small fixes bundled in
============================
docker-compose.yml: add the 6 third-party-fetcher opt-in env vars
(PREDICTION_MARKETS_ENABLED, FINANCIAL_ENABLED, FIMI_ENABLED,
NUFORC_ENABLED, NEWS_ENABLED, CROWDTHREAT_ENABLED). They were
documented in .env.example but never wired through compose, so setting
them in .env had no effect.
frontend/src/components/TopRightControls.tsx: fix 6 broken i18n keys
that were showing as raw "terminal.term1" / "terminal.cleanupDetail" /
"node.soloReady" placeholders in the INFONET TERMINAL modal. The
translation files have these strings under different key names; the
component now calls the right ones. Full-file sweep confirmed every
other t('...') key in the whole frontend resolves cleanly.
== Per-install operator handle for every third-party API call ==
Before this PR, every Shadowbroker install identified itself to
Wikipedia, Wikidata, Nominatim, GDELT, OpenMHz, Broadcastify,
weather.gov, NUFORC, Sentinel/Planetary Computer, TinyGS / CelesTrak,
Shodan, Finnhub, and others with a single project-wide User-Agent
("Shadowbroker/1.0" or "ShadowBroker-OSINT/1.0"). From the upstream's
perspective every install in the world looked like one giant scraper.
If one install misbehaved, the upstream's only recourse was to block
"Shadowbroker" as a whole.
PR #284 inadvertently doubled down on this in the frontend by
introducing a shared `WIKIMEDIA_API_USER_AGENT` constant. This PR
retrofits both backends to per-operator attribution.
New setting: OPERATOR_HANDLE (env var / settings UI / auto-gen)
New helper: network_utils.outbound_user_agent("purpose")
The handle is auto-generated as "operator-XXXXXX" on first call (the
"shadow-" prefix from earlier drafts was deliberately dropped — too
suspicious-looking for abuse-detection systems). Operators can
override via OPERATOR_HANDLE; the value is sanitized to lowercase
alphanumeric+dash+underscore and capped at 48 chars. Persisted to
backend/data/operator_handle.json so it survives container restarts.
Retrofitted call sites (every previously-MONSTER User-Agent):
- services/region_dossier.py (Wikipedia + Wikidata + Nominatim)
- services/geocode.py (Nominatim)
- services/sentinel_search.py (Microsoft Planetary Computer)
- services/feed_ingester.py (operator-curated RSS feeds)
- services/fetchers/earth_observation.py (weather.gov, NUFORC)
- services/fetchers/infrastructure.py
- services/fetchers/aircraft_database.py
- services/fetchers/route_database.py
- services/fetchers/trains.py
- services/fetchers/meshtastic_map.py
- services/shodan_connector.py
- services/unusual_whales_connector.py (Finnhub)
- services/tinygs_fetcher.py (CelesTrak + TinyGS)
- services/sar/sar_products_client.py
- services/geopolitics.py (GDELT)
- services/radio_intercept.py (Broadcastify + OpenMHz)
- routers/cctv.py + main.py (CCTV proxy)
- routers/ai_intel.py
- scripts/convert_power_plants.py (release-time data refresh)
Spoofed browser UAs removed (issues #289 / #290 / #291 — tg12 audit):
- cloudscraper-based Chrome impersonation against api.openmhz.com
-> replaced with honest requests + per-install UA
- Mozilla/5.0 spoofed UA on Broadcastify scrape
-> replaced with honest UA
- Mozilla/5.0 + fake first-party Referer on OpenMHz audio relay
-> replaced with honest UA
- cloudscraper dependency dropped from pyproject.toml + uv.lock
Frontend retrofit:
- new GET /api/settings/operator-handle endpoint (local-operator
gated) returns the install's handle
- frontend/src/lib/wikimediaClient.ts fetches the handle once on
first use, caches it for page lifetime, embeds it in the
Api-User-Agent for every Wikipedia / Wikidata browser-direct call
== GDELT GCS-direct fix ==
GDELT's data.gdeltproject.org is a CNAME to a Google Cloud Storage
bucket. GCS responds with the wildcard *.storage.googleapis.com cert
which legitimately does NOT cover the GDELT custom domain, so Python's
TLS verification correctly refuses the connection. Some networks
happen to route through a path where this works; many (notably Docker
Desktop's outbound NAT on local installs) do not. Verified on the
maintainer's local install: GDELT was unreachable; 1610 geopolitical
events / 48 export files were dropping silently.
Fix: services/geopolitics._gcs_direct_gdelt_url() rewrites any
data.gdeltproject.org URL to its GCS-direct equivalent
(storage.googleapis.com/data.gdeltproject.org/...) where the standard
GCS cert is genuinely valid. api.gdeltproject.org and every other host
are left untouched.
Confirmed live: backend log goes from
GDELT lastupdate failed: 500
to
Downloading 48 GDELT export files...
Downloaded 48/48 GDELT exports
GDELT parsed: 1610 conflict locations from 48 files
== Tests ==
backend/tests/test_per_operator_outbound_attribution.py (12 tests)
backend/tests/test_gdelt_gcs_direct_rewrite.py (6 tests)
backend/tests/test_region_dossier_wikimedia_ua.py (updated to
pin the helper + per-operator handle, not the old constant)
frontend/src/__tests__/utils/wikimediaClient.test.ts (rewritten
to mock /api/settings/operator-handle and assert per-operator UA)
Local: backend 114/114 security+audit+round7a suite green;
frontend 718/718 vitest suite green.
Credit: tg12 (external security audit, issues #289/#290/#291
relating to spoofed UAs); BigBodyCobain (operator-prefix call,
GDELT cloud-vs-local diagnosis).
Detected by Aeon + Semgrep (5x use-defused-xml ERROR).
Severity: medium
CWE-776 (billion laughs) / CWE-611 (XML external entity)
Five XML parse sites pass response bodies into the Python stdlib
xml.etree.ElementTree without protection against entity expansion
attacks. Python's ElementTree still permits internal entity references
by default (per the docs vulnerabilities table), so a malicious or
compromised upstream can ship a "billion laughs"-style payload that
expands to gigabytes in memory.
The user-controllable site is sb_monitor._parse_rss: the OpenClaw skill
exposes add_custom_feed(name, url, ...) to the agent, then
poll_custom_feeds fetches feed.url and passes the body to
xml.etree.ElementTree.fromstring with no host allowlist or
entity-bomb defence. The other four sites (psk_reporter_fetcher,
aircraft_database, cctv_pipeline x2) parse XML from hard-coded
upstreams (pskreporter.info, s3.opensky-network.org,
datos.madrid.es); defence-in-depth for upstream-compromise/MITM.
Switch all five call sites to defusedxml.ElementTree. Same
fromstring/find/findall/iter/findtext API, but rejects entity
references by default (raises defusedxml.EntitiesForbidden).
Confirmed locally that a 4-deep billion-laughs payload that
expands to 3000 chars under stdlib ET is rejected by defusedxml.
Added defusedxml>=0.7.1 to backend/pyproject.toml dependencies.
Co-authored-by: aeonframework <aeon-bot@aaronjmars.com>
External security audit by @tg12 (May 17, 2026) filed issues #201–#214
in addition to the #189–#200 batch already closed by PRs #227/#232/#260.
This PR closes all eight that are real security bugs (the other six in
the 201–214 range are either design discussions or upstream-abuse/TOS
concerns we're keeping intentional, see issue triage notes on each).
The user-facing principle for this PR: fix the security gap WITHOUT
introducing a single hostile error or behavior change for legitimate
users. Every fix follows the same template — fail forward, not loud.
When the secure path is harder than the insecure one, build a
fallback chain that ends in graceful degradation, not in a scary
modal or 422 response.
#205 — OpenMHZ audio redirect SSRF (services/radio_intercept.py)
Replaced requests.get(..., allow_redirects=True) with a manual
redirect loop that re-validates each hop's host against
_OPENMHZ_AUDIO_HOSTS. Same-host redirects (CDN edge selection)
still work, so legitimate audio playback is unaffected. Cross-host
redirects to disallowed hosts return a generic 502 which the
browser audio element handles gracefully. Cap at 5 hops.
#207 — infonet/status verify_signatures DoS (routers/mesh_public.py)
Silently downgrade verify_signatures=true to False for
unauthenticated callers. No error surfaced — the response shape is
identical, just without the O(n_events) signature verification.
Authenticated callers (scoped mesh.audit) still get the full path.
The frontend never passes this param so legitimate UI is unaffected.
#211 — thermal/verify expensive analysis (routers/sigint.py)
Added Depends(require_local_operator). Frontend has no direct
callers (verified by grep); Tauri/AI agents use scoped tokens that
pass the auth check. Anonymous abusers blocked silently — the
legitimate UI keeps working through the Next.js admin-key proxy.
#213, #214 — OpenMHZ calls/audio upstream abuse (routers/radio.py)
Added Depends(require_local_operator) to both. Browser users hit
these through the Next.js proxy at src/app/api/[...path]/route.ts
which injects X-Admin-Key, so the auth check passes transparently.
Direct attackers can no longer rotate sys_names to hammer
api.openmhz.com or relay arbitrary audio streams through the
backend's bandwidth.
#202 — overflights unbounded hours (routers/data.py)
Silently clamp `hours` to OVERFLIGHTS_MAX_HOURS (default 72,
configurable). NO 422 — clients asking for an absurd window get a
shorter window back with `requested_hours` and `effective_hours`
hint fields. Postel's law: liberal in what we accept, conservative
in what we compute.
#203 — Meshtastic callsign UA leak (services/fetchers/meshtastic_map.py)
Added MESHTASTIC_SEND_CALLSIGN_HEADER opt-out env var. Default is
TRUE — preserves existing operator behavior (callsign sent so
meshtastic.org can rate-limit per-install). Privacy-conscious
operators set it to false to suppress.
#206 — KiwiSDR upstream is HTTP-only (services/kiwisdr_fetcher.py)
Upstream rx.linkfanel.net doesn't speak HTTPS (verified — Apache
2.4.10 only on port 80). We can't fix the transport. Instead added
three layers:
1. Content validation on fetched data — reject responses with
<50 receivers or >5% malformed entries (likely MITM injection).
2. Existing disk cache fallback (already present).
3. NEW: bundled static directory at backend/data/kiwisdr_directory.json
shipping 798 known-good receivers. Used as last resort so the
KiwiSDR map layer always renders something useful.
#208 — Merkle proof DoS via /api/mesh/infonet/sync (services/mesh/mesh_hashchain.py)
The endpoint is part of the cross-node federation protocol — peers
legitimately call it without local-operator auth, so we can't add
Depends(). Instead made the underlying operation O(1) per proof
via a cached Merkle level structure on the Infonet instance:
- _merkle_levels_cache + _merkle_levels_for_event_count on each
Infonet instance
- _invalidate_merkle_cache() called from every chain mutation
point (append, ingest_events, apply_fork, cleanup_expired)
- _get_merkle_levels() does the lazy recompute on first read
after invalidation, then serves from cache thereafter
Effect: anonymous attackers hammering the proofs endpoint hit a
cached structure; the rebuild happens at most once per real chain
advance. Federation untouched.
#201 — Tor bundle SHA-256 bypass (services/tor_hidden_service.py)
Docker users were already covered — backend/Dockerfile installs
Tor via apt-get at build time (signed by Debian's package system).
No runtime download needed for the 80%-of-users case.
For Tauri desktop, replaced the single .sha256sum check with a
multi-source verification chain implemented in _verify_tor_bundle():
1. Try upstream .sha256sum (current behavior — fast path)
2. Try baked-in digest list at backend/data/tor_bundle_digests.json
(pinned per-version, maintainer-updated)
3. If neither source is REACHABLE: HTTPS-only fallback with a loud
warning (avoids breaking first-run onboarding while the
maintainer hasn't yet pinned a new Tor release)
A mismatch from a source that DID respond is always fatal — only
the "no source reachable" case falls back to HTTPS-only. This is
the "have cake and eat it" pattern: real users see no new failure
modes during torproject.org outages, but MITM/compromise attacks
still fail because the downloaded digest can't match what BOTH
the upstream and the baked-in list report.
Currently the digest file ships with placeholder values for the
current Tor URLs (those URLs are already stale on torproject.org
too). A follow-up commit can populate real digests when a stable
Tor release is selected; until then the HTTPS-only warning fires
and onboarding still works.
Tests (82 total, all passing):
test_openmhz_redirect_ssrf.py (5 tests) — #205
test_infonet_status_verify_gate.py (2 tests) — #207
test_overflights_clamp.py (5 tests) — #202
test_meshtastic_callsign_optout.py (3 tests) — #203
test_kiwisdr_fallback.py (6 tests) — #206
test_merkle_cache.py (6 tests) — #208
test_tor_bundle_verification.py (6 tests) — #201
test_control_surface_auth.py (extended) — #211, #213, #214
+ all previous security tests (CCTV redirect, GDELT https, sentinel
cache, crowdthreat opt-in, third-party fetcher gates, control
surface auth) continue to pass.
Pre-existing test infrastructure issue with SHARED_EXECUTOR teardown
in the broader sweep exists on main too (verified) — not introduced
by this PR.
Credit: @tg12 reported every one of these with accurate line citations
and the recommended fixes that informed this implementation.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
PR #227 hardened most Wormhole/Infonet control surfaces behind
require_local_operator and made the CrowdThreat fetcher opt-in. An
audit of the codebase against that PR's stated goals turned up four
classes of gap that the original change missed:
1. Two operator-only endpoints were left unprotected:
- POST /api/wormhole/join: calls bootstrap_wormhole_identity() and
flips the node into Tor mode, exactly the surface #227 hardened
on /api/wormhole/identity/bootstrap.
- POST /api/sigint/transmit: relays APRS-IS packets over radio
using operator-supplied credentials. Anything that reached the
API could transmit on the operator's authority.
Both now require_local_operator. test_control_surface_auth.py
extended with regression coverage for both.
2. Five third-party fetchers were still default-on, phoning home to
politically/commercially sensitive upstreams on every poll cycle:
- fimi.py -> euvsdisinfo.eu -> FIMI_ENABLED
- prediction_markets -> Polymarket + Kalshi -> PREDICTION_MARKETS_ENABLED
- financial.py -> Finnhub / yfinance -> FINANCIAL_ENABLED or FINNHUB_API_KEY
- nuforc_enrichment -> huggingface.co -> NUFORC_ENABLED
- news.py -> configured RSS feeds -> NEWS_ENABLED (default on, kill switch)
Same CrowdThreat-style pattern: explicit env-var opt-in, empty
the data slot and mark_fresh when disabled. New regression test
file test_third_party_fetchers_opt_in.py asserts each fetcher's
network entry point is not called when its gate is off.
3. The outbound User-Agent leaked both the operator's personal email
and a fork-specific GitHub URL on every fetcher request. Consolidated
to a single DEFAULT_USER_AGENT in network_utils.py, project-generic
by default (no contact info), overridable via SHADOWBROKER_USER_AGENT
for operators who want to identify themselves (e.g. for Nominatim or
weather.gov usage-policy compliance). Six call sites updated; the
Nominatim-specific override is preserved.
4. The same generic UA now also flows through the peer prekey lookup
in mesh_wormhole_prekey.py, so DM first-contact requests no longer
identify the caller as a Shadowbroker fork to the peer being
queried.
.env.example updated to document all new opt-in env vars.
Tests: backend/tests/test_control_surface_auth.py (extended),
backend/tests/test_crowdthreat_opt_in.py (unchanged, still passes),
backend/tests/test_third_party_fetchers_opt_in.py (new, 7 tests).
All 31 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Ship the v0.9.79 runtime refresh with transport lane isolation, Infonet secure-message address management, MeshChat MQTT controls, selected asset trail behavior, telemetry panel refinements, onboarding updates, and desktop/package metadata alignment.
Also ignore local graphify work products so analysis folders do not leak into future commits.
Gate messages now propagate via the Infonet hashchain as encrypted blobs — every node syncs them
through normal chain sync while only Gate members with MLS keys can decrypt. Added mesh reputation
system, peer push workers, voluntary Wormhole opt-in for node participation, fork recovery,
killwormhole scripts, obfuscated terminology, and hardened the self-updater to protect encryption
keys and chain state during updates.
New features: Shodan search, train tracking, Sentinel Hub imagery, 8 new intelligence layers,
CCTV expansion to 11,000+ cameras across 6 countries, Mesh Terminal CLI, prediction markets,
desktop-shell scaffold, and comprehensive mesh test suite (215 frontend + backend tests passing).
Community contributors: @wa1id, @AlborzNazari, @adust09, @Xpirix, @imqdcr, @csysp, @suranyami,
@chr0n1x, @johan-martensson, @singularfailure, @smithbh, @OrfeoTerkuci, @deuza, @tm-const,
@Elhard1, @ttulttul
Map ~35,000 power generation facilities from 164 countries using the
WRI Global Power Plant Database (CC BY 4.0). Follows the existing
datacenter layer pattern with clustered icon symbols, amber color
scheme, and click popups showing fuel type, capacity, and operator.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add 68 military bases (PLA, Russia, DPRK, ROC, Philippines, Australia)
with data-driven color coding (red/blue/green) on the map
- Add 6 news RSS feeds (Yonhap, Nikkei Asia, Taipei Times, Asia Times,
Defense News, Japan Times) and 15 geocoding keywords for islands,
straits, and disputed areas
- Extend ICAO country ranges for Russia, Australia, Philippines,
Singapore, DPRK and add Russian aircraft classification (fighters,
bombers, cargo, recon)
- Create PLAN/CCG vessel enrichment module (90+ ships) following
yacht_alert pattern for automatic MMSI-based identification
- Update frontend types and popup styling for adversary/allied/ROC
color distinction
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Infer country and military force (PLA, JSDF, ROK, ROC) from ICAO hex
address blocks when the flag field is Unknown. Extract and extend aircraft
model classification to cover East Asian fighters, cargo, recon, and
tanker types with hyphen-normalized matching.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 18 US military bases (Japan, Guam, South Korea, Hawaii, Diego Garcia)
as a toggleable map layer. Follows the existing data center layer pattern:
static JSON → backend fetcher → slow-tier API → frontend GeoJSON layer.
Includes red circle markers with labels, click popups showing operator
and branch info, and a toggle in the left panel.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 5 East Asia-focused RSS feeds (FocusTaiwan, Kyodo, SCMP, The Diplomat,
Stars and Stripes) and 22 geographic keywords (Taiwan Strait, South/East
China Sea, Okinawa, Guam, military bases, etc.) to improve coverage of
Taiwan contingency scenarios.
Refactor keyword matching into a pure _resolve_coords() function with
longest-match-first sorting so specific locations like "Taiwan Strait"
are not absorbed by generic "Taiwan".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fetch interval: 30min → 24h (TLEs only update a few times daily)
- Add If-Modified-Since header for conditional requests (304 support)
- Remove 10-thread parallel blitz on TLE fallback API → sequential with 1s delay
- Increase timeout 5s → 15s (be patient with a free service)
- SGP4 propagation still runs every 60s — satellite positions stay live
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Former-commit-id: 67b7654b6cc2d05c0a8ff00faad7c45c9cf2aa2d
BigBodyCobain