Persist OPENCLAW_HMAC_SECRET to data/openclaw.env so empty Docker env vars no longer block auth after UI bootstrap. Add verify_hmac.py, silence Rust warnings (#423), and document Docker signing (#424).
Co-authored-by: Cursor <cursoragent@cursor.com>
Expose observed aircraft/vessel paths, route enrichment, VIP metadata, datalink, and nearby context so agents can reconstruct movement without full telemetry dumps.
Co-authored-by: Cursor <cursoragent@cursor.com>
Parse common position, OOOI, performance, and ops formats across major carriers while hiding VDL binary fragments and duplicate frames from the dossier feed.
Co-authored-by: Cursor <cursoragent@cursor.com>
Bulk-ingest Airframes messages on a rate-limited staggered queue with instant cache lookups and priority per-aircraft refresh when opening a dossier; add Meshtastic manual SCAN PLANET control in the SIGINT panel.
Co-authored-by: Cursor <cursoragent@cursor.com>
Import oligarchs, royals, and curated celebrities from plane-alert-db while excluding joke tag labels from tracked_names. Sync plane_alert_db.json metadata, add import scripts, and map oligarch/royal/celebrity colors in the legend.
Co-authored-by: Cursor <cursoragent@cursor.com>
DGT migrated its public traffic site in 2026: the old image path
infocar.dgt.es/etraffic/data/camaras/{id}.jpg now 302-redirects to the
new etrafficWEB SPA and no longer serves JPEGs, so every DGT camera
showed FEED UNAVAILABLE (proxy and direct source both failed).
New working pattern: https://etraffic.dgt.es/camarasEtraffic/{id}.jpg
- cctv_pipeline.py: DGTNationalIngestor media_url + probe Referer
- routers/cctv.py & main.py: allowlist etraffic.dgt.es + dgt-spain proxy profile
Add Earthdata token entry on the SAR tab with accurate Mode B status, expose optional FIRMS_MAP_KEY in API settings, and remove the frontend operator-unlock gate that blocked localhost saves. Run network-heavy layer-enable fetches on a background executor with frontend retry polling so FIRMS toggles no longer freeze the single API worker.
Drop duplicate slow-tier weather/ukraine jobs, gate correlations when off, slim health probes, keyed layer-panel subscriptions, align backend layer defaults with the dashboard, and fetch CCTV/FIRMS/PSK/etc. synchronously on enable so toggles stay responsive without background prefetch waste.
Replace spoofable Host/Origin WebSocket auth with short-lived bootstrap
tokens minted over the existing local-operator HTTP path. Docker/browser
shell sessions prefetch a token before connecting; loopback peers remain
unchanged.
Also bump backend ws to 8.21.0 and refresh frontend lockfile to clear
npm audit findings (dev toolchain only for frontend).
Fixes#405, #406, #407
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gdelt): publish enriched copies instead of mutating live features
_enrich_gdelt_titles_background ran in a daemon thread that mutated the
nested properties dicts of GDELT features already published into
latest_data[gdelt]. HTTP readers hold live references to those dicts and
serialize them outside the data lock, so the in-place mutation raced the
serializer and raised RuntimeError: dictionary changed size during
iteration on /api/live-data/slow and /api/bootstrap/critical.
Enrich deep copies instead and atomically swap the top-level key under
_data_lock, with an identity guard so a newer fetch_gdelt() is not clobbered.
Honors the replace-don't-mutate contract documented in fetchers/_store.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(gdelt): regression test for background enrichment isolation
Asserts _enrich_gdelt_titles_background does not mutate already-published features and instead atomically swaps latest_data["gdelt"] with enriched copies (with the identity guard). Locks in the fix for the dictionary-changed-size race.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
get_latest_data_deepcopy_snapshot deep-copies layers outside the data lock; a writer mutating a nested object in place races it and raises "dictionary changed size during iteration" (500 on /api/health, /api/live-data). Two changes: (1) _merge_sigint_snapshot now shallow-copies each entry so latest_data["sigint"] no longer aliases the SIGINT bridge dicts or the meshtastic_map_nodes layer (the concrete offender); (2) the snapshot retries a few times as defense-in-depth for any other in-place mutator. Plus regression tests.
Adds connect-contact HTTP endpoint with cached-bundle support, subprocess contact send via docker cp bundle file, and direct Tor prekey fetch to avoid wedging single-worker uvicorn.
Co-authored-by: Cursor <cursoragent@cursor.com>
MLS export/reset and accept use live HTTP so uvicorn privacy-core state stays consistent; relay persistence and sender_seal fixes enable invite-accept-shared decrypt across onion peers. Adds participant/e2e compose overlays and harness recovery with optional Tor-only replicate mode.
Co-authored-by: Cursor <cursoragent@cursor.com>
Retry announce/manifest while Tor circuits warm on NODE and startup bootstrap.
Add verify_swarm_fresh_participant.py for empty-volume GHCR smoke tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
Auto-relay connect DMs with End Contact severing, signed fleet prekey lookup,
OpenClaw private Infonet channel intents, headless relay Tor bootstrap on redeploy,
and swarm/DM live verification scripts.
Co-authored-by: Cursor <cursoragent@cursor.com>
Ship sb-testnet fleet defaults, swarm/join API, NODE launcher registration step, and meshnode script defaults so users discover peers via the signed seed manifest without manual peer lists.
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed peer manifest pull/announce on the seed, immediate hashchain push for gate messages, seed-only Docker defaults, and stale-genesis sync diagnostics.
Co-authored-by: Cursor <cursoragent@cursor.com>
Rename Mesh Chat to Meshtastic Chat, embed the Infonet terminal with Arti/Tor warmup, improve the agent shell PTY (git in the backend image, operator PATH), and add docker-compose.override for local image builds. Gitignore Hermes Agent runtime installs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Uses a local-operator WebSocket bash session, keeps the map interactive, and SNAP docks the shell back into Mesh Chat instead of a floating blurred panel.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add Telegram OSINT with hourly incremental t.me scraping, metro geocoding
separate from news centroids, threat-intercept popup UI with inline media,
and HTML markers above alert boxes so pins stay clickable. Expose GFW_API_TOKEN
in onboarding and Settings Maritime; harden GFW/CCTV/geo fetchers. Port Osiris-
derived recon, SCM, entity graph, malware/cyber feeds, sanctions, and submarine
cable layers with tests and documentation.
Co-authored-by: Cursor <cursoragent@cursor.com>
Align full /api/live-data with slow-tier orjson options, remove dead main.py duplicate, cap slow batches to pool size, cancel queued work on timeout, and stop retrying HTTP 4xx/5xx.
Co-authored-by: Cursor <cursoragent@cursor.com>
Default python main.py to loopback, deep-copy dashboard snapshots outside the store lock with ETag on full live-data, and route GDELT/LiveUAMap/CCTV/slow-tier work through an isolated executor so Playwright jobs cannot starve fast-tier workers.
Co-authored-by: Cursor <cursoragent@cursor.com>
Operators enable Polymarket/Kalshi correlation from Global Threat Intercept with a consent dialog; polls use a jittered schedule separate from the slow tier. Right-click Sentinel imagery returns up to three signed scenes again.
Co-authored-by: Cursor <cursoragent@cursor.com>
Each install pulls ~60-day sightings from nuforc.org every Monday; disk cache
matches weekly cadence so users keep current pins between restarts.
Co-authored-by: Cursor <cursoragent@cursor.com>
Filter stale rows out of nuforc_recent_sightings.json on load; add requests-based
live scrape when curl is disabled; daily scheduler rebuild instead of weekly-only.
Co-authored-by: Cursor <cursoragent@cursor.com>
- User-Agent is per-install handle only (no Shadowbroker product token)
- LiveUAMap: Windows UI consent when enabling Global Incidents; env override
- Meshtastic callsign upstream header off by default (opt-in true)
- Expanded docs/OUTBOUND_DATA.md and README link for CCTV, basemap, Broadcastify
Co-authored-by: Cursor <cursoragent@cursor.com>
Operators can set DEEPSTATE_MIRROR_COMMIT for immutable frontline ingest; Madrid KML tries HTTPS then HTTP without changing camera image URLs or proxy Referers.
Co-authored-by: Cursor <cursoragent@cursor.com>
Private gate messages and offline DMs now ride the Infonet hashchain
as ciphertext-only events, replicated across nodes via private
transports (Tor onion / RNS / loopback) and decrypted only by parties
holding the gate or recipient keys.
Hashchain core (mesh_hashchain.py)
----------------------------------
* New ``append_private_gate_message`` and ``append_private_dm_message``
append paths with full signature verification, public-key binding,
revocation check, and replay protection in a dedicated sequence
domain (so a gate post does not consume the author's public broadcast
sequence, and a DM cannot replay-block a public message at sequence=1).
* Fork validation and full-chain validation now accept the gate
signature compatibility variants — older signatures that canonicalize
with/without epoch or reply_to still verify, so a re-sync from an
older peer doesn't reject still-valid history.
* DM hashchain spool: capped at 2 active sealed offline DMs per
recipient mailbox, plus a per-(sender, recipient) cap so one prolific
sender can't consume both slots. 1-hour TTL on the cap counter.
Spool intentionally small — it's an offline bootstrap channel,
not a persistent mailbox.
* Rebuild-state preserves the gate sequence domain across reloads so
a chain reload doesn't accidentally let an old gate sequence
replay-collide on next append.
Schema enforcement (mesh_schema.py)
-----------------------------------
* Private gate + DM payloads have closed allowlists of fields.
Plaintext keys (``message``, ``plaintext``, ``_local_plaintext``,
``_local_reply_to``) are explicit rejection-bait — they raise before
the event ever touches the chain.
* DM ciphertext + nonce must look like base64-ish sealed bytes;
obvious base64-encoded plaintext shapes are rejected.
* ``transport_lock`` required: DM hashchain spool requires
``private_strong``; gate accepts ``private``/``private_strong``/
``rns``/``onion``.
Defense-in-depth at the network layer (main.py + mesh_public.py)
----------------------------------------------------------------
* ``_infonet_sync_response_events`` now silently redacts private events
(gate_message + dm_message) unless the request looks like a loopback /
onion / RNS / private transport caller. If an operator accidentally
exposes :8000 to the public internet, an external puller gets
public events only — never ciphertext.
* ``_sync_from_peer`` raises ``PeerSyncRateLimited`` for 429 (handled
as 4-tuple return with retry_after_s) and ``PeerSyncHTTPError`` for
other non-200 statuses (handled by ``_run_public_sync_cycle`` to
honor server cooldown hints even outside the 429 path).
DM relay hydration (main.py)
-----------------------------
* New ``_hydrate_dm_relay_from_chain``: when accepted dm_message chain
events arrive on a node, they get deposited into the local DM relay
store with a deterministic sender_token_hash so re-sync of the same
event is idempotent. Recipients see the ciphertext as a normal DM
on their next poll and decrypt with their existing recipient key.
Other surfaces
--------------
* meshnode.bat / meshnode.sh now set ``MESH_INFONET_ALLOW_CLEARNET_SYNC=
false`` and the participant runtime flags by default so a freshly
spun-up node defaults to private-only sync.
* InfonetTerminal/InfonetShell.tsx adds a gate directory renderer for
the new private-gate workflow.
* docker-compose.relay.yml binds the relay backend to 127.0.0.1:8000
only; Tor's hidden service forwards onion traffic into 127.0.0.1.
Public clearnet :8000 stays off the network edge.
Tests
-----
* 7 new tests in test_private_gate_hashchain.py + test_private_dm_
hashchain.py covering: gate fork accepts ciphertext propagation,
gate fork rejects plaintext, append rejects plaintext before
normalize, append requires private_strong, append rejects
non-sealed ciphertext shape, DM spool 2-per-recipient + 1-per-pair
cap, DM hydration delivers to poll/claim.
* Updated test_mesh_node_bootstrap_runtime.py covers 429 backoff via
PeerSyncRateLimited 4-tuple AND PeerSyncHTTPError exception.
* Updated test_s14b_public_sync_gate_filter.py + test_s9b_gate_store_
hydration.py + test_gate_write_cutover.py cover the new private
redaction on public sync responses.
* test_private_gate_hashchain.py + test_private_dm_hashchain.py:
10 passed locally.
* Combined mesh-relevant suite (the 5 modified existing tests +
2 new): 17 passed.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Pre-fix the emissions tooltip only showed the per-hour *rate* — what most
users actually want is the cumulative *amount* burned. This adds running
totals computed by multiplying the model-based rate by the elapsed
observation time since we first saw the airframe.
New module ``flight_observations.py``:
* Tracks first_seen_at + last_seen_at per icao24 hex.
* Re-opens a fresh session when an aircraft is unseen for > 15 min
(treated as a new flight — landed and took off, or transited a dead
zone). Prevents the cumulative counter from resetting mid-flight if
the trail-rendering cache prunes the trail.
* Clamps elapsed time to 24h max so clock skew can't produce comically
large numbers.
* Pruned every 5 min via a new scheduler job (mirrors ais_prune cadence).
flights.py + military.py emission enrichment now also attaches:
* observed_seconds — how long we've been tracking this airframe.
* fuel_gallons_burned — rate * elapsed_h.
* co2_kg_emitted — rate * elapsed_h.
The existing per-hour rate fields stay in the dict for backward compat
and are shown as small secondary context in the tooltip.
Frontend EmissionsEstimateBlock (NewsFeed.tsx) now prominently shows
the cumulative totals with the rate as smaller context underneath plus
"Observed in flight for Xh Ym". When observed_seconds is 0 (first refresh)
it renders "Just observed · totals will appear on next refresh" instead
of a misleading "0 gal".
12 backend tests cover record/accumulate/reset, the 24h clamp, prune,
case-insensitive key normalization, and end-to-end emission integration
in _classify_and_publish.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When stream.aisstream.io is unreachable (cert outage, server down — see
2026-05-20 and 2026-05-23 events) the ships layer goes empty. This adds
a slow REST fallback to data.aishub.net so the layer stays populated in
degraded mode.
Behavior:
* Opt-in via AISHUB_USERNAME (free registration at aishub.net/api).
Without the env var the fetcher is a no-op.
* Default poll cadence 20 min — well inside their free-tier limits, gives
ships time to move enough to look "alive". Configurable via
AISHUB_POLL_INTERVAL_MINUTES, clamped to [1, 360].
* Internal gate: skips the poll entirely when the WebSocket primary is
currently connected. Stomping fresh live data with 20-min-old REST
data would be worse than leaving it alone.
* Vessels merge into the shared _vessels dict with source="aishub" so
the existing UI / health tooling can attribute the provider.
* Live data wins races: if a WebSocket update for the same MMSI lands in
the last 1s, we don't overwrite with the slower REST record.
Scheduler job runs every AISHUB_POLL_INTERVAL_MINUTES minutes alongside
the existing ais_prune job in data_fetcher.py.
24 tests cover gating (no-username, primary-connected), response parsing
(success / error / empty / malformed / unexpected shape), record
normalization (sentinels, missing fields, range checks, AIS @ padding),
poll interval clamping, and end-to-end merge with live-data-wins.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
On 2026-05-23, stream.aisstream.io went fully offline (TCP timeouts on port
443). The backend kept respawning the node WebSocket proxy every few
seconds with nothing arriving. From the operator's POV the ships layer
silently went empty — no banner, no log surfacing, no way to tell whether
it was their config / network / viewport filter / upstream.
Backend:
* ais_proxy_status() now also returns:
- connected (bool): true when a vessel message arrived in last 60s
- last_msg_age_seconds (int | None)
- proxy_spawn_count (int): proxy respawns — sustained growth without
connected means upstream is dead
* /api/health escalates top status to "degraded" when AIS_API_KEY is set
but the proxy is currently disconnected. Existing degraded_tls signal
preserved.
Frontend:
* useAisUpstreamHealth hook polls /api/health every 30s, derives the
outage state. Defensively only reports outage once spawn_count > 0 so
operators who haven't opted in don't see the banner.
* AisUpstreamBanner component renders a dismissible amber notice
"Ship data temporarily unavailable — AISStream upstream is offline"
mounted on the main app shell.
7 backend tests pin the status-shape contract and the /api/health
escalation behavior in both with-key and without-key configurations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pre-fix, adsb.lol records (the primary source for most flights) carried
no source marker. OpenSky records got is_opensky: True and supplementals
got supplemental_source, so any UI inspecting source labels saw
OpenSky/airplanes.live records as explicitly tagged and adsb.lol records
as "unlabeled" — making it look like adsb.lol wasn't being used at all
even though it's the primary source.
Changes:
* _fetch_adsb_lol_regions stamps source="adsb.lol" on each aircraft
before returning, so the tag survives the OpenSky dedupe-by-hex merge.
* OpenSky records get source="OpenSky" (alongside is_opensky=True for
back-compat).
* military fetcher tags source on both adsb.lol and airplanes.live
records before they're merged, and propagates source into the
military_flights and uavs output dicts.
* _classify_and_publish promotes the explicit source field into the
published flight dict. Falls back to legacy supplemental_source if
source is absent. Final fallback "adsb.lol" preserves prior behavior
for any caller synthesizing records without going through a fetcher.
8 new tests cover the published-dict propagation, OpenSky tagging,
supplemental fallback, explicit-wins precedence, default behavior, the
adsb.lol regional fetcher tagging, and the military output dict.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
