Private gate messages and offline DMs now ride the Infonet hashchain
as ciphertext-only events, replicated across nodes via private
transports (Tor onion / RNS / loopback) and decrypted only by parties
holding the gate or recipient keys.
Hashchain core (mesh_hashchain.py)
----------------------------------
* New ``append_private_gate_message`` and ``append_private_dm_message``
append paths with full signature verification, public-key binding,
revocation check, and replay protection in a dedicated sequence
domain (so a gate post does not consume the author's public broadcast
sequence, and a DM cannot replay-block a public message at sequence=1).
* Fork validation and full-chain validation now accept the gate
signature compatibility variants — older signatures that canonicalize
with/without epoch or reply_to still verify, so a re-sync from an
older peer doesn't reject still-valid history.
* DM hashchain spool: capped at 2 active sealed offline DMs per
recipient mailbox, plus a per-(sender, recipient) cap so one prolific
sender can't consume both slots. 1-hour TTL on the cap counter.
Spool intentionally small — it's an offline bootstrap channel,
not a persistent mailbox.
* Rebuild-state preserves the gate sequence domain across reloads so
a chain reload doesn't accidentally let an old gate sequence
replay-collide on next append.
Schema enforcement (mesh_schema.py)
-----------------------------------
* Private gate + DM payloads have closed allowlists of fields.
Plaintext keys (``message``, ``plaintext``, ``_local_plaintext``,
``_local_reply_to``) are explicit rejection-bait — they raise before
the event ever touches the chain.
* DM ciphertext + nonce must look like base64-ish sealed bytes;
obvious base64-encoded plaintext shapes are rejected.
* ``transport_lock`` required: DM hashchain spool requires
``private_strong``; gate accepts ``private``/``private_strong``/
``rns``/``onion``.
Defense-in-depth at the network layer (main.py + mesh_public.py)
----------------------------------------------------------------
* ``_infonet_sync_response_events`` now silently redacts private events
(gate_message + dm_message) unless the request looks like a loopback /
onion / RNS / private transport caller. If an operator accidentally
exposes :8000 to the public internet, an external puller gets
public events only — never ciphertext.
* ``_sync_from_peer`` raises ``PeerSyncRateLimited`` for 429 (handled
as 4-tuple return with retry_after_s) and ``PeerSyncHTTPError`` for
other non-200 statuses (handled by ``_run_public_sync_cycle`` to
honor server cooldown hints even outside the 429 path).
DM relay hydration (main.py)
-----------------------------
* New ``_hydrate_dm_relay_from_chain``: when accepted dm_message chain
events arrive on a node, they get deposited into the local DM relay
store with a deterministic sender_token_hash so re-sync of the same
event is idempotent. Recipients see the ciphertext as a normal DM
on their next poll and decrypt with their existing recipient key.
Other surfaces
--------------
* meshnode.bat / meshnode.sh now set ``MESH_INFONET_ALLOW_CLEARNET_SYNC=
false`` and the participant runtime flags by default so a freshly
spun-up node defaults to private-only sync.
* InfonetTerminal/InfonetShell.tsx adds a gate directory renderer for
the new private-gate workflow.
* docker-compose.relay.yml binds the relay backend to 127.0.0.1:8000
only; Tor's hidden service forwards onion traffic into 127.0.0.1.
Public clearnet :8000 stays off the network edge.
Tests
-----
* 7 new tests in test_private_gate_hashchain.py + test_private_dm_
hashchain.py covering: gate fork accepts ciphertext propagation,
gate fork rejects plaintext, append rejects plaintext before
normalize, append requires private_strong, append rejects
non-sealed ciphertext shape, DM spool 2-per-recipient + 1-per-pair
cap, DM hydration delivers to poll/claim.
* Updated test_mesh_node_bootstrap_runtime.py covers 429 backoff via
PeerSyncRateLimited 4-tuple AND PeerSyncHTTPError exception.
* Updated test_s14b_public_sync_gate_filter.py + test_s9b_gate_store_
hydration.py + test_gate_write_cutover.py cover the new private
redaction on public sync responses.
* test_private_gate_hashchain.py + test_private_dm_hashchain.py:
10 passed locally.
* Combined mesh-relevant suite (the 5 modified existing tests +
2 new): 17 passed.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Background
==========
PR #285 set up the seed -> cache -> GDELT model for the carrier tracker
to address audit issues #244/#245/#246. The GDELT half of that pipeline
hits api.gdeltproject.org's doc API for headline-region keyword
matching -- low precision (false centroid positions per #245) AND
unreliable (the host times out from some networks, including Docker
Desktop on Windows).
USNI publishes a weekly Fleet & Marine Tracker with explicit prose like:
"The Gerald R. Ford Carrier Strike Group is operating in the Red Sea"
"Aircraft carrier USS George Washington (CVN-73) is in port in
Yokosuka, Japan"
That is a strictly better source for U.S. Navy carrier positions:
authoritative, deterministically parseable, weekly cadence.
What this PR does
=================
New module: backend/services/fetchers/usni_fleet_tracker.py
- Pulls USNI's WordPress RSS feeds (site-wide + category, unioned).
- Picks the most recent fleet-tracker post by parsed pubDate.
- For each carrier in the registry, scans the article body for
"is operating in / is in port in / returned to / transiting" near
the carrier's name, hull code, or "<name> Carrier Strike Group"
variant. Captures the region/port phrase that follows.
- Maps the region phrase to coordinates via the existing
REGION_COORDS table, with a USNI-phrase alias table for the
specific wording USNI uses ("Yokosuka, Japan", "Norfolk, Va.",
"Naval Station San Diego", "5th Fleet AOR", etc.).
- Returns {hull: position_entry} with position_confidence="recent"
and position_source_at = the article's actual publication
timestamp (not now()).
Politeness
----------
Uses outbound_user_agent("usni-fleet-tracker") so USNI sees a
per-install Shadowbroker identifier (Round 7a / PR #292). The
article body pages return 403 to non-browser UAs; the WordPress RSS
feed serves the full <content:encoded> body and is the supported
aggregator path. No browser UA spoofing.
carrier_tracker.update_carrier_positions() now runs three phases:
1. Bootstrap from cache (or seed on first run).
2. USNI fleet tracker -- PRIMARY high-confidence source.
3. GDELT -- SECONDARY backfill; can NOT demote a "recent" USNI
position to an "approximate" GDELT headline match.
Verified live: 6 of 11 carriers picked up real May 18, 2026 positions
on first refresh (Eisenhower, Ford, Bush, Roosevelt, Lincoln,
Washington). The other 5 weren't mentioned in this week's article
(they're in port at homeports with no deployment changes) and kept
their cache entries -- which is the correct seed/cache contract from
PR #285.
Other small fixes bundled in
============================
docker-compose.yml: add the 6 third-party-fetcher opt-in env vars
(PREDICTION_MARKETS_ENABLED, FINANCIAL_ENABLED, FIMI_ENABLED,
NUFORC_ENABLED, NEWS_ENABLED, CROWDTHREAT_ENABLED). They were
documented in .env.example but never wired through compose, so setting
them in .env had no effect.
frontend/src/components/TopRightControls.tsx: fix 6 broken i18n keys
that were showing as raw "terminal.term1" / "terminal.cleanupDetail" /
"node.soloReady" placeholders in the INFONET TERMINAL modal. The
translation files have these strings under different key names; the
component now calls the right ones. Full-file sweep confirmed every
other t('...') key in the whole frontend resolves cleanly.
Before this change, every peer-push HMAC was derived from the single
fleet-shared MESH_PEER_PUSH_SECRET. The receiver could prove "this
request was signed by someone who knows the fleet secret" but it could
NOT prove which peer signed it. Any peer that knew the global secret
could compute the expected HMAC for any other peer URL and forge a
push pretending to be that peer.
Fix: introduce MESH_PEER_SECRETS, an optional comma-separated
url=secret map. When a peer URL appears in the map, only the listed
per-peer secret is accepted for it -- the global secret is ignored for
that specific URL. Peer A no longer knows peer B's secret, so peer A
cannot forge a push claiming to be peer B.
The new helper resolve_peer_key_for_url() in mesh_crypto.py wraps the
lookup and is called from every existing peer-push call site:
- backend/auth.py:_verify_peer_push_hmac (receiver)
- backend/main.py:_http_peer_push_loop (Infonet event push)
- backend/main.py:_http_gate_pull_loop (gate event pull)
- backend/main.py:_http_gate_push_loop (gate event push)
- backend/services/mesh/mesh_router.py (two transports, push)
- backend/services/mesh/mesh_hashchain.py (gate wire ref key)
- backend/services/mesh/mesh_wormhole_prekey.py (peer prekey lookup)
Zero hostility, by design:
- Single-peer installs leave MESH_PEER_SECRETS empty -> resolver falls
back to MESH_PEER_PUSH_SECRET -> behavior is byte-for-byte unchanged.
- Multi-peer installs that haven't migrated yet behave exactly as
before.
- Multi-peer installs that DO migrate set MESH_PEER_SECRETS on both
ends of each peering and immediately close the impersonation surface
for those URLs. Migration is incremental: unlisted peers keep using
the global secret.
Tests in backend/tests/test_per_peer_secret_resolver.py:
- env parsing (default, override, whitespace, malformed entries, cache)
- precedence: per-peer beats global
- migration window: unlisted peer falls back to global
- IMPERSONATION REFUSAL: peer A with global-secret-only cannot forge
HMAC for peer B that has a per-peer secret configured
- IMPERSONATION REFUSAL: peer A with its OWN per-peer secret cannot
forge HMAC for peer B
- positive control: legitimate peer B request verifies
- zero-behavior-change: single-peer install produces the same key bytes
as before the change
Credit: tg12 (external security audit, P1/High/High confidence)
Tightens the bridge-trust check so a connection on the Docker bridge
is only granted local-operator status when its source IP matches a
configured frontend container hostname (default: `frontend` + the
shipped `container_name` `shadowbroker-frontend`). Previously, when
`SHADOWBROKER_TRUST_DOCKER_BRIDGE_LOCAL_OPERATOR=1` was set, ANY IP
in the 172.16.0.0/12 range was granted local-operator privileges —
on a shared Docker host that included any unrelated container on the
same bridge.
Operators with renamed services can list new hostnames via the new
`SHADOWBROKER_TRUSTED_FRONTEND_HOSTS` env var (comma-separated). DNS
resolution is cached for 30s; if Docker DNS can't resolve any of the
configured names we fail closed and refuse the bridge entirely.
Single-user installs see no behavior change — the default-named
frontend container still resolves and is still trusted.
Credit: tg12 (external security audit)
Add Tor/onion runtime wiring and faster Infonet node status refresh.
Keep node bootstrap state clearer across Docker and local runtimes.
Use selected aircraft trail history for cumulative tracked-aircraft emissions.
Reduce cold-start stalls by raising the default backend memory limit, bounding heavy feed concurrency, preserving non-empty startup caches, and refreshing working news feeds. Fix the Next API proxy for Docker control-plane writes by stripping unsupported hop/body headers and forwarding small request bodies safely. Keep the dashboard dynamic so production users do not get stuck on a cached startup shell.
Allow the bundled Docker frontend proxy to reach local-operator endpoints through the private compose bridge without trusting LAN clients. This restores Time Machine, MeshChat key creation, AI pins/layers, and related local controls in Docker installs. Refresh first-run guidance so Docker users know to configure OpenSky and AIS keys through .env.
Seed safe static backend data into fresh Docker volumes, tighten Docker build-context exclusions, avoid optional env warnings, and make the frontend healthcheck use the IPv4 loopback path that works inside the container.
GHCR requires authentication even for public packages on some systems.
CI now pushes to both GHCR and Docker Hub. docker-compose.yml and Helm
chart point to Docker Hub where anonymous pulls always work. Build
directives kept as fallback for source-based builds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use cipher0's existing MESH_PEER_PUSH_SECRET so nodes connect
to the relay out of the box without configuration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 — Transport layer fix:
- Bake in default MESH_PEER_PUSH_SECRET so peer push, real-time
propagation, and pull-sync all work out of the box instead of
silently no-oping on an empty secret.
- Pass secret through docker-compose.yml for container deployments.
Phase 2 — Per-gate content keys:
- Generate a cryptographically random 32-byte secret per gate on
creation (and backfill existing gates on startup).
- Upgrade HKDF envelope encryption to use per-gate secret as IKM
so knowing a gate name alone no longer decrypts messages.
- 3-tier decryption fallback (phase2 key → legacy name-only →
legacy node-local) preserves backward compatibility.
- Expose gate_secret via list_gates API for authorized members.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add FINNHUB_API_KEY to docker-compose.yml so financial ticker works
in Docker deployments
- Update default layer config: planes/ships ON, satellites only for
space, no fire hotspots, military bases + internet outages for infra,
all SIGINT except HF digital spots
- Add MapLibre native clustering to APRS markers (matches Meshtastic)
with cluster radius 42, breaks apart at zoom 8
On a fresh Docker (or local) install, MESH_RELAY_PEERS was empty and
no bootstrap manifest existed, leaving the Infonet node with zero
peers to sync from — causing perpetual "RETRYING" status.
Set cipher0.shadowbroker.info:8000 as the default relay peer in both
the config defaults and docker-compose.yml so new installations sync
immediately after activating the wormhole.
docker compose pull was skipping with "No image to be pulled" because
the build: sections made Compose treat local builds as authoritative.
Moved build config to docker-compose.build.yml for developers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Users pulling pre-built images need the image: field. Increased backend
health check start_period from 30s to 60s with 5 retries to handle
slower startup environments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gate messages now propagate via the Infonet hashchain as encrypted blobs — every node syncs them
through normal chain sync while only Gate members with MLS keys can decrypt. Added mesh reputation
system, peer push workers, voluntary Wormhole opt-in for node participation, fork recovery,
killwormhole scripts, obfuscated terminology, and hardened the self-updater to protect encryption
keys and chain state during updates.
New features: Shodan search, train tracking, Sentinel Hub imagery, 8 new intelligence layers,
CCTV expansion to 11,000+ cameras across 6 countries, Mesh Terminal CLI, prediction markets,
desktop-shell scaffold, and comprehensive mesh test suite (215 frontend + backend tests passing).
Community contributors: @wa1id, @AlborzNazari, @adust09, @Xpirix, @imqdcr, @csysp, @suranyami,
@chr0n1x, @johan-martensson, @singularfailure, @smithbh, @OrfeoTerkuci, @deuza, @tm-const,
@Elhard1, @ttulttul
Change backend context from . to ./backend in docker-compose.
This is necessary for copying the pyproject.toml and uv.lock files from project root level
Two bugs introduced by the Next.js proxy Route Handler:
1. ERR_CONTENT_DECODING_FAILED — Node.js fetch() automatically
decompresses gzip/br responses from the backend, but the proxy was
still forwarding Content-Encoding and Content-Length headers to the
browser. The browser would then try to decompress already-decompressed
data and fail. Fixed by stripping Content-Encoding and Content-Length
from upstream response headers.
2. BACKEND_URL shell env leak into Docker Compose — docker-compose.yml
used ${BACKEND_URL:-http://backend:8000}, which was being overridden
by BACKEND_URL=http://localhost:8000 set in .mise.local.toml for local
dev. Inside the frontend container, localhost:8000 does not exist,
causing all proxied requests to return 502. Fixed by hardcoding
http://backend:8000 in docker-compose.yml so the shell environment
cannot override it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Former-commit-id: 036c62d2c0
Previously, NEXT_PUBLIC_API_URL was a build-time Next.js variable, making
it impossible to configure the backend URL in docker-compose `environment`
without rebuilding the image.
This change introduces a proper server-side proxy:
- next.config.ts: adds a rewrite rule that forwards all /api/* requests
to BACKEND_URL (read at server startup, not baked at build time).
Defaults to http://localhost:8000 so local dev works without config.
- api.ts: API_BASE is now an empty string — all fetch calls use relative
/api/... paths, which the Next.js server proxies to the backend.
- docker-compose.yml: replaces NEXT_PUBLIC_API_URL build arg with a
runtime BACKEND_URL env var defaulting to http://backend:8000, using
Docker's internal networking. Port 8000 no longer needs to be exposed.
- README: updates Docker setup docs, standalone compose example, and
environment variable reference to reflect BACKEND_URL.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Former-commit-id: a3b18e23c1
New features:
- Custom RSS Feed Manager: add/remove/prioritize up to 20 news sources
from the Settings panel with weight levels 1-5. Persists across restarts.
- Global Data Center Map Layer: 2,000+ DCs plotted worldwide with clustering,
server-rack icons, and automatic internet outage cross-referencing.
- Imperative map rendering: high-volume layers bypass React reconciliation
via direct setData() calls with debounced updates on dense layers.
- Enhanced /api/health with per-source freshness timestamps and counts.
Fixes:
- Data center coordinates fixed for 187 Southern Hemisphere entries
- Docker CORS_ORIGINS passthrough in docker-compose.yml
- Start scripts warn on Python 3.13+ compatibility
- Settings panel redesigned with tabbed UI (API Keys / News Feeds)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Former-commit-id: 950c308f04
Shadowbroker