Compare commits

...

7 Commits

Author SHA1 Message Date
BigBodyCobain 30aa30204f feat(infonet): private gate + DM hashchain spool with hardened propagation
Private gate messages and offline DMs now ride the Infonet hashchain
as ciphertext-only events, replicated across nodes via private
transports (Tor onion / RNS / loopback) and decrypted only by parties
holding the gate or recipient keys.

Hashchain core (mesh_hashchain.py)
----------------------------------

* New ``append_private_gate_message`` and ``append_private_dm_message``
  append paths with full signature verification, public-key binding,
  revocation check, and replay protection in a dedicated sequence
  domain (so a gate post does not consume the author's public broadcast
  sequence, and a DM cannot replay-block a public message at sequence=1).
* Fork validation and full-chain validation now accept the gate
  signature compatibility variants — older signatures that canonicalize
  with/without epoch or reply_to still verify, so a re-sync from an
  older peer doesn't reject still-valid history.
* DM hashchain spool: capped at 2 active sealed offline DMs per
  recipient mailbox, plus a per-(sender, recipient) cap so one prolific
  sender can't consume both slots. 1-hour TTL on the cap counter.
  Spool intentionally small — it's an offline bootstrap channel,
  not a persistent mailbox.
* Rebuild-state preserves the gate sequence domain across reloads so
  a chain reload doesn't accidentally let an old gate sequence
  replay-collide on next append.

Schema enforcement (mesh_schema.py)
-----------------------------------

* Private gate + DM payloads have closed allowlists of fields.
  Plaintext keys (``message``, ``plaintext``, ``_local_plaintext``,
  ``_local_reply_to``) are explicit rejection-bait — they raise before
  the event ever touches the chain.
* DM ciphertext + nonce must look like base64-ish sealed bytes;
  obvious base64-encoded plaintext shapes are rejected.
* ``transport_lock`` required: DM hashchain spool requires
  ``private_strong``; gate accepts ``private``/``private_strong``/
  ``rns``/``onion``.

Defense-in-depth at the network layer (main.py + mesh_public.py)
----------------------------------------------------------------

* ``_infonet_sync_response_events`` now silently redacts private events
  (gate_message + dm_message) unless the request looks like a loopback /
  onion / RNS / private transport caller. If an operator accidentally
  exposes :8000 to the public internet, an external puller gets
  public events only — never ciphertext.
* ``_sync_from_peer`` raises ``PeerSyncRateLimited`` for 429 (handled
  as 4-tuple return with retry_after_s) and ``PeerSyncHTTPError`` for
  other non-200 statuses (handled by ``_run_public_sync_cycle`` to
  honor server cooldown hints even outside the 429 path).

DM relay hydration (main.py)
-----------------------------

* New ``_hydrate_dm_relay_from_chain``: when accepted dm_message chain
  events arrive on a node, they get deposited into the local DM relay
  store with a deterministic sender_token_hash so re-sync of the same
  event is idempotent. Recipients see the ciphertext as a normal DM
  on their next poll and decrypt with their existing recipient key.

Other surfaces
--------------

* meshnode.bat / meshnode.sh now set ``MESH_INFONET_ALLOW_CLEARNET_SYNC=
  false`` and the participant runtime flags by default so a freshly
  spun-up node defaults to private-only sync.
* InfonetTerminal/InfonetShell.tsx adds a gate directory renderer for
  the new private-gate workflow.
* docker-compose.relay.yml binds the relay backend to 127.0.0.1:8000
  only; Tor's hidden service forwards onion traffic into 127.0.0.1.
  Public clearnet :8000 stays off the network edge.

Tests
-----

* 7 new tests in test_private_gate_hashchain.py + test_private_dm_
  hashchain.py covering: gate fork accepts ciphertext propagation,
  gate fork rejects plaintext, append rejects plaintext before
  normalize, append requires private_strong, append rejects
  non-sealed ciphertext shape, DM spool 2-per-recipient + 1-per-pair
  cap, DM hydration delivers to poll/claim.
* Updated test_mesh_node_bootstrap_runtime.py covers 429 backoff via
  PeerSyncRateLimited 4-tuple AND PeerSyncHTTPError exception.
* Updated test_s14b_public_sync_gate_filter.py + test_s9b_gate_store_
  hydration.py + test_gate_write_cutover.py cover the new private
  redaction on public sync responses.
* test_private_gate_hashchain.py + test_private_dm_hashchain.py:
  10 passed locally.
* Combined mesh-relevant suite (the 5 modified existing tests +
  2 new): 17 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 21:09:08 -06:00
Shadowbroker fb97042c01 Update README.md
Elaborated on Tor and Reticulum usage.
2026-05-24 11:08:05 -06:00
Shadowbroker 2616a6c9e3 Update README.md 2026-05-24 11:06:40 -06:00
Shadowbroker a930497e14 fix(start-scripts): find bundled privacy_core.dll next to script (#319) (#324)
* fix(start-scripts): find bundled privacy_core.dll next to script

start.bat and start.sh only checked the source-tree DLL path
(``privacy-core/target/release/privacy_core.dll``), not the bundled
location where MSI/AppImage/DMG installers stage the library directly
next to the script in backend-runtime/.

Users running start.bat from inside an MSI install dir (a documented
workaround when the desktop shell crashes) saw a scary "install Rust"
warning even though the DLL was sitting right next to them. See issue
#319 for the user-reported confusion.

Fix: add a fallback check for the bundled location before falling
through to the "build privacy-core from source" warning. Source-tree
behavior unchanged — the source path is still preferred when present.

Also re-stamps the v0.9.81 source archive: ``release_digests.json``
v0.9.81 zip hash updated to point at the rebuilt source archive that
contains these script changes. MSI/EXE/sig hashes are unchanged (the
scripts live at the repo root, not inside the desktop bundle).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(#319): bundle start.bat + start.sh into the MSI/EXE installers

Follow-up to the start-script DLL fallback fix in the prior commit.

ChrisMTheMan's report on #319 made it clear the workaround flow was:

  1. MSI install crashes on launch (different bug, fixed in v0.9.81)
  2. User goes looking for start.bat to launch the backend manually
  3. start.bat isn't in their install dir, so they go fetch it from GitHub
  4. They get a working script but it doesn't know about the bundled
     privacy_core.dll layout, so they see a scary "install Rust" warning

The prior commit fixed step 4. This commit fixes step 3 — start.bat and
start.sh now ship inside the MSI/EXE installers (staged into
backend-runtime/ next to the privacy_core.dll they expect to find).
After the rebuild lands, an MSI user looking for these scripts finds
them right inside their install dir, already pointing at the correct
bundled DLL location.

What changed
------------

* ``build-backend-runtime.cjs`` now has a ``stageStartScripts()`` step
  that copies start.bat and start.sh from the repo root into the
  staged backend-runtime/. Preserves the executable bit on .sh under
  POSIX.

* ``release_digests.json`` v0.9.81 block hashes refreshed for the
  rebuilt MSI / EXE / source-zip (the scripts being bundled changed
  the MSI/EXE contents; the source zip also includes the start-script
  fix from the prior commit).

  ShadowBroker_v0.9.81.zip                  6.06 MB
    af8c87ccdece8fbb9aadc6be63cce10d3fcba74e6d87ef83289dda6d555fd270
  ShadowBroker_0.9.81_x64_en-US.msi       122.4 MB
    8977c9a1c54e1f0d030436be9c4e3d81d766cc0080699eb747649095f360c7ff
  ShadowBroker_0.9.81_x64-setup.exe        76.5 MB
    4e866fa0423c0c2470ed32f4809167a7815dc23ee7762b69e95681c1f3a28250

Post-merge plan
---------------

Force-move the v0.9.81 tag to this commit and replace ALL release
assets on the GitHub release: zip, msi, exe, both .sig files,
latest.json, SHA256SUMS.txt, release-manifest.json.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 21:34:59 -06:00
Shadowbroker 2dc1fcc778 release: v0.9.81 — signed auto-update + admin_session race fix (#323)
What this release does
----------------------

1. Establishes a fresh Tauri updater signing keypair. The previous keypair
   (pubkey baked into v0.9.79 / v0.9.8) had no matching private key on
   any maintainer-controlled machine — every prior release shipped
   without signatures, so auto-update has never actually worked. v0.9.81
   rotates to a new pubkey and ships signed installers + latest.json so
   every release from here is a one-click upgrade.

2. Fixes the ``admin_session_required`` race in TopRightControls.tsx.
   The updateAction state used to default to ``auto_apply`` at React-init
   time. A click on the Update button before the async runtime probe
   completed went down the auto_apply path (POST /api/system/update),
   which throws ``admin_session_required`` on fresh sessions. Desktop
   installs now default to ``manual_download`` based on synchronous
   ``window.__TAURI__`` detection at useState init.

One-time cost for current installs
----------------------------------

Anyone on v0.9.79 or v0.9.8 will see the in-app Update button still
trigger the broken path on their existing install (the fix only takes
effect once they're ON v0.9.81). The MANUAL DOWNLOAD button in the
update dialog opens the GitHub release page, where they grab the .msi
and run it. After that one manual hop, all future updates are seamless.

Release artifacts
-----------------

  ShadowBroker_v0.9.81.zip                  6.06 MB
    42f8a51f9a5690d1e7349d90d8ecf2d163c9061d6cf90c69ee03647a785437ff
  ShadowBroker_0.9.81_x64_en-US.msi       122.4 MB
    a45b177c26c95d2b28d71592d7147e88ff4e104865f214fde11249d311ec9e25
  ShadowBroker_0.9.81_x64-setup.exe        76.5 MB
    eca884b9d37eeccd0f11c91dcc6f6ae1b3609d9dee72bd73c37c9a427babfef2

Plus .sig files for the .msi and .exe, plus a signed latest.json for
the Tauri updater endpoint.

Sizes match the v0.9.79 / v0.9.8 reference shape within drift for
the new TopRightControls patch.

release_digests.json keeps v0.9.79 + v0.9.8 blocks alongside v0.9.81
so operators still on those versions continue to validate cleanly
during the rollout transition.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 18:43:53 -06:00
Shadowbroker 896d1ae938 fix(#319,#296): v0.9.8 rebuild — bundle missing deps so backend launches (#322)
Issues #319 and #296 reported that the installed v0.9.79 Windows MSI/EXE
crashed on launch with:

    thread 'main' panicked ... failed to setup app: error encountered
    during setup hook: ShadowBroker cannot start: the bundled local
    backend failed to launch.
    technical detail: managed_backend_exited_early:exit code: 103

Root cause: ``backend/pyproject.toml`` declares ``defusedxml>=0.7.1`` and
``PySocks==1.7.1`` as runtime dependencies, but the venv used to build
v0.9.79 (and the initial v0.9.8 publish) had both missing. When
``services/fetchers/aircraft_database.py`` does
``import defusedxml.ElementTree`` at startup, Python raises
``ModuleNotFoundError`` and uvicorn exits, which Tauri reports as
``managed_backend_exited_early``.

Both packages now installed in the build venv. ``main.py`` imports
end-to-end with only the expected ``plane_alert_db.json not found``
warning (runtime-state file, populated on first launch).

Rebuilt artifacts on the maintainer's local machine:

    ShadowBroker_v0.9.8.zip                  6.06 MB
      183bb5cd62b9b9349d95df5ef7696cb6ca810ab4b991fa9dab6f898af4c7a175
    ShadowBroker_0.9.8_x64_en-US.msi       122.4 MB
      fe22f9d51e4360d74c18a7250c2fbb9ed4fa4c7a884b3ac0d04a21115466386b
    ShadowBroker_0.9.8_x64-setup.exe        76.5 MB
      94a0309862e9c81c92cdcbfea8eec9dbb97eef19ded82b26217b397defbc810c

After this merges, the v0.9.8 tag will be force-moved to this commit and
the GitHub release assets replaced so the integrity chain validates
against the working installers instead of the broken ones.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 16:48:45 -06:00
Shadowbroker 8dfa6a7199 release: v0.9.8 — Cumulative Fuel/CO2, AIS Resilience, Data-Layer Repair (#321)
Bumps every hardcoded 0.9.79 → 0.9.8 across backend, frontend,
desktop-shell, helm, lockfiles, test fixtures. Refreshes the in-app
ChangelogModal HEADLINE_FEATURES, NEW_FEATURES, and BUG_FIXES with the
v0.9.8 highlights.

Release artifacts built locally and hashed into release_digests.json:

  ShadowBroker_v0.9.8.zip                  6.06 MB
    d506f6b8462ccb12096f0cd9462233be58928094240416b65fb3127bdd1f3820
  ShadowBroker_0.9.8_x64_en-US.msi       122.4 MB
    d4be4cb68c3e6409fff54c225acdcdd08e27d5d6d2b31616d78d2a4f6812991d
  ShadowBroker_0.9.8_x64-setup.exe        76.5 MB
    1115d1f5cf37edd03ea2c21d821c7626e1bf3319c990402aaa0293bca46fea67

Sizes match the v0.9.79 reference shape (5.76 MB / 117 MB / 72.9 MB)
within expected drift for new code. The .zip is a `git archive` of the
v0.9.8 source tree (matching v0.9.79's approach).

Audit confirms no .env, .key, .venv-dir, or cache files leaked into the
backend-runtime bundle. Python 3.11.9 + 199 site-packages + privacy_core
all staged correctly.

Headline changes since v0.9.79:
* Cumulative fuel/CO2 per flight (#317) — running totals since first
  observation, not just per-hour rate.
* AIS maritime resilience (#314, #316) — outage banner + AISHub REST
  fallback when AISStream WebSocket primary is offline.
* Data-layer repair (#311, #312) — UAP fallback respects the 60-day
  cutoff; GPS jamming threshold tuning + nac_p=0 inclusion so the layer
  actually fires.
* Per-flight source attribution (#313) — source field on every record.
* Cross-node DM mailbox replication (#309).
* Infonet sync HTTP 429 honored (#310).

Test fixtures updated:
* test_per_operator_outbound_attribution.py — added v0.9.8 UA strings
  to the banned-aggregate-literals list (alongside v0.9.79).
* updateRuntime.test.ts — bumped asset filename fixtures to v0.9.8.

release_digests.json keeps the v0.9.79 block alongside v0.9.8 so
operators still on 0.9.79 validate cleanly during the rollout.

The accent narrowing fix in ChangelogModal (one feature uses 'purple',
two use 'cyan' so the renderer's `accent === 'purple'` comparison
still type-checks) is included.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 16:24:20 -06:00
43 changed files with 2314 additions and 450 deletions
+2 -2
View File
@@ -174,7 +174,7 @@ ShadowBroker v0.9.7 ships **InfoNet** (decentralized intelligence mesh + Soverei
| Channel | Privacy Status | Details |
|---|---|---|
| **Meshtastic / APRS** | **PUBLIC** | RF radio transmissions are public and interceptable by design. |
| **InfoNet Gate Chat** | **OBFUSCATED** | Messages are obfuscated with gate personas and canonical payload signing, but NOT end-to-end encrypted. Metadata is not hidden. |
| **InfoNet Gate Chat** | **OBFUSCATED** | Messages are obfuscated with gate personas and canonical payload signing, but NOT end-to-end encrypted. Metadata is not hidden despite being designed through Tor and Reticulum (Work in progress). |
| **Dead Drop DMs** | **STRONGEST CURRENT LANE** | Token-based epoch mailbox with SAS word verification. Strongest lane in this build, but not yet confidently private. |
| **Sovereign Shell governance** | **PUBLIC LEDGER** | Petitions, votes, upgrade hashes, and dispute stakes are signed events on a public hashchain. Pseudonymous via gate persona, but governance actions are intentionally observable. |
| **Privacy primitives (RingCT / stealth / DEX)** | **NOT YET WIRED** | Locked Protocol contracts are in place, but the cryptographic scheme has not been chosen. The privacy-core Rust crate is the integration target for a future sprint. |
@@ -199,7 +199,7 @@ The first decentralized intelligence communication and governance layer built di
**Communication layer (since v0.9.6):**
* **InfoNet Experimental Testnet** — A global, obfuscated message relay. Anyone running ShadowBroker can transmit and receive on the InfoNet. Messages pass through a Wormhole relay layer with gate personas, Ed25519 canonical payload signing, and transport obfuscation.
* **InfoNet Experimental Testnet** — A global, obfuscated message relay using Tor and Reticulum. Anyone running ShadowBroker can transmit and receive on the InfoNet. Messages pass through a Wormhole relay layer with gate personas, Ed25519 canonical payload signing, and transport obfuscation.
* **Mesh Chat Panel** — Three-tab interface: **INFONET** (gate chat with obfuscated transport), **MESH** (Meshtastic radio integration), **DEAD DROP** (peer-to-peer message exchange with token-based epoch mailboxes — strongest current lane).
* **Gate Persona System** — Pseudonymous identities with Ed25519 signing keys, prekey bundles, SAS word contact verification, and abuse reporting.
* **Mesh Terminal** — Built-in CLI: `send`, `dm`, market commands, gate state inspection. Draggable panel, minimizes to the top bar. Type `help` to see all commands.
+10
View File
@@ -36,5 +36,15 @@
"ShadowBroker_v0.9.79.zip": "f6877c1d66614525315ea82636ce9f7b41178332c4dbf90d27431a1ea1d9cd47",
"ShadowBroker_0.9.79_x64-setup.exe": "f7b676ada45cac7da05868b0a353678c9ee700e3abcf456a7c0c038c36da446f",
"ShadowBroker_0.9.79_x64_en-US.msi": "e0713c3cdda184cfbea750bfac0d62a35678fec00847e6476f2cac8e7e42046e"
},
"v0.9.8": {
"ShadowBroker_v0.9.8.zip": "183bb5cd62b9b9349d95df5ef7696cb6ca810ab4b991fa9dab6f898af4c7a175",
"ShadowBroker_0.9.8_x64-setup.exe": "94a0309862e9c81c92cdcbfea8eec9dbb97eef19ded82b26217b397defbc810c",
"ShadowBroker_0.9.8_x64_en-US.msi": "fe22f9d51e4360d74c18a7250c2fbb9ed4fa4c7a884b3ac0d04a21115466386b"
},
"v0.9.81": {
"ShadowBroker_v0.9.81.zip": "af8c87ccdece8fbb9aadc6be63cce10d3fcba74e6d87ef83289dda6d555fd270",
"ShadowBroker_0.9.81_x64-setup.exe": "4e866fa0423c0c2470ed32f4809167a7815dc23ee7762b69e95681c1f3a28250",
"ShadowBroker_0.9.81_x64_en-US.msi": "8977c9a1c54e1f0d030436be9c4e3d81d766cc0080699eb747649095f360c7ff"
}
}
+407 -96
View File
@@ -1,4 +1,4 @@
import os
import os
from dotenv import load_dotenv
load_dotenv()
@@ -8,18 +8,24 @@ import asyncio
import base64
import hmac
import importlib
import ipaddress
import secrets
import hashlib as _hashlib_mod
from dataclasses import dataclass, field
from typing import Any
from json import JSONDecodeError
APP_VERSION = "0.9.79"
APP_VERSION = "0.9.81"
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
_start_time = time.time()
_MESH_ONLY = os.environ.get("MESH_ONLY", "").strip().lower() in ("1", "true", "yes")
_HEADLESS_MESH_NODE_RUNTIME = os.environ.get("SHADOWBROKER_MESH_NODE_RUNTIME", "").strip().lower() in (
"1",
"true",
"yes",
)
_WARNED_LEGACY_DM_PUBKEY_LOOKUPS: set[str] = set()
@@ -1095,6 +1101,7 @@ _WORMHOLE_PUBLIC_PROFILE_FIELDS = {"wormhole_enabled"}
_PRIVATE_LANE_CONTROL_FIELDS = {"private_lane_tier", "private_lane_policy"}
_PUBLIC_RNS_STATUS_FIELDS = {"enabled", "ready", "configured_peers", "active_peers"}
_NODE_PUBLIC_EVENT_HOOK_REGISTERED = False
_NODE_RUNTIME_THREADS_STARTED = False
_INFONET_PRIVATE_TRANSPORT_LOCK = threading.Lock()
@@ -1184,6 +1191,49 @@ def _filter_infonet_sync_records(records: list[Any]) -> list[Any]:
]
def _infonet_peer_url_allowed(peer_url: str) -> bool:
if not _infonet_private_transport_required():
return True
return _is_private_infonet_transport(peer_transport_kind(peer_url))
def _filter_infonet_peer_urls(peer_urls: list[str]) -> list[str]:
if not _infonet_private_transport_required():
return peer_urls
return [peer_url for peer_url in peer_urls if _infonet_peer_url_allowed(peer_url)]
def _infonet_peer_requests_proxies(normalized_peer_url: str) -> dict[str, str] | None:
"""Return requests proxy settings for a sync/push peer, enforcing private policy."""
transport = peer_transport_kind(normalized_peer_url)
if _infonet_private_transport_required() and not _is_private_infonet_transport(transport):
raise RuntimeError(_infonet_private_transport_error())
if transport != "onion":
return None
if not bool(get_settings().MESH_ARTI_ENABLED):
raise RuntimeError("onion peer requests require Arti to be enabled")
from services.wormhole_supervisor import _check_arti_ready
if not _check_arti_ready():
raise RuntimeError("onion peer requests require a ready Arti transport")
socks_port = int(get_settings().MESH_ARTI_SOCKS_PORT or 9050)
proxy = f"socks5h://127.0.0.1:{socks_port}"
return {"http": proxy, "https": proxy}
def _local_infonet_peer_url() -> str:
"""Return this node's advertised peer URL for HMAC peer authentication."""
configured = normalize_peer_url(str(getattr(get_settings(), "MESH_PUBLIC_PEER_URL", "") or ""))
if configured:
return configured
try:
from services.tor_hidden_service import tor_service
return normalize_peer_url(str(tor_service.onion_address or ""))
except Exception:
return ""
def _ensure_infonet_private_transport_ready(reason: str = "") -> bool:
"""Warm the local onion transport before private Infonet sync.
@@ -1257,6 +1307,13 @@ def _refresh_node_peer_store(*, now: float | None = None) -> dict[str, Any]:
operator_peers = configured_relay_peer_urls()
bootstrap_seed_peers = _configured_bootstrap_seed_peer_urls()
skipped_clearnet_peers = 0
pruned_clearnet_peers = 0
if private_transport_required:
for key, record in list(store._records.items()):
if _is_private_infonet_transport(str(getattr(record, "transport", "") or "")):
continue
del store._records[key]
pruned_clearnet_peers += 1
for peer_url in operator_peers:
transport = peer_transport_kind(peer_url)
if not transport:
@@ -1364,6 +1421,7 @@ def _refresh_node_peer_store(*, now: float | None = None) -> dict[str, Any]:
"node_mode": mode,
"private_transport_required": private_transport_required,
"skipped_clearnet_peer_count": skipped_clearnet_peers,
"pruned_clearnet_peer_count": pruned_clearnet_peers,
"manifest_loaded": manifest is not None,
"manifest_signer_id": manifest.signer_id if manifest is not None else "",
"manifest_valid_until": int(manifest.valid_until or 0) if manifest is not None else 0,
@@ -1384,6 +1442,28 @@ def _materialize_local_infonet_state() -> None:
from services.mesh.mesh_hashchain import infonet
infonet.ensure_materialized()
try:
_hydrate_gate_store_from_chain(list(infonet.events))
_hydrate_dm_relay_from_chain(list(infonet.events))
except Exception:
pass
class PeerSyncHTTPError(RuntimeError):
def __init__(self, status_code: int, detail: str, *, retry_after_s: int = 0):
self.status_code = int(status_code or 0)
self.retry_after_s = int(retry_after_s or 0)
message = str(detail or f"HTTP {self.status_code}").strip()
if not message.upper().startswith("HTTP"):
message = f"HTTP {self.status_code}: {message}"
super().__init__(message)
def _parse_retry_after_seconds(value: str) -> int:
try:
return max(0, int(float(str(value or "").strip())))
except Exception:
return 0
def _peer_sync_response(peer_url: str, body: dict[str, Any]) -> dict[str, Any]:
@@ -1446,7 +1526,8 @@ def _peer_sync_response(peer_url: str, body: dict[str, Any]) -> dict[str, Any]:
raise ValueError(f"peer sync returned non-JSON response ({response.status_code})") from exc
if response.status_code != 200:
detail = str(payload.get("detail", "") or f"HTTP {response.status_code}").strip()
raise ValueError(detail or f"HTTP {response.status_code}")
retry_after_s = _parse_retry_after_seconds(response.headers.get("Retry-After", ""))
raise PeerSyncHTTPError(response.status_code, detail, retry_after_s=retry_after_s)
if not isinstance(payload, dict):
raise ValueError("peer sync returned malformed payload")
return payload
@@ -1485,6 +1566,46 @@ def _hydrate_gate_store_from_chain(events: list[dict]) -> int:
return count
def _hydrate_dm_relay_from_chain(events: list[dict]) -> int:
"""Copy accepted dm_message chain events into the local encrypted DM relay."""
import hashlib
from services.mesh.mesh_dm_relay import dm_relay
from services.mesh.mesh_hashchain import infonet
count = 0
for evt in events:
if evt.get("event_type") != "dm_message":
continue
event_id = str(evt.get("event_id", "") or "").strip()
if not event_id or event_id not in infonet.event_index:
continue
canonical = infonet.events[infonet.event_index[event_id]]
payload = canonical.get("payload") if isinstance(canonical.get("payload"), dict) else {}
sender_token_hash = hashlib.sha256(
f"hashchain-dm-sender|{event_id}|{canonical.get('node_id', '')}".encode("utf-8")
).hexdigest()
try:
result = dm_relay.deposit(
sender_id=str(canonical.get("node_id", "") or ""),
raw_sender_id=str(canonical.get("node_id", "") or ""),
recipient_id=str(payload.get("recipient_id", "") or ""),
ciphertext=str(payload.get("ciphertext", "") or ""),
msg_id=str(payload.get("msg_id", "") or ""),
delivery_class=str(payload.get("delivery_class", "") or ""),
recipient_token=str(payload.get("recipient_token", "") or "") or None,
sender_seal=str(payload.get("sender_seal", "") or ""),
sender_token_hash=sender_token_hash,
payload_format=str(payload.get("format", "dm1") or "dm1"),
session_welcome=str(payload.get("session_welcome", "") or ""),
)
if result.get("ok"):
count += 1
except Exception:
pass
return count
def _sync_from_peer(
peer_url: str,
*,
@@ -1538,6 +1659,7 @@ def _sync_from_peer(
return True, "", False, 0
result = infonet.ingest_events(events)
_hydrate_gate_store_from_chain(events)
_hydrate_dm_relay_from_chain(events)
rejected = list(result.get("rejected", []) or [])
if rejected:
return False, f"sync ingest rejected {len(rejected)} event(s)", False, 0
@@ -1600,6 +1722,8 @@ def _run_public_sync_cycle() -> SyncWorkerState:
last_error = "sync failed"
for record in peers:
retry_after_s = 0
http_status_code = 0
started = begin_sync(
current_state,
peer_url=record.peer_url,
@@ -1610,6 +1734,17 @@ def _run_public_sync_cycle() -> SyncWorkerState:
set_sync_state(started)
try:
ok, error, forked, retry_after_s = _sync_from_peer(record.peer_url)
except PeerSyncHTTPError as exc:
# _sync_from_peer catches PeerSyncRateLimited internally (4-tuple
# path for 429 with Retry-After). Other non-200 statuses surface
# here as PeerSyncHTTPError — pull retry_after_s + status off it
# so the cooldown calculation below can honor server hints even
# for non-429 throttling responses.
ok = False
error = str(exc)
forked = False
retry_after_s = int(exc.retry_after_s or 0)
http_status_code = int(exc.status_code or 0)
except Exception as exc:
ok = False
error = str(exc or type(exc).__name__)
@@ -1640,6 +1775,10 @@ def _run_public_sync_cycle() -> SyncWorkerState:
getattr(settings, "MESH_BOOTSTRAP_SEED_FAILURE_COOLDOWN_S", cooldown_s)
or cooldown_s
)
if http_status_code == 429:
failure_count = max(int(getattr(record, "failure_count", 0) or 0), current_state.consecutive_failures)
exponential_429_s = min(900, 60 * (2 ** min(failure_count, 4)))
cooldown_s = max(cooldown_s, retry_after_s, exponential_429_s)
store.mark_failure(
record.peer_url,
"sync",
@@ -1650,7 +1789,7 @@ def _run_public_sync_cycle() -> SyncWorkerState:
store.save()
failure_backoff_s = int(settings.MESH_SYNC_FAILURE_BACKOFF_S or 60)
if is_seed_peer:
failure_backoff_s = min(failure_backoff_s, max(1, cooldown_s))
failure_backoff_s = max(failure_backoff_s, max(1, cooldown_s))
updated = finish_sync(
started,
ok=False,
@@ -1750,7 +1889,7 @@ def _propagate_public_event_to_peers(event_dict: dict[str, Any]) -> None:
if not _participant_node_enabled():
return
if not authenticated_push_peer_urls():
if not _filter_infonet_peer_urls(authenticated_push_peer_urls()):
return
envelope = MeshEnvelope(
@@ -1784,6 +1923,45 @@ def _schedule_public_event_propagation(event_dict: dict[str, Any]) -> None:
).start()
def _infonet_node_runtime_requested() -> bool:
return (not _MESH_ONLY) or _HEADLESS_MESH_NODE_RUNTIME
def _start_infonet_node_runtime(reason: str = "startup") -> None:
"""Start sync/push/pull workers for participant nodes."""
global _NODE_PUBLIC_EVENT_HOOK_REGISTERED, _NODE_RUNTIME_THREADS_STARTED
if not _infonet_node_runtime_requested():
return
try:
from services.mesh.mesh_hashchain import register_public_event_append_hook
_materialize_local_infonet_state()
_refresh_node_peer_store()
if _node_runtime_supported():
if not _participant_node_enabled():
logger.info("Infonet participant auto-enabled for private seed sync")
_set_participant_node_enabled(True)
threading.Thread(
target=lambda: _ensure_infonet_private_transport_ready(reason),
daemon=True,
name="infonet-private-transport-warmup",
).start()
_NODE_SYNC_STOP.clear()
if not _NODE_RUNTIME_THREADS_STARTED:
threading.Thread(target=_public_infonet_sync_loop, daemon=True).start()
threading.Thread(target=_http_peer_push_loop, daemon=True).start()
threading.Thread(target=_http_gate_push_loop, daemon=True).start()
threading.Thread(target=_http_gate_pull_loop, daemon=True).start()
_NODE_RUNTIME_THREADS_STARTED = True
_kick_public_sync_background(reason)
if not _NODE_PUBLIC_EVENT_HOOK_REGISTERED:
register_public_event_append_hook(_schedule_public_event_propagation)
_NODE_PUBLIC_EVENT_HOOK_REGISTERED = True
except Exception as e:
logger.warning(f"Node bootstrap runtime failed to initialize: {e}")
# ─── Background HTTP Peer Push Worker ────────────────────────────────────
# Runs alongside the sync loop. Every PUSH_INTERVAL seconds, batches new
# Infonet events and sends them via HMAC-authenticated POST to push peers.
@@ -1791,6 +1969,7 @@ def _schedule_public_event_propagation(event_dict: dict[str, Any]) -> None:
_PEER_PUSH_INTERVAL_S = 10
_PEER_PUSH_BATCH_SIZE = 50
_peer_push_last_index: dict[str, int] = {} # peer_url → last pushed event index
_INFONET_SYNC_RATE_LIMIT = "600/minute"
def _http_peer_push_loop() -> None:
@@ -1812,7 +1991,7 @@ def _http_peer_push_loop() -> None:
# loop on the global secret being set — an install that only
# configures per-peer secrets is now valid.
peers = authenticated_push_peer_urls()
peers = _filter_infonet_peer_urls(authenticated_push_peer_urls())
if not peers:
_NODE_SYNC_STOP.wait(_PEER_PUSH_INTERVAL_S)
continue
@@ -1840,7 +2019,8 @@ def _http_peer_push_loop() -> None:
ensure_ascii=False,
).encode("utf-8")
peer_key = resolve_peer_key_for_url(normalized)
sender_url = _local_infonet_peer_url()
peer_key = resolve_peer_key_for_url(sender_url)
if not peer_key:
continue
import hmac as _hmac_mod2
@@ -1848,14 +2028,21 @@ def _http_peer_push_loop() -> None:
hmac_hex = _hmac_mod2.new(peer_key, body_bytes, _hashlib_mod2.sha256).hexdigest()
timeout = int(get_settings().MESH_RELAY_PUSH_TIMEOUT_S or 10)
resp = _requests.post(
f"{normalized}/api/mesh/infonet/peer-push",
data=body_bytes,
headers={
proxies = _infonet_peer_requests_proxies(normalized)
request_kwargs: dict[str, Any] = {
"data": body_bytes,
"headers": {
"Content-Type": "application/json",
"X-Peer-Url": sender_url,
"X-Peer-HMAC": hmac_hex,
},
timeout=timeout,
"timeout": timeout,
}
if proxies:
request_kwargs["proxies"] = proxies
resp = _requests.post(
f"{normalized}/api/mesh/infonet/peer-push",
**request_kwargs,
)
if resp.status_code == 200:
_peer_push_last_index[normalized] = last_idx + len(batch)
@@ -1895,7 +2082,7 @@ def _http_gate_pull_loop() -> None:
# Issue #256: per-peer key resolution; see _http_peer_push_loop.
peers = authenticated_push_peer_urls()
peers = _filter_infonet_peer_urls(authenticated_push_peer_urls())
if not peers:
_NODE_SYNC_STOP.wait(_GATE_PULL_INTERVAL_S)
continue
@@ -1905,7 +2092,8 @@ def _http_gate_pull_loop() -> None:
if not normalized:
continue
peer_key = resolve_peer_key_for_url(normalized)
sender_url = _local_infonet_peer_url()
peer_key = resolve_peer_key_for_url(sender_url)
if not peer_key:
continue
@@ -1925,14 +2113,21 @@ def _http_gate_pull_loop() -> None:
discovery_hmac = _hmac_pull.new(peer_key, discovery_body, _hashlib_pull.sha256).hexdigest()
timeout = int(get_settings().MESH_RELAY_PUSH_TIMEOUT_S or 10)
resp = _requests.post(
f"{normalized}/api/mesh/gate/peer-pull",
data=discovery_body,
headers={
proxies = _infonet_peer_requests_proxies(normalized)
discovery_kwargs: dict[str, Any] = {
"data": discovery_body,
"headers": {
"Content-Type": "application/json",
"X-Peer-Url": sender_url,
"X-Peer-HMAC": discovery_hmac,
},
timeout=timeout,
"timeout": timeout,
}
if proxies:
discovery_kwargs["proxies"] = proxies
resp = _requests.post(
f"{normalized}/api/mesh/gate/peer-pull",
**discovery_kwargs,
)
if resp.status_code != 200:
continue
@@ -1962,14 +2157,20 @@ def _http_gate_pull_loop() -> None:
pull_hmac = _hmac_pull.new(peer_key, pull_body, _hashlib_pull.sha256).hexdigest()
pull_resp = _requests.post(
f"{normalized}/api/mesh/gate/peer-pull",
data=pull_body,
headers={
pull_kwargs: dict[str, Any] = {
"data": pull_body,
"headers": {
"Content-Type": "application/json",
"X-Peer-Url": sender_url,
"X-Peer-HMAC": pull_hmac,
},
timeout=timeout,
"timeout": timeout,
}
if proxies:
pull_kwargs["proxies"] = proxies
pull_resp = _requests.post(
f"{normalized}/api/mesh/gate/peer-pull",
**pull_kwargs,
)
if pull_resp.status_code != 200:
continue
@@ -2020,7 +2221,7 @@ def _http_gate_push_loop() -> None:
# Issue #256: per-peer key resolution; see _http_peer_push_loop.
peers = authenticated_push_peer_urls()
peers = _filter_infonet_peer_urls(authenticated_push_peer_urls())
if not peers:
_NODE_SYNC_STOP.wait(_PEER_PUSH_INTERVAL_S)
continue
@@ -2033,7 +2234,8 @@ def _http_gate_push_loop() -> None:
if not normalized:
continue
peer_key = resolve_peer_key_for_url(normalized)
sender_url = _local_infonet_peer_url()
peer_key = resolve_peer_key_for_url(sender_url)
if not peer_key:
continue
@@ -2064,14 +2266,21 @@ def _http_gate_push_loop() -> None:
hmac_hex = _hmac_mod3.new(peer_key, body_bytes, _hashlib_mod3.sha256).hexdigest()
timeout = int(get_settings().MESH_RELAY_PUSH_TIMEOUT_S or 10)
resp = _requests.post(
f"{normalized}/api/mesh/gate/peer-push",
data=body_bytes,
headers={
proxies = _infonet_peer_requests_proxies(normalized)
request_kwargs: dict[str, Any] = {
"data": body_bytes,
"headers": {
"Content-Type": "application/json",
"X-Peer-Url": sender_url,
"X-Peer-HMAC": hmac_hex,
},
timeout=timeout,
"timeout": timeout,
}
if proxies:
request_kwargs["proxies"] = proxies
resp = _requests.post(
f"{normalized}/api/mesh/gate/peer-push",
**request_kwargs,
)
if resp.status_code == 200:
peer_counts[gate_id] = last + len(batch)
@@ -2413,32 +2622,8 @@ async def lifespan(app: FastAPI):
daemon=True,
name="wormhole-startup-sync",
).start()
try:
from services.mesh.mesh_hashchain import register_public_event_append_hook
_materialize_local_infonet_state()
_refresh_node_peer_store()
if _node_runtime_supported():
if not _participant_node_enabled():
logger.info("Infonet participant auto-enabled for private seed sync")
_set_participant_node_enabled(True)
threading.Thread(
target=lambda: _ensure_infonet_private_transport_ready("startup"),
daemon=True,
name="infonet-private-transport-warmup",
).start()
_NODE_SYNC_STOP.clear()
threading.Thread(target=_public_infonet_sync_loop, daemon=True).start()
_kick_public_sync_background("startup")
threading.Thread(target=_http_peer_push_loop, daemon=True).start()
threading.Thread(target=_http_gate_push_loop, daemon=True).start()
threading.Thread(target=_http_gate_pull_loop, daemon=True).start()
global _NODE_PUBLIC_EVENT_HOOK_REGISTERED
if not _NODE_PUBLIC_EVENT_HOOK_REGISTERED:
register_public_event_append_hook(_schedule_public_event_propagation)
_NODE_PUBLIC_EVENT_HOOK_REGISTERED = True
except Exception as e:
logger.warning(f"Node bootstrap runtime failed to initialize: {e}")
_start_infonet_node_runtime("startup")
if not _MESH_ONLY:
# Prime the static route/airport database from vrs-standing-data.adsb.lol
@@ -2677,6 +2862,91 @@ def _redact_public_event(event: dict) -> dict:
return _redact_vote_gate(_redact_key_rotate_payload(_redact_gate_metadata(event)))
def _is_loopback_host(host: str) -> bool:
value = str(host or "").strip().lower()
if not value:
return False
if value.startswith("[") and "]" in value:
value = value[1 : value.index("]")]
if ":" in value and value.count(":") == 1:
value = value.rsplit(":", 1)[0]
if value in {"localhost", "ip6-localhost"}:
return True
try:
return ipaddress.ip_address(value).is_loopback
except ValueError:
return False
def _is_onion_host(host: str) -> bool:
value = str(host or "").strip().lower()
if not value:
return False
if ":" in value and value.count(":") == 1:
value = value.rsplit(":", 1)[0]
return value.endswith(".onion")
def _forwarded_for_hosts(request) -> list[str]:
headers = getattr(request, "headers", {}) or {}
hosts: list[str] = []
x_forwarded_for = str(headers.get("x-forwarded-for", "") or "")
hosts.extend(part.strip() for part in x_forwarded_for.split(",") if part.strip())
forwarded = str(headers.get("forwarded", "") or "")
for section in forwarded.split(","):
for item in section.split(";"):
key, sep, value = item.strip().partition("=")
if sep and key.strip().lower() == "for":
hosts.append(value.strip().strip('"').strip("[]"))
return hosts
def _request_appears_private_infonet_transport(request) -> bool:
"""Return whether a sync request is safe to carry private ledger events.
This is intentionally fail-closed for the private event surface only. A
questionable request still gets public events; gate/DM ciphertext simply
stays out of the response.
"""
if not _infonet_private_transport_required() or request is None:
return False
forwarded_hosts = _forwarded_for_hosts(request)
if forwarded_hosts and any(not (_is_loopback_host(host) or _is_onion_host(host)) for host in forwarded_hosts):
return False
client = getattr(request, "client", None)
client_host = str(getattr(client, "host", "") or "")
headers = getattr(request, "headers", {}) or {}
host_header = str(headers.get("host", "") or "")
url_host = str(getattr(getattr(request, "url", None), "hostname", "") or "")
return any(
(
_is_loopback_host(client_host),
_is_loopback_host(host_header),
_is_loopback_host(url_host),
_is_onion_host(host_header),
_is_onion_host(url_host),
)
)
def _infonet_sync_response_events(events: list[dict], request=None) -> list[dict]:
"""Build the sync event surface for the current transport policy."""
include_private = _request_appears_private_infonet_transport(request)
response: list[dict] = []
for event in events:
if not isinstance(event, dict):
continue
event_type = str(event.get("event_type", "") or "")
if event_type in {"gate_message", "dm_message"}:
if include_private:
response.append(dict(event))
continue
response.append(_redact_public_event(event))
return response
def _trusted_gate_reply_to(event: dict) -> str:
if not isinstance(event, dict):
return ""
@@ -5261,32 +5531,15 @@ def _submit_gate_message_envelope(request: Request, gate_id: str, body: dict[str
if not cooldown_ok:
return {"ok": False, "detail": cooldown_reason}
# Advance sequence counter (replay protection) without appending to
# the public infonet chain — gate messages are private.
try:
from services.mesh.mesh_hashchain import infonet, gate_store
seq_ok, seq_reason = _validate_private_signed_sequence(
infonet,
sender_id,
sequence,
domain="gate_message",
)
if not seq_ok:
return {"ok": False, "detail": seq_reason}
except ValueError as exc:
return {"ok": False, "detail": str(exc)}
except Exception:
logger.exception("Failed to advance sequence for gate message")
return {"ok": False, "detail": "Failed to record gate message"}
gate_manager.record_message(gate_id)
_record_gate_post_cooldown(sender_id, gate_id)
logger.info("Encrypted gate message accepted on obfuscated gate plane")
# Build gate event and store in gate_store (private — not on public chain).
# Build and commit the encrypted gate event to the private Infonet ledger.
# The main hashchain is the durable propagation surface; gate_store is the
# local materialized view used by the existing decrypt/UI path.
try:
from services.mesh.mesh_hashchain import _private_gate_event_id
from services.mesh.mesh_hashchain import infonet
import time as _time
store_payload = dict(gate_payload)
@@ -5308,19 +5561,24 @@ def _submit_gate_message_envelope(request: Request, gate_id: str, body: dict[str
"public_key_algo": public_key_algo,
"protocol_version": protocol_version or PROTOCOL_VERSION,
}
gate_event["event_id"] = _private_gate_event_id(gate_id, sender_id, sequence, gate_event)
gate_event = infonet.append_private_gate_message(
node_id=sender_id,
payload=store_payload,
signature=signature,
sequence=sequence,
public_key=public_key,
public_key_algo=public_key_algo,
protocol_version=protocol_version or PROTOCOL_VERSION,
timestamp=float(gate_event.get("timestamp", 0) or 0),
)
except ValueError as exc:
return {"ok": False, "detail": str(exc)}
except Exception:
logger.exception("Failed to prepare private gate message for queued release")
logger.exception("Failed to append gate message to private Infonet ledger")
return {"ok": False, "detail": "Failed to record gate message"}
# Append to the local gate_store immediately. The gate_store is a
# per-node persistent ciphertext chain; writing to it is a local
# operation with no network dependency. Previously this happened only
# inside the release worker's attempt_private_release path, which
# meant messages sat in the outbox — invisible to the author and the
# gate UI — until the transport tier reached the release floor.
# Decoupling local visibility from network fan-out: append locally now,
# queue the release for network propagation when the lane is ready.
# Append to the local gate_store immediately so the author sees the same
# materialized gate view that peers will hydrate after private sync.
try:
from services.mesh.mesh_hashchain import gate_store
@@ -5447,7 +5705,7 @@ async def infonet_locator(request: Request, limit: int = Query(32, ge=4, le=128)
@app.post("/api/mesh/infonet/sync")
@limiter.limit("30/minute")
@limiter.limit(_INFONET_SYNC_RATE_LIMIT)
@mesh_write_exempt(MeshWriteExemption.PEER_GOSSIP)
async def infonet_sync_post(
request: Request,
@@ -5500,8 +5758,7 @@ async def infonet_sync_post(
elif matched_hash == GENESIS_HASH and len(locator) > 1:
forked = True
# Filter out legacy gate_message events — not part of the public sync surface.
events = [_redact_public_event(e) for e in events if e.get("event_type") != "gate_message"]
events = _infonet_sync_response_events(events, request=request)
response = {
"events": events,
@@ -5564,7 +5821,7 @@ async def mesh_rns_status(request: Request):
@app.get("/api/mesh/infonet/sync")
@limiter.limit("30/minute")
@limiter.limit(_INFONET_SYNC_RATE_LIMIT)
async def infonet_sync(
request: Request,
after_hash: str = "",
@@ -5602,8 +5859,7 @@ async def infonet_sync(
)
base = after_hash or GENESIS_HASH
events = infonet.get_events_after(base, limit=limit)
# Filter out legacy gate_message events — not part of the public sync surface.
events = [_redact_public_event(e) for e in events if e.get("event_type") != "gate_message"]
events = _infonet_sync_response_events(events, request=request)
return {
"events": events,
"after_hash": base,
@@ -5642,6 +5898,7 @@ async def infonet_ingest(request: Request):
result = infonet.ingest_events(events)
_hydrate_gate_store_from_chain(events)
_hydrate_dm_relay_from_chain(events)
return {"ok": True, **result}
@@ -5682,6 +5939,7 @@ async def infonet_peer_push(request: Request):
result = infonet.ingest_events(events)
_hydrate_gate_store_from_chain(events)
_hydrate_dm_relay_from_chain(events)
return {"ok": True, **result}
@@ -6241,6 +6499,12 @@ async def infonet_event(request: Request, event_id: str):
)
return _strip_gate_for_access(evt, access)
return {"ok": False, "detail": "Event not found"}
if evt.get("event_type") == "dm_message":
return await _private_plane_refusal_response(
request,
status_code=403,
payload=_private_plane_access_denied_payload(),
)
if evt.get("event_type") == "gate_message":
gate_id = str(evt.get("payload", {}).get("gate", "") or evt.get("gate", "") or "").strip()
access = _verify_gate_access(request, gate_id) if gate_id else ""
@@ -6265,7 +6529,7 @@ async def infonet_node_events(
from services.mesh.mesh_hashchain import infonet
events = infonet.get_events_by_node(node_id, limit=limit)
events = [e for e in events if e.get("event_type") != "gate_message"]
events = [e for e in events if e.get("event_type") not in {"gate_message", "dm_message"}]
events = [_redact_public_event(e) for e in infonet.decorate_events(events)]
events = _redact_public_node_history(
events,
@@ -6290,7 +6554,7 @@ async def infonet_events_by_type(
else:
events = list(reversed(infonet.events))
events = events[offset : offset + limit]
events = [e for e in events if e.get("event_type") != "gate_message"]
events = [e for e in events if e.get("event_type") not in {"gate_message", "dm_message"}]
events = [_redact_public_event(e) for e in infonet.decorate_events(events)]
return {
"events": events,
@@ -7028,6 +7292,7 @@ async def _dm_send_from_signed_request(request: Request):
relay_salt_hex = str(body.get("relay_salt", "") or "").strip().lower()
msg_id = str(body.get("msg_id", "")).strip()
timestamp = _safe_int(body.get("timestamp", 0) or 0)
sequence = _safe_int(body.get("sequence", 0) or 0)
nonce = str(body.get("nonce", "")).strip()
if not sender_id or not recipient_id or not ciphertext or not msg_id or not timestamp:
@@ -7101,7 +7366,7 @@ async def _dm_send_from_signed_request(request: Request):
ok_seq, seq_reason = _validate_private_signed_sequence(
infonet,
sender_id,
int(body.get("sequence", 0) or 0),
sequence,
domain="dm_send",
)
if not ok_seq:
@@ -7135,7 +7400,47 @@ async def _dm_send_from_signed_request(request: Request):
"sender_seal": sender_seal,
"relay_salt": relay_salt_hex,
}
hashchain_spool: dict[str, Any] = {"ok": False, "detail": "not attempted"}
try:
from services.mesh.mesh_hashchain import infonet
chain_payload = dict(prepared.payload if prepared is not None else {})
if not chain_payload:
chain_payload = {
"recipient_id": recipient_id,
"delivery_class": delivery_class,
"recipient_token": recipient_token if delivery_class == "shared" else "",
"ciphertext": ciphertext,
"msg_id": msg_id,
"timestamp": timestamp,
"format": payload_format,
}
chain_payload["transport_lock"] = "private_strong"
chain_event = infonet.append_private_dm_message(
node_id=sender_id,
payload=chain_payload,
signature=str(prepared.signature if prepared is not None else body.get("signature", "") or ""),
sequence=sequence,
public_key=str(prepared.public_key if prepared is not None else body.get("public_key", "") or ""),
public_key_algo=str(
prepared.public_key_algo if prepared is not None else body.get("public_key_algo", "") or ""
),
protocol_version=str(
prepared.protocol_version if prepared is not None else body.get("protocol_version", "") or ""
)
or PROTOCOL_VERSION,
timestamp=float(timestamp or time.time()),
)
_hydrate_dm_relay_from_chain([chain_event])
hashchain_spool = {
"ok": True,
"event_id": str(chain_event.get("event_id", "") or ""),
"limit": 2,
}
except Exception as exc:
hashchain_spool = {"ok": False, "detail": str(exc) or type(exc).__name__}
queued_result = _queue_dm_release(current_tier=tier, payload=release_payload)
queued_result["hashchain_spool"] = hashchain_spool
if transport_upgrade_pending:
queued_result["private_transport_pending"] = True
return queued_result
@@ -9111,6 +9416,11 @@ async def api_get_node_settings(request: Request):
async def api_set_node_settings(request: Request, body: NodeSettingsUpdate):
_refresh_node_peer_store()
if bool(body.enabled):
if _infonet_private_transport_required() and not _ensure_infonet_private_transport_ready("operator_enable"):
return JSONResponse(
{"ok": False, "detail": _infonet_private_transport_error()},
status_code=503,
)
try:
from services.transport_lane_isolation import disable_public_mesh_lane
@@ -9119,6 +9429,7 @@ async def api_set_node_settings(request: Request, body: NodeSettingsUpdate):
logger.warning("Failed to disable public Mesh while enabling private node: %s", exc)
result = _set_participant_node_enabled(bool(body.enabled))
if bool(body.enabled):
_start_infonet_node_runtime("operator_enable")
_kick_public_sync_background("operator_enable")
return result
+2 -2
View File
@@ -7,7 +7,7 @@ py-modules = []
[project]
name = "backend"
version = "0.9.79"
version = "0.9.81"
requires-python = ">=3.10"
dependencies = [
"apscheduler==3.10.3",
@@ -43,7 +43,7 @@ dev = ["pytest>=8.3.4", "pytest-asyncio==0.25.0", "ruff>=0.9.0", "black>=24.0.0"
[tool.ruff.lint]
# The current backend carries historical style debt in large legacy modules.
# Keep CI focused on actionable correctness checks for the v0.9.79 release.
# Keep CI focused on actionable correctness checks for the v0.9.81 release.
ignore = ["E401", "E402", "E701", "E731", "E741", "F401", "F402", "F541", "F811", "F841"]
[tool.black]
+2 -2
View File
@@ -1590,7 +1590,7 @@ async def agent_tool_manifest(request: Request):
return {
"ok": True,
"version": "0.9.79",
"version": "0.9.81",
"access_tier": access_tier,
"available_commands": available_commands,
"transport": {
@@ -2226,7 +2226,7 @@ async def api_capabilities(request: Request):
access_tier = str(get_settings().OPENCLAW_ACCESS_TIER or "restricted").strip().lower()
return {
"ok": True,
"version": "0.9.79",
"version": "0.9.81",
"auth": {
"method": "HMAC-SHA256",
"headers": ["X-SB-Timestamp", "X-SB-Nonce", "X-SB-Signature"],
+1 -1
View File
@@ -8,7 +8,7 @@ from services.data_fetcher import get_latest_data
from services.schemas import HealthResponse
import os
APP_VERSION = os.environ.get("_HEALTH_APP_VERSION", "0.9.79")
APP_VERSION = os.environ.get("_HEALTH_APP_VERSION", "0.9.81")
router = APIRouter()
+7
View File
@@ -55,6 +55,12 @@ def _hydrate_gate_store_from_chain(events: list) -> int:
return count
def _hydrate_dm_relay_from_chain(events: list) -> int:
import main as _m
return int(_m._hydrate_dm_relay_from_chain(events))
@router.post("/api/mesh/infonet/peer-push")
@limiter.limit("30/minute")
async def infonet_peer_push(request: Request):
@@ -82,6 +88,7 @@ async def infonet_peer_push(request: Request):
return {"ok": True, "accepted": 0, "duplicates": 0, "rejected": []}
result = infonet.ingest_events(events)
_hydrate_gate_store_from_chain(events)
_hydrate_dm_relay_from_chain(events)
return {"ok": True, **result}
+33 -8
View File
@@ -65,6 +65,7 @@ from services.mesh.mesh_signed_events import (
logger = logging.getLogger(__name__)
router = APIRouter()
_INFONET_SYNC_RATE_LIMIT = "600/minute"
def _signed_body(request: Request) -> dict[str, Any]:
@@ -263,6 +264,19 @@ def _redact_public_event(event: dict) -> dict:
return _redact_vote_gate(_redact_key_rotate_payload(_redact_gate_metadata(event)))
def _infonet_private_transport_required() -> bool:
import main as _m
return bool(_m._infonet_private_transport_required())
def _infonet_sync_response_events(events: list[dict], request=None) -> list[dict]:
"""Build the sync event surface for the current transport policy."""
import main as _m
return _m._infonet_sync_response_events(events, request=request)
def _trusted_gate_reply_to(event: dict) -> str:
if not isinstance(event, dict):
return ""
@@ -574,6 +588,12 @@ def _hydrate_gate_store_from_chain(events: list[dict]) -> int:
pass
return count
def _hydrate_dm_relay_from_chain(events: list[dict]) -> int:
import main as _m
return int(_m._hydrate_dm_relay_from_chain(events))
# --- Safe type helpers ---
def _safe_int(val, default=0):
@@ -1531,7 +1551,7 @@ async def infonet_locator(request: Request, limit: int = Query(32, ge=4, le=128)
@router.post("/api/mesh/infonet/sync")
@limiter.limit("30/minute")
@limiter.limit(_INFONET_SYNC_RATE_LIMIT)
@mesh_write_exempt(MeshWriteExemption.PEER_GOSSIP)
async def infonet_sync_post(
request: Request,
@@ -1584,8 +1604,7 @@ async def infonet_sync_post(
elif matched_hash == GENESIS_HASH and len(locator) > 1:
forked = True
# Filter out legacy gate_message events — not part of the public sync surface.
events = [_redact_public_event(e) for e in events if e.get("event_type") != "gate_message"]
events = _infonet_sync_response_events(events, request=request)
response = {
"events": events,
@@ -1646,7 +1665,7 @@ async def mesh_rns_status(request: Request):
@router.get("/api/mesh/infonet/sync")
@limiter.limit("30/minute")
@limiter.limit(_INFONET_SYNC_RATE_LIMIT)
async def infonet_sync(
request: Request,
after_hash: str = "",
@@ -1684,8 +1703,7 @@ async def infonet_sync(
)
base = after_hash or GENESIS_HASH
events = infonet.get_events_after(base, limit=limit)
# Filter out legacy gate_message events — not part of the public sync surface.
events = [_redact_public_event(e) for e in events if e.get("event_type") != "gate_message"]
events = _infonet_sync_response_events(events, request=request)
return {
"events": events,
"after_hash": base,
@@ -1724,6 +1742,7 @@ async def infonet_ingest(request: Request):
result = infonet.ingest_events(events)
_hydrate_gate_store_from_chain(events)
_hydrate_dm_relay_from_chain(events)
return {"ok": True, **result}
@@ -2279,6 +2298,12 @@ async def infonet_event(request: Request, event_id: str):
)
return _strip_gate_for_access(evt, access)
return {"ok": False, "detail": "Event not found"}
if evt.get("event_type") == "dm_message":
return await _private_plane_refusal_response(
request,
status_code=403,
payload=_private_plane_access_denied_payload(),
)
if evt.get("event_type") == "gate_message":
gate_id = str(evt.get("payload", {}).get("gate", "") or evt.get("gate", "") or "").strip()
access = _verify_gate_access(request, gate_id) if gate_id else ""
@@ -2303,7 +2328,7 @@ async def infonet_node_events(
from services.mesh.mesh_hashchain import infonet
events = infonet.get_events_by_node(node_id, limit=limit)
events = [e for e in events if e.get("event_type") != "gate_message"]
events = [e for e in events if e.get("event_type") not in {"gate_message", "dm_message"}]
events = [_redact_public_event(e) for e in infonet.decorate_events(events)]
events = _redact_public_node_history(
events,
@@ -2328,7 +2353,7 @@ async def infonet_events_by_type(
else:
events = list(reversed(infonet.events))
events = events[offset : offset + limit]
events = [e for e in events if e.get("event_type") != "gate_message"]
events = [e for e in events if e.get("event_type") not in {"gate_message", "dm_message"}]
events = [_redact_public_event(e) for e in infonet.decorate_events(events)]
return {
"events": events,
+1
View File
@@ -32,6 +32,7 @@ class Settings(BaseSettings):
MESH_ARTI_ENABLED: bool = False
MESH_ARTI_SOCKS_PORT: int = 9050
MESH_RELAY_PEERS: str = ""
MESH_PUBLIC_PEER_URL: str = ""
# Bootstrap seeds are discovery hints, not authoritative network roots.
# Nodes promote healthy discovered peers from the store/manifest over time.
MESH_BOOTSTRAP_SEED_PEERS: str = "http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"
+522 -83
View File
@@ -33,8 +33,9 @@ Each event contains:
Persistence: JSON file at backend/data/infonet.json
Encrypted gate chat events are intentionally kept off the public chain and
persisted separately via GateMessageStore.
Encrypted gate chat events are private-chain ciphertext records. They are
excluded from public read surfaces and replicated only over private Infonet
transports.
"""
import json
@@ -64,6 +65,8 @@ from services.mesh.mesh_schema import (
ACTIVE_PUBLIC_LEDGER_EVENT_TYPES,
PUBLIC_LEDGER_EVENT_TYPES,
validate_event_payload,
validate_private_dm_ledger_payload,
validate_private_gate_ledger_payload,
validate_protocol_fields,
validate_public_ledger_payload,
)
@@ -127,6 +130,12 @@ GATE_SEGMENT_MAX_COMPRESSED_BYTES = max(
int(os.environ.get("MESH_GATE_SEGMENT_MAX_COMPRESSED_BYTES", str(2 * 1024 * 1024)) or str(2 * 1024 * 1024)),
)
GATE_SEGMENT_STORAGE_VERSION = 1
DM_HASHCHAIN_SPOOL_LIMIT = max(1, int(os.environ.get("MESH_DM_HASHCHAIN_SPOOL_LIMIT", "2") or "2"))
DM_HASHCHAIN_SPOOL_SENDER_LIMIT = max(
1,
int(os.environ.get("MESH_DM_HASHCHAIN_SPOOL_SENDER_LIMIT", "1") or "1"),
)
DM_HASHCHAIN_SPOOL_TTL_S = max(60, int(os.environ.get("MESH_DM_HASHCHAIN_SPOOL_TTL_S", "3600") or "3600"))
_PUBLIC_EVENT_APPEND_HOOKS: list[Any] = []
_PUBLIC_EVENT_APPEND_HOOKS_LOCK = threading.Lock()
@@ -340,6 +349,32 @@ def _private_gate_event_id(
).hexdigest()
def _private_gate_signature_payload_variants(gate_id: str, event: dict[str, Any]) -> list[dict[str, Any]]:
payload = _private_gate_signature_payload(gate_id, event)
variants: list[dict[str, Any]] = [payload]
event_payload = event.get("payload") if isinstance(event.get("payload"), dict) else {}
reply_to = str(event_payload.get("reply_to", "") or "").strip()
if reply_to:
variants.append(_private_gate_signature_payload(gate_id, event, include_reply_to=False))
if "epoch" in payload:
no_epoch = dict(payload)
no_epoch.pop("epoch", None)
variants.append(no_epoch)
if reply_to:
no_epoch_no_reply = _private_gate_signature_payload(gate_id, event, include_reply_to=False)
no_epoch_no_reply.pop("epoch", None)
variants.append(no_epoch_no_reply)
deduped: list[dict[str, Any]] = []
seen: set[str] = set()
for variant in variants:
material = json.dumps(variant, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
if material in seen:
continue
seen.add(material)
deduped.append(variant)
return deduped
def _sanitize_private_gate_event(gate_id: str, event: dict[str, Any]) -> dict[str, Any]:
payload = event.get("payload") if isinstance(event.get("payload"), dict) else {}
sanitized = {
@@ -1568,11 +1603,18 @@ class Infonet:
def _rebuild_state(self) -> None:
self.event_index = {}
self.node_sequences = {}
# Keep private signed-write replay domains across public-chain
# rebuilds; these domains protect local side effects that are not
# represented as public Infonet events.
if not isinstance(getattr(self, "sequence_domains", None), dict):
self.sequence_domains = {}
# Keep private signed-write replay domains that are not represented
# on-chain, but rebuild the gate_message sequence domain from chain
# events so reloads/fork application do not mix it with public
# per-node message sequences.
preserved_domains = {}
if isinstance(getattr(self, "sequence_domains", None), dict):
preserved_domains = {
key: value
for key, value in self.sequence_domains.items()
if not str(key or "").endswith("|gate_message")
}
self.sequence_domains = dict(preserved_domains)
self.public_key_bindings = {}
self.revocations = {}
self._replay_filter = ReplayFilter()
@@ -1584,9 +1626,12 @@ class Infonet:
node_id = evt.get("node_id", "")
sequence = _safe_int(evt.get("sequence", 0) or 0, 0)
if node_id and sequence:
last = self.node_sequences.get(node_id, 0)
sequence_table, sequence_key = self._sequence_table_for_event(
evt.get("event_type", ""), node_id
)
last = sequence_table.get(sequence_key, 0)
if sequence > last:
self.node_sequences[node_id] = sequence
sequence_table[sequence_key] = sequence
public_key = str(evt.get("public_key", "") or "")
if public_key and node_id:
existing = self.public_key_bindings.get(public_key)
@@ -1898,6 +1943,295 @@ class Infonet:
self._save()
return True, "ok"
def _sequence_table_for_event(self, event_type: str, node_id: str) -> tuple[dict[str, int], str]:
normalized = str(event_type or "").strip().lower()
if normalized == "gate_message":
return self.sequence_domains, f"{node_id}|gate_message"
if normalized == "dm_message":
return self.sequence_domains, f"{node_id}|dm_message"
return self.node_sequences, node_id
def _dm_spool_target_key(self, payload: dict[str, Any]) -> tuple[str, str]:
delivery_class = str(payload.get("delivery_class", "") or "").strip().lower()
if delivery_class == "shared":
key = str(payload.get("recipient_token", "") or "").strip()
else:
key = str(payload.get("recipient_id", "") or "").strip()
return delivery_class, key
def _dm_spool_active_counts(
self,
payload: dict[str, Any],
*,
sender_id: str = "",
now: float | None = None,
) -> tuple[int, int]:
delivery_class, key = self._dm_spool_target_key(payload)
if not key:
return 0, 0
sender_id = str(sender_id or "").strip()
current = time.time() if now is None else float(now)
total_count = 0
sender_count = 0
for evt in reversed(self.events):
if evt.get("event_type") != "dm_message":
continue
evt_payload = evt.get("payload") if isinstance(evt.get("payload"), dict) else {}
evt_delivery_class, evt_key = self._dm_spool_target_key(evt_payload)
if evt_delivery_class != delivery_class:
continue
if evt_key != key:
continue
evt_ts = float(evt_payload.get("timestamp", evt.get("timestamp", 0)) or 0)
if evt_ts > 0 and current - evt_ts > DM_HASHCHAIN_SPOOL_TTL_S:
continue
total_count += 1
if sender_id and str(evt.get("node_id", "") or "").strip() == sender_id:
sender_count += 1
if total_count >= DM_HASHCHAIN_SPOOL_LIMIT and (
not sender_id or sender_count >= DM_HASHCHAIN_SPOOL_SENDER_LIMIT
):
break
return total_count, sender_count
def _dm_spool_active_count(self, payload: dict[str, Any], *, now: float | None = None) -> int:
total_count, _sender_count = self._dm_spool_active_counts(payload, now=now)
return total_count
def append_private_dm_message(
self,
*,
node_id: str,
payload: dict,
signature: str,
sequence: int,
public_key: str,
public_key_algo: str,
protocol_version: str = "",
timestamp: float = 0,
) -> dict:
"""Append an encrypted DM dead-drop message to the private Infonet ledger.
The event is a small offline spool, capped per mailbox target, so the
hashchain can carry a couple of sealed DMs without becoming an
unbounded global mailbox.
"""
event_type = "dm_message"
if sequence <= 0:
raise ValueError("sequence is required and must be > 0")
sequence_table, sequence_key = self._sequence_table_for_event(event_type, node_id)
last = sequence_table.get(sequence_key, 0)
if sequence <= last:
raise ValueError(f"Replay detected: sequence {sequence} <= last {last}")
raw_payload = dict(payload or {})
if "message" in raw_payload or "plaintext" in raw_payload or "_local_plaintext" in raw_payload:
raise ValueError("private DM ledger payload must not contain plaintext")
if str(raw_payload.get("transport_lock", "") or "").strip().lower() != "private_strong":
raise ValueError("DM hashchain spool requires private_strong transport_lock")
payload = normalize_payload(event_type, raw_payload)
ok, reason = validate_private_dm_ledger_payload(payload)
if not ok:
raise ValueError(reason)
total_count, sender_count = self._dm_spool_active_counts(payload, sender_id=node_id)
if sender_count >= DM_HASHCHAIN_SPOOL_SENDER_LIMIT:
raise ValueError("DM hashchain sender spool full for recipient")
if total_count >= DM_HASHCHAIN_SPOOL_LIMIT:
raise ValueError("DM hashchain spool full for recipient")
payload_json = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
if len(payload_json.encode("utf-8")) > MAX_PAYLOAD_BYTES:
raise ValueError("payload exceeds max size")
protocol_version = str(protocol_version or PROTOCOL_VERSION)
ok, reason = validate_protocol_fields(protocol_version, NETWORK_ID)
if not ok:
raise ValueError(reason)
if not (signature and public_key and public_key_algo):
raise ValueError("Missing signature fields")
algo = parse_public_key_algo(public_key_algo)
if not algo:
raise ValueError("Unsupported public_key_algo")
if not verify_node_binding(node_id, public_key):
raise ValueError("node_id mismatch")
bound, bind_reason = self._bind_public_key(public_key, node_id)
if not bound:
raise ValueError(bind_reason)
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=sequence,
payload=payload,
)
if not verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
raise ValueError("Invalid signature")
revoked, _info = self._revocation_status(public_key)
if revoked:
raise ValueError("public key is revoked")
event = ChainEvent(
prev_hash=self.head_hash,
event_type=event_type,
node_id=node_id,
payload=payload,
timestamp=float(timestamp or time.time()),
sequence=sequence,
signature=signature,
public_key=public_key,
public_key_algo=public_key_algo,
protocol_version=protocol_version,
)
event_dict = event.to_dict()
self._write_wal(event_dict)
self.events.append(event_dict)
self.event_index[event.event_id] = len(self.events) - 1
self.head_hash = event.event_id
sequence_table[sequence_key] = sequence
self._replay_filter.add(event.event_id)
self._invalidate_merkle_cache()
self._update_counters_for_event(event_dict)
self._save()
try:
from services.mesh.mesh_rns import rns_bridge
rns_bridge.publish_event(event_dict)
except Exception:
pass
_notify_public_event_append_hooks(event_dict)
logger.info(
f"Infonet append [dm_message] by {_redact_node(node_id)} seq={sequence} "
f"id={event.event_id[:16]}..."
)
return event_dict
def append_private_gate_message(
self,
*,
node_id: str,
payload: dict,
signature: str,
sequence: int,
public_key: str,
public_key_algo: str,
protocol_version: str = "",
timestamp: float = 0,
) -> dict:
"""Append an encrypted gate message to the private Infonet ledger.
Gate messages use their own sequence domain so a gate post cannot
consume or replay-block the author's public broadcast sequence.
"""
event_type = "gate_message"
if sequence <= 0:
raise ValueError("sequence is required and must be > 0")
sequence_table, sequence_key = self._sequence_table_for_event(event_type, node_id)
last = sequence_table.get(sequence_key, 0)
if sequence <= last:
raise ValueError(f"Replay detected: sequence {sequence} <= last {last}")
raw_payload = dict(payload or {})
if "message" in raw_payload or "_local_plaintext" in raw_payload or "_local_reply_to" in raw_payload:
raise ValueError("private gate ledger payload must not contain plaintext")
if str(raw_payload.get("transport_lock", "") or "").strip().lower() != "private_strong":
raise ValueError("gate messages require private_strong transport_lock")
payload = normalize_payload(event_type, raw_payload)
ok, reason = validate_private_gate_ledger_payload(payload)
if not ok:
raise ValueError(reason)
payload_json = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
if len(payload_json.encode("utf-8")) > MAX_PAYLOAD_BYTES:
raise ValueError("payload exceeds max size")
protocol_version = str(protocol_version or PROTOCOL_VERSION)
ok, reason = validate_protocol_fields(protocol_version, NETWORK_ID)
if not ok:
raise ValueError(reason)
if not (signature and public_key and public_key_algo):
raise ValueError("Missing signature fields")
algo = parse_public_key_algo(public_key_algo)
if not algo:
raise ValueError("Unsupported public_key_algo")
if not verify_node_binding(node_id, public_key):
raise ValueError("node_id mismatch")
bound, bind_reason = self._bind_public_key(public_key, node_id)
if not bound:
raise ValueError(bind_reason)
event_for_signature = {"payload": payload}
signature_ok = False
for signature_payload in _private_gate_signature_payload_variants(
str(payload.get("gate", "") or ""),
event_for_signature,
):
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=sequence,
payload=signature_payload,
)
if verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
signature_ok = True
break
if not signature_ok:
raise ValueError("Invalid signature")
revoked, _info = self._revocation_status(public_key)
if revoked:
raise ValueError("public key is revoked")
event = ChainEvent(
prev_hash=self.head_hash,
event_type=event_type,
node_id=node_id,
payload=payload,
timestamp=float(timestamp or time.time()),
sequence=sequence,
signature=signature,
public_key=public_key,
public_key_algo=public_key_algo,
protocol_version=protocol_version,
)
event_dict = event.to_dict()
self._write_wal(event_dict)
self.events.append(event_dict)
self.event_index[event.event_id] = len(self.events) - 1
self.head_hash = event.event_id
sequence_table[sequence_key] = sequence
self._replay_filter.add(event.event_id)
self._invalidate_merkle_cache()
self._update_counters_for_event(event_dict)
self._save()
try:
from services.mesh.mesh_rns import rns_bridge
rns_bridge.publish_event(event_dict)
except Exception:
pass
_notify_public_event_append_hooks(event_dict)
logger.info(
f"Infonet append [gate_message] by {_redact_node(node_id)} seq={sequence} "
f"id={event.event_id[:16]}..."
)
return event_dict
def append(
self,
event_type: str,
@@ -2078,6 +2412,18 @@ class Infonet:
if not event_id or not prev_hash:
rejected.append({"index": idx, "reason": "Missing event_id or prev_hash"})
continue
if event_id in self.event_index:
duplicates += 1
continue
if self._replay_filter.seen(event_id):
try:
from services.mesh.mesh_metrics import increment as metrics_inc
metrics_inc("ingest_replay_seen")
except Exception:
pass
duplicates += 1
continue
if prev_hash != expected_prev:
try:
from services.mesh.mesh_metrics import increment as metrics_inc
@@ -2096,25 +2442,14 @@ class Infonet:
pass
rejected.append({"index": idx, "reason": "network_id mismatch"})
continue
if event_id in self.event_index:
duplicates += 1
continue
if self._replay_filter.seen(event_id):
try:
from services.mesh.mesh_metrics import increment as metrics_inc
metrics_inc("ingest_replay_seen")
except Exception:
pass
duplicates += 1
continue
if prev_hash != self.head_hash:
rejected.append({"index": idx, "reason": "prev_hash does not match head"})
continue
if sequence <= 0:
rejected.append({"index": idx, "reason": "Invalid sequence"})
continue
last = self.node_sequences.get(node_id, 0)
sequence_table, sequence_key = self._sequence_table_for_event(event_type, node_id)
last = sequence_table.get(sequence_key, 0)
if sequence <= last:
rejected.append({"index": idx, "reason": "Replay detected"})
continue
@@ -2149,7 +2484,18 @@ class Infonet:
if not ok:
rejected.append({"index": idx, "reason": reason})
continue
ok, reason = validate_public_ledger_payload(event_type, payload)
if event_type == "gate_message":
ok, reason = validate_private_gate_ledger_payload(payload)
elif event_type == "dm_message":
ok, reason = validate_private_dm_ledger_payload(payload)
if ok:
total_count, sender_count = self._dm_spool_active_counts(payload, sender_id=str(evt.get("node_id", "") or ""))
if sender_count >= DM_HASHCHAIN_SPOOL_SENDER_LIMIT:
ok, reason = False, "DM hashchain sender spool full for recipient"
elif total_count >= DM_HASHCHAIN_SPOOL_LIMIT:
ok, reason = False, "DM hashchain spool full for recipient"
else:
ok, reason = validate_public_ledger_payload(event_type, payload)
if not ok:
rejected.append({"index": idx, "reason": reason})
continue
@@ -2225,7 +2571,7 @@ class Infonet:
pass
rejected.append({"index": idx, "reason": "public key is revoked"})
continue
last_seq = self.node_sequences.get(node_id, 0)
last_seq = sequence_table.get(sequence_key, 0)
if sequence <= last_seq:
try:
from services.mesh.mesh_metrics import increment as metrics_inc
@@ -2261,18 +2607,30 @@ class Infonet:
rejected.append({"index": idx, "reason": bind_reason})
continue
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=sequence,
payload=payload,
)
if not verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
if event_type == "gate_message":
signature_payloads = _private_gate_signature_payload_variants(
str(payload.get("gate", "") or ""),
evt,
)
else:
signature_payloads = [payload]
signature_ok = False
for signature_payload in signature_payloads:
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=sequence,
payload=signature_payload,
)
if verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
signature_ok = True
break
if not signature_ok:
try:
from services.mesh.mesh_metrics import increment as metrics_inc
@@ -2302,7 +2660,7 @@ class Infonet:
self.events.append(evt)
self.event_index[event_id] = len(self.events) - 1
self.head_hash = event_id
self.node_sequences[node_id] = sequence
sequence_table[sequence_key] = sequence
self._update_counters_for_event(evt)
accepted += 1
expected_prev = event_id
@@ -2365,6 +2723,7 @@ class Infonet:
verify_node_binding,
)
event_type = evt_dict.get("event_type", "")
node_id = evt_dict.get("node_id", "")
if not parse_public_key_algo(public_key_algo):
return False, f"Unsupported public_key_algo at index {i}"
@@ -2375,21 +2734,41 @@ class Infonet:
return False, f"public key binding conflict at index {i}"
seen_public_keys[public_key] = node_id
normalized = normalize_payload(
evt_dict.get("event_type", ""), evt_dict.get("payload", {})
)
sig_payload = build_signature_payload(
event_type=evt_dict.get("event_type", ""),
node_id=node_id,
sequence=_safe_int(evt_dict.get("sequence", 0) or 0, 0),
payload=normalized,
)
if not verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
payload = evt_dict.get("payload", {})
if event_type == "gate_message":
ok, reason = validate_private_gate_ledger_payload(payload)
if not ok:
return False, f"Invalid gate_message payload at index {i}: {reason}"
signature_payloads = _private_gate_signature_payload_variants(
str(payload.get("gate", "") or ""),
evt_dict,
)
elif event_type == "dm_message":
ok, reason = validate_private_dm_ledger_payload(payload)
if not ok:
return False, f"Invalid dm_message payload at index {i}: {reason}"
signature_payloads = [normalize_payload(event_type, payload)]
else:
signature_payloads = [
normalize_payload(event_type, payload)
]
signature_ok = False
for signature_payload in signature_payloads:
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=_safe_int(evt_dict.get("sequence", 0) or 0, 0),
payload=signature_payload,
)
if verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
signature_ok = True
break
if not signature_ok:
return False, f"Invalid signature at index {i}"
prev = evt_dict["event_id"]
@@ -2454,27 +2833,48 @@ class Infonet:
verify_node_binding,
)
event_type = evt_dict.get("event_type", "")
node_id = evt_dict.get("node_id", "")
if not parse_public_key_algo(public_key_algo):
return False, f"Unsupported public_key_algo at index {i}"
if not verify_node_binding(node_id, public_key):
return False, f"node_id mismatch at index {i}"
normalized = normalize_payload(
evt_dict.get("event_type", ""), evt_dict.get("payload", {})
)
sig_payload = build_signature_payload(
event_type=evt_dict.get("event_type", ""),
node_id=node_id,
sequence=_safe_int(evt_dict.get("sequence", 0) or 0, 0),
payload=normalized,
)
if not verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
payload = evt_dict.get("payload", {})
if event_type == "gate_message":
ok, reason = validate_private_gate_ledger_payload(payload)
if not ok:
return False, f"Invalid gate_message payload at index {i}: {reason}"
signature_payloads = _private_gate_signature_payload_variants(
str(payload.get("gate", "") or ""),
evt_dict,
)
elif event_type == "dm_message":
ok, reason = validate_private_dm_ledger_payload(payload)
if not ok:
return False, f"Invalid dm_message payload at index {i}: {reason}"
signature_payloads = [normalize_payload(event_type, payload)]
else:
signature_payloads = [
normalize_payload(event_type, payload)
]
signature_ok = False
for signature_payload in signature_payloads:
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=_safe_int(evt_dict.get("sequence", 0) or 0, 0),
payload=signature_payload,
)
if verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
signature_ok = True
break
if not signature_ok:
return False, f"Invalid signature at index {i}"
prev = evt_dict["event_id"]
@@ -2538,7 +2938,14 @@ class Infonet:
node_id = evt.get("node_id", "")
sequence = _safe_int(evt.get("sequence", 0) or 0, 0)
if node_id and sequence:
last_seq[node_id] = max(last_seq.get(node_id, 0), sequence)
sequence_key = (
f"{node_id}|gate_message"
if str(evt.get("event_type", "") or "").strip().lower() == "gate_message"
else f"{node_id}|dm_message"
if str(evt.get("event_type", "") or "").strip().lower() == "dm_message"
else node_id
)
last_seq[sequence_key] = max(last_seq.get(sequence_key, 0), sequence)
public_key = str(evt.get("public_key", "") or "")
if public_key and node_id:
seen_public_keys.setdefault(public_key, node_id)
@@ -2558,8 +2965,21 @@ class Infonet:
existing_idx = self.event_index.get(event_id)
if existing_idx is not None and existing_idx <= prev_index:
return False, "duplicate event_id"
payload = normalize_payload(event_type, dict(payload or {}))
if event_type == "gate_message":
payload = dict(payload or {})
elif event_type == "dm_message":
payload = normalize_payload(event_type, dict(payload or {}))
else:
payload = normalize_payload(event_type, dict(payload or {}))
ok, reason = validate_event_payload(event_type, payload)
if not ok:
return False, reason
if event_type == "gate_message":
ok, reason = validate_private_gate_ledger_payload(payload)
elif event_type == "dm_message":
ok, reason = validate_private_dm_ledger_payload(payload)
else:
ok, reason = validate_public_ledger_payload(event_type, payload)
if not ok:
return False, reason
proto = evt.get("protocol_version") or PROTOCOL_VERSION
@@ -2573,7 +2993,14 @@ class Infonet:
revoked, _info = self._revocation_status(public_key)
if revoked and event_type != "key_revoke":
return False, "public key revoked"
last = last_seq.get(node_id, 0)
sequence_key = (
f"{node_id}|gate_message"
if event_type == "gate_message"
else f"{node_id}|dm_message"
if event_type == "dm_message"
else node_id
)
last = last_seq.get(sequence_key, 0)
if sequence <= last:
return False, "sequence replay"
from services.mesh.mesh_crypto import (
@@ -2591,23 +3018,35 @@ class Infonet:
if existing and existing != node_id:
return False, "public key binding conflict"
seen_public_keys[public_key] = node_id
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=sequence,
payload=payload,
)
if not verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
if event_type == "gate_message":
signature_payloads = _private_gate_signature_payload_variants(
str(payload.get("gate", "") or ""),
evt,
)
else:
signature_payloads = [payload]
signature_ok = False
for signature_payload in signature_payloads:
sig_payload = build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=sequence,
payload=signature_payload,
)
if verify_signature(
public_key_b64=public_key,
public_key_algo=public_key_algo,
signature_hex=signature,
payload=sig_payload,
):
signature_ok = True
break
if not signature_ok:
return False, "invalid signature"
computed = ChainEvent.from_dict(evt).event_id
if computed != event_id:
return False, "event_id mismatch"
last_seq[node_id] = sequence
last_seq[sequence_key] = sequence
# Apply fork
self.events = prefix + ordered
@@ -276,5 +276,6 @@ def should_run_sync(
) -> bool:
current_time = int(now if now is not None else time.time())
if state.last_outcome == "running":
return False
started_at = int(state.last_sync_started_at or 0)
return started_at <= 0 or current_time - started_at >= 300
return int(state.next_sync_due_at or 0) <= current_time
+144
View File
@@ -2,6 +2,9 @@
from __future__ import annotations
import base64
import binascii
import math
from dataclasses import dataclass
from typing import Any, Callable
@@ -33,6 +36,58 @@ def _require_fields(payload: dict[str, Any], fields: tuple[str, ...]) -> tuple[b
return True, "ok"
def _decode_base64ish(value: Any) -> bytes | None:
raw = str(value or "").strip()
if not raw or any(ch.isspace() for ch in raw):
return None
padded = raw + ("=" * (-len(raw) % 4))
for altchars in (None, b"-_"):
try:
return base64.b64decode(padded.encode("ascii"), altchars=altchars, validate=True)
except (binascii.Error, UnicodeEncodeError, ValueError):
continue
return None
def _byte_entropy(data: bytes) -> float:
if not data:
return 0.0
counts = [0] * 256
for byte in data:
counts[byte] += 1
total = float(len(data))
return -sum((count / total) * math.log2(count / total) for count in counts if count)
def _validate_sealed_bytes_field(
payload: dict[str, Any],
field: str,
*,
min_bytes: int = 8,
entropy_floor: float = 2.5,
) -> tuple[bool, str]:
data = _decode_base64ish(payload.get(field, ""))
if data is None:
return False, f"{field} must be base64-encoded sealed bytes"
if len(data) < min_bytes:
return False, f"{field} is too short"
# Short test vectors and compact envelopes can be low entropy; only apply
# heuristics once there is enough material to distinguish a sealed blob
# from accidental base64-encoded plaintext.
if len(data) >= 32:
printable = sum(1 for byte in data if 32 <= byte <= 126 or byte in (9, 10, 13))
if printable / len(data) > 0.9:
try:
data.decode("utf-8")
return False, f"{field} looks like encoded plaintext"
except UnicodeDecodeError:
pass
if _byte_entropy(data) < entropy_floor:
return False, f"{field} entropy is too low for sealed bytes"
return True, "ok"
def _validate_message(payload: dict[str, Any]) -> tuple[bool, str]:
ok, reason = _require_fields(
payload, ("message", "destination", "channel", "priority", "ephemeral")
@@ -331,6 +386,7 @@ ACTIVE_PUBLIC_LEDGER_EVENT_TYPES: frozenset[str] = frozenset(
LEGACY_PUBLIC_LEDGER_EVENT_TYPES: frozenset[str] = frozenset(
{
"gate_message",
"dm_message",
}
)
"""Event types that exist historically on the public chain and must remain
@@ -425,6 +481,8 @@ def validate_event_payload(event_type: str, payload: dict[str, Any]) -> tuple[bo
def validate_public_ledger_payload(event_type: str, payload: dict[str, Any]) -> tuple[bool, str]:
if event_type == "gate_message":
return validate_private_gate_ledger_payload(payload)
if event_type not in PUBLIC_LEDGER_EVENT_TYPES and event_type not in _EXTENSION_VALIDATORS:
return False, f"{event_type} is not allowed on the public ledger"
forbidden = sorted(
@@ -441,6 +499,92 @@ def validate_public_ledger_payload(event_type: str, payload: dict[str, Any]) ->
return True, "ok"
_PRIVATE_GATE_LEDGER_ALLOWED_FIELDS: frozenset[str] = frozenset(
{
"gate",
"ciphertext",
"nonce",
"sender_ref",
"format",
"epoch",
"gate_envelope",
"envelope_hash",
"reply_to",
"transport_lock",
"signed_context",
}
)
def validate_private_gate_ledger_payload(payload: dict[str, Any]) -> tuple[bool, str]:
"""Validate ciphertext-only gate events for private Infonet replication."""
ok, reason = validate_event_payload("gate_message", payload)
if not ok:
return ok, reason
unexpected = sorted(
key
for key in payload.keys()
if str(key or "").strip().lower() not in _PRIVATE_GATE_LEDGER_ALLOWED_FIELDS
)
if unexpected:
return False, f"private gate ledger payload contains unsupported fields: {', '.join(unexpected)}"
if "message" in payload or "_local_plaintext" in payload or "_local_reply_to" in payload:
return False, "private gate ledger payload must not contain plaintext"
transport_lock = str(payload.get("transport_lock", "") or "").strip().lower()
if transport_lock and transport_lock not in {"private", "private_strong", "rns", "onion"}:
return False, "gate messages require private transport_lock"
ok, reason = _validate_sealed_bytes_field(payload, "ciphertext")
if not ok:
return ok, reason
ok, reason = _validate_sealed_bytes_field(payload, "nonce")
if not ok:
return ok, reason
return True, "ok"
_PRIVATE_DM_LEDGER_ALLOWED_FIELDS: frozenset[str] = frozenset(
{
"recipient_id",
"delivery_class",
"recipient_token",
"ciphertext",
"msg_id",
"timestamp",
"format",
"session_welcome",
"sender_seal",
"relay_salt",
"transport_lock",
"signed_context",
}
)
def validate_private_dm_ledger_payload(payload: dict[str, Any]) -> tuple[bool, str]:
"""Validate ciphertext-only DM dead-drop events for private Infonet replication."""
ok, reason = validate_event_payload("dm_message", payload)
if not ok:
return ok, reason
unexpected = sorted(
key
for key in payload.keys()
if str(key or "").strip().lower() not in _PRIVATE_DM_LEDGER_ALLOWED_FIELDS
)
if unexpected:
return False, f"private DM ledger payload contains unsupported fields: {', '.join(unexpected)}"
if "message" in payload or "plaintext" in payload or "_local_plaintext" in payload:
return False, "private DM ledger payload must not contain plaintext"
transport_lock = str(payload.get("transport_lock", "") or "").strip().lower()
if transport_lock != "private_strong":
return False, "DM hashchain spool requires private_strong transport_lock"
if not str(payload.get("ciphertext", "") or "").strip():
return False, "ciphertext cannot be empty"
ok, reason = _validate_sealed_bytes_field(payload, "ciphertext")
if not ok:
return ok, reason
return True, "ok"
def validate_protocol_fields(protocol_version: str, network_id: str) -> tuple[bool, str]:
if protocol_version != PROTOCOL_VERSION:
return False, "Unsupported protocol_version"
+42 -13
View File
@@ -87,11 +87,28 @@ def _run_gate_release_once(monkeypatch, *, transport_tier="private_strong"):
def _patch_for_successful_post(monkeypatch, module):
"""Apply standard monkeypatches so a gate_message post succeeds."""
import main
from services.mesh import mesh_hashchain
_setup_gate_outbox(monkeypatch)
monkeypatch.setattr(main, "_verify_gate_message_signed_write", lambda **kw: (True, "ok", kw.get("reply_to", "")))
monkeypatch.setattr(main, "_resolve_envelope_policy", lambda _gate_id: "envelope_disabled")
def _fake_private_gate_append(**kwargs):
return {
"event_id": f"ledger-ev-{kwargs.get('sequence', 0)}",
"event_type": "gate_message",
"node_id": kwargs["node_id"],
"payload": dict(kwargs["payload"]),
"timestamp": kwargs.get("timestamp", 0) or 123.0,
"sequence": kwargs["sequence"],
"signature": kwargs["signature"],
"public_key": kwargs["public_key"],
"public_key_algo": kwargs["public_key_algo"],
"protocol_version": kwargs.get("protocol_version", "infonet/2"),
}
monkeypatch.setattr(mesh_hashchain.infonet, "append_private_gate_message", _fake_private_gate_append)
from services.mesh.mesh_reputation import gate_manager, reputation_ledger
monkeypatch.setattr(gate_manager, "can_enter", lambda *a, **kw: (True, "ok"))
@@ -255,19 +272,30 @@ def test_gate_post_preserves_gate_envelope_in_store(monkeypatch):
def test_gate_post_advances_sequence(monkeypatch):
"""validate_and_set_sequence must be called to advance the counter."""
"""append_private_gate_message must receive the gate sequence."""
import main
from services.mesh import mesh_hashchain
_patch_for_successful_post(monkeypatch, main)
seq_calls = []
append_calls = []
def track_seq(node_id, seq, *, domain=""):
seq_calls.append((node_id, seq, domain))
return (True, "ok")
def track_private_append(**kwargs):
append_calls.append(kwargs)
return {
"event_id": "ev-seq",
"event_type": "gate_message",
"node_id": kwargs["node_id"],
"payload": dict(kwargs["payload"]),
"timestamp": kwargs.get("timestamp", 0) or 123.0,
"sequence": kwargs["sequence"],
"signature": kwargs["signature"],
"public_key": kwargs["public_key"],
"public_key_algo": kwargs["public_key_algo"],
"protocol_version": kwargs.get("protocol_version", "infonet/2"),
}
monkeypatch.setattr(mesh_hashchain.infonet, "validate_and_set_sequence", track_seq)
monkeypatch.setattr(mesh_hashchain.infonet, "append_private_gate_message", track_private_append)
monkeypatch.setattr(
mesh_hashchain.gate_store,
"append",
@@ -280,8 +308,9 @@ def test_gate_post_advances_sequence(monkeypatch):
assert result["ok"] is True
assert result["queued"] is True
assert len(seq_calls) == 1
assert seq_calls[0] == ("!sb_test1234567890", 42, "gate_message")
assert len(append_calls) == 1
assert append_calls[0]["node_id"] == "!sb_test1234567890"
assert append_calls[0]["sequence"] == 42
def test_gate_post_rejects_replay_via_sequence(monkeypatch):
@@ -290,11 +319,11 @@ def test_gate_post_rejects_replay_via_sequence(monkeypatch):
from services.mesh import mesh_hashchain
_patch_for_successful_post(monkeypatch, main)
monkeypatch.setattr(
mesh_hashchain.infonet,
"validate_and_set_sequence",
lambda node_id, seq: (False, "Replay detected: sequence 1 <= last 1"),
)
def reject_private_append(**_kwargs):
raise ValueError("Replay detected: sequence 1 <= last 1")
monkeypatch.setattr(mesh_hashchain.infonet, "append_private_gate_message", reject_private_append)
gate_id = "infonet"
body = _build_gate_message_body(gate_id, sequence=1)
@@ -117,3 +117,11 @@ def test_finish_solo_sync_marks_first_node_ready_without_peer_failure():
assert finished.next_sync_due_at == 500
assert should_run_sync(finished, now=499) is False
assert should_run_sync(finished, now=500) is True
def test_should_run_sync_recovers_stale_running_state():
fresh = SyncWorkerState(last_sync_started_at=100, last_outcome="running")
stale = SyncWorkerState(last_sync_started_at=100, last_outcome="running")
assert should_run_sync(fresh, now=399) is False
assert should_run_sync(stale, now=400) is True
@@ -8,6 +8,53 @@ from cryptography.hazmat.primitives.asymmetric import ed25519
from httpx import ASGITransport, AsyncClient
def test_onion_peer_requests_use_arti_socks_proxy(monkeypatch):
import main
from services import wormhole_supervisor
monkeypatch.setattr(main, "_infonet_private_transport_required", lambda: True)
monkeypatch.setattr(
main,
"get_settings",
lambda: SimpleNamespace(MESH_ARTI_ENABLED=True, MESH_ARTI_SOCKS_PORT=19050),
)
monkeypatch.setattr(wormhole_supervisor, "_check_arti_ready", lambda: True)
proxies = main._infonet_peer_requests_proxies("http://exampleabcd.onion:8000")
assert proxies == {
"http": "socks5h://127.0.0.1:19050",
"https": "socks5h://127.0.0.1:19050",
}
def test_private_peer_requests_reject_clearnet(monkeypatch):
import main
monkeypatch.setattr(main, "_infonet_private_transport_required", lambda: True)
try:
main._infonet_peer_requests_proxies("https://seed.example")
except RuntimeError as exc:
assert "private Infonet requires onion/RNS transport" in str(exc)
else:
raise AssertionError("clearnet peer was allowed while private transport is required")
def test_local_peer_url_prefers_configured_public_peer_url(monkeypatch):
import main
monkeypatch.setattr(
main,
"get_settings",
lambda: SimpleNamespace(
MESH_PUBLIC_PEER_URL="HTTP://LOCALPEEREXAMPLE.onion:8000/",
),
)
assert main._local_infonet_peer_url() == "http://localpeerexample.onion:8000"
def _write_signed_manifest(path, *, private_key):
from services.mesh.mesh_bootstrap_manifest import BOOTSTRAP_MANIFEST_VERSION
from services.mesh.mesh_crypto import canonical_json
@@ -142,6 +189,134 @@ def test_refresh_node_peer_store_suppresses_clearnet_seed_by_default(tmp_path, m
assert store.records_for_bucket("sync") == []
def test_refresh_node_peer_store_prunes_persisted_clearnet_records_in_private_mode(tmp_path, monkeypatch):
import main
from services.config import get_settings
from services.mesh import mesh_peer_store as peer_store_mod
peer_store_path = tmp_path / "peer_store.json"
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
store = peer_store_mod.PeerStore(peer_store_path)
store.upsert(
peer_store_mod.make_bootstrap_peer_record(
peer_url="https://node.shadowbroker.info",
transport="clearnet",
role="seed",
signer_id="shadowbroker-default",
now=1_749_999_900,
)
)
store.upsert(
peer_store_mod.make_sync_peer_record(
peer_url="https://node.shadowbroker.info",
transport="clearnet",
role="seed",
source="bundle",
now=1_749_999_900,
)
)
store.upsert(
peer_store_mod.make_push_peer_record(
peer_url="https://node.shadowbroker.info",
transport="clearnet",
role="relay",
now=1_749_999_900,
)
)
store.save()
onion_seed = "http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"
monkeypatch.setenv("MESH_RELAY_PEERS", "")
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", onion_seed)
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "")
get_settings.cache_clear()
try:
snapshot = main._refresh_node_peer_store(now=1_750_000_000)
store = peer_store_mod.PeerStore(peer_store_path)
store.load()
finally:
get_settings.cache_clear()
assert snapshot["private_transport_required"] is True
assert snapshot["pruned_clearnet_peer_count"] == 3
assert [record.peer_url for record in store.records()] == [onion_seed, onion_seed]
assert {record.bucket for record in store.records()} == {"bootstrap", "sync"}
assert all(record.transport == "onion" for record in store.records())
def test_infonet_peer_url_filter_excludes_clearnet_in_private_mode(monkeypatch):
import main
from services.config import get_settings
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
get_settings.cache_clear()
try:
assert main._filter_infonet_peer_urls(
[
"https://node.shadowbroker.info",
"http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000",
]
) == ["http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"]
finally:
get_settings.cache_clear()
def test_public_sync_cycle_backs_off_on_429_retry_after(tmp_path, monkeypatch):
import time
import main
from services.config import get_settings
from services.mesh import mesh_peer_store as peer_store_mod
peer_store_path = tmp_path / "peer_store.json"
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
onion_seed = "http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"
store = peer_store_mod.PeerStore(peer_store_path)
store.upsert(
peer_store_mod.make_sync_peer_record(
peer_url=onion_seed,
transport="onion",
role="seed",
source="bundle",
now=1_750_000_000,
)
)
store.save()
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
monkeypatch.setenv("MESH_SYNC_FAILURE_BACKOFF_S", "60")
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_FAILURE_COOLDOWN_S", "15")
get_settings.cache_clear()
monkeypatch.setattr(main, "_participant_node_enabled", lambda: True)
monkeypatch.setattr(main, "_ensure_infonet_private_transport_ready", lambda reason="": True)
monkeypatch.setattr(
main,
"_sync_from_peer",
lambda peer_url: (_ for _ in ()).throw(
main.PeerSyncHTTPError(429, "rate limited", retry_after_s=180)
),
)
main.set_sync_state(main.SyncWorkerState())
try:
before = int(time.time())
state = main._run_public_sync_cycle()
store = peer_store_mod.PeerStore(peer_store_path)
store.load()
finally:
get_settings.cache_clear()
main.set_sync_state(main.SyncWorkerState())
record = store.records_for_bucket("sync")[0]
assert state.last_error == "HTTP 429: rate limited"
assert state.next_sync_due_at >= before + 180
assert record.cooldown_until >= before + 180
def test_verify_peer_push_hmac_requires_allowlisted_peer(monkeypatch):
import hashlib
import hmac
@@ -225,3 +400,29 @@ def test_public_sync_cycle_allows_first_node_without_peers(tmp_path, monkeypatch
assert result.last_error == ""
assert result.last_peer_url == ""
assert result.consecutive_failures == 0
def test_headless_mesh_node_runtime_is_explicit(monkeypatch):
import main
monkeypatch.setattr(main, "_MESH_ONLY", True)
monkeypatch.setattr(main, "_HEADLESS_MESH_NODE_RUNTIME", False)
assert main._infonet_node_runtime_requested() is False
monkeypatch.setattr(main, "_HEADLESS_MESH_NODE_RUNTIME", True)
assert main._infonet_node_runtime_requested() is True
def test_meshnode_scripts_enable_private_hashchain_runtime():
from pathlib import Path
root = Path(__file__).resolve().parents[3]
bat = (root / "meshnode.bat").read_text(encoding="utf-8")
sh = (root / "meshnode.sh").read_text(encoding="utf-8")
for script in (bat, sh):
assert "SHADOWBROKER_MESH_NODE_RUNTIME=true" in script
assert "MESH_INFONET_ALLOW_CLEARNET_SYNC=false" in script
assert "MESH_ARTI_ENABLED=true" in script
assert "MESH_DM_HASHCHAIN_SPOOL_LIMIT=2" in script
assert "gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000" in script
@@ -0,0 +1,213 @@
import base64
import time
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
from services.config import get_settings
from services.mesh import mesh_crypto, mesh_dm_relay, mesh_hashchain, mesh_protocol, mesh_secure_storage
def _keypair():
private_key = ed25519.Ed25519PrivateKey.generate()
public_raw = private_key.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
public_key = base64.b64encode(public_raw).decode("utf-8")
node_id = mesh_crypto.derive_node_id(public_key)
return private_key, public_key, node_id
def _payload(recipient_id: str = "recipient-a", msg_id: str = "dm-1") -> dict:
return mesh_protocol.normalize_payload(
"dm_message",
{
"recipient_id": recipient_id,
"delivery_class": "request",
"recipient_token": "",
"ciphertext": base64.b64encode(f"cipher-{msg_id}".encode("utf-8")).decode("ascii"),
"msg_id": msg_id,
"timestamp": int(time.time()),
"format": "mls1",
"transport_lock": "private_strong",
},
)
def _signature(private_key, node_id: str, sequence: int, payload: dict) -> str:
signature_payload = mesh_crypto.build_signature_payload(
event_type="dm_message",
node_id=node_id,
sequence=sequence,
payload=payload,
)
return private_key.sign(signature_payload.encode("utf-8")).hex()
def _fresh_infonet(tmp_path, monkeypatch) -> mesh_hashchain.Infonet:
monkeypatch.setattr(mesh_hashchain, "DATA_DIR", tmp_path)
monkeypatch.setattr(mesh_hashchain, "CHAIN_FILE", tmp_path / "infonet.json")
monkeypatch.setattr(mesh_hashchain, "WAL_FILE", tmp_path / "infonet.wal")
return mesh_hashchain.Infonet()
def _fresh_relay(tmp_path, monkeypatch) -> mesh_dm_relay.DMRelay:
monkeypatch.setattr(mesh_dm_relay, "DATA_DIR", tmp_path)
monkeypatch.setattr(mesh_dm_relay, "RELAY_FILE", tmp_path / "dm_relay.json")
monkeypatch.setattr(mesh_secure_storage, "DATA_DIR", tmp_path)
monkeypatch.setattr(mesh_secure_storage, "MASTER_KEY_FILE", tmp_path / "wormhole_secure_store.key")
get_settings.cache_clear()
return mesh_dm_relay.DMRelay()
def test_private_dm_hashchain_spools_two_ciphertexts_per_recipient_from_distinct_senders(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
senders = [_keypair(), _keypair()]
for idx, (private_key, public_key, node_id) in enumerate(senders, start=1):
payload = _payload(msg_id=f"dm-{idx}")
event = inf.append_private_dm_message(
node_id=node_id,
payload=payload,
signature=_signature(private_key, node_id, 1, payload),
sequence=1,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
timestamp=float(payload["timestamp"]),
)
assert event["event_type"] == "dm_message"
private_key, public_key, node_id = _keypair()
third = _payload(msg_id="dm-3")
try:
inf.append_private_dm_message(
node_id=node_id,
payload=third,
signature=_signature(private_key, node_id, 1, third),
sequence=1,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
timestamp=float(third["timestamp"]),
)
except ValueError as exc:
assert "spool full" in str(exc)
else:
raise AssertionError("third DM spool event was accepted")
for _private_key, _public_key, sender_node_id in senders:
assert inf.sequence_domains[f"{sender_node_id}|dm_message"] == 1
assert inf.validate_chain(verify_signatures=True)[0] is True
def test_private_dm_hashchain_limits_one_active_spool_per_sender_recipient_pair(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
first = _payload(msg_id="dm-1")
inf.append_private_dm_message(
node_id=node_id,
payload=first,
signature=_signature(private_key, node_id, 1, first),
sequence=1,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
timestamp=float(first["timestamp"]),
)
second = _payload(msg_id="dm-2")
try:
inf.append_private_dm_message(
node_id=node_id,
payload=second,
signature=_signature(private_key, node_id, 2, second),
sequence=2,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
timestamp=float(second["timestamp"]),
)
except ValueError as exc:
assert "sender spool full" in str(exc)
else:
raise AssertionError("second DM from same sender to same recipient was accepted")
def test_private_dm_hashchain_rejects_plaintext(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
payload = _payload()
payload["message"] = "plaintext"
try:
inf.append_private_dm_message(
node_id=node_id,
payload=payload,
signature=_signature(private_key, node_id, 1, _payload()),
sequence=1,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
)
except ValueError as exc:
assert "plaintext" in str(exc)
else:
raise AssertionError("private DM append accepted plaintext")
def test_private_dm_hashchain_rejects_non_sealed_ciphertext_shape(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
payload = _payload()
payload["ciphertext"] = "not sealed plaintext"
try:
inf.append_private_dm_message(
node_id=node_id,
payload=payload,
signature=_signature(private_key, node_id, 1, payload),
sequence=1,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
)
except ValueError as exc:
assert "sealed bytes" in str(exc)
else:
raise AssertionError("private DM append accepted non-base64 ciphertext")
def test_hydrate_dm_relay_from_chain_delivers_to_poll_claim(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path / "chain", monkeypatch)
relay = _fresh_relay(tmp_path / "relay", monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", inf)
monkeypatch.setattr(mesh_dm_relay, "dm_relay", relay)
private_key, public_key, node_id = _keypair()
payload = _payload(recipient_id="recipient-a", msg_id="dm-chain-1")
event = inf.append_private_dm_message(
node_id=node_id,
payload=payload,
signature=_signature(private_key, node_id, 1, payload),
sequence=1,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
timestamp=float(payload["timestamp"]),
)
from main import _hydrate_dm_relay_from_chain
assert _hydrate_dm_relay_from_chain([event]) == 1
messages, more = relay.collect_claims(
"recipient-a",
[{"type": "requests", "token": "recipient-request-token"}],
limit=8,
)
assert more is False
assert [message["msg_id"] for message in messages] == ["dm-chain-1"]
assert messages[0]["ciphertext"] == payload["ciphertext"]
@@ -0,0 +1,269 @@
import base64
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
from services.mesh import mesh_crypto, mesh_hashchain, mesh_protocol
def _keypair():
private_key = ed25519.Ed25519PrivateKey.generate()
public_raw = private_key.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
public_key = base64.b64encode(public_raw).decode("utf-8")
node_id = mesh_crypto.derive_node_id(public_key)
return private_key, public_key, node_id
def _sign(private_key, *, event_type: str, node_id: str, sequence: int, payload: dict) -> str:
signature_payload = mesh_crypto.build_signature_payload(
event_type=event_type,
node_id=node_id,
sequence=sequence,
payload=payload,
)
return private_key.sign(signature_payload.encode("utf-8")).hex()
def _message_payload(text: str) -> dict:
return mesh_protocol.normalize_payload(
"message",
{
"message": text,
"destination": "broadcast",
"channel": "LongFast",
"priority": "normal",
"ephemeral": False,
},
)
def _gate_payload(gate_id: str = "ops-gate", *, epoch: int = 2, plaintext: bool = False) -> dict:
payload = {
"gate": gate_id,
"ciphertext": base64.b64encode(b"encrypted-gate-ciphertext").decode("ascii"),
"nonce": base64.b64encode(b"nonce-value-1234").decode("ascii"),
"sender_ref": "sender-ref-1",
"format": "mls1",
"transport_lock": "private_strong",
}
if epoch > 0:
payload["epoch"] = epoch
if plaintext:
payload["message"] = "this must never land on the chain"
return mesh_protocol.normalize_payload("gate_message", payload) if not plaintext else payload
def _gate_event(
private_key,
public_key: str,
node_id: str,
*,
sequence: int,
prev_hash: str,
payload: dict,
signature_payload: dict | None = None,
) -> dict:
signature = _sign(
private_key,
event_type="gate_message",
node_id=node_id,
sequence=sequence,
payload=signature_payload or payload,
)
return mesh_hashchain.ChainEvent(
prev_hash=prev_hash,
event_type="gate_message",
node_id=node_id,
payload=payload,
timestamp=1234.0 + sequence,
sequence=sequence,
signature=signature,
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
network_id=mesh_protocol.NETWORK_ID,
).to_dict()
def _fresh_infonet(tmp_path, monkeypatch) -> mesh_hashchain.Infonet:
monkeypatch.setattr(mesh_hashchain, "DATA_DIR", tmp_path)
monkeypatch.setattr(mesh_hashchain, "CHAIN_FILE", tmp_path / "infonet.json")
monkeypatch.setattr(mesh_hashchain, "WAL_FILE", tmp_path / "infonet.wal")
return mesh_hashchain.Infonet()
def test_private_gate_fork_uses_gate_sequence_domain_and_signature_variants(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
public_payload = _message_payload("public prefix")
public_event = inf.append(
event_type="message",
node_id=node_id,
payload=public_payload,
sequence=1,
signature=_sign(
private_key,
event_type="message",
node_id=node_id,
sequence=1,
payload=public_payload,
),
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
)
gate_payload = _gate_payload(epoch=3)
signature_payload = dict(gate_payload)
signature_payload.pop("epoch", None)
gate_event = _gate_event(
private_key,
public_key,
node_id,
sequence=1,
prev_hash=public_event["event_id"],
payload=gate_payload,
signature_payload=signature_payload,
)
ok, reason = inf.apply_fork([gate_event], gate_event["event_id"], proof_count=2, quorum=2)
assert ok is True, reason
assert inf.events[-1]["event_type"] == "gate_message"
assert inf.node_sequences[node_id] == 1
assert inf.sequence_domains[f"{node_id}|gate_message"] == 1
assert inf.validate_chain(verify_signatures=True)[0] is True
def test_private_gate_fork_rejects_plaintext_payload(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
public_payload = _message_payload("public prefix")
public_event = inf.append(
event_type="message",
node_id=node_id,
payload=public_payload,
sequence=1,
signature=_sign(
private_key,
event_type="message",
node_id=node_id,
sequence=1,
payload=public_payload,
),
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
)
plaintext_payload = _gate_payload(plaintext=True)
gate_event = _gate_event(
private_key,
public_key,
node_id,
sequence=1,
prev_hash=public_event["event_id"],
payload=plaintext_payload,
)
ok, reason = inf.apply_fork([gate_event], gate_event["event_id"], proof_count=2, quorum=2)
assert ok is False
assert "normalized" in reason or "plaintext" in reason
assert len(inf.events) == 1
assert "gate_message" not in inf.get_info()["event_types"]
def test_append_private_gate_message_rejects_plaintext_before_normalizing(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
payload = _gate_payload()
payload["message"] = "plaintext should not be silently dropped"
try:
inf.append_private_gate_message(
node_id=node_id,
payload=payload,
sequence=1,
signature=_sign(
private_key,
event_type="gate_message",
node_id=node_id,
sequence=1,
payload=_gate_payload(),
),
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
)
except ValueError as exc:
assert "plaintext" in str(exc)
else:
raise AssertionError("private gate append accepted plaintext")
assert inf.events == []
def test_append_private_gate_message_requires_private_strong_transport_lock(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
payload = _gate_payload()
payload.pop("transport_lock", None)
try:
inf.append_private_gate_message(
node_id=node_id,
payload=payload,
sequence=1,
signature=_sign(
private_key,
event_type="gate_message",
node_id=node_id,
sequence=1,
payload=_gate_payload(),
),
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
)
except ValueError as exc:
assert "private_strong" in str(exc)
else:
raise AssertionError("private gate append accepted missing transport_lock")
assert inf.events == []
def test_append_private_gate_message_rejects_non_sealed_ciphertext_shape(tmp_path, monkeypatch):
inf = _fresh_infonet(tmp_path, monkeypatch)
private_key, public_key, node_id = _keypair()
payload = _gate_payload()
payload["ciphertext"] = "not sealed plaintext"
try:
inf.append_private_gate_message(
node_id=node_id,
payload=payload,
sequence=1,
signature=_sign(
private_key,
event_type="gate_message",
node_id=node_id,
sequence=1,
payload=payload,
),
public_key=public_key,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
)
except ValueError as exc:
assert "sealed bytes" in str(exc)
else:
raise AssertionError("private gate append accepted non-base64 ciphertext")
assert inf.events == []
@@ -1,14 +1,12 @@
"""S14B Public Sync Gate Event Filter.
"""S14B private sync gate event policy.
Tests:
- GET /api/mesh/infonet/sync excludes gate_message when local infonet contains legacy gate_message plus public events
- POST /api/mesh/infonet/sync excludes gate_message under the same condition
- Both main app and router-served paths are covered
- Non-gate public redactions still hold (vote gate label stripped, key_rotate identity stripped)
- Do not overclaim that gate_message is removed from historical infonet storage or ingest
Private Infonet sync carries encrypted gate_message ledger events. If a node
is configured to allow clearnet-compatible sync, those gate events are filtered
out of the sync response.
"""
import asyncio
import base64
import json
from starlette.requests import Request
@@ -17,9 +15,6 @@ import main
from services.mesh import mesh_hashchain
# ── Helpers ──────────────────────────────────────────────────────────────
def _message_event() -> dict:
return {
"event_id": "msg-1",
@@ -83,6 +78,7 @@ def _gate_message_event() -> dict:
"nonce": "nonce-1",
"sender_ref": "sender-ref-1",
"format": "mls1",
"transport_lock": "private_strong",
},
"timestamp": 103.0,
"sequence": 4,
@@ -93,9 +89,31 @@ def _gate_message_event() -> dict:
}
class _FakeInfonet:
"""Minimal fake infonet with a gate_message among public events."""
def _dm_message_event() -> dict:
return {
"event_id": "dm-1",
"event_type": "dm_message",
"node_id": "!node-5",
"payload": {
"recipient_id": "recipient-a",
"delivery_class": "request",
"recipient_token": "",
"ciphertext": base64.b64encode(b"sealed-dm-ciphertext").decode("ascii"),
"msg_id": "dm-1",
"timestamp": 104,
"format": "mls1",
"transport_lock": "private_strong",
},
"timestamp": 104.0,
"sequence": 5,
"signature": "sig",
"public_key": "pub",
"public_key_algo": "Ed25519",
"protocol_version": "infonet/2",
}
class _FakeInfonet:
def __init__(self):
self.head_hash = "head-1"
self.events = [
@@ -113,12 +131,10 @@ class _FakeInfonet:
return int(getattr(limit, "default", 100) or 100)
def get_events_after(self, after_hash: str, limit=100):
resolved = self._limit_value(limit)
return [dict(e) for e in self.events[:resolved]]
return [dict(e) for e in self.events[: self._limit_value(limit)]]
def get_events_after_locator(self, locator: list[str], limit=100):
resolved = self._limit_value(limit)
return self.head_hash, 0, [dict(e) for e in self.events[:resolved]]
return self.head_hash, 0, [dict(e) for e in self.events[: self._limit_value(limit)]]
def get_merkle_proofs(self, start_index: int, count: int):
return {"root": "merkle-root", "total": len(self.events), "start": start_index, "proofs": []}
@@ -127,7 +143,7 @@ class _FakeInfonet:
return "merkle-root"
def _json_request(path: str, body: dict) -> Request:
def _json_request(path: str, body: dict, *, client_host: str = "127.0.0.1", headers: dict[str, str] | None = None) -> Request:
payload = json.dumps(body).encode("utf-8")
sent = {"value": False}
@@ -137,11 +153,14 @@ def _json_request(path: str, body: dict) -> Request:
sent["value"] = True
return {"type": "http.request", "body": payload, "more_body": False}
raw_headers = [(b"content-type", b"application/json")]
for key, value in dict(headers or {}).items():
raw_headers.append((key.lower().encode("ascii"), str(value).encode("ascii")))
return Request(
{
"type": "http",
"headers": [(b"content-type", b"application/json")],
"client": ("test", 12345),
"headers": raw_headers,
"client": (client_host, 12345),
"method": "POST",
"path": path,
},
@@ -149,20 +168,15 @@ def _json_request(path: str, body: dict) -> Request:
)
def _get_request(path: str) -> Request:
sent = {"value": False}
def _get_request(path: str, *, client_host: str = "127.0.0.1", headers: dict[str, str] | None = None) -> Request:
async def receive():
if sent["value"]:
return {"type": "http.request", "body": b"", "more_body": False}
sent["value"] = True
return {"type": "http.request", "body": b"", "more_body": False}
return Request(
{
"type": "http",
"headers": [],
"client": ("test", 12345),
"headers": [(key.lower().encode("ascii"), str(value).encode("ascii")) for key, value in dict(headers or {}).items()],
"client": (client_host, 12345),
"method": "GET",
"path": path,
},
@@ -170,120 +184,166 @@ def _get_request(path: str) -> Request:
)
# ── GET sync excludes gate_message (main app) ──────────────────────────
def _force_private_sync(monkeypatch):
monkeypatch.setattr(main, "_infonet_private_transport_required", lambda: True)
monkeypatch.setattr(main, "_request_appears_private_infonet_transport", lambda request: True)
def test_get_sync_excludes_gate_message(client, monkeypatch):
"""GET /api/mesh/infonet/sync must not return gate_message events."""
def _force_private_policy_only(monkeypatch):
monkeypatch.setattr(main, "_infonet_private_transport_required", lambda: True)
def _force_clearnet_sync(monkeypatch):
monkeypatch.setattr(main, "_infonet_private_transport_required", lambda: False)
def _event_types(events: list[dict]) -> list[str]:
return [str(e.get("event_type", "")) for e in events]
def test_private_sync_redacts_private_events_from_exposed_clearnet_request(monkeypatch):
_force_private_policy_only(monkeypatch)
request = _get_request("/api/mesh/infonet/sync", client_host="203.0.113.10")
events = main._infonet_sync_response_events(
[_message_event(), _gate_message_event(), _dm_message_event()],
request=request,
)
assert _event_types(events) == ["message"]
def test_private_sync_includes_private_events_for_loopback_request(monkeypatch):
_force_private_policy_only(monkeypatch)
request = _get_request("/api/mesh/infonet/sync", client_host="127.0.0.1")
events = main._infonet_sync_response_events(
[_message_event(), _gate_message_event(), _dm_message_event()],
request=request,
)
assert _event_types(events) == ["message", "gate_message", "dm_message"]
def test_private_sync_redacts_private_events_when_forwarded_for_is_clearnet(monkeypatch):
_force_private_policy_only(monkeypatch)
request = _get_request(
"/api/mesh/infonet/sync",
client_host="127.0.0.1",
headers={"x-forwarded-for": "198.51.100.44"},
)
events = main._infonet_sync_response_events(
[_message_event(), _gate_message_event(), _dm_message_event()],
request=request,
)
assert _event_types(events) == ["message"]
def test_get_sync_includes_gate_message_on_private_transport(client, monkeypatch):
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
resp = client.get("/api/mesh/infonet/sync")
data = resp.json()
event_types = [e["event_type"] for e in data["events"]]
assert "gate_message" not in event_types
assert "message" in event_types
assert "vote" in event_types
assert "key_rotate" in event_types
data = client.get("/api/mesh/infonet/sync").json()
assert "gate_message" in _event_types(data["events"])
assert data["count"] == 4
def test_get_sync_count_excludes_gate_message(client, monkeypatch):
"""GET sync count field must reflect filtered events (gate_message excluded)."""
def test_post_sync_includes_gate_message_on_private_transport(monkeypatch):
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
resp = client.get("/api/mesh/infonet/sync")
data = resp.json()
assert data["count"] == 3 # message, vote, key_rotate — not gate_message
# ── POST sync excludes gate_message (main app) ─────────────────────────
def test_post_sync_excludes_gate_message(monkeypatch):
"""POST /api/mesh/infonet/sync must not return gate_message events."""
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
result = asyncio.run(
main.infonet_sync_post(
_json_request("/api/mesh/infonet/sync", {"locator": ["head-1"]})
)
)
event_types = [e["event_type"] for e in result["events"]]
assert "gate_message" not in event_types
assert "message" in event_types
assert "vote" in event_types
assert "key_rotate" in event_types
assert "gate_message" in _event_types(result["events"])
assert result["count"] == 4
def test_post_sync_count_excludes_gate_message(monkeypatch):
"""POST sync count field must reflect filtered events."""
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
result = asyncio.run(
main.infonet_sync_post(
_json_request("/api/mesh/infonet/sync", {"locator": ["head-1"]})
)
)
assert result["count"] == 3
# ── Router-served paths ────────────────────────────────────────────────
def test_router_get_sync_excludes_gate_message(monkeypatch):
"""Router GET /api/mesh/infonet/sync must not return gate_message."""
def test_router_get_sync_includes_gate_message_on_private_transport(monkeypatch):
from routers.mesh_public import infonet_sync
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
result = asyncio.run(infonet_sync(_get_request("/api/mesh/infonet/sync")))
event_types = [e["event_type"] for e in result["events"]]
assert "gate_message" not in event_types
assert "message" in event_types
assert data_count_matches(result)
assert "gate_message" in _event_types(result["events"])
assert result["count"] == len(result["events"])
def test_router_post_sync_excludes_gate_message(monkeypatch):
"""Router POST /api/mesh/infonet/sync must not return gate_message."""
def test_router_post_sync_includes_gate_message_on_private_transport(monkeypatch):
from routers.mesh_public import infonet_sync_post
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
result = asyncio.run(
infonet_sync_post(
_json_request("/api/mesh/infonet/sync", {"locator": ["head-1"]})
)
)
event_types = [e["event_type"] for e in result["events"]]
assert "gate_message" not in event_types
assert "message" in event_types
assert data_count_matches(result)
assert "gate_message" in _event_types(result["events"])
assert result["count"] == len(result["events"])
def data_count_matches(result: dict) -> bool:
return result["count"] == len(result["events"])
def test_get_sync_excludes_gate_message_when_clearnet_sync_allowed(client, monkeypatch):
_force_clearnet_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
data = client.get("/api/mesh/infonet/sync").json()
assert "gate_message" not in _event_types(data["events"])
assert data["count"] == 3
# ── Non-gate redactions still hold ─────────────────────────────────────
def test_post_sync_excludes_gate_message_when_clearnet_sync_allowed(monkeypatch):
_force_clearnet_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
result = asyncio.run(
main.infonet_sync_post(
_json_request("/api/mesh/infonet/sync", {"locator": ["head-1"]})
)
)
assert "gate_message" not in _event_types(result["events"])
assert result["count"] == 3
def test_get_sync_still_redacts_vote_gate_label(client, monkeypatch):
"""Public sync must still strip gate label from vote payload."""
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
resp = client.get("/api/mesh/infonet/sync")
events = resp.json()["events"]
events = client.get("/api/mesh/infonet/sync").json()["events"]
vote = next(e for e in events if e["event_type"] == "vote")
assert "gate" not in vote.get("payload", {})
def test_get_sync_still_redacts_key_rotate_identity(client, monkeypatch):
"""Public sync must still strip old identity fields from key_rotate payload."""
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
resp = client.get("/api/mesh/infonet/sync")
events = resp.json()["events"]
events = client.get("/api/mesh/infonet/sync").json()["events"]
rotate = next(e for e in events if e["event_type"] == "key_rotate")
payload = rotate.get("payload", {})
assert "old_node_id" not in payload
assert "old_public_key" not in payload
assert "old_signature" not in payload
def test_post_sync_still_redacts_vote_and_rotate(monkeypatch):
"""POST sync must still apply standard public redactions to non-gate events."""
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _FakeInfonet(), raising=False)
result = asyncio.run(
main.infonet_sync_post(
_json_request("/api/mesh/infonet/sync", {"locator": ["head-1"]})
@@ -291,24 +351,17 @@ def test_post_sync_still_redacts_vote_and_rotate(monkeypatch):
)
vote = next(e for e in result["events"] if e["event_type"] == "vote")
rotate = next(e for e in result["events"] if e["event_type"] == "key_rotate")
assert "gate" not in vote.get("payload", {})
assert "old_node_id" not in rotate.get("payload", {})
# ── No overclaim ───────────────────────────────────────────────────────
def test_gate_message_still_in_fake_infonet_storage():
"""The filter does NOT remove gate_message from underlying storage.
This test documents that the infonet still holds gate_message events;
only the public sync response surface filters them out."""
fake = _FakeInfonet()
all_types = [e["event_type"] for e in fake.events]
assert "gate_message" in all_types
assert "gate_message" in _event_types(fake.events)
def test_sync_with_only_gate_messages_returns_empty(client, monkeypatch):
"""If infonet contains only gate_message events, sync returns empty list."""
def test_private_sync_with_only_gate_messages_returns_gate_events(client, monkeypatch):
class _GateOnlyInfonet:
head_hash = "head-1"
events = [_gate_message_event()]
@@ -325,8 +378,10 @@ def test_sync_with_only_gate_messages_returns_empty(client, monkeypatch):
def get_merkle_root(self):
return "r"
_force_private_sync(monkeypatch)
monkeypatch.setattr(mesh_hashchain, "infonet", _GateOnlyInfonet(), raising=False)
resp = client.get("/api/mesh/infonet/sync")
data = resp.json()
assert data["events"] == []
assert data["count"] == 0
data = client.get("/api/mesh/infonet/sync").json()
assert _event_types(data["events"]) == ["gate_message"]
assert data["count"] == 1
@@ -66,6 +66,20 @@ def _make_gate_message_event(priv, pub_b64, node_id, sequence, prev_hash, gate_i
return evt.to_dict()
def _make_gate_payload(gate_id="test-gate") -> dict:
return mesh_protocol.normalize_payload(
"gate_message",
{
"gate": gate_id,
"ciphertext": base64.b64encode(b"encrypted-data").decode(),
"nonce": base64.b64encode(b"nonce-value-1234").decode(),
"sender_ref": "sender-abc",
"format": "mls1",
"transport_lock": "private_strong",
},
)
@pytest.fixture()
def fresh_env(tmp_path, monkeypatch):
"""Set up isolated infonet + gate_store, return (infonet, gate_store)."""
@@ -89,6 +103,74 @@ def fresh_env(tmp_path, monkeypatch):
# ── Rejected gate_message must NOT hydrate gate_store ─────────────────────
def test_append_private_gate_message_uses_hashchain_gate_sequence(fresh_env):
"""Local gate posts become private hashchain events in a gate sequence domain."""
inf, _gs = fresh_env
priv, pub_b64, node_id = _make_keypair()
sequence = 1
payload = _make_gate_payload("test-gate")
sig_payload = mesh_crypto.build_signature_payload(
event_type="gate_message",
node_id=node_id,
sequence=sequence,
payload=payload,
)
signature = priv.sign(sig_payload.encode("utf-8")).hex()
event = inf.append_private_gate_message(
node_id=node_id,
payload=payload,
signature=signature,
sequence=sequence,
public_key=pub_b64,
public_key_algo="Ed25519",
protocol_version=mesh_protocol.PROTOCOL_VERSION,
timestamp=123.0,
)
assert event["event_type"] == "gate_message"
assert inf.head_hash == event["event_id"]
assert inf.sequence_domains[f"{node_id}|gate_message"] == sequence
assert inf.node_sequences.get(node_id, 0) == 0
assert event["payload"]["transport_lock"] == "private_strong"
def test_ingest_accepts_new_suffix_after_duplicate_prefix(fresh_env):
"""Peer-push batches may include events the receiver already has."""
inf, _gs = fresh_env
priv, pub_b64, node_id = _make_keypair()
evt1 = _make_gate_message_event(
priv,
pub_b64,
node_id,
sequence=1,
prev_hash=mesh_hashchain.GENESIS_HASH,
)
assert inf.ingest_events([evt1])["accepted"] == 1
evt2 = _make_gate_message_event(
priv,
pub_b64,
node_id,
sequence=2,
prev_hash=evt1["event_id"],
)
assert inf.ingest_events([evt2])["accepted"] == 1
evt3 = _make_gate_message_event(
priv,
pub_b64,
node_id,
sequence=3,
prev_hash=evt2["event_id"],
)
result = inf.ingest_events([evt1, evt2, evt3])
assert result["duplicates"] == 2
assert result["accepted"] == 1
assert result["rejected"] == []
assert inf.head_hash == evt3["event_id"]
def test_rejected_event_does_not_hydrate_gate_store(fresh_env):
"""A gate_message rejected by ingest must not appear in gate_store."""
inf, gs = fresh_env
@@ -238,6 +238,10 @@ class TestNoMonsterUserAgentRemains:
"ShadowBroker-FeedIngester/1.0",
"ShadowBroker/0.9.79 local Shodan connector",
"ShadowBroker/0.9.79 Finnhub connector",
"ShadowBroker/0.9.8 local Shodan connector",
"ShadowBroker/0.9.8 Finnhub connector",
"ShadowBroker/0.9.81 local Shodan connector",
"ShadowBroker/0.9.81 Finnhub connector",
"Mozilla/5.0 (compatible; ShadowBroker CCTV proxy)",
)
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@shadowbroker/desktop-shell",
"version": "0.9.79",
"version": "0.9.81",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@shadowbroker/desktop-shell",
"version": "0.9.79",
"version": "0.9.81",
"devDependencies": {
"typescript": "^5.6.0"
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@shadowbroker/desktop-shell",
"version": "0.9.79",
"version": "0.9.81",
"private": true,
"description": "ShadowBroker desktop shell packaging, runtime bridge, and release tooling",
"scripts": {
@@ -130,6 +130,45 @@ function stageBackendRuntime() {
});
stagePrivacyCoreArtifact();
stageReleaseAttestation();
stageStartScripts();
}
/**
* Copy ``start.bat`` and ``start.sh`` from the repo root into the
* staged backend-runtime/ so they sit next to ``privacy_core.dll``.
*
* Why: an MSI/EXE/AppImage user who wants to launch via the dev-style
* scripts (because the desktop shell is failing, or they prefer the
* browser frontend at localhost:3000) shouldn't have to clone the
* source repo just to get the scripts. Having them inside the install
* directory also means the bundled ``privacy_core.dll`` fallback in
* those scripts resolves to the SAME directory as the script, which
* is exactly the layout the v0.9.81 script update is looking for.
*
* Tracked from issue #319: users who fell back to start.bat from
* their MSI install dir had to go fetch it from GitHub, then saw a
* scary "install Rust" warning because the script didn't know where
* the bundled DLL was. Bundling the script removes both problems.
*/
function stageStartScripts() {
const scripts = ['start.bat', 'start.sh'];
for (const name of scripts) {
const src = path.join(repoRoot, name);
if (!fs.existsSync(src)) {
console.warn(`backend-runtime staged without ${name} (not at repo root)`);
continue;
}
const dst = path.join(outputDir, name);
fs.copyFileSync(src, dst);
// Preserve executable bit on POSIX systems for the .sh script.
if (name.endsWith('.sh') && process.platform !== 'win32') {
try {
fs.chmodSync(dst, 0o755);
} catch {
/* best-effort; not fatal on filesystems that don't honor chmod */
}
}
}
}
function stagePrivacyCoreArtifact() {
+1 -1
View File
@@ -4201,7 +4201,7 @@ dependencies = [
[[package]]
name = "shadowbroker-tauri-shell"
version = "0.9.79"
version = "0.9.81"
dependencies = [
"axum",
"base64 0.22.1",
@@ -1,6 +1,6 @@
[package]
name = "shadowbroker-tauri-shell"
version = "0.9.79"
version = "0.9.81"
edition = "2021"
[build-dependencies]
@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "ShadowBroker",
"version": "0.9.79",
"version": "0.9.81",
"identifier": "com.shadowbroker.desktop",
"build": {
"frontendDist": "../../../frontend/out",
@@ -38,7 +38,7 @@
},
"plugins": {
"updater": {
"pubkey": "dW50cnVzdGVkIGNvbW1lbnQ6IG1pbmlzaWduIHB1YmxpYyBrZXk6IEUxODExMjQ4MkJBMThFNTgKUldSWWpxRXJTQktCNFF3ZXNQbndUK0pVWUEwNDNuajcrUGI3ZEI4TWtDUDlQdHhudmlHUkNjQUUK",
"pubkey": "dW50cnVzdGVkIGNvbW1lbnQ6IG1pbmlzaWduIHB1YmxpYyBrZXk6IDVEMTFERDdCNjhBRTk3MDcKUldRSGw2NW9lOTBSWGRjS1ZobFN5TkZsd3NkZ2g2L09WZzU4aytTR2FtN3ZtR0ZKejlNNldTbFUK",
"endpoints": [
"https://github.com/BigBodyCobain/Shadowbroker/releases/latest/download/latest.json"
],
+1 -1
View File
@@ -6,7 +6,7 @@ services:
dockerfile: ./backend/Dockerfile
container_name: shadowbroker-relay
ports:
- "0.0.0.0:8000:8000"
- "127.0.0.1:8000:8000"
env_file: .env
volumes:
- relay_data:/app/data
+8
View File
@@ -24,8 +24,16 @@ services:
# Private Infonet bootstrap seeds. Seeds are discovery hints, not fixed roots.
- MESH_BOOTSTRAP_SEED_PEERS=${MESH_BOOTSTRAP_SEED_PEERS:-http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000}
- MESH_DEFAULT_SYNC_PEERS=${MESH_DEFAULT_SYNC_PEERS:-}
- MESH_SYNC_TIMEOUT_S=${MESH_SYNC_TIMEOUT_S:-5}
- MESH_RELAY_PUSH_TIMEOUT_S=${MESH_RELAY_PUSH_TIMEOUT_S:-45}
# Explicitly opt into HTTPS/IP-based peer sync. Default remains private transports only.
- MESH_INFONET_ALLOW_CLEARNET_SYNC=${MESH_INFONET_ALLOW_CLEARNET_SYNC:-false}
# Tor/Arti SOCKS transport for private .onion Infonet sync.
- MESH_ARTI_ENABLED=${MESH_ARTI_ENABLED:-false}
- MESH_ARTI_SOCKS_PORT=${MESH_ARTI_SOCKS_PORT:-9050}
# Operator-trusted sync/push peers. Leave empty unless you control the peer secret on both sides.
- MESH_RELAY_PEERS=${MESH_RELAY_PEERS:-}
- MESH_PUBLIC_PEER_URL=${MESH_PUBLIC_PEER_URL:-}
# Shared transport auth for operator peer push. Must be set to a unique secret per deployment.
- MESH_PEER_PUSH_SECRET=${MESH_PEER_PUSH_SECRET:-}
# Issue #256: optional per-peer HMAC secrets. Comma-separated
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "frontend",
"version": "0.9.79",
"version": "0.9.81",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "frontend",
"version": "0.9.79",
"version": "0.9.81",
"dependencies": {
"@mapbox/point-geometry": "^1.1.0",
"@tauri-apps/plugin-process": "^2.3.1",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "frontend",
"version": "0.9.79",
"version": "0.9.81",
"private": true,
"scripts": {
"dev": "node scripts/dev-all.cjs",
@@ -9,12 +9,12 @@ import {
} from '@/lib/updateRuntime';
const RELEASE: GitHubLatestRelease = {
html_url: 'https://github.com/BigBodyCobain/Shadowbroker/releases/tag/v0.9.79',
html_url: 'https://github.com/BigBodyCobain/Shadowbroker/releases/tag/v0.9.81',
assets: [
{ name: 'ShadowBroker_0.9.79_x64_en-US.msi', browser_download_url: 'https://example.test/windows.msi' },
{ name: 'ShadowBroker_0.9.79_x64-setup.exe', browser_download_url: 'https://example.test/windows-setup.exe' },
{ name: 'ShadowBroker_0.9.79_aarch64.dmg', browser_download_url: 'https://example.test/macos.dmg' },
{ name: 'ShadowBroker_0.9.79_amd64.AppImage', browser_download_url: 'https://example.test/linux.AppImage' },
{ name: 'ShadowBroker_0.9.81_x64_en-US.msi', browser_download_url: 'https://example.test/windows.msi' },
{ name: 'ShadowBroker_0.9.81_x64-setup.exe', browser_download_url: 'https://example.test/windows-setup.exe' },
{ name: 'ShadowBroker_0.9.81_aarch64.dmg', browser_download_url: 'https://example.test/macos.dmg' },
{ name: 'ShadowBroker_0.9.81_amd64.AppImage', browser_download_url: 'https://example.test/linux.AppImage' },
],
};
+48 -95
View File
@@ -20,129 +20,82 @@ import {
Heart,
} from 'lucide-react';
const CURRENT_VERSION = '0.9.79';
const CURRENT_VERSION = '0.9.81';
const STORAGE_KEY = `shadowbroker_changelog_v${CURRENT_VERSION}`;
const RELEASE_TITLE = 'Onboarding, Live Feeds, Mesh, and Agent Hardening';
const RELEASE_TITLE = 'Signed Auto-Update + Update Button Race Fix';
const HEADLINE_FEATURES = [
{
icon: <Bot size={20} className="text-purple-400" />,
icon: <KeyRound size={20} className="text-purple-400" />,
accent: 'purple' as const,
title: 'Agentic onboarding for OpenClaw-compatible agents',
subtitle: 'First-time setup now includes local/direct agent connection, access-tier selection, copyable HMAC setup, and optional Tor hidden-service prep.',
title: 'Signed Auto-Update Going Forward (one manual hop)',
subtitle: 'After installing v0.9.81, the in-app Update button finally works end-to-end. This release establishes a fresh signing key — every release from here is a one-click upgrade.',
details: [
'The onboarding flow can generate the local agent connection bundle through the existing HMAC API, point agents at /api/ai/tools, and let operators choose restricted read-only or full write access before connecting an agent.',
'Remote mode is labeled honestly: .onion exposes the signed HTTP agent API over Tor. Wormhole/MLS is not claimed as the current agent command transport.',
'The setup copy works for OpenClaw, Hermes, or any custom agent that implements the documented HMAC request contract.',
'tauri.conf.json now carries a fresh minisign pubkey (the previous keypair was generated before v0.9.79 shipped but the matching private key was lost before any release was actually signed, so no release before v0.9.81 has working auto-update).',
'The v0.9.81 release artifacts ship with a signed latest.json + .sig files so every install on v0.9.81 or later can verify and apply the next release automatically via the Tauri updater plugin.',
'One-time cost: if you are upgrading from v0.9.79 or v0.9.8, the click-Update path falls back to a manual download because the new pubkey does not match the one baked into your install. Click the MANUAL DOWNLOAD button in the update dialog → grab the .msi from the release page → run it → from then on auto-update works in-app.',
],
callToAction: 'OPEN FIRST-TIME SETUP -> AI AGENT',
callToAction: 'CLICK UPDATE → DOWNLOAD MSI ONCE → AUTO-UPDATE FOREVER',
},
{
icon: <Bot size={20} className="text-purple-400" />,
accent: 'purple' as const,
title: 'Agentic AI Channel — supports OpenClaw and any HMAC-signing agent',
subtitle: 'ShadowBroker now exposes a signed agent command channel. Bring your own agent (OpenClaw, Claude Code, GPT, LangChain, or a custom client) and drive the dashboard from any LLM that speaks the protocol.',
details: [
'A signed command channel (POST /api/ai/channel/command) plus a batched concurrent-execution endpoint (up to 20 tool calls per round-trip via /api/ai/channel/batch). Agents query flights, ships, SIGINT, news, and intel layers; reason over the live mesh; and run market or threat analyses without a human in the loop.',
'HMAC-SHA256 request signing with timestamp + nonce replay protection. Tier-gated access (restricted vs full) governs which read and write commands the agent can invoke. Every call is auditable through the channel log.',
'ShadowBroker does not bundle an LLM, an agent runtime, or model weights — it ships the protocol. Any agent that signs requests with the documented HMAC contract can connect. OpenClaw is the reference implementation.',
],
callToAction: 'CONNECT YOUR AGENT \u2192 /API/AI/CHANNEL/COMMAND',
},
{
icon: <Network size={20} className="text-cyan-400" />,
icon: <Network size={20} className="text-amber-400" />,
accent: 'cyan' as const,
title: 'InfoNet Testnet \u2014 Framework, Privacy, and a Path to Decentralized Intelligence',
subtitle: 'The testnet now ships its full governance economy and the runway for a privacy-preserving decentralized intelligence platform.',
title: 'AIS Maritime Resilience — Outage Banner + AISHub Fallback',
subtitle: 'When AISStreams WebSocket goes offline (as happened upstream in May 2026), the ships layer no longer goes silently empty.',
details: [
'Sovereign Shell views: petitions (governance DSL covers parameter updates and feature toggles), upgrade-hash voting (80% supermajority, 67% Heavy-Node activation), evidence submission, dispute markets, gate suspension and shutdown, and bootstrap eligible-node-one-vote. Every write action is a clickable form with verbatim diagnostics on rejection.',
'Privacy primitive runway: locked Protocol contracts for ring signatures, stealth addresses, shielded balances, and DEX matching. The privacy-core Rust crate is the integration target. Function Keys (anonymous citizenship proof) ship 5 of 6 pieces; only blind-signature issuance waits on a primitive decision.',
'Backbone: two-tier event state with epoch finality, identity rotation, progressive penalties, ramp milestones, and constitutional invariants enforced via MappingProxyType. Sprint 11+ wires the cryptographic primitives into the locked Protocols.',
'Still an experimental testnet \u2014 no privacy guarantee yet. Treat all channels as public until E2E and the privacy primitives ship.',
'AIS proxy health surfaces in /api/health: connected, last_msg_age_seconds, proxy_spawn_count. A dismissible amber banner explains the outage (“Ship data temporarily unavailable — AISStream upstream is offline”) instead of letting users assume their install is broken.',
'AISHub REST fallback (free tier at aishub.net/api). Polls every 20 minutes when the primary is disconnected and merges vessels into the same store with source: “aishub” so existing tooling attributes the provider.',
'Live data wins races: if the WebSocket reconnects mid-poll, fresh AISStream updates arent overwritten by stale REST records. Opt-in via AISHUB_USERNAME; cadence configurable via AISHUB_POLL_INTERVAL_MINUTES (clamped [1, 360]).',
],
callToAction: 'OPEN SOVEREIGN SHELL \u2192 PETITIONS \u2022 UPGRADES \u2022 GATES',
callToAction: 'SET AISHUB_USERNAME \u2192 RESTART BACKEND',
},
{
icon: <Shield size={20} className="text-cyan-400" />,
accent: 'cyan' as const,
title: 'Data-Layer Repair \u2014 UAP Cutoff + GPS Jamming Detection',
subtitle: 'Two long-broken layers fixed at the source. UFO sightings are actually recent now; GPS jamming zones actually fire.',
details: [
'UAP sightings: the Hugging Face NUFORC mirror fallback had no date cutoff, so when the live nuforc.org scrape failed the layer served 3-year-old reports as \u201crecent\u201d. Now drops rows older than 60 days and logs loudly when the mirror is fully stale. Scheduler moved daily \u2192 weekly (Mondays 12:00 UTC).',
'GPS jamming: three stacked filters meant the layer almost never lit up. nac_p == 0 (\u201cGPS lock lost\u201d) was filtered out as if it were an old transponder \u2014 it\u2019s actually the strongest jamming signal. Now counted. MIN_AIRCRAFT lowered 5 \u2192 3 so sparser hotspots clear; MIN_RATIO lowered 0.30 \u2192 0.20.',
'Both layers now surface their own outages via assert_canary so operators see broken vs empty, not silently stale.',
],
callToAction: 'TOGGLE UAP \u2022 GPS JAMMING LAYERS',
},
];
const NEW_FEATURES = [
{
icon: <Clock size={18} className="text-cyan-400" />,
title: 'Startup and Feed Responsiveness Pass',
desc: 'Map-critical feeds now lean on startup caches and priority preload behavior so the dashboard can paint before heavyweight synthesis jobs finish.',
},
{
icon: <Network size={18} className="text-green-400" />,
title: 'MeshChat MQTT Settings',
desc: 'Public MeshChat stays opt-in and now has an in-panel settings lane for broker, port, username, password, and channel PSK while remaining separated from Wormhole/private mode.',
icon: <Plane size={18} className="text-orange-400" />,
title: 'Cumulative Fuel & CO2 per Flight',
desc: 'Aircraft tooltip now shows how much fuel each plane has actually burned in the air since first observation, not just the per-hour rate. 15-minute gap between sightings resets the session; 24-hour clamp protects against clock skew; per-icao prune every 5 minutes keeps memory bounded.',
},
{
icon: <Plane size={18} className="text-cyan-400" />,
title: 'Selected Entity Trails',
desc: 'Flight and vessel trails are drawn only for selected assets, reducing global clutter while still exposing movement history for unknown-route entities.',
title: 'Per-Flight Source Attribution',
desc: 'Every aircraft record now carries a source field (adsb.lol, OpenSky, airplanes.live, adsb.fi) so consumers can attribute the data provider. Pre-fix, adsb.lol records were unmarked while OpenSky records were explicitly tagged, making it look like adsb.lol was unused even though it is the primary source.',
},
{
icon: <Plane size={18} className="text-amber-400" />,
title: 'Aircraft Detail Cards',
desc: 'Commercial aircraft stay airline-first, while private and general aviation aircraft can show model-focused Wiki context and imagery when available.',
icon: <Network size={18} className="text-green-400" />,
title: 'Cross-Node DM Mailbox Replication',
desc: 'Direct messages now replicate across mesh nodes when one party is offline. Per-(sender, recipient) anti-spam cap enforced as a network rule (not client-side) so source-code tampering cannot bypass it.',
},
{
icon: <Cpu size={18} className="text-purple-400" />,
title: 'AI Batch Command Channel',
desc: 'POST up to 20 tool calls in a single HTTP round-trip; the backend executes them concurrently and returns a fan-out result map. Cuts agent latency by an order of magnitude over sequential calls.',
},
{
icon: <Scale size={18} className="text-amber-400" />,
title: 'Governance DSL — Petition-Driven Parameter Changes',
desc: 'Type-safe payload executor for UPDATE_PARAM, BATCH_UPDATE_PARAMS, ENABLE_FEATURE, and DISABLE_FEATURE petitions. Tunable knobs change on-chain via a vote — no code deploys required.',
},
{
icon: <GitBranch size={18} className="text-purple-400" />,
title: 'Upgrade-Hash Governance',
desc: 'Protocol upgrades that need new logic (not just parameter changes) vote on a SHA-256 hash of the verified release. 80% supermajority, 40% quorum, 67% Heavy-Node activation. Lifecycle: signatures, voting, challenge window, awaiting readiness, activated.',
},
{
icon: <KeyRound size={18} className="text-purple-400" />,
title: 'Function Keys — Anonymous Citizenship Proof',
desc: 'A citizen proves "I am an Infonet citizen" without revealing their Infonet identity. 5 of 6 pieces shipped: nullifiers, challenge-response, two-phase commit receipts, enumerated denial codes, batched settlement. Issuance via blind signatures waits on a primitive decision.',
},
{
icon: <Shield size={18} className="text-cyan-400" />,
title: 'Privacy Primitive Runway',
desc: 'Locked Protocol contracts in services/infonet/privacy/contracts.py for ring signatures, stealth addresses, Pedersen commitments, range proofs, and DEX matching. The privacy-core Rust crate is the integration target — no caller of the privacy module needs to know which scheme is active.',
},
{
icon: <Layers size={18} className="text-blue-400" />,
title: 'Two-Tier State + Epoch Finality',
desc: 'Tier 1 events propagate CRDT-style for low latency; Tier 2 events require epoch finality before they can be acted on. Identity rotation, progressive penalties, ramp milestones, and constitutional invariants are enforced via MappingProxyType.',
},
{
icon: <Terminal size={18} className="text-cyan-400" />,
title: 'Sovereign Shell Write Surface',
desc: 'PetitionsView, UpgradeView, ResolutionView, GateShutdownView, BootstrapView, and FunctionKeyView each expose every Sprint 4-8 + 10 write action as a clickable form. Adaptive polling tightens to 8 seconds during active voting/challenge phases.',
},
{
icon: <Clock size={18} className="text-pink-400" />,
title: 'Time Machine — Snapshot Playback',
desc: 'Scrub backward through saved telemetry. Live polling pauses on entry to snapshot mode, the map redraws from the recorded snapshot, and moving entities interpolate between recorded frames. Hourly index lets you jump to any captured timestamp; pressing Live restores the current feed instantly.',
},
{
icon: <Satellite size={18} className="text-orange-400" />,
title: 'SAR Satellite Telemetry — ASF, OPERA, Copernicus',
desc: 'New SAR (Synthetic Aperture Radar) layer. Mode A (default-on) pulls free catalog metadata from the Alaska Satellite Facility — no account required. Mode B (two-step opt-in) ingests pre-processed ground-change anomalies from NASA OPERA, Copernicus EGMS, GFM, EMS, and UNOSAT — deformation, flood, and damage assessments. Integrates with OpenClaw so agents can read and act on SAR anomalies; broadcasts default to private-tier transport (Tor / RNS).',
icon: <Clock size={18} className="text-amber-400" />,
title: 'Infonet Sync — HTTP 429 Honored',
desc: 'When an upstream peer returns Retry-After, the node now waits exactly that long instead of retrying every 60 seconds and keeping the upstream rate-limit bucket permanently full. Exponential backoff on consecutive failures capped at 30 minutes.',
},
];
const BUG_FIXES = [
'Docker proxy and backend port handling hardened so changing the host backend port does not require changing the internal service contract.',
'Global Threat Intercept and live-data startup paths no longer wait on slow-tier synthesis before cached data can paint the UI.',
'MeshChat and Infonet statuses now separate public MQTT participation, private Wormhole mode, and local node bootstrap so the UI does not imply the wrong connection state.',
'Commercial aircraft detail cards no longer show a confusing model image alongside the airline card.',
'Sovereign Shell adaptive polling — voting and challenge windows refresh every 8 seconds while active, every 30 to 60 seconds when idle. Voting feels live without a websocket layer.',
'Per-row write actions (petitions, upgrades, disputes) hold isolated submission state so concurrent forms no longer share a single in-flight slot.',
'Verbatim diagnostic surfacing on every write button. The backend reason text is always shown on rejection no opaque "denied" toasts.',
'Evidence submission canonicalization matches Python repr() exactly, so client-side SHA-256 hashes round-trip cleanly through the chain.',
'Function Keys copy is context-agnostic — citizenship proof is described abstractly, not tied to a specific use case.',
'Post-cutover legacy mesh files (mesh_schema.py, mesh_signed_events.py, mesh_hashchain.py) hash-verified against the recorded baseline; the chain extension hook stays surgical.',
'Update button no longer throws "admin_session_required" on desktop installs. The initial updateAction now syncs to Tauri detection at React-init time (window.__TAURI__ is injected before mount), so a click before the async runtime probe completes opens the GitHub release page in a browser instead of POSTing to /api/system/update.',
'Desktop installer now bundles defusedxml + PySocks (declared in pyproject.toml but missing from the venv shipped with v0.9.79 and the initial v0.9.8 publish). Fixes the bundled-backend launch crash reported in #319 and #296 (managed_backend_exited_early:exit code: 103).',
'UAP layer no longer serves 3-year-old NUFORC sightings via the Hugging Face static-mirror fallback (60-day cutoff now applied to the fallback path too).',
'GPS jamming detection now counts nac_p == 0 (the actual GPS-lost signal) instead of filtering it out as an old-transponder artifact.',
'GPS jamming thresholds lowered (MIN_AIRCRAFT 5 → 3, MIN_RATIO 0.30 → 0.20) so sparser hotspots clear the bar without losing the 1-aircraft noise cushion.',
'AIS layer surfaces an outage banner when the AISStream WebSocket upstream is offline, instead of silently showing an empty ocean.',
'Flight emissions tooltip now shows cumulative fuel/CO2 since first observation, not just the per-hour rate.',
'Per-aircraft observation tracker (15-min reopen gap, 24-hour clamp) survives trail-rendering cache pruning so cumulative counters do not reset mid-flight.',
'UAP scheduler moved daily → weekly (Mondays 12:00 UTC) to match the layers rolling-window cadence and reduce upstream load.',
];
const CONTRIBUTORS = [
@@ -298,6 +298,33 @@ export default function InfonetShell({
setCurrentView(view);
};
const renderGateDirectory = (variant: 'landing' | 'command' = 'command') => (
<div
className={
variant === 'landing'
? 'w-full max-w-3xl border border-cyan-950/50 bg-black/20 px-4 py-3 text-left shadow-[0_0_18px_rgba(6,182,212,0.06)]'
: 'text-gray-400'
}
>
<p className={`${variant === 'landing' ? 'text-[11px]' : ''} text-gray-400 uppercase tracking-[0.18em]`}>
AVAILABLE OBFUSCATED GATES:
</p>
<div className={`grid grid-cols-1 sm:grid-cols-2 xl:grid-cols-3 ${variant === 'landing' ? 'gap-x-8 gap-y-1.5 mt-2' : 'gap-2 mt-2'}`}>
{GATES.map(gate => (
<button
key={gate}
type="button"
className="group flex min-h-[24px] items-center text-left text-gray-300 hover:text-white transition-colors"
onClick={() => handleNavigate('gate', gate)}
>
<span className="text-gray-500 mr-2 group-hover:text-cyan-400 transition-colors">[{'>'}]</span>
<span className="truncate group-hover:drop-shadow-[0_0_5px_rgba(6,182,212,0.8)]">{gate}</span>
</button>
))}
</div>
</div>
);
const openGateWhenReady = async (
gateTarget: string,
operation: () => Promise<void>,
@@ -471,19 +498,7 @@ export default function InfonetShell({
setHistory([]);
return;
} else if (trimmedCmd === 'gates') {
output = (
<div className="text-gray-400">
<p>AVAILABLE OBFUSCATED GATES:</p>
<div className="grid grid-cols-2 md:grid-cols-3 gap-2 mt-2">
{GATES.map(gate => (
<div key={gate} className="flex items-center cursor-pointer hover:text-gray-300 group" onClick={() => handleNavigate('gate', gate)}>
<span className="text-gray-500 mr-2 group-hover:text-cyan-400 transition-colors">[{'>'}]</span>
<span className="text-gray-300 group-hover:text-white transition-colors group-hover:drop-shadow-[0_0_5px_rgba(6,182,212,0.8)]">{gate}</span>
</div>
))}
</div>
</div>
);
output = renderGateDirectory('command');
} else if (trimmedCmd.startsWith('join ') || trimmedCmd.startsWith('g/')) {
const target = trimmedCmd.startsWith('g/') ? trimmedCmd.slice(2) : trimmedCmd.split(' ')[1];
if (GATES.includes(target)) {
@@ -661,6 +676,9 @@ export default function InfonetShell({
<p>Type <span className="text-green-400 font-bold">&apos;gates&apos;</span> or <span className="text-green-400 font-bold">g/</span> to view available chatrooms.</p>
</div>
<NetworkStats />
<div className="mt-5 w-full flex justify-center">
{renderGateDirectory('landing')}
</div>
</div>
<HashchainEvents />
+1 -1
View File
@@ -4,7 +4,7 @@ import React, { useState, useEffect } from 'react';
import { motion, AnimatePresence } from 'framer-motion';
import { X, ExternalLink, Key, Shield, Radar, Globe, Satellite, Ship, Radio, Bot, Copy, Check, Network } from 'lucide-react';
const CURRENT_ONBOARDING_VERSION = '0.9.79-agentic-onboarding-1';
const CURRENT_ONBOARDING_VERSION = '0.9.81-agentic-onboarding-1';
const STORAGE_KEY = `shadowbroker_onboarding_complete_v${CURRENT_ONBOARDING_VERSION}`;
const LEGACY_STORAGE_KEY = 'shadowbroker_onboarding_complete';
@@ -4,7 +4,7 @@ import React, { useEffect, useState } from 'react';
import { motion, AnimatePresence } from 'framer-motion';
import { Database, Clock, X } from 'lucide-react';
const CURRENT_VERSION = '0.9.79';
const CURRENT_VERSION = '0.9.81';
const STORAGE_KEY = `shadowbroker_startup_warmup_notice_v${CURRENT_VERSION}`;
interface StartupWarmupModalProps {
+13 -1
View File
@@ -91,7 +91,19 @@ export default function TopRightControls({
const [manualUpdateUrl, setManualUpdateUrl] = useState(DEFAULT_RELEASES_URL);
const [releasePageUrl, setReleasePageUrl] = useState(DEFAULT_RELEASES_URL);
const [dockerCommands, setDockerCommands] = useState('');
const [updateAction, setUpdateAction] = useState<UpdateActionKind>('auto_apply');
// Pre-detection initial value: the right action depends on the runtime.
// For desktop installs (Tauri webview), the default should be
// ``manual_download`` so that clicking Update before the async runtime
// probe completes opens the release page in a browser instead of POSTing
// to /api/system/update — which throws ``admin_session_required`` on
// fresh sessions and confused v0.9.79/v0.9.8 users with a cryptic error.
// ``window.__TAURI__`` is injected synchronously by Tauri before React
// mounts, so this check is safe to do at useState init time.
const initialUpdateAction: UpdateActionKind =
typeof window !== 'undefined' && (window as { __TAURI__?: unknown }).__TAURI__
? 'manual_download'
: 'auto_apply';
const [updateAction, setUpdateAction] = useState<UpdateActionKind>(initialUpdateAction);
const [updateDetail, setUpdateDetail] = useState(AUTO_UPDATE_DETAIL);
const pollRef = useRef<ReturnType<typeof setInterval> | null>(null);
const timeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);
+2 -2
View File
@@ -1,8 +1,8 @@
---
apiVersion: v2
name: shadowbroker
version: 0.9.79
appVersion: "0.9.79"
version: 0.9.81
appVersion: "0.9.81"
description: simple shadowbroker installation
type: application
+11 -3
View File
@@ -8,6 +8,7 @@ echo ===================================================
echo.
echo Lightweight node — syncs the Infonet chain only.
echo No map, no frontend, no data feeds.
echo Private hashchain relay: gate messages + offline DM spool.
echo Close this window to stop the node.
echo.
@@ -96,15 +97,22 @@ echo [*] Auto-enabling node participation...
if not exist "data\" mkdir data
echo {"enabled":true,"updated_at":0} > data\node.json
set MESH_ONLY=true
set SHADOWBROKER_MESH_NODE_RUNTIME=true
set MESH_NODE_MODE=participant
set MESH_INFONET_ALLOW_CLEARNET_SYNC=false
set MESH_ARTI_ENABLED=true
set MESH_DM_HASHCHAIN_SPOOL_LIMIT=2
set MESH_DM_HASHCHAIN_SPOOL_TTL_S=3600
if "%MESH_BOOTSTRAP_SEED_PEERS%"=="" set MESH_BOOTSTRAP_SEED_PEERS=http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000
echo.
echo ===================================================
echo Mesh node starting on port 8000
echo Mode: MESH_ONLY (no data feeds)
echo Relay: %MESH_RELAY_PEERS%
echo Bootstrap: %MESH_BOOTSTRAP_SEED_PEERS%
echo Press Ctrl+C to stop
echo ===================================================
echo.
set MESH_ONLY=true
set MESH_NODE_MODE=participant
python -m uvicorn main:app --host 0.0.0.0 --port 8000
+11 -3
View File
@@ -10,6 +10,7 @@ echo "==================================================="
echo ""
echo " Lightweight node — syncs the Infonet chain only."
echo " No map, no frontend, no data feeds."
echo " Private hashchain relay: gate messages + offline DM spool."
echo " Press Ctrl+C to stop."
echo ""
@@ -51,15 +52,22 @@ echo "[*] Auto-enabling node participation..."
mkdir -p data
echo '{"enabled":true,"updated_at":0}' > data/node.json
export MESH_ONLY=true
export SHADOWBROKER_MESH_NODE_RUNTIME=true
export MESH_NODE_MODE=participant
export MESH_INFONET_ALLOW_CLEARNET_SYNC=false
export MESH_ARTI_ENABLED=true
export MESH_DM_HASHCHAIN_SPOOL_LIMIT=2
export MESH_DM_HASHCHAIN_SPOOL_TTL_S=3600
export MESH_BOOTSTRAP_SEED_PEERS="${MESH_BOOTSTRAP_SEED_PEERS:-http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000}"
echo ""
echo "==================================================="
echo " Mesh node starting on port 8000"
echo " Mode: MESH_ONLY (no data feeds)"
echo " Relay: ${MESH_RELAY_PEERS:-default}"
echo " Bootstrap: ${MESH_BOOTSTRAP_SEED_PEERS}"
echo " Press Ctrl+C to stop"
echo "==================================================="
echo ""
export MESH_ONLY=true
export MESH_NODE_MODE=participant
python3 -m uvicorn main:app --host 0.0.0.0 --port 8000
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "shadowbroker"
version = "0.9.79"
version = "0.9.81"
readme = "README.md"
requires-python = ">=3.10"
dependencies = []
+8
View File
@@ -237,6 +237,14 @@ echo [*] Backend Node.js dependencies OK.
echo.
echo [*] Checking privacy-core shared library...
set "PRIVACY_CORE_DLL=%ROOT%\privacy-core\target\release\privacy_core.dll"
:: MSI/EXE installers stage privacy_core.dll directly in backend-runtime/
:: alongside this script. If somebody runs start.bat from an installed
:: app directory (no source checkout, no Rust toolchain), they shouldn't
:: see a spurious "install Rust" warning because the DLL is right next
:: to them — just at a different path than the source-tree build.
if not exist "%PRIVACY_CORE_DLL%" if exist "%ROOT%\privacy_core.dll" (
set "PRIVACY_CORE_DLL=%ROOT%\privacy_core.dll"
)
if not exist "%PRIVACY_CORE_DLL%" (
where cargo >nul 2>&1
if errorlevel 1 (
+11
View File
@@ -203,6 +203,17 @@ echo ""
echo "[*] Checking privacy-core shared library..."
PRIVACY_CORE_SO="$SCRIPT_DIR/privacy-core/target/release/libprivacy_core.so"
PRIVACY_CORE_DYLIB="$SCRIPT_DIR/privacy-core/target/release/libprivacy_core.dylib"
# MSI/AppImage/DMG installers stage the platform-specific shared library
# directly alongside this script (in backend-runtime/). If somebody runs
# start.sh from an installed app dir without Rust, they shouldn't see a
# spurious "install Rust" warning — the library is right next to them,
# just at a different path than the source-tree build.
if [ ! -f "$PRIVACY_CORE_SO" ] && [ -f "$SCRIPT_DIR/libprivacy_core.so" ]; then
PRIVACY_CORE_SO="$SCRIPT_DIR/libprivacy_core.so"
fi
if [ ! -f "$PRIVACY_CORE_DYLIB" ] && [ -f "$SCRIPT_DIR/libprivacy_core.dylib" ]; then
PRIVACY_CORE_DYLIB="$SCRIPT_DIR/libprivacy_core.dylib"
fi
if [ ! -f "$PRIVACY_CORE_SO" ] && [ ! -f "$PRIVACY_CORE_DYLIB" ]; then
if command -v cargo >/dev/null 2>&1; then
echo "[*] Building privacy-core release library..."
Generated
+2 -2
View File
@@ -74,7 +74,7 @@ wheels = [
[[package]]
name = "backend"
version = "0.9.79"
version = "0.9.81"
source = { editable = "backend" }
dependencies = [
{ name = "apscheduler" },
@@ -2231,7 +2231,7 @@ wheels = [
[[package]]
name = "shadowbroker"
version = "0.9.79"
version = "0.9.81"
source = { virtual = "." }
[package.metadata]