CalvinBackup/Shadowbroker
1
0
Fork 0
mirror of https://github.com/BigBodyCobain/Shadowbroker.git synced 2026-05-28 10:01:31 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
1d7fa5185a45dab61349dc339c2aec59e9692517
Shadowbroker/backend/tests/mesh/test_mesh_node_bootstrap_runtime.py
T
Shadowbroker 1d7fa5185a feat(infonet): private gate + DM hashchain spool with hardened propagation (#326) 
Private gate messages and offline DMs now ride the Infonet hashchain
as ciphertext-only events, replicated across nodes via private
transports (Tor onion / RNS / loopback) and decrypted only by parties
holding the gate or recipient keys.

Hashchain core (mesh_hashchain.py)
----------------------------------

* New ``append_private_gate_message`` and ``append_private_dm_message``
  append paths with full signature verification, public-key binding,
  revocation check, and replay protection in a dedicated sequence
  domain (so a gate post does not consume the author's public broadcast
  sequence, and a DM cannot replay-block a public message at sequence=1).
* Fork validation and full-chain validation now accept the gate
  signature compatibility variants — older signatures that canonicalize
  with/without epoch or reply_to still verify, so a re-sync from an
  older peer doesn't reject still-valid history.
* DM hashchain spool: capped at 2 active sealed offline DMs per
  recipient mailbox, plus a per-(sender, recipient) cap so one prolific
  sender can't consume both slots. 1-hour TTL on the cap counter.
  Spool intentionally small — it's an offline bootstrap channel,
  not a persistent mailbox.
* Rebuild-state preserves the gate sequence domain across reloads so
  a chain reload doesn't accidentally let an old gate sequence
  replay-collide on next append.

Schema enforcement (mesh_schema.py)
-----------------------------------

* Private gate + DM payloads have closed allowlists of fields.
  Plaintext keys (``message``, ``plaintext``, ``_local_plaintext``,
  ``_local_reply_to``) are explicit rejection-bait — they raise before
  the event ever touches the chain.
* DM ciphertext + nonce must look like base64-ish sealed bytes;
  obvious base64-encoded plaintext shapes are rejected.
* ``transport_lock`` required: DM hashchain spool requires
  ``private_strong``; gate accepts ``private``/``private_strong``/
  ``rns``/``onion``.

Defense-in-depth at the network layer (main.py + mesh_public.py)
----------------------------------------------------------------

* ``_infonet_sync_response_events`` now silently redacts private events
  (gate_message + dm_message) unless the request looks like a loopback /
  onion / RNS / private transport caller. If an operator accidentally
  exposes :8000 to the public internet, an external puller gets
  public events only — never ciphertext.
* ``_sync_from_peer`` raises ``PeerSyncRateLimited`` for 429 (handled
  as 4-tuple return with retry_after_s) and ``PeerSyncHTTPError`` for
  other non-200 statuses (handled by ``_run_public_sync_cycle`` to
  honor server cooldown hints even outside the 429 path).

DM relay hydration (main.py)
-----------------------------

* New ``_hydrate_dm_relay_from_chain``: when accepted dm_message chain
  events arrive on a node, they get deposited into the local DM relay
  store with a deterministic sender_token_hash so re-sync of the same
  event is idempotent. Recipients see the ciphertext as a normal DM
  on their next poll and decrypt with their existing recipient key.

Other surfaces
--------------

* meshnode.bat / meshnode.sh now set ``MESH_INFONET_ALLOW_CLEARNET_SYNC=
  false`` and the participant runtime flags by default so a freshly
  spun-up node defaults to private-only sync.
* InfonetTerminal/InfonetShell.tsx adds a gate directory renderer for
  the new private-gate workflow.
* docker-compose.relay.yml binds the relay backend to 127.0.0.1:8000
  only; Tor's hidden service forwards onion traffic into 127.0.0.1.
  Public clearnet :8000 stays off the network edge.

Tests
-----

* 7 new tests in test_private_gate_hashchain.py + test_private_dm_
  hashchain.py covering: gate fork accepts ciphertext propagation,
  gate fork rejects plaintext, append rejects plaintext before
  normalize, append requires private_strong, append rejects
  non-sealed ciphertext shape, DM spool 2-per-recipient + 1-per-pair
  cap, DM hydration delivers to poll/claim.
* Updated test_mesh_node_bootstrap_runtime.py covers 429 backoff via
  PeerSyncRateLimited 4-tuple AND PeerSyncHTTPError exception.
* Updated test_s14b_public_sync_gate_filter.py + test_s9b_gate_store_
  hydration.py + test_gate_write_cutover.py cover the new private
  redaction on public sync responses.
* test_private_gate_hashchain.py + test_private_dm_hashchain.py:
  10 passed locally.
* Combined mesh-relevant suite (the 5 modified existing tests +
  2 new): 17 passed.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 21:25:18 -06:00

429 lines
16 KiB
Python
Raw Blame History

import asyncio
import base64
import json
from types import SimpleNamespace
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ed25519
from httpx import ASGITransport, AsyncClient
def test_onion_peer_requests_use_arti_socks_proxy(monkeypatch):
import main
from services import wormhole_supervisor
monkeypatch.setattr(main, "_infonet_private_transport_required", lambda: True)
monkeypatch.setattr(
main,
"get_settings",
lambda: SimpleNamespace(MESH_ARTI_ENABLED=True, MESH_ARTI_SOCKS_PORT=19050),
)
monkeypatch.setattr(wormhole_supervisor, "_check_arti_ready", lambda: True)
proxies = main._infonet_peer_requests_proxies("http://exampleabcd.onion:8000")
assert proxies == {
"http": "socks5h://127.0.0.1:19050",
"https": "socks5h://127.0.0.1:19050",
}
def test_private_peer_requests_reject_clearnet(monkeypatch):
import main
monkeypatch.setattr(main, "_infonet_private_transport_required", lambda: True)
try:
main._infonet_peer_requests_proxies("https://seed.example")
except RuntimeError as exc:
assert "private Infonet requires onion/RNS transport" in str(exc)
else:
raise AssertionError("clearnet peer was allowed while private transport is required")
def test_local_peer_url_prefers_configured_public_peer_url(monkeypatch):
import main
monkeypatch.setattr(
main,
"get_settings",
lambda: SimpleNamespace(
MESH_PUBLIC_PEER_URL="HTTP://LOCALPEEREXAMPLE.onion:8000/",
),
)
assert main._local_infonet_peer_url() == "http://localpeerexample.onion:8000"
def _write_signed_manifest(path, *, private_key):
from services.mesh.mesh_bootstrap_manifest import BOOTSTRAP_MANIFEST_VERSION
from services.mesh.mesh_crypto import canonical_json
payload = {
"version": BOOTSTRAP_MANIFEST_VERSION,
"issued_at": 1_700_000_000,
"valid_until": 1_800_000_000,
"signer_id": "bootstrap-a",
"peers": [
{
"peer_url": "https://seed.example",
"transport": "clearnet",
"role": "seed",
"label": "Seed A",
}
],
}
signature = base64.b64encode(private_key.sign(canonical_json(payload).encode("utf-8"))).decode("utf-8")
path.write_text(json.dumps({**payload, "signature": signature}), encoding="utf-8")
def test_refresh_node_peer_store_promotes_manifest_peers_to_sync_only(tmp_path, monkeypatch):
import main
from services.config import get_settings
from services.mesh import mesh_bootstrap_manifest as manifest_mod
from services.mesh import mesh_peer_store as peer_store_mod
manifest_key = ed25519.Ed25519PrivateKey.generate()
manifest_pub = base64.b64encode(
manifest_key.public_key().public_bytes(
serialization.Encoding.Raw,
serialization.PublicFormat.Raw,
)
).decode("utf-8")
manifest_path = tmp_path / "bootstrap.json"
peer_store_path = tmp_path / "peer_store.json"
_write_signed_manifest(manifest_path, private_key=manifest_key)
monkeypatch.setattr(manifest_mod, "DEFAULT_BOOTSTRAP_MANIFEST_PATH", manifest_path)
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", manifest_pub)
monkeypatch.setenv("MESH_BOOTSTRAP_MANIFEST_PATH", str(manifest_path))
monkeypatch.setenv("MESH_RELAY_PEERS", "https://operator.example")
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "")
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
monkeypatch.setenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", "true")
get_settings.cache_clear()
try:
snapshot = main._refresh_node_peer_store(now=1_750_000_000)
store = peer_store_mod.PeerStore(peer_store_path)
store.load()
finally:
get_settings.cache_clear()
assert snapshot["manifest_loaded"] is True
assert snapshot["bootstrap_peer_count"] == 1
assert snapshot["sync_peer_count"] == 2
assert snapshot["push_peer_count"] == 1
assert [record.peer_url for record in store.records_for_bucket("bootstrap")] == ["https://seed.example"]
assert sorted(record.peer_url for record in store.records_for_bucket("sync")) == [
"https://operator.example",
"https://seed.example",
]
assert [record.peer_url for record in store.records_for_bucket("push")] == ["https://operator.example"]
def test_refresh_node_peer_store_adds_bootstrap_seed_as_pull_only_peer(tmp_path, monkeypatch):
import main
from services.config import get_settings
from services.mesh import mesh_peer_store as peer_store_mod
peer_store_path = tmp_path / "peer_store.json"
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
monkeypatch.setenv("MESH_RELAY_PEERS", "")
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "https://node.shadowbroker.info")
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
monkeypatch.setenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", "true")
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "")
get_settings.cache_clear()
try:
snapshot = main._refresh_node_peer_store(now=1_750_000_000)
store = peer_store_mod.PeerStore(peer_store_path)
store.load()
finally:
get_settings.cache_clear()
assert snapshot["manifest_loaded"] is False
assert snapshot["bootstrap_seed_peer_count"] == 1
assert snapshot["default_sync_peer_count"] == 1
assert snapshot["bootstrap_peer_count"] == 1
assert snapshot["sync_peer_count"] == 1
assert snapshot["push_peer_count"] == 0
assert [record.peer_url for record in store.records_for_bucket("bootstrap")] == [
"https://node.shadowbroker.info"
]
assert [record.peer_url for record in store.records_for_bucket("sync")] == [
"https://node.shadowbroker.info"
]
assert store.records_for_bucket("sync")[0].source == "bundle"
def test_refresh_node_peer_store_suppresses_clearnet_seed_by_default(tmp_path, monkeypatch):
import main
from services.config import get_settings
from services.mesh import mesh_peer_store as peer_store_mod
peer_store_path = tmp_path / "peer_store.json"
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
monkeypatch.setenv("MESH_RELAY_PEERS", "")
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", "https://node.shadowbroker.info")
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "")
get_settings.cache_clear()
try:
snapshot = main._refresh_node_peer_store(now=1_750_000_000)
store = peer_store_mod.PeerStore(peer_store_path)
store.load()
finally:
get_settings.cache_clear()
assert snapshot["private_transport_required"] is True
assert snapshot["skipped_clearnet_peer_count"] == 1
assert snapshot["bootstrap_peer_count"] == 0
assert snapshot["sync_peer_count"] == 0
assert "no clearnet sync fallback" in snapshot["last_bootstrap_error"]
assert store.records_for_bucket("bootstrap") == []
assert store.records_for_bucket("sync") == []
def test_refresh_node_peer_store_prunes_persisted_clearnet_records_in_private_mode(tmp_path, monkeypatch):
import main
from services.config import get_settings
from services.mesh import mesh_peer_store as peer_store_mod
peer_store_path = tmp_path / "peer_store.json"
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
store = peer_store_mod.PeerStore(peer_store_path)
store.upsert(
peer_store_mod.make_bootstrap_peer_record(
peer_url="https://node.shadowbroker.info",
transport="clearnet",
role="seed",
signer_id="shadowbroker-default",
now=1_749_999_900,
)
)
store.upsert(
peer_store_mod.make_sync_peer_record(
peer_url="https://node.shadowbroker.info",
transport="clearnet",
role="seed",
source="bundle",
now=1_749_999_900,
)
)
store.upsert(
peer_store_mod.make_push_peer_record(
peer_url="https://node.shadowbroker.info",
transport="clearnet",
role="relay",
now=1_749_999_900,
)
)
store.save()
onion_seed = "http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"
monkeypatch.setenv("MESH_RELAY_PEERS", "")
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_PEERS", onion_seed)
monkeypatch.setenv("MESH_DEFAULT_SYNC_PEERS", "")
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
monkeypatch.setenv("MESH_BOOTSTRAP_SIGNER_PUBLIC_KEY", "")
get_settings.cache_clear()
try:
snapshot = main._refresh_node_peer_store(now=1_750_000_000)
store = peer_store_mod.PeerStore(peer_store_path)
store.load()
finally:
get_settings.cache_clear()
assert snapshot["private_transport_required"] is True
assert snapshot["pruned_clearnet_peer_count"] == 3
assert [record.peer_url for record in store.records()] == [onion_seed, onion_seed]
assert {record.bucket for record in store.records()} == {"bootstrap", "sync"}
assert all(record.transport == "onion" for record in store.records())
def test_infonet_peer_url_filter_excludes_clearnet_in_private_mode(monkeypatch):
import main
from services.config import get_settings
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
get_settings.cache_clear()
try:
assert main._filter_infonet_peer_urls(
[
"https://node.shadowbroker.info",
"http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000",
]
) == ["http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"]
finally:
get_settings.cache_clear()
def test_public_sync_cycle_backs_off_on_429_retry_after(tmp_path, monkeypatch):
import time
import main
from services.config import get_settings
from services.mesh import mesh_peer_store as peer_store_mod
peer_store_path = tmp_path / "peer_store.json"
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
onion_seed = "http://gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000"
store = peer_store_mod.PeerStore(peer_store_path)
store.upsert(
peer_store_mod.make_sync_peer_record(
peer_url=onion_seed,
transport="onion",
role="seed",
source="bundle",
now=1_750_000_000,
)
)
store.save()
monkeypatch.delenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", raising=False)
monkeypatch.setenv("MESH_SYNC_FAILURE_BACKOFF_S", "60")
monkeypatch.setenv("MESH_BOOTSTRAP_SEED_FAILURE_COOLDOWN_S", "15")
get_settings.cache_clear()
monkeypatch.setattr(main, "_participant_node_enabled", lambda: True)
monkeypatch.setattr(main, "_ensure_infonet_private_transport_ready", lambda reason="": True)
monkeypatch.setattr(
main,
"_sync_from_peer",
lambda peer_url: (_ for _ in ()).throw(
main.PeerSyncHTTPError(429, "rate limited", retry_after_s=180)
),
)
main.set_sync_state(main.SyncWorkerState())
try:
before = int(time.time())
state = main._run_public_sync_cycle()
store = peer_store_mod.PeerStore(peer_store_path)
store.load()
finally:
get_settings.cache_clear()
main.set_sync_state(main.SyncWorkerState())
record = store.records_for_bucket("sync")[0]
assert state.last_error == "HTTP 429: rate limited"
assert state.next_sync_due_at >= before + 180
assert record.cooldown_until >= before + 180
def test_verify_peer_push_hmac_requires_allowlisted_peer(monkeypatch):
import hashlib
import hmac
import main
from services.config import get_settings
from services.mesh.mesh_crypto import _derive_peer_key
monkeypatch.setenv("MESH_PEER_PUSH_SECRET", "shared-secret")
get_settings.cache_clear()
monkeypatch.setattr(main, "authenticated_push_peer_urls", lambda *args, **kwargs: ["https://good.example"])
try:
body = b'{"events":[]}'
peer_url = "https://bad.example"
peer_key = _derive_peer_key("shared-secret", peer_url)
signature = hmac.new(peer_key, body, hashlib.sha256).hexdigest()
request = SimpleNamespace(
headers={"x-peer-url": peer_url, "x-peer-hmac": signature},
url=SimpleNamespace(scheme="https", netloc="bad.example"),
)
assert main._verify_peer_push_hmac(request, body) is False
finally:
get_settings.cache_clear()
def test_infonet_status_includes_node_runtime_snapshot(monkeypatch):
import main
from services import wormhole_supervisor
monkeypatch.setattr(main, "_check_scoped_auth", lambda *_args, **_kwargs: (True, "ok"))
monkeypatch.setattr(
wormhole_supervisor,
"get_wormhole_state",
lambda: {"configured": True, "ready": True, "arti_ready": True, "rns_ready": False},
)
monkeypatch.setattr(
main,
"_node_runtime_snapshot",
lambda: {
"node_mode": "participant",
"node_enabled": True,
"bootstrap": {"sync_peer_count": 2, "push_peer_count": 1},
"sync_runtime": {"last_outcome": "ok"},
"push_runtime": {"last_event_id": "evt-1"},
},
)
async def _run():
async with AsyncClient(transport=ASGITransport(app=main.app), base_url="http://test") as ac:
response = await ac.get("/api/mesh/infonet/status")
return response.json()
result = asyncio.run(_run())
assert result["node_mode"] == "participant"
assert result["node_enabled"] is True
assert result["bootstrap"]["sync_peer_count"] == 2
assert result["bootstrap"]["push_peer_count"] == 1
assert result["sync_runtime"]["last_outcome"] == "ok"
assert result["push_runtime"]["last_event_id"] == "evt-1"
def test_public_sync_cycle_allows_first_node_without_peers(tmp_path, monkeypatch):
import main
from services.config import get_settings
from services.mesh import mesh_peer_store as peer_store_mod
peer_store_path = tmp_path / "peer_store.json"
monkeypatch.setattr(peer_store_mod, "DEFAULT_PEER_STORE_PATH", peer_store_path)
monkeypatch.setattr(main, "_participant_node_enabled", lambda: True)
monkeypatch.setenv("MESH_INFONET_ALLOW_CLEARNET_SYNC", "true")
get_settings.cache_clear()
try:
result = main._run_public_sync_cycle()
finally:
get_settings.cache_clear()
assert result.last_outcome == "solo"
assert result.last_error == ""
assert result.last_peer_url == ""
assert result.consecutive_failures == 0
def test_headless_mesh_node_runtime_is_explicit(monkeypatch):
import main
monkeypatch.setattr(main, "_MESH_ONLY", True)
monkeypatch.setattr(main, "_HEADLESS_MESH_NODE_RUNTIME", False)
assert main._infonet_node_runtime_requested() is False
monkeypatch.setattr(main, "_HEADLESS_MESH_NODE_RUNTIME", True)
assert main._infonet_node_runtime_requested() is True
def test_meshnode_scripts_enable_private_hashchain_runtime():
from pathlib import Path
root = Path(__file__).resolve().parents[3]
bat = (root / "meshnode.bat").read_text(encoding="utf-8")
sh = (root / "meshnode.sh").read_text(encoding="utf-8")
for script in (bat, sh):
assert "SHADOWBROKER_MESH_NODE_RUNTIME=true" in script
assert "MESH_INFONET_ALLOW_CLEARNET_SYNC=false" in script
assert "MESH_ARTI_ENABLED=true" in script
assert "MESH_DM_HASHCHAIN_SPOOL_LIMIT=2" in script
assert "gqpbunqbgtkcqilvclm3xrkt3zowjyl3s62kkktvojgvxzizamvbrqid.onion:8000" in script
Reference in New Issue View Git Blame Copy Permalink