Align backend, desktop, helm, and frontend package versions for the Telegram OSINT and OpenClaw recon release.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add Telegram OSINT with hourly incremental t.me scraping, metro geocoding
separate from news centroids, threat-intercept popup UI with inline media,
and HTML markers above alert boxes so pins stay clickable. Expose GFW_API_TOKEN
in onboarding and Settings Maritime; harden GFW/CCTV/geo fetchers. Port Osiris-
derived recon, SCM, entity graph, malware/cyber feeds, sanctions, and submarine
cable layers with tests and documentation.
Co-authored-by: Cursor <cursoragent@cursor.com>
Align full /api/live-data with slow-tier orjson options, remove dead main.py duplicate, cap slow batches to pool size, cancel queued work on timeout, and stop retrying HTTP 4xx/5xx.
Co-authored-by: Cursor <cursoragent@cursor.com>
Default python main.py to loopback, deep-copy dashboard snapshots outside the store lock with ETag on full live-data, and route GDELT/LiveUAMap/CCTV/slow-tier work through an isolated executor so Playwright jobs cannot starve fast-tier workers.
Co-authored-by: Cursor <cursoragent@cursor.com>
Operators enable Polymarket/Kalshi correlation from Global Threat Intercept with a consent dialog; polls use a jittered schedule separate from the slow tier. Right-click Sentinel imagery returns up to three signed scenes again.
Co-authored-by: Cursor <cursoragent@cursor.com>
Each install pulls ~60-day sightings from nuforc.org every Monday; disk cache
matches weekly cadence so users keep current pins between restarts.
Co-authored-by: Cursor <cursoragent@cursor.com>
Filter stale rows out of nuforc_recent_sightings.json on load; add requests-based
live scrape when curl is disabled; daily scheduler rebuild instead of weekly-only.
Co-authored-by: Cursor <cursoragent@cursor.com>
- User-Agent is per-install handle only (no Shadowbroker product token)
- LiveUAMap: Windows UI consent when enabling Global Incidents; env override
- Meshtastic callsign upstream header off by default (opt-in true)
- Expanded docs/OUTBOUND_DATA.md and README link for CCTV, basemap, Broadcastify
Co-authored-by: Cursor <cursoragent@cursor.com>
Operators can set DEEPSTATE_MIRROR_COMMIT for immutable frontline ingest; Madrid KML tries HTTPS then HTTP without changing camera image URLs or proxy Referers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(ci): switch mirror-to-github job from PAT to per-repo SSH deploy key
GitHub fine-grained PATs are capped at 366 days, classic PATs would
need 'public_repo' (broader scope than needed). Per-repo SSH deploy
keys are tighter:
- Can ONLY push to BigBodyCobain/Shadowbroker (no access to anything
else, not even other repos owned by the same account).
- Never expire.
- Rotating == one-click delete on github.com/.../settings/keys.
Changes:
- New CI/CD variable GITHUB_MIRROR_SSH_KEY (File, Protected) holding
the ed25519 private half. Public half lives on the repo's deploy
keys with write access enabled.
- mirror-to-github before_script writes the key to ~/.ssh/id_ed25519,
pins github.com host fingerprints (ed25519 + ecdsa + rsa from the
2023-03-24 rotation) into ~/.ssh/known_hosts so we never trust a
MITM, then pushes via git@github.com:... instead of HTTPS.
- Job rule now gates on GITHUB_MIRROR_SSH_KEY (the new var) instead
of GITHUB_MIRROR_TOKEN (which never existed).
After this lands, every commit pushed directly to GitLab main will
mirror back to GitHub main automatically — closing the loop on
bi-directional sync.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(secret-scan): exempt SSH known_hosts entries from leaked-key detection
PR #331 introduced github.com host fingerprints pinned in
.gitlab-ci.yml's mirror-to-github before_script. The scanner flagged
them as embedded secrets and blocked CI:
BLOCKED: Embedded secrets/tokens found in:
.gitlab-ci.yml
133: github.com ssh-ed25519 AAAA...
135: github.com ssh-rsa AAAA...
These are PUBLIC host keys — the whole point of pinning known_hosts is
to publish the fingerprint widely so a MITM is detectable. They are
documented at https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints
and committing them is the correct, secure practice.
Fix: add a KNOWN_HOSTS_LINE regex to the content-scan block that
recognizes `<host-or-ip> [salt] <algo> AAAA...` shape lines (the
exact format used in ~/.ssh/known_hosts) and filters them out before
flagging the file. Bare `ssh-rsa AAAA...` lines without a host prefix
are still caught — only the host-key shape is exempt.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
The build-backend and build-frontend jobs were failing immediately after
identity verification finally allocated runners:
$ docker buildx create --use --name multiarch --driver docker-container
ERROR: could not create a builder instance with TLS data loaded from
environment. Please use `docker context create <context-name>` to create
a context for current environment and then create a builder instance
with context set to <context-name>
The dind service exports DOCKER_HOST=tcp://docker:2376 +
DOCKER_TLS_CERTDIR=/certs, but buildx --driver docker-container doesn't
read TLS from those env vars directly. Documented GitLab fix: create an
empty `docker context` (which inherits the current TLS env), then bind
buildx to that context name as a positional arg.
After this lands, the multi-arch buildx jobs should actually build and
push amd64 + arm64 images to
registry.gitlab.com/bigbodycobain/shadowbroker/backend:latest
registry.gitlab.com/bigbodycobain/shadowbroker/frontend:latest
Surfaced by the post-verification pipeline at
https://gitlab.com/bigbodycobain/Shadowbroker/-/pipelines/2550501798
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Pipelines on the GitLab mirror have been instant-failing with 0 jobs and
no started_at since the project was created — classic "shared runners
not allocated to unverified free-tier accounts" pattern. The account is
now identity-verified; this trivial comment bump exists solely to fire a
fresh pipeline that confirms runners now pick up the build-backend and
build-frontend jobs.
If the resulting pipeline produces real jobs that build the multi-arch
images and push them to registry.gitlab.com/bigbodycobain/shadowbroker/{backend,frontend},
the GitLab install path is at full parity with the GitHub one.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Re-cut v0.9.81 binaries from current main (which now includes the
private gate + DM hashchain spool from #326 and the gate-directory
test from #327). All three artifacts were signed with the same
minisign updater key as the original v0.9.81 release, so existing
v0.9.81 installs on Tauri auto-update accept the new bundles.
Updated hashes (verified against released assets):
- ShadowBroker_v0.9.81.zip f81f454bdc88e9a32c351df38212b8cfa624704d65764b971bb091eef62259c6
- ShadowBroker_0.9.81_x64-setup.exe 25e9a95d0d8ce959a7d08fe8e7406772ae24b596652793e81d1de5d02510a5a6
- ShadowBroker_0.9.81_x64_en-US.msi 34e655fc0c0f195ee4ac978f228a4b2b9d5565253b8771aca9ef4693409e9e70
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Adds the focused test Codex wrote alongside the gate-directory UI work
that already shipped in #326 (the `renderGateDirectory` helper used
both under the Infonet logo on the landing screen and as the output of
the `gates` command in the terminal).
The renderer itself is already on origin/main; this PR just ships the
test so CI catches regressions to the dual-variant render.
Verified locally:
- frontend npm run test:ci -- src/__tests__/mesh/infonetShellGateDirectory.test.tsx → 1/1 pass
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Private gate messages and offline DMs now ride the Infonet hashchain
as ciphertext-only events, replicated across nodes via private
transports (Tor onion / RNS / loopback) and decrypted only by parties
holding the gate or recipient keys.
Hashchain core (mesh_hashchain.py)
----------------------------------
* New ``append_private_gate_message`` and ``append_private_dm_message``
append paths with full signature verification, public-key binding,
revocation check, and replay protection in a dedicated sequence
domain (so a gate post does not consume the author's public broadcast
sequence, and a DM cannot replay-block a public message at sequence=1).
* Fork validation and full-chain validation now accept the gate
signature compatibility variants — older signatures that canonicalize
with/without epoch or reply_to still verify, so a re-sync from an
older peer doesn't reject still-valid history.
* DM hashchain spool: capped at 2 active sealed offline DMs per
recipient mailbox, plus a per-(sender, recipient) cap so one prolific
sender can't consume both slots. 1-hour TTL on the cap counter.
Spool intentionally small — it's an offline bootstrap channel,
not a persistent mailbox.
* Rebuild-state preserves the gate sequence domain across reloads so
a chain reload doesn't accidentally let an old gate sequence
replay-collide on next append.
Schema enforcement (mesh_schema.py)
-----------------------------------
* Private gate + DM payloads have closed allowlists of fields.
Plaintext keys (``message``, ``plaintext``, ``_local_plaintext``,
``_local_reply_to``) are explicit rejection-bait — they raise before
the event ever touches the chain.
* DM ciphertext + nonce must look like base64-ish sealed bytes;
obvious base64-encoded plaintext shapes are rejected.
* ``transport_lock`` required: DM hashchain spool requires
``private_strong``; gate accepts ``private``/``private_strong``/
``rns``/``onion``.
Defense-in-depth at the network layer (main.py + mesh_public.py)
----------------------------------------------------------------
* ``_infonet_sync_response_events`` now silently redacts private events
(gate_message + dm_message) unless the request looks like a loopback /
onion / RNS / private transport caller. If an operator accidentally
exposes :8000 to the public internet, an external puller gets
public events only — never ciphertext.
* ``_sync_from_peer`` raises ``PeerSyncRateLimited`` for 429 (handled
as 4-tuple return with retry_after_s) and ``PeerSyncHTTPError`` for
other non-200 statuses (handled by ``_run_public_sync_cycle`` to
honor server cooldown hints even outside the 429 path).
DM relay hydration (main.py)
-----------------------------
* New ``_hydrate_dm_relay_from_chain``: when accepted dm_message chain
events arrive on a node, they get deposited into the local DM relay
store with a deterministic sender_token_hash so re-sync of the same
event is idempotent. Recipients see the ciphertext as a normal DM
on their next poll and decrypt with their existing recipient key.
Other surfaces
--------------
* meshnode.bat / meshnode.sh now set ``MESH_INFONET_ALLOW_CLEARNET_SYNC=
false`` and the participant runtime flags by default so a freshly
spun-up node defaults to private-only sync.
* InfonetTerminal/InfonetShell.tsx adds a gate directory renderer for
the new private-gate workflow.
* docker-compose.relay.yml binds the relay backend to 127.0.0.1:8000
only; Tor's hidden service forwards onion traffic into 127.0.0.1.
Public clearnet :8000 stays off the network edge.
Tests
-----
* 7 new tests in test_private_gate_hashchain.py + test_private_dm_
hashchain.py covering: gate fork accepts ciphertext propagation,
gate fork rejects plaintext, append rejects plaintext before
normalize, append requires private_strong, append rejects
non-sealed ciphertext shape, DM spool 2-per-recipient + 1-per-pair
cap, DM hydration delivers to poll/claim.
* Updated test_mesh_node_bootstrap_runtime.py covers 429 backoff via
PeerSyncRateLimited 4-tuple AND PeerSyncHTTPError exception.
* Updated test_s14b_public_sync_gate_filter.py + test_s9b_gate_store_
hydration.py + test_gate_write_cutover.py cover the new private
redaction on public sync responses.
* test_private_gate_hashchain.py + test_private_dm_hashchain.py:
10 passed locally.
* Combined mesh-relevant suite (the 5 modified existing tests +
2 new): 17 passed.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(start-scripts): find bundled privacy_core.dll next to script
start.bat and start.sh only checked the source-tree DLL path
(``privacy-core/target/release/privacy_core.dll``), not the bundled
location where MSI/AppImage/DMG installers stage the library directly
next to the script in backend-runtime/.
Users running start.bat from inside an MSI install dir (a documented
workaround when the desktop shell crashes) saw a scary "install Rust"
warning even though the DLL was sitting right next to them. See issue
#319 for the user-reported confusion.
Fix: add a fallback check for the bundled location before falling
through to the "build privacy-core from source" warning. Source-tree
behavior unchanged — the source path is still preferred when present.
Also re-stamps the v0.9.81 source archive: ``release_digests.json``
v0.9.81 zip hash updated to point at the rebuilt source archive that
contains these script changes. MSI/EXE/sig hashes are unchanged (the
scripts live at the repo root, not inside the desktop bundle).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(#319): bundle start.bat + start.sh into the MSI/EXE installers
Follow-up to the start-script DLL fallback fix in the prior commit.
ChrisMTheMan's report on #319 made it clear the workaround flow was:
1. MSI install crashes on launch (different bug, fixed in v0.9.81)
2. User goes looking for start.bat to launch the backend manually
3. start.bat isn't in their install dir, so they go fetch it from GitHub
4. They get a working script but it doesn't know about the bundled
privacy_core.dll layout, so they see a scary "install Rust" warning
The prior commit fixed step 4. This commit fixes step 3 — start.bat and
start.sh now ship inside the MSI/EXE installers (staged into
backend-runtime/ next to the privacy_core.dll they expect to find).
After the rebuild lands, an MSI user looking for these scripts finds
them right inside their install dir, already pointing at the correct
bundled DLL location.
What changed
------------
* ``build-backend-runtime.cjs`` now has a ``stageStartScripts()`` step
that copies start.bat and start.sh from the repo root into the
staged backend-runtime/. Preserves the executable bit on .sh under
POSIX.
* ``release_digests.json`` v0.9.81 block hashes refreshed for the
rebuilt MSI / EXE / source-zip (the scripts being bundled changed
the MSI/EXE contents; the source zip also includes the start-script
fix from the prior commit).
ShadowBroker_v0.9.81.zip 6.06 MB
af8c87ccdece8fbb9aadc6be63cce10d3fcba74e6d87ef83289dda6d555fd270
ShadowBroker_0.9.81_x64_en-US.msi 122.4 MB
8977c9a1c54e1f0d030436be9c4e3d81d766cc0080699eb747649095f360c7ff
ShadowBroker_0.9.81_x64-setup.exe 76.5 MB
4e866fa0423c0c2470ed32f4809167a7815dc23ee7762b69e95681c1f3a28250
Post-merge plan
---------------
Force-move the v0.9.81 tag to this commit and replace ALL release
assets on the GitHub release: zip, msi, exe, both .sig files,
latest.json, SHA256SUMS.txt, release-manifest.json.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
What this release does
----------------------
1. Establishes a fresh Tauri updater signing keypair. The previous keypair
(pubkey baked into v0.9.79 / v0.9.8) had no matching private key on
any maintainer-controlled machine — every prior release shipped
without signatures, so auto-update has never actually worked. v0.9.81
rotates to a new pubkey and ships signed installers + latest.json so
every release from here is a one-click upgrade.
2. Fixes the ``admin_session_required`` race in TopRightControls.tsx.
The updateAction state used to default to ``auto_apply`` at React-init
time. A click on the Update button before the async runtime probe
completed went down the auto_apply path (POST /api/system/update),
which throws ``admin_session_required`` on fresh sessions. Desktop
installs now default to ``manual_download`` based on synchronous
``window.__TAURI__`` detection at useState init.
One-time cost for current installs
----------------------------------
Anyone on v0.9.79 or v0.9.8 will see the in-app Update button still
trigger the broken path on their existing install (the fix only takes
effect once they're ON v0.9.81). The MANUAL DOWNLOAD button in the
update dialog opens the GitHub release page, where they grab the .msi
and run it. After that one manual hop, all future updates are seamless.
Release artifacts
-----------------
ShadowBroker_v0.9.81.zip 6.06 MB
42f8a51f9a5690d1e7349d90d8ecf2d163c9061d6cf90c69ee03647a785437ff
ShadowBroker_0.9.81_x64_en-US.msi 122.4 MB
a45b177c26c95d2b28d71592d7147e88ff4e104865f214fde11249d311ec9e25
ShadowBroker_0.9.81_x64-setup.exe 76.5 MB
eca884b9d37eeccd0f11c91dcc6f6ae1b3609d9dee72bd73c37c9a427babfef2
Plus .sig files for the .msi and .exe, plus a signed latest.json for
the Tauri updater endpoint.
Sizes match the v0.9.79 / v0.9.8 reference shape within drift for
the new TopRightControls patch.
release_digests.json keeps v0.9.79 + v0.9.8 blocks alongside v0.9.81
so operators still on those versions continue to validate cleanly
during the rollout transition.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Issues #319 and #296 reported that the installed v0.9.79 Windows MSI/EXE
crashed on launch with:
thread 'main' panicked ... failed to setup app: error encountered
during setup hook: ShadowBroker cannot start: the bundled local
backend failed to launch.
technical detail: managed_backend_exited_early:exit code: 103
Root cause: ``backend/pyproject.toml`` declares ``defusedxml>=0.7.1`` and
``PySocks==1.7.1`` as runtime dependencies, but the venv used to build
v0.9.79 (and the initial v0.9.8 publish) had both missing. When
``services/fetchers/aircraft_database.py`` does
``import defusedxml.ElementTree`` at startup, Python raises
``ModuleNotFoundError`` and uvicorn exits, which Tauri reports as
``managed_backend_exited_early``.
Both packages now installed in the build venv. ``main.py`` imports
end-to-end with only the expected ``plane_alert_db.json not found``
warning (runtime-state file, populated on first launch).
Rebuilt artifacts on the maintainer's local machine:
ShadowBroker_v0.9.8.zip 6.06 MB
183bb5cd62b9b9349d95df5ef7696cb6ca810ab4b991fa9dab6f898af4c7a175
ShadowBroker_0.9.8_x64_en-US.msi 122.4 MB
fe22f9d51e4360d74c18a7250c2fbb9ed4fa4c7a884b3ac0d04a21115466386b
ShadowBroker_0.9.8_x64-setup.exe 76.5 MB
94a0309862e9c81c92cdcbfea8eec9dbb97eef19ded82b26217b397defbc810c
After this merges, the v0.9.8 tag will be force-moved to this commit and
the GitHub release assets replaced so the integrity chain validates
against the working installers instead of the broken ones.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Bumps every hardcoded 0.9.79 → 0.9.8 across backend, frontend,
desktop-shell, helm, lockfiles, test fixtures. Refreshes the in-app
ChangelogModal HEADLINE_FEATURES, NEW_FEATURES, and BUG_FIXES with the
v0.9.8 highlights.
Release artifacts built locally and hashed into release_digests.json:
ShadowBroker_v0.9.8.zip 6.06 MB
d506f6b8462ccb12096f0cd9462233be58928094240416b65fb3127bdd1f3820
ShadowBroker_0.9.8_x64_en-US.msi 122.4 MB
d4be4cb68c3e6409fff54c225acdcdd08e27d5d6d2b31616d78d2a4f6812991d
ShadowBroker_0.9.8_x64-setup.exe 76.5 MB
1115d1f5cf37edd03ea2c21d821c7626e1bf3319c990402aaa0293bca46fea67
Sizes match the v0.9.79 reference shape (5.76 MB / 117 MB / 72.9 MB)
within expected drift for new code. The .zip is a `git archive` of the
v0.9.8 source tree (matching v0.9.79's approach).
Audit confirms no .env, .key, .venv-dir, or cache files leaked into the
backend-runtime bundle. Python 3.11.9 + 199 site-packages + privacy_core
all staged correctly.
Headline changes since v0.9.79:
* Cumulative fuel/CO2 per flight (#317) — running totals since first
observation, not just per-hour rate.
* AIS maritime resilience (#314, #316) — outage banner + AISHub REST
fallback when AISStream WebSocket primary is offline.
* Data-layer repair (#311, #312) — UAP fallback respects the 60-day
cutoff; GPS jamming threshold tuning + nac_p=0 inclusion so the layer
actually fires.
* Per-flight source attribution (#313) — source field on every record.
* Cross-node DM mailbox replication (#309).
* Infonet sync HTTP 429 honored (#310).
Test fixtures updated:
* test_per_operator_outbound_attribution.py — added v0.9.8 UA strings
to the banned-aggregate-literals list (alongside v0.9.79).
* updateRuntime.test.ts — bumped asset filename fixtures to v0.9.8.
release_digests.json keeps the v0.9.79 block alongside v0.9.8 so
operators still on 0.9.79 validate cleanly during the rollout.
The accent narrowing fix in ChangelogModal (one feature uses 'purple',
two use 'cyan' so the renderer's `accent === 'purple'` comparison
still type-checks) is included.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
build-frontend-export.cjs stages a desktop-only frontend export tree and
strips the ``force-dynamic`` + ``revalidate`` directives from
``frontend/src/app/layout.tsx`` so Next's ``output: "export"`` can
prerender every route.
The strip regexes only matched LF (``\n``). Any Windows checkout without
``core.autocrlf=input`` has CRLF line endings, the strip silently
no-op'd, and the desktop build failed at the static-export step:
Error: Page with `dynamic = "force-dynamic"` couldn't be exported.
`output: "export"` requires all pages be renderable statically
because there is no runtime server to dynamically render routes
in this output format.
Export encountered an error on /_not-found/page: /_not-found
Reaches every Windows contributor who hasn't normalized line endings
locally. Replacing each ``\n`` in the strip regexes with ``\r?\n``
makes the strip CRLF-tolerant; LF behavior is unchanged.
Verified by running both regexes against the actual layout.tsx (302
bytes removed, force-dynamic + revalidate both gone) and against a
synthetic LF input (296 bytes removed, same outcome).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Pre-fix the emissions tooltip only showed the per-hour *rate* — what most
users actually want is the cumulative *amount* burned. This adds running
totals computed by multiplying the model-based rate by the elapsed
observation time since we first saw the airframe.
New module ``flight_observations.py``:
* Tracks first_seen_at + last_seen_at per icao24 hex.
* Re-opens a fresh session when an aircraft is unseen for > 15 min
(treated as a new flight — landed and took off, or transited a dead
zone). Prevents the cumulative counter from resetting mid-flight if
the trail-rendering cache prunes the trail.
* Clamps elapsed time to 24h max so clock skew can't produce comically
large numbers.
* Pruned every 5 min via a new scheduler job (mirrors ais_prune cadence).
flights.py + military.py emission enrichment now also attaches:
* observed_seconds — how long we've been tracking this airframe.
* fuel_gallons_burned — rate * elapsed_h.
* co2_kg_emitted — rate * elapsed_h.
The existing per-hour rate fields stay in the dict for backward compat
and are shown as small secondary context in the tooltip.
Frontend EmissionsEstimateBlock (NewsFeed.tsx) now prominently shows
the cumulative totals with the rate as smaller context underneath plus
"Observed in flight for Xh Ym". When observed_seconds is 0 (first refresh)
it renders "Just observed · totals will appear on next refresh" instead
of a misleading "0 gal".
12 backend tests cover record/accumulate/reset, the 24h clamp, prune,
case-insensitive key normalization, and end-to-end emission integration
in _classify_and_publish.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When stream.aisstream.io is unreachable (cert outage, server down — see
2026-05-20 and 2026-05-23 events) the ships layer goes empty. This adds
a slow REST fallback to data.aishub.net so the layer stays populated in
degraded mode.
Behavior:
* Opt-in via AISHUB_USERNAME (free registration at aishub.net/api).
Without the env var the fetcher is a no-op.
* Default poll cadence 20 min — well inside their free-tier limits, gives
ships time to move enough to look "alive". Configurable via
AISHUB_POLL_INTERVAL_MINUTES, clamped to [1, 360].
* Internal gate: skips the poll entirely when the WebSocket primary is
currently connected. Stomping fresh live data with 20-min-old REST
data would be worse than leaving it alone.
* Vessels merge into the shared _vessels dict with source="aishub" so
the existing UI / health tooling can attribute the provider.
* Live data wins races: if a WebSocket update for the same MMSI lands in
the last 1s, we don't overwrite with the slower REST record.
Scheduler job runs every AISHUB_POLL_INTERVAL_MINUTES minutes alongside
the existing ais_prune job in data_fetcher.py.
24 tests cover gating (no-username, primary-connected), response parsing
(success / error / empty / malformed / unexpected shape), record
normalization (sentinels, missing fields, range checks, AIS @ padding),
poll interval clamping, and end-to-end merge with live-data-wins.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
On 2026-05-23, stream.aisstream.io went fully offline (TCP timeouts on port
443). The backend kept respawning the node WebSocket proxy every few
seconds with nothing arriving. From the operator's POV the ships layer
silently went empty — no banner, no log surfacing, no way to tell whether
it was their config / network / viewport filter / upstream.
Backend:
* ais_proxy_status() now also returns:
- connected (bool): true when a vessel message arrived in last 60s
- last_msg_age_seconds (int | None)
- proxy_spawn_count (int): proxy respawns — sustained growth without
connected means upstream is dead
* /api/health escalates top status to "degraded" when AIS_API_KEY is set
but the proxy is currently disconnected. Existing degraded_tls signal
preserved.
Frontend:
* useAisUpstreamHealth hook polls /api/health every 30s, derives the
outage state. Defensively only reports outage once spawn_count > 0 so
operators who haven't opted in don't see the banner.
* AisUpstreamBanner component renders a dismissible amber notice
"Ship data temporarily unavailable — AISStream upstream is offline"
mounted on the main app shell.
7 backend tests pin the status-shape contract and the /api/health
escalation behavior in both with-key and without-key configurations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pre-fix, adsb.lol records (the primary source for most flights) carried
no source marker. OpenSky records got is_opensky: True and supplementals
got supplemental_source, so any UI inspecting source labels saw
OpenSky/airplanes.live records as explicitly tagged and adsb.lol records
as "unlabeled" — making it look like adsb.lol wasn't being used at all
even though it's the primary source.
Changes:
* _fetch_adsb_lol_regions stamps source="adsb.lol" on each aircraft
before returning, so the tag survives the OpenSky dedupe-by-hex merge.
* OpenSky records get source="OpenSky" (alongside is_opensky=True for
back-compat).
* military fetcher tags source on both adsb.lol and airplanes.live
records before they're merged, and propagates source into the
military_flights and uavs output dicts.
* _classify_and_publish promotes the explicit source field into the
published flight dict. Falls back to legacy supplemental_source if
source is absent. Final fallback "adsb.lol" preserves prior behavior
for any caller synthesizing records without going through a fetcher.
8 new tests cover the published-dict propagation, OpenSky tagging,
supplemental fallback, explicit-wins precedence, default behavior, the
adsb.lol regional fetcher tagging, and the military output dict.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three stacked filters meant the gps_jamming layer almost never lit up:
1. nac_p == 0 aircraft were dropped on the theory that "0 = old transponder."
That's only half right — modern Mode-S Enhanced Surveillance transponders
also fall back to nac_p=0 when they lose GPS lock entirely, which IS the
jamming signature we want to catch. Discarding them was discarding the
strongest signal. None (no field at all — typical for OpenSky-sourced
records) is still skipped because absence-of-data isn't evidence.
2. GPS_JAMMING_MIN_AIRCRAFT was 5 per 1°x1° cell. Jamming hotspots
(eastern Med, Russia/Ukraine border, Iran/Iraq) tend to have sparser
traffic because pilots avoid them. Lowered to 3.
3. GPS_JAMMING_MIN_RATIO was 0.30. Combined with the (preserved) -1 noise
cushion that made the effective bar high. Lowered to 0.20.
The 1-aircraft noise cushion is intact so a single quirky transponder
still can't flag a zone alone.
Also extracted the detector loop into a pure ``detect_gps_jamming_zones()``
function at module scope so it's testable in isolation (was previously
inlined inside ``_classify_and_publish``). The public signature accepts
threshold overrides for ad-hoc re-tuning without code edits.
16 new tests cover nac_p=0 inclusion, None-skip preservation, MIN_AIRCRAFT
lowering, MIN_RATIO lowering, noise cushion preservation, constant pinning,
override behavior, lon/lng key compatibility, and robustness to empty/None
inputs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The UAP sightings layer is sourced from a live scrape of nuforc.org with a
static Hugging Face CSV mirror (kcimc/NUFORC) as a fallback. The fallback
parsed every row, sorted by occurred-desc, and took the top 250 — with no
date cutoff. The HF mirror is a third-party snapshot that hasn't been
refreshed in years, so the "newest 250" rows it returns are from ~2022-23.
When the live path fails (Cloudflare 403, curl disabled on Windows, wdtNonce
regex stale, etc.) users see a map full of sightings from 3 years ago,
labeled as the "last 60 days" layer.
Changes:
* HF fallback now applies the same 60-day cutoff the live path uses. Rows
outside the window are dropped before take-top-N. If the mirror has
nothing inside the window the fallback returns [] (don't serve stale).
* When the HF mirror is fully stale a loud ERROR log fires with the count
of dropped rows so the operator can tell the mirror's the problem, not
a network issue.
* When BOTH live AND HF fallback produce 0 rows, fetch_uap_sightings now
trips assert_canary("uap_sightings", 0) so the health registry shows
the layer as broken instead of "fresh and empty for days."
* Scheduler moved from daily 12:00 UTC to weekly Mondays 12:00 UTC. The
layer is a rolling 60-day digest; refreshing once a week is enough
cadence for human-readable map exploration and keeps nuforc.org load
light.
6 new tests cover the cutoff filter, the doomsday-log path, the mixed-age
path, the both-paths-empty health failure, the positive fallback path, and
the scheduler cadence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
BigBodyCobain