Reported by @tg12. Pre-fix, the Settings panel stored real third-party
Copernicus CDSE client_id + client_secret in browser localStorage /
sessionStorage via the privacy storage helper, and the proxy routes
required those values to come back in every tile/token request body.
Any same-origin script (XSS, malicious browser extension, dev-tools
HAR export) had read access to the credentials.
This change moves them server-side, behind the same .env-backed admin
flow every other third-party API key (OpenSky, AIS Stream, Finnhub,
Shodan, …) already uses.
Backend
-------
backend/services/api_settings.py
* Added SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET entries to
API_REGISTRY. The existing GET/PUT /api/settings/api-keys flow
(already require_local_operator-gated, .env-backed) now manages
them — no new route surface.
backend/routers/tools.py
* /api/sentinel/token and /api/sentinel/tile resolve credentials via
a new _resolve_sentinel_credentials() helper: body fields win for
back-compat with any legacy callers, otherwise the helper reads
SENTINEL_CLIENT_ID / SENTINEL_CLIENT_SECRET from os.environ.
* When neither source has a value, the route returns 400 with a
friendly pointer ("Set SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET
in the API Keys panel") instead of the curt "required" message.
The user's standing rule against hostile errors applies.
* Function bodies only — decorator lines untouched, so this PR does
not conflict with #303 (which adds Depends(require_local_operator)
to the same routes).
Frontend
--------
frontend/src/lib/sentinelHub.ts — rewritten
* Removed: getSentinelCredentials / setSentinelCredentials /
clearSentinelCredentials / getSentinelCredentialStorageMode.
These were the browser-storage read/write helpers; their existence
was the bug.
* Added: checkBackendSentinelStatus(), refreshSentinelStatus(),
getCachedSentinelStatus(), and a kept-for-back-compat
hasSentinelCredentials() shim. Status is sourced from
/api/settings/api-keys (the same endpoint the API Keys panel
already uses), so we don't add a new route just for this read.
* Added: migrateLegacySentinelBrowserKeys() — one-shot, idempotent
helper that clears sb_sentinel_client_id / _secret / _instance_id
from BOTH localStorage and sessionStorage. We deliberately do NOT
auto-POST those legacy browser values to the backend; doing so
would silently migrate a secret across a trust boundary without
operator consent. Operators re-enter once in the API Keys panel
and the legacy keys get wiped here.
* fetchSentinelTile and getSentinelToken no longer send client_id /
client_secret in the request body. The backend uses .env.
frontend/src/components/SettingsPanel.tsx
* Dropped sb_sentinel_client_id / _secret / _instance_id from
PRIVACY_SENSITIVE_BROWSER_KEYS — they're no longer written.
* SentinelTab rewritten: removed the inline Client ID / Client Secret
inputs + Save / Clear / Test buttons. Replaced with a status panel
that calls checkBackendSentinelStatus() on mount, a one-click
"Open API Keys Panel" button, and a migration banner that appears
only when migrateLegacySentinelBrowserKeys() actually cleared
something.
* Setup guide STEP 3 now points to the API Keys panel instead of
the local form.
frontend/src/app/page.tsx
* Added a one-time useEffect that fires checkBackendSentinelStatus()
on mount so the cached value (which the synchronous
hasSentinelCredentials() shim reads) is populated before
MaplibreViewer's tile-URL memo runs.
Tests
-----
backend/tests/test_sentinel_credentials_server_side.py (new)
* API_REGISTRY surface — sentinel_client_id / sentinel_client_secret
are registered with the right env_keys, ALLOWED_ENV_KEYS lets
/api/settings/api-keys PUT them.
* Resolution order — body wins, env is fallback, neither → 400 with
the friendly pointer message, and NO upstream HTTP call when
neither source has credentials (asserted via
MagicMock(side_effect=AssertionError)).
* /api/sentinel/tile same shape.
frontend/src/__tests__/utils/sentinelHub.test.ts (new)
* migrateLegacySentinelBrowserKeys clears localStorage AND
sessionStorage, reports what it cleared, idempotent.
* fetchSentinelTile + getSentinelToken POST WITHOUT client_id /
client_secret in the body (plants leaked credentials in browser
storage first to prove they are NOT picked up).
* checkBackendSentinelStatus parses /api/settings/api-keys correctly:
true only when both keys is_set, false on partial config or
network errors.
All 7 backend tests + 8 frontend tests pass locally. The
test_no_new_duplicate_routes guard and the api-settings test suite
still pass.
Credit: @tg12 for the audit report.
== Per-install operator handle for every third-party API call ==
Before this PR, every Shadowbroker install identified itself to
Wikipedia, Wikidata, Nominatim, GDELT, OpenMHz, Broadcastify,
weather.gov, NUFORC, Sentinel/Planetary Computer, TinyGS / CelesTrak,
Shodan, Finnhub, and others with a single project-wide User-Agent
("Shadowbroker/1.0" or "ShadowBroker-OSINT/1.0"). From the upstream's
perspective every install in the world looked like one giant scraper.
If one install misbehaved, the upstream's only recourse was to block
"Shadowbroker" as a whole.
PR #284 inadvertently doubled down on this in the frontend by
introducing a shared `WIKIMEDIA_API_USER_AGENT` constant. This PR
retrofits both backends to per-operator attribution.
New setting: OPERATOR_HANDLE (env var / settings UI / auto-gen)
New helper: network_utils.outbound_user_agent("purpose")
The handle is auto-generated as "operator-XXXXXX" on first call (the
"shadow-" prefix from earlier drafts was deliberately dropped — too
suspicious-looking for abuse-detection systems). Operators can
override via OPERATOR_HANDLE; the value is sanitized to lowercase
alphanumeric+dash+underscore and capped at 48 chars. Persisted to
backend/data/operator_handle.json so it survives container restarts.
Retrofitted call sites (every previously-MONSTER User-Agent):
- services/region_dossier.py (Wikipedia + Wikidata + Nominatim)
- services/geocode.py (Nominatim)
- services/sentinel_search.py (Microsoft Planetary Computer)
- services/feed_ingester.py (operator-curated RSS feeds)
- services/fetchers/earth_observation.py (weather.gov, NUFORC)
- services/fetchers/infrastructure.py
- services/fetchers/aircraft_database.py
- services/fetchers/route_database.py
- services/fetchers/trains.py
- services/fetchers/meshtastic_map.py
- services/shodan_connector.py
- services/unusual_whales_connector.py (Finnhub)
- services/tinygs_fetcher.py (CelesTrak + TinyGS)
- services/sar/sar_products_client.py
- services/geopolitics.py (GDELT)
- services/radio_intercept.py (Broadcastify + OpenMHz)
- routers/cctv.py + main.py (CCTV proxy)
- routers/ai_intel.py
- scripts/convert_power_plants.py (release-time data refresh)
Spoofed browser UAs removed (issues #289 / #290 / #291 — tg12 audit):
- cloudscraper-based Chrome impersonation against api.openmhz.com
-> replaced with honest requests + per-install UA
- Mozilla/5.0 spoofed UA on Broadcastify scrape
-> replaced with honest UA
- Mozilla/5.0 + fake first-party Referer on OpenMHz audio relay
-> replaced with honest UA
- cloudscraper dependency dropped from pyproject.toml + uv.lock
Frontend retrofit:
- new GET /api/settings/operator-handle endpoint (local-operator
gated) returns the install's handle
- frontend/src/lib/wikimediaClient.ts fetches the handle once on
first use, caches it for page lifetime, embeds it in the
Api-User-Agent for every Wikipedia / Wikidata browser-direct call
== GDELT GCS-direct fix ==
GDELT's data.gdeltproject.org is a CNAME to a Google Cloud Storage
bucket. GCS responds with the wildcard *.storage.googleapis.com cert
which legitimately does NOT cover the GDELT custom domain, so Python's
TLS verification correctly refuses the connection. Some networks
happen to route through a path where this works; many (notably Docker
Desktop's outbound NAT on local installs) do not. Verified on the
maintainer's local install: GDELT was unreachable; 1610 geopolitical
events / 48 export files were dropping silently.
Fix: services/geopolitics._gcs_direct_gdelt_url() rewrites any
data.gdeltproject.org URL to its GCS-direct equivalent
(storage.googleapis.com/data.gdeltproject.org/...) where the standard
GCS cert is genuinely valid. api.gdeltproject.org and every other host
are left untouched.
Confirmed live: backend log goes from
GDELT lastupdate failed: 500
to
Downloading 48 GDELT export files...
Downloaded 48/48 GDELT exports
GDELT parsed: 1610 conflict locations from 48 files
== Tests ==
backend/tests/test_per_operator_outbound_attribution.py (12 tests)
backend/tests/test_gdelt_gcs_direct_rewrite.py (6 tests)
backend/tests/test_region_dossier_wikimedia_ua.py (updated to
pin the helper + per-operator handle, not the old constant)
frontend/src/__tests__/utils/wikimediaClient.test.ts (rewritten
to mock /api/settings/operator-handle and assert per-operator UA)
Local: backend 114/114 security+audit+round7a suite green;
frontend 718/718 vitest suite green.
Credit: tg12 (external security audit, issues #289/#290/#291
relating to spoofed UAs); BigBodyCobain (operator-prefix call,
GDELT cloud-vs-local diagnosis).
Wikimedia's User-Agent policy asks API clients to identify themselves
with a stable, contactable identifier so their operators can rate-limit
or coordinate. Before this change, ShadowBroker was sending:
- Backend (region_dossier.py): generic project default UA only; no
Api-User-Agent.
- Frontend (useRegionDossier.ts, WikiImage.tsx, NewsFeed.tsx): zero
identifying header at all; three separate copy-pasted anonymous
fetches with their own module-local caches.
Three separate components doing the same broken thing meant policy
fixes had to happen in three places, with no shared cache or kill
switch.
Fix (no UX change, zero hostility):
== Backend ==
`backend/services/region_dossier.py` now sets explicit `User-Agent` +
`Api-User-Agent` headers on every outbound Wikidata and Wikipedia
request via a new `_WIKIMEDIA_REQUEST_HEADERS` constant. The identifier
includes a contact path (issues page on the public GitHub repo).
== Frontend ==
New shared helper `frontend/src/lib/wikimediaClient.ts`:
- `fetchWikipediaSummary(title)` — single source of truth for Wikipedia
REST summary lookups, with one shared LRU cache (in-flight requests
deduplicated, 512-entry cap), `Api-User-Agent` on every fetch.
- `fetchWikidataSparql(query)` — same shape for Wikidata SPARQL.
- `WIKIMEDIA_API_USER_AGENT` — exported constant; one place to update
if Wikimedia ever asks us to back off.
Refactored three components to use the shared client:
- `frontend/src/hooks/useRegionDossier.ts` — fetchLeader() and
fetchLocalWikiSummary() now route through the shared helpers.
- `frontend/src/components/WikiImage.tsx` — uses fetchWikipediaSummary,
proper React state instead of module-mutation + forceUpdate trick.
- `frontend/src/components/NewsFeed.tsx` — same shape.
UX: byte-for-byte identical. Same thumbnails, same dossier content,
same load behavior. The only observable difference is the outgoing
request header.
Note on #239 (route duplication): an audit-grade inventory shows 166
main.py routes are shadowed by router modules. That cleanup is too
large to land safely in this PR; it will be staged as a separate
ladder of small PRs grouped by router module.
Tests:
- `backend/tests/test_region_dossier_wikimedia_ua.py` — 3 tests
asserting backend headers are present.
- `frontend/src/__tests__/utils/wikimediaClient.test.ts` — 9 tests
covering Api-User-Agent presence, shared cache, concurrent
deduplication, disambiguation/HTTP-error/network-error fallthroughs,
empty-input safety.
Local: backend 76/76 security suite green, frontend 716/716 vitest
suite green.
Credit: tg12 (external security audit).
This test asserts that clicking "Remove" on a contact:
1. Surfaces a toast "Removed contact: <name>."
2. Drops the contact from the visible list
The Remove handler in MessagesView dispatches a tight cluster of React
state updates in one event tick:
removeContact(peerId)
locallySavedContactIdsRef.current.delete(peerId)
setContacts(...)
setComposeError('')
setComposeStatus(`Removed contact: ${displayNameForPeer(...)}.`)
Locally those updates settle in <100ms and the toast appears under any
findByText default. Under GitHub Actions runner load — especially the
shared Node.js workers on the "CI Gate / Frontend Tests & Build" step
— the reconcile-and-paint cycle has been measured at ~1.4s, which
exceeds the 1s default findByText timeout.
This is a load-sensitive timing flake, not a real bug — the toast
always renders eventually because the state update chain is purely
synchronous and the displayed text comes from the closure's pre-update
contacts (so the "Remove Me" name is always available when the toast
finally renders).
Historical flake hits in CI on this exact assertion:
PR #226 (zh-CN i18n landing, exposed by i18n parse error)
PR #237 (GitLab mirror parity)
PR #261 (post-#227 audit gap closures)
PR #262 (AIS SPKI pinning — failed post-merge Docker Publish,
skipping image publication for commit 729ea78)
The last one is the worst — a post-merge flake that blocked the
Docker image for an actual security fix from being published. The
subsequent merge of #263 cumulatively re-published the image, but
that's by accident, not by design.
Fix: replace the 1-second findByText with waitFor + 5s timeout +
50ms polling. The 5s ceiling still surfaces a real "toast never
renders" regression with a clear error; it just doesn't get racy
under CI load anymore.
Validation:
Local sequential 10x run of just this test → 10 passed, 0 failed
Full vitest suite → 707 passed, 72 files
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
External audit by @tg12 found three coupled vulnerabilities in the
Next.js admin-auth surface that together let any webpage the operator
visits trigger arbitrary privileged backend calls:
#249/#254 — Cross-origin webpages can have process.env.ADMIN_KEY
injected into their forwarded backend requests just by
issuing fetch('http://localhost:3000/api/wormhole/...')
from a browser tab the operator has open. Full
identity-takeover CSRF.
#255 — When ADMIN_KEY is unset on the server (the default in
.env.example), the admin session route fell through to
GET /api/settings/privacy-profile to "verify" the user-
supplied key. That endpoint is public; it always returns
200 for any X-Admin-Key value. So arbitrary attacker
keys minted full admin session cookies on default
installs.
Both fixes preserve every legitimate UX path. Origin-header gating is
transparent to browser tabs on the dashboard's own host, transparent
to Tauri/native shells (no Origin), and transparent to server-to-
server callers (no Origin). Only cross-origin browser fetches with a
foreign Origin lose the injection.
frontend/src/app/api/[...path]/route.ts
Adds isSameOriginOrNonBrowser() — checks the Origin header against
the request's own Host. Allow if no Origin (native/server-to-
server), allow if Origin host == Host host (same-origin), reject
otherwise. The admin-key injection now requires EITHER a valid
session cookie (auth) OR same-origin-or-non-browser (CSRF guard).
frontend/src/app/api/admin/session/route.ts
verifyAdminKey() simplified to local-only string comparison. When
ADMIN_KEY is configured, the supplied key must match exactly.
When ADMIN_KEY is unset, minting is refused entirely with a clear
message pointing the operator at the backend's auto-trust-loopback
behavior (SHADOWBROKER_TRUST_DOCKER_BRIDGE_LOCAL_OPERATOR=1, the
Docker default — local users keep working without a session).
The previous round-trip to /api/settings/privacy-profile was both
the source of the bug AND useless on its own merits (the endpoint
is public). Removing it makes the validation honest about what
it's checking.
Tests:
frontend/src/__tests__/proxy/proxyAuthBypassChain.test.ts (new, 12)
Cross-origin fetch to sensitive route → no admin-key injection
Cross-origin POST to sensitive route → no admin-key injection
Same-origin fetch → admin-key injection works
No-Origin (server-to-server / native) → admin-key injection works
Valid session cookie on cross-origin → cookie auth wins
Malformed Origin → conservative reject
Non-sensitive routes unaffected
Mint with ADMIN_KEY unset → refused (no fetch happens)
Empty key → 400
Mint with matching ADMIN_KEY → success
Mint with mismatched key → 403
Mint never round-trips to the backend (local-only validation)
frontend/src/__tests__/desktop/adminSessionBoundary.test.ts (updated)
Three tests updated to reflect the new local-only validation
contract. The previous tests asserted fetchMock.toHaveBeenCalled
which validated the now-removed (and broken) backend round-trip.
Full frontend suite: 707 passed, 72 files. No regressions.
Credit: @tg12 for the report. The cross-origin CSRF angle was
non-obvious — they specifically called out that the proxy's
admin-key injection was an open door for any page running in the
operator's browser, which is exactly the right framing.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
PR #226 landed the i18n infrastructure and Chinese (zh-CN) translations.
This follow-up adds the safeguards that make accepting community
translations sustainable without exposing the project to subtle
state-aligned framing in future translation PRs.
Changes:
frontend/src/i18n/index.tsx (renamed from .ts)
- Add LOCALES registry: a single source of truth for available
languages and their NATIVE display names ("English", "中文 (简体)").
Adding a new language is now a one-entry change here plus a
JSON file.
- Add isLocale() guard so an unknown value in localStorage falls
through to navigator.language detection instead of corrupting
state.
- File renamed to .tsx because it contains JSX. Next.js tolerated
JSX in .ts but Vite/Oxc (used by vitest) does not.
frontend/src/components/SettingsPanel.tsx
Add a UI language picker to the Settings header — a small <select>
populated from LOCALES. Users no longer need the dev console to
switch languages. Locale change remains 100% client-side
(localStorage), no network call, no telemetry.
CONTRIBUTING.md (new)
Documents the translation-neutrality requirement that applies
symmetrically to all source countries:
- Translations must be technically faithful to the English source.
- Substitutions aligned with state propaganda from ANY country
(PRC, Russia, US, EU, etc.) will be rejected.
- The test is: "would a translator working strictly from the
English source produce this rendering?"
Also explains how translation PRs are reviewed and how to add
a new language.
.github/CODEOWNERS (new)
Auto-requests maintainer review on:
- /frontend/src/i18n/ (translation safety)
- /backend/auth.py, /backend/routers/wormhole.py,
/backend/services/mesh/, /backend/services/fetchers/
(the same paths recent security audits flagged as sensitive)
- /.github/workflows/, /.gitlab-ci.yml, /docker-compose*.yml,
/helm/ (build/deploy)
- /CONTRIBUTING.md, /.github/CODEOWNERS (policy itself)
frontend/src/__tests__/i18n/i18nProvider.test.tsx (new, 8 tests)
Locks in the i18n contract:
- LOCALES has both en and zh-CN with non-empty native labels
- Default English when navigator is English
- Auto-detect zh-CN when navigator language starts with "zh"
- localStorage preference overrides auto-detect
- setLocale persists to localStorage
- Unknown stored locale falls back to auto-detect
- Renders a real zh-CN translation (catches large-scale
translation removal in future PRs)
- Missing key falls back to the key itself
Note: i18n/index.tsx, the language toggle UI, the translation
policy, and the test suite together form a defense-in-depth setup.
The structural safety guarantee (no network calls, static JSON
bundled at build) is intact; this PR makes the social contract
around translations explicit and enforceable via branch
protection on CODEOWNERS-marked paths.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Each alert toast had a 5-second auto-dismiss timer that fired even
while the user was reading the card. This adds pause-on-hover: the
dismiss timer stops while the mouse is over a toast and restarts (full
lifetime) on mouse leave. The progress bar animation pauses with it,
so the visual matches the actual remaining time.
All other behavior is preserved: same cyber/mono styling, same spring
slide-in, same risk-color border + glow, same warning icon, same
LVL X/10 readout, same title/source layout, same click-to-fly + dismiss
on body click, same × dismiss button.
Implementation notes:
- Extract a ToastCard sub-component so each card can own its own
paused state (useState can't be array-indexed in the parent).
- Move the auto-dismiss timer out of useAlertToasts.ts and into
ToastCard. The hook previously scheduled the dismiss itself, which
meant the UI couldn't pause it — only the component knows whether
the user is interacting.
- Add tests covering: title/source/severity render, auto-dismiss
fires at 5s, hover pauses indefinitely, mouse-leave restarts the
full lifetime, × dismisses without flying, body-click flies +
dismisses.
This implements the genuine UX improvement that PR #234 was reaching
for, without #234's broken syntax, missing-field bug, duplicate
timer logic, or design regression.
Refs: #234
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Allow local-operator DM invite import without requiring a full admin session.
Prioritize bundled/bootstrap seed peers and shorten stale seed cooldowns for faster Infonet recovery.
Replace raw DM invite dumps with copyable signed-address controls, contact request handling, and safer sealed-send behavior while the private delivery route connects.
Ship the v0.9.79 runtime refresh with transport lane isolation, Infonet secure-message address management, MeshChat MQTT controls, selected asset trail behavior, telemetry panel refinements, onboarding updates, and desktop/package metadata alignment.
Also ignore local graphify work products so analysis folders do not leak into future commits.
Render the app shell dynamically so Next can attach per-request CSP nonces to its production scripts, preventing Docker from serving a static shell that cannot hydrate. Also gives the first-contact warmup test enough time in CI.
Gate messages now propagate via the Infonet hashchain as encrypted blobs — every node syncs them
through normal chain sync while only Gate members with MLS keys can decrypt. Added mesh reputation
system, peer push workers, voluntary Wormhole opt-in for node participation, fork recovery,
killwormhole scripts, obfuscated terminology, and hardened the self-updater to protect encryption
keys and chain state during updates.
New features: Shodan search, train tracking, Sentinel Hub imagery, 8 new intelligence layers,
CCTV expansion to 11,000+ cameras across 6 countries, Mesh Terminal CLI, prediction markets,
desktop-shell scaffold, and comprehensive mesh test suite (215 frontend + backend tests passing).
Community contributors: @wa1id, @AlborzNazari, @adust09, @Xpirix, @imqdcr, @csysp, @suranyami,
@chr0n1x, @johan-martensson, @singularfailure, @smithbh, @OrfeoTerkuci, @deuza, @tm-const,
@Elhard1, @ttulttul
- Add gsdf/msdf/asdf to known_branches in test_branch_values_are_known
- Add test_includes_jsdf_bases for Yonaguni, Naha, Kure
- Add test_colocated_bases_have_separate_entries for Misawa
- Add buildMilitaryBasesGeoJSON tests with ASDF branch validation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BigBodyCobain