mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-08-10 20:50:25 +02:00
00f02c26875fb5392d6d11ca94d8660a801b427d
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 BigBodyCobainandCursor
|
af9b3d08cc |
feat: Telegram OSINT map layer, Osiris intel ports, and maritime settings
Add Telegram OSINT with hourly incremental t.me scraping, metro geocoding separate from news centroids, threat-intercept popup UI with inline media, and HTML markers above alert boxes so pins stay clickable. Expose GFW_API_TOKEN in onboarding and Settings Maritime; harden GFW/CCTV/geo fetchers. Port Osiris- derived recon, SCM, entity graph, malware/cyber feeds, sanctions, and submarine cable layers with tests and documentation. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
BigBodyCobainandCursor
|
f08781bdc9 |
Route dossier, geocode, and Wikimedia through the backend (#351, #352, #360)
Proxy region dossier, Sentinel search, Wikipedia, and Wikidata via self-hosted APIs; remove LocateBar client-side Nominatim fallback; migrate legacy shadow- operator handles to operator- prefix. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
BigBodyCobain
|
b041b5e97c |
Fix #298: move Sentinel credentials from browser storage to backend .env
Reported by @tg12. Pre-fix, the Settings panel stored real third-party
Copernicus CDSE client_id + client_secret in browser localStorage /
sessionStorage via the privacy storage helper, and the proxy routes
required those values to come back in every tile/token request body.
Any same-origin script (XSS, malicious browser extension, dev-tools
HAR export) had read access to the credentials.
This change moves them server-side, behind the same .env-backed admin
flow every other third-party API key (OpenSky, AIS Stream, Finnhub,
Shodan, …) already uses.
Backend
-------
backend/services/api_settings.py
* Added SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET entries to
API_REGISTRY. The existing GET/PUT /api/settings/api-keys flow
(already require_local_operator-gated, .env-backed) now manages
them — no new route surface.
backend/routers/tools.py
* /api/sentinel/token and /api/sentinel/tile resolve credentials via
a new _resolve_sentinel_credentials() helper: body fields win for
back-compat with any legacy callers, otherwise the helper reads
SENTINEL_CLIENT_ID / SENTINEL_CLIENT_SECRET from os.environ.
* When neither source has a value, the route returns 400 with a
friendly pointer ("Set SENTINEL_CLIENT_ID and SENTINEL_CLIENT_SECRET
in the API Keys panel") instead of the curt "required" message.
The user's standing rule against hostile errors applies.
* Function bodies only — decorator lines untouched, so this PR does
not conflict with #303 (which adds Depends(require_local_operator)
to the same routes).
Frontend
--------
frontend/src/lib/sentinelHub.ts — rewritten
* Removed: getSentinelCredentials / setSentinelCredentials /
clearSentinelCredentials / getSentinelCredentialStorageMode.
These were the browser-storage read/write helpers; their existence
was the bug.
* Added: checkBackendSentinelStatus(), refreshSentinelStatus(),
getCachedSentinelStatus(), and a kept-for-back-compat
hasSentinelCredentials() shim. Status is sourced from
/api/settings/api-keys (the same endpoint the API Keys panel
already uses), so we don't add a new route just for this read.
* Added: migrateLegacySentinelBrowserKeys() — one-shot, idempotent
helper that clears sb_sentinel_client_id / _secret / _instance_id
from BOTH localStorage and sessionStorage. We deliberately do NOT
auto-POST those legacy browser values to the backend; doing so
would silently migrate a secret across a trust boundary without
operator consent. Operators re-enter once in the API Keys panel
and the legacy keys get wiped here.
* fetchSentinelTile and getSentinelToken no longer send client_id /
client_secret in the request body. The backend uses .env.
frontend/src/components/SettingsPanel.tsx
* Dropped sb_sentinel_client_id / _secret / _instance_id from
PRIVACY_SENSITIVE_BROWSER_KEYS — they're no longer written.
* SentinelTab rewritten: removed the inline Client ID / Client Secret
inputs + Save / Clear / Test buttons. Replaced with a status panel
that calls checkBackendSentinelStatus() on mount, a one-click
"Open API Keys Panel" button, and a migration banner that appears
only when migrateLegacySentinelBrowserKeys() actually cleared
something.
* Setup guide STEP 3 now points to the API Keys panel instead of
the local form.
frontend/src/app/page.tsx
* Added a one-time useEffect that fires checkBackendSentinelStatus()
on mount so the cached value (which the synchronous
hasSentinelCredentials() shim reads) is populated before
MaplibreViewer's tile-URL memo runs.
Tests
-----
backend/tests/test_sentinel_credentials_server_side.py (new)
* API_REGISTRY surface — sentinel_client_id / sentinel_client_secret
are registered with the right env_keys, ALLOWED_ENV_KEYS lets
/api/settings/api-keys PUT them.
* Resolution order — body wins, env is fallback, neither → 400 with
the friendly pointer message, and NO upstream HTTP call when
neither source has credentials (asserted via
MagicMock(side_effect=AssertionError)).
* /api/sentinel/tile same shape.
frontend/src/__tests__/utils/sentinelHub.test.ts (new)
* migrateLegacySentinelBrowserKeys clears localStorage AND
sessionStorage, reports what it cleared, idempotent.
* fetchSentinelTile + getSentinelToken POST WITHOUT client_id /
client_secret in the body (plants leaked credentials in browser
storage first to prove they are NOT picked up).
* checkBackendSentinelStatus parses /api/settings/api-keys correctly:
true only when both keys is_set, false on partial config or
network errors.
All 7 backend tests + 8 frontend tests pass locally. The
test_no_new_duplicate_routes guard and the api-settings test suite
still pass.
Credit: @tg12 for the audit report.
|
||
|
ShadowbrokerandGitHub
|
76750caa92 |
Round 7a: per-operator outbound attribution + GDELT GCS-direct fix (#292)
== Per-install operator handle for every third-party API call ==
Before this PR, every Shadowbroker install identified itself to
Wikipedia, Wikidata, Nominatim, GDELT, OpenMHz, Broadcastify,
weather.gov, NUFORC, Sentinel/Planetary Computer, TinyGS / CelesTrak,
Shodan, Finnhub, and others with a single project-wide User-Agent
("Shadowbroker/1.0" or "ShadowBroker-OSINT/1.0"). From the upstream's
perspective every install in the world looked like one giant scraper.
If one install misbehaved, the upstream's only recourse was to block
"Shadowbroker" as a whole.
PR #284 inadvertently doubled down on this in the frontend by
introducing a shared `WIKIMEDIA_API_USER_AGENT` constant. This PR
retrofits both backends to per-operator attribution.
New setting: OPERATOR_HANDLE (env var / settings UI / auto-gen)
New helper: network_utils.outbound_user_agent("purpose")
The handle is auto-generated as "operator-XXXXXX" on first call (the
"shadow-" prefix from earlier drafts was deliberately dropped — too
suspicious-looking for abuse-detection systems). Operators can
override via OPERATOR_HANDLE; the value is sanitized to lowercase
alphanumeric+dash+underscore and capped at 48 chars. Persisted to
backend/data/operator_handle.json so it survives container restarts.
Retrofitted call sites (every previously-MONSTER User-Agent):
- services/region_dossier.py (Wikipedia + Wikidata + Nominatim)
- services/geocode.py (Nominatim)
- services/sentinel_search.py (Microsoft Planetary Computer)
- services/feed_ingester.py (operator-curated RSS feeds)
- services/fetchers/earth_observation.py (weather.gov, NUFORC)
- services/fetchers/infrastructure.py
- services/fetchers/aircraft_database.py
- services/fetchers/route_database.py
- services/fetchers/trains.py
- services/fetchers/meshtastic_map.py
- services/shodan_connector.py
- services/unusual_whales_connector.py (Finnhub)
- services/tinygs_fetcher.py (CelesTrak + TinyGS)
- services/sar/sar_products_client.py
- services/geopolitics.py (GDELT)
- services/radio_intercept.py (Broadcastify + OpenMHz)
- routers/cctv.py + main.py (CCTV proxy)
- routers/ai_intel.py
- scripts/convert_power_plants.py (release-time data refresh)
Spoofed browser UAs removed (issues #289 / #290 / #291 — tg12 audit):
- cloudscraper-based Chrome impersonation against api.openmhz.com
-> replaced with honest requests + per-install UA
- Mozilla/5.0 spoofed UA on Broadcastify scrape
-> replaced with honest UA
- Mozilla/5.0 + fake first-party Referer on OpenMHz audio relay
-> replaced with honest UA
- cloudscraper dependency dropped from pyproject.toml + uv.lock
Frontend retrofit:
- new GET /api/settings/operator-handle endpoint (local-operator
gated) returns the install's handle
- frontend/src/lib/wikimediaClient.ts fetches the handle once on
first use, caches it for page lifetime, embeds it in the
Api-User-Agent for every Wikipedia / Wikidata browser-direct call
== GDELT GCS-direct fix ==
GDELT's data.gdeltproject.org is a CNAME to a Google Cloud Storage
bucket. GCS responds with the wildcard *.storage.googleapis.com cert
which legitimately does NOT cover the GDELT custom domain, so Python's
TLS verification correctly refuses the connection. Some networks
happen to route through a path where this works; many (notably Docker
Desktop's outbound NAT on local installs) do not. Verified on the
maintainer's local install: GDELT was unreachable; 1610 geopolitical
events / 48 export files were dropping silently.
Fix: services/geopolitics._gcs_direct_gdelt_url() rewrites any
data.gdeltproject.org URL to its GCS-direct equivalent
(storage.googleapis.com/data.gdeltproject.org/...) where the standard
GCS cert is genuinely valid. api.gdeltproject.org and every other host
are left untouched.
Confirmed live: backend log goes from
GDELT lastupdate failed: 500
to
Downloading 48 GDELT export files...
Downloaded 48/48 GDELT exports
GDELT parsed: 1610 conflict locations from 48 files
== Tests ==
backend/tests/test_per_operator_outbound_attribution.py (12 tests)
backend/tests/test_gdelt_gcs_direct_rewrite.py (6 tests)
backend/tests/test_region_dossier_wikimedia_ua.py (updated to
pin the helper + per-operator handle, not the old constant)
frontend/src/__tests__/utils/wikimediaClient.test.ts (rewritten
to mock /api/settings/operator-handle and assert per-operator UA)
Local: backend 114/114 security+audit+round7a suite green;
frontend 718/718 vitest suite green.
Credit: tg12 (external security audit, issues #289/#290/#291
relating to spoofed UAs); BigBodyCobain (operator-prefix call,
GDELT cloud-vs-local diagnosis).
|
||
|
ShadowbrokerandGitHub
|
0fee36e8f7 |
Fix #218/#219/#220: identify ShadowBroker on Wikipedia + Wikidata calls (#284)
Wikimedia's User-Agent policy asks API clients to identify themselves with a stable, contactable identifier so their operators can rate-limit or coordinate. Before this change, ShadowBroker was sending: - Backend (region_dossier.py): generic project default UA only; no Api-User-Agent. - Frontend (useRegionDossier.ts, WikiImage.tsx, NewsFeed.tsx): zero identifying header at all; three separate copy-pasted anonymous fetches with their own module-local caches. Three separate components doing the same broken thing meant policy fixes had to happen in three places, with no shared cache or kill switch. Fix (no UX change, zero hostility): == Backend == `backend/services/region_dossier.py` now sets explicit `User-Agent` + `Api-User-Agent` headers on every outbound Wikidata and Wikipedia request via a new `_WIKIMEDIA_REQUEST_HEADERS` constant. The identifier includes a contact path (issues page on the public GitHub repo). == Frontend == New shared helper `frontend/src/lib/wikimediaClient.ts`: - `fetchWikipediaSummary(title)` — single source of truth for Wikipedia REST summary lookups, with one shared LRU cache (in-flight requests deduplicated, 512-entry cap), `Api-User-Agent` on every fetch. - `fetchWikidataSparql(query)` — same shape for Wikidata SPARQL. - `WIKIMEDIA_API_USER_AGENT` — exported constant; one place to update if Wikimedia ever asks us to back off. Refactored three components to use the shared client: - `frontend/src/hooks/useRegionDossier.ts` — fetchLeader() and fetchLocalWikiSummary() now route through the shared helpers. - `frontend/src/components/WikiImage.tsx` — uses fetchWikipediaSummary, proper React state instead of module-mutation + forceUpdate trick. - `frontend/src/components/NewsFeed.tsx` — same shape. UX: byte-for-byte identical. Same thumbnails, same dossier content, same load behavior. The only observable difference is the outgoing request header. Note on #239 (route duplication): an audit-grade inventory shows 166 main.py routes are shadowed by router modules. That cleanup is too large to land safely in this PR; it will be staged as a separate ladder of small PRs grouped by router module. Tests: - `backend/tests/test_region_dossier_wikimedia_ua.py` — 3 tests asserting backend headers are present. - `frontend/src/__tests__/utils/wikimediaClient.test.ts` — 9 tests covering Api-User-Agent presence, shared cache, concurrent deduplication, disambiguation/HTTP-error/network-error fallthroughs, empty-input safety. Local: backend 76/76 security suite green, frontend 716/716 vitest suite green. Credit: tg12 (external security audit). |
||
|
anoracleofra-code
|
668ce16dc7 |
v0.9.6: InfoNet hashchain, Wormhole gate encryption, mesh reputation, 16 community contributors
Gate messages now propagate via the Infonet hashchain as encrypted blobs — every node syncs them through normal chain sync while only Gate members with MLS keys can decrypt. Added mesh reputation system, peer push workers, voluntary Wormhole opt-in for node participation, fork recovery, killwormhole scripts, obfuscated terminology, and hardened the self-updater to protect encryption keys and chain state during updates. New features: Shodan search, train tracking, Sentinel Hub imagery, 8 new intelligence layers, CCTV expansion to 11,000+ cameras across 6 countries, Mesh Terminal CLI, prediction markets, desktop-shell scaffold, and comprehensive mesh test suite (215 frontend + backend tests passing). Community contributors: @wa1id, @AlborzNazari, @adust09, @Xpirix, @imqdcr, @csysp, @suranyami, @chr0n1x, @johan-martensson, @singularfailure, @smithbh, @OrfeoTerkuci, @deuza, @tm-const, @Elhard1, @ttulttul |
||
|
anoracleofra-codeandClaude Opus 4.6
|
90c2e90e2c |
v0.9.5: The Voltron Update — modular architecture, stable IDs, parallelized boot
- Parallelized startup (60s → 15s) via ThreadPoolExecutor - Adaptive polling engine with ETag caching (no more bbox interrupts) - useCallback optimization for interpolation functions - Sliding LAYERS/INTEL edge panels replace bulky Record Panel - Modular fetcher architecture (flights, geo, infrastructure, financial, earth_observation) - Stable entity IDs for GDELT & News popups (PR #63, credit @csysp) - Admin auth (X-Admin-Key), rate limiting (slowapi), auto-updater - Docker Swarm secrets support, env_check.py validation - 85+ vitest tests, CI pipeline, geoJSON builder extraction - Server-side viewport bbox filtering reduces payloads 80%+ Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Former-commit-id: f2883150b5bc78ebc139d89cc966a76f7d7c0408 |