CalvinBackup/Shadowbroker
1
0
Fork 0
mirror of https://github.com/BigBodyCobain/Shadowbroker.git synced 2026-09-17 14:42:20 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
71550b4adf868de4800bdc1244ee0a41f9880c77
Shadowbroker/docs/OUTBOUND_DATA.md
T
C3B2W23andClaude Fable 5.1 8f169f1ecc Add runtime CARTO_API_KEY for basemap tiles 
CARTO now requires an API key for its basemap tiles; without one every
tile in the DEFAULT dark/light map carries an "API KEY REQUIRED"
watermark. The tile URLs were hardcoded in mapStyles.ts with no way to
supply a key, and because the frontend ships as a prebuilt image a
NEXT_PUBLIC_ variable would be baked in empty for every Docker user.

- New frontend-local route GET /api/basemap-config reads CARTO_API_KEY
  from the frontend container's environment at request time (same
  pattern as BACKEND_URL), so no image rebuild is needed.
- useBasemapConfig() fetches it once per page load; MaplibreViewer
  builds the MapLibre style from it via buildBasemapStyle(theme, key)
  and defers the map's first style load until the config has settled,
  avoiding a burst of unkeyed tile requests followed by a style swap.
- Tile URLs move to CARTO's documented rastertiles/ path with ?key=
  appended when configured. Unkeyed URLs serve byte-identical tiles to
  the old path, so deployments without a key behave exactly as before.
- CARTO_API_KEY wired through docker-compose.yml and documented in
  .env.example, README (data source table + frontend env table) and
  docs/OUTBOUND_DATA.md.
- Tests cover the route (unset / set / trimmed) and the style builder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 14:54:01 -07:00

121 lines
6.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Outbound data and third-party exposure
Shadowbroker is **self-hosted**: each install uses its own backend egress IP. This document is the operator-facing record for GitHub audit issues **#348–#366** (tg12): what contacts third parties, why, and how to opt out without losing unrelated features.
## Architecture
| Path | Who calls third parties |
|------|-------------------------|
| Map UI → `/api/*` → fetchers | **This install’s backend** |
| Basemap tiles / fonts | **Operator’s browser** (CARTO, demotiles.maplibre.org) |
| CCTV still/video proxy | **Backend** (Referer/Origin set per agency — see #349) |
---
## Issue disposition summary
| Issue | Status | Approach |
|-------|--------|----------|
| **#351** | Fixed | Region dossier via backend proxy |
| **#352** | Fixed | Geocode via `/api/geocode` only |
| **#360** | Fixed | Wikipedia/Wikidata via backend |
| **#362** | Fixed | `DEEPSTATE_MIRROR_COMMIT` optional pin |
| **#363** | Fixed | Madrid KML HTTPS-first |
| **#364** | Fixed | KiwiSDR HTTPS-first + validation |
| **#348** | Accepted + gated | Windows UI opt-in; env override; stealth documented |
| **#349** | Accepted + documented | Agency-required Referer on backend proxy only |
| **#350** | Mitigated | Callsign in UA **off by default**; opt-in `MESHTASTIC_SEND_CALLSIGN_HEADER=true` |
| **#354** | Accepted + documented | Default basemap CDN; optional self-hosted tiles |
| **#361** | Mitigated | UA is **install handle only** (`operator-…`), not shared `Shadowbroker/` token |
| **#366** | Accepted + documented | Honest per-install scrape; feature degrades if blocked |
---
## Per-install User-Agent (#361)
- **Code:** `backend/services/network_utils.py` — `outbound_user_agent()`, `OPERATOR_HANDLE`
- **Sent:** `operator-7f3a92` or `your-handle (purpose: nominatim)` — **no** shared app product name
- **Why:** Upstreams can rate-limit **one install**; a block on `operator-abc123` does not require blocking every Shadowbroker user
- **Override:** `SHADOWBROKER_USER_AGENT` replaces the entire string
- **Note:** The same handle across Wikipedia, Broadcastify, etc. still correlates **your** traffic across those sites — that is intentional per-install attribution, not anonymity
---
## LiveUAMap scraper (#348)
- **Layer:** `global_incidents` (LiveUAMap map pins; **GDELT** text still loads without LiveUAMap)
- **Code:** `backend/services/liveuamap_scraper.py` (Playwright + stealth for Turnstile)
- **Windows:** Scraper **off** until you enable **Global Incidents** and confirm the UI dialog → `backend/data/liveuamap_scraper_opt_in.json`
- **Linux/macOS:** Scraper runs when the layer is on (unless env forces off)
- **API:** `GET /api/liveuamap/scraper-status`, `POST /api/liveuamap/scraper-opt-in`
- **Env:** `SHADOWBROKER_ENABLE_LIVEUAMAP_SCRAPER=true|false` overrides UI on all platforms
- **Honesty:** Backend-only; no browser-direct LiveUAMap from end users. Stealth remains a functional tradeoff for Turnstile; disable layer or env if unacceptable
## UAP sightings (NUFORC map layer)
- **Window:** last **60 days** (`NUFORC_RECENT_DAYS`, ~2 months) from **live** nuforc.org
- **Cadence:** **Weekly** (Monday 12:00 UTC) per install; typical yield **~400–500** geocoded pins
- **Between weeks:** `backend/data/nuforc_recent_sightings.json` (7-day TTL) so restarts do not wipe the layer
- **Immediate pull:** admin `GET /api/refresh` on that install
- **Not used for map pins:** stale Hugging Face mirror (frozen ~2023) unless live is down and mirror happens to have in-window rows
---
## CCTV proxy Referer / Origin (#349)
- **Code:** `backend/routers/cctv.py`, `backend/main.py`
- **Behavior:** Backend proxies streams and sets `Referer` / `Origin` each agency expects (e.g. `https://511ga.org/cctv`, `https://informo.madrid.es/`)
- **Exposure:** Agency sees **backend IP**, not each viewer’s browser
- **Not removed:** Without these headers, most public DOT/city feeds return 403 — this is not end-user browser impersonation, it is the same headers a normal browser session would send to play the feed
---
## Meshtastic map callsign (#350)
- **Layer:** `sigint_meshtastic` must be active for `fetch_meshtastic_nodes()`
- **Default:** `MESHTASTIC_SEND_CALLSIGN_HEADER=false` — callsign **not** sent to `meshtastic.liamcottle.net` unless you set `true`
- **Optional:** `MESHTASTIC_OPERATOR_CALLSIGN` for local display; header only when explicitly enabled
---
## Basemap CDN (#354)
- **Code:** `frontend/src/components/map/styles/mapStyles.ts`, `frontend/public/map-style.json`
- **Hosts:** `*.basemaps.cartocdn.com`, `demotiles.maplibre.org`
- **Exposure:** **Browser** loads tiles (client IP + pan/zoom), not the backend
- **API key:** CARTO requires a key for basemap tiles. `CARTO_API_KEY` is set on the frontend container and served to the browser by the frontend-local route `/api/basemap-config` (read at request time, never proxied to the backend). The browser then sends it to `*.basemaps.cartocdn.com` as a `?key=` query parameter on every tile request. Unset it to keep the previous unkeyed behavior (watermarked tiles).
- **Mitigation:** Self-host raster tiles and point MapLibre `sources` at your tile server (operator choice; not required for core features)
---
## Broadcastify top feeds (#366)
- **Code:** `backend/services/radio_intercept.py`
- **Behavior:** Backend fetches `https://www.broadcastify.com/listen/top` with per-install handle UA; parses public HTML for feed metadata and CDN stream URLs
- **Exposure:** Your backend IP; 5-minute cache
- **If blocked:** Panel shows empty list — feature not removed from the app
- **Not:** Fake Chrome UA or cloudscraper bypass (removed in Round 7a)
---
## Ukraine frontline mirror (#362)
- **Layer:** `ukraine_frontline` / `frontlines`
- **Pin:** `DEEPSTATE_MIRROR_COMMIT`, optional `DEEPSTATE_MIRROR_REPO`
## Madrid CCTV (#363) / KiwiSDR (#364)
- Madrid: HTTPS-first KML catalog; image URLs unchanged
- KiwiSDR: HTTPS-first directory fetch; shape validation + bundled fallback
---
## Operator checklist
1. Set `OPERATOR_HANDLE` if you want a recognizable name on upstream logs.
2. Pin `DEEPSTATE_MIRROR_COMMIT` for reproducible frontlines (optional).
3. Windows: enable Global Incidents in UI only if you accept LiveUAMap server contact.
4. Set `SHADOWBROKER_ENABLE_LIVEUAMAP_SCRAPER=false` to forbid LiveUAMap entirely.
5. Set `MESHTASTIC_SEND_CALLSIGN_HEADER=true` only if you want callsign sent upstream.
6. Self-host map tiles if basemap CDN exposure matters.
Reference in New Issue View Git Blame Copy Permalink