mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-08-28 05:10:25 +02:00
v0.9.0: in-app auto-updater, ship toggle split, stable entity IDs, performance fixes
New features: - In-app auto-updater with confirmation dialog, manual download fallback, restart polling, and protected file safety net - Ship layers split into 4 independent toggles (Military/Carriers, Cargo/Tankers, Civilian, Cruise/Passenger) with per-category counts - Stable entity IDs using MMSI/callsign instead of volatile array indices - Dismissible threat alert bubbles (session-scoped, survives data refresh) Performance: - GDELT title fetching is now non-blocking (background enrichment) - Removed duplicate startup fetch jobs - Docker healthcheck start_period 15s → 90s Bug fixes: - Removed fake intelligence assessment generator (OSINT-only policy) - Fixed carrier tracker GDELT 429/TypeError crash - Fixed ETag collision (full payload hash) - Added concurrent /api/refresh guard Contributors: @imqdcr (ship split + stable IDs), @csysp (dismissible alerts, PR #48) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Former-commit-id: a2c4c67da54345393f70a9b33b52e7e4fd6c049f
This commit is contained in:
anoracleofra-code
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
1eb2b21647
commit
fc9eff865e
@@ -0,0 +1,257 @@
|
||||
# ShadowBroker Release Protocol
|
||||
|
||||
> This document exists because API keys were leaked in release zips v0.5.0, v0.6.0, and briefly v0.8.0.
|
||||
> Follow this exactly. No shortcuts.
|
||||
|
||||
---
|
||||
|
||||
## Pre-Release Checklist
|
||||
|
||||
### 1. Bump the Version
|
||||
|
||||
- **`frontend/package.json`** — update `"version"` field
|
||||
- **`frontend/src/components/ChangelogModal.tsx`** — update `CURRENT_VERSION` and `STORAGE_KEY`
|
||||
- **Update `NEW_FEATURES`, `BUG_FIXES`, and `CONTRIBUTORS` arrays** in the changelog modal
|
||||
|
||||
### 2. Pull Remote Changes First
|
||||
|
||||
```bash
|
||||
git pull --rebase origin main
|
||||
```
|
||||
|
||||
If there are merge conflicts, resolve them carefully. **Do not blindly delete files during rebase** — this is how the API proxy route (`frontend/src/app/api/[...path]/route.ts`) was accidentally deleted and broke the entire app.
|
||||
|
||||
After resolving conflicts, verify critical files still exist:
|
||||
```bash
|
||||
ls frontend/src/app/api/\[...path\]/route.ts # API proxy — app is dead without this
|
||||
ls backend/main.py
|
||||
ls frontend/src/app/page.tsx
|
||||
```
|
||||
|
||||
### 3. Test Before Committing
|
||||
|
||||
```bash
|
||||
# Backend
|
||||
cd backend && python -c "import main; print('Backend OK')"
|
||||
|
||||
# Frontend
|
||||
cd frontend && npm run build
|
||||
```
|
||||
|
||||
If the backend fails with a missing module, install it:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Building the Release Zip
|
||||
|
||||
### The Command
|
||||
|
||||
Run from the project root (`live-risk-dashboard/`):
|
||||
|
||||
```bash
|
||||
7z a -tzip ../ShadowBroker_vX.Y.Z.zip \
|
||||
-xr!node_modules -xr!.next -xr!__pycache__ -xr!venv -xr!.git -xr!.git_backup \
|
||||
-xr!*.pyc -xr!*.db -xr!*.sqlite -xr!*.xlsx \
|
||||
-xr!.env -xr!.env.local -xr!.env.production -xr!.env.development \
|
||||
-xr!carrier_cache.json -xr!ais_cache.json \
|
||||
-xr!tmp_fast.json -xr!dump.json -xr!debug_fast.json \
|
||||
-xr!nyc_sample.json -xr!nyc_full.json \
|
||||
-xr!server_logs.txt -xr!server_logs2.txt -xr!xlsx_analysis.txt -xr!liveua_test.html \
|
||||
-xr!merged.txt -xr!recent_commits.txt \
|
||||
-xr!build_error.txt -xr!build_logs*.txt -xr!build_output.txt -xr!errors.txt \
|
||||
-xr!geocode_log.txt -xr!tsconfig.tsbuildinfo \
|
||||
-xr!ShadowBroker_v*.zip \
|
||||
.
|
||||
```
|
||||
|
||||
### Critical Exclusions (NEVER ship these)
|
||||
|
||||
| Pattern | Why |
|
||||
|---------|-----|
|
||||
| `.env` | **Contains real API keys** (OpenSky, AIS Stream) |
|
||||
| `.env.local` | **Contains real API keys** (TomTom, etc.) |
|
||||
| `.env.production` / `.env.development` | May contain secrets |
|
||||
| `carrier_cache.json` / `ais_cache.json` | Runtime cache, not source |
|
||||
| `node_modules/` / `__pycache__/` / `.next/` | Build artifacts |
|
||||
| `*.db` / `*.sqlite` / `*.xlsx` | Data files, not source |
|
||||
| `ShadowBroker_v*.zip` | Previous release zips sitting in the project dir |
|
||||
|
||||
### What SHOULD Be in the Zip
|
||||
|
||||
| File | Required |
|
||||
|------|----------|
|
||||
| `frontend/src/app/api/[...path]/route.ts` | **YES** — API proxy, app is dead without it |
|
||||
| `backend/.env.example` | YES — template for users |
|
||||
| `.env.example` | YES — template for users |
|
||||
| `backend/data/plane_alert_db.json` | YES — aircraft database |
|
||||
| `backend/data/datacenters*.json` | YES — data center layer |
|
||||
| `backend/data/tracked_names.json` | YES — tracked aircraft names |
|
||||
| `frontend/src/lib/airlines.json` | YES — airline codes |
|
||||
| `start.bat` / `start.sh` | YES — launcher scripts |
|
||||
|
||||
### Do NOT Use
|
||||
|
||||
- **`git archive`** — includes tracked junk, misses untracked essential files
|
||||
- **`Compress-Archive` (PowerShell)** — has lock file issues, no exclusion control
|
||||
- **Gemini's zip script** — included test files, debug outputs, `.env` with real keys, and 30+ unnecessary files
|
||||
|
||||
---
|
||||
|
||||
## Post-Build Audit (MANDATORY)
|
||||
|
||||
**Before uploading, always scan the zip for leaks:**
|
||||
|
||||
```bash
|
||||
# Check for .env files (should only show .env.example files)
|
||||
7z l ShadowBroker_vX.Y.Z.zip | grep -i "\.env" | grep "....A"
|
||||
|
||||
# Check for anything with "secret", "key", "token", "credential" in the filename
|
||||
7z l ShadowBroker_vX.Y.Z.zip | grep -iE "secret|api.key|credential|token" | grep "....A"
|
||||
|
||||
# Check the largest files (look for unexpected blobs)
|
||||
7z l ShadowBroker_vX.Y.Z.zip | grep "....A" | awk '{print $4, $NF}' | sort -rn | head -15
|
||||
|
||||
# Verify the API proxy route exists
|
||||
7z l ShadowBroker_vX.Y.Z.zip | grep "route.ts"
|
||||
```
|
||||
|
||||
**Expected results:**
|
||||
- `.env` files: ONLY `.env.example` and `next-env.d.ts`
|
||||
- No files with "secret"/"credential" in the name
|
||||
- Largest files: `plane_alert_db.json` (~4.6MB), `datacenters_geocoded.json` (~1.2MB), `airlines.json` (~800KB)
|
||||
- `route.ts` exists under `frontend/src/app/api/[...path]/`
|
||||
- **Total zip size: ~1.7MB** (as of v0.8.0). If it's 5MB+ something leaked.
|
||||
|
||||
---
|
||||
|
||||
## Commit, Tag, and Push
|
||||
|
||||
```bash
|
||||
# Stage specific files (NEVER use git add -A)
|
||||
git add <specific files>
|
||||
|
||||
# Commit
|
||||
git commit -m "v0.X.0: brief description of release"
|
||||
|
||||
# Tag
|
||||
git tag v0.X.0
|
||||
|
||||
# Push (pull first if remote has new commits)
|
||||
git pull --rebase origin main
|
||||
git push origin main --tags
|
||||
|
||||
# If the tag was created before rebase, re-tag on the new HEAD:
|
||||
git tag -f v0.X.0
|
||||
git push origin v0.X.0 --force
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Creating the GitHub Release
|
||||
|
||||
### Via GitHub API (when `gh` CLI is unavailable)
|
||||
|
||||
```python
|
||||
# 1. Create the release
|
||||
import urllib.request, json
|
||||
|
||||
body = {
|
||||
"tag_name": "v0.X.0",
|
||||
"name": "v0.X.0 — Title Here",
|
||||
"body": "Release notes here...",
|
||||
"draft": False,
|
||||
"prerelease": False
|
||||
}
|
||||
|
||||
# Write to a temp file to avoid JSON escaping hell in bash
|
||||
with open("release_body.json", "w") as f:
|
||||
json.dump(body, f)
|
||||
|
||||
# POST to GitHub API...
|
||||
|
||||
# 2. Upload the zip asset to the release
|
||||
# Use the upload_url from the release response
|
||||
```
|
||||
|
||||
### Via `gh` CLI (if installed)
|
||||
|
||||
```bash
|
||||
gh release create v0.X.0 ../ShadowBroker_v0.X.0.zip \
|
||||
--title "v0.X.0 — Title" \
|
||||
--notes-file RELEASE_NOTES.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Post-Release Verification
|
||||
|
||||
After uploading, download the release zip from GitHub and verify it:
|
||||
|
||||
```bash
|
||||
# Download what GitHub is actually serving
|
||||
curl -L -o /tmp/verify.zip "https://github.com/BigBodyCobain/Shadowbroker/releases/download/v0.X.0/ShadowBroker_v0.X.0.zip"
|
||||
|
||||
# Scan for leaks (same audit as above)
|
||||
7z l /tmp/verify.zip | grep -i "\.env" | grep "....A"
|
||||
|
||||
# Compare hash to your local copy
|
||||
md5sum /tmp/verify.zip ../ShadowBroker_v0.X.0.zip
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## If You Discover a Leak
|
||||
|
||||
### Immediate Actions
|
||||
|
||||
1. **Rebuild the zip** without the leaked file
|
||||
2. **Delete the old asset** from the GitHub release via API
|
||||
3. **Upload the clean zip** as a replacement
|
||||
4. **Rotate ALL leaked keys immediately:**
|
||||
- OpenSky: https://opensky-network.org/
|
||||
- AIS Stream: https://aisstream.io/
|
||||
- Any other keys found in the leak
|
||||
5. **Audit ALL other releases** — leaks tend to exist in multiple versions
|
||||
|
||||
### Audit All Releases Script
|
||||
|
||||
```python
|
||||
import urllib.request, json
|
||||
|
||||
TOKEN = "your_token"
|
||||
headers = {"Authorization": f"token {TOKEN}", "Accept": "application/vnd.github+json"}
|
||||
|
||||
# Get all releases
|
||||
req = urllib.request.Request(
|
||||
"https://api.github.com/repos/BigBodyCobain/Shadowbroker/releases",
|
||||
headers=headers
|
||||
)
|
||||
releases = json.loads(urllib.request.urlopen(req).read())
|
||||
|
||||
for r in releases:
|
||||
for asset in r.get("assets", []):
|
||||
# Download via API
|
||||
req2 = urllib.request.Request(
|
||||
asset["url"],
|
||||
headers={**headers, "Accept": "application/octet-stream"}
|
||||
)
|
||||
data = urllib.request.urlopen(req2).read()
|
||||
filename = f"/tmp/{r['tag_name']}.zip"
|
||||
with open(filename, "wb") as f:
|
||||
f.write(data)
|
||||
print(f"Downloaded {r['tag_name']}: {len(data)} bytes")
|
||||
# Then run 7z l on each to check for .env files
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Lessons Learned (v0.8.0 Incident)
|
||||
|
||||
1. **Rebasing can silently delete files.** After `git pull --rebase`, always verify that critical files like the API proxy route still exist.
|
||||
2. **The zip command must explicitly exclude `.env` and `.env.local`.** These files are not in `.gitignore` patterns that 7z understands — you must pass `-xr!.env -xr!.env.local` every time.
|
||||
3. **Always audit the zip before uploading.** A 10-second grep saves a key rotation.
|
||||
4. **Never trust another tool's zip output.** Gemini's zip included `.env` with real keys, 30+ test files, debug outputs, and sample JSON dumps.
|
||||
5. **2,000+ stars means 2,000+ potential eyes on every release.** Treat every zip as if it will be decompiled line by line.
|
||||
Reference in New Issue
Block a user