feat(v1.9.0.0): gbrain-sync — cross-machine gstack memory (#1151)

* feat(gbrain-sync): queue primitives + writer shims

Adds bin/gstack-brain-enqueue (atomic append to sync queue) and
bin/gstack-jsonl-merge (git merge driver, ts-sort with SHA-256 fallback).
Wires one backgrounded enqueue call into learnings-log, timeline-log,
review-log, and developer-profile --migrate. question-log and
question-preferences stay local per Codex v2 decision.

gstack-config gains gbrain_sync_mode (off/artifacts-only/full) and
gbrain_sync_mode_prompted keys, plus GSTACK_HOME env alignment so
tests don't leak into real ~/.gstack/config.yaml.

* feat(gbrain-sync): --once drain + secret scan + push

bin/gstack-brain-sync is the core sync binary. Subcommands: --once
(drain queue, allowlist-filter, privacy-class-filter, secret-scan
staged diff, commit with template, push with fetch+merge retry),
--status, --skip-file <path>, --drop-queue --yes, --discover-new
(cursor-based detection of artifact writes that skip the shim).

Secret regex families: AWS keys, GitHub tokens (ghp_/gho_/ghu_/ghs_/
ghr_/github_pat_), OpenAI sk-, PEM blocks, JWTs, bearer-token-in-JSON.
On hit: unstage, preserve queue, print remediation hint (--skip-file
or edit), exit clean. No daemon — invoked by preamble at skill
boundaries.

* feat(gbrain-sync): init, restore, uninstall, consumer registry

bin/gstack-brain-init: idempotent first-run. git init ~/.gstack/,
.gitignore=*, canonical .brain-allowlist + .brain-privacy-map.json,
pre-commit secret-scan hook (defense-in-depth), merge driver registration
via git config, gh repo create --private OR arbitrary --remote <url>,
initial push, ~/.gstack-brain-remote.txt for new-machine discovery,
GBrain consumer registration via HTTP POST.

bin/gstack-brain-restore: safe new-machine bootstrap. Refuses clobber
of existing allowlisted files, clones to staging, rsync-copies tracked
files, re-registers merge drivers (required — not cloned from remote),
rehydrates consumers.json, prompts for per-consumer tokens.

bin/gstack-brain-uninstall: clean off-ramp. Removes .git + .brain-*
files + consumers.json + config keys. Preserves user data (learnings,
plans, retros, profile). Optional --delete-remote for GitHub repos.

bin/gstack-brain-consumer + bin/gstack-brain-reader (symlink alias):
registry management. Internal 'consumer' term; user-facing 'reader'
per DX review decision.

* feat(gbrain-sync): preamble block — privacy gate + boundary sync

scripts/resolvers/preamble/generate-brain-sync-block.ts emits bash that
runs at every skill invocation:
- Detects ~/.gstack-brain-remote.txt on machines without local .git
  and surfaces a restore-available hint (does NOT auto-run restore).
- Runs gstack-brain-sync --once at skill start to drain any pending
  writes (and at skill end via prose instruction).
- Once-per-day auto-pull (cached via .brain-last-pull) for append-only
  JSONL files.
- Emits BRAIN_SYNC: status line every skill run.

Also emits prose for the host LLM to fire the one-time privacy
stop-gate (full / artifacts-only / off) when gbrain is detected and
gbrain_sync_mode_prompted is false. Wired into preamble.ts composition.

* test(gbrain-sync): 27-test consolidated suite

test/brain-sync.test.ts covers:
- Config: validation, defaults, GSTACK_HOME env isolation
- Enqueue: no-op gates, skip list, concurrent atomicity, JSON escape
- JSONL merge driver: 3-way + ts-sort + SHA-256 fallback
- Init + sync: canonical file creation, merge driver registration,
  push-reject + fetch+merge retry path
- Init refuses different remote (idempotency)
- Cross-machine restore round-trip (machine A write → machine B sees)
- Secret scan across all 6 regex families (AWS, GH, OpenAI, PEM, JWT,
  bearer-JSON). --skip-file unblock remediation
- Uninstall removes sync config, preserves user data
- --discover-new idempotence via mtime+size cursor

Behaviors verified via integration smokes during implementation. Known
follow-up: bun-test 5s default timeout needs 30s wrapper for
spawnSync-heavy tests.

* docs(gbrain-sync): user guide + error lookup + README section

docs/gbrain-sync.md: setup walkthrough, privacy modes, cross-machine
workflow, secret protection, two-machine conflict handling, uninstall,
troubleshooting reference.

docs/gbrain-sync-errors.md: problem/cause/fix index for every
user-visible error. Patterned on Rust's error docs + Stripe's API
error reference.

README.md: 'Cross-machine memory with GBrain sync' section near the
top (discovery moment), plus docs-table entry.

* chore: bump version and changelog (v1.7.0.0)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: regenerate SKILL.md files for gbrain-sync preamble block

Re-runs bun run gen:skill-docs after adding generateBrainSyncBlock
to scripts/resolvers/preamble.ts in a2aa8a07. CI check-freshness
caught the drift. All 36 SKILL.md files regenerated with the new
skill-start bash block + privacy-gate prose + skill-end sync
instructions baked in.

* fix(test): session-awareness reads AskUserQuestion Format from a Tier 2+ SKILL.md

The test was reading ROOT/SKILL.md (browse skill, Tier 1) which never
contained '## AskUserQuestion Format' — that section is only emitted
for Tier 2+ skills by scripts/resolvers/preamble.ts. As a result the
agent was prompted with an empty format guide and only emitted
'RECOMMENDATION' intermittently, making the test flaky.

Pre-existing on main (same ROOT/SKILL.md shape there) — surfaced now
because the agent run didn't hit the RECOMMENDATION/recommend/option a
fallback strings in this particular attempt.

Fix: read from office-hours/SKILL.md (Tier 3, always has the section)
with a fallback that scans for the first top-level skill dir whose
SKILL.md contains the header. Future template moves won't break this
test again.

* chore: bump to v1.9.0.0 for gbrain-sync landing

Changes just the VERSION + package.json + CHANGELOG header (1.7.0.0 → 1.9.0.0
and date 2026-04-22 → 2026-04-23). No code changes. User call: land gbrain-sync
as a bigger-signal release above main's 1.6.4.0, skipping 1.8.0.0.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-23 17:54:54 -07:00
committed by GitHub
parent d75402bbd2
commit 9dbaf906cf
61 changed files with 6171 additions and 210 deletions
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env bash
# gstack-brain-consumer — manage the consumer (reader) registry.
#
# Consumer = a reader that ingests the gstack-brain git repo as a source of
# session memory. v1 primary consumer is GBrain; later versions can register
# Codex, OpenClaw, or third-party readers.
#
# NOTE ON NAMING: internally this helper uses "consumer" (correct data-model
# term). User-facing copy and the alias `gstack-brain-reader` use "reader"
# (matches user mental model: "what's reading my brain?").
#
# Usage:
# gstack-brain-consumer add <name> --ingest-url <url> --token <token>
# gstack-brain-consumer list
# gstack-brain-consumer remove <name>
# gstack-brain-consumer test <name>
#
# Env:
# GSTACK_HOME — override ~/.gstack
set -euo pipefail
GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}"
CONSUMERS_FILE="$GSTACK_HOME/consumers.json"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONFIG_BIN="$SCRIPT_DIR/gstack-config"
ensure_file() {
mkdir -p "$GSTACK_HOME"
if [ ! -f "$CONSUMERS_FILE" ]; then
echo '{"consumers": []}' > "$CONSUMERS_FILE"
fi
}
get_remote_url() {
git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo ""
}
sub_add() {
local name="" url="" token=""
local positional=""
while [ $# -gt 0 ]; do
case "$1" in
--ingest-url) url="$2"; shift 2 ;;
--token) token="$2"; shift 2 ;;
--) shift; break ;;
-*) echo "Unknown flag: $1" >&2; exit 1 ;;
*) positional="$1"; shift ;;
esac
done
name="$positional"
if [ -z "$name" ] || [ -z "$url" ]; then
echo "Usage: gstack-brain-consumer add <name> --ingest-url <url> [--token <token>]" >&2
exit 1
fi
ensure_file
# Upsert in consumers.json, store token in gstack-config under `<name>_token`.
python3 - "$CONSUMERS_FILE" "$name" "$url" <<'PYEOF'
import sys, json
path, name, url = sys.argv[1:4]
try:
with open(path) as f:
data = json.load(f)
except Exception:
data = {"consumers": []}
entry = {"name": name, "ingest_url": url, "status": "unknown", "token_ref": f"{name}_token"}
cs = data.setdefault("consumers", [])
for i, c in enumerate(cs):
if c.get("name") == name:
cs[i] = entry
break
else:
cs.append(entry)
with open(path, "w") as f:
json.dump(data, f, indent=2)
f.write("\n")
print(f"registered consumer: {name}")
PYEOF
if [ -n "$token" ]; then
"$CONFIG_BIN" set "${name}_token" "$token"
echo "token stored: gstack-config get ${name}_token to retrieve"
fi
# Attempt registration with remote (HTTP POST).
sub_test "$name"
}
sub_list() {
if [ ! -f "$CONSUMERS_FILE" ]; then
echo '{"consumers": []}'
return 0
fi
cat "$CONSUMERS_FILE"
}
sub_remove() {
local name="${1:-}"
if [ -z "$name" ]; then
echo "Usage: gstack-brain-consumer remove <name>" >&2
exit 1
fi
ensure_file
python3 - "$CONSUMERS_FILE" "$name" <<'PYEOF'
import sys, json
path, name = sys.argv[1:3]
try:
with open(path) as f:
data = json.load(f)
except Exception:
data = {"consumers": []}
before = len(data.get("consumers", []))
data["consumers"] = [c for c in data.get("consumers", []) if c.get("name") != name]
after = len(data["consumers"])
with open(path, "w") as f:
json.dump(data, f, indent=2)
f.write("\n")
print(f"removed: {before - after} entry(ies)")
PYEOF
}
sub_test() {
local name="${1:-}"
if [ -z "$name" ]; then
echo "Usage: gstack-brain-consumer test <name>" >&2
exit 1
fi
ensure_file
# Look up the consumer by name.
local info
info=$(python3 - "$CONSUMERS_FILE" "$name" <<'PYEOF'
import sys, json
path, name = sys.argv[1:3]
try:
with open(path) as f:
data = json.load(f)
except Exception:
data = {"consumers": []}
for c in data.get("consumers", []):
if c.get("name") == name:
print(c.get("ingest_url", ""))
sys.exit(0)
sys.exit(1)
PYEOF
) || { echo "No such consumer: $name" >&2; exit 1; }
local url="$info"
local token
token=$("$CONFIG_BIN" get "${name}_token" 2>/dev/null || echo "")
if [ -z "$url" ] || [ -z "$token" ]; then
echo "consumer '$name': url or token missing; cannot test"
return 0
fi
local repo_url
repo_url=$(get_remote_url)
echo "Testing $name at ${url%/}/ingest-repo ..."
local resp
resp=$(curl -sS -X POST "${url%/}/ingest-repo" \
-H "Authorization: Bearer $token" \
-H "Content-Type: application/json" \
--data "{\"repo_url\":\"$repo_url\"}" \
-w "\n%{http_code}" 2>&1 || echo -e "\ncurl-error")
local code
code=$(echo "$resp" | tail -1)
if [ "$code" = "200" ] || [ "$code" = "201" ] || [ "$code" = "204" ]; then
echo "ok (HTTP $code)"
# Update status in consumers.json.
python3 - "$CONSUMERS_FILE" "$name" "ok" <<'PYEOF'
import sys, json
path, name, status = sys.argv[1:4]
with open(path) as f: data = json.load(f)
for c in data.get("consumers", []):
if c.get("name") == name:
c["status"] = status
with open(path, "w") as f: json.dump(data, f, indent=2); f.write("\n")
PYEOF
else
echo "failed (HTTP $code)"
python3 - "$CONSUMERS_FILE" "$name" "error" <<'PYEOF'
import sys, json
path, name, status = sys.argv[1:4]
with open(path) as f: data = json.load(f)
for c in data.get("consumers", []):
if c.get("name") == name:
c["status"] = status
with open(path, "w") as f: json.dump(data, f, indent=2); f.write("\n")
PYEOF
fi
}
case "${1:-}" in
add) shift; sub_add "$@" ;;
list) sub_list ;;
remove) shift; sub_remove "$@" ;;
test) shift; sub_test "$@" ;;
--help|-h|"") sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//' ;;
*) echo "Unknown subcommand: $1" >&2; exit 1 ;;
esac
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
# gstack-brain-enqueue — atomically append a path to the GBrain sync queue.
#
# Usage:
# gstack-brain-enqueue <file-path>
#
# Called by writer scripts (gstack-learnings-log, gstack-timeline-log, etc.)
# after their local write. Fire-and-forget; failures are silent (never blocks
# the writer). Queue is drained by `gstack-brain-sync --once` invoked from the
# preamble at skill START and END boundaries.
#
# No-op when:
# - gbrain_sync_mode is off (the default)
# - ~/.gstack/.git doesn't exist (feature not initialized)
# - <file-path> matches a line in ~/.gstack/.brain-skip.txt
#
# Env:
# GSTACK_HOME — override ~/.gstack state directory (aligns with writers).
# Tests use GSTACK_HOME=/tmp/test-$$ for isolation.
#
# Concurrency: POSIX append is atomic up to PIPE_BUF (~4KB Linux, 512 BSD).
# Queue lines are ~200 bytes, safe under concurrent callers.
# No `-e` — writer shims rely on this never failing loudly.
set -uo pipefail
FILE="${1:-}"
[ -z "$FILE" ] && exit 0
GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}"
QUEUE="$GSTACK_HOME/.brain-queue.jsonl"
SKIP_FILE="$GSTACK_HOME/.brain-skip.txt"
# Fast exits: no git repo, no sync.
[ ! -d "$GSTACK_HOME/.git" ] && exit 0
# Check sync mode. off → silent no-op.
SCRIPT_DIR="$(cd "$(dirname "$0")" 2>/dev/null && pwd)"
MODE=$("$SCRIPT_DIR/gstack-config" get gbrain_sync_mode 2>/dev/null || echo off)
[ "$MODE" = "off" ] && exit 0
# User-maintained skip list (for secret-scan false positives).
if [ -f "$SKIP_FILE" ]; then
if grep -Fxq "$FILE" "$SKIP_FILE" 2>/dev/null; then
exit 0
fi
fi
# JSON-escape the file path (backslash + quotes only; paths shouldn't have other specials).
ESC_FILE=$(printf '%s' "$FILE" | sed 's/\\/\\\\/g; s/"/\\"/g')
TS=$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo "")
printf '{"file":"%s","ts":"%s"}\n' "$ESC_FILE" "$TS" >> "$QUEUE" 2>/dev/null
exit 0
+360
View File
@@ -0,0 +1,360 @@
#!/usr/bin/env bash
# gstack-brain-init — set up ~/.gstack/ as a git repo that syncs to GBrain.
#
# Usage:
# gstack-brain-init [--remote <url>]
#
# Interactive by default. Pass --remote to skip the remote prompt.
#
# Idempotent: safe to re-run. If ~/.gstack/.git already exists AND points at
# the same remote, reconfigures drivers/hooks/attributes without clobbering
# history. If it points at a DIFFERENT remote, refuses and suggests
# `gstack-brain-uninstall` first.
#
# What it does:
# 1. git init ~/.gstack/ (or verify existing repo points at the right remote)
# 2. Write .gitignore = "*" (ignore everything; allowlist is explicit)
# 3. Write .brain-allowlist (canonical paths to sync)
# 4. Write .brain-privacy-map.json (paths → privacy class)
# 5. Write .gitattributes (register JSONL + union merge drivers)
# 6. git config merge.jsonl-append.driver + merge.union.driver
# 7. Install .git/hooks/pre-commit (defense-in-depth secret scan)
# 8. Prompt for remote (default: gh repo create --private gstack-brain-$USER)
# 9. Initial commit + push
# 10. Write ~/.gstack-brain-remote.txt (URL-only, safe to share)
# 11. Register GBrain consumer (HTTP POST if GBRAIN_URL set; else defer)
#
# Env:
# GSTACK_HOME — override ~/.gstack
# GBRAIN_URL — GBrain ingest endpoint base URL (for consumer registration)
set -euo pipefail
GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONFIG_BIN="$SCRIPT_DIR/gstack-config"
REMOTE_FILE="$HOME/.gstack-brain-remote.txt"
CONSUMERS_FILE="$GSTACK_HOME/consumers.json"
REMOTE_URL=""
while [ $# -gt 0 ]; do
case "$1" in
--remote) REMOTE_URL="$2"; shift 2 ;;
--help|-h) sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "Unknown flag: $1" >&2; exit 1 ;;
esac
done
# ---- preconditions ----
mkdir -p "$GSTACK_HOME"
EXISTING_REMOTE=""
if [ -d "$GSTACK_HOME/.git" ]; then
EXISTING_REMOTE=$(git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "")
if [ -n "$EXISTING_REMOTE" ] && [ -n "$REMOTE_URL" ] && [ "$EXISTING_REMOTE" != "$REMOTE_URL" ]; then
cat >&2 <<EOF
gstack-brain-init: ~/.gstack/ is already a git repo pointing at:
$EXISTING_REMOTE
You asked to init with:
$REMOTE_URL
Refusing to overwrite. To switch remotes, first run:
gstack-brain-uninstall
(or edit the remote manually with: git -C ~/.gstack remote set-url origin <url>)
EOF
exit 1
fi
fi
# ---- choose the remote ----
if [ -z "$REMOTE_URL" ] && [ -n "$EXISTING_REMOTE" ]; then
REMOTE_URL="$EXISTING_REMOTE"
echo "Using existing remote: $REMOTE_URL"
fi
if [ -z "$REMOTE_URL" ]; then
# Interactive prompt. Default: gh repo create (if available).
echo "gstack-brain-init will create a private git repo that holds your"
echo "gstack session memory across machines and lets GBrain index it."
echo
if command -v gh >/dev/null 2>&1 && gh auth status >/dev/null 2>&1; then
DEFAULT_NAME="gstack-brain-${USER:-$(whoami)}"
echo "Default: gh will create a private repo named '$DEFAULT_NAME' under your account."
printf "Press Enter to accept, or paste a custom git URL: "
read -r REPLY || REPLY=""
if [ -z "$REPLY" ]; then
echo "Creating GitHub repo: $DEFAULT_NAME ..."
if ! gh repo create "$DEFAULT_NAME" --private --description "gstack session memory" --source "$GSTACK_HOME" 2>/dev/null; then
# Maybe the repo already exists; try to fetch its URL.
REMOTE_URL=$(gh repo view "$DEFAULT_NAME" --json sshUrl -q .sshUrl 2>/dev/null || echo "")
if [ -z "$REMOTE_URL" ]; then
echo "Failed to create or find '$DEFAULT_NAME'. Try --remote <url>." >&2
exit 1
fi
echo "Repo already exists; using $REMOTE_URL"
else
REMOTE_URL=$(gh repo view "$DEFAULT_NAME" --json sshUrl -q .sshUrl 2>/dev/null || echo "")
fi
else
REMOTE_URL="$REPLY"
fi
else
echo "(gh CLI not found or not authenticated; provide a git URL directly)"
printf "Paste a private git URL (e.g. git@github.com:you/gstack-brain.git): "
read -r REMOTE_URL || REMOTE_URL=""
if [ -z "$REMOTE_URL" ]; then
echo "No URL provided. Aborting." >&2
exit 1
fi
fi
fi
# ---- verify remote reachable ----
echo "Verifying remote connectivity: $REMOTE_URL"
if ! git ls-remote "$REMOTE_URL" >/dev/null 2>&1; then
cat >&2 <<EOF
Remote not reachable: $REMOTE_URL
This could mean:
- Wrong URL
- Not authenticated (GitHub: gh auth status; GitLab: glab auth status)
- Network issue
Fix and re-run gstack-brain-init.
EOF
exit 1
fi
# ---- git init ----
if [ ! -d "$GSTACK_HOME/.git" ]; then
git -C "$GSTACK_HOME" init -q -b main 2>/dev/null || git -C "$GSTACK_HOME" init -q
# If -b main wasn't supported, rename.
git -C "$GSTACK_HOME" branch -M main 2>/dev/null || true
fi
if [ -z "$(git -C "$GSTACK_HOME" remote 2>/dev/null)" ]; then
git -C "$GSTACK_HOME" remote add origin "$REMOTE_URL"
else
git -C "$GSTACK_HOME" remote set-url origin "$REMOTE_URL"
fi
# ---- write canonical files (idempotent) ----
cat > "$GSTACK_HOME/.gitignore" <<'EOF'
# gstack-brain sync: ignore-everything base. Paths are included explicitly via
# .brain-allowlist and `git add -f` from gstack-brain-sync. Do not edit.
*
EOF
cat > "$GSTACK_HOME/.brain-allowlist" <<'EOF'
# Canonical allowlist of paths that gstack-brain-sync will publish.
# One glob per line. Anything not matching stays local.
# Do not edit directly; managed by gstack-brain-init. User additions go below
# the marker and survive re-init.
projects/*/learnings.jsonl
projects/*/*-reviews.jsonl
projects/*/ceo-plans/*.md
projects/*/ceo-plans/*/*.md
projects/*/designs/*.md
projects/*/designs/*/*.md
projects/*/timeline.jsonl
retros/*.md
developer-profile.json
builder-journey.md
builder-profile.jsonl
# NOT synced (per Codex v2 review — machine-local UX state):
# projects/*/question-preferences.json (per-machine UX preferences)
# projects/*/question-log.jsonl (audit/derivation log stays with preferences)
# projects/*/question-events.jsonl (same)
# ---- USER ADDITIONS BELOW ---- (survives re-init; above is managed)
EOF
cat > "$GSTACK_HOME/.brain-privacy-map.json" <<'EOF'
[
{"pattern": "projects/*/learnings.jsonl", "class": "artifact"},
{"pattern": "projects/*/*-reviews.jsonl", "class": "artifact"},
{"pattern": "projects/*/ceo-plans/*.md", "class": "artifact"},
{"pattern": "projects/*/ceo-plans/*/*.md", "class": "artifact"},
{"pattern": "projects/*/designs/*.md", "class": "artifact"},
{"pattern": "projects/*/designs/*/*.md", "class": "artifact"},
{"pattern": "retros/*.md", "class": "artifact"},
{"pattern": "builder-journey.md", "class": "artifact"},
{"pattern": "projects/*/timeline.jsonl", "class": "behavioral"},
{"pattern": "developer-profile.json", "class": "behavioral"},
{"pattern": "builder-profile.jsonl", "class": "behavioral"}
]
EOF
cat > "$GSTACK_HOME/.gitattributes" <<'EOF'
# gstack-brain: merge drivers for cross-machine sync conflicts.
# Matching driver must be registered in local git config; gstack-brain-init
# and gstack-brain-restore run `git config merge.<name>.driver ...` after init.
*.jsonl merge=jsonl-append
retros/*.md merge=union
projects/*/designs/**/*.md merge=union
projects/*/ceo-plans/**/*.md merge=union
EOF
# ---- register merge drivers in local git config ----
git -C "$GSTACK_HOME" config merge.jsonl-append.driver "$SCRIPT_DIR/gstack-jsonl-merge %O %A %B"
git -C "$GSTACK_HOME" config merge.jsonl-append.name "gstack JSONL append-only merger"
git -C "$GSTACK_HOME" config merge.union.driver "cat %A %B > %A.merged && mv %A.merged %A"
git -C "$GSTACK_HOME" config merge.union.name "union concat"
# ---- install pre-commit hook (defense-in-depth) ----
HOOK="$GSTACK_HOME/.git/hooks/pre-commit"
mkdir -p "$(dirname "$HOOK")"
cat > "$HOOK" <<'HOOK_EOF'
#!/usr/bin/env bash
# gstack-brain pre-commit hook — secret-scan defense-in-depth.
# The primary scanner runs inside gstack-brain-sync BEFORE staging. This hook
# catches any manual `git commit` a user might accidentally run against the
# brain repo.
set -uo pipefail
python3 -c "
import sys, re, subprocess
try:
out = subprocess.check_output(['git', 'diff', '--cached'], stderr=subprocess.DEVNULL).decode('utf-8', 'replace')
except Exception:
sys.exit(0)
patterns = [
('aws-access-key', re.compile(r'AKIA[0-9A-Z]{16}')),
('github-token', re.compile(r'\b(gh[pousr]_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,})')),
('openai-key', re.compile(r'\bsk-[A-Za-z0-9_-]{20,}')),
('pem-block', re.compile(r'-----BEGIN [A-Z ]{3,}-----')),
('jwt', re.compile(r'\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b')),
('bearer-token-json',
re.compile(r'\"(authorization|api[_-]?key|apikey|token|secret|password)\"\s*:\s*\"[A-Za-z0-9_./+=-]{16,}\"',
re.IGNORECASE)),
]
for name, rx in patterns:
if rx.search(out):
sys.stderr.write(f'gstack-brain pre-commit: refusing commit — {name} detected in staged diff.\n')
sys.stderr.write('Either edit the offending file, or if intentional, run:\n')
sys.stderr.write(' gstack-brain-sync --skip-file <path> (to permanently exclude)\n')
sys.exit(1)
sys.exit(0)
"
HOOK_EOF
chmod +x "$HOOK"
# ---- initial commit (idempotent; skips if already committed) ----
cd "$GSTACK_HOME"
git add -f .gitignore .brain-allowlist .brain-privacy-map.json .gitattributes
# Only commit if the index has changes from HEAD (if there is a HEAD).
if git rev-parse HEAD >/dev/null 2>&1; then
if ! git diff --cached --quiet 2>/dev/null; then
git -c user.email="gstack@localhost" -c user.name="gstack-brain-init" \
commit -q -m "chore: gstack-brain-init (refresh sync config)"
fi
else
# First commit ever.
git -c user.email="gstack@localhost" -c user.name="gstack-brain-init" \
commit -q -m "chore: gstack-brain-init"
fi
# ---- initial push ----
if ! git push -q -u origin main 2>/dev/null; then
# Maybe the default branch is master, or the remote has existing content.
# Try to resolve: fetch + fast-forward merge + push.
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
if git fetch origin 2>/dev/null && git pull --ff-only origin "$CURRENT_BRANCH" 2>/dev/null; then
git push -q -u origin "$CURRENT_BRANCH" || {
echo "Push to $REMOTE_URL failed. The remote may have divergent content." >&2
echo "Try: cd ~/.gstack && git pull --rebase origin $CURRENT_BRANCH && git push origin $CURRENT_BRANCH" >&2
exit 1
}
else
# Couldn't fetch/merge; print what to do.
echo "Push to $REMOTE_URL failed and fetch/merge didn't help." >&2
echo "Manual recovery: cd ~/.gstack && git status, then push once conflicts are resolved." >&2
exit 1
fi
fi
# ---- write the remote-url helper file (outside ~/.gstack/, survives restore) ----
echo "$REMOTE_URL" > "$REMOTE_FILE"
chmod 600 "$REMOTE_FILE"
# ---- register GBrain consumer ----
mkdir -p "$GSTACK_HOME"
CONSUMER_STATUS="pending"
GBRAIN_URL_VAL="${GBRAIN_URL:-$("$CONFIG_BIN" get gbrain_url 2>/dev/null || echo "")}"
GBRAIN_TOKEN_VAL="${GBRAIN_TOKEN:-$("$CONFIG_BIN" get gbrain_token 2>/dev/null || echo "")}"
if [ -n "$GBRAIN_URL_VAL" ] && [ -n "$GBRAIN_TOKEN_VAL" ]; then
# Try the HTTP handoff.
HTTP_RESP=$(curl -sS -X POST "${GBRAIN_URL_VAL%/}/ingest-repo" \
-H "Authorization: Bearer $GBRAIN_TOKEN_VAL" \
-H "Content-Type: application/json" \
--data "{\"repo_url\":\"$REMOTE_URL\"}" \
-w "\n%{http_code}" 2>&1 || echo -e "\ncurl-error")
HTTP_CODE=$(echo "$HTTP_RESP" | tail -1)
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ] || [ "$HTTP_CODE" = "204" ]; then
CONSUMER_STATUS="ok"
echo "GBrain consumer registered: $GBRAIN_URL_VAL"
else
echo "GBrain ingest endpoint returned HTTP $HTTP_CODE; will retry on next skill run."
fi
elif [ -z "$GBRAIN_URL_VAL" ]; then
echo "(GBRAIN_URL not configured; skipping consumer registration. Set it with:"
echo " gstack-config set gbrain_url <url>"
echo " gstack-config set gbrain_token <token>"
echo " then run: gstack-brain-consumer add gbrain --ingest-url <url> --token <token>)"
fi
# Write consumers.json — the canonical registry. Tokens are NOT stored here;
# they stay in gstack-config (machine-local). This file IS synced so a new
# machine knows which consumers exist and can prompt for tokens.
python3 - "$CONSUMERS_FILE" "$GBRAIN_URL_VAL" "$CONSUMER_STATUS" <<'PYEOF'
import sys, json, os
path, url, status = sys.argv[1:4]
try:
with open(path) as f:
data = json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
data = {"consumers": []}
# Upsert GBrain entry.
entry = {"name": "gbrain", "ingest_url": url, "status": status, "token_ref": "gbrain_token"}
updated = False
for i, c in enumerate(data.get("consumers", [])):
if c.get("name") == "gbrain":
data["consumers"][i] = entry
updated = True
break
if not updated:
data.setdefault("consumers", []).append(entry)
with open(path, "w") as f:
json.dump(data, f, indent=2)
f.write("\n")
PYEOF
# Stage and commit consumers.json in the same session.
cd "$GSTACK_HOME"
git add -f consumers.json 2>/dev/null || true
if ! git diff --cached --quiet 2>/dev/null; then
git -c user.email="gstack@localhost" -c user.name="gstack-brain-init" \
commit -q -m "chore: register GBrain consumer"
git push -q origin HEAD 2>/dev/null || true
fi
# ---- done ----
cat <<EOF
gstack-brain-init complete.
Repo: $GSTACK_HOME (git)
Remote: $REMOTE_URL
Remote URL also saved at: $REMOTE_FILE
Sync happens automatically at the start and end of each skill (no daemon).
Check status anytime with:
gstack-brain-sync --status
To activate sync, the next skill you run will ask you one question about
privacy mode (sync everything / artifacts only / off).
New machine? On the other laptop, put a copy of:
$REMOTE_FILE
in that machine's home directory, then run: gstack-brain-restore
EOF
+1
View File
@@ -0,0 +1 @@
gstack-brain-consumer
+235
View File
@@ -0,0 +1,235 @@
#!/usr/bin/env bash
# gstack-brain-restore — bootstrap a new machine from an existing brain repo.
#
# Usage:
# gstack-brain-restore [<git-remote-url>]
#
# If no URL is given, reads from ~/.gstack-brain-remote.txt (written by
# gstack-brain-init on the original machine). Copy that file to the new
# machine before running this command.
#
# Safety gates (refuses with clear message):
# - ~/.gstack/.git already exists with a DIFFERENT remote
# - ~/.gstack/ contains non-allowlisted, non-gitignored user files
# that would be clobbered by restore
#
# What it does:
# 1. Clone the remote to a staging directory
# 2. Validate the repo is gstack-brain-shaped (.brain-allowlist, .gitattributes)
# 3. rsync-copy tracked files into ~/.gstack/ with skip-if-same-hash
# 4. Move staging's .git into ~/.gstack/.git
# 5. Register local git config merge drivers (they don't clone from remote)
# 6. Rehydrate consumers.json endpoints; prompt for tokens
#
# Env:
# GSTACK_HOME — override ~/.gstack
set -euo pipefail
GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONFIG_BIN="$SCRIPT_DIR/gstack-config"
REMOTE_FILE="$HOME/.gstack-brain-remote.txt"
REMOTE_URL="${1:-}"
if [ -z "$REMOTE_URL" ]; then
if [ -f "$REMOTE_FILE" ]; then
REMOTE_URL=$(head -1 "$REMOTE_FILE" | tr -d '[:space:]')
fi
fi
if [ -z "$REMOTE_URL" ]; then
cat >&2 <<EOF
gstack-brain-restore: no remote URL provided.
Provide one of:
gstack-brain-restore <git-url>
or put the URL in $REMOTE_FILE (copy from the original machine)
EOF
exit 1
fi
# ---- safety gates ----
if [ -d "$GSTACK_HOME/.git" ]; then
EXISTING_REMOTE=$(git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "")
if [ -n "$EXISTING_REMOTE" ] && [ "$EXISTING_REMOTE" != "$REMOTE_URL" ]; then
cat >&2 <<EOF
gstack-brain-restore: ~/.gstack/.git already points at:
$EXISTING_REMOTE
You asked to restore from:
$REMOTE_URL
Refusing to overwrite. Run 'gstack-brain-uninstall' first or pass a matching URL.
EOF
exit 1
fi
fi
# ---- clone to staging ----
STAGING=$(mktemp -d "${TMPDIR:-/tmp}/gstack-brain-restore.XXXXXX")
trap 'rm -rf "$STAGING" 2>/dev/null' EXIT
echo "Cloning $REMOTE_URL to staging..."
if ! git clone --quiet "$REMOTE_URL" "$STAGING/repo" 2>/dev/null; then
echo "Clone failed. Check:" >&2
echo " - URL is correct: $REMOTE_URL" >&2
echo " - Auth: gh auth status (github) / glab auth status (gitlab)" >&2
exit 1
fi
# ---- validate shape ----
if [ ! -f "$STAGING/repo/.brain-allowlist" ] || [ ! -f "$STAGING/repo/.gitattributes" ]; then
cat >&2 <<EOF
gstack-brain-restore: $REMOTE_URL does not look like a gstack-brain repo.
Missing: .brain-allowlist and/or .gitattributes
This command only works on repos created by gstack-brain-init.
EOF
exit 1
fi
# ---- validate target ~/.gstack/ has no non-gitignored user files ----
mkdir -p "$GSTACK_HOME"
if [ ! -d "$GSTACK_HOME/.git" ]; then
# No existing git → check if we'd clobber anything allowlisted.
# Read the new allowlist globs and see if any existing files would collide.
CLOBBER_RISK=$(python3 - "$GSTACK_HOME" "$STAGING/repo/.brain-allowlist" <<'PYEOF'
import sys, os, fnmatch
home, allowlist_path = sys.argv[1:3]
try:
with open(allowlist_path) as f:
globs = [l.strip() for l in f if l.strip() and not l.lstrip().startswith('#')]
except FileNotFoundError:
globs = []
risks = []
for root, dirs, files in os.walk(home):
dirs[:] = [d for d in dirs if d != '.git']
for name in files:
full = os.path.join(root, name)
rel = os.path.relpath(full, home)
for g in globs:
if fnmatch.fnmatchcase(rel, g):
risks.append(rel)
break
for r in risks[:5]:
print(r)
if len(risks) > 5:
print(f"...and {len(risks) - 5} more")
sys.exit(0 if not risks else 2)
PYEOF
) || true
if [ -n "$CLOBBER_RISK" ]; then
cat >&2 <<EOF
gstack-brain-restore: ~/.gstack/ has existing allowlisted files that would
be clobbered by restore:
$CLOBBER_RISK
Back these up first, or run this command on a machine with an empty
~/.gstack/. If these files are from an earlier gstack session on THIS
machine, you probably want to run gstack-brain-init instead (to create a
new brain repo with this machine's state).
EOF
exit 1
fi
fi
# ---- copy tracked files in ----
echo "Copying tracked files into ~/.gstack/ ..."
# Use git-ls-tree to get exact tracked file list (avoids staged/untracked files).
cd "$STAGING/repo"
git ls-tree -r --name-only HEAD | while IFS= read -r rel_path; do
src="$STAGING/repo/$rel_path"
dst="$GSTACK_HOME/$rel_path"
mkdir -p "$(dirname "$dst")"
# Skip if identical (content hash). Otherwise copy.
if [ -f "$dst" ] && cmp -s "$src" "$dst"; then
continue
fi
cp "$src" "$dst"
done
# ---- move .git into place ----
if [ -d "$GSTACK_HOME/.git" ]; then
# Existing .git with matching remote — just fetch + fast-forward.
git -C "$GSTACK_HOME" fetch origin >/dev/null 2>&1 || true
else
mv "$STAGING/repo/.git" "$GSTACK_HOME/.git"
fi
# ---- register merge drivers (local git config; don't survive clones) ----
git -C "$GSTACK_HOME" config merge.jsonl-append.driver "$SCRIPT_DIR/gstack-jsonl-merge %O %A %B"
git -C "$GSTACK_HOME" config merge.jsonl-append.name "gstack JSONL append-only merger"
git -C "$GSTACK_HOME" config merge.union.driver "cat %A %B > %A.merged && mv %A.merged %A"
git -C "$GSTACK_HOME" config merge.union.name "union concat"
# ---- install pre-commit hook (same as init) ----
HOOK="$GSTACK_HOME/.git/hooks/pre-commit"
mkdir -p "$(dirname "$HOOK")"
cat > "$HOOK" <<'HOOK_EOF'
#!/usr/bin/env bash
set -uo pipefail
python3 -c "
import sys, re, subprocess
try:
out = subprocess.check_output(['git', 'diff', '--cached'], stderr=subprocess.DEVNULL).decode('utf-8', 'replace')
except Exception:
sys.exit(0)
patterns = [
('aws-access-key', re.compile(r'AKIA[0-9A-Z]{16}')),
('github-token', re.compile(r'\b(gh[pousr]_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,})')),
('openai-key', re.compile(r'\bsk-[A-Za-z0-9_-]{20,}')),
('pem-block', re.compile(r'-----BEGIN [A-Z ]{3,}-----')),
('jwt', re.compile(r'\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b')),
('bearer-token-json',
re.compile(r'\"(authorization|api[_-]?key|apikey|token|secret|password)\"\s*:\s*\"[A-Za-z0-9_./+=-]{16,}\"',
re.IGNORECASE)),
]
for name, rx in patterns:
if rx.search(out):
sys.stderr.write(f'gstack-brain pre-commit: refusing commit — {name} detected.\n')
sys.exit(1)
sys.exit(0)
"
HOOK_EOF
chmod +x "$HOOK"
# ---- rehydrate consumers, prompt for tokens ----
if [ -f "$GSTACK_HOME/consumers.json" ]; then
echo ""
echo "Consumer registry restored. Tokens are machine-local and NOT synced."
echo "Run these for each consumer to re-enter tokens:"
python3 - "$GSTACK_HOME/consumers.json" <<'PYEOF'
import sys, json
try:
with open(sys.argv[1]) as f:
data = json.load(f)
except Exception:
sys.exit(0)
for c in data.get("consumers", []):
name = c.get("name", "")
token_ref = c.get("token_ref", f"{name}_token")
print(f" gstack-config set {token_ref} <your-token>")
PYEOF
fi
# ---- write remote helper file if missing ----
if [ ! -f "$REMOTE_FILE" ]; then
echo "$REMOTE_URL" > "$REMOTE_FILE"
chmod 600 "$REMOTE_FILE"
echo ""
echo "Wrote $REMOTE_FILE for future skill-run auto-detection."
fi
cat <<EOF
gstack-brain-restore complete.
Local: $GSTACK_HOME
Remote: $REMOTE_URL
Next skill run will ask about privacy mode (one-time question) and then
sync automatically at skill boundaries.
Status anytime: gstack-brain-sync --status
EOF
+447
View File
@@ -0,0 +1,447 @@
#!/usr/bin/env bash
# gstack-brain-sync — drain queue, commit allowlisted paths, push to remote.
#
# Usage:
# gstack-brain-sync --once drain queue, commit, push (default)
# gstack-brain-sync --status print sync health as JSON
# gstack-brain-sync --skip-file <p> add <p> to ~/.gstack/.brain-skip.txt
# gstack-brain-sync --drop-queue --yes clear queue without committing
# gstack-brain-sync --discover-new scan allowlist dirs, enqueue changed files
#
# Invoked by the preamble at skill START and END boundaries. No persistent
# daemon. Typical run <1s when queue empty; ~200-800ms with network push.
#
# Singleton enforcement: flock on ~/.gstack/.brain-sync.lock. Concurrent
# invocations queue and serialize.
#
# Env:
# GSTACK_HOME — override ~/.gstack (aligns with writers).
set -uo pipefail
GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}"
QUEUE="$GSTACK_HOME/.brain-queue.jsonl"
ALLOWLIST="$GSTACK_HOME/.brain-allowlist"
PRIVACY_MAP="$GSTACK_HOME/.brain-privacy-map.json"
SKIP_FILE="$GSTACK_HOME/.brain-skip.txt"
STATUS_FILE="$GSTACK_HOME/.brain-sync-status.json"
LAST_PUSH_FILE="$GSTACK_HOME/.brain-last-push"
LOCK_FILE="$GSTACK_HOME/.brain-sync.lock"
DISCOVER_CURSOR="$GSTACK_HOME/.brain-discover-cursor"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONFIG_BIN="$SCRIPT_DIR/gstack-config"
# Remote-specific hint for auth errors (branch on origin URL).
remote_auth_hint() {
local url
url=$(git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "")
case "$url" in
*github.com*|*@github.*) echo "run: gh auth status (and gh auth refresh if needed)" ;;
*gitlab*) echo "run: glab auth status" ;;
*) echo "check 'git remote -v' and your credentials" ;;
esac
}
write_status() {
# args: status_code message [extra_json_blob]
local code="$1"
local msg="$2"
local extra="${3:-{\}}"
local ts
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo "")
python3 - "$STATUS_FILE" "$code" "$msg" "$ts" "$extra" <<'PYEOF' 2>/dev/null || true
import json, sys
path, code, msg, ts, extra = sys.argv[1:6]
try:
extra_obj = json.loads(extra) if extra else {}
except Exception:
extra_obj = {}
data = {"status": code, "message": msg, "ts": ts, **extra_obj}
with open(path, "w") as f:
json.dump(data, f)
f.write("\n")
PYEOF
}
# Read config; return 0 if sync active, 1 otherwise.
sync_active() {
if [ ! -d "$GSTACK_HOME/.git" ]; then
return 1
fi
local mode
mode=$("$CONFIG_BIN" get gbrain_sync_mode 2>/dev/null || echo off)
[ "$mode" = "off" ] && return 1
return 0
}
# Secret regex families — stdin scan. Exits 0 clean, 1 if hit.
# Echoes the matching pattern family name on hit. Uses python3 -c (not
# heredoc) so sys.stdin stays available for the diff content.
secret_scan_stdin() {
python3 -c "
import sys, re
patterns = [
('aws-access-key', re.compile(r'AKIA[0-9A-Z]{16}')),
('github-token', re.compile(r'\\b(gh[pousr]_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,})')),
('openai-key', re.compile(r'\\bsk-[A-Za-z0-9_-]{20,}')),
('pem-block', re.compile(r'-----BEGIN [A-Z ]{3,}-----')),
('jwt', re.compile(r'\\beyJ[A-Za-z0-9_-]{10,}\\.[A-Za-z0-9_-]{10,}\\.[A-Za-z0-9_-]{10,}\\b')),
('bearer-token-json',
re.compile(r'\"(authorization|api[_-]?key|apikey|token|secret|password)\"\\s*:\\s*\"[A-Za-z0-9_./+=-]{16,}\"',
re.IGNORECASE)),
]
text = sys.stdin.read()
for name, rx in patterns:
m = rx.search(text)
if m:
snippet = m.group(0)
if len(snippet) > 30:
snippet = snippet[:30] + '...'
print(name + ':' + snippet)
sys.exit(1)
sys.exit(0)
"
}
# Compute matched allowlisted, privacy-filtered path set from queue.
# Output: newline-delimited relative paths that should be staged.
compute_paths_to_stage() {
local mode="$1"
python3 - "$GSTACK_HOME" "$QUEUE" "$ALLOWLIST" "$PRIVACY_MAP" "$SKIP_FILE" "$mode" <<'PYEOF'
import sys, json, os, fnmatch, glob
gstack_home, queue, allowlist_path, privacy_path, skip_path, mode = sys.argv[1:7]
def load_lines(path):
try:
with open(path) as f:
return [l.strip() for l in f if l.strip() and not l.lstrip().startswith("#")]
except FileNotFoundError:
return []
def load_privacy_map(path):
try:
with open(path) as f:
data = json.load(f)
# Expected: [{"pattern": "glob", "class": "artifact" | "behavioral"}]
return data if isinstance(data, list) else []
except (FileNotFoundError, json.JSONDecodeError):
return []
allowlist_globs = load_lines(allowlist_path)
privacy_map = load_privacy_map(privacy_path)
skip_lines = set(load_lines(skip_path))
# Read queue; collect unique file paths.
queue_paths = set()
try:
with open(queue) as f:
for line in f:
line = line.strip()
if not line:
continue
try:
obj = json.loads(line)
p = obj.get("file")
if isinstance(p, str):
queue_paths.add(p)
except json.JSONDecodeError:
continue
except FileNotFoundError:
pass
def path_matches_any(path, globs):
for pattern in globs:
if fnmatch.fnmatchcase(path, pattern):
return True
return False
def privacy_class(path, mapping):
for entry in mapping:
pat = entry.get("pattern")
if pat and fnmatch.fnmatchcase(path, pat):
return entry.get("class", "artifact")
# Default class when no pattern matches: artifact (safe default).
return "artifact"
# mode filter: 'off' → nothing; 'artifacts-only' → only artifact class;
# 'full' → both classes.
def mode_allows(cls, mode):
if mode == "off":
return False
if mode == "artifacts-only":
return cls == "artifact"
return True # full
final = []
for p in sorted(queue_paths):
if p in skip_lines:
continue
# Must be under GSTACK_HOME root. Reject absolute + reject ../ escape.
if p.startswith("/") or ".." in p.split("/"):
continue
# Must match at least one allowlist glob.
if not path_matches_any(p, allowlist_globs):
continue
# Must survive privacy mode filter.
cls = privacy_class(p, privacy_map)
if not mode_allows(cls, mode):
continue
# Must exist on disk — can't stage what isn't there.
if not os.path.exists(os.path.join(gstack_home, p)):
continue
final.append(p)
for p in final:
print(p)
PYEOF
}
subcmd_once() {
if ! sync_active; then
# Silent no-op when feature not initialized / disabled.
exit 0
fi
# Singleton lock via atomic mkdir. `flock(1)` isn't on macOS by default;
# `mkdir` is atomic on every POSIX filesystem. If another --once is already
# running, skip (don't wait) — the next skill boundary will catch up.
local lock_dir="${LOCK_FILE}.d"
if ! mkdir "$lock_dir" 2>/dev/null; then
# Is the lock stale? Check the pidfile inside. If process is dead, clear it.
if [ -f "$lock_dir/pid" ]; then
local lock_pid
lock_pid=$(cat "$lock_dir/pid" 2>/dev/null || echo "")
if [ -n "$lock_pid" ] && ! kill -0 "$lock_pid" 2>/dev/null; then
# Stale lock — clear and retry once.
rm -rf "$lock_dir" 2>/dev/null || true
if ! mkdir "$lock_dir" 2>/dev/null; then
exit 0
fi
else
# Lock is held by a live process.
exit 0
fi
else
# Lock dir without pidfile — treat as held; don't touch.
exit 0
fi
fi
echo "$$" > "$lock_dir/pid" 2>/dev/null || true
local mode
mode=$("$CONFIG_BIN" get gbrain_sync_mode 2>/dev/null || echo off)
local paths_file
paths_file=$(mktemp /tmp/brain-sync-paths.XXXXXX) || { rm -rf "$lock_dir" 2>/dev/null; write_status "error" "mktemp failed"; exit 1; }
# Single trap covers both: lock cleanup AND tempfile cleanup.
trap 'rm -f "$paths_file" 2>/dev/null; rm -rf "$lock_dir" 2>/dev/null || true' EXIT INT TERM
compute_paths_to_stage "$mode" > "$paths_file"
if [ ! -s "$paths_file" ]; then
# Nothing to stage. Clear any stale queue entries and exit.
: > "$QUEUE"
write_status "idle" "no allowlisted changes in queue"
exit 0
fi
# Stage with git add -f (forces past .gitignore=*) explicit paths only.
while IFS= read -r p; do
[ -z "$p" ] && continue
git -C "$GSTACK_HOME" add -f -- "$p" 2>/dev/null || true
done < "$paths_file"
# Secret-scan staged diff.
local scan_out
scan_out=$(git -C "$GSTACK_HOME" diff --cached 2>/dev/null | secret_scan_stdin || true)
if [ -n "$scan_out" ]; then
# Hit — unstage, preserve queue, write loud status.
git -C "$GSTACK_HOME" reset HEAD -- . >/dev/null 2>&1 || true
local hint
hint="secret pattern detected ($scan_out). Remediation: review the staged file, then run: gstack-brain-sync --skip-file <path> OR edit the content."
write_status "blocked" "$hint"
echo "BRAIN_SYNC: blocked: $scan_out" >&2
exit 0
fi
# Commit with template message.
local n ts
n=$(wc -l < "$paths_file" | tr -d ' ')
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
local msg="sync: $n file(s) | $ts"
git -C "$GSTACK_HOME" -c user.email="gstack@localhost" -c user.name="gstack-brain-sync" \
commit -q -m "$msg" 2>/dev/null || {
# Nothing to commit (e.g. all files already committed).
: > "$QUEUE"
write_status "idle" "queue drained but no new changes to commit"
exit 0
}
# Push. On reject, fetch + merge (merge driver handles JSONL) + retry once.
local push_err
push_err=$(git -C "$GSTACK_HOME" push origin HEAD 2>&1 >/dev/null) || {
# Check if this is an auth error first — no point retrying.
if echo "$push_err" | grep -qiE "auth|permission|403|401|forbidden"; then
local hint
hint=$(remote_auth_hint)
write_status "push_failed" "push failed: auth error. fix: $hint"
echo "BRAIN_SYNC: push failed: auth. fix: $hint" >&2
# Queue cleared because the commit exists locally; next push will send it.
: > "$QUEUE"
exit 0
fi
# Try a fetch-and-merge + retry.
if git -C "$GSTACK_HOME" fetch origin 2>/dev/null; then
local branch
branch=$(git -C "$GSTACK_HOME" rev-parse --abbrev-ref HEAD 2>/dev/null || echo main)
if git -C "$GSTACK_HOME" merge --no-edit "origin/$branch" >/dev/null 2>&1; then
if git -C "$GSTACK_HOME" push origin HEAD 2>/dev/null; then
: > "$QUEUE"
date -u +%Y-%m-%dT%H:%M:%SZ > "$LAST_PUSH_FILE"
write_status "ok" "pushed $n file(s) after rebase"
exit 0
fi
fi
fi
write_status "push_failed" "push failed: $(printf '%s' "$push_err" | head -1)"
: > "$QUEUE"
exit 0
}
# Success: clear queue, update last-push.
: > "$QUEUE"
date -u +%Y-%m-%dT%H:%M:%SZ > "$LAST_PUSH_FILE"
write_status "ok" "pushed $n file(s)"
exit 0
}
subcmd_status() {
if [ -f "$STATUS_FILE" ]; then
cat "$STATUS_FILE"
else
echo '{"status":"unknown","message":"no status file yet"}'
fi
# Supplemental info (not in status file).
local queue_depth=0
[ -f "$QUEUE" ] && queue_depth=$(wc -l < "$QUEUE" | tr -d ' ')
local last_push="never"
[ -f "$LAST_PUSH_FILE" ] && last_push=$(cat "$LAST_PUSH_FILE" 2>/dev/null || echo never)
local mode
mode=$("$CONFIG_BIN" get gbrain_sync_mode 2>/dev/null || echo off)
printf '{"queue_depth":%s,"last_push":"%s","mode":"%s"}\n' "$queue_depth" "$last_push" "$mode"
}
subcmd_skip_file() {
local path="${1:-}"
if [ -z "$path" ]; then
echo "Usage: gstack-brain-sync --skip-file <path>" >&2
exit 1
fi
mkdir -p "$GSTACK_HOME"
# Avoid duplicate entries.
if [ -f "$SKIP_FILE" ] && grep -Fxq "$path" "$SKIP_FILE"; then
echo "already in skip list: $path"
exit 0
fi
echo "$path" >> "$SKIP_FILE"
echo "added to skip list: $path"
echo "(future writers will not enqueue this path; existing queue entries ignored on next --once)"
}
subcmd_drop_queue() {
local force="${1:-}"
if [ "$force" != "--yes" ]; then
echo "Refusing: --drop-queue discards pending syncs. Pass --yes to confirm." >&2
exit 1
fi
if [ ! -f "$QUEUE" ]; then
echo "queue already empty"
exit 0
fi
local n
n=$(wc -l < "$QUEUE" | tr -d ' ')
: > "$QUEUE"
echo "dropped $n queue entries"
}
subcmd_discover_new() {
if ! sync_active; then
exit 0
fi
# Walk allowlist globs; enqueue any file where mtime+size differs from cursor.
python3 - "$GSTACK_HOME" "$ALLOWLIST" "$DISCOVER_CURSOR" "$SCRIPT_DIR/gstack-brain-enqueue" <<'PYEOF' 2>/dev/null || true
import sys, os, json, glob, fnmatch, subprocess, hashlib
gstack_home, allowlist_path, cursor_path, enqueue_bin = sys.argv[1:5]
def load_lines(path):
try:
with open(path) as f:
return [l.strip() for l in f if l.strip() and not l.lstrip().startswith("#")]
except FileNotFoundError:
return []
def load_cursor(path):
try:
with open(path) as f:
return json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
return {}
def save_cursor(path, data):
try:
with open(path, "w") as f:
json.dump(data, f)
except OSError:
pass
allowlist = load_lines(allowlist_path)
cursor = load_cursor(cursor_path)
new_cursor = dict(cursor)
# Walk all files under gstack_home, match against allowlist.
for root, dirs, files in os.walk(gstack_home):
# Skip .git and .brain-* state files.
if ".git" in root.split(os.sep):
continue
for name in files:
full = os.path.join(root, name)
rel = os.path.relpath(full, gstack_home)
if rel.startswith(".brain-"):
continue
matched = any(fnmatch.fnmatchcase(rel, pat) for pat in allowlist)
if not matched:
continue
try:
st = os.stat(full)
key = f"{int(st.st_mtime)}:{st.st_size}"
except OSError:
continue
prev = cursor.get(rel)
if prev != key:
# Enqueue via the shim (respects sync mode + skip list).
subprocess.run([enqueue_bin, rel], check=False)
new_cursor[rel] = key
save_cursor(cursor_path, new_cursor)
PYEOF
}
# -------- dispatch --------
case "${1:-}" in
--once|"") subcmd_once ;;
--status) subcmd_status ;;
--skip-file) shift; subcmd_skip_file "${1:-}" ;;
--drop-queue) shift; subcmd_drop_queue "${1:-}" ;;
--discover-new) subcmd_discover_new ;;
--help|-h)
sed -n '2,18p' "$0" | sed 's/^# \{0,1\}//'
;;
*)
echo "Unknown subcommand: $1" >&2
echo "Run: gstack-brain-sync --help" >&2
exit 1
;;
esac
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env bash
# gstack-brain-uninstall — clean off-ramp for gstack-brain sync.
#
# Usage:
# gstack-brain-uninstall [--yes] [--delete-remote]
#
# Removes the git layer from ~/.gstack/ and clears sync config. Your local
# gstack memory (learnings, timelines, etc.) is NOT touched — this is an
# uninstall-sync command, not a delete-data command.
#
# Flags:
# --yes Skip the confirmation prompt.
# --delete-remote Also delete the GitHub repo via `gh repo delete`
# (interactive unless --yes is also passed).
#
# What it removes (in ~/.gstack/):
# .git/ — the sync repo's git data
# .gitignore — canonical ignore-all marker
# .gitattributes — merge driver declarations
# .brain-allowlist — sync path list
# .brain-privacy-map.json — sync privacy classifier
# .brain-queue.jsonl — pending queue
# .brain-discover-cursor — discover-new cursor
# .brain-last-push — timestamp marker
# .brain-skip.txt — user-maintained skip list
# .brain-sync.lock.d/ — lock dir (if present)
# .brain-sync-status.json — health status
# consumers.json — consumer/reader registry
#
# What it clears (via gstack-config):
# gbrain_sync_mode → off
# gbrain_sync_mode_prompted → false (so user re-prompts on re-init)
#
# What it does NOT touch:
# Project data (projects/*, retros/*, developer-profile.json, etc.)
# Consumer tokens in gstack-config (<name>_token keys)
# ~/.gstack-brain-remote.txt in your home directory
# The actual remote git repo (unless --delete-remote)
set -euo pipefail
GSTACK_HOME="${GSTACK_HOME:-$HOME/.gstack}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONFIG_BIN="$SCRIPT_DIR/gstack-config"
REMOTE_FILE="$HOME/.gstack-brain-remote.txt"
ASSUME_YES=0
DELETE_REMOTE=0
while [ $# -gt 0 ]; do
case "$1" in
--yes|-y) ASSUME_YES=1; shift ;;
--delete-remote) DELETE_REMOTE=1; shift ;;
--help|-h) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "Unknown flag: $1" >&2; exit 1 ;;
esac
done
if [ ! -d "$GSTACK_HOME/.git" ]; then
echo "gstack-brain-uninstall: nothing to do (~/.gstack/.git doesn't exist)."
exit 0
fi
REMOTE_URL=$(git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "")
# ---- confirmation ----
if [ "$ASSUME_YES" != "1" ]; then
cat <<EOF
This will remove gstack-brain sync from this machine:
- Remove ~/.gstack/.git and sync config files
- Clear gbrain_sync_mode in gstack-config
- Remote: $REMOTE_URL will be $([ "$DELETE_REMOTE" = "1" ] && echo "DELETED" || echo "kept")
Local memory (learnings, plans, etc.) is NOT touched.
EOF
printf "Proceed? [y/N] "
read -r reply
case "$reply" in
y|Y|yes|Yes) ;;
*) echo "Aborted."; exit 0 ;;
esac
fi
# ---- delete remote if requested ----
if [ "$DELETE_REMOTE" = "1" ] && [ -n "$REMOTE_URL" ]; then
case "$REMOTE_URL" in
*github.com*|*@github*)
if command -v gh >/dev/null 2>&1; then
# Extract owner/repo from URL.
REPO_SLUG=$(echo "$REMOTE_URL" | sed -E 's#.*[:/]([^/:]+/[^/]+)(\.git)?$#\1#' | sed 's/\.git$//')
if [ -n "$REPO_SLUG" ]; then
echo "Deleting GitHub repo: $REPO_SLUG"
if [ "$ASSUME_YES" = "1" ]; then
gh repo delete "$REPO_SLUG" --yes 2>/dev/null || echo "gh repo delete failed; continuing local uninstall"
else
gh repo delete "$REPO_SLUG" 2>/dev/null || echo "gh repo delete failed; continuing local uninstall"
fi
fi
else
echo "--delete-remote requires the gh CLI. Skipping remote deletion."
fi
;;
*)
echo "--delete-remote only supports github.com remotes. Delete manually if needed: $REMOTE_URL"
;;
esac
fi
# ---- remove sync files ----
echo "Removing git layer and sync config files..."
rm -rf "$GSTACK_HOME/.git" 2>/dev/null || true
rm -f "$GSTACK_HOME/.gitignore" 2>/dev/null || true
rm -f "$GSTACK_HOME/.gitattributes" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-allowlist" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-privacy-map.json" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-queue.jsonl" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-discover-cursor" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-last-push" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-last-pull" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-skip.txt" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-sync-status.json" 2>/dev/null || true
rm -rf "$GSTACK_HOME/.brain-sync.lock.d" 2>/dev/null || true
rm -f "$GSTACK_HOME/consumers.json" 2>/dev/null || true
# ---- clear config keys ----
"$CONFIG_BIN" set gbrain_sync_mode off >/dev/null 2>&1 || true
"$CONFIG_BIN" set gbrain_sync_mode_prompted false >/dev/null 2>&1 || true
# ---- leave remote-helper file alone unless user asked to delete remote ----
if [ "$DELETE_REMOTE" = "1" ]; then
rm -f "$REMOTE_FILE" 2>/dev/null || true
else
if [ -f "$REMOTE_FILE" ]; then
echo "(keeping $REMOTE_FILE — remove manually if you want to forget the URL)"
fi
fi
cat <<EOF
gstack-brain uninstall complete.
Sync is off. ~/.gstack/ is a plain directory again.
Your project data, learnings, and profile are untouched.
To re-enable sync later: gstack-brain-init
EOF
+26 -4
View File
@@ -8,10 +8,11 @@
# gstack-config defaults — show just the defaults table
#
# Env overrides (for testing):
# GSTACK_STATE_DIR — override ~/.gstack state directory
# GSTACK_HOME — override ~/.gstack state directory (aligns with writer scripts)
# GSTACK_STATE_DIR — legacy alias for GSTACK_HOME (kept for backwards compat)
set -euo pipefail
STATE_DIR="${GSTACK_STATE_DIR:-$HOME/.gstack}"
STATE_DIR="${GSTACK_HOME:-${GSTACK_STATE_DIR:-$HOME/.gstack}}"
CONFIG_FILE="$STATE_DIR/config.yaml"
# Annotated header for new config files. Written once on first `set`.
@@ -59,6 +60,19 @@ CONFIG_HEADER='# gstack configuration — edit freely, changes take effect on ne
# # Unknown values default to "default" with a warning.
# # See docs/designs/PLAN_TUNING_V1.md for rationale.
#
# ─── GBrain sync (v1.7+) ─────────────────────────────────────────────
# gbrain_sync_mode: off # off | artifacts-only | full
# # off — no sync (default)
# # artifacts-only — sync plans/designs/retros/learnings only
# # (skip behavioral data: question-log,
# # developer-profile, timeline)
# # full — sync everything allowlisted
# # Set by the first-run privacy stop-gate. See docs/gbrain-sync.md.
#
# gbrain_sync_mode_prompted: false
# # Set to true once the privacy gate has asked the user.
# # Flip back to false to be re-prompted.
#
# ─── Advanced ────────────────────────────────────────────────────────
# codex_reviews: enabled # disabled = skip Codex adversarial reviews in /ship
# gstack_contributor: false # true = file field reports when gstack misbehaves
@@ -83,6 +97,8 @@ lookup_default() {
gstack_contributor) echo "false" ;;
skip_eng_review) echo "false" ;;
cross_project_learnings) echo "" ;; # intentionally empty → unset triggers first-time prompt
gbrain_sync_mode) echo "off" ;;
gbrain_sync_mode_prompted) echo "false" ;;
*) echo "" ;;
esac
}
@@ -114,6 +130,10 @@ case "${1:-}" in
echo "Warning: explain_level '$VALUE' not recognized. Valid values: default, terse. Using default." >&2
VALUE="default"
fi
if [ "$KEY" = "gbrain_sync_mode" ] && [ "$VALUE" != "off" ] && [ "$VALUE" != "artifacts-only" ] && [ "$VALUE" != "full" ]; then
echo "Warning: gbrain_sync_mode '$VALUE' not recognized. Valid values: off, artifacts-only, full. Using off." >&2
VALUE="off"
fi
mkdir -p "$STATE_DIR"
# Write annotated header on first creation
if [ ! -f "$CONFIG_FILE" ]; then
@@ -142,7 +162,8 @@ case "${1:-}" in
echo "# ─── Active values (including defaults for unset keys) ───"
for KEY in proactive routing_declined telemetry auto_upgrade update_check \
skill_prefix checkpoint_mode checkpoint_push codex_reviews \
gstack_contributor skip_eng_review; do
gstack_contributor skip_eng_review gbrain_sync_mode \
gbrain_sync_mode_prompted; do
VALUE=$(grep -E "^${KEY}:" "$CONFIG_FILE" 2>/dev/null | tail -1 | awk '{print $2}' | tr -d '[:space:]' || true)
SOURCE="default"
if [ -n "$VALUE" ]; then
@@ -157,7 +178,8 @@ case "${1:-}" in
echo "# gstack-config defaults"
for KEY in proactive routing_declined telemetry auto_upgrade update_check \
skill_prefix checkpoint_mode checkpoint_push codex_reviews \
gstack_contributor skip_eng_review; do
gstack_contributor skip_eng_review gbrain_sync_mode \
gbrain_sync_mode_prompted; do
printf ' %-24s %s\n' "$KEY:" "$(lookup_default "$KEY")"
done
;;
+4
View File
@@ -101,6 +101,10 @@ do_migrate() {
mv "$TMPOUT" "$PROFILE_FILE"
trap - EXIT
# gbrain-sync: enqueue the migrated file for cross-machine sync (no-op if off).
SCRIPT_DIR_E="$(cd "$(dirname "$0")" && pwd)"
"$SCRIPT_DIR_E/gstack-brain-enqueue" "developer-profile.json" 2>/dev/null &
# Archive the legacy file.
local TS
TS="$(date +%Y-%m-%d-%H%M%S)"
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env bash
# gstack-jsonl-merge — git merge driver for append-only JSONL files.
#
# Usage (called by git, not by users):
# gstack-jsonl-merge <base> <ours> <theirs>
#
# Registered in local git config by bin/gstack-brain-init and
# bin/gstack-brain-restore:
# git config merge.jsonl-append.driver \
# "$GSTACK_BIN/gstack-jsonl-merge %O %A %B"
#
# Behavior:
# Concatenate base + ours + theirs, dedup exact-duplicate lines, sort by
# ISO "ts" field when present, fall back to SHA-256 of the line for
# deterministic order. Write result to <ours> (the %A file per the git
# merge-driver contract).
#
# Two machines appending to the same JSONL file between pushes produces
# a same-line conflict at the file tail. This driver resolves it cleanly:
# both appends survive, ordered by wall-clock timestamp where available,
# content hash otherwise.
#
# Exit codes:
# 0 — merge succeeded, result written to <ours>
# 1 — error; git treats as conflict and stops the merge
set -uo pipefail
if [ "$#" -lt 3 ]; then
echo "gstack-jsonl-merge: expected 3 args (base ours theirs), got $#" >&2
exit 1
fi
BASE="$1"
OURS="$2"
THEIRS="$3"
TMP=$(mktemp /tmp/gstack-jsonl-merge.XXXXXX) || exit 1
trap 'rm -f "$TMP" 2>/dev/null || true' EXIT
python3 - "$BASE" "$OURS" "$THEIRS" > "$TMP" <<'PYEOF'
import sys, json, hashlib
paths = sys.argv[1:4] # base, ours, theirs
seen = {} # line content -> sort_key
for path in paths:
try:
with open(path, 'r', encoding='utf-8') as f:
for line in f:
line = line.rstrip('\n')
if not line:
continue
if line in seen:
continue
# Prefer ISO ts field for sort; fall back to SHA-256.
sort_key = None
try:
obj = json.loads(line)
ts = obj.get('ts') or obj.get('timestamp')
if isinstance(ts, str):
sort_key = (0, ts)
except (json.JSONDecodeError, ValueError, TypeError):
pass
if sort_key is None:
h = hashlib.sha256(line.encode('utf-8')).hexdigest()
sort_key = (1, h)
seen[line] = sort_key
except FileNotFoundError:
# Absent base / absent ours / absent theirs are all valid.
continue
except OSError:
# Permission / IO errors are fatal — caller sees non-zero exit.
sys.exit(1)
# Timestamp-ordered entries first (group 0), then hash-ordered (group 1).
for line, _ in sorted(seen.items(), key=lambda item: item[1]):
print(line)
PYEOF
_PYEXIT=$?
if [ "$_PYEXIT" != "0" ]; then
exit 1
fi
mv "$TMP" "$OURS" || exit 1
trap - EXIT
exit 0
+3
View File
@@ -84,3 +84,6 @@ if [ $? -ne 0 ] || [ -z "$VALIDATED" ]; then
fi
echo "$VALIDATED" >> "$GSTACK_HOME/projects/$SLUG/learnings.jsonl"
# gbrain-sync: enqueue for cross-machine sync (no-op if sync is off).
"$SCRIPT_DIR/gstack-brain-enqueue" "projects/$SLUG/learnings.jsonl" 2>/dev/null &
+4
View File
@@ -165,3 +165,7 @@ if [ $VALIDATE_RC -ne 0 ] || [ -z "$VALIDATED" ]; then
fi
echo "$VALIDATED" >> "$GSTACK_HOME/projects/$SLUG/question-log.jsonl"
# NOTE: question-log.jsonl is deliberately NOT enqueued for gbrain-sync.
# Per Codex v2 review, audit/derivation data stays local alongside the
# question-preferences.json it annotates.
+3
View File
@@ -16,3 +16,6 @@ if ! printf '%s' "$INPUT" | bun -e "JSON.parse(await Bun.stdin.text())" 2>/dev/n
fi
echo "$INPUT" >> "$GSTACK_HOME/projects/$SLUG/$BRANCH-reviews.jsonl"
# gbrain-sync: enqueue for cross-machine sync (no-op if sync is off).
"$SCRIPT_DIR/gstack-brain-enqueue" "projects/$SLUG/$BRANCH-reviews.jsonl" 2>/dev/null &
+7 -1
View File
@@ -2,7 +2,10 @@
# gstack-timeline-log — append a timeline event to the project timeline
# Usage: gstack-timeline-log '{"skill":"review","event":"started","branch":"main"}'
#
# Session timeline: local-only, never sent anywhere.
# Session timeline: local by default. If the user enables `gbrain_sync_mode`
# with the `full` (not `artifacts-only`) privacy tier — via the first-run
# stop-gate from `gstack-brain-init` or the preamble — timeline events are
# published to the user's private GBrain sync repo. See docs/gbrain-sync.md.
# Required fields: skill, event (started|completed).
# Optional: branch, outcome, duration_s, session, ts.
# Validation failure → skip silently (non-blocking).
@@ -32,3 +35,6 @@ if ! printf '%s' "$INPUT" | bun -e "const j=JSON.parse(await Bun.stdin.text());
fi
echo "$INPUT" >> "$GSTACK_HOME/projects/$SLUG/timeline.jsonl"
# gbrain-sync: enqueue for cross-machine sync (no-op if sync is off).
"$SCRIPT_DIR/gstack-brain-enqueue" "projects/$SLUG/timeline.jsonl" 2>/dev/null &