Files
gstack/bin/gstack-brain-restore
T
Garry Tanandcapy-ai[bot] 90fded4dd0 Brain sync scans binary files and split authorization values for secrets (GST-2292) (#4276)
* Brain sync scans binary files and split authorization values for secrets (GST-2292)

Git showed a file with a NUL byte or UTF-16 text as "Binary files differ",
so the push-time secret scan never saw an AWS key inside it. The sync and the
pre-commit hook now diff with --text --no-textconv, the scanner also reads
NUL-bearing lines as UTF-16 and with NULs removed, and a section git still
did not diff as text is held as binary-not-scanned.

A Bearer value added under an unchanged "authorization": line was missed
because the + marker broke the match; each hunk's new text is now scanned as
one block too. The docs say plainly that an already-committed secret stays
held after deletion and only --skip-file clears it.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Secret scan catches split auth values in UTF-16 files and under diff.context=0

Review-Finding: p1/red-team#1 reader=red-team
Review-Finding: p1/red-team#2 reader=red-team
Review-Finding: p1/red-team#3 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Sync docs say a secret already committed in ~/.gstack stays held until you skip the file

Approved docs-audit repair: approval a-029ea2b533be (GStack Minder GSTA-36, lane brief item 3).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Secret scan keeps the plain git diff as one reading beside the text reading

The sync and the pre-commit hooks had replaced git's plain staged diff with
a --text --no-textconv -U3 diff. That lost two things the old scan held: an
unchanged line that the user's diff.context showed (5 lines away from an
edit), and a secret that only a configured textconv driver decodes. The
scanner now takes both diffs and holds a file when either reading matches;
binary-not-scanned applies to the text reading only.

Review-Finding: sol-audit-201c6a0#1 reader=cross-family-audit
Review-Finding: sol-audit-201c6a0#2 reader=cross-family-audit
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Secret scan decodes JSON escapes in UTF-16 text and keeps its diff files private

A \u-escaped AWS key inside a UTF-16 JSON file passed both scans, because
the UTF-16 and NUL-stripped readings skipped the JSON decoding every other
reading gets. The sync's two scan diffs, which now carry binary bytes, were
written with the user's umask (0644 under /tmp); they are created 0600.

Review-Finding: p3/outside-voice#1 reader=outside-voice
Review-Finding: p3/outside-voice#2 reader=outside-voice
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-10 06:22:58 +00:00

262 lines
9.8 KiB
Bash
Executable File

#!/usr/bin/env bash
# gstack-brain-restore — bootstrap a new machine from an existing brain repo.
#
# Usage:
# gstack-brain-restore [<git-remote-url>]
#
# If no URL is given, reads from ~/.gstack-brain-remote.txt (written by
# gstack-brain-init on the original machine). Copy that file to the new
# machine before running this command.
#
# Safety gates (refuses with clear message):
# - ~/.gstack/.git already exists with a DIFFERENT remote
# - ~/.gstack/ contains non-allowlisted, non-gitignored user files
# that would be clobbered by restore
#
# What it does:
# 1. Clone the remote to a staging directory
# 2. Validate the repo is gstack-brain-shaped (.brain-allowlist, .gitattributes)
# 3. rsync-copy tracked files into ~/.gstack/ with skip-if-same-hash
# 4. Move staging's .git into ~/.gstack/.git
# 5. Register local git config merge drivers (they don't clone from remote)
# 6. Wire the cloned brain into gbrain via gstack-gbrain-source-wireup
# (best-effort; restore continues even if gbrain wireup fails)
#
# Env:
# GSTACK_HOME — override ~/.gstack
# Heredoc delivery guard. bash 5.2+ writes a heredoc body <=64KiB through a
# pipe in the forked child before exec, with no reader on the other end. On
# macOS under pipe-KVA pressure a fresh pipe gets a 512-byte buffer, so any
# body >=512B blocks write() forever and the script hangs at startup with no
# output. Compat level 50 restores the tempfile path. These scripts are
# bash-3.2-clean, so the compat level costs them nothing. Not exported: the
# guard is per-script, and it survives `bash script.sh` call sites that
# bypass the shebang.
BASH_COMPAT=50
set -euo pipefail
. "$(dirname "$0")/gstack-state-root.sh" 2>/dev/null || { echo "$0: cannot resolve the gstack state root: $(dirname "$0")/gstack-state-root.sh is missing. fix: reinstall with ./setup or /gstack-upgrade (docs/state-root.md)" >&2; exit 1; }
gstack_state_root_select
GSTACK_HOME="$_gstack_sr_root"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CONFIG_BIN="$SCRIPT_DIR/gstack-config"
# Egress receipt helpers (_receipted_git): fail-open for user-directed
# git ops against the user's own artifacts remote.
. "$SCRIPT_DIR/gstack-egress-lib.sh"
# v1.27.0.0+ canonical name; brain-remote is the legacy fallback during the
# migration window. The migration script renames the file in place.
if [ -f "$HOME/.gstack-artifacts-remote.txt" ]; then
REMOTE_FILE="$HOME/.gstack-artifacts-remote.txt"
else
REMOTE_FILE="$HOME/.gstack-brain-remote.txt"
fi
REMOTE_URL="${1:-}"
if [ -z "$REMOTE_URL" ]; then
if [ -f "$REMOTE_FILE" ]; then
REMOTE_URL=$(head -1 "$REMOTE_FILE" | tr -d '[:space:]')
fi
fi
if [ -z "$REMOTE_URL" ]; then
cat >&2 <<EOF
gstack-brain-restore: no remote URL provided.
Provide one of:
gstack-brain-restore <git-url>
or put the URL in $REMOTE_FILE (copy from the original machine)
EOF
exit 1
fi
# ---- safety gates ----
if [ -d "$GSTACK_HOME/.git" ]; then
EXISTING_REMOTE=$(git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "")
if [ -n "$EXISTING_REMOTE" ] && [ "$EXISTING_REMOTE" != "$REMOTE_URL" ]; then
cat >&2 <<EOF
gstack-brain-restore: ~/.gstack/.git already points at:
$EXISTING_REMOTE
You asked to restore from:
$REMOTE_URL
Refusing to overwrite. Run 'gstack-brain-uninstall' first or pass a matching URL.
EOF
exit 1
fi
fi
# ---- drain lock ----
# Every gstack writer to the artifacts repo's git state takes the same lock;
# the wireup this script runs at the end inherits it.
. "$SCRIPT_DIR/gstack-brain-lock.sh"
if ! gstack_brain_lock_acquire "$GSTACK_HOME" wait; then
echo "gstack-brain-restore: another gstack sync (pid ${_GSTACK_BRAIN_LOCK_HOLDER:-unknown}) is using $GSTACK_HOME; re-run when it finishes" >&2
exit 1
fi
# ---- clone to staging ----
STAGING=$(mktemp -d "${TMPDIR:-/tmp}/gstack-brain-restore.XXXXXX")
trap 'rm -rf "$STAGING" 2>/dev/null; gstack_brain_lock_release' EXIT
echo "Cloning $REMOTE_URL to staging..."
RESTORE_HOST="${REMOTE_URL#*://}"; RESTORE_HOST="${RESTORE_HOST#*@}"; RESTORE_HOST="${RESTORE_HOST%%[/:]*}"
if ! _receipted_git open brain-restore "${RESTORE_HOST:-unknown}" brain-restore-clone "user ran gstack-brain-restore" \
bash -c 'git clone --quiet "$1" "$2" 2>/dev/null' _ "$REMOTE_URL" "$STAGING/repo"; then
echo "Clone failed. Check:" >&2
echo " - URL is correct: $REMOTE_URL" >&2
echo " - Auth: gh auth status (github) / glab auth status (gitlab)" >&2
exit 1
fi
# ---- validate shape ----
if [ ! -f "$STAGING/repo/.brain-allowlist" ] || [ ! -f "$STAGING/repo/.gitattributes" ]; then
cat >&2 <<EOF
gstack-brain-restore: $REMOTE_URL does not look like a gstack-brain repo.
Missing: .brain-allowlist and/or .gitattributes
This command only works on repos created by gstack-brain-init.
EOF
exit 1
fi
# ---- validate target ~/.gstack/ has no non-gitignored user files ----
mkdir -p "$GSTACK_HOME"
if [ ! -d "$GSTACK_HOME/.git" ]; then
# No existing git → check if we'd clobber anything allowlisted.
# Read the new allowlist globs and see if any existing files would collide.
CLOBBER_RISK=$(python3 - "$GSTACK_HOME" "$STAGING/repo/.brain-allowlist" <<'PYEOF'
import sys, os, fnmatch
home, allowlist_path = sys.argv[1:3]
try:
with open(allowlist_path) as f:
globs = [l.strip() for l in f if l.strip() and not l.lstrip().startswith('#')]
except FileNotFoundError:
globs = []
risks = []
for root, dirs, files in os.walk(home):
dirs[:] = [d for d in dirs if d != '.git']
for name in files:
full = os.path.join(root, name)
rel = os.path.relpath(full, home)
for g in globs:
if fnmatch.fnmatchcase(rel, g):
risks.append(rel)
break
for r in risks[:5]:
print(r)
if len(risks) > 5:
print(f"...and {len(risks) - 5} more")
sys.exit(0 if not risks else 2)
PYEOF
) || true
if [ -n "$CLOBBER_RISK" ]; then
cat >&2 <<EOF
gstack-brain-restore: ~/.gstack/ has existing allowlisted files that would
be clobbered by restore:
$CLOBBER_RISK
Back these up first, or run this command on a machine with an empty
~/.gstack/. If these files are from an earlier gstack session on THIS
machine, you probably want to run gstack-brain-init instead (to create a
new brain repo with this machine's state).
EOF
exit 1
fi
fi
# ---- copy tracked files in ----
echo "Copying tracked files into ~/.gstack/ ..."
# Use git-ls-tree to get exact tracked file list (avoids staged/untracked files).
cd "$STAGING/repo"
git ls-tree -r --name-only HEAD | while IFS= read -r rel_path; do
src="$STAGING/repo/$rel_path"
dst="$GSTACK_HOME/$rel_path"
mkdir -p "$(dirname "$dst")"
# Skip if identical (content hash). Otherwise copy.
if [ -f "$dst" ] && cmp -s "$src" "$dst"; then
continue
fi
cp "$src" "$dst"
done
# ---- move .git into place ----
if [ -d "$GSTACK_HOME/.git" ]; then
# Existing .git with matching remote — just fetch + fast-forward.
_receipted_git open brain-restore "${RESTORE_HOST:-unknown}" brain-restore-fetch "user ran gstack-brain-restore" \
bash -c 'git -C "$1" fetch origin >/dev/null 2>&1' _ "$GSTACK_HOME" || true
else
mv "$STAGING/repo/.git" "$GSTACK_HOME/.git"
fi
# ---- register merge drivers (local git config; don't survive clones) ----
git -C "$GSTACK_HOME" config merge.jsonl-append.driver "$SCRIPT_DIR/gstack-jsonl-merge %O %A %B"
git -C "$GSTACK_HOME" config merge.jsonl-append.name "gstack JSONL append-only merger"
git -C "$GSTACK_HOME" config merge.union.driver "cat %A %B > %A.merged && mv %A.merged %A"
git -C "$GSTACK_HOME" config merge.union.name "union concat"
# ---- install pre-commit hook (same as init) ----
HOOK="$GSTACK_HOME/.git/hooks/pre-commit"
mkdir -p "$(dirname "$HOOK")"
cp "$SCRIPT_DIR/gstack-brain-secret-scan.py" "$(dirname "$HOOK")/gstack-brain-secret-scan.py"
cat > "$HOOK" <<'HOOK_EOF'
#!/usr/bin/env bash
# gstack-brain pre-commit hook: secret-scan defense-in-depth. It runs a copy of the
# scanner gstack-brain-sync runs before staging (same families, the whole
# staged diff, JSON string values decoded), written beside this hook by
# gstack-artifacts-init and gstack-brain-restore, so a manual `git commit` against
# the artifacts repo is held the same way.
set -uo pipefail
SCAN="$(dirname "$0")/gstack-brain-secret-scan.py"
if [ ! -f "$SCAN" ]; then
echo "gstack-brain pre-commit: refusing commit — the secret scanner $SCAN is missing; rerun gstack-brain-restore to reinstall this hook." >&2
exit 1
fi
TEXT=$(mktemp) || { echo "gstack-brain pre-commit: refusing commit — could not create a temporary file for the secret scan." >&2; exit 1; }
trap 'rm -f "$TEXT"' EXIT
if ! git diff --cached --no-renames --no-color --no-ext-diff --text --no-textconv -U3 > "$TEXT"; then
echo "gstack-brain pre-commit: refusing commit — git could not read the staged diff for the secret scan." >&2
exit 1
fi
hits=$(git diff --cached --no-renames --no-color --no-ext-diff | python3 "$SCAN" gate - "$TEXT")
rc=$?
[ "$rc" -eq 0 ] && exit 0
if [ "$rc" -eq 3 ]; then
echo "gstack-brain pre-commit: refusing commit — $hits detected in staged diff." >&2
else
echo "gstack-brain pre-commit: refusing commit — the secret scan failed (exit $rc)." >&2
fi
exit 1
HOOK_EOF
chmod +x "$HOOK"
# ---- write remote helper file if missing ----
if [ ! -f "$REMOTE_FILE" ]; then
echo "$REMOTE_URL" > "$REMOTE_FILE"
chmod 600 "$REMOTE_FILE"
echo ""
echo "Wrote $REMOTE_FILE for future skill-run auto-detection."
fi
# ---- wire the cloned brain into gbrain (best-effort) ----
WIREUP_BIN="$SCRIPT_DIR/gstack-gbrain-source-wireup"
if [ -x "$WIREUP_BIN" ]; then
"$WIREUP_BIN" || >&2 echo "WARNING: gbrain wireup failed; run $WIREUP_BIN manually after fixing prereqs"
fi
cat <<EOF
gstack-brain-restore complete.
Local: $GSTACK_HOME
Remote: $REMOTE_URL
Next skill run will ask about privacy mode (one-time question) and then
sync automatically at skill boundaries.
Status anytime: gstack-brain-sync --status
EOF