mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-10 18:50:18 +01:00
* Brain sync scans binary files and split authorization values for secrets (GST-2292) Git showed a file with a NUL byte or UTF-16 text as "Binary files differ", so the push-time secret scan never saw an AWS key inside it. The sync and the pre-commit hook now diff with --text --no-textconv, the scanner also reads NUL-bearing lines as UTF-16 and with NULs removed, and a section git still did not diff as text is held as binary-not-scanned. A Bearer value added under an unchanged "authorization": line was missed because the + marker broke the match; each hunk's new text is now scanned as one block too. The docs say plainly that an already-committed secret stays held after deletion and only --skip-file clears it. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Secret scan catches split auth values in UTF-16 files and under diff.context=0 Review-Finding: p1/red-team#1 reader=red-team Review-Finding: p1/red-team#2 reader=red-team Review-Finding: p1/red-team#3 reader=red-team Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Sync docs say a secret already committed in ~/.gstack stays held until you skip the file Approved docs-audit repair: approval a-029ea2b533be (GStack Minder GSTA-36, lane brief item 3). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Secret scan keeps the plain git diff as one reading beside the text reading The sync and the pre-commit hooks had replaced git's plain staged diff with a --text --no-textconv -U3 diff. That lost two things the old scan held: an unchanged line that the user's diff.context showed (5 lines away from an edit), and a secret that only a configured textconv driver decodes. The scanner now takes both diffs and holds a file when either reading matches; binary-not-scanned applies to the text reading only. Review-Finding: sol-audit-201c6a0#1 reader=cross-family-audit Review-Finding: sol-audit-201c6a0#2 reader=cross-family-audit Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Secret scan decodes JSON escapes in UTF-16 text and keeps its diff files private A \u-escaped AWS key inside a UTF-16 JSON file passed both scans, because the UTF-16 and NUL-stripped readings skipped the JSON decoding every other reading gets. The sync's two scan diffs, which now carry binary bytes, were written with the user's umask (0644 under /tmp); they are created 0600. Review-Finding: p3/outside-voice#1 reader=outside-voice Review-Finding: p3/outside-voice#2 reader=outside-voice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
262 lines
9.8 KiB
Bash
Executable File
262 lines
9.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# gstack-brain-restore — bootstrap a new machine from an existing brain repo.
|
|
#
|
|
# Usage:
|
|
# gstack-brain-restore [<git-remote-url>]
|
|
#
|
|
# If no URL is given, reads from ~/.gstack-brain-remote.txt (written by
|
|
# gstack-brain-init on the original machine). Copy that file to the new
|
|
# machine before running this command.
|
|
#
|
|
# Safety gates (refuses with clear message):
|
|
# - ~/.gstack/.git already exists with a DIFFERENT remote
|
|
# - ~/.gstack/ contains non-allowlisted, non-gitignored user files
|
|
# that would be clobbered by restore
|
|
#
|
|
# What it does:
|
|
# 1. Clone the remote to a staging directory
|
|
# 2. Validate the repo is gstack-brain-shaped (.brain-allowlist, .gitattributes)
|
|
# 3. rsync-copy tracked files into ~/.gstack/ with skip-if-same-hash
|
|
# 4. Move staging's .git into ~/.gstack/.git
|
|
# 5. Register local git config merge drivers (they don't clone from remote)
|
|
# 6. Wire the cloned brain into gbrain via gstack-gbrain-source-wireup
|
|
# (best-effort; restore continues even if gbrain wireup fails)
|
|
#
|
|
# Env:
|
|
# GSTACK_HOME — override ~/.gstack
|
|
|
|
# Heredoc delivery guard. bash 5.2+ writes a heredoc body <=64KiB through a
|
|
# pipe in the forked child before exec, with no reader on the other end. On
|
|
# macOS under pipe-KVA pressure a fresh pipe gets a 512-byte buffer, so any
|
|
# body >=512B blocks write() forever and the script hangs at startup with no
|
|
# output. Compat level 50 restores the tempfile path. These scripts are
|
|
# bash-3.2-clean, so the compat level costs them nothing. Not exported: the
|
|
# guard is per-script, and it survives `bash script.sh` call sites that
|
|
# bypass the shebang.
|
|
BASH_COMPAT=50
|
|
|
|
set -euo pipefail
|
|
|
|
. "$(dirname "$0")/gstack-state-root.sh" 2>/dev/null || { echo "$0: cannot resolve the gstack state root: $(dirname "$0")/gstack-state-root.sh is missing. fix: reinstall with ./setup or /gstack-upgrade (docs/state-root.md)" >&2; exit 1; }
|
|
gstack_state_root_select
|
|
GSTACK_HOME="$_gstack_sr_root"
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
CONFIG_BIN="$SCRIPT_DIR/gstack-config"
|
|
|
|
# Egress receipt helpers (_receipted_git): fail-open for user-directed
|
|
# git ops against the user's own artifacts remote.
|
|
. "$SCRIPT_DIR/gstack-egress-lib.sh"
|
|
# v1.27.0.0+ canonical name; brain-remote is the legacy fallback during the
|
|
# migration window. The migration script renames the file in place.
|
|
if [ -f "$HOME/.gstack-artifacts-remote.txt" ]; then
|
|
REMOTE_FILE="$HOME/.gstack-artifacts-remote.txt"
|
|
else
|
|
REMOTE_FILE="$HOME/.gstack-brain-remote.txt"
|
|
fi
|
|
|
|
REMOTE_URL="${1:-}"
|
|
if [ -z "$REMOTE_URL" ]; then
|
|
if [ -f "$REMOTE_FILE" ]; then
|
|
REMOTE_URL=$(head -1 "$REMOTE_FILE" | tr -d '[:space:]')
|
|
fi
|
|
fi
|
|
|
|
if [ -z "$REMOTE_URL" ]; then
|
|
cat >&2 <<EOF
|
|
gstack-brain-restore: no remote URL provided.
|
|
|
|
Provide one of:
|
|
gstack-brain-restore <git-url>
|
|
or put the URL in $REMOTE_FILE (copy from the original machine)
|
|
EOF
|
|
exit 1
|
|
fi
|
|
|
|
# ---- safety gates ----
|
|
if [ -d "$GSTACK_HOME/.git" ]; then
|
|
EXISTING_REMOTE=$(git -C "$GSTACK_HOME" remote get-url origin 2>/dev/null || echo "")
|
|
if [ -n "$EXISTING_REMOTE" ] && [ "$EXISTING_REMOTE" != "$REMOTE_URL" ]; then
|
|
cat >&2 <<EOF
|
|
gstack-brain-restore: ~/.gstack/.git already points at:
|
|
$EXISTING_REMOTE
|
|
|
|
You asked to restore from:
|
|
$REMOTE_URL
|
|
|
|
Refusing to overwrite. Run 'gstack-brain-uninstall' first or pass a matching URL.
|
|
EOF
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
# ---- drain lock ----
|
|
# Every gstack writer to the artifacts repo's git state takes the same lock;
|
|
# the wireup this script runs at the end inherits it.
|
|
. "$SCRIPT_DIR/gstack-brain-lock.sh"
|
|
if ! gstack_brain_lock_acquire "$GSTACK_HOME" wait; then
|
|
echo "gstack-brain-restore: another gstack sync (pid ${_GSTACK_BRAIN_LOCK_HOLDER:-unknown}) is using $GSTACK_HOME; re-run when it finishes" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# ---- clone to staging ----
|
|
STAGING=$(mktemp -d "${TMPDIR:-/tmp}/gstack-brain-restore.XXXXXX")
|
|
trap 'rm -rf "$STAGING" 2>/dev/null; gstack_brain_lock_release' EXIT
|
|
|
|
echo "Cloning $REMOTE_URL to staging..."
|
|
RESTORE_HOST="${REMOTE_URL#*://}"; RESTORE_HOST="${RESTORE_HOST#*@}"; RESTORE_HOST="${RESTORE_HOST%%[/:]*}"
|
|
if ! _receipted_git open brain-restore "${RESTORE_HOST:-unknown}" brain-restore-clone "user ran gstack-brain-restore" \
|
|
bash -c 'git clone --quiet "$1" "$2" 2>/dev/null' _ "$REMOTE_URL" "$STAGING/repo"; then
|
|
echo "Clone failed. Check:" >&2
|
|
echo " - URL is correct: $REMOTE_URL" >&2
|
|
echo " - Auth: gh auth status (github) / glab auth status (gitlab)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# ---- validate shape ----
|
|
if [ ! -f "$STAGING/repo/.brain-allowlist" ] || [ ! -f "$STAGING/repo/.gitattributes" ]; then
|
|
cat >&2 <<EOF
|
|
gstack-brain-restore: $REMOTE_URL does not look like a gstack-brain repo.
|
|
Missing: .brain-allowlist and/or .gitattributes
|
|
|
|
This command only works on repos created by gstack-brain-init.
|
|
EOF
|
|
exit 1
|
|
fi
|
|
|
|
# ---- validate target ~/.gstack/ has no non-gitignored user files ----
|
|
mkdir -p "$GSTACK_HOME"
|
|
if [ ! -d "$GSTACK_HOME/.git" ]; then
|
|
# No existing git → check if we'd clobber anything allowlisted.
|
|
# Read the new allowlist globs and see if any existing files would collide.
|
|
CLOBBER_RISK=$(python3 - "$GSTACK_HOME" "$STAGING/repo/.brain-allowlist" <<'PYEOF'
|
|
import sys, os, fnmatch
|
|
home, allowlist_path = sys.argv[1:3]
|
|
try:
|
|
with open(allowlist_path) as f:
|
|
globs = [l.strip() for l in f if l.strip() and not l.lstrip().startswith('#')]
|
|
except FileNotFoundError:
|
|
globs = []
|
|
risks = []
|
|
for root, dirs, files in os.walk(home):
|
|
dirs[:] = [d for d in dirs if d != '.git']
|
|
for name in files:
|
|
full = os.path.join(root, name)
|
|
rel = os.path.relpath(full, home)
|
|
for g in globs:
|
|
if fnmatch.fnmatchcase(rel, g):
|
|
risks.append(rel)
|
|
break
|
|
for r in risks[:5]:
|
|
print(r)
|
|
if len(risks) > 5:
|
|
print(f"...and {len(risks) - 5} more")
|
|
sys.exit(0 if not risks else 2)
|
|
PYEOF
|
|
) || true
|
|
if [ -n "$CLOBBER_RISK" ]; then
|
|
cat >&2 <<EOF
|
|
gstack-brain-restore: ~/.gstack/ has existing allowlisted files that would
|
|
be clobbered by restore:
|
|
|
|
$CLOBBER_RISK
|
|
|
|
Back these up first, or run this command on a machine with an empty
|
|
~/.gstack/. If these files are from an earlier gstack session on THIS
|
|
machine, you probably want to run gstack-brain-init instead (to create a
|
|
new brain repo with this machine's state).
|
|
EOF
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
# ---- copy tracked files in ----
|
|
echo "Copying tracked files into ~/.gstack/ ..."
|
|
# Use git-ls-tree to get exact tracked file list (avoids staged/untracked files).
|
|
cd "$STAGING/repo"
|
|
git ls-tree -r --name-only HEAD | while IFS= read -r rel_path; do
|
|
src="$STAGING/repo/$rel_path"
|
|
dst="$GSTACK_HOME/$rel_path"
|
|
mkdir -p "$(dirname "$dst")"
|
|
# Skip if identical (content hash). Otherwise copy.
|
|
if [ -f "$dst" ] && cmp -s "$src" "$dst"; then
|
|
continue
|
|
fi
|
|
cp "$src" "$dst"
|
|
done
|
|
|
|
# ---- move .git into place ----
|
|
if [ -d "$GSTACK_HOME/.git" ]; then
|
|
# Existing .git with matching remote — just fetch + fast-forward.
|
|
_receipted_git open brain-restore "${RESTORE_HOST:-unknown}" brain-restore-fetch "user ran gstack-brain-restore" \
|
|
bash -c 'git -C "$1" fetch origin >/dev/null 2>&1' _ "$GSTACK_HOME" || true
|
|
else
|
|
mv "$STAGING/repo/.git" "$GSTACK_HOME/.git"
|
|
fi
|
|
|
|
# ---- register merge drivers (local git config; don't survive clones) ----
|
|
git -C "$GSTACK_HOME" config merge.jsonl-append.driver "$SCRIPT_DIR/gstack-jsonl-merge %O %A %B"
|
|
git -C "$GSTACK_HOME" config merge.jsonl-append.name "gstack JSONL append-only merger"
|
|
git -C "$GSTACK_HOME" config merge.union.driver "cat %A %B > %A.merged && mv %A.merged %A"
|
|
git -C "$GSTACK_HOME" config merge.union.name "union concat"
|
|
|
|
# ---- install pre-commit hook (same as init) ----
|
|
HOOK="$GSTACK_HOME/.git/hooks/pre-commit"
|
|
mkdir -p "$(dirname "$HOOK")"
|
|
cp "$SCRIPT_DIR/gstack-brain-secret-scan.py" "$(dirname "$HOOK")/gstack-brain-secret-scan.py"
|
|
cat > "$HOOK" <<'HOOK_EOF'
|
|
#!/usr/bin/env bash
|
|
# gstack-brain pre-commit hook: secret-scan defense-in-depth. It runs a copy of the
|
|
# scanner gstack-brain-sync runs before staging (same families, the whole
|
|
# staged diff, JSON string values decoded), written beside this hook by
|
|
# gstack-artifacts-init and gstack-brain-restore, so a manual `git commit` against
|
|
# the artifacts repo is held the same way.
|
|
set -uo pipefail
|
|
SCAN="$(dirname "$0")/gstack-brain-secret-scan.py"
|
|
if [ ! -f "$SCAN" ]; then
|
|
echo "gstack-brain pre-commit: refusing commit — the secret scanner $SCAN is missing; rerun gstack-brain-restore to reinstall this hook." >&2
|
|
exit 1
|
|
fi
|
|
TEXT=$(mktemp) || { echo "gstack-brain pre-commit: refusing commit — could not create a temporary file for the secret scan." >&2; exit 1; }
|
|
trap 'rm -f "$TEXT"' EXIT
|
|
if ! git diff --cached --no-renames --no-color --no-ext-diff --text --no-textconv -U3 > "$TEXT"; then
|
|
echo "gstack-brain pre-commit: refusing commit — git could not read the staged diff for the secret scan." >&2
|
|
exit 1
|
|
fi
|
|
hits=$(git diff --cached --no-renames --no-color --no-ext-diff | python3 "$SCAN" gate - "$TEXT")
|
|
rc=$?
|
|
[ "$rc" -eq 0 ] && exit 0
|
|
if [ "$rc" -eq 3 ]; then
|
|
echo "gstack-brain pre-commit: refusing commit — $hits detected in staged diff." >&2
|
|
else
|
|
echo "gstack-brain pre-commit: refusing commit — the secret scan failed (exit $rc)." >&2
|
|
fi
|
|
exit 1
|
|
HOOK_EOF
|
|
chmod +x "$HOOK"
|
|
|
|
# ---- write remote helper file if missing ----
|
|
if [ ! -f "$REMOTE_FILE" ]; then
|
|
echo "$REMOTE_URL" > "$REMOTE_FILE"
|
|
chmod 600 "$REMOTE_FILE"
|
|
echo ""
|
|
echo "Wrote $REMOTE_FILE for future skill-run auto-detection."
|
|
fi
|
|
|
|
# ---- wire the cloned brain into gbrain (best-effort) ----
|
|
WIREUP_BIN="$SCRIPT_DIR/gstack-gbrain-source-wireup"
|
|
if [ -x "$WIREUP_BIN" ]; then
|
|
"$WIREUP_BIN" || >&2 echo "WARNING: gbrain wireup failed; run $WIREUP_BIN manually after fixing prereqs"
|
|
fi
|
|
|
|
cat <<EOF
|
|
|
|
gstack-brain-restore complete.
|
|
Local: $GSTACK_HOME
|
|
Remote: $REMOTE_URL
|
|
|
|
Next skill run will ask about privacy mode (one-time question) and then
|
|
sync automatically at skill boundaries.
|
|
|
|
Status anytime: gstack-brain-sync --status
|
|
EOF
|