From bcece3fbcdf0e84e029caac5d04524af3ff6ba41 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Tue, 17 Nov 2026 17:05:25 +0800 Subject: [PATCH] Make gstack's shell locks hold when mkdir reports a race it lost (GST-2247) (#4262) * Make gstack's shell locks hold when mkdir reports a race it lost (GST-2247) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Fix lock tests and fixtures for gstack_lock_take; trim setup-gbrain lock prose Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Ship the lock helper with every copied runtime and fixture that runs a lock The Claude Code wrapper's copied runtime now includes bin/gstack-lock.sh, and bin/gstack-render-claude.sh sources it beside itself, so gstack-config set works in a copied runtime and the render lock is taken by every caller. Test fixtures that copy single bin scripts now copy the helper too, and the mkdir lock guard is registered as a repo-wide docs reader. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Release a lock whose owner file could not be fully written gstack_lock_take created the owner file with a noclobber redirect and, when that redirect failed, treated an existing file as a competing holder. If the file was created but the write then failed (disk full, quota, ulimit -f), the helper found its own empty file, returned 1 and left a lock nobody held; freeze's state lock has no stale recovery, so it stayed busy until removed by hand. The helper now tells "opened but write failed" from "already held", removes its own file and directory, and returns 3. The inline copy in gstack-ff-update is updated byte for byte, and a new test under ulimit -f 0 fails on the old helper. Review-Finding: s11/outside-voice#1 reader=outside-voice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Describe the lease lock as gstack_lock_take's O_EXCL owner file Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- USING_GBRAIN_WITH_GSTACK.md | 2 +- bin/gstack-brain-lock.sh | 15 +- bin/gstack-browser-ensure | 4 +- bin/gstack-capy-session | 2 +- bin/gstack-codex-probe | 20 +- bin/gstack-config | 14 +- bin/gstack-developer-profile | 22 ++- bin/gstack-ff-update | 28 ++- bin/gstack-install-registry.sh | 13 +- bin/gstack-lease | 27 +-- bin/gstack-lock.sh | 48 +++++ bin/gstack-memorable | 12 +- bin/gstack-render-claude.sh | 13 +- bin/gstack-session-update | 27 +-- bin/gstack-settings-hook | 16 +- bin/gstack-upgrade | 6 +- ...ell-locks-hold-when-mkdir-lies-gst-2247.md | 5 + docs/capy-install.md | 8 +- docs/troubleshooting.md | 30 +++ freeze/bin/freeze-state.sh | 5 +- lib/claude-code-migration.ts | 2 +- lib/docs-only.ts | 2 +- setup | 25 +-- setup-gbrain/SKILL.md | 11 +- setup-gbrain/SKILL.md.tmpl | 11 +- test/capy-skill-wrapper.test.ts | 2 +- test/codex-probe-cache.test.ts | 9 +- test/freeze-owned-lifecycle.test.ts | 2 +- test/gen-skill-docs.test.ts | 2 +- test/gstack-browser-ensure.test.ts | 1 + test/gstack-ff-update.test.ts | 7 + test/gstack-lease.test.ts | 3 +- test/gstack-lock-race.test.ts | 186 ++++++++++++++++++ test/gstack-settings-hook-symlink.test.ts | 9 +- test/gstack-upgrade-bin.test.ts | 2 +- test/helpers/ship-measure-seeded-fixture.ts | 8 +- test/mkdir-lock-guard.test.ts | 131 ++++++++++++ test/plan-mode-preamble-render.test.ts | 2 +- test/relink-disabled-router.test.ts | 2 +- test/relink.test.ts | 1 + test/session-update-autostash.test.ts | 2 +- test/session-update-hook-hold.test.ts | 2 +- test/setup-claude-code-migration.test.ts | 2 +- test/setup-gbrain-fixture.test.ts | 6 +- test/setup-playwright-best-effort.test.ts | 2 + test/setup-production-deps.test.ts | 4 +- test/setup-skill-hooks-gate.test.ts | 2 +- test/team-mode.test.ts | 2 +- test/uninstall.test.ts | 14 +- test/update-channel.test.ts | 4 +- 50 files changed, 621 insertions(+), 154 deletions(-) create mode 100644 bin/gstack-lock.sh create mode 100644 changelog.d/capy-shell-locks-hold-when-mkdir-lies-gst-2247.md create mode 100644 test/gstack-lock-race.test.ts create mode 100644 test/mkdir-lock-guard.test.ts diff --git a/USING_GBRAIN_WITH_GSTACK.md b/USING_GBRAIN_WITH_GSTACK.md index d4a59d370..457bdeea3 100644 --- a/USING_GBRAIN_WITH_GSTACK.md +++ b/USING_GBRAIN_WITH_GSTACK.md @@ -243,7 +243,7 @@ Gbrain itself ships with these that gstack wraps: |---|---| | `~/.gbrain/config.json` | Engine (pglite/postgres), database URL or path, API keys. Mode 0600. Written by `gbrain init`. When `GBRAIN_HOME` is set, gbrain and gstack's helpers read `$GBRAIN_HOME/.gbrain/config.json` instead. | | `~/.gstack/gbrain-repo-policy.json` | Per-remote trust triad. Schema v2. Mode 0600. | -| `~/.gstack/.setup-gbrain.lock.d` | Concurrent-run lock (atomic mkdir). Released on normal exit + SIGINT. | +| `~/.gstack/.setup-gbrain.lock.d` | Concurrent-run lock: a directory whose `pid` file is created with O_EXCL (`gstack_lock_take`, `bin/gstack-lock.sh`), so it holds even where `mkdir` misreports a race. Released on normal exit + SIGINT. | | `~/.gstack/.brain-queue.d/` | Pending sync records for gstack memory sync — maildir-style spool, one file per record. A legacy `.brain-queue.jsonl` from older releases migrates automatically on the next drain. | | `~/.gstack/.brain-last-push` | Timestamp of last sync push (for `/health` scoring) | | `~/.gstack-artifacts-remote.txt` | URL of your gstack memory sync remote (safe to copy between machines; legacy name `~/.gstack-brain-remote.txt` still read) | diff --git a/bin/gstack-brain-lock.sh b/bin/gstack-brain-lock.sh index 2f60ac942..110caebb0 100644 --- a/bin/gstack-brain-lock.sh +++ b/bin/gstack-brain-lock.sh @@ -23,10 +23,14 @@ # gstack_brain_lock_release # Removes the lock only when this process took it and still owns it. # -# The lock is a directory (mkdir is atomic on every POSIX filesystem; flock(1) -# is not on macOS) holding `pid` and `owner`. A lock whose pid is dead is -# stale and is cleared. A lock directory with no pid file is a holder killed -# between mkdir and the pid write once it is older than 10 minutes. +# The lock is a directory holding `pid` and `owner`, taken with +# gstack_lock_take (bin/gstack-lock.sh), which creates `pid` with O_EXCL: +# mkdir's exit status is not a lock on Ubuntu 26.04's uutils mkdir, and +# flock(1) is not on macOS. A lock whose pid is dead is stale and is cleared. +# A lock directory with no pid file is a holder killed between mkdir and the +# pid write once it is older than 10 minutes. + +. "$(dirname "${BASH_SOURCE[0]}")/gstack-lock.sh" 2>/dev/null || gstack_lock_take() { echo "gstack: $(dirname "${BASH_SOURCE[0]}")/gstack-lock.sh is missing; reinstall with ./setup" >&2; return 2; } _GSTACK_BRAIN_LOCK_DIR="" _GSTACK_BRAIN_LOCK_MINE=0 @@ -34,10 +38,9 @@ _GSTACK_BRAIN_LOCK_HOLDER="" _gstack_brain_lock_try_once() { local dir="$1" pid - if mkdir "$dir" 2>/dev/null; then + if gstack_lock_take "$dir" pid "$$"; then GSTACK_BRAIN_LOCK_TOKEN="$$-${RANDOM:-0}-$(date +%s 2>/dev/null || echo 0)" export GSTACK_BRAIN_LOCK_TOKEN - printf '%s\n' "$$" > "$dir/pid" 2>/dev/null || true printf '%s\n' "$GSTACK_BRAIN_LOCK_TOKEN" > "$dir/owner" 2>/dev/null || true _GSTACK_BRAIN_LOCK_MINE=1 return 0 diff --git a/bin/gstack-browser-ensure b/bin/gstack-browser-ensure index 0f87e3274..95e1d1824 100755 --- a/bin/gstack-browser-ensure +++ b/bin/gstack-browser-ensure @@ -101,8 +101,9 @@ BUN=$(command -v bun 2>/dev/null) || fail "Chromium is not installed and bun is # Same lock ./setup takes (#2136), so two installs never race on the cache. LOCK="${TMPDIR:-/tmp}/gstack-playwright-install.lock" +. "$ROOT/bin/gstack-lock.sh" 2>/dev/null || fail "$ROOT/bin/gstack-lock.sh is missing" "reinstall with cd $ROOT && ./setup" tries=0 -while ! mkdir "$LOCK" 2>/dev/null; do +while ! gstack_lock_take "$LOCK" pid "$$"; do holder=$(cat "$LOCK/pid" 2>/dev/null || true) case "$holder" in ''|*[!0-9]*) holder=0 ;; esac if [ "$holder" -eq 0 ] || ! kill -0 "$holder" 2>/dev/null; then rm -rf "$LOCK" 2>/dev/null; continue; fi @@ -110,7 +111,6 @@ while ! mkdir "$LOCK" 2>/dev/null; do [ "$tries" -ge "$TIMEOUT" ] && fail "another gstack install holds the Chromium lock ($LOCK, pid $holder)" "wait for it, or remove a stale lock: rm -rf \"$LOCK\"" sleep 1 done -echo "$$" > "$LOCK/pid" 2>/dev/null || true trap 'rm -rf "$LOCK" 2>/dev/null' EXIT INT TERM OVERRIDE="" diff --git a/bin/gstack-capy-session b/bin/gstack-capy-session index 314390708..64b83926f 100755 --- a/bin/gstack-capy-session +++ b/bin/gstack-capy-session @@ -283,7 +283,7 @@ EOF_OWN report_behind "$RERUN" export GSTACK_CAPY_BEHIND_BLOCKED="$REASON" elif [ -n "$SWITCH" ] && ! { SETUP_LOCK="$("$ROOT/bin/gstack-paths" --get GSTACK_STATE_ROOT 2>/dev/null)/.setup-lock"; [ "$SETUP_LOCK" != /.setup-lock ] \ - && mkdir -p "$(dirname "$SETUP_LOCK")" 2>/dev/null && mkdir "$SETUP_LOCK" 2>/dev/null && echo "$$" > "$SETUP_LOCK/pid"; }; then + && mkdir -p "$(dirname "$SETUP_LOCK")" 2>/dev/null && . "$ROOT/bin/gstack-lock.sh" && gstack_lock_take "$SETUP_LOCK" pid "$$"; }; then # The switch and the fast-forward happen under gstack-upgrade's setup lock, # taken here and handed over in GSTACK_SETUP_LOCK_OWNER, so a concurrent # updater never sees the checkout between the two. diff --git a/bin/gstack-codex-probe b/bin/gstack-codex-probe index 9f0a4f115..afbcadab0 100755 --- a/bin/gstack-codex-probe +++ b/bin/gstack-codex-probe @@ -103,6 +103,7 @@ fi # === end gstack self-locate === _GSTACK_CODEX_BIN="$_gstack_helper_dir" unset _gstack_helper _gstack_helper_dir _gstack_helper_shell +. "$_GSTACK_CODEX_BIN/gstack-lock.sh" 2>/dev/null || gstack_lock_take() { return 2; } # --- Auth probe ------------------------------------------------------------- @@ -631,19 +632,16 @@ _gstack_codex_probe_cache_store() { _gstack_codex_lock_acquire() { # $1 lock directory, $2 seconds to wait, $3 seconds after which even a live - # owner is abandoned. mkdir is atomic on every platform bash runs on. A dead - # owner on this host is recovered at once; a missing owner file is the + # owner is abandoned. gstack_lock_take (bin/gstack-lock.sh) creates the owner + # file with O_EXCL, since mkdir's exit status is not a lock on Ubuntu 26.04's + # uutils mkdir. A dead owner on this host is recovered at once; a missing owner file is the # creation window and becomes stale after 5 seconds. 0 = held, 1 = timed # out, 2 = the lock or its owner record cannot be created. local _dir="$1" _tries=$(( ${2:-45} * 10 )) _stale="${3:-90}" _host _owner _pid _ohost _ts _now _i=0 _dead _host=$(uname -n 2>/dev/null || echo unknown) while :; do - if mkdir "$_dir" 2>/dev/null; then - printf '%s %s %s\n' "$$" "$_host" "$(date +%s 2>/dev/null || echo 0)" > "$_dir/owner" 2>/dev/null && return 0 - rm -f "$_dir/owner" 2>/dev/null - rmdir "$_dir" 2>/dev/null - return 2 - fi + gstack_lock_take "$_dir" owner "$$ $_host $(date +%s 2>/dev/null || echo 0)" + case $? in 0) return 0 ;; 3) return 2 ;; esac if [ ! -d "$_dir" ]; then { [ -e "$_dir" ] || [ -L "$_dir" ]; } && return 2 { [ -d "${_dir%/*}" ] && [ -w "${_dir%/*}" ] && [ -x "${_dir%/*}" ]; } || return 2 @@ -677,15 +675,15 @@ _gstack_codex_lock_break() { # Breakers serialize on $1.break and remove the lock only if it still has the # owner they judged abandoned, so a lock re-taken meanwhile survives. local _dir="$1" _seen="$2" _bt _now - if mkdir "$_dir.break" 2>/dev/null; then + if gstack_lock_take "$_dir.break" pid "$$"; then [ "$(cat "$_dir/owner" 2>/dev/null)" = "$_seen" ] && rm -rf "$_dir" 2>/dev/null - rmdir "$_dir.break" 2>/dev/null + rm -rf "$_dir.break" 2>/dev/null return 0 fi _now=$(date +%s 2>/dev/null || echo 0) _bt=$(stat -c %Y "$_dir.break" 2>/dev/null || stat -f %m "$_dir.break" 2>/dev/null || echo "$_now") case "$_bt" in ''|*[!0-9]*) _bt="$_now" ;; esac - [ $((_now - _bt)) -ge 10 ] && rmdir "$_dir.break" 2>/dev/null + [ $((_now - _bt)) -ge 10 ] && rm -rf "$_dir.break" 2>/dev/null return 0 } diff --git a/bin/gstack-config b/bin/gstack-config index 8eb26cf72..89c4a8150 100755 --- a/bin/gstack-config +++ b/bin/gstack-config @@ -552,18 +552,21 @@ read_config_value() { # ── Config writes: one lock, one read-modify-write rename ────────────────── # Every mutation of config.yaml (set, unset, the transcript_repos pair, the -# user-slug writer) holds an atomic mkdir lock and replaces the file with one +# user-slug writer) holds one lock directory and replaces the file with one # rename, so concurrent writers never lose each other's keys and an -# interrupted write leaves the old file. The pid file holds " "; -# a lock whose holder is dead or older than 10 s is taken over. No flock(1): -# stock macOS does not ship it. +# interrupted write leaves the old file. The lock is taken with +# gstack_lock_take (bin/gstack-lock.sh), which creates the pid file with +# O_EXCL because mkdir's exit status is not a lock on Ubuntu 26.04's uutils +# mkdir. The pid file holds " "; a lock whose holder is dead or +# older than 10 s is taken over. No flock(1): stock macOS does not ship it. CONFIG_LOCK="$STATE_DIR/.config-yaml.lock" _config_locked="" config_lock() { mkdir -p "$STATE_DIR" + . "$(dirname "$0")/gstack-lock.sh" 2>/dev/null || { echo "Error: $(dirname "$0")/gstack-lock.sh is missing. Existing value left unchanged. fix: reinstall with ./setup or /gstack-upgrade" >&2; exit 1; } local waited=0 held pid ts stale - while ! mkdir "$CONFIG_LOCK" 2>/dev/null; do + while ! gstack_lock_take "$CONFIG_LOCK" pid "$$ $(date +%s)"; do if [ ! -w "$STATE_DIR" ]; then echo "Error: cannot write to $STATE_DIR (its config lock could not be created). Existing value left unchanged." >&2 exit 1 @@ -589,7 +592,6 @@ config_lock() { fi sleep 0.05 done - printf '%s %s\n' "$$" "$(date +%s)" > "$CONFIG_LOCK/pid" _config_locked=1 trap 'config_unlock' EXIT } diff --git a/bin/gstack-developer-profile b/bin/gstack-developer-profile index 624dc723c..2db44df2e 100755 --- a/bin/gstack-developer-profile +++ b/bin/gstack-developer-profile @@ -66,16 +66,18 @@ shift || true # changes and renames the whole file, and gstack-question-log starts a # background --derive after every append, so writers overlap: a shared temp # name made renames fail and an unlocked rewrite dropped a concurrent -# --log-session. Writers hold a mkdir lock (.lock/, whose owner file +# --log-session. Writers hold a lock directory (.lock/, whose owner file # names the holder's pid and a random token; release is ownership-checked) # from before their read until after their rename, and write a unique temp # file. A lock whose holder process is gone is taken over at once; one older # than 5 minutes is taken over even when its pid is alive, since no writer -# runs that long and the pid may have been reused. Takeovers are serialized by a second mkdir lock +# runs that long and the pid may have been reused. Takeovers are serialized by a second lock # (.lock.takeover/) and re-read the owner under it, so two waiters # cannot both take over or remove a lock a third writer just took. A lock or # takeover directory older than 30 s with no owner belongs to a writer that -# died before writing one; cleanup removes a takeover lock this writer holds. A writer that cannot get the lock within the +# died before writing one; cleanup removes a takeover lock this writer holds. Both locks +# are taken with gstack_lock_take (bin/gstack-lock.sh): mkdir's exit status is not +# a lock on Ubuntu 26.04's uutils mkdir, so the owner file is created with O_EXCL. A writer that cannot get the lock within the # give-up window writes nothing and exits 5. Writers outside this binary # (gstack-distill-apply, /plan-tune's declared edits) do not take it yet. # ----------------------------------------------------------------------- @@ -87,7 +89,7 @@ PROFILE_TMP="" cleanup() { if [ -n "$PROFILE_TMP" ]; then rm -f "$PROFILE_TMP"; fi - if [ -n "$HOLDS_TAKEOVER" ]; then rmdir "$TAKEOVER_DIR" 2>/dev/null || true; fi + if [ -n "$HOLDS_TAKEOVER" ]; then rm -rf "$TAKEOVER_DIR" 2>/dev/null || true; fi if [ -n "$LOCK_TOKEN" ] && [ "$(cat "$LOCK_DIR/owner" 2>/dev/null || true)" = "$LOCK_TOKEN" ]; then rm -rf "$LOCK_DIR" fi @@ -117,8 +119,9 @@ acquire_lock() { # GSTACK_PROFILE_LOCK_TIMEOUT_MS: test-only override for the give-up. local give_up_ms="${GSTACK_PROFILE_LOCK_TIMEOUT_MS:-10000}" waited_ms=0 started=$SECONDS vanished=0 owner local token="$$-$RANDOM$RANDOM" - until mkdir "$LOCK_DIR" 2>/dev/null; do - # mkdir failed with no lock to wait for: retry once in case a holder just + . "$SCRIPT_DIR/gstack-lock.sh" 2>/dev/null || { echo "gstack-developer-profile: $SCRIPT_DIR/gstack-lock.sh is missing, so nothing was written (exit 5). fix: reinstall with ./setup or /gstack-upgrade" >&2; exit 5; } + until gstack_lock_take "$LOCK_DIR" owner "$token"; do + # The take failed with no lock to wait for: retry once in case a holder just # released it, then report the directory instead of waiting out the window. if [ ! -e "$LOCK_DIR" ]; then if [ "$vanished" -eq 0 ]; then vanished=1; continue; fi @@ -128,13 +131,13 @@ acquire_lock() { vanished=0 owner=$(cat "$LOCK_DIR/owner" 2>/dev/null || true) if [ -d "$LOCK_DIR" ] && lock_abandoned "$owner"; then - if [ -d "$TAKEOVER_DIR" ] && [ "$(dir_age_s "$TAKEOVER_DIR")" -gt 30 ]; then rmdir "$TAKEOVER_DIR" 2>/dev/null || true; fi - if mkdir "$TAKEOVER_DIR" 2>/dev/null; then + if [ -d "$TAKEOVER_DIR" ] && [ "$(dir_age_s "$TAKEOVER_DIR")" -gt 30 ]; then rm -rf "$TAKEOVER_DIR" 2>/dev/null || true; fi + if gstack_lock_take "$TAKEOVER_DIR" pid "$$"; then HOLDS_TAKEOVER=1 if [ "$(cat "$LOCK_DIR/owner" 2>/dev/null || true)" = "$owner" ] && lock_abandoned "$owner"; then rm -rf "$LOCK_DIR" 2>/dev/null || true fi - rmdir "$TAKEOVER_DIR" 2>/dev/null || true + rm -rf "$TAKEOVER_DIR" 2>/dev/null || true HOLDS_TAKEOVER="" if [ ! -e "$LOCK_DIR" ]; then continue; fi fi @@ -146,7 +149,6 @@ acquire_lock() { sleep 0.05 waited_ms=$(( waited_ms + 50 )) done - printf '%s\n' "$token" > "$LOCK_DIR/owner" LOCK_TOKEN="$token" } diff --git a/bin/gstack-ff-update b/bin/gstack-ff-update index 8ee22267a..a385fe80e 100755 --- a/bin/gstack-ff-update +++ b/bin/gstack-ff-update @@ -9,7 +9,8 @@ # the new revision (git show FETCH_HEAD:bin/gstack-ff-update) and run it in a # private temp directory, so it sources nothing. The lock and the # flatten-and-strip sanitizer are inlined; test/gstack-ff-update.test.ts pins -# them to their twins in bin/gstack-session-update. +# them to their twins in bin/gstack-session-update, and gstack_lock_take to +# bin/gstack-lock.sh. # # Inside an install-scoped lock (/gstack-ff.lock): # 1. The working tree is captured as-is through a private index (tracked @@ -115,18 +116,38 @@ LOCK_DIR="$COMMON/gstack-ff.lock" LOCK_TTL_MINUTES=30 MYPID=$$ OWN_INDEX_LOCK="" TMP="" HB_PID="" +# Twin of bin/gstack-lock.sh (test-pinned): mkdir's exit status is not a lock +# on Ubuntu 26.04's uutils mkdir, so the pid file is created with O_EXCL. +gstack_lock_take() { + local _gl_dir="$1" _gl_file="${2:-pid}" _gl_content="${3:-$$}" + if [ -e "$_gl_dir" ] || [ -L "$_gl_dir" ]; then return 1; fi + mkdir "$_gl_dir" 2>/dev/null || : + if [ ! -d "$_gl_dir" ] || [ -L "$_gl_dir" ]; then + if [ -e "$_gl_dir" ] || [ -L "$_gl_dir" ]; then return 1; fi + return 2 + fi + local _gl_opened + _gl_opened=$(set -o noclobber; { echo o; printf '%s\n' "$_gl_content" >&3; } 2>/dev/null 3> "$_gl_dir/$_gl_file") && return 0 + if [ "$_gl_opened" = o ]; then + rm -f "$_gl_dir/$_gl_file"; rmdir "$_gl_dir" 2>/dev/null || : + return 3 + fi + if [ -e "$_gl_dir/$_gl_file" ] || [ -L "$_gl_dir/$_gl_file" ] || [ ! -d "$_gl_dir" ]; then return 1; fi + rmdir "$_gl_dir" 2>/dev/null || : + return 3 +} lock_is_expired() { _hb="$LOCK_DIR/pid" [ -f "$_hb" ] || _hb="$LOCK_DIR" [ -n "$(find "$_hb" -maxdepth 0 -mmin +$LOCK_TTL_MINUTES 2>/dev/null)" ] } lock_try() { - mkdir "$LOCK_DIR" 2>/dev/null && return 0 + gstack_lock_take "$LOCK_DIR" pid "$MYPID" && return 0 _lp=$(cat "$LOCK_DIR/pid" 2>/dev/null) if lock_is_expired || { [ "${_lp:-0}" -gt 0 ] 2>/dev/null && ! kill -0 "$_lp" 2>/dev/null; }; then mv "$LOCK_DIR" "$LOCK_DIR.reap.$$" 2>/dev/null || return 1 rm -rf "$LOCK_DIR.reap.$$" 2>/dev/null - mkdir "$LOCK_DIR" 2>/dev/null && return 0 + gstack_lock_take "$LOCK_DIR" pid "$MYPID" && return 0 fi return 1 } @@ -145,7 +166,6 @@ until lock_try; do [ "$_wait" -gt 0 ] || refuse 75 ff-busy "another gstack update is moving $DIR (lock $LOCK_DIR held by pid $(cat "$LOCK_DIR/pid" 2>/dev/null || echo unknown)); nothing was changed, retry in a minute" sleep 1; _wait=$((_wait - 1)) done -echo "$MYPID" > "$LOCK_DIR/pid" 2>/dev/null trap cleanup_ff_update EXIT trap 'exit 129' HUP trap 'exit 130' INT diff --git a/bin/gstack-install-registry.sh b/bin/gstack-install-registry.sh index a56caf180..7f53f9330 100644 --- a/bin/gstack-install-registry.sh +++ b/bin/gstack-install-registry.sh @@ -23,10 +23,12 @@ # render render directory, or 'committed' # # Callers must set GSTACK_STATE_ROOT first (bin/gstack-state-root.sh). Bash 3.2 -# clean; no flock (macOS lacks it), so the lock is an atomic mkdir. +# clean; no flock (macOS lacks it), so the lock is gstack_lock_take +# (bin/gstack-lock.sh). # Same-path and containment checks (bin/gstack-path-spelling.sh). . "$(dirname "${BASH_SOURCE[0]}")/gstack-path-spelling.sh" 2>/dev/null || true +. "$(dirname "${BASH_SOURCE[0]}")/gstack-lock.sh" 2>/dev/null || gstack_lock_take() { echo "gstack: $(dirname "${BASH_SOURCE[0]}")/gstack-lock.sh is missing; reinstall with ./setup" >&2; return 2; } # gstack_install_move_hint SRC HOST — when HOST's runtime root ROOT is SRC (the # checkout) or contains it, `./setup --host HOST` would replace the checkout @@ -53,15 +55,17 @@ gstack_install_registry_file() { printf '%s/installs.tsv\n' "${GSTACK_STATE_ROOT:?gstack-install-registry: GSTACK_STATE_ROOT is not set}" } -# _gstack_registry_lock — take the registry lock (mkdir is atomic). A lock whose -# owner pid is gone is stale and is broken. Gives up after ~10 s. +# _gstack_registry_lock — take the registry lock (gstack_lock_take, which +# creates the pid file with O_EXCL; mkdir's exit status is not a lock on Ubuntu +# 26.04's uutils mkdir). A lock whose owner pid is gone is stale and is broken. +# Gives up after ~10 s. _gstack_registry_lock() { local file lock tries owner file="$(gstack_install_registry_file)" || return 1 lock="$file.lock" mkdir -p "$(dirname "$file")" || return 1 tries=0 - while ! mkdir "$lock" 2>/dev/null; do + while ! gstack_lock_take "$lock" pid "$$"; do owner="$(cat "$lock/pid" 2>/dev/null || true)" if [ -n "$owner" ] && ! kill -0 "$owner" 2>/dev/null; then rm -rf "$lock" @@ -74,7 +78,6 @@ _gstack_registry_lock() { fi sleep 0.1 done - echo "$$" > "$lock/pid" } _gstack_registry_unlock() { diff --git a/bin/gstack-lease b/bin/gstack-lease index 25b508700..ea6b05f42 100755 --- a/bin/gstack-lease +++ b/bin/gstack-lease @@ -55,6 +55,8 @@ BIN="$(cd "$(dirname "$0")" && pwd -P)" gstack_harness_pid() { printf '%s\n' "$1"; } gstack_shared_harness() { [ "$(ps -o comm= -p "$1" 2>/dev/null | head -1 | sed 's|.*/||; s/^-//')" = capy-server ]; } } +# shellcheck source=bin/gstack-lock.sh +. "$BIN/gstack-lock.sh" 2>/dev/null || gstack_lock_take() { echo "gstack-lease: $BIN/gstack-lock.sh is missing; reinstall with ./setup" >&2; return 2; } VERB="${1:-}"; [ $# -gt 0 ] && shift ROOT="" PID="" EXCLUDE="" FROM="" SHARED=0 RUN="" @@ -119,21 +121,22 @@ state() { # The lease directory's lock: touch's publish and prune, and reclaim's # reread-and-delete, run under it, so a reclaim never deletes a renewal and a # lease never disappears while it is live. The lock is a directory, .lock, -# whose owner file names its holder (` `); mkdir is -# atomic everywhere gstack runs, Git Bash on Windows included, where ln -s to a -# target that does not exist always fails (#3330). A taker publishes its token -# as a claim (.lock.claim.) before its mkdir and renames the claim into -# the lock as its owner after, so a lock without an owner is either being +# whose owner file names its holder (` `). It is taken +# with gstack_lock_take (bin/gstack-lock.sh), which creates the owner file +# with O_EXCL: mkdir's exit status is not a lock on Ubuntu 26.04's uutils +# mkdir, and ln -s to a target that does not exist always fails on Git Bash +# (#3330). A taker publishes its token as a claim (.lock.claim.) before +# its mkdir and creates the owner file after, so a lock without an owner is either being # taken by a live claimant or orphaned by a dead one. A lock is broken only # when its holder is dead or its pid now belongs to another process, never by # age, and unlock removes only the caller's own lock. An ownerless lock is # broken once no claimant lives and it is at least 2 s old (and unchanged -# while it was checked), so a taker between its mkdir and its owner rename is +# while it was checked), so a taker between its mkdir and its owner create is # never taken for a crash, even when its claim landed after the claims were # read. A claim is honored while its claimant lives; one that records no # start identity (so a reused pid cannot be told apart) is abandoned after 2 # minutes, since a live taker holds one for milliseconds. Known limit: on such -# a host a taker stalled over 2 minutes between its mkdir and its owner rename +# a host a taker stalled over 2 minutes between its mkdir and its owner create # can be broken and then publish into its successor's lock. An owner whose # pid is 0, zero-padded or not a number counts as a dead holder, and a lock # whose owner file exists but cannot be read is never treated as ownerless. A symlink .lock left by an older @@ -148,10 +151,7 @@ LOCK="$DIR/.lock" TOKEN="" take() { if [ -e "$1" ] || [ -L "$1" ]; then return 1; fi { printf '%s\n' "$TOKEN" > "$1.claim.$$"; } 2>/dev/null || { rm -f "$1.claim.$$"; return 1; } - if mkdir "$1" 2>/dev/null; then - [ ! -e "$1/owner" ] && mv -f "$1.claim.$$" "$1/owner" && return 0 - [ -e "$1/owner" ] || rm -rf "$1" - fi + if gstack_lock_take "$1" owner "$TOKEN"; then rm -f "$1.claim.$$"; return 0; fi rm -f "$1.claim.$$" return 1 } @@ -162,8 +162,9 @@ mtime() { stat -c %Y "$1" 2>/dev/null || stat -f %m "$1" 2>/dev/null; } # missing or readable and empty, has no live claimant, is at least 2 s old and kept # its mtime throughout. The clock is read before the first mtime, so a pause # only makes the lock look younger, and a lock made since (a new generation) -# has a newer mtime. Claims are read before the owner, so a taker that renames -# its claim in between is seen as the owner. An empty claim is a taker that +# has a newer mtime. Claims are read before the owner, and a taker removes its +# claim only after its owner file is written, so a taker that finishes in +# between is seen as the owner. An empty claim is a taker that # has not reached its mkdir, so it never made this lock. An mtime a minute or # more in the future (clock skew) counts as old. orphaned() { diff --git a/bin/gstack-lock.sh b/bin/gstack-lock.sh new file mode 100644 index 000000000..fee3e71ad --- /dev/null +++ b/bin/gstack-lock.sh @@ -0,0 +1,48 @@ +# shellcheck shell=bash +# gstack-lock.sh — the one way gstack shell code takes a lock directory. +# +# Source it, then call: +# +# gstack_lock_take DIR [FILE] [CONTENT] +# Take the lock DIR by creating DIR/FILE (default `pid`) holding CONTENT +# (default $$). Returns 0 when this shell took the lock, 1 when it is held +# (DIR already existed, or another process created FILE first), 2 when DIR +# could not be created (missing or unwritable parent, read-only filesystem) +# and 3 when DIR was made but FILE could not be created or fully written +# in it (disk full, quota, a file-size limit; FILE and DIR are removed +# again, so a failed write never leaves a lock nobody holds). Neither 2 +# nor 3 is evidence of a competing holder. +# +# Why not `mkdir DIR && ...`: mkdir's exit status is not a lock. Rust +# coreutils (uutils) 0.8.0, the mkdir on Ubuntu 26.04 and on GitHub's +# ubuntu-latest from 2026-11-09, reports success when another process created +# the directory first, so two racing callers could both think they held it. +# Here mkdir only makes the directory and its status is ignored. The arbiter is +# the shell's noclobber redirect, which bash, zsh and dash open with +# O_CREAT|O_EXCL themselves (no external binary), so the kernel lets exactly one +# caller create FILE. flock(1) is not used: stock macOS does not ship it. +# +# A DIR that exists before the call is never taken, even with no FILE in it, so +# a lock left by a holder killed between mkdir and FILE stays held until the +# caller's own stale-lock recovery removes it, exactly as before. Callers keep +# their own waiting, staleness, takeover and release logic; the owner still +# releases with `rm -rf DIR`. Safe under set -e and set -u, and noclobber never +# leaks into the caller's shell. +gstack_lock_take() { + local _gl_dir="$1" _gl_file="${2:-pid}" _gl_content="${3:-$$}" + if [ -e "$_gl_dir" ] || [ -L "$_gl_dir" ]; then return 1; fi + mkdir "$_gl_dir" 2>/dev/null || : + if [ ! -d "$_gl_dir" ] || [ -L "$_gl_dir" ]; then + if [ -e "$_gl_dir" ] || [ -L "$_gl_dir" ]; then return 1; fi + return 2 + fi + local _gl_opened + _gl_opened=$(set -o noclobber; { echo o; printf '%s\n' "$_gl_content" >&3; } 2>/dev/null 3> "$_gl_dir/$_gl_file") && return 0 + if [ "$_gl_opened" = o ]; then + rm -f "$_gl_dir/$_gl_file"; rmdir "$_gl_dir" 2>/dev/null || : + return 3 + fi + if [ -e "$_gl_dir/$_gl_file" ] || [ -L "$_gl_dir/$_gl_file" ] || [ ! -d "$_gl_dir" ]; then return 1; fi + rmdir "$_gl_dir" 2>/dev/null || : + return 3 +} diff --git a/bin/gstack-memorable b/bin/gstack-memorable index 558767b14..0556c0101 100755 --- a/bin/gstack-memorable +++ b/bin/gstack-memorable @@ -105,11 +105,14 @@ _lock_release() { _lock_acquire() { mkdir -p "$STATE_DIR/locks" 2>/dev/null || { _err "cannot create $STATE_DIR/locks (state directory not writable; nothing can be recorded there)"; return 5; } local tries=0 mtime now stale judged moved owner_pid - while ! mkdir "$LOCK_DIR" 2>/dev/null; do + . "$SCRIPT_DIR/gstack-lock.sh" 2>/dev/null || { _err "$SCRIPT_DIR/gstack-lock.sh is missing (fix: reinstall with ./setup or /gstack-upgrade)"; return 5; } + while ! gstack_lock_take "$LOCK_DIR" owner "$$"; do tries=$((tries + 1)) - # Staleness from the directory's own mtime (set atomically by the holder's - # mkdir), never from a file written after it: a contender that looks in - # the gap between mkdir and bookkeeping must wait, not reclaim. GNU stat + # Staleness from the directory's own mtime (set by the holder's mkdir and + # owner create), never from the owner's content: a contender that looks in + # the gap between mkdir and the owner file must wait, not reclaim. The + # owner file is the lock (gstack_lock_take creates it with O_EXCL, since + # mkdir's exit status is not a lock on Ubuntu 26.04's uutils mkdir). GNU stat # first, BSD stat second, garbage -> no takeover (same idiom as # bin/gstack-settings-hook). mtime="$(stat -c %Y "$LOCK_DIR" 2>/dev/null || stat -f %m "$LOCK_DIR" 2>/dev/null || echo "")" @@ -143,7 +146,6 @@ _lock_acquire() { if [ "$tries" -ge "$LOCK_TRIES" ]; then _err "another gstack-memorable is running (lock $LOCK_DIR; stale but not reclaimable if older than ${LOCK_STALE_S}s); try again"; return 5; fi sleep "$LOCK_SLEEP" done - printf '%s\n' "$$" > "$LOCK_DIR/owner" LOCK_HELD=1 trap _lock_release EXIT } diff --git a/bin/gstack-render-claude.sh b/bin/gstack-render-claude.sh index c7b06c4cd..f626461cc 100644 --- a/bin/gstack-render-claude.sh +++ b/bin/gstack-render-claude.sh @@ -1,5 +1,6 @@ # gstack-render-claude.sh — the Claude skill render shared by ./setup and -# `gstack-config gbrain-refresh`. Source it; it only defines functions. +# `gstack-config gbrain-refresh`. Source it; it only defines functions, and +# sources bin/gstack-lock.sh beside it for the render lock. # bash-3.2-clean and safe under `set -euo pipefail`. # # Overlay selection is opt-in: the overlay is the family of the persisted @@ -26,6 +27,8 @@ # their tested copy) and may define `log` and `bun_cmd`; plain echo and bun # are used otherwise. +. "$(dirname "${BASH_SOURCE[0]}")/gstack-lock.sh" 2>/dev/null || gstack_lock_take() { echo "gstack: $(dirname "${BASH_SOURCE[0]}")/gstack-lock.sh is missing; reinstall with ./setup" >&2; return 2; } + _gstack_render_say() { if declare -F log >/dev/null 2>&1; then log "$@"; else echo "$@"; fi } @@ -84,12 +87,13 @@ gstack_claude_overlay_field() { } # gstack_render_lock DIR / gstack_render_unlock DIR — serialize the swap and -# record write of concurrent renders (mkdir lock, no flock). A lock whose -# holder is gone is broken; after 30 s the swap proceeds unlocked. +# record write of concurrent renders (gstack_lock_take from bin/gstack-lock.sh, +# no flock). A lock whose holder is gone is broken; after 30 s the swap +# proceeds unlocked. gstack_render_lock() { local lock="$1.lock" i=0 pid mkdir -p "$(dirname "$1")" 2>/dev/null || true - while ! mkdir "$lock" 2>/dev/null; do + while ! gstack_lock_take "$lock" pid "$$"; do pid="$(cat "$lock/pid" 2>/dev/null || true)" if [ -n "$pid" ] && ! kill -0 "$pid" 2>/dev/null; then rm -rf "$lock" @@ -99,7 +103,6 @@ gstack_render_lock() { [ "$i" -lt 300 ] || return 0 sleep 0.1 done - echo "$$" > "$lock/pid" } gstack_render_unlock() { rm -rf "$1.lock" diff --git a/bin/gstack-session-update b/bin/gstack-session-update index 60ac42c99..78a8cdfa5 100755 --- a/bin/gstack-session-update +++ b/bin/gstack-session-update @@ -315,7 +315,10 @@ CALLER_PID=$(gstack_harness_pid "$PPID") # holder touches the pidfile at step boundaries (after the pull, after # setup), so a legitimately-slow run keeps itself alive. The TTL also # bounds the missing/empty-pidfile states: inside the window they mean - # "just acquired, between mkdir and echo" and are respected. + # "just acquired, between mkdir and the pid write" and are respected. + # The take itself is gstack_lock_take (bin/gstack-lock.sh): it creates the + # pidfile with O_EXCL, because mkdir's exit status is not a lock on Ubuntu + # 26.04's uutils mkdir (two racing updaters both "won" it). LOCK_TTL_MINUTES=30 lock_is_expired() { _hb="$LOCK_DIR/pid" @@ -327,11 +330,18 @@ CALLER_PID=$(gstack_harness_pid "$PPID") # mkdir (one rm can land between the other's rm and mkdir). `mv` of the lock # dir is atomic — exactly one contender's mv succeeds; the loser's mv fails # (ENOENT) and it backs off. The winner reaps the moved-aside dir at leisure. - if ! mkdir "$LOCK_DIR" 2>/dev/null; then + # + # The pidfile holds the HOLDER's PID (see #2613 note above; macOS ships + # bash 3.2 with no BASHPID — the sh child's $PPID IS this subshell, so the + # fallback is exact there). MYPID is captured once, before the take, so the + # trap below can prove ownership before removing. + MYPID="${BASHPID:-$(sh -c 'echo $PPID')}" + . "$(cd "$(dirname "$0")" && pwd)/gstack-lock.sh" 2>/dev/null || { log_entry "SKIP lock_helper_missing"; exit 0; } + if ! gstack_lock_take "$LOCK_DIR" pid "$MYPID"; then if lock_is_expired; then mv "$LOCK_DIR" "$LOCK_DIR.reap.$$" 2>/dev/null || { log_entry "SKIP lock_contested"; exit 0; } rm -rf "$LOCK_DIR.reap.$$" 2>/dev/null - mkdir "$LOCK_DIR" 2>/dev/null || { log_entry "SKIP lock_contested"; exit 0; } + gstack_lock_take "$LOCK_DIR" pid "$MYPID" || { log_entry "SKIP lock_contested"; exit 0; } log_entry "RECLAIMED lock_ttl_expired" elif [ -f "$LOCK_DIR/pid" ]; then LOCK_PID=$(cat "$LOCK_DIR/pid" 2>/dev/null || echo 0) @@ -339,7 +349,7 @@ CALLER_PID=$(gstack_harness_pid "$PPID") # Stale lock — mv aside atomically (see reclaim note above), re-acquire mv "$LOCK_DIR" "$LOCK_DIR.reap.$$" 2>/dev/null || { log_entry "SKIP lock_contested"; exit 0; } rm -rf "$LOCK_DIR.reap.$$" 2>/dev/null - mkdir "$LOCK_DIR" 2>/dev/null || { log_entry "SKIP lock_contested"; exit 0; } + gstack_lock_take "$LOCK_DIR" pid "$MYPID" || { log_entry "SKIP lock_contested"; exit 0; } else # Live holder — or an empty/non-numeric pidfile inside the TTL # window (the -gt test fails on garbage, landing here by design). @@ -347,19 +357,12 @@ CALLER_PID=$(gstack_harness_pid "$PPID") exit 0 fi else - # Missing pidfile inside the TTL window: just-acquired (mkdir→echo race). + # Missing pidfile inside the TTL window: just-acquired (mkdir→pid race). log_entry "SKIP locked_no_pid" exit 0 fi fi - # Write the HOLDER's PID for stale lock detection (see #2613 note above; - # macOS ships bash 3.2 with no BASHPID — the sh child's $PPID IS this - # subshell, so the fallback is exact there). MYPID is captured once at - # write time so the trap below can prove ownership before removing. - MYPID="${BASHPID:-$(sh -c 'echo $PPID')}" - echo "$MYPID" > "$LOCK_DIR/pid" 2>/dev/null - # In-flight heartbeat: the step-boundary touches below only fire AFTER the # pull / setup return, so a legitimately-slow step (cold clone, huge setup) # older than the TTL got reclaimed while ALIVE. This background loop diff --git a/bin/gstack-settings-hook b/bin/gstack-settings-hook index f7005384c..bf63cd84e 100755 --- a/bin/gstack-settings-hook +++ b/bin/gstack-settings-hook @@ -47,8 +47,10 @@ # treated as an owned legacy stray. # # Mutation safety: -# - every mutation runs under a mkdir lock (.lock/) with an owner -# token; release is ownership-checked; stale locks (>30s) are taken over +# - every mutation runs under a lock directory (.lock/) whose owner +# token file is created with O_EXCL (gstack_lock_take, bin/gstack-lock.sh; +# mkdir's exit status is not a lock on Ubuntu 26.04's uutils mkdir); +# release is ownership-checked; stale locks (>30s) are taken over # via atomic rename. On lock give-up the mutation is SKIPPED with a warning # (the next setup retries — the system is convergent). # - parse failure fails CLOSED: a corrupt settings.json is never overwritten @@ -353,18 +355,18 @@ _acquire_lock() { local stale_after_s=30 # lock older than this belongs to a crashed holder local poll_ms=50 # retry cadence; sleep below derives from this token="$$-$RANDOM$RANDOM" + . "$(cd "$(dirname "$0")" && pwd)/gstack-lock.sh" 2>/dev/null || { echo "gstack-settings-hook: $(dirname "$0")/gstack-lock.sh is missing -- skipping this mutation, exit 5 (fix: reinstall with ./setup)" >&2; return 1; } while :; do - if mkdir "$_LOCK_DIR" 2>/dev/null; then - printf '%s\n' "$token" > "$_LOCK_DIR/owner" 2>/dev/null || true + if gstack_lock_take "$_LOCK_DIR" owner "$token"; then _LOCK_TOKEN="$token" trap _release_lock EXIT trap 'exit 129' INT trap 'exit 143' TERM return 0 fi - # mkdir failed but no lock dir exists: either a holder released between - # our mkdir and this check, or NOT contention (unwritable parent, read-only - # fs, missing directory). Retry mkdir once; a second failure with still no + # The take failed but no lock dir exists: either a holder released between + # our take and this check, or NOT contention (unwritable parent, read-only + # fs, missing directory). Retry the take once; a second failure with still no # lock dir is the environment, and waiting cannot help, so give up loudly # now instead of spinning out the full timeout. if [ ! -e "$_LOCK_DIR" ]; then diff --git a/bin/gstack-upgrade b/bin/gstack-upgrade index 6f933b424..a335b0dd5 100755 --- a/bin/gstack-upgrade +++ b/bin/gstack-upgrade @@ -251,8 +251,10 @@ if [ -n "${GSTACK_SETUP_LOCK_OWNER:-}" ] && [ "$(cat "$LOCK_DIR/pid" 2>/dev/null LOCK_HANDED=1 else mkdir -p "$GSTACK_STATE_ROOT" 2>/dev/null - mkdir "$LOCK_DIR" 2>/dev/null || refuse "another gstack updater holds $LOCK_DIR (pid $(cat "$LOCK_DIR/pid" 2>/dev/null || echo unknown))" "wait for it to finish, then re-run" - echo "$$" > "$LOCK_DIR/pid" + . "$ROOT/bin/gstack-lock.sh" 2>/dev/null || refuse "$ROOT/bin/gstack-lock.sh is missing" "reinstall with cd $ROOT && ./setup" + # gstack_lock_take creates the pid file with O_EXCL: mkdir's exit status is + # not a lock on Ubuntu 26.04's uutils mkdir. + gstack_lock_take "$LOCK_DIR" pid "$$" || refuse "another gstack updater holds $LOCK_DIR (pid $(cat "$LOCK_DIR/pid" 2>/dev/null || echo unknown))" "wait for it to finish, then re-run" trap '[ "$(cat "$LOCK_DIR/pid" 2>/dev/null)" = "$$" ] && rm -rf "$LOCK_DIR" 2>/dev/null' EXIT fi diff --git a/changelog.d/capy-shell-locks-hold-when-mkdir-lies-gst-2247.md b/changelog.d/capy-shell-locks-hold-when-mkdir-lies-gst-2247.md new file mode 100644 index 000000000..baa8e6560 --- /dev/null +++ b/changelog.d/capy-shell-locks-hold-when-mkdir-lies-gst-2247.md @@ -0,0 +1,5 @@ +--- +important: blocker +--- +- **gstack's shell locks hold again on Ubuntu 26.04, where `mkdir` reports success when another process won the race (GST-2247).** Ubuntu 26.04, and GitHub's `ubuntu-latest` from 2026-11-09, ships Rust coreutils (uutils) 0.8.0 as `mkdir`. It exits 0 even when the directory already existed because a racing process just made it, so every `mkdir "$LOCK"` lock in gstack could admit two holders: two updates moving one checkout at once, or two `gstack-config set` calls where one dropped the other's key (a test with eight parallel writers under a uutils-style `mkdir` lost six of the eight keys). Every lock now goes through one helper, `gstack_lock_take` in `bin/gstack-lock.sh`: `mkdir` only makes the directory, and the holder is whoever creates the lock's pid or owner file with the shell's noclobber redirect, which bash, zsh and dash open `O_CREAT|O_EXCL`, so the kernel admits exactly one. No `flock` (macOS has none). This covers the session-start updater, `gstack-upgrade`, the Capy session pin switch, `gstack-ff-update` (inlined, pinned to the helper by a test, because it must run standalone), setup's dependency-migration and Chromium-install locks, `gstack-browser-ensure`, `gstack-config`, the install registry, the Claude render lock, the developer profile and its takeover lock, `gstack-memorable`, the settings.json hook, the brain-sync drain lock, the Codex model-probe lock and its breaker, the lease directory lock, freeze's state lock and /setup-gbrain's concurrent-run lock. The Claude Code wrapper's copied runtime (`CLAUDE_CODE_RUNTIME_FILES`) and `bin/gstack-render-claude.sh` now carry or source the helper themselves, so a copied runtime and every script that sources the render library still take the lock. Each caller keeps its own waiting, stale-lock recovery, timeout and release. If the owner file is created but can't be fully written (disk full, a quota or a file-size limit), the helper removes it and the directory again and returns 3, so a failed write never leaves a lock that nobody holds. A lock directory that already exists is still never taken, even with no owner file in it, so a lock left by a killed holder waits for the caller's existing stale rule, as before. +- `test/gstack-lock-race.test.ts` runs the old pattern and the helper against a PATH shim that behaves like uutils 0.8.0 `mkdir`, so it runs on any runner image: the old pattern lets two of four racers win, the helper admits exactly one in each of 60 races and never overlaps across take, hold and release loops, and parallel `gstack-config` writers keep every key. `test/mkdir-lock-guard.test.ts` fails when shipped shell, a template, a workflow, a Markdown bash fence or TypeScript that embeds shell decides anything on a non-`-p` `mkdir`'s exit status; it found 26 such sites before this change, 12 more than the first report listed. The /setup-gbrain lock test now expects `mkdir:` on stderr only when the parent cannot be created, because the helper never runs `mkdir` on a path that already exists; no other assertion, threshold or timeout changed. diff --git a/docs/capy-install.md b/docs/capy-install.md index 9c6e40e5a..df3aeaac0 100644 --- a/docs/capy-install.md +++ b/docs/capy-install.md @@ -254,9 +254,11 @@ by rename, prune) and reclaim (reread, then delete only the record judged stale) run under one lock in the lease directory, a `.lock` directory whose `owner` file names its holder's pid and start identity, so a renewal is never deleted and a live lease never disappears from a concurrent scan. The lock is a -directory because `mkdir` is atomic on every host, while Git Bash on Windows -cannot `ln -s` to a target that does not exist (#3330). A taker publishes a -claim before its `mkdir` and renames it in as the owner after, so an ownerless +directory taken with `gstack_lock_take` (`bin/gstack-lock.sh`), which creates +the `owner` file with O_EXCL, because `mkdir`'s exit status is no lock on +Ubuntu 26.04's uutils `mkdir` and Git Bash on Windows cannot `ln -s` to a +target that does not exist (#3330). A taker publishes a claim before its +`mkdir` and removes it only after its owner file is written, so an ownerless lock is broken only once no claimant lives and the lock is at least 2 s old and unchanged while checked, which keeps a taker between its two steps safe. A claim that records no start identity is abandoned after 2 minutes, so a diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 84064e69a..49b423b6f 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1882,6 +1882,36 @@ A `tracked-by-project` or `version-reverted` cause has its own section: listed changes, check out the default branch, move local commits to another branch), then re-run `bin/gstack-upgrade`. A reinstall that failed after the fast-forward is completed by the next run (`reinstall pending from `). +For `another gstack updater holds /.setup-lock (pid )`, wait +for that update to finish. If no gstack update is running (`kill -0 ` +fails, or the pid reads `unknown` because the holder was killed before it +wrote one), remove the lock with `rm -rf /.setup-lock` and +re-run. See [how gstack's shell locks work](#shell-locks). + + +### Two gstack updates, setups or config writes ran at once (Ubuntu 26.04, uutils `mkdir`) + +**Meaning.** gstack's shell locks (the updater's `.setup-lock`, the config +lock `.config-yaml.lock`, the install registry, the developer profile, +settings.json, the Chromium install, the brain-sync drain, freeze and the +lease directory) are lock directories. Older gstack held one when +`mkdir ` succeeded. Rust coreutils (uutils) 0.8.0, the `mkdir` on +Ubuntu 26.04 and on GitHub's `ubuntu-latest` from 2026-11-09, reports success +even when another process created the directory first, so two racing callers +could both hold the lock: two updates moving one checkout, or two +`gstack-config set` calls where one write dropped the other's key. GNU and +BSD `mkdir` never did this. + +**Fix.** Upgrade gstack (`/gstack-upgrade`). Every lock is now taken by +`gstack_lock_take` in `bin/gstack-lock.sh`: `mkdir` only makes the +directory, and the holder is whoever creates its `pid` (or `owner`) file +with the shell's noclobber redirect, which opens it `O_CREAT|O_EXCL`, so +exactly one caller wins whatever `mkdir` reports. Waiting, stale-lock +recovery and release are unchanged. A lock directory with no owner file +inside, left by a process killed while taking it, still counts as held until +the caller's own stale-lock rule clears it; when nothing is running, remove it +with `rm -rf `. Re-set any config key you set during a concurrent write +on 26.04 (`gstack-config get ` shows what stuck). ### `gstack-upgrade: live gstack sessions hold this installation: pid

(m), ...; nothing was changed; fix: ... (UPGRADE_BUSY)` / `gstack-upgrade: a shared harness (capy-server, pid

) holds a lease m old; ... (UPGRADE_BUSY)` diff --git a/freeze/bin/freeze-state.sh b/freeze/bin/freeze-state.sh index ad61b80cd..98582fa11 100755 --- a/freeze/bin/freeze-state.sh +++ b/freeze/bin/freeze-state.sh @@ -15,7 +15,8 @@ mutex="$STATE_DIR/.freeze-mutation.lock" action="${1:-}" case "$action" in acquire|set|release|clear) ;; *) echo 'Usage: freeze-state.sh acquire|set DIRECTORY | release OWNER | clear' >&2; exit 2 ;; esac -if ! mkdir "$mutex" 2>/dev/null; then +. "$_here/../../bin/gstack-lock.sh" 2>/dev/null || { echo "FREEZE_ERROR: $_here/../../bin/gstack-lock.sh is missing. No boundary changed. Fix: reinstall with ./setup or /gstack-upgrade." >&2; exit 1; } +if ! gstack_lock_take "$mutex" pid "$$"; then echo 'FREEZE_BUSY: another writer or an interrupted mutation owns the state lock. Retry after it finishes; if abandoned, inspect it with the user before recovery. No boundary changed.' >&2 exit 1 fi @@ -23,7 +24,7 @@ temp="" finish() { local rc=$? [ -z "$temp" ] || rm -f -- "$temp" - rmdir "$mutex" + rm -rf -- "$mutex" exit "$rc" } trap finish EXIT diff --git a/lib/claude-code-migration.ts b/lib/claude-code-migration.ts index 3dd45f229..31c7da30f 100644 --- a/lib/claude-code-migration.ts +++ b/lib/claude-code-migration.ts @@ -9,7 +9,7 @@ const OLD = 'gstack-claude'; const NEXT = 'gstack-claude-code'; const BANNER = '