fix(gstack-lease): take the lease lock with mkdir, not ln -s, so Git Bash on Windows stops waiting out 300 tries (#3330)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-25 23:01:59 +00:00
1 parent c23d7d48fe
commit 03acc162d1
5 files changed
+302 -48

No files matched your search

+16
View File
@@ -1,5 +1,21 @@
# Changelog
## [Unreleased]
**Skills start at once on Windows again: the workflow lease lock no longer depends on symlinks (#3330).** `bin/gstack-lease` took its lock with `ln -s "<holder token>" .lock`. In Git for Windows' default symlink mode (`MSYS` unset), `ln -s` cannot link to a target that does not exist, so the lock was never taken. Every skill start then waited out all 300 tries (0.1 s apart, plus process start costs) before `gstack-skill-start` printed anything, and still wrote no lease. One report measured 499 s per skill start. The error also blamed a "live process" with an empty pid.
### Fixed
- **`bin/gstack-lease`**: the lock is now a `.lock` directory taken with `mkdir`, which is atomic on every host gstack runs on, with an `owner` file naming the holder's pid and start identity. A taker writes a claim (`.lock.claim.<pid>`) before its `mkdir` and renames it in as the owner after, so a lock left ownerless by a taker killed between the two steps is broken once every claimant is dead, never by age. A lock is removed by renaming it aside first, so nobody sees it half-removed. Stale-holder semantics are unchanged: a lock is broken only when its holder is dead or its pid now belongs to another process, breakers still serialize on `.lock.break` and reread before removing, and unlock removes only the caller's own lock. A symlink `.lock` left by an older gstack is still read and broken the same way, and it excludes the new lock at the same path, so mixed versions never both hold it.
- When the lock cannot be taken although nobody holds it, `touch` now fails after three quick retries with `gstack-lease: cannot take the lease directory lock <path>` instead of waiting out every try and naming an empty pid. `gstack-skill-start` stays fail-open, as before.
- `docs/capy-install.md` describes the directory lock.
### Tests
- `test/gstack-lease-git-bash.test.ts` (new, in the curated windows-free-tests lane, launched through `lib/script-invocation.ts`, so Git Bash on `windows-latest`): `touch` with the default 300 tries writes its lease and leaves no lock; a host whose `ln -s` fails like Git Bash's still takes the lock (with a control that the shim really fails `ln -s`); a lock held by a live Git Bash process is kept and named, and a dead holder's lock is broken. The last two fail against the old bin.
- `test/gstack-lease.test.ts`: the dead-holder, live-holder and dead-breaker cases run against both the directory lock and the legacy symlink lock. The race cases inject at `mkdir` instead of `ln`. New cases: a lock held by a running gstack-lease excludes a second taker, which names it; 16 parallel touches all land and leave nothing behind; an ownerless lock is kept while a claimant lives and broken once none does; a lock that cannot be taken while nobody holds it fails in under 10 s. Seven of these fail against the old bin.
### Gate integrity
- No threshold, budget, timeout, trial count, case kind or assertion was weakened. Three existing race tests moved their fault injection from `ln` to `mkdir`, because the lock no longer calls `ln`. Their assertions are unchanged.
## [1.91.221.0] - 2026-10-25
**The session-start updater no longer skips because of its own session's workflow lease, and the autostash tests stop failing their first pass in Free Tests.** Since #3385, `gstack-session-update` skips the fast-forward while another session holds a workflow lease, and it means to ignore the session that started it (`--exclude-ancestry-of`). The check runs in the forked background updater, though, and passed the hook's own `$$`, which has exited by then, so the ancestry walk excluded nothing. A Claude Code process that had already run a skill (and so held a lease) and then started a session again in the same process (`/clear`, resume or compaction) logged `SKIP leased pid=<its own pid>` and never updated. `test/session-update-autostash.test.ts` hit the same gap plus a fixture leak: its first two fixtures spawned the updater with the whole parent environment, so it read the test preload's `GSTACK_LEASE_DIR`, which every test file in the same Bun process shares. A lease another file left there gave "Received: SKIP leased pid=3196/1m" where UPDATING or PULL_FAILED was expected, in 13 Free Tests runs since #3385 merged (each passed on the serial retry).
+85 -26
View File
@@ -31,7 +31,7 @@
# runtime root's .source-path, else ../). A root that is not a git checkout
# exits 2 and writes nothing. GSTACK_LEASE_DIR overrides the lease directory
# (hermetic tests; a dev checkout's own tests must not lease it). Bash 3.2
# plus git, ps, find, ln, readlink.
# plus git, ps, find, mkdir, mv, readlink.
set -u
BIN="$(cd "$(dirname "$0")" && pwd -P)"
# shellcheck source=bin/gstack-install-verdict.sh
@@ -99,43 +99,97 @@ state() {
}
# The lease directory's lock: touch's publish and prune, and reclaim's
# reread-and-delete, run under it, so a reclaim never deletes a renewal and a
# lease never disappears while it is live. The lock is a symlink whose target
# names its holder (`<pid> <start identity>`): ln -s creates it and its owner
# in one atomic step. It is broken only when that holder is dead or its pid
# now belongs to another process, never by age, and unlock removes only the
# caller's own lock. A live holder makes lock() give up after
# GSTACK_LEASE_LOCK_TRIES tries 0.1 s apart (default 300).
# lease never disappears while it is live. The lock is a directory, .lock,
# whose owner file names its holder (`<pid> <start identity>`); mkdir is
# atomic everywhere gstack runs, Git Bash on Windows included, where ln -s to a
# target that does not exist always fails (#3330). A taker publishes its token
# as a claim (.lock.claim.<pid>) before its mkdir and renames the claim into
# the lock as its owner after, so a lock without an owner is either being
# taken by a live claimant or orphaned by a dead one. A lock is broken only
# when its holder (or, ownerless, every claimant) is dead or its pid now
# belongs to another process, never by age, and unlock removes only the
# caller's own lock. A symlink .lock left by an older gstack names its holder
# the same way and excludes a directory lock at the same path. A live holder
# makes lock() give up after GSTACK_LEASE_LOCK_TRIES tries 0.1 s apart
# (default 300).
LOCK="$DIR/.lock" TOKEN=""
# take PATH — create the lock PATH owned by TOKEN; 1 when it exists.
take() {
printf '%s\n' "$TOKEN" > "$1.claim.$$" || return 1
if mkdir "$1" 2>/dev/null; then
mv -f "$1.claim.$$" "$1/owner" && return 0
rm -rf "$1"
fi
rm -f "$1.claim.$$"
return 1
}
# owner_of PATH — the token the lock PATH names, empty when it names none.
owner_of() { if [ -L "$1" ]; then readlink "$1" 2>/dev/null; else head -1 "$1/owner" 2>/dev/null; fi; }
# orphaned PATH — PATH is a lock directory without an owner and no live
# claimant. Claims are read before the owner, so a taker that renames its
# claim in between is seen as the owner. An empty claim is a taker that has
# not reached its mkdir, so it never made this lock.
orphaned() {
[ -d "$1" ] && [ ! -L "$1" ] || return 1
for _c in "$1".claim.*; do
_t=$(head -1 "$_c" 2>/dev/null)
[ -n "$_t" ] && ! holder_gone "$_t" && return 1
done
[ ! -e "$1/owner" ]
}
# drop PATH — remove a lock in one step: a directory is renamed aside first,
# so nobody sees it ownerless on the way out.
drop() {
if [ -L "$1" ]; then rm -f "$1"; return; fi
mv "$1" "$1.gone.$$" 2>/dev/null && rm -rf "$1.gone.$$"
}
lock() {
[ -n "$TOKEN" ] || TOKEN="$$ $(identity $$)"
_tries=0
until ln -s "$TOKEN" "$LOCK" 2>/dev/null; do
_tries=$((_tries + 1))
LOCK_OWNER=$(readlink "$LOCK" 2>/dev/null)
[ -n "$LOCK_OWNER" ] && holder_gone "$LOCK_OWNER" && break_lock "$LOCK_OWNER" && continue
_tries=0 _gone=0
until take "$LOCK"; do
LOCK_OWNER=$(owner_of "$LOCK")
if [ ! -e "$LOCK" ] && [ ! -L "$LOCK" ]; then
# Released since take, or take itself cannot work here: never wait out
# the tries for a lock nobody holds.
_gone=$((_gone + 1))
[ "$_gone" -ge 3 ] && return 1
continue
fi
_gone=0 _tries=$((_tries + 1))
if [ -n "$LOCK_OWNER" ]; then holder_gone "$LOCK_OWNER" && break_lock "$LOCK_OWNER" && continue
elif orphaned "$LOCK"; then break_lock "" && continue
fi
[ "$_tries" -ge "${GSTACK_LEASE_LOCK_TRIES:-300}" ] && return 1
sleep 0.1
done
}
holder_gone() { case "$(state "${1%% *}" "${1#* }" 0)" in dead|stale) return 0 ;; esac; return 1; }
# break_lock OWNER — remove the lock only if it still names OWNER. Breakers
# take a second symlink, .lock.break, and reread the lock under it, so a lock
# taken since the verdict is never removed; a .lock.break whose holder is gone
# is cleared. Returns 1 when another breaker holds .lock.break. Known limit:
# that clear is not atomic, so two gstack-lease processes killed in a row (one
# holding .lock, the next inside this function) can let a reclaim delete a
# renewal; one crash is always safe.
# break_lock OWNER — remove the lock only if it still names OWNER (empty: is
# still orphaned). Breakers take a second lock, .lock.break, and reread the
# lock under it, so a lock taken since the verdict is never removed; a
# .lock.break whose holder is gone is cleared. Returns 1 when another breaker
# holds .lock.break. Known limit: that clear is not atomic, so two
# gstack-lease processes killed in a row (one holding .lock, the next inside
# this function) can let a reclaim delete a renewal; one crash is always safe.
break_lock() {
if ! ln -s "$TOKEN" "$LOCK.break" 2>/dev/null; then
_b=$(readlink "$LOCK.break" 2>/dev/null)
[ -n "$_b" ] && holder_gone "$_b" && [ "$(readlink "$LOCK.break" 2>/dev/null)" = "$_b" ] && rm -f "$LOCK.break"
if ! take "$LOCK.break"; then
_b=$(owner_of "$LOCK.break")
if [ -n "$_b" ]; then
holder_gone "$_b" && [ "$(owner_of "$LOCK.break")" = "$_b" ] && drop "$LOCK.break"
else
orphaned "$LOCK.break" && drop "$LOCK.break"
fi
return 1
fi
[ "$(readlink "$LOCK" 2>/dev/null)" = "$1" ] && rm -f "$LOCK"
rm -f "$LOCK.break"
if [ -n "$1" ]; then
[ "$(owner_of "$LOCK")" = "$1" ] && drop "$LOCK"
else
orphaned "$LOCK" && drop "$LOCK"
fi
drop "$LOCK.break"
return 0
}
unlock() { [ "$(readlink "$LOCK" 2>/dev/null)" = "$TOKEN" ] && rm -f "$LOCK"; return 0; }
unlock() { [ "$(owner_of "$LOCK")" = "$TOKEN" ] && { drop "$LOCK" || rm -rf "$LOCK"; }; return 0; }
# reclaim FILE RECORD — delete a stale lease only if, under the lock, it still
# holds the record judged stale; a renewal since then keeps it.
reclaim() {
@@ -153,7 +207,12 @@ case "$VERB" in
touch)
[ -n "$PID" ] || PID=$(gstack_harness_pid "${FROM:-$PPID}")
mkdir -p "$DIR" || exit 2
lock || { echo "gstack-lease: the lease directory is locked by live process ${LOCK_OWNER%% *}; lease $PID not renewed" >&2; exit 3; }
lock || {
if [ -n "${LOCK_OWNER:-}" ]; then echo "gstack-lease: the lease directory is locked by live process ${LOCK_OWNER%% *}; lease $PID not renewed" >&2
else echo "gstack-lease: cannot take the lease directory lock $LOCK; lease $PID not renewed" >&2
fi
exit 3
}
_tmp="$DIR/.$PID.$$"
identity "$PID" > "$_tmp" && mv -f "$_tmp" "$DIR/$PID" || { rm -f "$_tmp"; unlock; exit 2; }
find "$DIR" -mmin +120 -type f -exec rm -f {} + 2>/dev/null
+10 -5
View File
@@ -120,11 +120,16 @@ lease …`). Stale needs proof: two well-formed identities from the same backend
that differ (`ps` is read under `LC_ALL=C TZ=UTC0`). An identity that cannot be
read, a malformed record or a backend change keeps the lease. `touch` (publish
by rename, prune) and reclaim (reread, then delete only the record judged
stale) run under one lock in the lease directory, a symlink naming its holder's
pid and start identity, so a renewal is never deleted and a live lease never
disappears from a concurrent scan. The lock is broken only when its holder is
dead or its pid was reused, never by age, and breakers serialize on a second
symlink and reread the lock before removing it. One killed gstack-lease process
stale) run under one lock in the lease directory, a `.lock` directory whose
`owner` file names its holder's pid and start identity, so a renewal is never
deleted and a live lease never disappears from a concurrent scan. The lock is a
directory because `mkdir` is atomic on every host, while Git Bash on Windows
cannot `ln -s` to a target that does not exist (#3330). A taker publishes a
claim before its `mkdir` and renames it in as the owner after, so an ownerless
lock is broken only once every claimant is dead. The lock is broken only when
its holder is dead or its pid was reused, never by age, and breakers serialize
on a second lock and reread the lock before removing it. A symlink `.lock` left
by an older gstack is honored the same way. One killed gstack-lease process
is always safe; two killed in a row, one holding the lock and the next while
breaking it, can still let a reclaim delete a renewal. While a live process holds it,
`touch` exits 3 without writing (`gstack-skill-start` reports that the
+86
View File
@@ -0,0 +1,86 @@
/**
* #3330: gstack-lease took its lock with `ln -s <holder token> .lock`. Git
* for Windows' default symlink mode (MSYS unset) cannot link to a target that
* does not exist, so the lock was never taken and every skill start waited out
* 300 tries (one report: 499 s) and still wrote no lease. The lock is now a
* directory (mkdir) with an owner file. test/gstack-lease.test.ts pins the full
* contract but is POSIX-bound (mode bits, sleep children), so this file is the
* one the curated windows-free-tests lane runs: it launches the bin through
* lib/script-invocation.ts, which is Git Bash on windows-latest.
*/
import { afterEach, describe, expect, test } from 'bun:test';
import { spawnSync } from 'child_process';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { BIN_DIR, runBashSync, runScriptSync } from '../lib/script-invocation';
const LEASE = path.join(BIN_DIR, 'gstack-lease');
const fwd = (p: string) => p.replace(/\\/g, '/');
const dirs: string[] = [];
afterEach(() => {
for (const d of dirs.splice(0)) fs.rmSync(d, { recursive: true, force: true });
});
// No inherited lease dir from the test preload, and Git Bash's default symlink mode (no winsymlinks).
const { GSTACK_LEASE_DIR: _preload, MSYS: _msys, CYGWIN: _cygwin, GSTACK_LEASE_LOCK_TRIES: _tries, ...baseEnv } = process.env as Record<string, string>;
function checkout() {
const dir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-lease-gb-')));
dirs.push(dir);
const r = spawnSync('git', ['init', '-q', dir], { encoding: 'utf8', timeout: 30_000 });
if (r.status !== 0) throw new Error(r.stderr);
return { dir, leases: path.join(dir, '.git', 'gstack-leases') };
}
/** A BASH_ENV file in which `ln -s` behaves as Git Bash's default: linking to a missing target fails. */
function gitBashLn(dir: string): string {
const file = path.join(dir, 'git-bash-ln.sh');
fs.writeFileSync(file, 'ln() { if [ "$1" = -s ] && [ ! -e "$2" ]; then echo "ln: failed to create symbolic link \'$3\': No such file or directory" >&2; return 1; fi; command ln "$@"; }\n');
return file;
}
describe('gstack-lease under Git Bash semantics (#3330)', () => {
test('touch takes the lock and writes the lease at once with the default tries, leaving no lock behind', () => {
const { dir, leases } = checkout();
const started = Date.now();
const r = runScriptSync(LEASE, ['touch', '--root', fwd(dir), '--pid', '4242'], { encoding: 'utf8', env: baseEnv, timeout: 120_000 });
expect(r.status, `${r.error ?? ''}${r.stdout}${r.stderr}`).toBe(0);
expect(Date.now() - started).toBeLessThan(25_000);
expect(fs.readdirSync(leases)).toEqual(['4242']);
});
test('a host whose ln -s cannot link to a missing target still takes the lock (negative control: the shim really fails ln -s)', () => {
const { dir, leases } = checkout();
const shim = gitBashLn(dir);
const probe = runBashSync('ln -s "1 2" "$PROBE"', { encoding: 'utf8', env: { ...baseEnv, BASH_ENV: fwd(shim), PROBE: fwd(path.join(dir, 'probe')) }, timeout: 30_000 });
expect(probe.status, `${probe.error ?? ''}${probe.stderr}`).not.toBe(0);
expect(probe.stderr).toContain('failed to create symbolic link');
const r = runScriptSync(LEASE, ['touch', '--root', fwd(dir), '--pid', '4243'], { encoding: 'utf8', env: { ...baseEnv, BASH_ENV: fwd(shim), GSTACK_LEASE_LOCK_TRIES: '3' }, timeout: 120_000 });
expect(r.status, `${r.error ?? ''}${r.stdout}${r.stderr}`).toBe(0);
expect(fs.readdirSync(leases)).toEqual(['4243']);
});
test('a lock held by a live Git Bash process is kept (touch exits 3 naming it); a dead holder\'s lock is broken', () => {
const { dir, leases } = checkout();
const script = [
'set -u',
'mkdir -p "$LEASES/.lock" && printf "%s \\n" "$$" > "$LEASES/.lock/owner"',
'GSTACK_LEASE_LOCK_TRIES=2 bash "$LEASE" touch --root "$ROOT" --pid 4244; echo "held=$? holder=$$"',
'( exit 0 ) & dead=$!; wait "$dead"',
'printf "%s \\n" "$dead" > "$LEASES/.lock/owner"',
'GSTACK_LEASE_LOCK_TRIES=2 bash "$LEASE" touch --root "$ROOT" --pid 4245; echo "dead=$?"',
].join('\n');
const r = runBashSync(script, {
encoding: 'utf8', timeout: 120_000,
env: { ...baseEnv, LEASE: fwd(LEASE), ROOT: fwd(dir), LEASES: fwd(leases) },
});
const out = `${r.error ?? ''}${r.stdout}${r.stderr}`;
expect(r.status, out).toBe(0);
const holder = /held=(\d+) holder=(\d+)/.exec(r.stdout);
expect(holder?.[1], out).toBe('3');
expect(r.stderr).toContain(`locked by live process ${holder?.[2]}`);
expect(r.stdout, out).toContain('dead=0');
expect(fs.readdirSync(leases).sort()).toEqual(['4245']);
});
});
+105 -17
View File
@@ -59,6 +59,19 @@ function otherStart(recorded: string): string {
return recorded.startsWith('linux ') ? recorded.replace(/\d+$/, (n) => String(Number(n) + 1)) : 'ps Thu Jan 1 00:00:00 1970';
}
/** A directory lock as gstack-lease takes it: <path>/owner holds the holder's token. */
function plantLock(lockPath: string, token: string) {
fs.mkdirSync(lockPath, { recursive: true });
fs.writeFileSync(path.join(lockPath, 'owner'), token + '\n');
}
const ownerOf = (lockPath: string) => fs.readFileSync(path.join(lockPath, 'owner'), 'utf8').trim();
/** Each lock layout a holder can leave: the directory lock, and the symlink lock an older gstack took. */
const LAYOUTS: Array<[string, (lockPath: string, token: string) => void]> = [
['directory lock', plantLock],
['legacy symlink lock', (lockPath, token) => fs.symlinkSync(token, lockPath)],
];
/** A BASH_ENV file whose functions shadow commands inside the lease script (fault injection). */
function bashEnv(dir: string, body: string): string {
const file = path.join(dir, 'bash-env.sh');
@@ -160,7 +173,7 @@ describe('gstack-lease', () => {
const renewed = path.join(dir, 'renewed');
fs.writeFileSync(renewed, fresh);
// The holder renews just before reclaim takes the lock, as if touch ran after the stale verdict.
const racing = bashEnv(dir, `ln() { case "$*" in *"/.lock") command cp "${renewed}" "${file}" ;; esac; command ln "$@"; }\n`);
const racing = bashEnv(dir, `mkdir() { case "$*" in *"/.lock") command cp "${renewed}" "${file}" ;; esac; command mkdir "$@"; }\n`);
const r = run(['live', '--root', dir], { BASH_ENV: racing });
expect(r.status, r.stdout + r.stderr).toBe(0);
expect(r.stderr).not.toContain('reclaimed');
@@ -168,23 +181,24 @@ describe('gstack-lease', () => {
expect(fs.readdirSync(leases).filter((n) => n.startsWith('.'))).toEqual([]);
});
test('a lock whose holder died, or whose pid now belongs to another process, is broken at once', () => {
test.each(LAYOUTS)('a %s whose holder died, or whose pid now belongs to another process, is broken at once', (_layout, plant) => {
const { dir, leases } = checkout();
const holder = sleeper();
fs.mkdirSync(leases, { recursive: true });
fs.symlinkSync(`${deadPid()} `, path.join(leases, '.lock'));
plant(path.join(leases, '.lock'), `${deadPid()} `);
const r = run(['touch', '--root', dir, '--pid', String(holder)], { GSTACK_LEASE_LOCK_TRIES: '3' });
expect(r.status, r.stdout + r.stderr).toBe(0);
expect(fs.existsSync(path.join(leases, '.lock'))).toBe(false);
const identity = fs.readFileSync(path.join(leases, String(holder)), 'utf8').trim();
expect(identity).toMatch(/^(linux|ps) /);
fs.symlinkSync(`${holder} ${otherStart(identity)}`, path.join(leases, '.lock'));
plant(path.join(leases, '.lock'), `${holder} ${otherStart(identity)}`);
const again = run(['touch', '--root', dir, '--pid', String(holder)], { GSTACK_LEASE_LOCK_TRIES: '3' });
expect(again.status, again.stdout + again.stderr).toBe(0);
expect(fs.existsSync(path.join(leases, '.lock'))).toBe(false);
expect(fs.readdirSync(leases).filter((n) => n.startsWith('.'))).toEqual([]);
});
test('a live holder\'s lock is never broken, however old: touch exits 3 without writing and a reclaim keeps the lease', () => {
test.each(LAYOUTS)('a live holder\'s %s is never broken, however old: touch exits 3 without writing and a reclaim keeps the lease', (layout, plant) => {
const { dir, leases } = checkout();
const owner = sleeper();
const holder = sleeper();
@@ -192,9 +206,10 @@ describe('gstack-lease', () => {
expect(run(['touch', '--root', dir, '--pid', String(holder)]).status).toBe(0);
const ownerToken = `${owner} ${fs.readFileSync(path.join(leases, String(owner)), 'utf8').trim()}`;
const lockPath = path.join(leases, '.lock');
fs.symlinkSync(ownerToken, lockPath);
plant(lockPath, ownerToken);
const old = new Date(Date.now() - 5 * 60_000);
fs.lutimesSync(lockPath, old, old);
if (layout === 'directory lock') fs.utimesSync(path.join(lockPath, 'owner'), old, old);
const holderFile = path.join(leases, String(holder));
const stale = otherStart(fs.readFileSync(holderFile, 'utf8').trim()) + '\n';
fs.writeFileSync(holderFile, stale);
@@ -207,7 +222,7 @@ describe('gstack-lease', () => {
expect(l.stderr).not.toContain('reclaimed');
expect(l.stdout).toContain(`${holder} `);
expect(fs.existsSync(holderFile)).toBe(true);
expect(fs.readlinkSync(lockPath)).toBe(ownerToken);
expect(layout === 'directory lock' ? ownerOf(lockPath) : fs.readlinkSync(lockPath)).toBe(ownerToken);
});
test('two breakers of one dead lock: the second rereads under .lock.break and leaves the lock the first one took', () => {
@@ -217,21 +232,21 @@ describe('gstack-lease', () => {
expect(run(['touch', '--root', dir, '--pid', String(owner)]).status).toBe(0);
const ownerToken = `${owner} ${fs.readFileSync(path.join(leases, String(owner)), 'utf8').trim()}`;
const lockPath = path.join(leases, '.lock');
fs.symlinkSync(`${deadPid()} `, lockPath);
plantLock(lockPath, `${deadPid()} `);
// Between this breaker's dead verdict and its .lock.break, another breaker removes the dead lock and takes it.
const first = bashEnv(dir, `ln() { case "$*" in *"/.lock.break") command rm -f "${lockPath}"; command ln -s "${ownerToken}" "${lockPath}" ;; esac; command ln "$@"; }\n`);
const first = bashEnv(dir, `mkdir() { case "$*" in *"/.lock.break") command rm -rf "${lockPath}"; command mkdir "${lockPath}"; printf '%s\\n' "${ownerToken}" > "${lockPath}/owner" ;; esac; command mkdir "$@"; }\n`);
const t = run(['touch', '--root', dir, '--pid', String(holder)], { BASH_ENV: first, GSTACK_LEASE_LOCK_TRIES: '3' });
expect(t.status, t.stdout + t.stderr).toBe(3);
expect(fs.readlinkSync(lockPath)).toBe(ownerToken);
expect(fs.existsSync(path.join(leases, '.lock.break'))).toBe(false);
expect(ownerOf(lockPath)).toBe(ownerToken);
expect(fs.readdirSync(leases).filter((n) => n.startsWith('.')).sort()).toEqual(['.lock']);
});
test('a .lock.break left by a dead breaker is cleared and the dead lock is still broken', () => {
test.each(LAYOUTS)('a .lock.break (%s) left by a dead breaker is cleared and the dead lock is still broken', (_layout, plant) => {
const { dir, leases } = checkout();
const holder = sleeper();
fs.mkdirSync(leases, { recursive: true });
fs.symlinkSync(`${deadPid()} `, path.join(leases, '.lock'));
fs.symlinkSync(`${deadPid()} `, path.join(leases, '.lock.break'));
plant(path.join(leases, '.lock'), `${deadPid()} `);
plant(path.join(leases, '.lock.break'), `${deadPid()} `);
const r = run(['touch', '--root', dir, '--pid', String(holder)], { GSTACK_LEASE_LOCK_TRIES: '5' });
expect(r.status, r.stdout + r.stderr).toBe(0);
expect(fs.existsSync(path.join(leases, String(holder)))).toBe(true);
@@ -249,10 +264,83 @@ describe('gstack-lease', () => {
fs.writeFileSync(holderFile, otherStart(fs.readFileSync(holderFile, 'utf8').trim()) + '\n');
const lockPath = path.join(leases, '.lock');
// While reclaim deletes the stale lease under its lock, the lock changes hands.
const swap = bashEnv(dir, `rm() { if [ "$*" = "-f ${holderFile}" ]; then command rm -f "${lockPath}"; command ln -s "${ownerToken}" "${lockPath}"; fi; command rm "$@"; }\n`);
const swap = bashEnv(dir, `rm() { if [ "$*" = "-f ${holderFile}" ]; then command rm -rf "${lockPath}"; command mkdir "${lockPath}"; printf '%s\\n' "${ownerToken}" > "${lockPath}/owner"; fi; command rm "$@"; }\n`);
const r = run(['live', '--root', dir, '--exclude-ancestry-of', String(owner)], { BASH_ENV: swap });
expect(r.stderr).toContain(`reclaimed stale lease ${holder}`);
expect(fs.readlinkSync(lockPath)).toBe(ownerToken);
expect(ownerOf(lockPath)).toBe(ownerToken);
});
test('a lock held by a running gstack-lease excludes a second taker, which names it; both leases land once it is released', async () => {
const { dir, leases } = checkout();
const [first, second] = [sleeper(), sleeper()];
// The first touch holds the lock across its prune (find runs under the lock).
const slow = bashEnv(dir, 'find() { sleep 3; command find "$@"; }\n');
const holder = spawn('bash', [LEASE, 'touch', '--root', dir, '--pid', String(first)], { env: { ...baseEnv, BASH_ENV: slow }, stdio: 'ignore' });
children.push(holder);
const exited = new Promise<number | null>((resolve) => holder.on('exit', resolve));
const owner = path.join(leases, '.lock', 'owner');
for (let i = 0; i < 100 && !fs.existsSync(owner); i++) await Bun.sleep(20);
expect(fs.readFileSync(owner, 'utf8')).toMatch(new RegExp(`^${holder.pid} `));
const blocked = run(['touch', '--root', dir, '--pid', String(second)], { GSTACK_LEASE_LOCK_TRIES: '3' });
expect(blocked.status, blocked.stdout + blocked.stderr).toBe(3);
expect(blocked.stderr).toContain(`locked by live process ${holder.pid};`);
expect(await exited).toBe(0);
expect(run(['touch', '--root', dir, '--pid', String(second)]).status).toBe(0);
expect(fs.readdirSync(leases).sort()).toEqual([String(first), String(second)].sort());
});
test('parallel touches all take the lock in turn and leave no lock, claim or temp file behind', async () => {
const { dir, leases } = checkout();
const pids = Array.from({ length: 16 }, (_, i) => 4300 + i);
const codes = await Promise.all(pids.map((p) => new Promise<number | null>((resolve) => {
const c = spawn('bash', [LEASE, 'touch', '--root', dir, '--pid', String(p)], { env: baseEnv, stdio: 'ignore' });
children.push(c);
c.on('exit', resolve);
})));
expect(codes).toEqual(pids.map(() => 0));
expect(fs.readdirSync(leases).sort()).toEqual(pids.map(String).sort());
});
// #3330: a taker killed between its mkdir and its owner rename leaves an ownerless lock.
test('an ownerless lock is broken once every claimant is gone, and kept while a live claimant may still own it', () => {
const { dir, leases } = checkout();
const holder = sleeper();
const lockPath = path.join(leases, '.lock');
fs.mkdirSync(lockPath, { recursive: true });
fs.writeFileSync(path.join(leases, `.lock.claim.${holder}`), `${holder} \n`);
const kept = run(['touch', '--root', dir, '--pid', String(holder)], { GSTACK_LEASE_LOCK_TRIES: '3' });
expect(kept.status, kept.stdout + kept.stderr).toBe(3);
expect(kept.stderr).toContain('cannot take the lease directory lock');
expect(fs.existsSync(path.join(leases, String(holder)))).toBe(false);
expect(fs.readdirSync(lockPath)).toEqual([]);
const dead = deadPid();
fs.renameSync(path.join(leases, `.lock.claim.${holder}`), path.join(leases, `.lock.claim.${dead}`));
fs.writeFileSync(path.join(leases, `.lock.claim.${dead}`), `${dead} \n`);
fs.writeFileSync(path.join(leases, '.lock.claim.1'), '');
const broken = run(['touch', '--root', dir, '--pid', String(holder)], { GSTACK_LEASE_LOCK_TRIES: '3' });
expect(broken.status, broken.stdout + broken.stderr).toBe(0);
expect(fs.existsSync(path.join(leases, String(holder)))).toBe(true);
expect(fs.existsSync(lockPath)).toBe(false);
fs.rmSync(path.join(leases, `.lock.claim.${dead}`));
fs.mkdirSync(lockPath);
const unclaimed = run(['touch', '--root', dir, '--pid', String(holder)], { GSTACK_LEASE_LOCK_TRIES: '3' });
expect(unclaimed.status, unclaimed.stdout + unclaimed.stderr).toBe(0);
expect(fs.existsSync(lockPath)).toBe(false);
});
test('a lock that cannot be taken while nobody holds it fails at once instead of waiting out the tries', () => {
const { dir, leases } = checkout();
const holder = sleeper();
const broken = bashEnv(dir, 'mkdir() { case "$*" in *"/.lock") return 1 ;; esac; command mkdir "$@"; }\n');
const started = Date.now();
const r = run(['touch', '--root', dir, '--pid', String(holder)], { BASH_ENV: broken });
expect(r.status, r.stdout + r.stderr).toBe(3);
expect(Date.now() - started).toBeLessThan(10_000);
expect(r.stderr).toContain('cannot take the lease directory lock');
expect(r.stderr).not.toContain('locked by live process');
expect(fs.readdirSync(leases)).toEqual([]);
});
test('a lease without a recorded identity (older gstack) counts while fresh and is stale past the 120-minute window', () => {
@@ -333,7 +421,7 @@ describe('gstack-lease', () => {
fs.writeFileSync(path.join(state, 'config.yaml'), 'update_check: false\n');
const leaseDir = path.join(state, 'leases');
fs.mkdirSync(leaseDir);
fs.symlinkSync(`${sleeper()} `, path.join(leaseDir, '.lock'));
plantLock(path.join(leaseDir, '.lock'), `${sleeper()} `);
const r = spawnSync(START, ['--skill', 'testskill', '--parent-pid', '424244'], {
encoding: 'utf8', cwd: home, env: { PATH: process.env.PATH!, HOME: home, GSTACK_HOME: state, GSTACK_LEASE_DIR: leaseDir, GSTACK_LEASE_LOCK_TRIES: '2' }, timeout: 30_000,
});