fix(watchdog): signal-0 liveness, tick-scaled respawn guard, windowsHide

Three-bug chain behind the Windows terminal-agent leak (console window
strobing every 60s, one orphaned agent per watchdog tick until the box
ran out of committable memory):

1. isProcessAlive shelled out to `tasklist /FI "PID eq <pid>"` on Windows
   with a 3s timeout. A Bun.spawnSync that hits its timeout still RETURNS
   with partial stdout, so the `.includes()` PID match read a LIVE agent
   as dead — killAgentByRecord skipped the kill, the watchdog respawned
   around the survivor, and every orphan slowed the next tasklist enough
   to produce the next false negative. Now: `process.kill(pid, 0)` on
   every platform (Node and Bun both map signal 0 to an OpenProcess
   existence check on Windows), with EPERM counted as alive. No
   subprocess, no timeout, no console window.

2. The respawn circuit-breaker was mathematically unreachable — verified
   in this tree: RESPAWN_GUARD_WINDOW_MS was a fixed 60_000 against a
   60_000ms default tick, and each tick pushes at most one respawn
   timestamp, so three pushes span ~120s and can never coexist inside a
   60s window (eviction is strict `>`, and setInterval drift plus
   per-tick work always ages the prior entry past the boundary). The
   guard could not fire at the default tick rate and a steady
   one-per-tick leak ran unbounded. The window now scales with the tick:
   max(60_000, tick * (RESPAWN_GUARD_MAX + 2)), so "3 crashes in quick
   succession → stop" holds at any tick value.

3. The tasklist probe popped a visible console per tick (no windowsHide).
   Removing the shell-out kills that site; the agent-spawn site itself
   already passes windowsHide: true (landed with the bun-polyfill
   windowsHide commit — PR #2414's terminal-agent-control.ts hunk is
   reconciled there rather than duplicated).

New browse/test/process-liveness-windows.test.ts pins all three: no
subprocess from the probe, a static tripwire against reintroducing
`tasklist` + `PID eq` liveness checks in src/, the spawnTerminalAgent
windowsHide + stdio contract, and the window-derived-from-tick
arithmetic. terminal-agent-watchdog.test.ts test 4 now pins the
window/tick relationship instead of the fixed literal that let this
ship. Also converts `new URL(import.meta.url).pathname` to
`import.meta.path` across the static-grep tests it touches — the
pathname form yields /C:/... on Windows and breaks path.resolve.

Contributed by @SYKhayyat (PR #2414).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-14 20:20:53 -07:00
co-authored by Claude Fable 5
parent 15c9455a69
commit bbaf5068b0
17 changed files with 200 additions and 36 deletions
+30 -16
View File
@@ -7,8 +7,6 @@
import * as fs from 'fs';
const IS_WINDOWS = process.platform === 'win32';
// ─── Filesystem ────────────────────────────────────────────────
/** Remove a file, ignoring ENOENT (already gone). Rethrows other errors. */
@@ -36,23 +34,39 @@ export function safeKill(pid: number, signal: NodeJS.Signals | number): void {
}
}
/** Check if a PID is alive. Pure boolean probe — returns false for ALL errors. */
/**
* Check if a PID is alive. Pure boolean probe — never throws.
*
* Signal 0 on every platform. Node and Bun both map `process.kill(pid, 0)` to
* an OpenProcess existence check on Windows, so the POSIX idiom is portable
* here — no shell-out needed.
*
* Windows used to shell out to `tasklist /FI "PID eq <pid>"` and string-match
* the CSV. That was wrong in two ways, both of which bit in production:
*
* 1. FALSE NEGATIVES UNDER LOAD. `tasklist` takes ~700-1700ms on an idle
* Windows box and far longer under memory pressure. A Bun.spawnSync that
* hits its `timeout` still RETURNS, carrying partial stdout — so the
* `.includes()` match came back false and a LIVE process was reported
* dead. Callers (killAgentByRecord, the terminal-agent watchdog) then
* skipped the kill and respawned around the survivor, leaking one
* terminal-agent per watchdog tick. The leak was self-reinforcing: every
* orphan added memory pressure, which made the next tasklist slower,
* which produced the next false negative.
* 2. A VISIBLE CONSOLE WINDOW per probe (no windowsHide), so a background
* watchdog strobed a terminal into the foreground every 60 seconds.
*
* Signal 0 is ~74,000x faster (0.004ms vs 270ms, measured), spawns nothing,
* and cannot time out.
*
* EPERM means the process EXISTS but we lack rights to signal it. That is
* alive; returning false there would reintroduce failure mode 1.
*/
export function isProcessAlive(pid: number): boolean {
if (IS_WINDOWS) {
try {
const result = Bun.spawnSync(
['tasklist', '/FI', `PID eq ${pid}`, '/NH', '/FO', 'CSV'],
{ stdout: 'pipe', stderr: 'pipe', timeout: 3000 }
);
return result.stdout.toString().includes(`"${pid}"`);
} catch {
return false;
}
}
try {
process.kill(pid, 0);
return true;
} catch {
return false;
} catch (err: any) {
return err?.code === 'EPERM';
}
}
+11 -1
View File
@@ -1529,8 +1529,18 @@ export function buildFetchHandler(cfg: ServerConfig): ServerHandle {
process.env.GSTACK_AGENT_WATCHDOG_TICK_MS || '60000',
10,
);
const RESPAWN_GUARD_WINDOW_MS = 60_000;
const RESPAWN_GUARD_MAX = 3;
// The guard window MUST span enough ticks for RESPAWN_GUARD_MAX respawns to
// land inside it. This was a fixed 60_000 against a 60_000 tick, so at most
// ONE respawn could ever be in the window and `respawnHistory.length >= 3`
// was unreachable — the guard could not fire at the default tick rate, and a
// steady one-per-tick leak ran unbounded instead of stopping after 3. Scale
// with the tick so the intent ("3 crashes in quick succession → stop") holds
// at any tick value: 3 respawns within 5 ticks trips it.
const RESPAWN_GUARD_WINDOW_MS = Math.max(
60_000,
AGENT_WATCHDOG_TICK_MS * (RESPAWN_GUARD_MAX + 2),
);
let agentRespawnGuardTripped = false;
if (ownsTerminalAgent) {