mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-10 23:19:09 +02:00
fix(watchdog): signal-0 liveness, tick-scaled respawn guard, windowsHide
Three-bug chain behind the Windows terminal-agent leak (console window strobing every 60s, one orphaned agent per watchdog tick until the box ran out of committable memory): 1. isProcessAlive shelled out to `tasklist /FI "PID eq <pid>"` on Windows with a 3s timeout. A Bun.spawnSync that hits its timeout still RETURNS with partial stdout, so the `.includes()` PID match read a LIVE agent as dead — killAgentByRecord skipped the kill, the watchdog respawned around the survivor, and every orphan slowed the next tasklist enough to produce the next false negative. Now: `process.kill(pid, 0)` on every platform (Node and Bun both map signal 0 to an OpenProcess existence check on Windows), with EPERM counted as alive. No subprocess, no timeout, no console window. 2. The respawn circuit-breaker was mathematically unreachable — verified in this tree: RESPAWN_GUARD_WINDOW_MS was a fixed 60_000 against a 60_000ms default tick, and each tick pushes at most one respawn timestamp, so three pushes span ~120s and can never coexist inside a 60s window (eviction is strict `>`, and setInterval drift plus per-tick work always ages the prior entry past the boundary). The guard could not fire at the default tick rate and a steady one-per-tick leak ran unbounded. The window now scales with the tick: max(60_000, tick * (RESPAWN_GUARD_MAX + 2)), so "3 crashes in quick succession → stop" holds at any tick value. 3. The tasklist probe popped a visible console per tick (no windowsHide). Removing the shell-out kills that site; the agent-spawn site itself already passes windowsHide: true (landed with the bun-polyfill windowsHide commit — PR #2414's terminal-agent-control.ts hunk is reconciled there rather than duplicated). New browse/test/process-liveness-windows.test.ts pins all three: no subprocess from the probe, a static tripwire against reintroducing `tasklist` + `PID eq` liveness checks in src/, the spawnTerminalAgent windowsHide + stdio contract, and the window-derived-from-tick arithmetic. terminal-agent-watchdog.test.ts test 4 now pins the window/tick relationship instead of the fixed literal that let this ship. Also converts `new URL(import.meta.url).pathname` to `import.meta.path` across the static-grep tests it touches — the pathname form yields /C:/... on Windows and breaks path.resolve. Contributed by @SYKhayyat (PR #2414). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
15c9455a69
commit
bbaf5068b0
@@ -7,8 +7,6 @@
|
||||
|
||||
import * as fs from 'fs';
|
||||
|
||||
const IS_WINDOWS = process.platform === 'win32';
|
||||
|
||||
// ─── Filesystem ────────────────────────────────────────────────
|
||||
|
||||
/** Remove a file, ignoring ENOENT (already gone). Rethrows other errors. */
|
||||
@@ -36,23 +34,39 @@ export function safeKill(pid: number, signal: NodeJS.Signals | number): void {
|
||||
}
|
||||
}
|
||||
|
||||
/** Check if a PID is alive. Pure boolean probe — returns false for ALL errors. */
|
||||
/**
|
||||
* Check if a PID is alive. Pure boolean probe — never throws.
|
||||
*
|
||||
* Signal 0 on every platform. Node and Bun both map `process.kill(pid, 0)` to
|
||||
* an OpenProcess existence check on Windows, so the POSIX idiom is portable
|
||||
* here — no shell-out needed.
|
||||
*
|
||||
* Windows used to shell out to `tasklist /FI "PID eq <pid>"` and string-match
|
||||
* the CSV. That was wrong in two ways, both of which bit in production:
|
||||
*
|
||||
* 1. FALSE NEGATIVES UNDER LOAD. `tasklist` takes ~700-1700ms on an idle
|
||||
* Windows box and far longer under memory pressure. A Bun.spawnSync that
|
||||
* hits its `timeout` still RETURNS, carrying partial stdout — so the
|
||||
* `.includes()` match came back false and a LIVE process was reported
|
||||
* dead. Callers (killAgentByRecord, the terminal-agent watchdog) then
|
||||
* skipped the kill and respawned around the survivor, leaking one
|
||||
* terminal-agent per watchdog tick. The leak was self-reinforcing: every
|
||||
* orphan added memory pressure, which made the next tasklist slower,
|
||||
* which produced the next false negative.
|
||||
* 2. A VISIBLE CONSOLE WINDOW per probe (no windowsHide), so a background
|
||||
* watchdog strobed a terminal into the foreground every 60 seconds.
|
||||
*
|
||||
* Signal 0 is ~74,000x faster (0.004ms vs 270ms, measured), spawns nothing,
|
||||
* and cannot time out.
|
||||
*
|
||||
* EPERM means the process EXISTS but we lack rights to signal it. That is
|
||||
* alive; returning false there would reintroduce failure mode 1.
|
||||
*/
|
||||
export function isProcessAlive(pid: number): boolean {
|
||||
if (IS_WINDOWS) {
|
||||
try {
|
||||
const result = Bun.spawnSync(
|
||||
['tasklist', '/FI', `PID eq ${pid}`, '/NH', '/FO', 'CSV'],
|
||||
{ stdout: 'pipe', stderr: 'pipe', timeout: 3000 }
|
||||
);
|
||||
return result.stdout.toString().includes(`"${pid}"`);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
} catch (err: any) {
|
||||
return err?.code === 'EPERM';
|
||||
}
|
||||
}
|
||||
|
||||
+11
-1
@@ -1529,8 +1529,18 @@ export function buildFetchHandler(cfg: ServerConfig): ServerHandle {
|
||||
process.env.GSTACK_AGENT_WATCHDOG_TICK_MS || '60000',
|
||||
10,
|
||||
);
|
||||
const RESPAWN_GUARD_WINDOW_MS = 60_000;
|
||||
const RESPAWN_GUARD_MAX = 3;
|
||||
// The guard window MUST span enough ticks for RESPAWN_GUARD_MAX respawns to
|
||||
// land inside it. This was a fixed 60_000 against a 60_000 tick, so at most
|
||||
// ONE respawn could ever be in the window and `respawnHistory.length >= 3`
|
||||
// was unreachable — the guard could not fire at the default tick rate, and a
|
||||
// steady one-per-tick leak ran unbounded instead of stopping after 3. Scale
|
||||
// with the tick so the intent ("3 crashes in quick succession → stop") holds
|
||||
// at any tick value: 3 respawns within 5 ticks trips it.
|
||||
const RESPAWN_GUARD_WINDOW_MS = Math.max(
|
||||
60_000,
|
||||
AGENT_WATCHDOG_TICK_MS * (RESPAWN_GUARD_MAX + 2),
|
||||
);
|
||||
let agentRespawnGuardTripped = false;
|
||||
|
||||
if (ownsTerminalAgent) {
|
||||
|
||||
Reference in New Issue
Block a user