Test runs whose workers start in their own session are no longer killed as stalled (GST-3213) (#4533)

/ship's test check no longer kills a quiet, working test run as stalled
when the test command starts its workers in their own session, as pnpm
11's `pnpm exec` does for Vitest.

Why: the stall check read CPU time from the command's process group
only, where just the idle launcher was left; it now counts every process
descended from the command.

Nothing to do: a truly hung run is still killed at the same stall
timeout, and every threshold is unchanged.

Co-authored-by: dtoelke <14206719+dtoelke@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-19 16:54:44 +00:00
1 parent 104fc72757
commit 50cc282273
8 files changed
+156 -36

No files matched your search

@@ -0,0 +1 @@
- **/ship no longer kills a working test run as stalled when the test command starts its workers in their own session.** The stall check read CPU time from the test command's process group only, so a quiet suite run through a launcher that starts its workers in a new session (pnpm 11's `pnpm exec` running Vitest) looked idle and was killed as `PREGATE_STALLED`. It now counts every process descended from the command, and a stall or timeout kill reaches those workers too. A truly hung run is still killed at the same stall timeout. Reported by @dtoelke in #4477.
+12 -6
View File
@@ -540,14 +540,18 @@ test-tier command:
a one-file run (each serial file) reports its file from the launch for any
runner. Other runners (node --test, vitest, jest, pytest, playwright test, flutter test, dart test, cargo test) get heartbeats
only. A heartbeat follows each 60 s with no progress line;
- is killed as **stalled** when its log does not grow and its process group
- is killed as **stalled** when its log does not grow and its process tree
stays under a tenth of one CPU for the stall timeout (`--stall-timeout <s>`, else
`stall_timeout_s`, else 300; `0` disables): the check is `incomplete` with
`PREGATE_STALLED` and names the last file started; the next serial file
still runs. The group counts as busy when its CPU time, sampled each second,
still runs. The process tree is the command's process group plus every
descendant reached by parent links, whatever session it moved to, so a
worker a launcher starts in its own session (pnpm 11's `pnpm exec` running
Vitest) counts. The tree counts as busy when its CPU time, sampled each second,
grew by at least a tenth of the wall time, so a quiet suite that is working
(FOUN-63: a parallel `check.sh` silent for 9 minutes) is not killed, while
a hang that waits on nothing is. A hang that keeps burning CPU is left to
a hang that waits on nothing is. A descendant reparented away from the tree
(a double-forked daemon) no longer counts. A hang that keeps burning CPU is left to
the timeout. Linux reads `/proc` (exited children's time included once
reaped); macOS and other BSDs read `ps -S`; a failed read skips that
sample. Windows reads no CPU time here and uses log growth alone. The `--remote` gate is exempt: a
@@ -558,9 +562,11 @@ test-tier command:
serial files that each fit still pass): the check is `incomplete`, its detail
names which of the three set the timeout, and the next serial file still runs.
A kill is SIGTERM to the group, then SIGKILL after 5 s (a group that ignores
SIGTERM still dies). SIGINT or SIGTERM to `gstack-pregate` itself SIGKILLs every
running group, writes no `pregate.json` and exits 130 or 143. Windows has no
A stall or timeout kill is SIGTERM to the process group of every tree member,
then SIGKILL to the same groups after 5 s (a group that ignores SIGTERM still
dies, and a worker in its own session is not left running). After a normal exit
only the command's own group is killed. SIGINT or SIGTERM to `gstack-pregate`
itself SIGKILLs every running tree, writes no `pregate.json` and exits 130 or 143. Windows has no
process groups here: the kill is `taskkill /T /F` on the shell's tree, which
misses a descendant that was reparented or started detached.
+2 -2
View File
@@ -4846,9 +4846,9 @@ missing checks needs a full `gstack-pregate --allow-repo-commands` (neither
<a id="pregate-stalled"></a>
### `PREGATE_STALLED`
**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log, and its process group used under a tenth of one CPU (Windows: log growth alone), for the stall timeout (default 300 s,
**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log, and its process tree (its process group plus every descendant, whatever session it moved to) used under a tenth of one CPU (Windows: log growth alone), for the stall timeout (default 300 s,
`stall_timeout_s` in the base branch's `.gstack/pregate.json` or the user-owned file, or
`--stall-timeout`). pregate killed its process group and names the last test
`--stall-timeout`). pregate killed its process tree and names the last test
file started. The check is incomplete, never pass.
**Fix.** Open the printed log, find the hang in the named file (a blocked
+1 -1
View File
@@ -90,7 +90,7 @@ async function run(ctx: PregateContext): Promise<CheckResult> {
for (const r of runs) {
if (r.carryNone) lines.push(`CI_CARRY: none guard=${r.guard.id} (${r.carryNone})`);
if (r.prior) { lines.push(`GUARD: ${r.guard.id} ${r.prior}`); continue; }
lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`);
lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process tree killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`);
if (r.exit !== 0) lines.push(...r.tail.map(l => ` ${l}`));
if (r.environment) lines.push(` ${r.guard.id} ${environmentLine(r.environment)}`);
}
+1 -1
View File
@@ -453,7 +453,7 @@ function unfinished(ctx: PregateContext, r: Awaited<ReturnType<typeof runLocal>>
const timedOut = r.runs.filter(x => x.outcome === 'timeout');
const rest = r.failed ? `; ${r.failed} failed` : '';
if (stalled.length) {
return { code: 'PREGATE_STALLED', detail: `no test output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed; last file started: ${stalled.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest}`, fix: `read ${r.log} for the stalled file; fix the hang, or raise --stall-timeout (stall_timeout_s in ${ctx.configFile}; 0 disables) for a test that waits quietly for longer` };
return { code: 'PREGATE_STALLED', detail: `no test output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process tree killed; last file started: ${stalled.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest}`, fix: `read ${r.log} for the stalled file; fix the hang, or raise --stall-timeout (stall_timeout_s in ${ctx.configFile}; 0 disables) for a test that waits quietly for longer` };
}
const launch = r.runs.find(x => x.outcome === 'spawn-failed');
if (launch) {
+65 -23
View File
@@ -3,11 +3,14 @@
* own process group, stdout and stderr appended to a log file (no pipes, so
* a backgrounded grandchild holding the output can never keep the check
* waiting), supervised asynchronously. A stall is `stallMs` with no log growth
* and no CPU activity in the process group (GST-1095: a quiet parallel suite
* that is busy working is alive; a hang that keeps burning CPU is left to the
* deadline); the deadline is the check's `--timeout`. Either kills the whole
* process group (SIGTERM, then SIGKILL after a grace period) and the run is
* incomplete. After a normal exit the group's leftover members are killed.
* and no CPU activity in the process tree: the runner's process group plus
* every descendant by parent links, whatever session it moved to (GST-1095: a
* quiet parallel suite that is busy working is alive, #4477: so is a worker
* `pnpm exec` starts in its own session; a hang that keeps burning CPU is left
* to the deadline); the deadline is the check's `--timeout`. Either kills the
* group of every tree member (SIGTERM, then SIGKILL after a grace period) and
* the run is incomplete. After a normal exit only the runner's group's leftover
* members are killed: a worker in its own session is no longer linked to it.
*
* Progress: a runner with a progress adapter (`bun test`: its `<file>:`
* headers) reports `PREGATE_TEST: <n>/<total> <file> start|pass|fail <s>`
@@ -16,7 +19,7 @@
* with no progress line; with `lastLine` it names the last line the command printed.
*
* Platforms: POSIX signals the negative pid (the group `detached` creates
* with setsid). Windows has no process groups here: the kill is
* with setsid) and each tree member's own group. Windows has no process groups here: the kill is
* `taskkill /T /F` on the shell's tree, which misses a descendant that was
* reparented or started detached.
*/
@@ -70,12 +73,12 @@ const USER_HZ = 100;
const CPU_SAMPLE_MS = 1_000;
const CPU_BUSY_SHARE = 0.1;
/** What a stall means beyond silence, for the PREGATE_STALLED detail and the guard line; Windows reads no CPU time. */
export const STALL_IDLE = process.platform === 'win32' ? '' : ' and its process group used under a tenth of one CPU';
export const STALL_IDLE = process.platform === 'win32' ? '' : ' and its process tree used under a tenth of one CPU';
/** SIGKILL every supervised process group still running (the CLI's SIGINT/SIGTERM handler, before it exits). */
/** SIGKILL every supervised process tree still running (the CLI's SIGINT/SIGTERM handler, before it exits). */
export function cancelActive(): number {
const pids = [...ACTIVE];
for (const pid of pids) killGroup(pid, 'SIGKILL');
for (const pid of pids) killTree(pid, 'SIGKILL');
ACTIVE.clear();
return pids.length;
}
@@ -88,29 +91,67 @@ export function killGroup(pid: number, signal: NodeJS.Signals): void {
try { process.kill(-pid, signal); } catch { /* group already gone */ }
}
/** CPU time in ms used by the process group `pgid`, reaped children included where the platform reports them; null where it cannot be read (Windows). */
export function groupCpuMs(pgid: number): number | null {
export interface Proc { pid: number; ppid: number; pgid: number; cpuMs: number }
/** Every process with its parent, group and CPU time in ms (reaped children included where the platform reports them); null where it cannot be read (Windows, a failed `ps`). */
export function processTable(): Proc[] | null {
if (process.platform === 'win32') return null;
const out: Proc[] = [];
if (fs.existsSync('/proc/self/stat')) {
let ticks = 0;
for (const name of fs.readdirSync('/proc')) {
if (!/^\d+$/.test(name)) continue;
let stat: string;
try { stat = fs.readFileSync(`/proc/${name}/stat`, 'utf8'); } catch { continue; }
const f = stat.slice(stat.lastIndexOf(')') + 2).split(' ');
if (Number(f[2]) === pgid) ticks += Number(f[11]) + Number(f[12]) + Number(f[13]) + Number(f[14]);
out.push({ pid: Number(name), ppid: Number(f[1]), pgid: Number(f[2]), cpuMs: (Number(f[11]) + Number(f[12]) + Number(f[13]) + Number(f[14])) * 1000 / USER_HZ });
}
return ticks * 1000 / USER_HZ;
return out;
}
const ps = spawnSync('ps', ['-A', '-S', '-o', 'pgid=,time='], { encoding: 'utf8', timeout: 15_000 });
const ps = spawnSync('ps', ['-A', '-S', '-o', 'pid=,ppid=,pgid=,time='], { encoding: 'utf8', timeout: 15_000 });
if (ps.status !== 0) return null;
let ms = 0;
for (const line of ps.stdout.split('\n')) {
const [id, time] = line.trim().split(/\s+/);
const [pid, ppid, pgid, time] = line.trim().split(/\s+/);
const t = time && /^(?:(\d+)-)?(?:(\d+):)?(\d+):(\d+(?:\.\d+)?)$/.exec(time);
if (Number(id) === pgid && t) ms += (((Number(t[1] ?? 0) * 24 + Number(t[2] ?? 0)) * 60 + Number(t[3])) * 60 + Number(t[4])) * 1000;
if (t) out.push({ pid: Number(pid), ppid: Number(ppid), pgid: Number(pgid), cpuMs: (((Number(t[1] ?? 0) * 24 + Number(t[2] ?? 0)) * 60 + Number(t[3])) * 60 + Number(t[4])) * 1000 });
}
return ms;
return out;
}
/** The runner `root`'s process tree: the members of its process group plus every descendant by parent links, whatever session or group it moved to. */
export function processTree(root: number, procs: readonly Proc[]): Proc[] {
const children = new Map<number, Proc[]>();
for (const p of procs) { const c = children.get(p.ppid); if (c) c.push(p); else children.set(p.ppid, [p]); }
const tree = new Map<number, Proc>();
const queue = procs.filter(p => p.pgid === root);
while (queue.length) {
const p = queue.pop()!;
if (tree.has(p.pid)) continue;
tree.set(p.pid, p);
queue.push(...(children.get(p.pid) ?? []));
}
return [...tree.values()];
}
/** CPU time in ms used by the runner `root`'s process tree; null where it cannot be read (Windows). */
export function treeCpuMs(root: number): number | null {
const procs = processTable();
return procs && processTree(root, procs).reduce((ms, p) => ms + p.cpuMs, 0);
}
/**
* Signal the runner `root`'s process group and the group of each tree member
* outside it, so a stall or timeout kill leaves no worker that moved to its own
* session running. `known` carries groups an earlier call found, for a runner
* that has since exited. Never signals pid 1's group or this process's own
* group. Returns every group signalled. Windows: `taskkill /T` on the shell.
*/
export function killTree(root: number, signal: NodeJS.Signals, known: Iterable<number> = []): Set<number> {
const groups = new Set<number>([root, ...known]);
const procs = processTable();
const own = procs?.find(p => p.pid === process.pid)?.pgid;
if (procs) for (const p of processTree(root, procs)) groups.add(p.pgid);
for (const g of groups) if (g > 1 && g !== own && g !== process.pid) killGroup(g, signal);
return groups;
}
/** The adapter that reads per-file progress from this runner's output, or null when it has none. */
@@ -194,18 +235,19 @@ export function supervise(o: SuperviseOptions): Promise<Supervised> {
let outcome: Supervised['outcome'] | null = null;
let settled = false;
let escalate: ReturnType<typeof setTimeout> | null = null;
let killed = new Set<number>();
const stop = (why: Supervised['outcome']) => {
if (outcome || !child.pid) return;
outcome = why;
killGroup(child.pid, 'SIGTERM');
escalate = setTimeout(() => killGroup(child.pid!, 'SIGKILL'), graceMs);
killed = killTree(child.pid, 'SIGTERM');
escalate = setTimeout(() => { killed = killTree(child.pid!, 'SIGKILL', killed); }, graceMs);
};
const finish = (exit: number | null, signal: string | null, error?: string) => {
if (settled) return;
settled = true;
clearInterval(timer);
if (escalate) clearTimeout(escalate);
if (child.pid) { ACTIVE.delete(child.pid); if (process.platform !== 'win32') killGroup(child.pid, 'SIGKILL'); }
if (child.pid) { ACTIVE.delete(child.pid); if (process.platform !== 'win32') { if (outcome) killTree(child.pid, 'SIGKILL', killed); else killGroup(child.pid, 'SIGKILL'); } }
try { size = fs.statSync(o.log).size; } catch { /* log removed */ }
readNew();
const done: Supervised['outcome'] = error ? 'spawn-failed' : (outcome ?? 'exited');
@@ -220,7 +262,7 @@ export function supervise(o: SuperviseOptions): Promise<Supervised> {
if (outcome) return;
if (now >= o.deadline) { stop('timeout'); return; }
if (o.stallMs > 0 && process.platform !== 'win32' && now - cpu.at >= CPU_SAMPLE_MS) {
const used = groupCpuMs(child.pid!);
const used = treeCpuMs(child.pid!);
if (used !== null && used - cpu.ms >= CPU_BUSY_SHARE * (now - cpu.at)) lastBusy = now;
cpu = { at: now, ms: used ?? cpu.ms };
}
+1 -1
View File
@@ -411,7 +411,7 @@ export const RESULT_CODES = {
},
PREGATE_STALLED: {
anchor: 'pregate-stalled',
summary: 'a test command printed nothing for the stall timeout while its process group used under a tenth of one CPU (log growth alone on Windows), so the group was killed; the check is incomplete, never pass',
summary: 'a test command printed nothing for the stall timeout while its process tree used under a tenth of one CPU (log growth alone on Windows), so the tree was killed; the check is incomplete, never pass',
fix: 'read the lane log for the last file started, fix the hang, or raise `--stall-timeout` (`stall_timeout_s`; 0 disables) for a test that waits quietly for longer, then rerun `gstack-pregate --stage tests`',
},
PREGATE_RECEIPT_INVALID: {
+73 -2
View File
@@ -13,12 +13,13 @@ import * as os from 'node:os';
import * as path from 'node:path';
import { EXIT } from '../lib/headless-artifacts';
import { buildContext, withFingerprint } from '../lib/pregate/context';
import { adapterFor, supervise, type SuperviseOptions } from '../lib/pregate/supervise';
import { adapterFor, processTree, supervise, type SuperviseOptions } from '../lib/pregate/supervise';
import { BASE_PREGATE, BIN, cleanup, commit, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/pregate-fixture';
const dirs: string[] = [];
const repos: FixtureRepo[] = [];
afterEach(() => { while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); while (repos.length) cleanup(repos.pop()!); });
const workers: number[] = [];
afterEach(() => { while (workers.length) { try { process.kill(workers.pop()!, 'SIGKILL'); } catch { /* already gone */ } } while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); while (repos.length) cleanup(repos.pop()!); });
function scratch(): string { const d = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-supervise-')); dirs.push(d); return d; }
function alive(pid: number): boolean { try { process.kill(pid, 0); return true; } catch { return false; } }
@@ -42,6 +43,27 @@ function run(dir: string, command: string, o: Partial<SuperviseOptions> = {}) {
.then(r => ({ ...r, lines, wall: Date.now() - started, log: fs.readFileSync(log, 'utf8') }));
}
/** A runner that starts one worker in a new session (a detached spawn, as `pnpm exec` starts Vitest under pnpm 11), prints nothing while the worker runs and exits with it. `<mode>.pids` holds the worker's pid, the worker's group and the runner's group. */
function ownSessionRunner(dir: string, mode: 'busy' | 'forever' | 'idle'): string {
const work = { busy: 'const end = Date.now() + 3500; while (Date.now() < end) {}', forever: 'for (;;) {}', idle: 'setTimeout(() => {}, 1e9)' }[mode];
const script = path.join(dir, `${mode}-runner.js`);
fs.writeFileSync(script, [
"const { spawn, execFileSync } = require('node:child_process');",
"const group = pid => execFileSync('ps', ['-o', 'pgid=', '-p', String(pid)], { encoding: 'utf8', timeout: 5000 }).trim();",
`const w = spawn(process.execPath, ['-e', ${JSON.stringify(work)}], { detached: true, stdio: 'ignore' });`,
`require('node:fs').writeFileSync(${JSON.stringify(path.join(dir, `${mode}.pids`))}, [w.pid, group(w.pid), group(process.pid)].join(' '));`,
"w.on('exit', code => { console.log('worker done'); process.exit(code ?? 1); });",
'',
].join('\n'));
return `${process.execPath} ${script}`;
}
async function pidsFrom(file: string): Promise<number[]> {
await pidFrom(file);
const pids = fs.readFileSync(file, 'utf8').trim().split(' ').map(Number);
workers.push(pids[0]!);
return pids;
}
describe('supervise: process group, log file, stall and deadline', () => {
test('a backgrounded child holding stdout does not keep the run waiting, and is killed with the group after the exit', async () => {
const dir = scratch();
@@ -81,6 +103,55 @@ describe('supervise: process group, log file, stall and deadline', () => {
expect(forever.exit).toBeNull();
}, 20_000);
// Value: protects=a quiet suite whose runner starts its workers in their own session (pnpm 11's `pnpm exec` running Vitest) is measured
// as busy, and a hung tree of that shape is still killed as stalled at the same stall timeout, worker included; fails_when=the stall check
// reads CPU of the runner's process group alone and kills the busy lane, or the kill signals only that group and orphans the worker;
// why_new=PR #4477: a monorepo suite ended PREGATE_STALLED three times at one tree while green outside pregate; seam=none;
// no_claim=a worker reparented away from the tree (a double fork) is not measured
test('the process tree is the runner group plus every descendant by parent links; a process reparented away is not in it', () => {
const procs = [
{ pid: 100, ppid: 1, pgid: 100, cpuMs: 10 },
{ pid: 101, ppid: 100, pgid: 100, cpuMs: 20 },
{ pid: 200, ppid: 101, pgid: 200, cpuMs: 300 },
{ pid: 201, ppid: 200, pgid: 200, cpuMs: 400 },
{ pid: 300, ppid: 1, pgid: 300, cpuMs: 5000 },
{ pid: 400, ppid: 1, pgid: 400, cpuMs: 7 },
];
expect(processTree(100, procs).map(p => p.pid).sort()).toEqual([100, 101, 200, 201]);
expect(processTree(999, procs)).toEqual([]);
});
test('a silent worker the runner starts in its own session keeps the run alive while it uses CPU', async () => {
const dir = scratch();
const busy = await run(dir, ownSessionRunner(dir, 'busy'), { stallMs: 1500 });
const [worker, workerGroup, runnerGroup] = await pidsFrom(path.join(dir, 'busy.pids'));
expect(workerGroup).toBe(worker);
expect(workerGroup).not.toBe(runnerGroup);
expect(busy.outcome).toBe('exited');
expect(busy.exit).toBe(0);
expect(busy.log).toContain('worker done');
}, 20_000);
test('an own-session worker that never stops working is killed at the deadline with its runner', async () => {
const dir = scratch();
const forever = await run(dir, ownSessionRunner(dir, 'forever'), { stallMs: 1500, deadline: Date.now() + 4000 });
const [worker] = await pidsFrom(path.join(dir, 'forever.pids'));
expect(forever.outcome).toBe('timeout');
expect(forever.exit).toBeNull();
expect(forever.wall).toBeLessThan(9000);
expect(await goneSoon(worker!)).toBe(true);
}, 20_000);
test('a tree whose runner and own-session worker both sit idle is killed as stalled at the stall timeout, worker included', async () => {
const dir = scratch();
const idle = await run(dir, ownSessionRunner(dir, 'idle'), { stallMs: 700 });
const [worker] = await pidsFrom(path.join(dir, 'idle.pids'));
expect(idle.outcome).toBe('stalled');
expect(idle.wall).toBeGreaterThanOrEqual(700);
expect(idle.wall).toBeLessThan(5000);
expect(await goneSoon(worker!)).toBe(true);
}, 20_000);
test('a group that ignores SIGTERM is SIGKILLed after the grace period', async () => {
const dir = scratch();
const r = await run(dir, `trap '' TERM; echo $$ > ${dir}/stubborn.pid; while :; do sleep 0.1; done`, { stallMs: 500, graceMs: 400 });