From 50cc28227367598e6adcb2d60124ca9992f1f8f7 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 20 Nov 2026 00:54:44 +0800 Subject: [PATCH] Test runs whose workers start in their own session are no longer killed as stalled (GST-3213) (#4533) /ship's test check no longer kills a quiet, working test run as stalled when the test command starts its workers in their own session, as pnpm 11's `pnpm exec` does for Vitest. Why: the stall check read CPU time from the command's process group only, where just the idle launcher was left; it now counts every process descended from the command. Nothing to do: a truly hung run is still killed at the same stall timeout, and every threshold is unchanged. Co-authored-by: dtoelke <14206719+dtoelke@users.noreply.github.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- .../stall-check-counts-own-session-workers.md | 1 + docs/pregate.md | 18 ++-- docs/troubleshooting.md | 4 +- lib/pregate/checks/guards.ts | 2 +- lib/pregate/checks/lanes.ts | 2 +- lib/pregate/supervise.ts | 88 ++++++++++++++----- lib/result-codes.ts | 2 +- test/pregate-supervise.test.ts | 75 +++++++++++++++- 8 files changed, 156 insertions(+), 36 deletions(-) create mode 100644 changelog.d/stall-check-counts-own-session-workers.md diff --git a/changelog.d/stall-check-counts-own-session-workers.md b/changelog.d/stall-check-counts-own-session-workers.md new file mode 100644 index 000000000..ac9395b69 --- /dev/null +++ b/changelog.d/stall-check-counts-own-session-workers.md @@ -0,0 +1 @@ +- **/ship no longer kills a working test run as stalled when the test command starts its workers in their own session.** The stall check read CPU time from the test command's process group only, so a quiet suite run through a launcher that starts its workers in a new session (pnpm 11's `pnpm exec` running Vitest) looked idle and was killed as `PREGATE_STALLED`. It now counts every process descended from the command, and a stall or timeout kill reaches those workers too. A truly hung run is still killed at the same stall timeout. Reported by @dtoelke in #4477. diff --git a/docs/pregate.md b/docs/pregate.md index d9858f862..2dfc4058a 100644 --- a/docs/pregate.md +++ b/docs/pregate.md @@ -540,14 +540,18 @@ test-tier command: a one-file run (each serial file) reports its file from the launch for any runner. Other runners (node --test, vitest, jest, pytest, playwright test, flutter test, dart test, cargo test) get heartbeats only. A heartbeat follows each 60 s with no progress line; -- is killed as **stalled** when its log does not grow and its process group +- is killed as **stalled** when its log does not grow and its process tree stays under a tenth of one CPU for the stall timeout (`--stall-timeout `, else `stall_timeout_s`, else 300; `0` disables): the check is `incomplete` with `PREGATE_STALLED` and names the last file started; the next serial file - still runs. The group counts as busy when its CPU time, sampled each second, + still runs. The process tree is the command's process group plus every + descendant reached by parent links, whatever session it moved to, so a + worker a launcher starts in its own session (pnpm 11's `pnpm exec` running + Vitest) counts. The tree counts as busy when its CPU time, sampled each second, grew by at least a tenth of the wall time, so a quiet suite that is working (FOUN-63: a parallel `check.sh` silent for 9 minutes) is not killed, while - a hang that waits on nothing is. A hang that keeps burning CPU is left to + a hang that waits on nothing is. A descendant reparented away from the tree + (a double-forked daemon) no longer counts. A hang that keeps burning CPU is left to the timeout. Linux reads `/proc` (exited children's time included once reaped); macOS and other BSDs read `ps -S`; a failed read skips that sample. Windows reads no CPU time here and uses log growth alone. The `--remote` gate is exempt: a @@ -558,9 +562,11 @@ test-tier command: serial files that each fit still pass): the check is `incomplete`, its detail names which of the three set the timeout, and the next serial file still runs. -A kill is SIGTERM to the group, then SIGKILL after 5 s (a group that ignores -SIGTERM still dies). SIGINT or SIGTERM to `gstack-pregate` itself SIGKILLs every -running group, writes no `pregate.json` and exits 130 or 143. Windows has no +A stall or timeout kill is SIGTERM to the process group of every tree member, +then SIGKILL to the same groups after 5 s (a group that ignores SIGTERM still +dies, and a worker in its own session is not left running). After a normal exit +only the command's own group is killed. SIGINT or SIGTERM to `gstack-pregate` +itself SIGKILLs every running tree, writes no `pregate.json` and exits 130 or 143. Windows has no process groups here: the kill is `taskkill /T /F` on the shell's tree, which misses a descendant that was reparented or started detached. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 04c1d63c4..76d862b96 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -4846,9 +4846,9 @@ missing checks needs a full `gstack-pregate --allow-repo-commands` (neither ### `PREGATE_STALLED` -**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log, and its process group used under a tenth of one CPU (Windows: log growth alone), for the stall timeout (default 300 s, +**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log, and its process tree (its process group plus every descendant, whatever session it moved to) used under a tenth of one CPU (Windows: log growth alone), for the stall timeout (default 300 s, `stall_timeout_s` in the base branch's `.gstack/pregate.json` or the user-owned file, or -`--stall-timeout`). pregate killed its process group and names the last test +`--stall-timeout`). pregate killed its process tree and names the last test file started. The check is incomplete, never pass. **Fix.** Open the printed log, find the hang in the named file (a blocked diff --git a/lib/pregate/checks/guards.ts b/lib/pregate/checks/guards.ts index c8d40feb0..2a2a9c137 100644 --- a/lib/pregate/checks/guards.ts +++ b/lib/pregate/checks/guards.ts @@ -90,7 +90,7 @@ async function run(ctx: PregateContext): Promise { for (const r of runs) { if (r.carryNone) lines.push(`CI_CARRY: none guard=${r.guard.id} (${r.carryNone})`); if (r.prior) { lines.push(`GUARD: ${r.guard.id} ${r.prior}`); continue; } - lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`); + lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process tree killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`); if (r.exit !== 0) lines.push(...r.tail.map(l => ` ${l}`)); if (r.environment) lines.push(` ${r.guard.id} ${environmentLine(r.environment)}`); } diff --git a/lib/pregate/checks/lanes.ts b/lib/pregate/checks/lanes.ts index ee1a0e736..334a70207 100644 --- a/lib/pregate/checks/lanes.ts +++ b/lib/pregate/checks/lanes.ts @@ -453,7 +453,7 @@ function unfinished(ctx: PregateContext, r: Awaited> const timedOut = r.runs.filter(x => x.outcome === 'timeout'); const rest = r.failed ? `; ${r.failed} failed` : ''; if (stalled.length) { - return { code: 'PREGATE_STALLED', detail: `no test output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed; last file started: ${stalled.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest}`, fix: `read ${r.log} for the stalled file; fix the hang, or raise --stall-timeout (stall_timeout_s in ${ctx.configFile}; 0 disables) for a test that waits quietly for longer` }; + return { code: 'PREGATE_STALLED', detail: `no test output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process tree killed; last file started: ${stalled.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest}`, fix: `read ${r.log} for the stalled file; fix the hang, or raise --stall-timeout (stall_timeout_s in ${ctx.configFile}; 0 disables) for a test that waits quietly for longer` }; } const launch = r.runs.find(x => x.outcome === 'spawn-failed'); if (launch) { diff --git a/lib/pregate/supervise.ts b/lib/pregate/supervise.ts index 49331283d..ce8618595 100644 --- a/lib/pregate/supervise.ts +++ b/lib/pregate/supervise.ts @@ -3,11 +3,14 @@ * own process group, stdout and stderr appended to a log file (no pipes, so * a backgrounded grandchild holding the output can never keep the check * waiting), supervised asynchronously. A stall is `stallMs` with no log growth - * and no CPU activity in the process group (GST-1095: a quiet parallel suite - * that is busy working is alive; a hang that keeps burning CPU is left to the - * deadline); the deadline is the check's `--timeout`. Either kills the whole - * process group (SIGTERM, then SIGKILL after a grace period) and the run is - * incomplete. After a normal exit the group's leftover members are killed. + * and no CPU activity in the process tree: the runner's process group plus + * every descendant by parent links, whatever session it moved to (GST-1095: a + * quiet parallel suite that is busy working is alive, #4477: so is a worker + * `pnpm exec` starts in its own session; a hang that keeps burning CPU is left + * to the deadline); the deadline is the check's `--timeout`. Either kills the + * group of every tree member (SIGTERM, then SIGKILL after a grace period) and + * the run is incomplete. After a normal exit only the runner's group's leftover + * members are killed: a worker in its own session is no longer linked to it. * * Progress: a runner with a progress adapter (`bun test`: its `:` * headers) reports `PREGATE_TEST: / start|pass|fail ` @@ -16,7 +19,7 @@ * with no progress line; with `lastLine` it names the last line the command printed. * * Platforms: POSIX signals the negative pid (the group `detached` creates - * with setsid). Windows has no process groups here: the kill is + * with setsid) and each tree member's own group. Windows has no process groups here: the kill is * `taskkill /T /F` on the shell's tree, which misses a descendant that was * reparented or started detached. */ @@ -70,12 +73,12 @@ const USER_HZ = 100; const CPU_SAMPLE_MS = 1_000; const CPU_BUSY_SHARE = 0.1; /** What a stall means beyond silence, for the PREGATE_STALLED detail and the guard line; Windows reads no CPU time. */ -export const STALL_IDLE = process.platform === 'win32' ? '' : ' and its process group used under a tenth of one CPU'; +export const STALL_IDLE = process.platform === 'win32' ? '' : ' and its process tree used under a tenth of one CPU'; -/** SIGKILL every supervised process group still running (the CLI's SIGINT/SIGTERM handler, before it exits). */ +/** SIGKILL every supervised process tree still running (the CLI's SIGINT/SIGTERM handler, before it exits). */ export function cancelActive(): number { const pids = [...ACTIVE]; - for (const pid of pids) killGroup(pid, 'SIGKILL'); + for (const pid of pids) killTree(pid, 'SIGKILL'); ACTIVE.clear(); return pids.length; } @@ -88,29 +91,67 @@ export function killGroup(pid: number, signal: NodeJS.Signals): void { try { process.kill(-pid, signal); } catch { /* group already gone */ } } -/** CPU time in ms used by the process group `pgid`, reaped children included where the platform reports them; null where it cannot be read (Windows). */ -export function groupCpuMs(pgid: number): number | null { +export interface Proc { pid: number; ppid: number; pgid: number; cpuMs: number } + +/** Every process with its parent, group and CPU time in ms (reaped children included where the platform reports them); null where it cannot be read (Windows, a failed `ps`). */ +export function processTable(): Proc[] | null { if (process.platform === 'win32') return null; + const out: Proc[] = []; if (fs.existsSync('/proc/self/stat')) { - let ticks = 0; for (const name of fs.readdirSync('/proc')) { if (!/^\d+$/.test(name)) continue; let stat: string; try { stat = fs.readFileSync(`/proc/${name}/stat`, 'utf8'); } catch { continue; } const f = stat.slice(stat.lastIndexOf(')') + 2).split(' '); - if (Number(f[2]) === pgid) ticks += Number(f[11]) + Number(f[12]) + Number(f[13]) + Number(f[14]); + out.push({ pid: Number(name), ppid: Number(f[1]), pgid: Number(f[2]), cpuMs: (Number(f[11]) + Number(f[12]) + Number(f[13]) + Number(f[14])) * 1000 / USER_HZ }); } - return ticks * 1000 / USER_HZ; + return out; } - const ps = spawnSync('ps', ['-A', '-S', '-o', 'pgid=,time='], { encoding: 'utf8', timeout: 15_000 }); + const ps = spawnSync('ps', ['-A', '-S', '-o', 'pid=,ppid=,pgid=,time='], { encoding: 'utf8', timeout: 15_000 }); if (ps.status !== 0) return null; - let ms = 0; for (const line of ps.stdout.split('\n')) { - const [id, time] = line.trim().split(/\s+/); + const [pid, ppid, pgid, time] = line.trim().split(/\s+/); const t = time && /^(?:(\d+)-)?(?:(\d+):)?(\d+):(\d+(?:\.\d+)?)$/.exec(time); - if (Number(id) === pgid && t) ms += (((Number(t[1] ?? 0) * 24 + Number(t[2] ?? 0)) * 60 + Number(t[3])) * 60 + Number(t[4])) * 1000; + if (t) out.push({ pid: Number(pid), ppid: Number(ppid), pgid: Number(pgid), cpuMs: (((Number(t[1] ?? 0) * 24 + Number(t[2] ?? 0)) * 60 + Number(t[3])) * 60 + Number(t[4])) * 1000 }); } - return ms; + return out; +} + +/** The runner `root`'s process tree: the members of its process group plus every descendant by parent links, whatever session or group it moved to. */ +export function processTree(root: number, procs: readonly Proc[]): Proc[] { + const children = new Map(); + for (const p of procs) { const c = children.get(p.ppid); if (c) c.push(p); else children.set(p.ppid, [p]); } + const tree = new Map(); + const queue = procs.filter(p => p.pgid === root); + while (queue.length) { + const p = queue.pop()!; + if (tree.has(p.pid)) continue; + tree.set(p.pid, p); + queue.push(...(children.get(p.pid) ?? [])); + } + return [...tree.values()]; +} + +/** CPU time in ms used by the runner `root`'s process tree; null where it cannot be read (Windows). */ +export function treeCpuMs(root: number): number | null { + const procs = processTable(); + return procs && processTree(root, procs).reduce((ms, p) => ms + p.cpuMs, 0); +} + +/** + * Signal the runner `root`'s process group and the group of each tree member + * outside it, so a stall or timeout kill leaves no worker that moved to its own + * session running. `known` carries groups an earlier call found, for a runner + * that has since exited. Never signals pid 1's group or this process's own + * group. Returns every group signalled. Windows: `taskkill /T` on the shell. + */ +export function killTree(root: number, signal: NodeJS.Signals, known: Iterable = []): Set { + const groups = new Set([root, ...known]); + const procs = processTable(); + const own = procs?.find(p => p.pid === process.pid)?.pgid; + if (procs) for (const p of processTree(root, procs)) groups.add(p.pgid); + for (const g of groups) if (g > 1 && g !== own && g !== process.pid) killGroup(g, signal); + return groups; } /** The adapter that reads per-file progress from this runner's output, or null when it has none. */ @@ -194,18 +235,19 @@ export function supervise(o: SuperviseOptions): Promise { let outcome: Supervised['outcome'] | null = null; let settled = false; let escalate: ReturnType | null = null; + let killed = new Set(); const stop = (why: Supervised['outcome']) => { if (outcome || !child.pid) return; outcome = why; - killGroup(child.pid, 'SIGTERM'); - escalate = setTimeout(() => killGroup(child.pid!, 'SIGKILL'), graceMs); + killed = killTree(child.pid, 'SIGTERM'); + escalate = setTimeout(() => { killed = killTree(child.pid!, 'SIGKILL', killed); }, graceMs); }; const finish = (exit: number | null, signal: string | null, error?: string) => { if (settled) return; settled = true; clearInterval(timer); if (escalate) clearTimeout(escalate); - if (child.pid) { ACTIVE.delete(child.pid); if (process.platform !== 'win32') killGroup(child.pid, 'SIGKILL'); } + if (child.pid) { ACTIVE.delete(child.pid); if (process.platform !== 'win32') { if (outcome) killTree(child.pid, 'SIGKILL', killed); else killGroup(child.pid, 'SIGKILL'); } } try { size = fs.statSync(o.log).size; } catch { /* log removed */ } readNew(); const done: Supervised['outcome'] = error ? 'spawn-failed' : (outcome ?? 'exited'); @@ -220,7 +262,7 @@ export function supervise(o: SuperviseOptions): Promise { if (outcome) return; if (now >= o.deadline) { stop('timeout'); return; } if (o.stallMs > 0 && process.platform !== 'win32' && now - cpu.at >= CPU_SAMPLE_MS) { - const used = groupCpuMs(child.pid!); + const used = treeCpuMs(child.pid!); if (used !== null && used - cpu.ms >= CPU_BUSY_SHARE * (now - cpu.at)) lastBusy = now; cpu = { at: now, ms: used ?? cpu.ms }; } diff --git a/lib/result-codes.ts b/lib/result-codes.ts index ae294da79..c5ef134a3 100644 --- a/lib/result-codes.ts +++ b/lib/result-codes.ts @@ -411,7 +411,7 @@ export const RESULT_CODES = { }, PREGATE_STALLED: { anchor: 'pregate-stalled', - summary: 'a test command printed nothing for the stall timeout while its process group used under a tenth of one CPU (log growth alone on Windows), so the group was killed; the check is incomplete, never pass', + summary: 'a test command printed nothing for the stall timeout while its process tree used under a tenth of one CPU (log growth alone on Windows), so the tree was killed; the check is incomplete, never pass', fix: 'read the lane log for the last file started, fix the hang, or raise `--stall-timeout` (`stall_timeout_s`; 0 disables) for a test that waits quietly for longer, then rerun `gstack-pregate --stage tests`', }, PREGATE_RECEIPT_INVALID: { diff --git a/test/pregate-supervise.test.ts b/test/pregate-supervise.test.ts index df6f2fb73..adc91fe7b 100644 --- a/test/pregate-supervise.test.ts +++ b/test/pregate-supervise.test.ts @@ -13,12 +13,13 @@ import * as os from 'node:os'; import * as path from 'node:path'; import { EXIT } from '../lib/headless-artifacts'; import { buildContext, withFingerprint } from '../lib/pregate/context'; -import { adapterFor, supervise, type SuperviseOptions } from '../lib/pregate/supervise'; +import { adapterFor, processTree, supervise, type SuperviseOptions } from '../lib/pregate/supervise'; import { BASE_PREGATE, BIN, cleanup, commit, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/pregate-fixture'; const dirs: string[] = []; const repos: FixtureRepo[] = []; -afterEach(() => { while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); while (repos.length) cleanup(repos.pop()!); }); +const workers: number[] = []; +afterEach(() => { while (workers.length) { try { process.kill(workers.pop()!, 'SIGKILL'); } catch { /* already gone */ } } while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); while (repos.length) cleanup(repos.pop()!); }); function scratch(): string { const d = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-supervise-')); dirs.push(d); return d; } function alive(pid: number): boolean { try { process.kill(pid, 0); return true; } catch { return false; } } @@ -42,6 +43,27 @@ function run(dir: string, command: string, o: Partial = {}) { .then(r => ({ ...r, lines, wall: Date.now() - started, log: fs.readFileSync(log, 'utf8') })); } +/** A runner that starts one worker in a new session (a detached spawn, as `pnpm exec` starts Vitest under pnpm 11), prints nothing while the worker runs and exits with it. `.pids` holds the worker's pid, the worker's group and the runner's group. */ +function ownSessionRunner(dir: string, mode: 'busy' | 'forever' | 'idle'): string { + const work = { busy: 'const end = Date.now() + 3500; while (Date.now() < end) {}', forever: 'for (;;) {}', idle: 'setTimeout(() => {}, 1e9)' }[mode]; + const script = path.join(dir, `${mode}-runner.js`); + fs.writeFileSync(script, [ + "const { spawn, execFileSync } = require('node:child_process');", + "const group = pid => execFileSync('ps', ['-o', 'pgid=', '-p', String(pid)], { encoding: 'utf8', timeout: 5000 }).trim();", + `const w = spawn(process.execPath, ['-e', ${JSON.stringify(work)}], { detached: true, stdio: 'ignore' });`, + `require('node:fs').writeFileSync(${JSON.stringify(path.join(dir, `${mode}.pids`))}, [w.pid, group(w.pid), group(process.pid)].join(' '));`, + "w.on('exit', code => { console.log('worker done'); process.exit(code ?? 1); });", + '', + ].join('\n')); + return `${process.execPath} ${script}`; +} +async function pidsFrom(file: string): Promise { + await pidFrom(file); + const pids = fs.readFileSync(file, 'utf8').trim().split(' ').map(Number); + workers.push(pids[0]!); + return pids; +} + describe('supervise: process group, log file, stall and deadline', () => { test('a backgrounded child holding stdout does not keep the run waiting, and is killed with the group after the exit', async () => { const dir = scratch(); @@ -81,6 +103,55 @@ describe('supervise: process group, log file, stall and deadline', () => { expect(forever.exit).toBeNull(); }, 20_000); + // Value: protects=a quiet suite whose runner starts its workers in their own session (pnpm 11's `pnpm exec` running Vitest) is measured + // as busy, and a hung tree of that shape is still killed as stalled at the same stall timeout, worker included; fails_when=the stall check + // reads CPU of the runner's process group alone and kills the busy lane, or the kill signals only that group and orphans the worker; + // why_new=PR #4477: a monorepo suite ended PREGATE_STALLED three times at one tree while green outside pregate; seam=none; + // no_claim=a worker reparented away from the tree (a double fork) is not measured + test('the process tree is the runner group plus every descendant by parent links; a process reparented away is not in it', () => { + const procs = [ + { pid: 100, ppid: 1, pgid: 100, cpuMs: 10 }, + { pid: 101, ppid: 100, pgid: 100, cpuMs: 20 }, + { pid: 200, ppid: 101, pgid: 200, cpuMs: 300 }, + { pid: 201, ppid: 200, pgid: 200, cpuMs: 400 }, + { pid: 300, ppid: 1, pgid: 300, cpuMs: 5000 }, + { pid: 400, ppid: 1, pgid: 400, cpuMs: 7 }, + ]; + expect(processTree(100, procs).map(p => p.pid).sort()).toEqual([100, 101, 200, 201]); + expect(processTree(999, procs)).toEqual([]); + }); + + test('a silent worker the runner starts in its own session keeps the run alive while it uses CPU', async () => { + const dir = scratch(); + const busy = await run(dir, ownSessionRunner(dir, 'busy'), { stallMs: 1500 }); + const [worker, workerGroup, runnerGroup] = await pidsFrom(path.join(dir, 'busy.pids')); + expect(workerGroup).toBe(worker); + expect(workerGroup).not.toBe(runnerGroup); + expect(busy.outcome).toBe('exited'); + expect(busy.exit).toBe(0); + expect(busy.log).toContain('worker done'); + }, 20_000); + + test('an own-session worker that never stops working is killed at the deadline with its runner', async () => { + const dir = scratch(); + const forever = await run(dir, ownSessionRunner(dir, 'forever'), { stallMs: 1500, deadline: Date.now() + 4000 }); + const [worker] = await pidsFrom(path.join(dir, 'forever.pids')); + expect(forever.outcome).toBe('timeout'); + expect(forever.exit).toBeNull(); + expect(forever.wall).toBeLessThan(9000); + expect(await goneSoon(worker!)).toBe(true); + }, 20_000); + + test('a tree whose runner and own-session worker both sit idle is killed as stalled at the stall timeout, worker included', async () => { + const dir = scratch(); + const idle = await run(dir, ownSessionRunner(dir, 'idle'), { stallMs: 700 }); + const [worker] = await pidsFrom(path.join(dir, 'idle.pids')); + expect(idle.outcome).toBe('stalled'); + expect(idle.wall).toBeGreaterThanOrEqual(700); + expect(idle.wall).toBeLessThan(5000); + expect(await goneSoon(worker!)).toBe(true); + }, 20_000); + test('a group that ignores SIGTERM is SIGKILLed after the grace period', async () => { const dir = scratch(); const r = await run(dir, `trap '' TERM; echo $$ > ${dir}/stubborn.pid; while :; do sleep 0.1; done`, { stallMs: 500, graceMs: 400 });