mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-22 08:01:32 +01:00
Test runs whose workers start in their own session are no longer killed as stalled (GST-3213) (#4533)
/ship's test check no longer kills a quiet, working test run as stalled when the test command starts its workers in their own session, as pnpm 11's `pnpm exec` does for Vitest. Why: the stall check read CPU time from the command's process group only, where just the idle launcher was left; it now counts every process descended from the command. Nothing to do: a truly hung run is still killed at the same stall timeout, and every threshold is unchanged. Co-authored-by: dtoelke <14206719+dtoelke@users.noreply.github.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
8 files changed
+156
-36
No files matched your search
@@ -0,0 +1 @@
|
||||
- **/ship no longer kills a working test run as stalled when the test command starts its workers in their own session.** The stall check read CPU time from the test command's process group only, so a quiet suite run through a launcher that starts its workers in a new session (pnpm 11's `pnpm exec` running Vitest) looked idle and was killed as `PREGATE_STALLED`. It now counts every process descended from the command, and a stall or timeout kill reaches those workers too. A truly hung run is still killed at the same stall timeout. Reported by @dtoelke in #4477.
|
||||
+12
-6
@@ -540,14 +540,18 @@ test-tier command:
|
||||
a one-file run (each serial file) reports its file from the launch for any
|
||||
runner. Other runners (node --test, vitest, jest, pytest, playwright test, flutter test, dart test, cargo test) get heartbeats
|
||||
only. A heartbeat follows each 60 s with no progress line;
|
||||
- is killed as **stalled** when its log does not grow and its process group
|
||||
- is killed as **stalled** when its log does not grow and its process tree
|
||||
stays under a tenth of one CPU for the stall timeout (`--stall-timeout <s>`, else
|
||||
`stall_timeout_s`, else 300; `0` disables): the check is `incomplete` with
|
||||
`PREGATE_STALLED` and names the last file started; the next serial file
|
||||
still runs. The group counts as busy when its CPU time, sampled each second,
|
||||
still runs. The process tree is the command's process group plus every
|
||||
descendant reached by parent links, whatever session it moved to, so a
|
||||
worker a launcher starts in its own session (pnpm 11's `pnpm exec` running
|
||||
Vitest) counts. The tree counts as busy when its CPU time, sampled each second,
|
||||
grew by at least a tenth of the wall time, so a quiet suite that is working
|
||||
(FOUN-63: a parallel `check.sh` silent for 9 minutes) is not killed, while
|
||||
a hang that waits on nothing is. A hang that keeps burning CPU is left to
|
||||
a hang that waits on nothing is. A descendant reparented away from the tree
|
||||
(a double-forked daemon) no longer counts. A hang that keeps burning CPU is left to
|
||||
the timeout. Linux reads `/proc` (exited children's time included once
|
||||
reaped); macOS and other BSDs read `ps -S`; a failed read skips that
|
||||
sample. Windows reads no CPU time here and uses log growth alone. The `--remote` gate is exempt: a
|
||||
@@ -558,9 +562,11 @@ test-tier command:
|
||||
serial files that each fit still pass): the check is `incomplete`, its detail
|
||||
names which of the three set the timeout, and the next serial file still runs.
|
||||
|
||||
A kill is SIGTERM to the group, then SIGKILL after 5 s (a group that ignores
|
||||
SIGTERM still dies). SIGINT or SIGTERM to `gstack-pregate` itself SIGKILLs every
|
||||
running group, writes no `pregate.json` and exits 130 or 143. Windows has no
|
||||
A stall or timeout kill is SIGTERM to the process group of every tree member,
|
||||
then SIGKILL to the same groups after 5 s (a group that ignores SIGTERM still
|
||||
dies, and a worker in its own session is not left running). After a normal exit
|
||||
only the command's own group is killed. SIGINT or SIGTERM to `gstack-pregate`
|
||||
itself SIGKILLs every running tree, writes no `pregate.json` and exits 130 or 143. Windows has no
|
||||
process groups here: the kill is `taskkill /T /F` on the shell's tree, which
|
||||
misses a descendant that was reparented or started detached.
|
||||
|
||||
|
||||
@@ -4846,9 +4846,9 @@ missing checks needs a full `gstack-pregate --allow-repo-commands` (neither
|
||||
<a id="pregate-stalled"></a>
|
||||
### `PREGATE_STALLED`
|
||||
|
||||
**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log, and its process group used under a tenth of one CPU (Windows: log growth alone), for the stall timeout (default 300 s,
|
||||
**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log, and its process tree (its process group plus every descendant, whatever session it moved to) used under a tenth of one CPU (Windows: log growth alone), for the stall timeout (default 300 s,
|
||||
`stall_timeout_s` in the base branch's `.gstack/pregate.json` or the user-owned file, or
|
||||
`--stall-timeout`). pregate killed its process group and names the last test
|
||||
`--stall-timeout`). pregate killed its process tree and names the last test
|
||||
file started. The check is incomplete, never pass.
|
||||
|
||||
**Fix.** Open the printed log, find the hang in the named file (a blocked
|
||||
|
||||
@@ -90,7 +90,7 @@ async function run(ctx: PregateContext): Promise<CheckResult> {
|
||||
for (const r of runs) {
|
||||
if (r.carryNone) lines.push(`CI_CARRY: none guard=${r.guard.id} (${r.carryNone})`);
|
||||
if (r.prior) { lines.push(`GUARD: ${r.guard.id} ${r.prior}`); continue; }
|
||||
lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`);
|
||||
lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process tree killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`);
|
||||
if (r.exit !== 0) lines.push(...r.tail.map(l => ` ${l}`));
|
||||
if (r.environment) lines.push(` ${r.guard.id} ${environmentLine(r.environment)}`);
|
||||
}
|
||||
|
||||
@@ -453,7 +453,7 @@ function unfinished(ctx: PregateContext, r: Awaited<ReturnType<typeof runLocal>>
|
||||
const timedOut = r.runs.filter(x => x.outcome === 'timeout');
|
||||
const rest = r.failed ? `; ${r.failed} failed` : '';
|
||||
if (stalled.length) {
|
||||
return { code: 'PREGATE_STALLED', detail: `no test output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed; last file started: ${stalled.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest}`, fix: `read ${r.log} for the stalled file; fix the hang, or raise --stall-timeout (stall_timeout_s in ${ctx.configFile}; 0 disables) for a test that waits quietly for longer` };
|
||||
return { code: 'PREGATE_STALLED', detail: `no test output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process tree killed; last file started: ${stalled.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest}`, fix: `read ${r.log} for the stalled file; fix the hang, or raise --stall-timeout (stall_timeout_s in ${ctx.configFile}; 0 disables) for a test that waits quietly for longer` };
|
||||
}
|
||||
const launch = r.runs.find(x => x.outcome === 'spawn-failed');
|
||||
if (launch) {
|
||||
|
||||
+65
-23
@@ -3,11 +3,14 @@
|
||||
* own process group, stdout and stderr appended to a log file (no pipes, so
|
||||
* a backgrounded grandchild holding the output can never keep the check
|
||||
* waiting), supervised asynchronously. A stall is `stallMs` with no log growth
|
||||
* and no CPU activity in the process group (GST-1095: a quiet parallel suite
|
||||
* that is busy working is alive; a hang that keeps burning CPU is left to the
|
||||
* deadline); the deadline is the check's `--timeout`. Either kills the whole
|
||||
* process group (SIGTERM, then SIGKILL after a grace period) and the run is
|
||||
* incomplete. After a normal exit the group's leftover members are killed.
|
||||
* and no CPU activity in the process tree: the runner's process group plus
|
||||
* every descendant by parent links, whatever session it moved to (GST-1095: a
|
||||
* quiet parallel suite that is busy working is alive, #4477: so is a worker
|
||||
* `pnpm exec` starts in its own session; a hang that keeps burning CPU is left
|
||||
* to the deadline); the deadline is the check's `--timeout`. Either kills the
|
||||
* group of every tree member (SIGTERM, then SIGKILL after a grace period) and
|
||||
* the run is incomplete. After a normal exit only the runner's group's leftover
|
||||
* members are killed: a worker in its own session is no longer linked to it.
|
||||
*
|
||||
* Progress: a runner with a progress adapter (`bun test`: its `<file>:`
|
||||
* headers) reports `PREGATE_TEST: <n>/<total> <file> start|pass|fail <s>`
|
||||
@@ -16,7 +19,7 @@
|
||||
* with no progress line; with `lastLine` it names the last line the command printed.
|
||||
*
|
||||
* Platforms: POSIX signals the negative pid (the group `detached` creates
|
||||
* with setsid). Windows has no process groups here: the kill is
|
||||
* with setsid) and each tree member's own group. Windows has no process groups here: the kill is
|
||||
* `taskkill /T /F` on the shell's tree, which misses a descendant that was
|
||||
* reparented or started detached.
|
||||
*/
|
||||
@@ -70,12 +73,12 @@ const USER_HZ = 100;
|
||||
const CPU_SAMPLE_MS = 1_000;
|
||||
const CPU_BUSY_SHARE = 0.1;
|
||||
/** What a stall means beyond silence, for the PREGATE_STALLED detail and the guard line; Windows reads no CPU time. */
|
||||
export const STALL_IDLE = process.platform === 'win32' ? '' : ' and its process group used under a tenth of one CPU';
|
||||
export const STALL_IDLE = process.platform === 'win32' ? '' : ' and its process tree used under a tenth of one CPU';
|
||||
|
||||
/** SIGKILL every supervised process group still running (the CLI's SIGINT/SIGTERM handler, before it exits). */
|
||||
/** SIGKILL every supervised process tree still running (the CLI's SIGINT/SIGTERM handler, before it exits). */
|
||||
export function cancelActive(): number {
|
||||
const pids = [...ACTIVE];
|
||||
for (const pid of pids) killGroup(pid, 'SIGKILL');
|
||||
for (const pid of pids) killTree(pid, 'SIGKILL');
|
||||
ACTIVE.clear();
|
||||
return pids.length;
|
||||
}
|
||||
@@ -88,29 +91,67 @@ export function killGroup(pid: number, signal: NodeJS.Signals): void {
|
||||
try { process.kill(-pid, signal); } catch { /* group already gone */ }
|
||||
}
|
||||
|
||||
/** CPU time in ms used by the process group `pgid`, reaped children included where the platform reports them; null where it cannot be read (Windows). */
|
||||
export function groupCpuMs(pgid: number): number | null {
|
||||
export interface Proc { pid: number; ppid: number; pgid: number; cpuMs: number }
|
||||
|
||||
/** Every process with its parent, group and CPU time in ms (reaped children included where the platform reports them); null where it cannot be read (Windows, a failed `ps`). */
|
||||
export function processTable(): Proc[] | null {
|
||||
if (process.platform === 'win32') return null;
|
||||
const out: Proc[] = [];
|
||||
if (fs.existsSync('/proc/self/stat')) {
|
||||
let ticks = 0;
|
||||
for (const name of fs.readdirSync('/proc')) {
|
||||
if (!/^\d+$/.test(name)) continue;
|
||||
let stat: string;
|
||||
try { stat = fs.readFileSync(`/proc/${name}/stat`, 'utf8'); } catch { continue; }
|
||||
const f = stat.slice(stat.lastIndexOf(')') + 2).split(' ');
|
||||
if (Number(f[2]) === pgid) ticks += Number(f[11]) + Number(f[12]) + Number(f[13]) + Number(f[14]);
|
||||
out.push({ pid: Number(name), ppid: Number(f[1]), pgid: Number(f[2]), cpuMs: (Number(f[11]) + Number(f[12]) + Number(f[13]) + Number(f[14])) * 1000 / USER_HZ });
|
||||
}
|
||||
return ticks * 1000 / USER_HZ;
|
||||
return out;
|
||||
}
|
||||
const ps = spawnSync('ps', ['-A', '-S', '-o', 'pgid=,time='], { encoding: 'utf8', timeout: 15_000 });
|
||||
const ps = spawnSync('ps', ['-A', '-S', '-o', 'pid=,ppid=,pgid=,time='], { encoding: 'utf8', timeout: 15_000 });
|
||||
if (ps.status !== 0) return null;
|
||||
let ms = 0;
|
||||
for (const line of ps.stdout.split('\n')) {
|
||||
const [id, time] = line.trim().split(/\s+/);
|
||||
const [pid, ppid, pgid, time] = line.trim().split(/\s+/);
|
||||
const t = time && /^(?:(\d+)-)?(?:(\d+):)?(\d+):(\d+(?:\.\d+)?)$/.exec(time);
|
||||
if (Number(id) === pgid && t) ms += (((Number(t[1] ?? 0) * 24 + Number(t[2] ?? 0)) * 60 + Number(t[3])) * 60 + Number(t[4])) * 1000;
|
||||
if (t) out.push({ pid: Number(pid), ppid: Number(ppid), pgid: Number(pgid), cpuMs: (((Number(t[1] ?? 0) * 24 + Number(t[2] ?? 0)) * 60 + Number(t[3])) * 60 + Number(t[4])) * 1000 });
|
||||
}
|
||||
return ms;
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The runner `root`'s process tree: the members of its process group plus every descendant by parent links, whatever session or group it moved to. */
|
||||
export function processTree(root: number, procs: readonly Proc[]): Proc[] {
|
||||
const children = new Map<number, Proc[]>();
|
||||
for (const p of procs) { const c = children.get(p.ppid); if (c) c.push(p); else children.set(p.ppid, [p]); }
|
||||
const tree = new Map<number, Proc>();
|
||||
const queue = procs.filter(p => p.pgid === root);
|
||||
while (queue.length) {
|
||||
const p = queue.pop()!;
|
||||
if (tree.has(p.pid)) continue;
|
||||
tree.set(p.pid, p);
|
||||
queue.push(...(children.get(p.pid) ?? []));
|
||||
}
|
||||
return [...tree.values()];
|
||||
}
|
||||
|
||||
/** CPU time in ms used by the runner `root`'s process tree; null where it cannot be read (Windows). */
|
||||
export function treeCpuMs(root: number): number | null {
|
||||
const procs = processTable();
|
||||
return procs && processTree(root, procs).reduce((ms, p) => ms + p.cpuMs, 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Signal the runner `root`'s process group and the group of each tree member
|
||||
* outside it, so a stall or timeout kill leaves no worker that moved to its own
|
||||
* session running. `known` carries groups an earlier call found, for a runner
|
||||
* that has since exited. Never signals pid 1's group or this process's own
|
||||
* group. Returns every group signalled. Windows: `taskkill /T` on the shell.
|
||||
*/
|
||||
export function killTree(root: number, signal: NodeJS.Signals, known: Iterable<number> = []): Set<number> {
|
||||
const groups = new Set<number>([root, ...known]);
|
||||
const procs = processTable();
|
||||
const own = procs?.find(p => p.pid === process.pid)?.pgid;
|
||||
if (procs) for (const p of processTree(root, procs)) groups.add(p.pgid);
|
||||
for (const g of groups) if (g > 1 && g !== own && g !== process.pid) killGroup(g, signal);
|
||||
return groups;
|
||||
}
|
||||
|
||||
/** The adapter that reads per-file progress from this runner's output, or null when it has none. */
|
||||
@@ -194,18 +235,19 @@ export function supervise(o: SuperviseOptions): Promise<Supervised> {
|
||||
let outcome: Supervised['outcome'] | null = null;
|
||||
let settled = false;
|
||||
let escalate: ReturnType<typeof setTimeout> | null = null;
|
||||
let killed = new Set<number>();
|
||||
const stop = (why: Supervised['outcome']) => {
|
||||
if (outcome || !child.pid) return;
|
||||
outcome = why;
|
||||
killGroup(child.pid, 'SIGTERM');
|
||||
escalate = setTimeout(() => killGroup(child.pid!, 'SIGKILL'), graceMs);
|
||||
killed = killTree(child.pid, 'SIGTERM');
|
||||
escalate = setTimeout(() => { killed = killTree(child.pid!, 'SIGKILL', killed); }, graceMs);
|
||||
};
|
||||
const finish = (exit: number | null, signal: string | null, error?: string) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearInterval(timer);
|
||||
if (escalate) clearTimeout(escalate);
|
||||
if (child.pid) { ACTIVE.delete(child.pid); if (process.platform !== 'win32') killGroup(child.pid, 'SIGKILL'); }
|
||||
if (child.pid) { ACTIVE.delete(child.pid); if (process.platform !== 'win32') { if (outcome) killTree(child.pid, 'SIGKILL', killed); else killGroup(child.pid, 'SIGKILL'); } }
|
||||
try { size = fs.statSync(o.log).size; } catch { /* log removed */ }
|
||||
readNew();
|
||||
const done: Supervised['outcome'] = error ? 'spawn-failed' : (outcome ?? 'exited');
|
||||
@@ -220,7 +262,7 @@ export function supervise(o: SuperviseOptions): Promise<Supervised> {
|
||||
if (outcome) return;
|
||||
if (now >= o.deadline) { stop('timeout'); return; }
|
||||
if (o.stallMs > 0 && process.platform !== 'win32' && now - cpu.at >= CPU_SAMPLE_MS) {
|
||||
const used = groupCpuMs(child.pid!);
|
||||
const used = treeCpuMs(child.pid!);
|
||||
if (used !== null && used - cpu.ms >= CPU_BUSY_SHARE * (now - cpu.at)) lastBusy = now;
|
||||
cpu = { at: now, ms: used ?? cpu.ms };
|
||||
}
|
||||
|
||||
+1
-1
@@ -411,7 +411,7 @@ export const RESULT_CODES = {
|
||||
},
|
||||
PREGATE_STALLED: {
|
||||
anchor: 'pregate-stalled',
|
||||
summary: 'a test command printed nothing for the stall timeout while its process group used under a tenth of one CPU (log growth alone on Windows), so the group was killed; the check is incomplete, never pass',
|
||||
summary: 'a test command printed nothing for the stall timeout while its process tree used under a tenth of one CPU (log growth alone on Windows), so the tree was killed; the check is incomplete, never pass',
|
||||
fix: 'read the lane log for the last file started, fix the hang, or raise `--stall-timeout` (`stall_timeout_s`; 0 disables) for a test that waits quietly for longer, then rerun `gstack-pregate --stage tests`',
|
||||
},
|
||||
PREGATE_RECEIPT_INVALID: {
|
||||
|
||||
@@ -13,12 +13,13 @@ import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { EXIT } from '../lib/headless-artifacts';
|
||||
import { buildContext, withFingerprint } from '../lib/pregate/context';
|
||||
import { adapterFor, supervise, type SuperviseOptions } from '../lib/pregate/supervise';
|
||||
import { adapterFor, processTree, supervise, type SuperviseOptions } from '../lib/pregate/supervise';
|
||||
import { BASE_PREGATE, BIN, cleanup, commit, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/pregate-fixture';
|
||||
|
||||
const dirs: string[] = [];
|
||||
const repos: FixtureRepo[] = [];
|
||||
afterEach(() => { while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); while (repos.length) cleanup(repos.pop()!); });
|
||||
const workers: number[] = [];
|
||||
afterEach(() => { while (workers.length) { try { process.kill(workers.pop()!, 'SIGKILL'); } catch { /* already gone */ } } while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); while (repos.length) cleanup(repos.pop()!); });
|
||||
|
||||
function scratch(): string { const d = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-supervise-')); dirs.push(d); return d; }
|
||||
function alive(pid: number): boolean { try { process.kill(pid, 0); return true; } catch { return false; } }
|
||||
@@ -42,6 +43,27 @@ function run(dir: string, command: string, o: Partial<SuperviseOptions> = {}) {
|
||||
.then(r => ({ ...r, lines, wall: Date.now() - started, log: fs.readFileSync(log, 'utf8') }));
|
||||
}
|
||||
|
||||
/** A runner that starts one worker in a new session (a detached spawn, as `pnpm exec` starts Vitest under pnpm 11), prints nothing while the worker runs and exits with it. `<mode>.pids` holds the worker's pid, the worker's group and the runner's group. */
|
||||
function ownSessionRunner(dir: string, mode: 'busy' | 'forever' | 'idle'): string {
|
||||
const work = { busy: 'const end = Date.now() + 3500; while (Date.now() < end) {}', forever: 'for (;;) {}', idle: 'setTimeout(() => {}, 1e9)' }[mode];
|
||||
const script = path.join(dir, `${mode}-runner.js`);
|
||||
fs.writeFileSync(script, [
|
||||
"const { spawn, execFileSync } = require('node:child_process');",
|
||||
"const group = pid => execFileSync('ps', ['-o', 'pgid=', '-p', String(pid)], { encoding: 'utf8', timeout: 5000 }).trim();",
|
||||
`const w = spawn(process.execPath, ['-e', ${JSON.stringify(work)}], { detached: true, stdio: 'ignore' });`,
|
||||
`require('node:fs').writeFileSync(${JSON.stringify(path.join(dir, `${mode}.pids`))}, [w.pid, group(w.pid), group(process.pid)].join(' '));`,
|
||||
"w.on('exit', code => { console.log('worker done'); process.exit(code ?? 1); });",
|
||||
'',
|
||||
].join('\n'));
|
||||
return `${process.execPath} ${script}`;
|
||||
}
|
||||
async function pidsFrom(file: string): Promise<number[]> {
|
||||
await pidFrom(file);
|
||||
const pids = fs.readFileSync(file, 'utf8').trim().split(' ').map(Number);
|
||||
workers.push(pids[0]!);
|
||||
return pids;
|
||||
}
|
||||
|
||||
describe('supervise: process group, log file, stall and deadline', () => {
|
||||
test('a backgrounded child holding stdout does not keep the run waiting, and is killed with the group after the exit', async () => {
|
||||
const dir = scratch();
|
||||
@@ -81,6 +103,55 @@ describe('supervise: process group, log file, stall and deadline', () => {
|
||||
expect(forever.exit).toBeNull();
|
||||
}, 20_000);
|
||||
|
||||
// Value: protects=a quiet suite whose runner starts its workers in their own session (pnpm 11's `pnpm exec` running Vitest) is measured
|
||||
// as busy, and a hung tree of that shape is still killed as stalled at the same stall timeout, worker included; fails_when=the stall check
|
||||
// reads CPU of the runner's process group alone and kills the busy lane, or the kill signals only that group and orphans the worker;
|
||||
// why_new=PR #4477: a monorepo suite ended PREGATE_STALLED three times at one tree while green outside pregate; seam=none;
|
||||
// no_claim=a worker reparented away from the tree (a double fork) is not measured
|
||||
test('the process tree is the runner group plus every descendant by parent links; a process reparented away is not in it', () => {
|
||||
const procs = [
|
||||
{ pid: 100, ppid: 1, pgid: 100, cpuMs: 10 },
|
||||
{ pid: 101, ppid: 100, pgid: 100, cpuMs: 20 },
|
||||
{ pid: 200, ppid: 101, pgid: 200, cpuMs: 300 },
|
||||
{ pid: 201, ppid: 200, pgid: 200, cpuMs: 400 },
|
||||
{ pid: 300, ppid: 1, pgid: 300, cpuMs: 5000 },
|
||||
{ pid: 400, ppid: 1, pgid: 400, cpuMs: 7 },
|
||||
];
|
||||
expect(processTree(100, procs).map(p => p.pid).sort()).toEqual([100, 101, 200, 201]);
|
||||
expect(processTree(999, procs)).toEqual([]);
|
||||
});
|
||||
|
||||
test('a silent worker the runner starts in its own session keeps the run alive while it uses CPU', async () => {
|
||||
const dir = scratch();
|
||||
const busy = await run(dir, ownSessionRunner(dir, 'busy'), { stallMs: 1500 });
|
||||
const [worker, workerGroup, runnerGroup] = await pidsFrom(path.join(dir, 'busy.pids'));
|
||||
expect(workerGroup).toBe(worker);
|
||||
expect(workerGroup).not.toBe(runnerGroup);
|
||||
expect(busy.outcome).toBe('exited');
|
||||
expect(busy.exit).toBe(0);
|
||||
expect(busy.log).toContain('worker done');
|
||||
}, 20_000);
|
||||
|
||||
test('an own-session worker that never stops working is killed at the deadline with its runner', async () => {
|
||||
const dir = scratch();
|
||||
const forever = await run(dir, ownSessionRunner(dir, 'forever'), { stallMs: 1500, deadline: Date.now() + 4000 });
|
||||
const [worker] = await pidsFrom(path.join(dir, 'forever.pids'));
|
||||
expect(forever.outcome).toBe('timeout');
|
||||
expect(forever.exit).toBeNull();
|
||||
expect(forever.wall).toBeLessThan(9000);
|
||||
expect(await goneSoon(worker!)).toBe(true);
|
||||
}, 20_000);
|
||||
|
||||
test('a tree whose runner and own-session worker both sit idle is killed as stalled at the stall timeout, worker included', async () => {
|
||||
const dir = scratch();
|
||||
const idle = await run(dir, ownSessionRunner(dir, 'idle'), { stallMs: 700 });
|
||||
const [worker] = await pidsFrom(path.join(dir, 'idle.pids'));
|
||||
expect(idle.outcome).toBe('stalled');
|
||||
expect(idle.wall).toBeGreaterThanOrEqual(700);
|
||||
expect(idle.wall).toBeLessThan(5000);
|
||||
expect(await goneSoon(worker!)).toBe(true);
|
||||
}, 20_000);
|
||||
|
||||
test('a group that ignores SIGTERM is SIGKILLed after the grace period', async () => {
|
||||
const dir = scratch();
|
||||
const r = await run(dir, `trap '' TERM; echo $$ > ${dir}/stubborn.pid; while :; do sleep 0.1; done`, { stallMs: 500, graceMs: 400 });
|
||||
|
||||
Reference in new issue
Block a user