mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-10 02:30:39 +01:00
fix(pregate): a quiet suite that is busy on CPU is not stalled (GST-1095)
The stall timer looked only at log growth, so a parallel check.sh that prints nothing for 9 minutes while working ended PREGATE_STALLED (FOUN-63). supervise now samples the process group's CPU time each second and treats growth of at least a tenth of wall time as liveness. An idle hang still stalls; a hang that burns CPU is left to --timeout. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
f1cbdbb483
commit
bb19bfb13e
9 files changed
+87
-19
No files matched your search
+10
-5
@@ -85,7 +85,7 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`).
|
||||
| `serial_patterns` | test-file globs the `lanes` check runs one process per file, apart from the unit files (default `**/*.serial.test.*`) |
|
||||
| `sealed` | sealed or eval output that must never reach a public destination; read by the pre-push hook and `gstack-pregate sealed` (see [sealed output](#sealed)) |
|
||||
| `price_file` | repo-relative path of the paid gate's call-plan data file (`GSTACK_EVAL_CALLS v1`); `gstack-eval-plan price-gate` prices from it and never executes anything ([docs/eval-calls.md](eval-calls.md)) |
|
||||
| `stall_timeout_s` | seconds a test command may print nothing before its process group is killed and the check is `incomplete` (`PREGATE_STALLED`); default 300, `0` disables; `--stall-timeout` overrides; the `--remote` gate is exempt (see [supervision](#supervision)) |
|
||||
| `stall_timeout_s` | seconds a test command may print nothing and use no CPU before its process group is killed and the check is `incomplete` (`PREGATE_STALLED`); default 300, `0` disables; `--stall-timeout` overrides; the `--remote` gate is exempt (see [supervision](#supervision)) |
|
||||
| `timeout_s` | seconds each runner process may take (the unit-file group, each serial file, each guard without its own `timeout_s`, and the remote gate's deadline) when `--timeout` is not passed; 1 to 3600, default 600. `gstack-ship-preflight` and the `finish` fallback run pregate without `--timeout`, so this is how a repo with long lanes sets theirs |
|
||||
| `remote_gate` | the repo's remote test gate command; under 8 cores the local test tier becomes a `requires-remote` obligation naming it, and `--remote` runs it (see [remote gate](#remote-gate)) |
|
||||
| `run_profiles` | name → `{ workflow, events, jobs, artifact, max_age_hours?, os? }`: the GitHub Actions runs `clear --run` verifies as evidence on the identical tree; a profile named after a lane (`free-tests.yml/macos-named-regressions`) clears that lane (see [run evidence](#run-evidence)). `--from-run` and the main-push reuse job verify runs against them too ([from-run](#from-run), [CI reuse](#ci-reuse)) |
|
||||
@@ -447,10 +447,15 @@ test-tier command:
|
||||
a one-file run (each serial file) reports its file from the launch for any
|
||||
runner. Other runners (node --test, vitest, jest, pytest, playwright test, flutter test, dart test, cargo test) get heartbeats
|
||||
only. A heartbeat follows each 60 s with no progress line;
|
||||
- is killed as **stalled** when its log does not grow for the stall timeout
|
||||
(`--stall-timeout <s>`, else `stall_timeout_s`, else 300; `0` disables): the
|
||||
check is `incomplete` with `PREGATE_STALLED` and names the last file
|
||||
started; the next serial file still runs. The `--remote` gate is exempt: a
|
||||
- is killed as **stalled** when its log does not grow and its process group
|
||||
uses no CPU for the stall timeout (`--stall-timeout <s>`, else
|
||||
`stall_timeout_s`, else 300; `0` disables): the check is `incomplete` with
|
||||
`PREGATE_STALLED` and names the last file started; the next serial file
|
||||
still runs. The group counts as busy when its CPU time, sampled each second,
|
||||
grew by at least a tenth of the wall time, so a quiet suite that is working
|
||||
(FOUN-63: a parallel `check.sh` silent for 9 minutes) is not killed, while
|
||||
a hang that waits on nothing is. A hang that keeps burning CPU is left to
|
||||
the timeout. Windows reads no CPU time here and uses log growth alone. The `--remote` gate is exempt: a
|
||||
gate that waits on CI is quiet by design, so only its timeout bounds it, and
|
||||
its heartbeats add `; last output: <the gate's last line>`;
|
||||
- is killed as **timed out** when that runner process reaches its timeout
|
||||
|
||||
@@ -3843,14 +3843,14 @@ the lane log, then rerun or split the selection. A tool missing from PATH is
|
||||
<a id="pregate-stalled"></a>
|
||||
### `PREGATE_STALLED`
|
||||
|
||||
**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log for the stall timeout (default 300 s,
|
||||
**Meaning.** A test command the lanes check (or a guard) launched wrote nothing to its log and its process group used no CPU for the stall timeout (default 300 s,
|
||||
`stall_timeout_s` in the base branch's `.gstack/pregate.json` or the user-owned file, or
|
||||
`--stall-timeout`). pregate killed its process group and names the last test
|
||||
file started. The check is incomplete, never pass.
|
||||
|
||||
**Fix.** Open the printed log, find the hang in the named file (a blocked
|
||||
child, a server that never exits, a synchronous wait), fix it and rerun
|
||||
`gstack-pregate --stage tests`. A test that is legitimately quiet for longer
|
||||
`gstack-pregate --stage tests`. A test that legitimately waits quietly for longer
|
||||
needs a larger `--stall-timeout`; `0` disables the stall kill and leaves only
|
||||
`--timeout`. The `--remote` gate never stalls: it is bounded by `--timeout`
|
||||
(else `timeout_s`), so a gate that waits on CI needs a timeout long enough for
|
||||
|
||||
Reference in new issue
Block a user