test-impact: tracer and reuse keys for free tests; pregate keeps main's always-run list

GST-167 PR 2: lib/test-impact/{trace,key}.ts and scripts/test-impact.ts trace a free test file under strace and key it at a tree over its read set, import closure, lockfile, configs, runner and key code and runtime; anything uncertain is ineligible. Nothing in CI changes yet.

GST-233: pregate unions the merge base's .gstack/always-run.json with the branch's, so a branch cannot drop main's whole-tree tests.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-11-01 14:54:28 +00:00
1 parent 83ae96e086
commit 32c8a85005
13 files changed
+1061 -12

No files matched your search

+2
View File
@@ -154,6 +154,8 @@ Review each entry before committing the file on the base branch.
`{ "tests": [globs] }` names whole-tree tests that scan every file of a kind (tripwires, ratchets, conformance and fixture inventories). A diff-selected run never maps a touched file to them, so they used to pass the selected gate and then fail CI (GST-98). When a branch selects any test, owes a lane, or edits this file, the `lanes` selection adds every tracked test file a glob matches. `--explain` prints each one as `<test> (always_run, .gstack/always-run.json)`, and a glob that matches no test as `<glob> matches no test file`. A docs-only branch adds nothing.
The globs are the union of the working tree's list and the merge base's list (GST-233), so a branch can add entries but never drop one main already has. A test only the merge base's list names prints as `<test> (always_run, merge base .gstack/always-run.json)`. `scripts/test-impact.ts` takes the same union with its `--base` ref and never reuses a result for those tests ([test-impact.md](test-impact.md)).
The list lives in its own file, not in `pregate.json`, because older pregate readers reject unknown keys there. A repository that ignores `.gstack/` (gstack does) tracks the file with `git add -f`, as it does `pregate.json`. A missing file means an empty list. A file that is not JSON, or whose `tests` is not a list of non-empty strings, stops the check with that message. gstack's own list matches 16 test files; the 13 with a seed in `scripts/free-test-durations.json` (recorded 2026-10-19) total about 13 s there, and `bun-entry-isolation`, `launch-env-echo-ratchet` and `review-packets-removed` have no seed yet.
## `gstack-pregate`
+40
View File
@@ -0,0 +1,40 @@
# Test impact: traced reuse keys for free tests
GST-167 is the plan to skip a free test on a pull request only when main already passed that exact test with the exact same inputs. PR 2 adds the pieces that decide what "the same inputs" means. Nothing here changes what CI runs yet: PR 3 records main receipts and reports in shadow, and PR 4 is the first change that would skip anything.
## Commands
`scripts/test-impact.ts` is a Bun CLI:
- `trace [--out <dir>] <test-file>...` runs each file with `bun test` under strace and writes its read set to `<dir>` (default `.context/test-impact/traces`). It needs Linux with strace installed.
- `key --tree <ref> [--base <ref>] [--traces <dir>] [--json] <test-file>...` prints each file's key at `<ref>`, or why it has none.
- `explain [--tree <ref>] [--base <ref>] [--traces <dir>] <test-file>` prints every input of one file's key and why it is or isn't eligible.
With `--base`, the CLI loads `lib/test-impact/key.ts` from that ref and runs it over the tree, so a branch can't change the rules that decide what it skips. Without `--base`, it uses this checkout's code and says so. A base ref with no key code makes every file ineligible.
## What the trace records
The recorder runs `strace -f -y -qq -v` with `openat, open, execve, execveat, getdents64, readlink, readlinkat, stat, lstat, newfstatat, statx, access, faccessat, faccessat2, chdir, fchdir, clone, clone3, fork, vfork, connect, bind, sendto, getsockname` (strace 6.8 on x86_64 Linux accepts the whole set) and gives the run a private TMPDIR. `lib/test-impact/trace.ts` keeps every repo path the test or any child process opened, ran, checked or probed. A failed open counts, because a file that shows up later changes what the test sees. For each repo directory it listed, the trace keeps the sorted names. Paths outside the repo, in /proc and /dev, under node_modules and in the private TMPDIR are dropped.
## What the key covers
`lib/test-impact/key.ts` reads everything at the given tree through git, never from the working tree. The key is a sha256 over:
- each traced path's bytes (read with `git cat-file --batch`), or that it is a directory, or that it is absent;
- the names in each traced directory listing, at the tree;
- the test's forward import-graph closure from `lib/pregate/import-graph.ts`, up to 12 hops;
- `bun.lock`, every `tsconfig*.json` and the `dependencies` in `.gstack/pregate.json`;
- the key code itself: the tracer, the key, the CLI, the free-test runner and planner (`scripts/test-free-shards.ts`), the shard engine, the import graph and `lib/pregate/lane-selection.ts`;
- the runtime (platform, architecture, Bun version) and the values of `CI`, `GITHUB_ACTIONS`, `TZ`, `LANG`, `LC_ALL`, `NODE_ENV`, `EVALS` and `EVALS_TIER`.
## What is never reused
A file is ineligible, so it always runs, when:
- it is on the always-run list in the base branch's `.gstack/always-run.json` or the tree's (the union, so a branch can add entries but never drop main's);
- it has no trace, a trace older than 7 days, or a trace whose run failed;
- its trace shows an inet connect or send to anything other than a loopback port the same run opened, any access under the real HOME other than a PATH tool lookup, shared /tmp state, or a read of the repository's own `.git`;
- its import-graph walk reaches the 12-hop cap. The walk fails closed: a deeper chain makes the test ineligible instead of cutting the closure short;
- the tree changes any key-code file against the base. Then every file runs.
Limits: strace can't see which environment variables a test reads, so only the allowlist above is in the key, and a test that depends on another variable has to read it through a file the trace sees or stay off the reuse path. Timing assertions, flaky history and the shard's execution grouping are for PR 3 and PR 4, which have the receipts to check them against. Windows has no tracer, so the Windows lane stays full.