mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-07 17:28:23 +01:00
Garry Tanandcapy-ai[bot]
a60c99dbd7
* test-impact: tracer and reuse keys for free tests; pregate keeps main's always-run list
GST-167 PR 2: lib/test-impact/{trace,key}.ts and scripts/test-impact.ts trace a free test file under strace and key it at a tree over its read set, import closure, lockfile, configs, runner and key code and runtime; anything uncertain is ineligible. Nothing in CI changes yet.
GST-233: pregate unions the merge base's .gstack/always-run.json with the branch's, so a branch cannot drop main's whole-tree tests.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: pre-landing review fixes (untracked reads ineligible, tool bytes and runner imports in the key)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: adversarial review fixes (creation proof, PATH reads, bound loopback endpoints, unix sockets, env and runtime binding, future traces)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: graph checkout ignores the tree's export attributes; the tracer runs the free runner's own command and sandbox
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: write the graph's tree copy blob by blob so .gitattributes can't rewrite or hide files
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: refuse unsafe tree paths before materializeTree writes; trace the CLI key test on the real runtime
materializeTree now checks every path before writing: a .., . or .git segment, an empty segment, a backslash, a duplicate path or an entry under a symlink entry refuses the tree, and symlinks are written last. Failing-first tests on 89d4728 showed a '..' tree writing into os.tmpdir() and a symlink-then-blob tree writing through the link.
The key --base CLI test traced under a hard-coded bun 1.4.4 while the CLI keys on the real runtime (CI runs bun 1.4.2), so the runtime check added in 6b4e6c5 made it ineligible; it now traces on the real runtime at the current time.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
43 lines
5.1 KiB
Markdown
43 lines
5.1 KiB
Markdown
# Test impact: traced reuse keys for free tests
|
|
|
|
GST-167 is the plan to skip a free test on a pull request only when main already passed that exact test with the exact same inputs. PR 2 adds the pieces that decide what "the same inputs" means. Nothing here changes what CI runs yet: PR 3 records main receipts and reports in shadow, and PR 4 is the first change that would skip anything.
|
|
|
|
## Commands
|
|
|
|
`scripts/test-impact.ts` is a Bun CLI:
|
|
|
|
- `trace [--out <dir>] <test-file>...` runs each file under strace with the free runner's own `bun test` arguments and shard sandbox, and writes its read set to `<dir>` (default `.context/test-impact/traces`). It needs Linux with strace installed.
|
|
- `key --tree <ref> [--base <ref>] [--traces <dir>] [--json] <test-file>...` prints each file's key at `<ref>`, or why it has none.
|
|
- `explain [--tree <ref>] [--base <ref>] [--traces <dir>] <test-file>` prints every input of one file's key and why it is or isn't eligible.
|
|
|
|
With `--base`, the CLI loads `lib/test-impact/key.ts` from that ref and runs it over the tree, so a branch can't change the rules that decide what it skips. The CLI that does the loading comes from the checkout, so CI wiring (PR 3 and 4) has to run the base ref's copy of `scripts/test-impact.ts`, never the branch's. Without `--base`, it uses this checkout's code and says so. A base ref with no key code makes every file ineligible.
|
|
|
|
## What the trace records
|
|
|
|
The recorder runs `strace -f -y -qq -v` with `openat, open, execve, execveat, getdents64, readlink, readlinkat, stat, lstat, newfstatat, statx, access, faccessat, faccessat2, chdir, fchdir, clone, clone3, fork, vfork, connect, bind, sendto, sendmsg, sendmmsg, getsockname` (strace 6.8 on x86_64 Linux accepts the whole set) and gives the run a private TMPDIR. `lib/test-impact/trace.ts` keeps every repo path the test or any child process opened, ran, checked or probed. A failed open counts, because a file that shows up later changes what the test sees. For each repo directory it listed, the trace keeps the sorted names. Paths outside the repo, in /proc and /dev, under node_modules and in the private TMPDIR are dropped.
|
|
|
|
## What the key covers
|
|
|
|
`lib/test-impact/key.ts` reads everything at the given tree through git, never from the working tree. The graph runs over a copy of that tree written blob by blob, with no attribute processing, so the tree's own `.gitattributes` (`export-ignore`, `working-tree-encoding`, filters) can't hide an import from it. The key is a sha256 over:
|
|
|
|
- each traced path's bytes (read with `git cat-file --batch`), or that it is a directory, or that it is absent;
|
|
- the names in each traced directory listing, at the tree;
|
|
- the test's forward import-graph closure from `lib/pregate/import-graph.ts`, up to 12 hops;
|
|
- `bun.lock`, `bunfig.toml`, every `tsconfig*.json` and the `dependencies` in `.gstack/pregate.json`;
|
|
- the key code itself: the tracer, the key, the CLI, the free-test runner and planner (`scripts/test-free-shards.ts`) with every repo module it imports (shard engine, HOME guard, CI health, Windows curation, the paid/free split, the state root), the import graph and `lib/pregate/lane-selection.ts`;
|
|
- the bytes of each executable outside the repo the trace ran (git, bash, tar and so on), read on the machine computing the key;
|
|
- the runtime (platform, architecture, Bun version) and the values of `CI`, `GITHUB_ACTIONS`, `TZ`, `LANG`, `LC_ALL`, `NODE_ENV`, `EVALS` and `EVALS_TIER`.
|
|
|
|
## What is never reused
|
|
|
|
A file is ineligible, so it always runs, when:
|
|
|
|
- it is on the always-run list in the base branch's `.gstack/always-run.json` or the tree's (the union, so a branch can add entries but never drop main's);
|
|
- it has no trace, a trace older than 7 days or dated more than 5 minutes in the future, a trace whose run failed, or a trace made under other values of the allowlisted variables or on another runtime;
|
|
- its trace shows an inet connect or send to anything other than a loopback endpoint a socket in the same run bound, a unix socket outside the repo and the private TMPDIR (or an abstract one), any access under the real HOME other than a PATH tool lookup or the tool itself, a listing of HOME, shared /tmp state, or a read of the repository's own `.git`;
|
|
- its trace read a repo path the tree doesn't track, such as a build output under `dist/`. A probe of a missing path and a file the run created or truncated itself don't count;
|
|
- its import-graph walk reaches the 12-hop cap. The walk fails closed: a deeper chain makes the test ineligible instead of cutting the closure short;
|
|
- the tree changes any key-code file against the base. Then every file runs.
|
|
|
|
Limits: strace can't see which environment variables a test reads, so only the allowlist above is in the key, and a test that depends on another variable has to read it through a file the trace sees or stay off the reuse path. The recorder reuses the runner's arguments and sandbox but not its home-write guard, which only watches and changes nothing a test reads. Timing assertions, flaky history and the shard's execution grouping are for PR 3 and PR 4, which have the receipts to check them against. Windows has no tracer, so the Windows lane stays full.
|