diff --git a/CHANGELOG.md b/CHANGELOG.md index 611f5a6b0..f67c5f462 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,28 @@ # Changelog +## [1.91.254.0] - 2026-10-26 + +**Pregate ergonomics for a repository that is not gstack (Lane D, GSTA-112; plan harness-codify E4–E6; Captain issues 6, 11, 12).** Three Captains lost their first `/ship` pass to `PREGATE_NO_LANE` in a repo that had no `.gstack/pregate.json` yet (FOUN #292, harness-evals #81/#82); the non-downgradable `secrets` check refused 1,800 dummy per-trial tokens, 11-digit Actions run ids read as phone numbers and a public Firebase key every bundle rebuild re-adds (harness-evals #81/#83, GSTA-102, GSTA-106); a lane whose runner could not start read as the diff's failure and was waived by test name (truther #617); and pregate's logs under `os.tmpdir()` were shared between parallel runs and wiped by a Capy wake (gbrain 16). + +### Added +- **No policy → warn plus a starter, never a hard `PREGATE_NO_LANE`** (`lib/pregate/starter.ts`, `lib/pregate/checks/lanes.ts`). When the base branch carries no `.gstack/pregate.json` and no user-owned policy exists, a touched file with no lane is a `warn` row under the new code `PREGATE_NO_POLICY`; the selected tests still run (a red lane is still `fail`, an unknown platform matrix is still a `requires-remote` obligation), and the table prints a starter policy drafted from the repository: the detected runner, the test directories, the CI lanes (naming a planner-computed matrix that needs `selection`), the JSON between `PREGATE_STARTER: begin` and `end` with a `dependencies` entry per unmapped touched file mapped to the tests nearest it or sharing its stem (never `[]`), and a `TODO` line per file nothing covers. `run` writes nothing; `gstack-pregate init` is the explicit write. Once any policy exists (base copy, user-owned file, or `--policy-from head`), an unmapped file is `PREGATE_NO_LANE` as before. +- **`allowlist` in `.gstack/pregate.json`** (`lib/pregate/allowlist.ts`, `lib/pregate/config.ts`, `lib/pregate/checks/secrets.ts`): `[{ finding, paths?, pattern?, reason }]`, where `finding` is a gstack-redact finding id or one of the categories `pii`, `internal`, `hygiene`, `legal`; `paths` are contained globs; `pattern` is a regex matched at the finding's position in the added line and ending at a token boundary; one scope is required and `reason` always is (64 entries at most). The `secrets` row says `N allowlisted ()`, lists each allowlisted finding with its entry and reason, and records the entries' digest as `inputs.allowlist`. **`env.value` (a credential-named environment variable's real value in the added lines, #3449) and the whole `secret` category can never be allowlisted:** validation refuses the entry (`PREGATE_CONFIG_INVALID` at `allowlist[i].finding`) and the matcher never consults the list for `env.value`, so a stale or hand-edited policy cannot silence it; the scanner's own `engine.*` refusals (an oversize input fails closed) are refused and kept the same way. A `pattern` entry that named a finding the scanner gave no position for is reported as not tested (the finding is kept) instead of silently inert. An unparsed scanner report keeps its exit-code verdict. +- **Environment failures reported apart from diff failures** (`lib/pregate/environment.ts`; new code `PREGATE_ENVIRONMENT` with the troubleshooting anchor `pregate-environment`). A runner or guard binary missing from PATH (exit 127, `command not found`, a spawn `ENOENT`, a tool `not on PATH`), a network error or a quota refusal before any test was counted, a full disk, an unverified runtime or gstack's own scanner not finishing is `incomplete` with `cause: "environment"` and `environment: { kind, evidence }` in `pregate.json`, an `environment: — ` table line and a fix; `PREGATE_LANE_FAILED`, `PREGATE_GUARD_FAILED` and `PREGATE_SECRETS` carry `cause: "diff"`. `environment.proof` records how the cause was established: `launch` (a spawn failure, a binary off PATH, a runtime or tool refusal; the fix ends "the diff is not at fault") or `output` (a rule matched the command's text before any test was counted, which the diff's own output could have written; the table line says `(inferred from output)` and the fix asks for confirmation on another machine or with `--remote`). The classifier is conservative: a local lane that exited non-zero is read from its output only when its runner reported no test summary and was not seen starting a test file; a counted failure is the diff's whatever its log says; exit 127 and `command not found` are `missing-binary` only when the command's own binary is off PATH, so a runner, guard or gate script the diff controls that exits 127 is the diff's; a remote gate is environment only when it printed no `PREGATE_REMOTE:` line and the cause sits in its first 40 lines, before any test output could stream; a runner killed by a signal, stalls and timeouts keep `PREGATE_INCOMPLETE` / `PREGATE_STALLED`. `gstack-pregate verify` prints `PREGATE_ENVIRONMENT: check(s) blocked by the environment (), not by the diff` when every incomplete check is environment-caused, naming any check inferred from output alone, and names the kinds beside `PREGATE_INCOMPLETE` otherwise, so a lane waives by cause and proof, never by test name. +- **One scratch directory per run under the state root** (`lib/pregate/run-scratch.ts`): `/projects//pregate/runs//` (0700; `--run ` names it, else `--<4 hex>`) holds the secrets scan's added-line copies (deleted after the scan), `lanes/lanes.log`, `guards/.log` and the remote gate's log and file list, never `os.tmpdir()`. `pregate.json` records it as `scratch`, the table prints `scratch: `, a taken `--run` name takes a numbered sibling (`-2` … `-9`) rather than sharing the directory; runs older than 7 days under the same project are pruned when a new run starts; an unwritable state root falls back to a private per-run directory under `os.tmpdir()` and says so. + +### Fixed +- **`literals` no longer floods on a common module stem or a one-digit value** (`lib/pregate/checks/literals.ts` `consumersOf`). A test file counted as a consumer of a changed constant when it merely contained the source file's stem as a bare word (`install`, `retry`), so `GSTACK_CAPY_WRAPPER=2 → 3` in `capy/skill/gstack/scripts/install.sh` flagged 575 test files (`PREGATE_LITERALS`; GSTA-114, #3475). The module now counts only as a path fragment (`/install.sh`, `/retry.js`, `/retry'`), and an old value shorter than two characters needs the constant's name in the file. The same edit now flags the 5 files that name `GSTACK_CAPY_WRAPPER`. Regression: `test/pregate-selection.test.ts` "consumersOf: a module stem is a path fragment…". + +### Changed +- `docs/pregate.md`: the `allowlist` row, the "No policy yet" paragraph, three sections (allowlist, environment failures, scratch), the synopsis (`--run `), the stage-1 `guards` and stage-2 `lanes` bullets and the Publication code list. `docs/troubleshooting.md`: `PREGATE_NO_POLICY` and `PREGATE_ENVIRONMENT` anchors; `PREGATE_NO_LANE` and `PREGATE_INCOMPLETE` point at them; `PREGATE_GUARD_FAILED` carves out a guard whose own binary is off PATH, `PREGATE_SECRETS` points at the allowlist, `PREGATE_INCOMPLETE` names a signal-killed runner. `lib/headless-artifacts.ts`: the pregate schema accepts `cause`, `environment` and `scratch` (additive). `gstack-pregate --help` documents `--run`. +- Existing pregate tests whose fixtures had no policy and asserted the hard `PREGATE_NO_LANE` (`test/pregate-selection.test.ts`, `pregate-docs-lane-review.test.ts`, `pregate-non-js.test.ts`, `pregate-rust.test.ts`) now assert the no-policy warning and that the file is still named as unmapped; the Dart missing-SDK cases in `test/pregate-dart.test.ts` assert `PREGATE_ENVIRONMENT` and the new fix text. + +### Tests +- `test/pregate-starter.test.ts`: no policy → exit 0, `warn`/`PREGATE_NO_POLICY`, the mapped test still ran, the starter validates, maps by stem, and prints the TODO, test directories and CI lanes; a planner-computed platform lane keeps `requires-remote` with the code; a failing selected test is still `PREGATE_LANE_FAILED`; a base or user-owned policy keeps `PREGATE_NO_LANE`; `run` leaves no file and `init` writes one `validate` accepts. Red before the change: exit 1 with `PREGATE_NO_LANE`. +- `test/pregate-allowlist.test.ts`: validation (id or category with a scope and a reason; `env.value`, the `secret` wildcard, an empty scope, a bad regex and an uncontained glob refused); a dummy `sk-trial-…` token allowlisted by path and pattern and named, failing outside the path or under a pattern that covers only its head; a category entry for a run id; `env.value` survives a `pattern: ".*"` entry at the matcher and the config naming it is refused before any check runs. Red before: `PREGATE_SECRETS` on the dummy token and `allowlist` an unknown key. +- `test/pregate-environment.test.ts`: the classifier's vocabulary; a missing runner binary, a network error with no summary, a quota refusal from the remote gate (obligation uncleared) and a missing guard binary are `PREGATE_ENVIRONMENT` with their kinds and `verify` says so; a counted test failure beside an `ECONNREFUSED` line and a guard that exits 2 stay `cause: "diff"`. Red before: `PREGATE_LANE_FAILED` with no cause. +- `test/pregate-scratch.test.ts`: two `--run` ids get two directories under the fixture's state root, logs and the record name them, an unnamed run is unique, the secrets copies are gone after the scan, and prune keeps fresh runs and the current one. Red before: logs under `/tmp/gstack-pregate-lanes-*` and no `scratch` field. + ## [1.91.253.0] - 2026-10-26 **`/captain` (also `/crew`): the crew operating model that keeps many agents merging into one base branch without turning it red, as a gstack skill with templates and a team guide.** Teams running a controller agent over worker threads had the model only in one crew's private notes. It now ships generic: one controller merges and lanes never merge; a READY block (full head SHA, base, CI counts, pregate, gate evidence, `SHIP_RECEIPT` at the pin, screen choice with justification, receipt path, logged flaky cases) is the only merge trigger, and a green PR page or a cheap test tier never is; merge trains (build, verdict, per-slot stamp on call, verify, merge with the expected head, verify again, prune), the red-train rules (in-run flaky rerun once, bisect on the red cases, exactly one unchanged rebuild, one infra re-dispatch), the tree rule, the backstop and the freeze, and the post-merge babysit; the digest shape (Needs you / New / In progress with ages); the three approval shapes (the per-thread record, the plan ask with before → after and a "do it" reply, the merge ask with captioned before/after images); and the daily flaky-fix pass (ledger format, the gate runs' flake records and the free suites' retries as sources, red-before-green fix PRs, one named infra rerun allowance, never thresholds, trials or kinds). Where crews that run this model disagree (squash or merge, lane or controller stamps, gate-ahead, the severe race, a gate run id in the READY) the skill ships both as options for the crew rules file. Every repo-specific name is a parameter (`REPO`, `BASE`, `REQUIRED_CHECKS`, `FULL_GATE_CHECK`, `MERGE_METHOD`, `STAMP_FILES`, `KNOWN_INFRA_FLAKES`). diff --git a/VERSION b/VERSION index a7099c900..515e9323d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.253.0 +1.91.254.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 4d2a029e7..2bbd84735 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.253.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.254.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-doctor-components.sh b/bin/gstack-doctor-components.sh index 0ab402984..624dea855 100644 --- a/bin/gstack-doctor-components.sh +++ b/bin/gstack-doctor-components.sh @@ -282,6 +282,8 @@ gstack_result_anchor() { QUEUE_SCREEN_STALE_REF) printf '%s' 'queue-screen-stale-ref' ;; READY_SCREEN_CHOICE_REQUIRED) printf '%s' 'ready-screen-choice-required' ;; PREGATE_TIMING_ASSERTION) printf '%s' 'pregate-timing-assertion' ;; + PREGATE_NO_POLICY) printf '%s' 'pregate-no-policy' ;; + PREGATE_ENVIRONMENT) printf '%s' 'pregate-environment' ;; APPROVAL_CLASSES_INVALID) printf '%s' 'approval-classes-invalid' ;; APPROVAL_CLASS_FOUND) printf '%s' 'approval-class-found' ;; APPROVAL_CITATION_MISSING) printf '%s' 'approval-citation-missing' ;; diff --git a/bin/gstack-pregate b/bin/gstack-pregate index 8ce0a2083..2246ec682 100755 --- a/bin/gstack-pregate +++ b/bin/gstack-pregate @@ -75,6 +75,8 @@ const HELP = `gstack-pregate — cheap checks before anything paid or remote (do PREGATE_REMOTE: tree= tests= failed=0 line with n > 0 --spend-cap the cap --remote passes to the gate (default \$GSTACK_SPEND_CAP_USD); spend is reported as spent, reserved and unknown + --run the run label recorded in pregate.json and the name of this run's scratch directory (a taken name gets a numbered sibling, -2 … -9) + (/projects//pregate/runs/: lane and guard logs, scan files; default a unique id) --explain print each selected test's provenance and each uncertain mapping --json one JSON object on stdout; a refusal before any check runs (unavailable base, invalid pin file, WTREE_UNAVAILABLE) is {ok:false,status:'refused',code,detail,fix,docs} with exit 3, @@ -147,7 +149,7 @@ async function cmdRun(root: string, rest: string[], json: boolean): Promise` copy or the user-owned file below is honoured, so a downgrade is reviewed (or is the user's own) before it takes effect. A repo that @@ -105,6 +106,23 @@ instead of falling back to defaults. A base copy git cannot read (and instead of handing the run to the user file. `gstack-pregate validate` / `init` cover the working-tree file. + +**No policy yet.** A repository with no `.gstack/pregate.json` on the base +branch and no user-owned file runs under the defaults, and a touched file that +maps to no test is then a **warning**, not the hard `PREGATE_NO_LANE`: the +`lanes` row is `warn` with `PREGATE_NO_POLICY`, the selected tests still run +(a red lane is still `fail`, an unknown platform matrix is still a +`requires-remote` obligation), and the table prints a starter policy drafted +from the repository (`PREGATE_STARTER:` lines: the detected runner, the test +directories, the CI lanes, then the JSON between `begin` and `end`, with a +`dependencies` entry per unmapped touched file mapped to the tests nearest it +or sharing its stem, never `[]`, and a `TODO` line for each file nothing +covers). `run` writes nothing; `gstack-pregate init` is the explicit write. +Once any policy exists (committed on the base, or the user-owned file), an +unmapped file is `PREGATE_NO_LANE` again: the warning is for the first /ship +in a new repository (three Captains lost their first pass to it), never a way +to hide missing coverage. `--policy-from head` with a head policy is strict too. + `gstack-pregate init` writes a draft built from the repo, not a fixed example. It prints the languages it found and the runner each language's tests would get, then maps every file the import graph cannot place to the tests of the @@ -118,7 +136,7 @@ Review each entry before committing the file on the base branch. ``` gstack-pregate [--stage preflight|tests|all] [--only ] [--allow-repo-commands] [--policy-from base|head] - [--timeout ] [--stall-timeout ] [--remote --spend-cap ] [--explain] [--json] + [--timeout ] [--stall-timeout ] [--remote --spend-cap ] [--run ] [--explain] [--json] gstack-pregate verify | clear --remote --run-url | lanes | validate | init ``` @@ -150,7 +168,9 @@ Two stages with separate clocks, both printed: touched file (renames included) runs its `command` in its own process, under the verified launch Bun, with its own `timeout_s` (default `--timeout`). A non-zero exit fails pregate naming the guard and its `fix` - (`PREGATE_GUARD_FAILED`); a timeout is `incomplete`; without + (`PREGATE_GUARD_FAILED`); a guard that could not start, or whose own binary + is off PATH (exit 127), is the machine's, `PREGATE_ENVIRONMENT` + (`incomplete`), not a guard failure; a timeout is `incomplete`; without `--allow-repo-commands` the selected guards are listed and the check is `incomplete`. Guards are read from the base branch's copy (or the user-owned file when the base carries none) and cannot be @@ -179,8 +199,11 @@ Two stages with separate clocks, both printed: `baseUrl`) tries the exact path, then `.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`, `.cjs`, `.mts`, `.cts`, then `index.ts`, `index.tsx`, `index.js`, `index.jsx` and `index.mjs` inside the directory; the first file that exists wins. - A touched source file with no lane is `FAIL lanes — has no test lane`, - never a pass. Markdown that no test depends on is a docs/no-tests empty lane + A touched source file with no lane is `FAIL lanes — has no test lane` + once any policy exists (the base copy, the user-owned file or `--policy-from + head`); with no policy anywhere it is the `warn` `PREGATE_NO_POLICY` with a + starter printed ([no policy](#no-policy)); never a pass. Markdown that no + test depends on is a docs/no-tests empty lane instead (`docs/no-tests: N documentation file(s)`), including when the base branch has no `.gstack/pregate.json`. The dependency is the same import graph as every other file, so a test that reads the Markdown (a fixture or a doc) @@ -268,7 +291,8 @@ The record is `.gstack/tmp/pregate.json` (`gstack-artifact validate --as pregate **Publication.** `gstack-ship-receipt write --pregate .gstack/tmp/pregate.json` (Step 18) refuses unless the record is current for the tree being published (`PREGATE_STALE`: rerun after the final stamp), green (`PREGATE_LANE_FAILED`, -`PREGATE_INCOMPLETE`) and every `requires-remote` obligation carries a receipt +`PREGATE_INCOMPLETE`, or `PREGATE_ENVIRONMENT` when every incomplete check is +the machine's) and every `requires-remote` obligation carries a receipt (`PREGATE_REMOTE_UNCLEARED`: dispatch the job, then `gstack-pregate clear --remote --run-url `). The ship receipt then carries `pregate` and `requires_remote`. `gstack-pregate verify` @@ -427,6 +451,128 @@ guard is a *command* with its own fix line. Older gstack refuses unknown keys that reads them ships. The worked gbrain configuration is in [gbrain worked configs](#gbrain-worked-configs). + +## Allowlist: benign findings the secrets check accepts + +The `secrets` check cannot be downgraded, and that is right for a key. It also +refused 1,800 per-trial dummy tokens in eval receipts (`"key": "sk-trial-"` +reads as `env.kv`), an 11-digit Actions run id read as a phone number, and +excalidraw.com's public Firebase key that every rebuild of a bundle re-adds +(harness-evals #81/#83, GSTA-102, GSTA-106). `allowlist` in the policy names +those, each with its scope and its reason: + +```json +{ "allowlist": [ + { "finding": "env.kv", "paths": ["evals/**/receipts/*.json"], "pattern": "sk-trial-[0-9a-f-]+", "reason": "dummy per-trial meter ids" }, + { "finding": "pii.phone.e164", "pattern": "\\d{11}", "reason": "GitHub Actions run ids in reports" }, + { "finding": "google.api_key", "paths": ["lib/render/dist/bundle.js"], "pattern": "AIza[0-9A-Za-z_-]{35}", "reason": "the vendor's public web key, restricted server-side" } ] } +``` + +- `finding` is a gstack-redact finding id (`env.kv`, `google.api_key`, + `pii.phone.e164`, `internal.hostname`, …) or one of the categories `pii`, + `internal`, `hygiene`, `legal`. A secret is always named by its id; the + `secret` category cannot be allowlisted whole. +- `paths` are contained globs over the added file's repo path; `pattern` is a + regular expression matched **at the finding's position in the added line** + and ending at a token boundary, so a dummy token's shape never covers a real + key that shares its line or starts the same way. One of the two is required; + `reason` always is, and the record repeats it. +- **`env.value` is never allowlisted.** It is the real value of a + credential-named variable of the environment pregate ran in (#3449), found in + the added lines whatever its shape. Validation refuses an entry naming it + (`PREGATE_CONFIG_INVALID`), and the matcher never consults the list for it, + so a stale or hand-edited policy cannot silence it either. The scanner's own + `engine.*` refusals (an oversize input fails closed) are refused and kept the + same way: they are the scanner declining its input, not a finding in it. +- A `pattern` entry needs the finding's position. When the scanner gave none, + the entry is inert for that finding and the row says so (`allowlist[0] + env.kv not tested on env.kv (): the scanner gave no position for its + pattern; the finding is kept`) rather than silently keeping it. + +The `secrets` row says `N allowlisted ()` and lists each allowlisted +finding with the entry (`allowlisted env.kv (evals/receipts/t1.json, added line +3) by allowlist[0] env.kv: dummy per-trial meter ids`); `inputs.allowlist` +carries the entries' digest, so a changed list is a changed input. The list is +read from the base branch's copy (reviewed) or the user-owned file, like every +other key; an unparsed scanner report keeps its exit-code verdict and is never +allowlisted. + + +## Environment failures, apart from diff failures + +A lane that could not run is not a lane that failed. When a check could not run +its command because of the machine (truther #617: a root runner with a vitest +config that found no file, a corpus fetch that stalled on one host and passed +on another), the check is `incomplete` under its own code, +**`PREGATE_ENVIRONMENT`**, with `cause: "environment"` and an +`environment: { kind, evidence, proof }` object in `pregate.json`, an +`environment: — ` table line, and a fix that says so. A +failure the branch caused carries `cause: "diff"` (`PREGATE_LANE_FAILED`, +`PREGATE_GUARD_FAILED`, `PREGATE_SECRETS`). Kinds: + +| Kind | Evidence | +|---|---| +| `missing-binary` | exit 127, `command not found`, a spawn `ENOENT`, a runner tool `not on PATH` (dart, flutter, cargo, a package manager) | +| `network` | `ENOTFOUND`, `ECONNREFUSED`, `ECONNRESET`, `ETIMEDOUT`, `EAI_AGAIN`, `Could not resolve host`, `fetch failed`, a TLS handshake or certificate failure | +| `quota` | `quota`, `429 Too Many Requests`, `rate limit`, `insufficient_quota` (a vCPU quota on the remote backend) | +| `disk` | `ENOSPC`, `No space left on device`, `EDQUOT` | +| `runtime` | the install verdict or the pinned Bun refused (`BUN_RESOLUTION_DRIFT`, `INSTALL_UNVERIFIED`) | +| `tool` | gstack's own scanner (gstack-redact) did not finish or exited outside its contract | + +The classifier is conservative. A local lane that exited non-zero is read from +its output **only when its runner reported no test summary** (nothing ran) and +the runner was not seen starting a test file; a run whose summary counts `N +ran, M failed` is the diff's failure whatever its log says, so a test that +failed on `ECONNREFUSED` is still a red test. A remote gate that exited +without a `PREGATE_REMOTE:` line is read from its **preamble only** (its first +40 lines, before any test output could stream): a VM refused for quota or an +unreachable API there never ran the tests, so it is environment and the +obligation stays uncleared, while a `quota` in a streamed test name is not. A +guard that ran and said no is the diff's. Exit 127 and `command not found` +count as `missing-binary` only when the command's **own** binary is off +PATH: a runner, guard or gate script the diff controls that exits 127 (a +package script naming a tool that is not installed) is the diff's failure. A +stall or a timeout keeps `PREGATE_STALLED` / `PREGATE_INCOMPLETE`: a hang +has no single cause, and so does a runner killed by a signal. + +`proof` records how the cause was established. `launch` is certain: the +command could not start, its binary is off PATH, the runtime or a gstack tool +refused; the fix ends `the diff is not at fault`. `output` means a rule +matched the command's text before any test was counted, which the diff's own +output could have written (a test that prints `fetch failed` and exits the +process before the summary); the table line says `(inferred from output)` and +the fix asks for confirmation on another machine or with `--remote` before +the diff is treated as clear. Neither proof ever passes the check. + +`gstack-pregate verify` reports `PREGATE_ENVIRONMENT: check(s) blocked by +the environment (), not by the diff` when every incomplete check is +environment-caused, naming any check whose cause was inferred from output +alone, and names the kinds beside the generic `PREGATE_INCOMPLETE` otherwise; +`--json` carries `cause` and `environment` per check. A lane that waives such +a run cites the kind, the evidence and the proof, not a test name, and clears +the lane remotely (`--remote`, or `gstack-pregate clear` with a run URL) +rather than on the broken machine. Incomplete never passes. + + +## Scratch: one directory per run, under the state root + +Every run owns `/projects//pregate/runs//` (mode +0700; `` is `--run ` reduced to `[A-Za-z0-9._-]`, else +`--<4 hex>`). The secrets scan's added-line copies +(`secrets/`, deleted after the scan), the lane log (`lanes/lanes.log`), each +guard's log (`guards/.log`) and the remote gate's log and file list +(`remote/`) live there, never under `os.tmpdir()`: two pregates in parallel +(two lanes on one machine, a subagent beside its parent) never share a path, +and the log a `fix:` line names survives a host that wipes `/tmp` between +wakes. `pregate.json` records the directory as `scratch` and the table prints +`scratch: `. A `--run ` whose directory already exists (a finished run +under the same name, or a second run beside a live one) is never shared, since +a second run would truncate the first's logs: the run takes `-2` … `-9` +and the record names the directory it got. Runs older than 7 days under the +same project are pruned when a new run starts (the current run never is). An +unwritable state root falls back to a private directory under `os.tmpdir()`, +still one per run, and `--explain` says so. + ## Timing assertions diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4091c3291..3f081d302 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2997,7 +2997,11 @@ names it). `secret` category, in the lines this branch adds. **Fix.** Rotate the credential, remove it from the added lines (history -included), rerun. Not downgradable. +included), rerun. Not downgradable. A benign finding (a dummy token, a run id +read as a phone number, a vendor's public key) is named in the policy's +`allowlist` by finding id or non-secret category with a path glob or an +anchored pattern and a reason ([docs/pregate.md](pregate.md#allowlist)); +`env.value`, the whole `secret` category and `engine.*` never. ### `PREGATE_STRAYS` @@ -3052,8 +3056,10 @@ Other non-Markdown documentation goes in `empty_lanes` on the base branch. Dart files under `integration_test/` or `test_driver/` need a device, so pregate never runs them: declare `"**/integration_test/**": []` (and `"**/test_driver/**": []`) under `dependencies` and let CI run them. -With no policy yet, `gstack-pregate init` drafts the `dependencies` entries -from the repo and lists the files it could not place. `--explain` prints every mapping and why. For a file with no lane it also prints the import +With no policy anywhere (no base copy, no user-owned file) an unmapped file is +the warning [`PREGATE_NO_POLICY`](#pregate-no-policy) with a starter policy +printed, not this error; `gstack-pregate init` writes that draft and lists the +files it could not place. `--explain` prints every mapping and why. For a file with no lane it also prints the import specifiers in test files that resolved to no repo file (for example a `tsconfig.json` `paths` alias such as `@/lib/x` whose target is missing) and any `extends` hop that was not followed (only relative `extends`, at most 4 hops). If @@ -3072,14 +3078,19 @@ should have found it, report the specifier as a resolver gap. ### `PREGATE_INCOMPLETE` -**Meaning.** A check timed out, its tool is unavailable, repo commands were not -allowed, or the working tree changed while the check ran. Incomplete is never -pass and blocks publication. +**Meaning.** A check timed out, a runner process was killed by a signal (out of +memory, an outside kill) and ended without an exit code, repo commands were not +allowed, no runner is declared or detected, or the working tree changed while +the check ran. Incomplete is never pass and blocks publication. A tool missing from PATH, a +network or quota error before any test ran, or a full disk is +[`PREGATE_ENVIRONMENT`](#pregate-environment) instead, so the machine's failure +reads apart from the diff's. -**Fix.** Pass `--allow-repo-commands`, install the tool (for a Dart package the -message names `not on PATH: dart` or `flutter`; fvm also works when the repo pins -it), commit the `pubspec.yaml` of a Dart package the message says has none (or -declare `runner`), raise `--timeout`, or keep the tree still, then rerun. +**Fix.** Pass `--allow-repo-commands`, commit the `pubspec.yaml` of a Dart +package the message says has none (or declare `runner`), raise `--timeout`, or +keep the tree still, then rerun. A runner killed by a signal never passes: read +the lane log, then rerun or split the selection. A tool missing from PATH is +[`PREGATE_ENVIRONMENT`](#pregate-environment), with its own fix. ### `PREGATE_STALLED` @@ -3110,6 +3121,54 @@ as attested rather than validated. file with `--receipt`, or let `gstack-pregate --stage tests --remote --spend-cap --allow-repo-commands` run and validate it. + +### `PREGATE_NO_POLICY` + +**Meaning.** The repository has no pregate policy yet (no `.gstack/pregate.json` +on the base branch and no user-owned `/projects//pregate.json`) +and a touched file maps to no test lane. Under no policy that is a `warn`, not +`PREGATE_NO_LANE`: the selected tests still ran (a red lane is still `fail`), +and the table printed a starter policy drafted from the repository +(`PREGATE_STARTER:` lines: runner, test directories, CI lanes, the JSON between +`begin` and `end`, a `TODO` per file nothing covers). `run` wrote nothing. +Details: [docs/pregate.md](pregate.md#no-policy). + +**Fix.** Review the starter, write it with `gstack-pregate init` (it refuses to +overwrite an existing file without `--force`), edit each `TODO` (a test that +names the file, a `dependencies` entry, or `[]` for a file with no behaviour), +run `gstack-pregate validate`, then commit `.gstack/pregate.json` on the base +branch. From then on an unmapped file is `PREGATE_NO_LANE`. + + +### `PREGATE_ENVIRONMENT` + +**Meaning.** A check could not run its command because of the machine, not the +diff: a runner or guard binary missing from PATH (exit 127, `command not +found`, a tool `not on PATH`), a network error or a quota refusal before any +test was counted, a full disk, an unverified runtime, or gstack's own scanner +not finishing. The check is `incomplete` (never pass) with `cause: +"environment"` and `environment: { kind, evidence, proof }` in `pregate.json`, +and the table prints `environment: — `. A test that failed and +whose log also names a connection error is **not** this code: the runner's own +summary counted a failure, so it is `PREGATE_LANE_FAILED` with `cause: +"diff"`; nor is a runner, guard or gate script whose own binary is present but +exits 127. `proof: "launch"` is certain (the command could not start, its +binary is off PATH, the runtime or a gstack tool refused); `proof: "output"` +means a rule matched the command's text before any test was counted, which the +diff's own output could have written, so the line says `(inferred from output)` +and the fix asks for confirmation elsewhere. Kinds and the rule: +[docs/pregate.md](pregate.md#environment). + +**Fix.** Repair what the `environment:` line names on this machine (install the +tool, restore the network or the quota, free the disk, repair the runtime), or +run the lane where it can run: `gstack-pregate --stage tests --remote +--spend-cap --allow-repo-commands`, or the CI job and `gstack-pregate +clear --remote --run-url `. A lane that waives the run +cites the kind, the evidence and the proof, never a test name; `verify` prints +`PREGATE_ENVIRONMENT: check(s) blocked by the environment (), not by +the diff`, naming any check inferred from output alone, so a coordinator can +tell the two apart and knows which to confirm elsewhere first. + ### `PREGATE_STALE` @@ -3351,7 +3410,10 @@ CLI golden test) from the base branch's `.gstack/pregate.json`, or from the user-owned file when the base carries none, matched a file the branch touched and exited non-zero. Guards cover what the import graph cannot see: manifests, subprocess goldens, registries. They cannot be -downgraded to warn. +downgraded to warn. A guard that could not start, or whose own binary is off +PATH (exit 127), is the machine's: `PREGATE_ENVIRONMENT` (`incomplete`), not +this code; a guard whose binary is present but whose script exits 127 is this +code. **Fix.** Read the guard's lines in the table (exit code, output tail, log path) and do what its `fix` line says, commit, then rerun diff --git a/lib/headless-artifacts.ts b/lib/headless-artifacts.ts index 4b5cfb408..0a4de2773 100644 --- a/lib/headless-artifacts.ts +++ b/lib/headless-artifacts.ts @@ -67,6 +67,8 @@ export interface PregateCheck { id: string; status: 'pass' | 'fail' | 'warn' | 'incomplete' | 'requires-remote' | 'not_installed'; ms?: number; detail?: string; // C-pregate (C8): stage, the input identity the check ran against, the fix clause and result code, and per-item lines. stage?: 'preflight' | 'tests'; inputs?: Record; fix?: string; code?: string; lines?: string[]; + // Lane D (GSTA-112): a non-pass names its cause; an environment cause carries its kind and evidence. + cause?: 'diff' | 'environment'; environment?: { kind: string; evidence: string; proof?: 'launch' | 'output' }; } export interface PregateRemote { lane: string; platform: string; workflow: string; job: string; tests: string[]; receipt: string | null; cleared_by?: string; command?: string } export interface PregateFile { @@ -77,6 +79,8 @@ export interface PregateFile { policy_digest?: string; /** Under the defaults, the user-owned policy path that was looked for and absent; verify checks it is still absent. */ policy_user_path?: string; + /** This run's scratch directory under the state root (logs, scan files). */ + scratch?: string; } export interface ShipReceipt { schema_version: 1; head: string; base: string; pr?: number; version?: string; gated_tree?: string; @@ -192,8 +196,8 @@ export const ARTIFACT_SCHEMAS = { type: 'object', required: ['schema_version', 'tree', 'checks'], properties: { schema_version: { const: 1 }, run: str, tree: str, stage1_ms: num, stage2_ms: num, - checks: { type: 'array', items: { type: 'object', required: ['id', 'status'], properties: { id: str, status: { enum: ['pass', 'fail', 'warn', 'incomplete', 'requires-remote', 'not_installed'] }, ms: num, detail: str, stage: { enum: ['preflight', 'tests'] }, inputs: { type: 'object' }, fix: str, code: str, lines: strList } } }, - head: str, base: str, generated_at: str, policy: str, policy_digest: str, policy_user_path: str, registry: str, verdict: str, + checks: { type: 'array', items: { type: 'object', required: ['id', 'status'], properties: { id: str, status: { enum: ['pass', 'fail', 'warn', 'incomplete', 'requires-remote', 'not_installed'] }, ms: num, detail: str, stage: { enum: ['preflight', 'tests'] }, inputs: { type: 'object' }, fix: str, code: str, lines: strList, cause: { enum: ['diff', 'environment'] }, environment: { type: 'object', required: ['kind', 'evidence'], properties: { kind: str, evidence: str } } } } }, + head: str, base: str, generated_at: str, policy: str, policy_digest: str, policy_user_path: str, registry: str, verdict: str, scratch: str, requires_remote: { type: 'array', items: { type: 'object', required: ['lane', 'platform', 'tests', 'receipt'], properties: { lane: str, platform: str, workflow: str, job: str, tests: strList, receipt: nullableStr, cleared_by: str, command: str } } }, }, }, diff --git a/lib/pregate/allowlist.ts b/lib/pregate/allowlist.ts new file mode 100644 index 000000000..7355d36af --- /dev/null +++ b/lib/pregate/allowlist.ts @@ -0,0 +1,75 @@ +/** + * pregate/allowlist — the policy's `allowlist` applied to the secrets check's + * findings (Lane D, GSTA-112; harness-evals #81/#83, GSTA-102, GSTA-106). An + * entry names a gstack-redact finding id or a non-secret category, and scopes + * it by path globs and/or a regular expression matched at the finding's + * position in the added line (anchored there, so a dummy token's shape never + * covers a real key that happens to share its line). `env.value`, the real + * value of a credential-named environment variable, is never consulted + * against the list: validation refuses an entry naming it, and this matcher + * keeps such a finding whatever the entries say (#3449). + */ +import { matchesAny } from '../ship-policy'; +import { sha256 } from './context'; +import { ALLOWLISTABLE_CATEGORIES, nonAllowlistable, type AllowlistEntry } from './config'; + +export interface AllowlistFinding { + id: string; category: string; severity: string; file: string; line?: number; col?: number; + /** The added line the finding sits on (for `pattern`); absent when the scanner gave no position. */ + lineText?: string; +} +export interface Allowlisted { finding: F; index: number; entry: AllowlistEntry } + +/** Whether `entry` names this finding: its id, or its category when the category may be named whole. */ +function names(entry: AllowlistEntry, f: AllowlistFinding): boolean { + if (entry.finding === f.id) return true; + return (ALLOWLISTABLE_CATEGORIES as readonly string[]).includes(entry.finding) && entry.finding === f.category; +} + +/** Whether a finding has the position a `pattern` needs: the added line and a 1-based column. */ +const positioned = (f: AllowlistFinding) => f.lineText !== undefined && !!f.col; + +/** The pattern, anchored at the finding's column in its added line and ending at a token boundary (so a short pattern never covers the head of a longer value); a finding with no position never matches a pattern. */ +function patternHits(entry: AllowlistEntry, f: AllowlistFinding): boolean { + if (entry.pattern === undefined) return true; + if (!positioned(f)) return false; + let re: RegExp; + try { re = new RegExp(`^(?:${entry.pattern})(?![\\w-])`); } catch { return false; } + return re.test(f.lineText!.slice(f.col! - 1)); +} + +/** + * Split findings into the ones an entry covers (with the entry) and the ones + * kept; env.value and engine.* are always kept. `untested` lists kept findings + * that a pattern entry named by id and path but could not test because the + * scanner gave no position, so the table says why the entry was inert. + */ +export function applyAllowlist(findings: readonly F[], entries: readonly AllowlistEntry[]): { kept: F[]; allowlisted: Allowlisted[]; untested: Allowlisted[] } { + const kept: F[] = []; + const allowlisted: Allowlisted[] = []; + const untested: Allowlisted[] = []; + for (const f of findings) { + const scoped = (e: AllowlistEntry) => names(e, f) && (!e.paths || matchesAny(e.paths, f.file)); + const index = nonAllowlistable(f.id) ? -1 : entries.findIndex(e => scoped(e) && patternHits(e, f)); + if (index >= 0) { allowlisted.push({ finding: f, index, entry: entries[index]! }); continue; } + kept.push(f); + if (nonAllowlistable(f.id) || positioned(f)) continue; + const named = entries.findIndex(e => e.pattern !== undefined && scoped(e)); + if (named >= 0) untested.push({ finding: f, index: named, entry: entries[named]! }); + } + return { kept, allowlisted, untested }; +} + +/** sha256 of the entries, for the check's `inputs.allowlist`. */ +export const allowlistDigest = (entries: readonly AllowlistEntry[]) => sha256(JSON.stringify(entries)); + +/** One table line per finding a pattern entry named but could not test: the finding is kept. */ +export function describeUntested(a: Allowlisted): string { + return `allowlist[${a.index}] ${a.entry.finding} not tested on ${a.finding.id} (${a.finding.file}): the scanner gave no position for its pattern; the finding is kept`; +} + +/** One table line per allowlisted finding: what, where, and which entry (with its reason) covered it. */ +export function describeAllowlisted(a: Allowlisted): string { + const where = a.finding.line ? ` (${a.finding.file}, added line ${a.finding.line})` : ` (${a.finding.file})`; + return `allowlisted ${a.finding.id}${where} by allowlist[${a.index}] ${a.entry.finding}: ${a.entry.reason}`; +} diff --git a/lib/pregate/checks/guards.ts b/lib/pregate/checks/guards.ts index dc28c8144..3f6ca1c46 100644 --- a/lib/pregate/checks/guards.ts +++ b/lib/pregate/checks/guards.ts @@ -13,16 +13,18 @@ * spawns the CLI as a subprocess, so no import graph selects it. */ import * as fs from 'node:fs'; -import * as os from 'node:os'; import * as path from 'node:path'; import { matchesAny } from '../../ship-policy'; import type { Guard } from '../config'; import { baseInputs, sha256 } from '../context'; +import { classifyCommandEnvironment, classifyEnvironment, environmentLine, environmentResult, launchCause } from '../environment'; import { launchRuntime } from '../runtime'; +import { checkScratch } from '../run-scratch'; import { readRange, supervise } from '../supervise'; +import type { EnvironmentCause } from '../types'; import type { CheckDef, CheckResult, PregateContext } from '../types'; -interface GuardRun { guard: Guard; exit: number | null; timedOut: boolean; stalled: boolean; tail: string[]; log: string } +interface GuardRun { guard: Guard; exit: number | null; timedOut: boolean; stalled: boolean; tail: string[]; log: string; environment: EnvironmentCause | null } async function runGuard(ctx: PregateContext, guard: Guard, env: NodeJS.ProcessEnv, dir: string): Promise { const timeoutMs = guard.timeout_s ? guard.timeout_s * 1000 : ctx.timeoutMs; @@ -30,7 +32,10 @@ async function runGuard(ctx: PregateContext, guard: Guard, env: NodeJS.ProcessEn fs.writeFileSync(log, ''); const r = await supervise({ command: guard.command, cwd: ctx.repoRoot, env: { ...env, CI: env.CI ?? '1' }, log, deadline: Date.now() + timeoutMs, stallMs: ctx.stallTimeoutMs, label: `guard ${guard.id}`, progress: ctx.progress }); const output = readRange(log, r.start, r.end).trim(); - return { guard, exit: r.exit, timedOut: r.outcome !== 'exited', stalled: r.outcome === 'stalled', tail: output ? output.split('\n').slice(-15) : [], log }; + // A guard that could not start, or whose own binary is off PATH (exit 127), is the machine's failure; a guard that ran and said no is the diff's, + // whatever its output mentions (a test about rate limits must not read as a quota refusal, and a `bun run lint` whose script names a missing tool is the diff's). + const environment = r.outcome === 'spawn-failed' ? { ...(classifyEnvironment(r.error ?? '', null) ?? launchCause('missing-binary', r.error ?? 'the guard could not be started')), proof: 'launch' as const } : r.outcome === 'exited' && r.exit === 127 ? classifyCommandEnvironment(guard.command, output, r.exit, env) : null; + return { guard, exit: r.exit, timedOut: r.outcome !== 'exited' && r.outcome !== 'spawn-failed', stalled: r.outcome === 'stalled', tail: output ? output.split('\n').slice(-15) : [], log, environment }; } async function run(ctx: PregateContext): Promise { @@ -45,20 +50,23 @@ async function run(ctx: PregateContext): Promise { return { ...base, status: 'incomplete', code: 'REPO_COMMANDS_NOT_ALLOWED', lines: listed, detail: `${selected.length} guard(s) not run (pass --allow-repo-commands): ${selected.map(g => g.id).join(', ')}`, fix: 'rerun with --allow-repo-commands (what a lane runs before reporting done, and what /ship passes)' }; } const runtime = launchRuntime(ctx.repoRoot, ctx.env); - if (runtime.refusal) return { ...base, ...runtime.refusal, lines: [`runtime: ${runtime.label}`] }; - const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-pregate-guards-')); + if (runtime.refusal) return { ...base, ...runtime.refusal, cause: 'environment', environment: launchCause('runtime', runtime.refusal.detail), lines: [`runtime: ${runtime.label}`, environmentLine(launchCause('runtime', runtime.refusal.detail))] }; + const dir = checkScratch(ctx.scratch, 'guards'); const runs: GuardRun[] = []; for (const g of selected) runs.push(await runGuard(ctx, g, runtime.env, dir)); const lines = [`runtime: ${runtime.label}`]; for (const r of runs) { lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s; process group killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`); if (r.exit !== 0) lines.push(...r.tail.map(l => ` ${l}`)); + if (r.environment) lines.push(` ${r.guard.id} ${environmentLine(r.environment)}`); } - const failed = runs.filter(r => !r.timedOut && r.exit !== 0); + const blocked = runs.filter(r => r.environment); + const failed = runs.filter(r => !r.timedOut && !r.environment && r.exit !== 0); const timedOut = runs.filter(r => r.timedOut); const withRuns = { ...base, lines, inputs: { ...inputs, runtime: runtime.label } }; const fixOf = (rs: GuardRun[]) => rs.map(r => `${r.guard.id}: ${r.guard.fix ?? `read ${r.log}, fix the failure, rerun gstack-pregate --only guards`}`).join('; '); - if (failed.length) return { ...withRuns, status: 'fail', code: 'PREGATE_GUARD_FAILED', detail: `${failed.length} guard(s) failed: ${failed.map(r => r.guard.id).join(', ')}${timedOut.length ? `; timed out: ${timedOut.map(r => r.guard.id).join(', ')}` : ''}`, fix: fixOf(failed) }; + if (failed.length) return { ...withRuns, status: 'fail', cause: 'diff', code: 'PREGATE_GUARD_FAILED', detail: `${failed.length} guard(s) failed: ${failed.map(r => r.guard.id).join(', ')}${timedOut.length ? `; timed out: ${timedOut.map(r => r.guard.id).join(', ')}` : ''}${blocked.length ? `; blocked by the environment: ${blocked.map(r => r.guard.id).join(', ')}` : ''}`, fix: fixOf(failed) }; + if (blocked.length) return environmentResult(withRuns, blocked[0]!.environment!, `${blocked.length} guard(s) could not run on this machine: ${blocked.map(r => `${r.guard.id} (${r.environment!.kind})`).join(', ')}${timedOut.length ? `; timed out: ${timedOut.map(r => r.guard.id).join(', ')}` : ''}`); if (timedOut.length) return { ...withRuns, status: 'incomplete', code: timedOut.some(r => r.stalled) ? 'PREGATE_STALLED' : 'PREGATE_INCOMPLETE', detail: `${timedOut.length} guard(s) did not finish: ${timedOut.map(r => `${r.guard.id} (${r.stalled ? 'stalled' : 'timed out'})`).join(', ')}`, fix: 'fix the hang, or raise the guard\'s timeout_s, --timeout or --stall-timeout; an unfinished guard never passes' }; return { ...withRuns, detail: `${runs.length} guard(s) passed: ${runs.map(r => r.guard.id).join(', ')}` }; } diff --git a/lib/pregate/checks/lanes.ts b/lib/pregate/checks/lanes.ts index 2b135254d..1dafa8ddd 100644 --- a/lib/pregate/checks/lanes.ts +++ b/lib/pregate/checks/lanes.ts @@ -4,7 +4,10 @@ * `set -o pipefail` with the exit code and the runner's own summary judged * (zero tests ran is a failure, exit 0 or not); platform lanes this machine * cannot run become persisted `requires-remote` obligations. A touched file - * with no lane is `FAIL lanes — has no test lane`, never a pass. + * with no lane is `FAIL lanes — has no test lane` once any policy + * exists; with no policy anywhere (no base `.gstack/pregate.json`, no + * user-owned file) it is the `warn` PREGATE_NO_POLICY with a starter printed + * (lib/pregate/starter.ts), never a pass either way. * Markdown no test depends on is the docs/no-tests empty lane instead * (lib/pregate/docs-lane.ts): it does not fail, and unmapped code still does. * A release manifest whose diff against the merge base changes only version @@ -15,10 +18,12 @@ * p0-4-windows-lane-unrun (a Windows-only change merged with its lane unrun). */ import * as fs from 'node:fs'; -import * as os from 'node:os'; import * as path from 'node:path'; import { baseInputs, sha256 } from '../context'; import { PREGATE_CONFIG_PATH } from '../config'; +import { classifyCommandEnvironment, classifyEnvironment, environmentLine, environmentResult, launchCause } from '../environment'; +import { checkScratch } from '../run-scratch'; +import { starterPolicy } from '../starter'; import { isDeviceDart, listRepoFiles, readRepoFile } from '../import-graph'; import { selectLanes, type Selection } from '../lane-selection'; import { launchRuntime } from '../runtime'; @@ -219,11 +224,12 @@ export function planRuns(ctx: PregateContext, files: readonly string[]): RunPlan } /** Why `plan` leaves files unrun: their count, the missing SDKs, `.dart` files outside any package, and the fix. */ -function planGap(ctx: PregateContext, plan: RunPlan, total: number): Pick { +function planGap(ctx: PregateContext, plan: RunPlan, total: number): Pick { const { unclaimed, missing, noPubspec } = plan; const detail = `no test runner detected for ${unclaimed.length} of ${total} selected file(s): ${unclaimed.slice(0, 8).join(' ')}${unclaimed.length > 8 ? ' …' : ''}${missing.length ? `; not on PATH: ${missing.join(', ')}` : ''}${noPubspec.length ? `; no pubspec.yaml above ${noPubspec.length} .dart file(s)` : ''}`; - const fix = missing.length ? `install ${missing.join(' and ')}${fvmPinnedFor(ctx.repoRoot, unclaimed) ? ' (or fvm, which this repo pins)' : ''} and rerun` - : noPubspec.length ? `commit the Dart package's pubspec.yaml, or declare \`runner\` (with {files}) in ${ctx.configFile}` + // A tool missing from PATH is this machine's gap, not the diff's; no runner declared or no pubspec is the repo's. + if (missing.length) return { code: 'PREGATE_ENVIRONMENT', cause: 'environment', environment: launchCause('missing-binary', `not on PATH: ${missing.join(', ')}`), detail, fix: `install ${missing.join(' and ')}${fvmPinnedFor(ctx.repoRoot, unclaimed) ? ' (or fvm, which this repo pins)' : ''} on this machine and rerun, or run the lane remotely; the diff is not at fault` }; + const fix = noPubspec.length ? `commit the Dart package's pubspec.yaml, or declare \`runner\` (with {files}) in ${ctx.configFile}` : `declare \`runner\` (with {files}) in ${ctx.configFile}`; return { code: 'PREGATE_INCOMPLETE', detail, fix }; } @@ -270,7 +276,7 @@ export function parseSummary(raw: string, runner = ''): RunnerSummary { return (dartRun ? null : parseDart(output)) ?? parseBats(output) ?? { ran: null, failed: null }; } -interface LocalRun { exit: number | null; outcome: Supervised['outcome']; ran: number | null; failed: number | null; tail: string[]; command: string; files: string[]; lastFile: string | null; ms: number } +interface LocalRun { exit: number | null; outcome: Supervised['outcome']; signal: string | null; observedStart: boolean; ran: number | null; failed: number | null; tail: string[]; runner: string; command: string; files: string[]; lastFile: string | null; ms: number; error?: string } const quote = (f: string) => `'${f.replace(/'/g, `'\\''`)}'`; @@ -282,7 +288,7 @@ async function runOnce(ctx: PregateContext, env: NodeJS.ProcessEnv, g: RunGroup, const r = await supervise({ command: run, cwd: path.join(ctx.repoRoot, g.cwd), env: { ...env, CI: env.CI ?? '1' }, log, deadline: Date.now() + ctx.timeoutMs, stallMs: ctx.stallTimeoutMs, label: `lanes (${shown.length === 1 ? shown[0] : `${shown.length} files`})`, files, adapter: adapterFor(g.runner), counter, progress: ctx.progress }); const output = readRange(log, r.start, r.end); const summary = parseSummary(output, g.runner); - return { exit: r.exit, outcome: r.outcome, ...summary, tail: output.trim().split('\n').slice(-25), command, files: shown, lastFile: r.lastFile && g.cwd ? `${g.cwd}/${r.lastFile}` : r.lastFile, ms: r.ms }; + return { exit: r.exit, outcome: r.outcome, signal: r.signal, observedStart: r.observedStart, ...summary, tail: output.trim().split('\n').slice(-25), runner: g.runner, command, files: shown, lastFile: r.lastFile && g.cwd ? `${g.cwd}/${r.lastFile}` : r.lastFile, ms: r.ms, ...(r.error ? { error: r.error } : {}) }; } /** @@ -297,7 +303,7 @@ async function runOnce(ctx: PregateContext, env: NodeJS.ProcessEnv, g: RunGroup, * count must not hide it). */ async function runLocal(ctx: PregateContext, env: NodeJS.ProcessEnv, groups: RunGroup[]) { - const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-pregate-lanes-')); + const dir = checkScratch(ctx.scratch, 'lanes'); const log = path.join(dir, 'lanes.log'); fs.writeFileSync(log, ''); const counter: Counter = { done: 0, total: groups.reduce((n, g) => n + g.files.length, 0) }; @@ -316,19 +322,28 @@ async function runLocal(ctx: PregateContext, env: NodeJS.ProcessEnv, groups: Run } /** Why the local lane did not finish: a stall, a runner process past --timeout or a launch failure, with the last file started. */ -function unfinished(ctx: PregateContext, r: Awaited>, total: number): Pick { +function unfinished(ctx: PregateContext, r: Awaited>, total: number): Pick { const stalled = r.runs.filter(x => x.outcome === 'stalled'); const timedOut = r.runs.filter(x => x.outcome === 'timeout'); const rest = r.failed ? `; ${r.failed} failed` : ''; if (stalled.length) { return { code: 'PREGATE_STALLED', detail: `no test output for ${ctx.stallTimeoutMs / 1000} s; process group killed; last file started: ${stalled.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest}`, fix: `read ${r.log} for the stalled file; fix the hang, or raise --stall-timeout (stall_timeout_s in ${ctx.configFile}; 0 disables) for a test that is quiet for longer` }; } - if (!timedOut.length) return { code: 'PREGATE_INCOMPLETE', detail: `the runner could not be launched (${total} files${rest})`, fix: 'read the lane log, fix the runner command, then rerun gstack-pregate --stage tests' }; + const launch = r.runs.find(x => x.outcome === 'spawn-failed'); + if (launch) { + const environment = { ...(classifyEnvironment(launch.error ?? '', null) ?? launchCause('missing-binary', launch.error ?? 'the runner could not be started')), proof: 'launch' as const }; + return { environment, detail: `the runner could not be launched (${total} files${rest}): ${environment.evidence}` }; + } + if (!timedOut.length) { + // The runner started and died without an exit code (a signal: OOM kill, SIGSEGV, an outside kill): incomplete, not the machine's missing binary. + const killed = r.runs.find(x => x.outcome === 'exited' && x.exit === null); + return { code: 'PREGATE_INCOMPLETE', detail: `the runner process ended without an exit code (killed by ${killed?.signal ?? 'a signal'}; ${total} files; last file started: ${killed?.lastFile ?? 'none reported'}${rest})`, fix: `read ${r.log}; a runner killed by a signal (out of memory, an outside kill) never passes; rerun, or split the selection` }; + } return { code: 'PREGATE_INCOMPLETE', detail: `a runner process timed out after ${ctx.timeoutMs} ms (--timeout applies to each runner process; ${total} files; last file started: ${timedOut.map(x => x.lastFile ?? `(none reported; ${x.files.length} file(s))`).join(', ')}${rest})`, fix: 'raise --timeout or split the selection; a timed-out lane never passes' }; } -function describe(sel: Selection): string[] { - const lines = sel.noLane.map(f => `FAIL lanes — ${f} has no test lane`); +function describe(sel: Selection, noPolicy = false): string[] { + const lines = noPolicy ? [`no policy: ${sel.noLane.join(' ')} have no test lane (a warning under no policy; the starter below maps them)`] : sel.noLane.map(f => `FAIL lanes — ${f} has no test lane`); if (sel.emptyLane.length) lines.push(`docs/no-tests (empty lane): ${sel.emptyLane.join(' ')}`); if (sel.inert.length) lines.push(`no-tests (assets and tool config): ${sel.inert.join(' ')}`); if (sel.stampOnly.length) lines.push(`version-stamp-only (no lane needed): ${sel.stampOnly.join(' ')} — only version values changed against the merge base; a manifest whose other content changed still needs a lane`); @@ -355,6 +370,8 @@ function remoteObligation(ctx: PregateContext, sel: Selection): RemoteObligation } export const REMOTE_GATE_LANE = 'pregate.json/remote_gate'; +/** How many leading lines of a remote gate's output are read for an environment cause: its preamble, before any test output could stream. */ +export const REMOTE_PREAMBLE_LINES = 40; /** The `clear` command for an obligation: the remote gate's own output for remote_gate, a run URL otherwise. */ export const clearHint = (lane: string) => `gstack-pregate clear --remote ${lane} ${lane === REMOTE_GATE_LANE ? '--receipt ' : '--run-url '}`; export interface RemoteReport { tree: string; tests: number; failed: number; spent?: string } @@ -386,7 +403,7 @@ async function runRemote(ctx: PregateContext, sel: Selection, base: CheckResult) if (!ctx.config.remote_gate) return { ...base, status: 'incomplete', code: 'PREGATE_INCOMPLETE', detail: `--remote runs only a declared remote gate; ${ctx.configLabel} has no remote_gate`, fix: `declare remote_gate (a command that prints PREGATE_REMOTE: tree=… tests=… failed=…) in ${ctx.configFile === PREGATE_CONFIG_PATH ? `${PREGATE_CONFIG_PATH} on the base branch` : ctx.configFile}, or run the tests locally without --remote` }; const obligation = remoteObligation(ctx, sel); const cap = ctx.remote!.spendCapUsd; - const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-pregate-remote-')); + const dir = checkScratch(ctx.scratch, 'remote'); const log = path.join(dir, 'remote.log'); const filesFile = path.join(dir, 'files.txt'); fs.writeFileSync(log, ''); @@ -402,21 +419,20 @@ async function runRemote(ctx: PregateContext, sel: Selection, base: CheckResult) const why = r.outcome !== 'exited' ? `the remote gate did not finish (${r.outcome === 'stalled' ? `no output for ${ctx.stallTimeoutMs / 1000} s` : r.outcome})` : !reported ? `the remote gate exited ${r.exit} without a PREGATE_REMOTE: line` : remoteReportProblem(reported, ctx.wtree) ?? (r.exit !== 0 ? `the remote gate exited ${r.exit}` : null); + // A gate that printed no report and names a quota, network or missing-tool error in its preamble never ran the tests: the environment, not the diff. + // Only the gate's first lines are read (a VM refusal or a network error comes before any test streams); exit 127 counts only when the gate's own binary is off PATH. + const environment = why && !reported && r.outcome !== 'stalled' && r.outcome !== 'timeout' + ? r.outcome === 'spawn-failed' ? { ...(classifyEnvironment(r.error ?? '', null) ?? launchCause('missing-binary', r.error ?? 'the remote gate could not be started')), proof: 'launch' as const } : classifyCommandEnvironment(obligation.command!, output.split('\n').slice(0, REMOTE_PREAMBLE_LINES).join('\n'), r.exit, ctx.env) + : null; + if (environment) return environmentResult({ ...base, inputs, lines: runLines, remote: [...sel.remote, obligation] }, environment, `${why}; requires-remote ${obligation.lane} stays uncleared`); if (why) return { ...base, inputs, lines: runLines, status: 'incomplete', remote: [...sel.remote, obligation], code: r.outcome === 'stalled' ? 'PREGATE_STALLED' : 'PREGATE_INCOMPLETE', detail: `${why}; requires-remote ${obligation.lane} stays uncleared`, fix: `read ${log}; the obligation clears only on PREGATE_REMOTE: tree=${ctx.wtree.slice(0, 12)}… tests>0 failed=0 with exit 0` }; const cleared = { ...obligation, receipt: `remote-gate tree=${ctx.wtree.slice(0, 12)} tests=${reported!.tests}`, cleared_by: `gstack-pregate --remote ${new Date().toISOString()}` }; return { ...base, inputs, lines: runLines, status: 'requires-remote', remote: [...sel.remote, cleared], detail: `remote gate ${obligation.lane}: ${reported!.tests} tests, 0 failed on tree ${ctx.wtree.slice(0, 12)}; ${spend}` }; } -async function run(ctx: PregateContext): Promise { - const sel = selectLanes(ctx); - ctx.explain.push(...sel.provenance); - for (const [lane, set] of Object.entries(sel.laneSets)) ctx.explain.push(`lane ${lane}: ${set === '?' ? 'file set unknown' : `${set.length} files`}`); - const inputs = baseInputs(ctx, { selection: sha256(sel.tests.join('\n')), localTests: sha256(sel.localTests.join('\n')), remote: sha256(sel.remote.map(r => `${r.lane}:${r.tests.join(',')}`).join('\n')) }); - const lines = describe(sel); - const base: CheckResult = { id: 'lanes', stage: 'tests', status: 'pass', inputs, lines, remote: sel.remote }; - if (sel.noLane.length) { - return { ...base, status: 'fail', code: 'PREGATE_NO_LANE', detail: `${sel.noLane.length} touched file(s) with no test lane: ${sel.noLane.join(' ')}`, fix: noLaneFix(ctx, sel.noLane) }; - } +/** The lanes verdict for a selection: the selected tests run (locally, remotely or as obligations) and the result names the diff or the machine. */ +async function runSelected(ctx: PregateContext, sel: Selection, base: CheckResult): Promise { + const { inputs, lines = [] } = base; if (sel.tests.length === 0) { const classes = [sel.emptyLane.length ? `${sel.emptyLane.length} documentation file(s)` : '', sel.inert.length ? `${sel.inert.length} asset or tool-config file(s)` : '', sel.stampOnly.length ? `${sel.stampOnly.length} version-stamp-only manifest(s)` : ''].filter(Boolean); const label = sel.emptyLane.length ? 'docs/no-tests' : sel.inert.length ? 'no-tests' : 'version-stamp-only'; @@ -435,23 +451,61 @@ async function run(ctx: PregateContext): Promise { // Runnable groups run even when other files are unclaimed: a real failure is reported as one, and an all-green run is still incomplete. const plan = planRuns(ctx, sel.localTests); const gap = plan.unclaimed.length ? planGap(ctx, plan, sel.localTests.length) : null; - if (!plan.groups.length) return { ...base, status: 'incomplete', ...gap }; + if (!plan.groups.length) return { ...base, status: 'incomplete', ...gap, ...(gap?.environment ? { lines: [...lines, environmentLine(gap.environment)] } : {}) }; const { groups } = plan; const total = groups.reduce((n, g) => n + g.files.length, 0); const runtime = launchRuntime(ctx.repoRoot, ctx.env); - if (runtime.refusal) return { ...base, ...runtime.refusal, lines: [`runtime: ${runtime.label}`] }; + if (runtime.refusal) return { ...base, ...runtime.refusal, cause: 'environment', environment: launchCause('runtime', runtime.refusal.detail), lines: [`runtime: ${runtime.label}`, environmentLine(launchCause('runtime', runtime.refusal.detail))] }; const r = await runLocal(ctx, runtime.env, groups); const adapters = [...new Set(groups.map(g => adapterFor(g.runner)).filter(Boolean))]; const runLines = [...lines, `runtime: ${runtime.label}`, `progress: ${adapters.join(', ') || 'none (no progress adapter for this runner; serial files and heartbeats only)'}`, `ran: ${r.command.length > 200 ? `${r.command.slice(0, 200)}…` : r.command}`, `log: ${r.log}`, ...r.tail.map(l => ` ${l}`)]; const withRun = { ...base, lines: runLines, inputs: { ...inputs, command: sha256(r.command), runtime: runtime.label } }; - if (r.exit === null) return { ...withRun, status: 'incomplete', ...unfinished(ctx, r, total) }; - if (r.exit !== 0) return { ...withRun, status: 'fail', code: 'PREGATE_LANE_FAILED', detail: `local lane exited ${r.exit} (${r.failed ?? '?'} failed of ${r.ran ?? '?'} ran; ${total} files; pipefail)`, fix: `read ${r.log}, fix the in-branch failure, rerun gstack-pregate --stage tests` }; - if (r.ran === 0) return { ...withRun, status: 'fail', code: 'PREGATE_LANE_FAILED', detail: `ZERO-RUN: the runner reported 0 tests${r.zero ? ` in ${r.zero}` : ''} (${total} selected file(s))`, fix: 'a bad selector, a missing file or a skipped describe; exit 0 is not a pass' }; + if (r.exit === null) { + const why = unfinished(ctx, r, total); + return why.environment ? environmentResult(withRun, why.environment, why.detail!) : { ...withRun, status: 'incomplete', ...why }; + } + if (r.exit !== 0) { + // The runner's own summary decides: a counted failure is the diff's. Only a run that reported no test at all, and whose runner was not seen + // starting a test file (a test that exits the process before the summary cannot relabel itself), is read from its output; exit 127 counts + // only when the runner's binary is off PATH. + const environment = r.ran === null && r.bad && !r.bad.observedStart ? classifyCommandEnvironment(r.bad.runner, r.tail.join('\n'), r.exit, runtime.env) : null; + if (environment) return environmentResult(withRun, environment, `the runner exited ${r.exit} before any test was counted (${total} files): ${environment.evidence}`); + return { ...withRun, status: 'fail', cause: 'diff', code: 'PREGATE_LANE_FAILED', detail: `local lane exited ${r.exit} (${r.failed ?? '?'} failed of ${r.ran ?? '?'} ran; ${total} files; pipefail)`, fix: `read ${r.log}, fix the in-branch failure, rerun gstack-pregate --stage tests` }; + } + if (r.ran === 0) return { ...withRun, status: 'fail', cause: 'diff', code: 'PREGATE_LANE_FAILED', detail: `ZERO-RUN: the runner reported 0 tests${r.zero ? ` in ${r.zero}` : ''} (${total} selected file(s))`, fix: 'a bad selector, a missing file or a skipped describe; exit 0 is not a pass' }; const detail = `${r.ran ?? 'count unavailable'} tests ran in ${total} file(s)${r.serial ? ` (${r.serial} serial)` : ''}, exit 0${sel.remote.length ? `; ${sel.remote.length} requires-remote obligation(s)` : ''}`; - if (gap) return { ...withRun, status: 'incomplete', ...gap, detail: `${detail}; ${gap.detail}` }; + if (gap) return { ...withRun, status: 'incomplete', ...gap, detail: `${detail}; ${gap.detail}`, ...(gap.environment ? { lines: [...runLines, environmentLine(gap.environment)] } : {}) }; return { ...withRun, status: sel.remote.length ? 'requires-remote' : 'pass', detail }; } +/** + * With no policy anywhere (no base `.gstack/pregate.json`, no user-owned + * file), an unmapped file is a warning with a starter policy, not a hard + * failure: the selected tests still ran and their verdict stands; a green run + * becomes `warn` (PREGATE_NO_POLICY) so the first /ship in a new repo is not + * blocked before the owner has written a lane policy. + */ +function withStarter(ctx: PregateContext, sel: Selection, result: CheckResult): CheckResult { + const lines = [...(result.lines ?? []), ...starterPolicy(ctx, sel.noLane, detectRunner(ctx, sel.localTests.length ? sel.localTests : sel.tests))]; + const noted = `${sel.noLane.length} touched file(s) with no test lane under no policy (starter printed)`; + if (result.status !== 'pass') return { ...result, lines, detail: `${result.detail}; ${noted}`, ...(result.status === 'requires-remote' && !result.code ? { code: 'PREGATE_NO_POLICY' } : {}) }; + return { ...result, status: 'warn', code: 'PREGATE_NO_POLICY', lines, detail: `${result.detail}; ${noted}`, fix: `review the starter above, write it with gstack-pregate init, edit each TODO, then commit ${PREGATE_CONFIG_PATH} on the base branch; from then on an unmapped file is PREGATE_NO_LANE` }; +} + +async function run(ctx: PregateContext): Promise { + const sel = selectLanes(ctx); + ctx.explain.push(...sel.provenance); + for (const [lane, set] of Object.entries(sel.laneSets)) ctx.explain.push(`lane ${lane}: ${set === '?' ? 'file set unknown' : `${set.length} files`}`); + const inputs = baseInputs(ctx, { selection: sha256(sel.tests.join('\n')), localTests: sha256(sel.localTests.join('\n')), remote: sha256(sel.remote.map(r => `${r.lane}:${r.tests.join(',')}`).join('\n')) }); + const noPolicy = sel.noLane.length > 0 && ctx.configKind === 'none'; + const base: CheckResult = { id: 'lanes', stage: 'tests', status: 'pass', inputs, lines: describe(sel, noPolicy), remote: sel.remote }; + if (sel.noLane.length && !noPolicy) { + return { ...base, status: 'fail', cause: 'diff', code: 'PREGATE_NO_LANE', detail: `${sel.noLane.length} touched file(s) with no test lane: ${sel.noLane.join(' ')}`, fix: noLaneFix(ctx, sel.noLane) }; + } + const result = await runSelected(ctx, sel, base); + return noPolicy ? withStarter(ctx, sel, result) : result; +} + function noLaneFix(ctx: PregateContext, noLane: readonly string[]): string { const device = noLane.some(isDeviceDart) ? `; pregate does not run device tests, so declare "**/integration_test/**": [] (and "**/test_driver/**": []) and let CI run them` : ''; return `add a test that names the file, or declare its lane (or []) under dependencies — or empty_lanes for other documentation — in ${ctx.configFile}${device}`; diff --git a/lib/pregate/checks/literals.ts b/lib/pregate/checks/literals.ts index bca67b438..034de66db 100644 --- a/lib/pregate/checks/literals.ts +++ b/lib/pregate/checks/literals.ts @@ -59,17 +59,27 @@ export function changedLiterals(file: string, oldText: string, newText: string): function escape(s: string): string { return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); } function unquote(v: string): string { return /^['"`]/.test(v) ? v.slice(1, -1) : v; } -/** Test files that still carry the old literal beside the constant's name or its module stem. */ +/** + * Test files that still carry the old literal beside the constant's name or + * its module. The module counts as its full file name (`install.sh`) or as a + * path fragment (`/retry.js`, `/retry'`): a bare stem such as `install` or `retry` is an + * English word in half the test tree, and a one-character old value (`2`) + * sits in every test, so that value needs the constant's name (a wrapper + * version 2 → 3 flagged 575 consumers through the stem alone). + */ export function consumersOf(repoRoot: string, testFiles: readonly string[], change: ChangedLiteral): string[] { const value = unquote(change.old); const valueRe = /^-?\d/.test(value) ? new RegExp(`(? ({ file, text: lines.join('\n') })).filter(p => p.text.trim()); } - -type Finding = { id?: string; description?: string; severity?: string; category?: string; line?: number; file: string }; +type Finding = { id?: string; description?: string; severity?: string; category?: string; line?: number; col?: number; file: string }; +type Scanned = Finding & AllowlistFinding; function run(ctx: PregateContext): CheckResult { const parts = addedLines(ctx); - const inputs = baseInputs(ctx, { added_lines: sha256(JSON.stringify(parts)) }); + const inputs = baseInputs(ctx, { added_lines: sha256(JSON.stringify(parts)), ...(ctx.config.allowlist.length ? { allowlist: allowlistDigest(ctx.config.allowlist) } : {}) }); if (!parts.length) return { id: 'secrets', stage: 'preflight', status: 'pass', detail: 'no added lines', inputs }; - const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-pregate-secrets-')); + // A run killed mid-scan under the same --run id must not keep its added-line copies: clear them before writing new ones. + fs.rmSync(path.join(ctx.scratch, 'secrets'), { recursive: true, force: true }); + const tmp = checkScratch(ctx.scratch, 'secrets'); + const base: CheckResult = { id: 'secrets', stage: 'preflight', status: 'incomplete', inputs }; try { - const findings: Finding[] = []; + const findings: Scanned[] = []; let high = false; let medium = false; + let parsedAll = true; for (const [i, part] of parts.entries()) { const file = path.join(tmp, String(i), path.basename(part.file) || 'added.txt'); fs.mkdirSync(path.dirname(file), { mode: 0o700 }); fs.writeFileSync(file, part.text, { mode: 0o600 }); const r = runScriptSync(path.join(BIN_DIR, 'gstack-redact'), ['--from-file', file, '--json', '--max-bytes', String(64 * 1024 * 1024)], { cwd: ctx.repoRoot, encoding: 'utf8', timeout: Math.min(ctx.timeoutMs, 300_000), maxBuffer: 64 * 1024 * 1024, env: ctx.env }); - if (r.error || r.status === null) return { id: 'secrets', stage: 'preflight', status: 'incomplete', inputs, code: 'PREGATE_INCOMPLETE', detail: `gstack-redact did not finish on ${part.file || 'added lines'} (${r.error?.message ?? 'killed'})`, fix: 'rerun; raise --timeout for very large diffs' }; - if (r.status !== 0 && r.status !== 2 && r.status !== 3) return { id: 'secrets', stage: 'preflight', status: 'incomplete', inputs, code: 'PREGATE_INCOMPLETE', detail: `gstack-redact exited ${r.status} on ${part.file || 'added lines'}: ${(r.stderr ?? '').trim().split('\n').pop() ?? ''}`, fix: 'fix the scanner invocation; an incomplete scan never passes' }; + if (r.error || r.status === null) return environmentResult(base, launchCause('tool', `gstack-redact did not finish on ${part.file || 'added lines'} (${r.error?.message ?? 'killed'})`), `gstack-redact did not finish on ${part.file || 'added lines'}; an incomplete scan never passes`); + if (r.status !== 0 && r.status !== 2 && r.status !== 3) return environmentResult(base, launchCause('tool', `gstack-redact exited ${r.status} on ${part.file || 'added lines'}: ${(r.stderr ?? '').trim().split('\n').pop() ?? ''}`), `gstack-redact exited ${r.status}; an incomplete scan never passes`); high ||= r.status === 3; medium ||= r.status === 2; - try { findings.push(...(JSON.parse(r.stdout).findings ?? []).map((f: Omit) => ({ ...f, file: part.file }))); } catch { /* non-JSON output: judged by exit status */ } + const lines = part.text.split('\n'); + try { findings.push(...(JSON.parse(r.stdout).findings ?? []).map((f: Omit) => ({ ...f, id: f.id ?? 'finding', category: f.category ?? '?', severity: f.severity ?? '?', file: part.file, lineText: f.line ? lines[f.line - 1] : undefined }))); } catch { parsedAll = false; } } - const render = (f: Finding) => `${f.severity} ${f.id ?? 'finding'} [${f.category ?? '?'}]: ${f.description ?? ''}${f.line ? ` (${f.file ? `${f.file}, ` : ''}added line ${f.line})` : ''}`; + // The allowlist applies only to findings the scanner reported in full; an unparsed report keeps its exit-code verdict. + const applied = parsedAll && ctx.config.allowlist.length ? applyAllowlist(findings, ctx.config.allowlist) : { kept: findings, allowlisted: [], untested: [] }; + const kept = applied.kept; + if (parsedAll) { high = kept.some(f => f.severity === 'HIGH'); medium = kept.some(f => f.severity === 'MEDIUM'); } + const render = (f: Scanned) => `${f.severity} ${f.id} [${f.category}]: ${f.description ?? ''}${f.line ? ` (${f.file ? `${f.file}, ` : ''}added line ${f.line})` : ''}`; + const allowLines = [...applied.allowlisted.map(describeAllowlisted), ...applied.untested.map(describeUntested)]; + const allowNote = applied.allowlisted.length ? `; ${applied.allowlisted.length} allowlisted (${[...new Set(applied.allowlisted.map(a => a.finding.id))].join(', ')})` : ''; // HIGH and any MEDIUM credential fail; a MEDIUM hygiene/internal/pii/legal // hit in code (a `.local` property reads as a hostname) is listed as warn: // there is no user to answer gstack-redact's per-finding question here. - const blocking = findings.filter(f => f.severity === 'HIGH' || (f.severity === 'MEDIUM' && f.category === 'secret')); - const advisory = findings.filter(f => f.severity === 'MEDIUM' && f.category !== 'secret'); + const blocking = kept.filter(f => f.severity === 'HIGH' || (f.severity === 'MEDIUM' && f.category === 'secret')); + const advisory = kept.filter(f => f.severity === 'MEDIUM' && f.category !== 'secret'); const lineCount = parts.reduce((n, p) => n + p.text.split('\n').length, 0); - if (!high && !medium) return { id: 'secrets', stage: 'preflight', status: 'pass', inputs, detail: `${lineCount} added lines clean${findings.length ? ` (${findings.length} WARN)` : ''}` }; - if (blocking.length) return { id: 'secrets', stage: 'preflight', status: 'fail', inputs, lines: [...blocking, ...advisory].map(render), code: 'PREGATE_SECRETS', detail: `${blocking.length} credential finding(s) in added lines (${blocking.map(f => f.id).join(', ')})`, fix: 'rotate and remove the credential from the added lines (history included); secrets cannot be downgraded' }; - if (high) return { id: 'secrets', stage: 'preflight', status: 'fail', inputs, lines: findings.map(render), code: 'PREGATE_SECRETS', detail: 'HIGH finding(s) in added lines', fix: 'rotate and remove the credential from the added lines (history included); secrets cannot be downgraded' }; - return { id: 'secrets', stage: 'preflight', status: 'warn', inputs, lines: advisory.map(render), code: 'PREGATE_SECRETS', detail: `${advisory.length} MEDIUM non-credential finding(s) in added lines (${[...new Set(advisory.map(f => f.id))].join(', ')})`, fix: 'read each listed line; gstack-post asks about the same findings when the PR body quotes them' }; + const fail = (lines: string[], detail: string): CheckResult => ({ ...base, status: 'fail', cause: 'diff', lines: [...lines, ...allowLines], code: 'PREGATE_SECRETS', detail: `${detail}${allowNote}`, fix: 'rotate and remove the credential from the added lines (history included); secrets cannot be downgraded (a benign finding is allowlisted by id, path and pattern in the policy, never env.value)' }); + if (!high && !medium) return { ...base, status: 'pass', ...(allowLines.length ? { lines: allowLines } : {}), detail: `${lineCount} added lines clean${kept.length ? ` (${kept.length} WARN)` : ''}${allowNote}` }; + if (blocking.length) return fail([...blocking, ...advisory].map(render), `${blocking.length} credential finding(s) in added lines (${blocking.map(f => f.id).join(', ')})`); + if (high) return fail(kept.map(render), 'HIGH finding(s) in added lines'); + return { ...base, status: 'warn', cause: 'diff', lines: [...advisory.map(render), ...allowLines], code: 'PREGATE_SECRETS', detail: `${advisory.length} MEDIUM non-credential finding(s) in added lines (${[...new Set(advisory.map(f => f.id))].join(', ')})${allowNote}`, fix: 'read each listed line; gstack-post asks about the same findings when the PR body quotes them; a benign one is allowlisted by id or category, path and pattern in the policy' }; } finally { fs.rmSync(tmp, { recursive: true, force: true }); } } diff --git a/lib/pregate/config.ts b/lib/pregate/config.ts index e1f2b3e20..8e20540fa 100644 --- a/lib/pregate/config.ts +++ b/lib/pregate/config.ts @@ -33,6 +33,12 @@ * remote_gate the repo's remote test gate command: under 8 cores the test * tier is a requires-remote obligation naming it, and * `--remote` runs it (it prints `PREGATE_REMOTE: tree=… tests=… failed=…`) + * allowlist `{ finding, paths?, pattern?, reason }` entries the `secrets` + * check accepts as benign: a gstack-redact finding id or a + * non-secret category, scoped by path globs and/or a regex + * anchored at the finding (lib/pregate/allowlist.ts); never + * `env.value` (a configured credential's real value) and never + * the whole `secret` category * * A repo that keeps `.gstack/` out of git can hold the same file at * `/projects//pregate.json` (lib/state-root.ts), honoured @@ -50,6 +56,12 @@ import { containedRelative, globToRegExp } from '../ship-policy'; export const PREGATE_CONFIG_PATH = '.gstack/pregate.json'; export const PREGATE_CONFIG_SCHEMA = 'https://github.com/garrytan/gstack/docs/pregate.md#pregate-v1'; export const NON_DOWNGRADABLE = ['secrets', 'guards'] as const; +/** Finding ids (or categories) no allowlist entry may name: the real value of a credential-named environment variable (#3449), and the secret wildcard. */ +export const NON_ALLOWLISTABLE = ['env.value', 'secret'] as const; +/** Whether no entry may name this finding id: the list above, plus the scanner's own `engine.*` refusals (an oversize input fails closed and is never benign). */ +export const nonAllowlistable = (id: string) => (NON_ALLOWLISTABLE as readonly string[]).includes(id) || id.startsWith('engine.'); +/** Categories an allowlist entry may name whole; a secret is always named by its id. */ +export const ALLOWLISTABLE_CATEGORIES = ['pii', 'internal', 'hygiene', 'legal'] as const; export const DEFAULT_SERIAL_PATTERNS = ['**/*.serial.test.*']; export interface PregateConfig { @@ -67,17 +79,23 @@ export interface PregateConfig { price_file: string | null; stall_timeout_s: number | null; remote_gate: string | null; + allowlist: AllowlistEntry[]; } export interface Guard { id: string; when: string[]; command: string; fix?: string; timeout_s?: number } -export const DEFAULT_CONFIG: PregateConfig = { runner: null, dependencies: {}, empty_lanes: [], selection: {}, warn: [], strays: { allow: [] }, local: [], guards: [], serial_patterns: DEFAULT_SERIAL_PATTERNS, price_file: null, stall_timeout_s: null, remote_gate: null }; +export interface AllowlistEntry { finding: string; paths?: string[]; pattern?: string; reason: string } +export const DEFAULT_CONFIG: PregateConfig = { runner: null, dependencies: {}, empty_lanes: [], selection: {}, warn: [], strays: { allow: [] }, local: [], guards: [], serial_patterns: DEFAULT_SERIAL_PATTERNS, price_file: null, stall_timeout_s: null, remote_gate: null, allowlist: [] }; export interface ConfigError { code: ResultCodeName; path: string; message: string } export const MAX_LIST = 256; const MAX_COMMAND = 512; -const KEYS = ['runner', 'dependencies', 'empty_lanes', 'selection', 'warn', 'strays', 'local', 'guards', 'serial_patterns', 'price_file', 'sealed', 'stall_timeout_s', 'remote_gate']; +const KEYS = ['runner', 'dependencies', 'empty_lanes', 'selection', 'warn', 'strays', 'local', 'guards', 'serial_patterns', 'price_file', 'sealed', 'stall_timeout_s', 'remote_gate', 'allowlist']; const LANE_ID = /^[A-Za-z0-9_.-]+\.ya?ml\/[A-Za-z0-9_-]+$/; const MAX_GUARDS = 32; const GUARD_KEYS = ['id', 'when', 'command', 'fix', 'timeout_s']; +const MAX_ALLOWLIST = 64; +const MAX_PATTERN = 256; +const ALLOWLIST_KEYS = ['finding', 'paths', 'pattern', 'reason']; +const FINDING_ID = /^[a-z][a-z0-9_]*(?:\.[a-z][a-z0-9_]*)*$/; function globError(g: string): string | null { const why = containedRelative(g); @@ -113,6 +131,30 @@ function guardErrors(raw: unknown, errors: ConfigError[]): void { }); } +function allowlistErrors(raw: unknown, errors: ConfigError[]): void { + const bad = (p: string, message: string) => errors.push({ code: 'PREGATE_CONFIG_INVALID', path: p, message }); + if (!Array.isArray(raw)) { bad('allowlist', 'must be an array of { finding, paths?, pattern?, reason }'); return; } + if (raw.length > MAX_ALLOWLIST) bad('allowlist', `more than ${MAX_ALLOWLIST} entries`); + raw.forEach((e, i) => { + const at = `allowlist[${i}]`; + if (!e || typeof e !== 'object' || Array.isArray(e)) { bad(at, 'must be an object { finding, paths?, pattern?, reason }'); return; } + const o = e as Record; + for (const k of Object.keys(o)) if (!ALLOWLIST_KEYS.includes(k)) bad(`${at}.${k}`, `unknown key; valid keys: ${ALLOWLIST_KEYS.join(', ')}`); + if (typeof o.finding !== 'string' || !FINDING_ID.test(o.finding)) bad(`${at}.finding`, 'must be a gstack-redact finding id (env.kv, google.api_key, pii.phone.e164, …) or one of the categories pii, internal, hygiene, legal'); + else if (nonAllowlistable(o.finding)) bad(`${at}.finding`, `${o.finding} cannot be allowlisted (${o.finding === 'env.value' ? 'the real value of a credential-named environment variable is never benign' : o.finding === 'secret' ? 'name the secret finding by its id, never the whole category' : 'an engine.* finding is the scanner refusing its input, not a finding in it'})`); + if (!('paths' in o) && !('pattern' in o)) bad(at, 'needs paths, pattern or both: an entry with no scope would allowlist every finding of that kind'); + if ('paths' in o) { + if (!Array.isArray(o.paths) || o.paths.length === 0 || o.paths.some(p => typeof p !== 'string')) bad(`${at}.paths`, 'must be a non-empty array of globs'); + else (o.paths as string[]).forEach((g, j) => { const why = globError(g); if (why) bad(`${at}.paths[${j}]`, `${JSON.stringify(g)} ${why}`); }); + } + if ('pattern' in o) { + if (typeof o.pattern !== 'string' || !o.pattern.trim() || o.pattern.length > MAX_PATTERN) bad(`${at}.pattern`, `must be a regular expression under ${MAX_PATTERN} characters, matched at the finding's position in the added line`); + else { try { new RegExp(o.pattern); } catch (err: any) { bad(`${at}.pattern`, `not a valid regular expression: ${err.message}`); } } + } + if (typeof o.reason !== 'string' || !o.reason.trim()) bad(`${at}.reason`, 'must say why the finding is benign (the record repeats it)'); + }); +} + export function validateConfig(raw: unknown, knownChecks: readonly string[] = []): ConfigError[] { const errors: ConfigError[] = []; if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return [{ code: 'PREGATE_CONFIG_INVALID', path: '', message: 'config must be a JSON object' }]; @@ -135,6 +177,7 @@ export function validateConfig(raw: unknown, knownChecks: readonly string[] = [] if (why) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: 'remote_gate', message: why }); } if ('guards' in obj) guardErrors(obj.guards, errors); + if ('allowlist' in obj) allowlistErrors(obj.allowlist, errors); if ('sealed' in obj) for (const e of sealedErrors(obj.sealed)) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: e.path, message: e.message }); if ('serial_patterns' in obj) { const list = obj.serial_patterns; @@ -201,7 +244,7 @@ export function parseConfig(text: string, knownChecks: readonly string[] = []): const errors = validateConfig(raw, knownChecks); if (errors.length) return { config: DEFAULT_CONFIG, errors }; const o = raw as Partial; - return { config: { $schema: o.$schema, runner: o.runner ?? null, dependencies: o.dependencies ?? {}, empty_lanes: o.empty_lanes ?? [], selection: o.selection ?? {}, warn: o.warn ?? [], strays: { allow: o.strays?.allow ?? [] }, local: o.local ?? [], guards: o.guards ?? [], serial_patterns: o.serial_patterns ?? DEFAULT_SERIAL_PATTERNS, price_file: o.price_file ?? null, stall_timeout_s: o.stall_timeout_s ?? null, remote_gate: o.remote_gate ?? null }, errors: [] }; + return { config: { $schema: o.$schema, runner: o.runner ?? null, dependencies: o.dependencies ?? {}, empty_lanes: o.empty_lanes ?? [], selection: o.selection ?? {}, warn: o.warn ?? [], strays: { allow: o.strays?.allow ?? [] }, local: o.local ?? [], guards: o.guards ?? [], serial_patterns: o.serial_patterns ?? DEFAULT_SERIAL_PATTERNS, price_file: o.price_file ?? null, stall_timeout_s: o.stall_timeout_s ?? null, remote_gate: o.remote_gate ?? null, allowlist: o.allowlist ?? [] }, errors: [] }; } export type ConfigLoad = { config: PregateConfig; errors: ConfigError[]; present: boolean; label: string; kind: 'base' | 'head' | 'user' | 'none'; /** sha256 of the user file's bytes (kind 'user' only). */ digest?: string } | { unavailable: true; base: string; detail: string }; @@ -282,6 +325,7 @@ export function explainConfig(): string { 'price_file: repo-relative path of the GSTACK_EVAL_CALLS v1 call-plan file `gstack-eval-plan price-gate` prices a paid gate from; never executed (docs/eval-calls.md)', 'stall_timeout_s: seconds without test output before the lanes check kills the runner\'s process group and records it incomplete (default 300; 0 disables; --stall-timeout overrides)', 'remote_gate: the repo\'s remote test gate command; under 8 cores the test tier becomes a requires-remote obligation naming it, and gstack-pregate --remote runs it (it must print PREGATE_REMOTE: tree= tests= failed=) (docs/pregate.md#remote-gate)', + `allowlist: [{ finding, paths?, pattern?, reason }]: benign findings the secrets check accepts and names as allowlisted. finding is a gstack-redact id (env.kv, google.api_key, pii.phone.e164, …) or a category (${ALLOWLISTABLE_CATEGORIES.join(', ')}); paths are globs; pattern is a regex matched at the finding's position in the added line; one of the two is required, reason always. Never ${NON_ALLOWLISTABLE.join(' or ')} or engine.* (docs/pregate.md#allowlist)`, 'no .gstack/pregate.json on the base: /projects//pregate.json is read instead (state root: GSTACK_STATE_ROOT, else GSTACK_HOME, else GSTACK_STATE_DIR, else a gstack plugin data dir, else ~/.gstack, per docs/state-root.md; run --explain prints the resolved path; outside the repo, so a PR cannot reach it; a path inside the checkout is ignored)', ].join('\n') + '\n'; } diff --git a/lib/pregate/context.ts b/lib/pregate/context.ts index ba9b4e532..02829b208 100644 --- a/lib/pregate/context.ts +++ b/lib/pregate/context.ts @@ -21,6 +21,7 @@ import { policyUnavailableLine, resolveBaseBranch } from '../ship-policy'; import { PREGATE_CONFIG_PATH, isConfigUnavailable, loadConfigFromBase, loadConfigFromHead, loadConfigFromUser, type ConfigError, type ConfigLoad, type PregateConfig } from './config'; import type { CheckResult, PregateContext, TouchedFile } from './types'; import { discoverLanes } from './workflows'; +import { createScratch, scratchRunId } from './run-scratch'; import type { RegistryError } from '../regen'; export interface BuildContextOptions { @@ -32,6 +33,9 @@ export interface BuildContextOptions { invocation?: InvocationOptions; /** `/projects//pregate.json`: read only when the base carries no pregate.json, and never from a relative path or one inside the repository. */ userConfigPath?: string; + /** `--run `: names this run's scratch directory under `scratchRoot` (default `/projects//pregate/runs`); unset, a unique id. */ + runId?: string; + scratchRoot?: string; } export const DEFAULT_STALL_TIMEOUT_S = 300; @@ -175,6 +179,23 @@ export function defaultUserConfigPath(root: string): string { return path.join(resolveStateRoot(), 'projects', slugFromEnvironment(undefined, root), 'pregate.json'); } +/** `/projects//pregate/runs`: where every run's scratch directory lives (lib/pregate/run-scratch.ts), from the caller's environment. */ +export function defaultScratchRoot(root: string, env: NodeJS.ProcessEnv = process.env): string { + const stateRoot = resolveStateRoot(env); + return path.join(stateRoot, 'projects', slugFromEnvironment(stateRoot, root), 'pregate', 'runs'); +} + +/** This run's scratch directory, created under the state root; an unwritable state root falls back to a private directory under os.tmpdir() (still one per run). */ +function openScratch(o: BuildContextOptions, env: NodeJS.ProcessEnv, explain: string[]): string { + const runId = scratchRunId(o.runId); + try { return createScratch(o.scratchRoot ?? defaultScratchRoot(o.repoRoot, env), runId); } + catch (e: any) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), `gstack-pregate-${runId}-`)); + explain.push(`scratch: state root not writable (${e?.message ?? e}); using ${dir}`); + return dir; + } +} + /** Why a user policy path must not be read, or null: a relative path (no HOME) or one a PR could commit is never trusted. */ export function userConfigRejection(repoRoot: string, file: string): string | null { if (!path.isAbsolute(file)) return 'not an absolute path'; @@ -209,6 +230,7 @@ export function buildContext(o: BuildContextOptions): ContextLoad { const fp = wtreeFingerprint({ cwd: o.repoRoot, env, invocation: o.invocation }); if ('error' in fp) return { refused: wtreeRefusal(fp) }; const wtree = fp.tree; + const scratch = openScratch(o, env, explain); const platform = o.platform ?? process.platform; const config = configLoad.config as PregateConfig; const lanes = discoverLanes(o.repoRoot, platform); @@ -217,7 +239,7 @@ export function buildContext(o: BuildContextOptions): ContextLoad { return { ctx: { repoRoot: o.repoRoot, base, baseRef, baseSha: baseSha.out, mergeBase, head, wtree, touched: touchedFiles(o.repoRoot, mergeBase), - config, configLabel: configLoad.label, configFile: configLoad.kind === 'user' && o.userConfigPath ? o.userConfigPath : PREGATE_CONFIG_PATH, ...(configLoad.kind === 'user' && configLoad.digest ? { configDigest: configLoad.digest } : {}), ...(absentUserPolicy ? { absentUserPolicy } : {}), registry: registryLoad.registry as Registry, registryLabel: registryLoad.source.label, + config, configLabel: configLoad.label, configKind: configLoad.kind, scratch, configFile: configLoad.kind === 'user' && o.userConfigPath ? o.userConfigPath : PREGATE_CONFIG_PATH, ...(configLoad.kind === 'user' && configLoad.digest ? { configDigest: configLoad.digest } : {}), ...(absentUserPolicy ? { absentUserPolicy } : {}), registry: registryLoad.registry as Registry, registryLabel: registryLoad.source.label, lanes, platform, allowRepoCommands: o.allowRepoCommands, timeoutMs: o.timeoutMs ?? 600_000, stallTimeoutMs: o.stallTimeoutMs ?? (config.stall_timeout_s ?? DEFAULT_STALL_TIMEOUT_S) * 1000, progress: o.progress ?? (line => process.stderr.write(`${line}\n`)), diff --git a/lib/pregate/environment.ts b/lib/pregate/environment.ts new file mode 100644 index 000000000..da70ab24d --- /dev/null +++ b/lib/pregate/environment.ts @@ -0,0 +1,92 @@ +/** + * pregate/environment — tells a failure the machine caused from one the diff + * caused (Lane D, GSTA-112; truther #617). A check that could not run its + * command (a runner or guard binary missing from PATH, a network error or a + * quota refusal before any test ran, a full disk) is `incomplete` under + * PREGATE_ENVIRONMENT with `cause: 'environment'`, a kind and the line that + * proves it, so a lane waives by cause and a reader never mistakes it for the + * branch's fault. The classifier reads a command's output and exit code only; + * the caller decides when to ask it: a local lane whose runner printed a test + * summary is the diff's failure whatever its log says, and only a run with no + * summary (nothing ran) is classified from its output, and only before any + * test file started. A missing binary is the machine's only when the command's + * own binary is off PATH: a script the diff controls that exits 127 or prints + * "command not found" is the diff's failure (`classifyCommandEnvironment`). + */ +import type { CheckResult, EnvironmentCause, EnvironmentKind } from './types'; + +/** A cause established at launch: a spawn failure, a binary off PATH, a runtime or tool refusal. */ +export const launchCause = (kind: EnvironmentKind, evidence: string): EnvironmentCause => ({ kind, evidence, proof: 'launch' }); + +const ANSI = /\x1b\[[0-9;?]*[A-Za-z]/g; +/** The shell's own "no such command" text (POSIX and cmd.exe); exit 127 says the same without text. */ +const NOT_FOUND = /command not found|is not recognized as an internal or external command/; +const RULES: Array<{ kind: EnvironmentKind; re: RegExp }> = [ + { kind: 'missing-binary', re: new RegExp(`${NOT_FOUND.source}|No such file or directory: ['"]?[\\w./-]+|spawn \\S+ ENOENT|\\bENOENT\\b.*\\bspawn\\b|not on PATH`) }, + { kind: 'quota', re: /\bquota\b|\b429 Too Many Requests|status(?: code)?[:= ]+429\b|HTTP(?:\/[\d.]+)? 429\b|\brate.?limit(?:ed)?\b|insufficient_quota/i }, + { kind: 'network', re: /\bE(?:NOTFOUND|CONNREFUSED|CONNRESET|TIMEDOUT|HOSTUNREACH|NETUNREACH|AI_AGAIN|PIPE)\b|getaddrinfo|Could not resolve host|Connection refused|Connection reset|Network is unreachable|fetch failed|TLS handshake|certificate verify failed|UNABLE_TO_GET_ISSUER_CERT|Temporary failure in name resolution/ }, + { kind: 'disk', re: /\bENOSPC\b|No space left on device|\bEDQUOT\b|Disk quota exceeded/ }, +]; + +/** The first line (trimmed, capped) of ANSI-free `text` that matches `re`, else its first line. */ +function evidenceLine(text: string, re: RegExp): string { + const lines = text.split(/\r?\n/).map(l => l.trim()).filter(Boolean); + return (lines.find(l => re.test(l)) ?? lines[0] ?? '').replace(/^(?:Error|error|ERROR):\s*/, '').slice(0, 200); +} + +/** + * The environment cause a command's output and exit code show, or null for an + * ordinary failure. Exit 127 (the shell found no such command) is + * missing-binary with or without text; a disk rule outranks quota (EDQUOT is a + * disk quota); otherwise the first rule in order whose text appears wins. + * Exit 127 is proved at launch; a text match is `proof: 'output'`, which the + * diff's own output could have written, and the fix says so. + */ +export function classifyEnvironment(output: string, exit: number | null): EnvironmentCause | null { + const text = output.replace(ANSI, ''); + if (exit === 127 || NOT_FOUND.test(text)) return { kind: 'missing-binary', evidence: evidenceLine(text, RULES[0]!.re) || 'exit 127: the shell found no such command', proof: exit === 127 ? 'launch' : 'output' }; + const disk = RULES.find(r => r.kind === 'disk')!; + if (disk.re.test(text)) return { kind: 'disk', evidence: evidenceLine(text, disk.re), proof: 'output' }; + for (const rule of RULES) if (rule.kind !== 'disk' && rule.re.test(text)) return { kind: rule.kind, evidence: evidenceLine(text, rule.re), proof: 'output' }; + return null; +} + +/** The binary a shell command runs: the first word after a leading `(cd … &&` and any `NAME=value` assignments. */ +export function commandBinary(command: string): string | null { + const words = command.replace(/^\(\s*cd\s+(?:'[^']*'|"[^"]*"|\S+)\s*&&\s*/, '').trim().split(/\s+/); + return words.find(w => w && !/^[A-Za-z_][A-Za-z0-9_]*=/.test(w)) ?? null; +} + +/** + * `classifyEnvironment` for a command that started and exited: missing-binary + * holds only when the command's own binary is off `env.PATH`. A runner or + * guard that resolved and then exited 127, or printed "command not found" from + * a script the diff controls, is the diff's failure. + */ +export function classifyCommandEnvironment(command: string, output: string, exit: number | null, env: NodeJS.ProcessEnv): EnvironmentCause | null { + const cause = classifyEnvironment(output, exit); + if (cause?.kind !== 'missing-binary') return cause; + const bin = commandBinary(command); + return bin && Bun.which(bin, { PATH: env.PATH ?? '' }) ? null : { ...cause, proof: 'launch' }; +} + +const FIX: Record = { + 'missing-binary': 'install the named tool on this machine (or put it on PATH), or run the lane remotely (--remote, or clear it with a run URL)', + network: 'check the network, proxy or registry this machine reaches, then rerun; or run the lane remotely', + quota: 'wait for the quota or raise it, free the backend\'s capacity, then rerun', + disk: 'free disk space (or move the state root) and rerun', + runtime: 'repair the runtime the line names (the install verdict or the pinned Bun), then rerun', + tool: 'repair or reinstall the named gstack tool, then rerun', +}; +/** The fix's last clause: a launch-proved cause clears the diff; an output-inferred one asks for confirmation elsewhere, since the diff's own output could have written the line. */ +export const PROOF_NOTE = { + launch: 'the diff is not at fault', + output: 'inferred from the output alone (no test was counted), which the diff could have written: confirm on another machine or with --remote before treating the diff as clear', +} as const; + +/** An `incomplete` result under PREGATE_ENVIRONMENT: cause, kind, evidence, an `environment:` line and the kind's fix. */ +export function environmentResult(base: CheckResult, cause: EnvironmentCause, detail: string): CheckResult { + return { ...base, status: 'incomplete', code: 'PREGATE_ENVIRONMENT', cause: 'environment', environment: cause, detail, fix: `${FIX[cause.kind]}; ${PROOF_NOTE[cause.proof]}`, lines: [...(base.lines ?? []), environmentLine(cause)] }; +} + +export const environmentLine = (cause: EnvironmentCause) => `environment: ${cause.kind} — ${cause.evidence}${cause.proof === 'output' ? ' (inferred from output)' : ''}`; diff --git a/lib/pregate/index.ts b/lib/pregate/index.ts index d6a52462f..d6cdaa44f 100644 --- a/lib/pregate/index.ts +++ b/lib/pregate/index.ts @@ -57,7 +57,7 @@ export async function runPregate(ctx: PregateContext, o: RunOptions): Promise c.remote ?? []).map(r => ({ ...r })); const file: PregateFile = { schema_version: 1, tree: ctx.wtree, head: ctx.head, base: `${ctx.baseRef}@${ctx.baseSha.slice(0, 12)}`, generated_at: new Date().toISOString(), - policy: ctx.configLabel, ...(ctx.configDigest ? { policy_digest: ctx.configDigest } : {}), ...(ctx.absentUserPolicy ? { policy_user_path: ctx.absentUserPolicy } : {}), registry: ctx.registryLabel, run: o.run, + policy: ctx.configLabel, ...(ctx.configDigest ? { policy_digest: ctx.configDigest } : {}), ...(ctx.absentUserPolicy ? { policy_user_path: ctx.absentUserPolicy } : {}), registry: ctx.registryLabel, run: o.run, scratch: ctx.scratch, stage1_ms: o.stage === 'tests' ? undefined : clocks.preflight, stage2_ms: o.stage === 'preflight' ? undefined : clocks.tests, checks: checks.map(c => { const { remote: _r, ...rest } = c; return rest; }), requires_remote: remote, @@ -85,6 +85,7 @@ export function renderTable(file: PregateFile, opts: { lines?: boolean } = { lin const clocks = [ file.stage1_ms !== undefined ? `stage 1 (mechanical preflight): ${file.stage1_ms} ms` : null, file.stage2_ms !== undefined ? `stage 2 (test execution): ${file.stage2_ms} ms` : null, + file.scratch ? `scratch: ${file.scratch}` : null, ].filter(Boolean); const remote = (file.requires_remote ?? []).map(r => { const run = r.command ? `run ${r.command.length > 200 ? `${r.command.slice(0, 200)}… (full command in pregate.json)` : r.command}, then ` : ''; @@ -117,7 +118,7 @@ export function readPregateFile(file: string): { file: PregateFile | null; error return { file: errors.length ? null : (parsed as PregateFile), errors }; } -export interface Verification { ok: boolean; code?: 'PREGATE_STALE' | 'PREGATE_REMOTE_UNCLEARED' | 'PREGATE_INCOMPLETE' | 'PREGATE_LANE_FAILED'; message: string; current: boolean; uncleared: PregateRemote[] } +export interface Verification { ok: boolean; code?: 'PREGATE_STALE' | 'PREGATE_REMOTE_UNCLEARED' | 'PREGATE_INCOMPLETE' | 'PREGATE_ENVIRONMENT' | 'PREGATE_LANE_FAILED'; message: string; current: boolean; uncleared: PregateRemote[] } /** * Publication check: the recorded tree must equal the tree now (results @@ -136,7 +137,14 @@ export function verifyPregate(repoRoot: string, file: PregateFile, treeNow: Wtre const failed = file.checks.filter(c => c.status === 'fail'); if (failed.length) return { ok: false, code: 'PREGATE_LANE_FAILED', message: `${failed.length} check(s) failed: ${failed.map(c => c.id).join(', ')}`, current, uncleared }; const incomplete = file.checks.filter(c => c.status === 'incomplete' || c.status === 'not_installed'); - if (incomplete.length) return { ok: false, code: 'PREGATE_INCOMPLETE', message: `${incomplete.length} check(s) incomplete: ${incomplete.map(c => c.id).join(', ')}`, current, uncleared }; + // An environment cause is named apart from the diff: a lane waives it by kind, never by test name. + const environment = incomplete.filter(c => c.cause === 'environment'); + if (environment.length === incomplete.length && environment.length) { + const inferred = environment.filter(c => c.environment?.proof === 'output'); + const caveat = inferred.length ? `; ${inferred.map(c => c.id).join(', ')} inferred from output alone (no test was counted): confirm on another machine or with --remote before treating the diff as clear` : ''; + return { ok: false, code: 'PREGATE_ENVIRONMENT', message: `${environment.length} check(s) blocked by the environment (${[...new Set(environment.map(c => c.environment?.kind ?? 'unknown'))].join(', ')}), not by the diff: ${environment.map(c => `${c.id}: ${c.environment?.evidence ?? c.detail}`).join('; ')}${caveat}`, current, uncleared }; + } + if (incomplete.length) return { ok: false, code: 'PREGATE_INCOMPLETE', message: `${incomplete.length} check(s) incomplete: ${incomplete.map(c => c.id).join(', ')}${environment.length ? ` (blocked by the environment: ${environment.map(c => `${c.id} ${c.environment?.kind}`).join(', ')})` : ''}`, current, uncleared }; if (uncleared.length) return { ok: false, code: 'PREGATE_REMOTE_UNCLEARED', message: `${uncleared.length} requires-remote obligation(s) without a receipt: ${uncleared.map(r => r.lane).join(', ')}`, current, uncleared }; return { ok: true, message: `pregate ${file.verdict} is current for tree ${file.tree.slice(0, 12)}`, current, uncleared }; } diff --git a/lib/pregate/init-draft.ts b/lib/pregate/init-draft.ts index 4be30a1f3..d09913525 100644 --- a/lib/pregate/init-draft.ts +++ b/lib/pregate/init-draft.ts @@ -27,7 +27,7 @@ const LANGUAGES: Array<[string, RegExp]> = [ ['go', /\.go$/], ['ruby', /\.rb$/], ['rust', /\.rs$/], ['shell', /\.(?:sh|bash|bats)$/], ]; /** Above this many tests a directory's tests are no longer a useful suggestion. */ -const MAX_SUGGESTED = 25; +export const MAX_SUGGESTED = 25; const MAX_TODO_LINES = 50; export interface InitDraft { @@ -40,7 +40,7 @@ export interface InitDraft { } /** The tests under the nearest ancestor directory of `file` (not the repo root) that has 1..MAX_SUGGESTED tests. */ -function nearestTests(tests: readonly string[], file: string): string[] { +export function nearestTests(tests: readonly string[], file: string): string[] { for (let dir = path.posix.dirname(file); dir !== '.' && dir !== ''; dir = path.posix.dirname(dir)) { const under = tests.filter(t => t.startsWith(`${dir}/`)); if (under.length > MAX_SUGGESTED) return []; diff --git a/lib/pregate/run-scratch.ts b/lib/pregate/run-scratch.ts new file mode 100644 index 000000000..ed94258fe --- /dev/null +++ b/lib/pregate/run-scratch.ts @@ -0,0 +1,69 @@ +/** + * pregate/run-scratch — one scratch directory per run under the state root + * (Lane D, GSTA-112): `/projects//pregate/runs//`. + * Every check keeps its working files there (the secrets scan's added-line + * copies, the lane and guard logs, the remote gate's file list) instead of + * os.tmpdir(): two pregates running at once never share a path (a taken + * `--run ` takes a numbered sibling), and the log a `fix:` line names + * survives a host that wipes /tmp between wakes (gbrain 16). The record names + * the directory. Runs older than the retention are pruned + * best-effort when a new run starts; the current run is never pruned. + */ +import { randomBytes } from 'node:crypto'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +export const SCRATCH_RETENTION_MS = 7 * 24 * 3600 * 1000; +const MAX_RUN_ID = 96; + +/** A filesystem-safe run id: `--run ` reduced to [A-Za-z0-9._-], else `--<4 hex>`. */ +export function scratchRunId(requested?: string, pid: number = process.pid): string { + const cleaned = (requested ?? '').replace(/[^A-Za-z0-9._-]+/g, '-').replace(/^[.-]+|[.-]+$/g, '').slice(0, MAX_RUN_ID); + if (cleaned) return cleaned; + const stamp = new Date().toISOString().replace(/[-:]/g, '').replace(/\.\d{3}Z$/, 'Z'); + return `${stamp}-${pid}-${randomBytes(2).toString('hex')}`; +} + +/** How many numbered siblings a taken run id may fall back to before the run refuses. */ +const MAX_RUN_ID_SIBLINGS = 9; + +/** + * `/`, created 0700 (and its parents), after pruning old + * siblings. The leaf is created without `recursive`, so a directory that + * already exists (a second `--run ` beside a live run, or a finished one + * under the same name) is never shared: the run takes `-2` … `-9` + * instead, and the record names the directory it got. + */ +export function createScratch(runsRoot: string, runId: string): string { + fs.mkdirSync(runsRoot, { recursive: true, mode: 0o700 }); + pruneScratch(runsRoot, runId); + for (let n = 1; n <= MAX_RUN_ID_SIBLINGS; n++) { + const dir = path.join(runsRoot, n === 1 ? runId : `${runId}-${n}`); + try { fs.mkdirSync(dir, { mode: 0o700 }); } catch (e: any) { if (e?.code === 'EEXIST') continue; throw e; } + try { fs.chmodSync(dir, 0o700); } catch { /* best effort on hosts without POSIX modes */ } + return dir; + } + throw new Error(`run directory ${path.join(runsRoot, runId)} and its ${MAX_RUN_ID_SIBLINGS - 1} numbered siblings all exist; choose another --run id`); +} + +/** Remove sibling run directories older than the retention, never `keep` (the current run); every failure is ignored. */ +export function pruneScratch(runsRoot: string, keep: string, now: number = Date.now()): void { + let names: string[]; + try { names = fs.readdirSync(runsRoot); } catch { return; } + for (const name of names) { + if (name === keep) continue; + const dir = path.join(runsRoot, name); + try { + const st = fs.lstatSync(dir); + if (!st.isDirectory() || now - st.mtimeMs < SCRATCH_RETENTION_MS) continue; + fs.rmSync(dir, { recursive: true, force: true }); + } catch { /* another run may hold it; leave it */ } + } +} + +/** A fresh subdirectory of the run's scratch for one check (`lanes`, `guards`, `secrets`, `remote`). */ +export function checkScratch(scratch: string, check: string): string { + const dir = path.join(scratch, check); + fs.mkdirSync(dir, { recursive: true, mode: 0o700 }); + return dir; +} diff --git a/lib/pregate/starter.ts b/lib/pregate/starter.ts new file mode 100644 index 000000000..8ed5a3954 --- /dev/null +++ b/lib/pregate/starter.ts @@ -0,0 +1,55 @@ +/** + * pregate/starter — the policy a repository with none would start from + * (Lane D, GSTA-112; FOUN #292, harness-evals #81/#82). When no + * `.gstack/pregate.json` exists on the base branch and no user-owned file is + * present, the lanes check does not fail a touched file with no lane + * (PREGATE_NO_LANE); it warns (PREGATE_NO_POLICY) and prints this starter: + * the detected runner, a `dependencies` entry for each unmapped touched file + * (the tests nearest it, else the tests sharing its stem; never `[]`, which + * would clear a file without a test), the test directories and the CI lanes + * seen. Nothing is written here; `gstack-pregate init` is the explicit write. + */ +import * as path from 'node:path'; +import { PREGATE_CONFIG_PATH, PREGATE_CONFIG_SCHEMA } from './config'; +import { isTestFile, listRepoFiles } from './import-graph'; +import { MAX_SUGGESTED, nearestTests } from './init-draft'; +import type { PregateContext } from './types'; + +const MAX_DIRS = 8; + + +/** `init`'s nearest-directory suggestion, else the tests whose basename starts with the file's stem. */ +export function nearestTestsFor(tests: readonly string[], file: string): string[] { + const nearest = nearestTests(tests, file); + if (nearest.length) return nearest; + const stem = path.posix.basename(file).replace(/\.[^.]+$/, ''); + if (!stem) return []; + const byStem = tests.filter(t => { const b = path.posix.basename(t); return b === stem || b.startsWith(`${stem}.`) || b.startsWith(`${stem}-`) || b.startsWith(`${stem}_`); }); + return byStem.length <= MAX_SUGGESTED ? byStem : []; +} + +/** The starter for `noLane` (touched files with no lane) under `runner` (detected by the lanes check; null when none). */ +export function starterPolicy(ctx: Pick, noLane: readonly string[], runner: string | null): string[] { + const tests = listRepoFiles(ctx.repoRoot).filter(isTestFile).sort(); + const dependencies: Record = {}; + const todo: string[] = []; + for (const file of noLane) { + const suggested = nearestTestsFor(tests, file); + if (suggested.length) dependencies[file] = suggested; else todo.push(file); + } + const json = JSON.stringify({ $schema: PREGATE_CONFIG_SCHEMA, runner, dependencies }, null, 2); + const dirCounts = new Map(); + for (const t of tests) { const d = path.posix.dirname(t); dirCounts.set(d, (dirCounts.get(d) ?? 0) + 1); } + const dirs = [...dirCounts].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])).slice(0, MAX_DIRS).map(([d]) => d); + const lanes = ctx.lanes.filter(l => l.runsTests && !l.manual).map(l => `${l.id} (${l.platform}${l.matrixExpression && l.testFiles.length === 0 && !ctx.config.selection[l.id] ? ', matrix computed; declare selection' : ''})`); + return [ + `PREGATE_STARTER: no ${PREGATE_CONFIG_PATH} on ${ctx.baseRef}${ctx.absentUserPolicy ? ` and no user-owned policy at ${ctx.absentUserPolicy}` : ''}; a starter drafted from this repository follows. Review it, write it with gstack-pregate init, edit each TODO, then commit it on the base branch (from then on an unmapped file is PREGATE_NO_LANE)`, + `PREGATE_STARTER: runner ${runner ?? 'null (none detected; set runner with {files})'}`, + `PREGATE_STARTER: test directories: ${dirs.join(', ') || '(no test files found)'}${dirCounts.size > MAX_DIRS ? ` (+${dirCounts.size - MAX_DIRS} more)` : ''}`, + `PREGATE_STARTER: CI lanes: ${lanes.join(', ') || '(no test job in .github/workflows)'}`, + `PREGATE_STARTER: begin ${PREGATE_CONFIG_PATH}`, + ...json.split('\n'), + 'PREGATE_STARTER: end', + ...todo.map(f => `TODO ${f}: no test lane and no nearby test; add a test, map it under dependencies, or declare [] if no test can cover it`), + ]; +} diff --git a/lib/pregate/supervise.ts b/lib/pregate/supervise.ts index 0d6aac3a0..17639f97f 100644 --- a/lib/pregate/supervise.ts +++ b/lib/pregate/supervise.ts @@ -54,6 +54,8 @@ export interface Supervised { start: number; end: number; lastFile: string | null; + /** Whether the progress adapter saw a test file start in the runner's own output (a single file is marked started by the supervisor, which proves nothing). */ + observedStart: boolean; error?: string; ms: number; } @@ -109,6 +111,7 @@ export function supervise(o: SuperviseOptions): Promise { let lastProgress = started; let current: { file: string; at: number; failed: boolean } | null = null; let lastFile: string | null = null; + let observedStart = false; const emit = (line: string) => { lastProgress = Date.now(); o.progress(line); }; const startFile = (file: string) => { current = { file, at: Date.now(), failed: false }; lastFile = file; emit(`PREGATE_TEST: ${counter.done + 1}/${counter.total} ${file} start`); }; const finishFile = (failed?: boolean) => { @@ -125,7 +128,7 @@ export function supervise(o: SuperviseOptions): Promise { partial = lines.pop() ?? ''; for (const line of lines) { const header = /^(?:::group::)?(\S.*):$/.exec(line); - if (header && files.has(header[1]!)) { finishFile(); startFile(header[1]!); continue; } + if (header && files.has(header[1]!)) { observedStart = true; finishFile(); startFile(header[1]!); continue; } if (/^\s*\d+\s+pass$/.test(line) || /^Ran \d+ tests? across/.test(line) || /^\s*\d+\s+tests? failed:?$/.test(line)) { finishFile(); continue; } if (current && /^\(fail\)\s/.test(line)) current.failed = true; } @@ -134,11 +137,11 @@ export function supervise(o: SuperviseOptions): Promise { return new Promise(resolve => { let fd: number; try { fd = fs.openSync(o.log, 'a'); } - catch (e: any) { resolve({ outcome: 'spawn-failed', exit: null, signal: null, start, end: start, lastFile: null, error: e.message, ms: 0 }); return; } + catch (e: any) { resolve({ outcome: 'spawn-failed', exit: null, signal: null, start, end: start, lastFile: null, observedStart: false, error: e.message, ms: 0 }); return; } const inv = bashCommandInvocation(`set -o pipefail; ${o.command}`); if (!inv.ok) { fs.closeSync(fd); - resolve({ outcome: 'spawn-failed', exit: null, signal: null, start, end: start, lastFile: null, error: inv.error, ms: Date.now() - started }); + resolve({ outcome: 'spawn-failed', exit: null, signal: null, start, end: start, lastFile: null, observedStart: false, error: inv.error, ms: Date.now() - started }); return; } let child: ReturnType; @@ -146,7 +149,7 @@ export function supervise(o: SuperviseOptions): Promise { child = spawn(inv.cmd, inv.argv, { cwd: o.cwd, env: o.env, stdio: ['ignore', fd, fd], detached: process.platform !== 'win32', windowsHide: true }); } catch (e: any) { fs.closeSync(fd); - resolve({ outcome: 'spawn-failed', exit: null, signal: null, start, end: start, lastFile: null, error: e.message, ms: Date.now() - started }); + resolve({ outcome: 'spawn-failed', exit: null, signal: null, start, end: start, lastFile: null, observedStart: false, error: e.message, ms: Date.now() - started }); return; } fs.closeSync(fd); @@ -170,7 +173,7 @@ export function supervise(o: SuperviseOptions): Promise { readNew(); const done: Supervised['outcome'] = error ? 'spawn-failed' : (outcome ?? 'exited'); if (current) finishFile(done !== 'exited' || exit !== 0 ? true : undefined); - resolve({ outcome: done, exit: done === 'exited' ? exit : null, signal, start, end: size, lastFile, error, ms: Date.now() - started }); + resolve({ outcome: done, exit: done === 'exited' ? exit : null, signal, start, end: size, lastFile, observedStart, error, ms: Date.now() - started }); }; const timer = setInterval(() => { const now = Date.now(); diff --git a/lib/pregate/types.ts b/lib/pregate/types.ts index b93fbe140..e357242f7 100644 --- a/lib/pregate/types.ts +++ b/lib/pregate/types.ts @@ -33,6 +33,10 @@ export interface PregateContext { touched: TouchedFile[]; config: PregateConfig; configLabel: string; + /** Where the policy came from: the base branch, the unreviewed head, the user-owned file, or nowhere (defaults). */ + configKind: 'base' | 'head' | 'user' | 'none'; + /** This run's scratch directory under the state root (lib/pregate/run-scratch.ts): logs and scan files, never os.tmpdir(). */ + scratch: string; /** The policy file to edit: the user-owned path when it is the active policy, else `.gstack/pregate.json`. */ configFile: string; /** sha256 of the user-owned policy's bytes when it is active; recorded as `inputs.policy` (a base copy is pinned by `inputs.base`). */ @@ -86,8 +90,15 @@ export interface CheckResult extends PregateCheck { /** Per-file or per-item lines for the table body. */ lines?: string[]; remote?: RemoteObligation[]; + /** With `cause: 'environment'` (declared on PregateCheck): the kind and the line that proves it. */ + environment?: EnvironmentCause; } +export type EnvironmentKind = 'missing-binary' | 'network' | 'quota' | 'disk' | 'runtime' | 'tool'; +/** How the cause was established: `launch` (the command could not start, its binary is off PATH, the runtime or a gstack tool refused), or `output` (a rule matched the command's text before any test was counted, which the diff's own output could have written). */ +export type EnvironmentProof = 'launch' | 'output'; +export interface EnvironmentCause { kind: EnvironmentKind; evidence: string; proof: EnvironmentProof } + export interface CheckDef { id: string; stage: Stage; diff --git a/lib/result-codes-release.ts b/lib/result-codes-release.ts index d44dbf65b..3691ab304 100644 --- a/lib/result-codes-release.ts +++ b/lib/result-codes-release.ts @@ -159,6 +159,17 @@ export const RELEASE_RESULT_CODES = { summary: 'an added or changed test line compares an elapsed wall-clock or CPU time against a bound, or sleeps and then asserts (a warning, not a block)', fix: 'assert on the outcome, an injected clock or a forced deadline instead; when the bound is the contract, add `gstack-timing-ok: ` in a comment on the line or the line above', }, + // Lane D (GSTA-112): the no-policy warning and the environment-cause verdict. + PREGATE_NO_POLICY: { + anchor: 'pregate-no-policy', + summary: 'the repository has no pregate policy yet (no .gstack/pregate.json on the base branch and no user-owned file) and a touched file maps to no test lane; a warn, with a starter policy printed', + fix: 'review the printed starter, write it with `gstack-pregate init`, edit the TODO entries, then commit .gstack/pregate.json on the base branch; from then on an unmapped file is PREGATE_NO_LANE', + }, + PREGATE_ENVIRONMENT: { + anchor: 'pregate-environment', + summary: 'a check could not run because of the machine, not the diff: a runner or guard binary missing from PATH, a network error or quota refusal before any test ran, a full disk, or an unverified runtime; incomplete, never pass', + fix: 'read the `environment:` line (kind and evidence), repair that on this machine or run the lane remotely (`--remote`, `gstack-pregate clear`), then rerun; a waiver cites the kind, never a test name', + }, // Field list 2, W2 (approval classes) and W3 (state-machine round-trip matrix). APPROVAL_CLASSES_INVALID: { anchor: 'approval-classes-invalid', diff --git a/package.json b/package.json index 91e735d54..4d97b4095 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.253", + "version": "1.91.254", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/scripts/lib/windows-curation.ts b/scripts/lib/windows-curation.ts index fe98f272f..0b8123809 100644 --- a/scripts/lib/windows-curation.ts +++ b/scripts/lib/windows-curation.ts @@ -66,6 +66,10 @@ export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }> { file: 'test/pregate-invalidation.test.ts', reason: 'expects the Linux-host lanes verdict (the windows lane as the requires-remote obligation); on Windows the Linux lane is the obligation instead; gstack-pregate itself launches through lib/script-invocation.ts' }, { file: 'test/pregate-selection.test.ts', reason: 'its gstack-pregate (bin) cases expect the Linux-host lanes verdict (a local lane run); on Windows the fixture\'s Linux lane is a requires-remote obligation' }, { file: 'test/pregate-stamp-only.test.ts', reason: 'its gstack-pregate (bin) cases expect the Linux-host lanes verdict for the mapped-file and test-reads-manifest fixtures, and its mode-change case needs a POSIX exec bit and symlink; the stamp rule it drives is POSIX-run like the other pregate fixtures' }, + { file: 'test/pregate-starter.test.ts', reason: 'expects the Linux-host lanes verdict (the fixture\'s Linux lane runs locally and the no-policy row is warn); on Windows that lane is a requires-remote obligation; gstack-pregate itself launches through lib/script-invocation.ts' }, + { file: 'test/pregate-allowlist.test.ts', reason: 'spawns bin/gstack-pregate through test/helpers/pregate-fixture.ts runBin and gstack-redact through the bin shebang; the allowlist rules it drives are POSIX-run like the other pregate fixtures' }, + { file: 'test/pregate-environment.test.ts', reason: 'drives exit-127 and spawn-failure classification through a POSIX shell (bash\'s command-not-found exit) and expects the Linux-host lanes verdict; on Windows the fixture\'s Linux lane is a requires-remote obligation' }, + { file: 'test/pregate-scratch.test.ts', reason: 'asserts the 0700 mode of the per-run scratch directory and the Linux-host lanes verdict (lane and guard logs written locally); Windows fakes mode bits and makes the Linux lane a requires-remote obligation' }, { file: 'test/pregate-bats-nested.test.ts', reason: 'expects the bats lane to run locally, which it does on a Linux host; on Windows the fixture\'s Linux lane is a requires-remote obligation' }, { file: 'test/pregate-docs-lane-review.test.ts', reason: 'spawns bin/gstack-pregate through test/helpers/pregate-fixture.ts runBin (shebang via CreateProcess); the docs-lane selection rules it drives are POSIX-run like the other pregate fixtures' }, { file: 'test/pregate-dart.test.ts', reason: 'its dart, flutter and fvm stand-ins are bash scripts on a colon-joined PATH, which Bun.which and CreateProcess cannot use on Windows, and its containment case needs a POSIX symlink; the Dart lane code it drives is POSIX-run like the other pregate fixtures' }, diff --git a/test/pregate-allowlist.test.ts b/test/pregate-allowlist.test.ts new file mode 100644 index 000000000..5c5ea19a9 --- /dev/null +++ b/test/pregate-allowlist.test.ts @@ -0,0 +1,128 @@ +/** + * Lane D (GSTA-112, E5): `allowlist` in the pregate policy lets a repo accept + * benign findings the `secrets` check would otherwise refuse (dummy per-trial + * tokens in receipts, a run id read as a phone number, a known public key in + * a rebuilt bundle: harness-evals #81/#83, GSTA-102, GSTA-106), by finding id + * or category, path and a pattern anchored at the finding. The real value of a + * credential-named environment variable (`env.value`, #3449) can never be + * allowlisted: validation refuses the entry, and the matcher ignores one. + */ +import { afterEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { EXIT } from '../lib/headless-artifacts'; +import { applyAllowlist, type AllowlistFinding } from '../lib/pregate/allowlist'; +import { NON_ALLOWLISTABLE, validateConfig } from '../lib/pregate/config'; +import { CHECK_IDS } from '../lib/pregate/index'; +import { BASE_PREGATE, cleanup, commit, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/pregate-fixture'; + +const repos: FixtureRepo[] = []; +const fixture = (pregate: Record | null) => { const r = makeFixtureRepo({ pregate }); repos.push(r); return r; }; +afterEach(() => { while (repos.length) cleanup(repos.pop()!); }); +const secrets = (repo: FixtureRepo, env: Record = {}) => runBin(repo, 'gstack-pregate', ['--base', 'main', '--only', 'secrets'], env); +const row = (repo: FixtureRepo) => JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')).checks.find((c: any) => c.id === 'secrets'); +const codes = (c: unknown) => validateConfig(c, CHECK_IDS).map(e => `${e.code}:${e.path}`); + +const TRIAL = '{\n "trial": 1,\n "key": "sk-trial-9f8e7d6c-5b4a-3f2e-1d0c-7b6a5f4e3d2c"\n}\n'; +const TRIAL_ENTRY = { finding: 'env.kv', paths: ['evals/**'], pattern: 'sk-trial-[0-9a-f-]+', reason: 'dummy per-trial meter ids in eval receipts' }; + +describe('allowlist: policy validation', () => { + // Value: protects=the key's shape: finding id or non-secret category, paths and/or pattern, a reason; the env.value rule and + // the secret wildcard are refused at validation; fails_when=an entry without a reason or scope is accepted, or env.value can be + // named; why_new=the key is new; seam=none; no_claim=regex runtime cost + test('accepts an id or category entry with a reason and a scope; refuses env.value, the secret wildcard, an empty scope and a bad regex', () => { + expect(codes({ allowlist: [TRIAL_ENTRY] })).toEqual([]); + expect(codes({ allowlist: [{ finding: 'pii', pattern: '\\d{11}', reason: 'GitHub run ids' }] })).toEqual([]); + expect(codes({ allowlist: [{ finding: 'env.kv', paths: ['evals/**'] }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].reason']); + expect(codes({ allowlist: [{ finding: 'env.kv', reason: 'x' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0]']); + expect(codes({ allowlist: [{ finding: 'env.value', pattern: '.*', reason: 'never' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].finding']); + expect(codes({ allowlist: [{ finding: 'secret', pattern: '.*', reason: 'never' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].finding']); + expect(codes({ allowlist: [{ finding: 'env.kv', pattern: '(', reason: 'x' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].pattern']); + expect(codes({ allowlist: [{ finding: 'env.kv', paths: ['../x'], reason: 'x' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].paths[0]']); + expect(codes({ allowlist: { finding: 'env.kv' } })).toEqual(['PREGATE_CONFIG_INVALID:allowlist']); + expect(codes({ allowlist: Array.from({ length: 65 }, () => TRIAL_ENTRY) })).toContain('PREGATE_CONFIG_INVALID:allowlist'); + expect(codes({ allowlist: [{ ...TRIAL_ENTRY, extra: 1 }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].extra']); + expect(codes({ allowlist: [{ finding: 'env.kv', paths: [], reason: 'x' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].paths']); + expect(codes({ allowlist: [{ finding: 'env.kv', pattern: 'a'.repeat(257), reason: 'x' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].pattern']); + // A pattern entry never covers a finding the scanner gave no position; a paths-only entry does. + const positionless: AllowlistFinding = { id: 'env.kv', category: 'secret', severity: 'MEDIUM', file: 'evals/x.json' }; + const inert = applyAllowlist([positionless], [TRIAL_ENTRY]); + expect(inert.kept).toHaveLength(1); + expect(inert.untested.map(u => [u.index, u.finding.id])).toEqual([[0, 'env.kv']]); + expect(applyAllowlist([positionless], [{ finding: 'env.kv', paths: ['evals/**'], reason: 'r' }]).allowlisted).toHaveLength(1); + expect(NON_ALLOWLISTABLE).toContain('env.value'); + // The scanner's own refusal (an oversize input fails closed) is never a benign finding: refused by validation, kept by the matcher. + expect(codes({ allowlist: [{ finding: 'engine.input_too_large', paths: ['**'], reason: 'x' }] })).toEqual(['PREGATE_CONFIG_INVALID:allowlist[0].finding']); + const refusal: AllowlistFinding = { id: 'engine.input_too_large', category: 'engine', severity: 'HIGH', file: 'big.json' }; + expect(applyAllowlist([refusal], [{ finding: 'engine.input_too_large', paths: ['**'], reason: 'x' }]).kept).toHaveLength(1); + }); +}); + +describe('allowlist: the secrets check', () => { + // Value: protects=a benign dummy token in the allowlisted path with the allowlisted shape passes and is named as allowlisted; + // the same token outside the path, or a shape the pattern does not cover, still fails PREGATE_SECRETS; fails_when=the + // allowlist is ignored (the pre-change behaviour: 1,800 dummy tokens failed harness-evals #81) or it silences a finding its + // scope does not cover; why_new=the key is new; seam=none; no_claim=none + test('a dummy trial token is allowlisted by path and pattern, and named; outside the scope it still fails', () => { + const repo = fixture({ ...BASE_PREGATE, allowlist: [TRIAL_ENTRY] }); + commit(repo, { 'evals/receipts/t1.json': TRIAL }); + const ok = secrets(repo); + expect(ok.status).toBe(EXIT.ok); + expect(ok.err).not.toContain('PREGATE_SECRETS'); + const r = row(repo); + expect(r.status).toBe('pass'); + expect(r.detail).toContain('1 allowlisted (env.kv)'); + expect(r.lines.join('\n')).toContain('allowlisted env.kv (evals/receipts/t1.json, added line 3) by allowlist[0] env.kv: dummy per-trial meter ids in eval receipts'); + expect(r.inputs.allowlist).toMatch(/^[0-9a-f]{64}$/); + commit(repo, { 'notes/t2.json': TRIAL }); + const outside = secrets(repo); + expect(outside.status).toBe(EXIT.fail); + expect(outside.err).toContain('(PREGATE_SECRETS)'); + expect(row(repo).lines.join('\n')).toContain('notes/t2.json'); + const shape = fixture({ ...BASE_PREGATE, allowlist: [{ ...TRIAL_ENTRY, pattern: 'sk-trial-[0-9]+' }] }); + commit(shape, { 'evals/receipts/t1.json': TRIAL }); + expect(secrets(shape).status).toBe(EXIT.fail); + }); + + // Value: protects=a category entry covers an advisory pii hit (a run id read as a phone number) and the WARN count drops; + // fails_when=category matching is id-only; why_new=new; seam=none; no_claim=none + test('a category entry silences the advisory pii finding for a run id', () => { + const repo = fixture({ ...BASE_PREGATE, allowlist: [{ finding: 'pii', pattern: '\\d{11}', reason: 'GitHub Actions run ids in reports' }] }); + commit(repo, { 'docs/runs.md': 'run 40302077688 passed\n' }); + expect(secrets(repo).status).toBe(EXIT.ok); + const r = row(repo); + expect([r.status, r.code]).toEqual(['pass', undefined]); + expect(r.detail).toContain('1 allowlisted (pii.phone.e164)'); + const control = fixture(BASE_PREGATE); + commit(control, { 'docs/runs.md': 'run 40302077688 passed\n' }); + secrets(control); + expect([row(control).status, row(control).code]).toEqual(['warn', 'PREGATE_SECRETS']); + expect(row(control).detail).toContain('pii.phone.e164'); + }); + + // Value: protects=env.value is never allowlisted: a config naming it is refused before any check runs, and the matcher keeps an + // env.value finding even when an entry's scope would match it; fails_when=either path lets a configured credential's value + // through; why_new=the rule is new; seam=none; no_claim=none + test('env.value survives every allowlist: the config is refused, and the matcher never drops it', () => { + const value = 'q8Zt3vRm7Lk2Xn9Pw4Cs6Hd1Yf5Gb0Je'; + const refused = fixture({ ...BASE_PREGATE, allowlist: [{ finding: 'env.value', pattern: '.*', reason: 'no' }] }); + commit(refused, { 'lib/leak.txt': `token=${value}\n` }); + const r = secrets(refused, { FIXTURE_API_TOKEN: value }); + expect(r.status).toBe(EXIT.refused); + expect(r.err).toContain('(PREGATE_CONFIG_INVALID)'); + expect(r.err).toContain('allowlist[0].finding'); + const findings: AllowlistFinding[] = [ + { id: 'env.value', category: 'secret', severity: 'HIGH', file: 'lib/leak.txt', line: 1, col: 7, lineText: `token=${value}` }, + { id: 'env.kv', category: 'secret', severity: 'MEDIUM', file: 'lib/leak.txt', line: 1, col: 7, lineText: `token=${value}` }, + ]; + const wide = [{ finding: 'env.value', pattern: '.*', reason: 'no' }, { finding: 'env.kv', pattern: '.*', reason: 'wide' }]; + const applied = applyAllowlist(findings, wide); + expect(applied.kept.map(f => f.id)).toEqual(['env.value']); + expect(applied.allowlisted.map(a => a.finding.id)).toEqual(['env.kv']); + const live = fixture({ ...BASE_PREGATE, allowlist: [{ finding: 'env.kv', pattern: '.*', reason: 'wide' }] }); + commit(live, { 'lib/leak.txt': `token=${value}\n` }); + const leak = secrets(live, { FIXTURE_API_TOKEN: value }); + expect(leak.status).toBe(EXIT.fail); + expect(row(live).lines.join('\n')).toContain('env.value'); + }); +}); diff --git a/test/pregate-dart.test.ts b/test/pregate-dart.test.ts index bae591bec..0162b2375 100644 --- a/test/pregate-dart.test.ts +++ b/test/pregate-dart.test.ts @@ -533,8 +533,9 @@ describe('gstack-pregate on a Dart repo with a committed policy', () => { for (const tool of ['fvm', 'dart', 'flutter']) fs.rmSync(path.join(bin, tool)); const missing = run(repo, env); expect(missing.status).not.toBe(EXIT.ok); - expect(missing.out).toMatch(/^tests {6}lanes {5}incomplete .*not on PATH: dart/m); - expect(missing.out).toContain('fix: install dart (or fvm, which this repo pins) and rerun'); + expect(missing.out).toMatch(/^tests {6}lanes {5}incomplete .*not on PATH: dart.*\(PREGATE_ENVIRONMENT\)/m); + expect(missing.out).toContain('fix: install dart (or fvm, which this repo pins) on this machine and rerun, or run the lane remotely; the diff is not at fault'); + expect(missing.out).toContain('environment: missing-binary — not on PATH: dart'); }, 30_000); // Value: protects=without an fvm pin the missing-SDK fix names only the SDK, never fvm; fails_when=fvmPinnedFor always reports a pin @@ -546,8 +547,8 @@ describe('gstack-pregate on a Dart repo with a committed policy', () => { commit(repo, { 'packages/core/lib/core.dart': 'int one() => 2;\n' }); const r = run(repo, { PATH: bin }); expect(r.status).not.toBe(EXIT.ok); - expect(r.out).toMatch(/^tests {6}lanes {5}incomplete .*not on PATH: dart/m); - expect(r.out).toContain('fix: install dart and rerun'); + expect(r.out).toMatch(/^tests {6}lanes {5}incomplete .*not on PATH: dart.*\(PREGATE_ENVIRONMENT\)/m); + expect(r.out).toContain('fix: install dart on this machine and rerun, or run the lane remotely; the diff is not at fault'); expect(r.out).not.toContain('(or fvm'); }, 30_000); @@ -612,7 +613,8 @@ describe('gstack-pregate on a Dart repo with a committed policy', () => { const r = run(repo, env); expect(r.status).not.toBe(EXIT.ok); expect(r.out).toMatch(/^tests {6}lanes {5}incomplete .*2 tests ran in 2 file\(s\), exit 0; no test runner detected for 1 of 3 selected file\(s\): packages\/ui\/test\/ui_test\.dart; not on PATH: flutter/m); - expect(r.out).toContain('fix: install flutter and rerun'); + expect(r.out).toContain('fix: install flutter on this machine and rerun, or run the lane remotely; the diff is not at fault'); + expect(r.out).toContain('(PREGATE_ENVIRONMENT)'); expect(fs.readFileSync(/log: (\S+)/.exec(r.out)![1]!, 'utf8')).toContain('ran in core: test/core_test.dart'); }, 30_000); diff --git a/test/pregate-docs-lane-review.test.ts b/test/pregate-docs-lane-review.test.ts index 1a4fd7731..bda824e3d 100644 --- a/test/pregate-docs-lane-review.test.ts +++ b/test/pregate-docs-lane-review.test.ts @@ -27,7 +27,7 @@ describe('docs/no-tests empty lane: reviewer edges (#3215)', () => { expect(r.out).toMatch(/docs\/no-tests: 1 documentation file\(s\)/); }); - test('without a pin file, a template source and an unread test-tree Markdown file still fail closed; a renamed or deleted Markdown file is docs', () => { + test('without a pin file, a template source and an unread test-tree Markdown file are still unmapped (the no-policy warning, never docs); a renamed or deleted Markdown file is docs', () => { const repo = fixture(null); repo.git('checkout', '-q', 'main'); commit(repo, { 'notes/old.md': '# old\n', 'notes/gone.md': '# gone\n' }, 'notes'); @@ -42,10 +42,11 @@ describe('docs/no-tests empty lane: reviewer edges (#3215)', () => { expect(docs.out).toContain('notes/gone.md: docs/no-tests (empty lane)'); commit(repo, { 'skills/review/SKILL.md.tmpl': '# {{PREAMBLE}}\n', 'test/notes.md': '# notes no test reads\n' }); const r = lanes(repo); - expect(r.status).toBe(EXIT.fail); - expect(r.err).toContain('(PREGATE_NO_LANE)'); - expect(r.out).toContain('FAIL lanes — skills/review/SKILL.md.tmpl has no test lane'); - expect(r.out).toContain('FAIL lanes — test/notes.md has no test lane'); + expect(r.err).not.toContain('PREGATE_NO_LANE'); + expect(r.out).toContain('(PREGATE_NO_POLICY)'); + expect(r.out).toContain('no policy: skills/review/SKILL.md.tmpl test/notes.md have no test lane'); + expect(r.out).not.toContain('skills/review/SKILL.md.tmpl: docs/no-tests'); + expect(r.out).not.toContain('test/notes.md: docs/no-tests'); }); test('empty_lanes is read from the base branch only: the same key added on the branch does not clear a file', () => { diff --git a/test/pregate-environment.test.ts b/test/pregate-environment.test.ts new file mode 100644 index 000000000..78ae04ebf --- /dev/null +++ b/test/pregate-environment.test.ts @@ -0,0 +1,186 @@ +/** + * Lane D (GSTA-112, E6): a pregate failure the environment caused (a runner + * binary missing from PATH, a network error before any test ran, a quota + * refusal from the remote gate, a guard that cannot start) is classified and + * reported apart from a failure the diff caused, under its own result code + * `PREGATE_ENVIRONMENT` with a kind and the evidence, so a lane can waive by + * cause instead of by test name (truther #617). A failing test whose log also + * mentions a network error is still a diff failure: the runner's own summary + * decides, never the log alone. + */ +import { afterEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { EXIT } from '../lib/headless-artifacts'; +import { classifyCommandEnvironment, classifyEnvironment, commandBinary } from '../lib/pregate/environment'; +import { verifyPregate } from '../lib/pregate/index'; +import { BASE_PREGATE, cleanup, commit, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/pregate-fixture'; + +const repos: FixtureRepo[] = []; +const fixture = (pregate: Record, baseFiles?: Record) => { const r = makeFixtureRepo({ pregate, baseFiles }); repos.push(r); return r; }; +afterEach(() => { while (repos.length) cleanup(repos.pop()!); }); +const pregate = (repo: FixtureRepo, args: string[]) => runBin(repo, 'gstack-pregate', ['--base', 'main', '--allow-repo-commands', '--only', 'lanes,guards', ...args]); +const readOut = (repo: FixtureRepo) => JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')); +const check = (repo: FixtureRepo, id: string) => readOut(repo).checks.find((c: any) => c.id === id); +const UTIL_EDIT = { 'lib/util.js': 'exports.add = (a, b) => b + a;\n' }; + +describe('classifyEnvironment', () => { + // Value: protects=the classifier's vocabulary and its silence on ordinary failures; fails_when=a plain assertion failure or a + // non-zero exit with a test summary reads as environment, or exit 127 / ENOTFOUND / quota text does not; why_new=new module; + // seam=none; no_claim=completeness of the vocabulary + test('names missing-binary, network, quota and disk, and stays null for an ordinary failure', () => { + expect(classifyEnvironment('bash: line 1: vitest: command not found', 127)?.kind).toBe('missing-binary'); + expect(classifyEnvironment('', 127)).toEqual({ kind: 'missing-binary', evidence: 'exit 127: the shell found no such command', proof: 'launch' }); + expect(classifyEnvironment('write failed: Disk quota exceeded (EDQUOT)', 1)).toEqual({ kind: 'disk', evidence: 'write failed: Disk quota exceeded (EDQUOT)', proof: 'output' }); + expect(classifyEnvironment("'nope' is not recognized as an internal or external command", 1)?.kind).toBe('missing-binary'); + expect(classifyEnvironment('Error: getaddrinfo ENOTFOUND registry.example.invalid', 1)).toEqual({ kind: 'network', evidence: 'getaddrinfo ENOTFOUND registry.example.invalid', proof: 'output' }); + expect(classifyEnvironment('curl: (7) Failed to connect to host port 443: Connection refused', 7)?.kind).toBe('network'); + expect(classifyEnvironment('ubicloud: vcpu quota exceeded (528/512)', 1)?.kind).toBe('quota'); + expect(classifyEnvironment('HTTP 429 Too Many Requests', 1)?.kind).toBe('quota'); + expect(classifyEnvironment('ENOSPC: no space left on device, write', 1)?.kind).toBe('disk'); + expect(classifyEnvironment('AssertionError: expected 3 to be 4', 1)).toBeNull(); + expect(classifyEnvironment('✖ fetches the thing\n ECONNREFUSED 127.0.0.1:3000\nℹ tests 1\nℹ fail 1', 1)?.kind).toBe('network'); + }); + + // Value: protects=exit 127 (or "command not found" text) is the machine's only when the command's own binary is off PATH: a script + // the diff controls that exits 127 is the diff's; fails_when=a present runner exiting 127 reads as missing-binary; why_new=the veto + // is new (adversarial review); seam=none; no_claim=none + test('a missing binary is the machine\'s only when the command\'s own binary is off PATH', () => { + expect(commandBinary("(cd 'pkg' && UBI_OWNER=x node run.js a b)")).toBe('node'); + expect(commandBinary('FOO=1 BAR=2 bun test x')).toBe('bun'); + expect(commandBinary('')).toBeNull(); + const env = { PATH: process.env.PATH }; + expect(classifyCommandEnvironment('node scripts/x.js', '', 127, env)).toBeNull(); + expect(classifyCommandEnvironment('node scripts/x.js', 'bash: vitest: command not found', 127, env)).toBeNull(); + expect(classifyCommandEnvironment('gstack-fixture-runner-that-does-not-exist x', '', 127, env)).toMatchObject({ kind: 'missing-binary', proof: 'launch' }); + expect(classifyCommandEnvironment('node scripts/x.js', 'Error: getaddrinfo ENOTFOUND registry.example.invalid', 1, env)).toMatchObject({ kind: 'network', proof: 'output' }); + }); +}); + +describe('environment failures are reported apart from diff failures', () => { + // Value: protects=a runner whose binary is missing is PREGATE_ENVIRONMENT missing-binary (incomplete, cause environment) rather + // than a lane failure that reads as the diff's fault; fails_when=the lane is PREGATE_LANE_FAILED with no cause (the behaviour + // before this change); why_new=the code and cause are new; seam=none; no_claim=none + test('a runner binary missing from PATH is PREGATE_ENVIRONMENT missing-binary, incomplete, cause environment', () => { + const repo = fixture({ ...BASE_PREGATE, runner: 'gstack-fixture-runner-that-does-not-exist {files}' }); + commit(repo, UTIL_EDIT); + const r = pregate(repo, []); + expect(r.status).toBe(EXIT.fail); + expect(r.err).toContain('(PREGATE_ENVIRONMENT)'); + expect(r.err).not.toContain('PREGATE_LANE_FAILED'); + const c = check(repo, 'lanes'); + expect([c.status, c.code, c.cause, c.environment.kind, c.environment.proof]).toEqual(['incomplete', 'PREGATE_ENVIRONMENT', 'environment', 'missing-binary', 'launch']); + expect(c.environment.evidence).toContain('gstack-fixture-runner-that-does-not-exist'); + expect(c.fix).toContain('the diff is not at fault'); + expect(r.out).toContain('environment: missing-binary'); + const verify = runBin(repo, 'gstack-pregate', ['verify']); + expect(verify.status).toBe(EXIT.fail); + expect(verify.err).toContain('(PREGATE_ENVIRONMENT)'); + expect(verify.err).toContain('blocked by the environment (missing-binary), not by the diff'); + }); + + // Value: protects=a runner that dies on a network error before reporting any test is environment/network, while a test that + // failed and whose log also names a connection error stays a diff failure (its summary counted a failure); fails_when=either + // direction is misread; why_new=the summary-first rule is new; seam=none; no_claim=none + test('a network error with no test summary is environment; a counted test failure beside a network line is still the diff', () => { + const net = fixture({ ...BASE_PREGATE, runner: 'node scripts/net.js {files}' }, { 'scripts/net.js': "console.error('Error: getaddrinfo ENOTFOUND registry.example.invalid');\nprocess.exit(1);\n" }); + commit(net, UTIL_EDIT); + pregate(net, []); + const c = check(net, 'lanes'); + expect([c.status, c.code, c.cause, c.environment.kind]).toEqual(['incomplete', 'PREGATE_ENVIRONMENT', 'environment', 'network']); + const diff = fixture({ ...BASE_PREGATE, runner: 'node scripts/fail.js {files}' }, { 'scripts/fail.js': "console.log('✖ talks to the server\\n Error: connect ECONNREFUSED 127.0.0.1:3000\\nℹ tests 1\\nℹ pass 0\\nℹ fail 1');\nprocess.exit(1);\n" }); + commit(diff, UTIL_EDIT); + const r = pregate(diff, []); + expect(r.err).toContain('(PREGATE_LANE_FAILED)'); + const d = check(diff, 'lanes'); + expect([d.status, d.code, d.cause]).toEqual(['fail', 'PREGATE_LANE_FAILED', 'diff']); + expect(d.environment).toBeUndefined(); + }); + + // Value: protects=a test in the diff cannot clear itself: a runner whose binary is present but exits 127 is PREGATE_LANE_FAILED + // (diff), and a bun test that prints a network line and exits the process before the summary is at most an output-inferred + // environment cause whose fix and verify message ask for confirmation elsewhere, never "the diff is not at fault"; + // fails_when=exit 127 from a present runner reads as missing-binary, or the spoof's fix clears the diff; why_new=the binary veto + // and the launch/output proof are new (adversarial review); seam=none; no_claim=a spoof with no summary cannot be told from a + // real network failure by the output alone, which is exactly what the proof records + test('a present runner that exits 127 is the diff\'s; a test that exits with a network line is only an output-inferred cause', () => { + const exit127 = fixture({ ...BASE_PREGATE, runner: 'node scripts/x.js {files}' }, { 'scripts/x.js': 'process.exit(127);\n' }); + commit(exit127, UTIL_EDIT); + pregate(exit127, []); + const c = check(exit127, 'lanes'); + expect([c.status, c.code, c.cause, c.environment]).toEqual(['fail', 'PREGATE_LANE_FAILED', 'diff', undefined]); + if (!Bun.which('bun')) return; + const spoof = fixture({ ...BASE_PREGATE, runner: 'bun test {files}' }, { 'test/util.test.js': "const { test } = require('bun:test');\nrequire('../lib/util.js');\ntest('spoof', () => { console.log('Received: fetch failed'); process.exit(1); });\n" }); + commit(spoof, UTIL_EDIT); + pregate(spoof, []); + const s = check(spoof, 'lanes'); + expect([s.status, s.code, s.environment.kind, s.environment.proof]).toEqual(['incomplete', 'PREGATE_ENVIRONMENT', 'network', 'output']); + expect(s.fix).not.toContain('the diff is not at fault'); + expect(s.fix).toContain('confirm on another machine or with --remote'); + expect(s.lines.join('\n')).toContain('environment: network — Received: fetch failed (inferred from output)'); + const verify = runBin(spoof, 'gstack-pregate', ['verify']); + expect(verify.err).toContain('lanes inferred from output alone'); + }); + + // Value: protects=a remote gate is classified from its preamble only: quota text streamed from the tests past the first lines + // is not the environment; fails_when=a test name with "quota" waives the gate; why_new=new (adversarial review); seam=none; no_claim=none + test('a remote gate whose test stream names a quota past the preamble is PREGATE_INCOMPLETE, not environment', () => { + const repo = fixture({ ...BASE_PREGATE, remote_gate: 'node scripts/gate.js' }, { 'scripts/gate.js': "for (let i = 0; i < 60; i++) console.log('ok test ' + i);\nconsole.log('✗ billing > enforces quota per user');\nprocess.exit(1);\n" }); + commit(repo, UTIL_EDIT); + pregate(repo, ['--remote', '--spend-cap', '1']); + const c = check(repo, 'lanes'); + expect([c.status, c.code, c.environment]).toEqual(['incomplete', 'PREGATE_INCOMPLETE', undefined]); + }); + + // Value: protects=a remote gate refused by a quota (no PREGATE_REMOTE line) is environment/quota and the obligation stays + // uncleared; fails_when=it is a generic PREGATE_INCOMPLETE; why_new=new; seam=none; no_claim=none + test('a remote gate that hit a quota is PREGATE_ENVIRONMENT quota and leaves the obligation uncleared', () => { + const repo = fixture({ ...BASE_PREGATE, remote_gate: 'node scripts/gate.js' }, { 'scripts/gate.js': "console.error('ubicloud: vcpu quota exceeded (528/512)');\nprocess.exit(1);\n" }); + commit(repo, UTIL_EDIT); + const r = pregate(repo, ['--remote', '--spend-cap', '1']); + expect(r.status).toBe(EXIT.fail); + expect(r.err).toContain('(PREGATE_ENVIRONMENT)'); + const c = check(repo, 'lanes'); + expect([c.status, c.cause, c.environment.kind]).toEqual(['incomplete', 'environment', 'quota']); + expect(readOut(repo).requires_remote.find((o: any) => o.lane === 'pregate.json/remote_gate').receipt).toBeNull(); + }); + + // Value: protects=guards classify the same way: a guard whose command is missing is environment, a guard that fails on its + // own terms is the diff; fails_when=a missing guard binary is PREGATE_GUARD_FAILED; why_new=new; seam=none; no_claim=none + test('a guard whose binary is missing is environment; a guard that fails is the diff', () => { + // The red guard's output names a rate limit and a refused connection: a guard that ran and said no is the diff's whatever it printed. + const repo = fixture({ ...BASE_PREGATE, guards: [{ id: 'absent', when: ['lib/**'], command: 'gstack-fixture-guard-that-does-not-exist --check' }, { id: 'red', when: ['lib/**'], command: 'node -e "console.log(\'FAIL api > retries after a rate limit (ECONNREFUSED fixture)\'); process.exit(2)"', fix: 'fix it' }] }); + commit(repo, UTIL_EDIT); + const r = pregate(repo, ['--only', 'guards']); + expect(r.status).toBe(EXIT.fail); + const g = check(repo, 'guards'); + expect([g.status, g.code, g.cause]).toEqual(['fail', 'PREGATE_GUARD_FAILED', 'diff']); + expect(g.lines.join('\n')).toContain('environment: missing-binary'); + expect(g.lines.join('\n')).not.toContain('red environment:'); + // A guard whose own binary is present but whose script exits 127 (a package script naming a missing tool) is the diff's. + const present = fixture({ ...BASE_PREGATE, guards: [{ id: 'script', when: ['lib/**'], command: 'node -e "process.exit(127)"', fix: 'fix it' }] }); + commit(present, UTIL_EDIT); + pregate(present, ['--only', 'guards']); + const p = check(present, 'guards'); + expect([p.status, p.code, p.cause, p.environment]).toEqual(['fail', 'PREGATE_GUARD_FAILED', 'diff', undefined]); + const only = fixture({ ...BASE_PREGATE, guards: [{ id: 'absent', when: ['lib/**'], command: 'gstack-fixture-guard-that-does-not-exist --check' }] }); + commit(only, UTIL_EDIT); + pregate(only, ['--only', 'guards']); + const o = check(only, 'guards'); + expect([o.status, o.code, o.cause, o.environment.kind]).toEqual(['incomplete', 'PREGATE_ENVIRONMENT', 'environment', 'missing-binary']); + }); +}); + +describe('verify: a mixed incomplete record stays PREGATE_INCOMPLETE and names the environment-blocked check', () => { + test('one environment-caused check beside a plain incomplete one is not promoted to PREGATE_ENVIRONMENT', () => { + const file: any = { schema_version: 1, tree: 'T', checks: [ + { id: 'lanes', status: 'incomplete', cause: 'environment', environment: { kind: 'network', evidence: 'ENOTFOUND x' } }, + { id: 'guards', status: 'incomplete', code: 'REPO_COMMANDS_NOT_ALLOWED' }, + ] }; + const v = verifyPregate('/nonexistent', file, { tree: 'T' } as any); + expect(v.code).toBe('PREGATE_INCOMPLETE'); + expect(v.message).toContain('(blocked by the environment: lanes network)'); + const all: any = { ...file, checks: [file.checks[0]] }; + expect(verifyPregate('/nonexistent', all, { tree: 'T' } as any).code).toBe('PREGATE_ENVIRONMENT'); + }); +}); diff --git a/test/pregate-invalidation.test.ts b/test/pregate-invalidation.test.ts index 731d18941..61a46b0fc 100644 --- a/test/pregate-invalidation.test.ts +++ b/test/pregate-invalidation.test.ts @@ -45,7 +45,7 @@ describe('pregate invalidation', () => { // Re-running on the moved tree makes it current again. expect(run(repo, []).status).toBe(EXIT.ok); expect(receipt(repo).status).toBe(EXIT.ok); - }); + }, 30_000); test('an uncleared requires-remote obligation is PREGATE_REMOTE_UNCLEARED until `clear` records a receipt, which the ship receipt then carries', () => { const repo = fixture(); @@ -69,7 +69,7 @@ describe('pregate invalidation', () => { const ok = receipt(repo); expect(ok.status).toBe(EXIT.ok); expect(JSON.parse(ok.out)).toMatchObject({ pregate: expect.stringMatching(/^requires-remote cleared @/), requires_remote: ['windows.yml/windows-shard: https://github.com/o/r/actions/runs/42'] }); - }); + }, 30_000); test('a failed check and an incomplete check both refuse publication; a missing or invalid pregate.json is not evidence', () => { const repo = fixture(); @@ -89,5 +89,5 @@ describe('pregate invalidation', () => { const bad = receipt(none); expect(bad.status).toBe(EXIT.fail); expect(bad.err).toContain('(ARTIFACT_SCHEMA)'); - }); + }, 30_000); }); diff --git a/test/pregate-non-js.test.ts b/test/pregate-non-js.test.ts index bf162617d..439efb431 100644 --- a/test/pregate-non-js.test.ts +++ b/test/pregate-non-js.test.ts @@ -72,28 +72,28 @@ describe('Django files loaded by name have a lane (#3266)', () => { const plain = fixture(DJANGO_APP); commit(plain, { 'orders/admin.py': 'from django.contrib import admin\nadmin.site.site_header = "Orders"\n' }); const r = lanes(plain); - expect(r.status).toBe(EXIT.fail); - expect(r.out).toContain('FAIL lanes — orders/admin.py has no test lane'); + expect(r.out).toContain('(PREGATE_NO_POLICY)'); + expect(r.out).toContain('no policy: orders/admin.py have no test lane'); expect(r.out).not.toContain('(framework)'); const quoted = fixture({ ...DJANGO_APP, 'docs_example.py': 'EXAMPLE = """\nINSTALLED_APPS = []\n"""\n# ROOT_URLCONF = "config.urls"\n' }); commit(quoted, { 'orders/admin.py': 'from django.contrib import admin\nadmin.site.site_header = "Orders"\n' }); const q = lanes(quoted); - expect(q.out).toContain('FAIL lanes — orders/admin.py has no test lane'); + expect(q.out).toContain('no policy: orders/admin.py have no test lane'); expect(q.out).not.toContain('(framework)'); const noDjango = fixture({ 'app/settings.py': 'INSTALLED_APPS = ["app.plugins.Core"]\n', 'app/__init__.py': '', 'app/plugins.py': 'class Core: pass\n', 'app/admin.py': 'X = 1\n', 'app/models.py': 'Y = 2\n', 'tests/test_app.py': 'def test_x():\n assert True\n' }); commit(noDjango, { 'app/admin.py': 'X = 2\n' }); const n = lanes(noDjango); - expect(n.out).toContain('FAIL lanes — app/admin.py has no test lane'); + expect(n.out).toContain('no policy: app/admin.py have no test lane'); expect(n.out).not.toContain('(framework)'); const django = fixture({ ...DJANGO_APP, 'config/settings.py': SETTINGS }); commit(django, { 'scripts/oneoff.py': 'print("not loaded by Django or any test")\n' }); const bad = lanes(django); - expect(bad.status).toBe(EXIT.fail); - expect(bad.err).toContain('(PREGATE_NO_LANE)'); - expect(bad.out).toContain('FAIL lanes — scripts/oneoff.py has no test lane'); + expect(bad.out).toContain('(PREGATE_NO_POLICY)'); + expect(bad.out).toContain('no policy: scripts/oneoff.py have no test lane'); + expect(bad.out).not.toMatch(/^tests {6}lanes {5}pass /m); }); test('DJANGO_SETTINGS_MODULE in pytest.ini names the settings module when it assigns neither INSTALLED_APPS nor ROOT_URLCONF directly', () => { @@ -148,8 +148,10 @@ describe('assets, licence text and tool config take an empty lane (#3267)', () = '.gstack/icon.svg': '\n', }); const r = lanes(repo); - expect(r.status).toBe(EXIT.fail); - for (const f of ['.nvmrc', '.env.test', '.github/workflows/lint.yml', 'notes/sketch.txt', 'lib/orphan.js', '.rspec', '.husky/pre-commit', 'styles/colors.css', 'src/__image_snapshots__/home.png', '.gstack/icon.svg']) expect(r.out).toContain(`FAIL lanes — ${f} has no test lane`); + const unmapped = ['.nvmrc', '.env.test', '.github/workflows/lint.yml', 'notes/sketch.txt', 'lib/orphan.js', '.rspec', '.husky/pre-commit', 'styles/colors.css', 'src/__image_snapshots__/home.png', '.gstack/icon.svg']; + const noPolicy = /^\s*lanes: no policy: (.*) have no test lane/m.exec(r.out)![1]!.split(' '); + for (const f of unmapped) expect(noPolicy).toContain(f); + expect(r.out).toContain('(PREGATE_NO_POLICY)'); expect(r.out).not.toContain('no-tests (assets and tool config)'); const named = fixture({ 'assets/logo.svg': '\n', 'test/logo.test.js': "const test = require('node:test');\nconst fs = require('node:fs');\ntest('logo', () => { fs.readFileSync('assets/logo.svg', 'utf8'); });\n" }); diff --git a/test/pregate-rust.test.ts b/test/pregate-rust.test.ts index cec5aeab6..802ade053 100644 --- a/test/pregate-rust.test.ts +++ b/test/pregate-rust.test.ts @@ -365,7 +365,8 @@ describe('gstack-pregate on a Rust workspace', () => { const repo = rustRepo(); commit(repo, { 'engine/crates/lonely/src/lib.rs': 'pub fn alone() { }\n' }); const r = run(repo, cargoShim(repo)); - expect(r.status).not.toBe(EXIT.ok); - expect(r.out).toContain('FAIL lanes — engine/crates/lonely/src/lib.rs has no test lane'); + expect(r.out).toContain('(PREGATE_NO_POLICY)'); + expect(r.out).toContain('no policy: engine/crates/lonely/src/lib.rs have no test lane'); + expect(r.out).not.toMatch(/^tests {6}lanes {5}pass /m); }, 60_000); }); diff --git a/test/pregate-scratch.test.ts b/test/pregate-scratch.test.ts new file mode 100644 index 000000000..cbd4dc2f8 --- /dev/null +++ b/test/pregate-scratch.test.ts @@ -0,0 +1,100 @@ +/** + * Lane D (GSTA-112, E6b): every run gets its own scratch directory under the + * state root (`/projects//pregate/runs//`) for the + * secrets scan files, lane and guard logs and the remote gate's files, instead + * of os.tmpdir(): two pregates in parallel never share a path, the logs a + * `fix:` line names survive a Capy wake that wipes /tmp (gbrain 16), and the + * record names the directory. Old runs are pruned best-effort. + */ +import { afterEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { EXIT } from '../lib/headless-artifacts'; +import { pruneScratch, scratchRunId } from '../lib/pregate/run-scratch'; +import { BASE_PREGATE, cleanup, commit, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/pregate-fixture'; + +const repos: FixtureRepo[] = []; +afterEach(() => { while (repos.length) cleanup(repos.pop()!); }); +const SLUG = 'scratchslug'; +const pregate = (repo: FixtureRepo, args: string[]) => runBin(repo, 'gstack-pregate', ['--base', 'main', '--allow-repo-commands', ...args], { GSTACK_PROJECT_SLUG: SLUG }); +const readOut = (repo: FixtureRepo) => JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')); +const runsDir = (repo: FixtureRepo) => path.join(repo.home, 'projects', SLUG, 'pregate', 'runs'); + +describe('per-run scratch under the state root', () => { + // Value: protects=lane and guard logs and the record's scratch path sit under the state root's per-run directory, named by + // --run, and two runs never share one; fails_when=logs land under os.tmpdir() (the behaviour before this change) or two runs + // collide; why_new=the scratch layout is new; seam=none; no_claim=none + test('logs and the record name the run directory; two --run ids get two directories', () => { + const repo = makeFixtureRepo({ pregate: { ...BASE_PREGATE, guards: [{ id: 'echo', when: ['lib/**'], command: 'node -e "console.log(1)"' }] } }); + repos.push(repo); + commit(repo, { 'lib/util.js': 'exports.add = (a, b) => b + a;\n' }); + const a = pregate(repo, ['--run', 'run-a', '--only', 'lanes,guards']); + expect(a.status).toBe(EXIT.ok); + const dirA = path.join(runsDir(repo), 'run-a'); + expect(readOut(repo).scratch).toBe(dirA); + expect(a.out).toContain(`scratch: ${dirA}`); + const logLines = a.out.split('\n').filter(l => /\blog \S+|\(log \S+\)/.test(l)); + expect(logLines.length).toBeGreaterThan(0); + for (const l of logLines) { expect(l).toContain(dirA); expect(l).not.toMatch(/gstack-pregate-(?:lanes|guards|secrets|remote)-/); } + expect(fs.existsSync(path.join(dirA, 'lanes', 'lanes.log'))).toBe(true); + expect(fs.existsSync(path.join(dirA, 'guards', 'echo.log'))).toBe(true); + expect((fs.statSync(dirA).mode & 0o777)).toBe(0o700); + pregate(repo, ['--run', 'run-b', '--only', 'lanes']); + expect(readOut(repo).scratch).toBe(path.join(runsDir(repo), 'run-b')); + expect(fs.existsSync(path.join(dirA, 'lanes', 'lanes.log'))).toBe(true); + // A taken --run id is never shared (a second run would truncate the first's logs): it takes a numbered sibling and the record names it. + const again = pregate(repo, ['--run', 'run-a', '--only', 'lanes']); + expect(readOut(repo).scratch).toBe(`${dirA}-2`); + expect(again.out).toContain(`scratch: ${dirA}-2`); + expect(fs.existsSync(path.join(dirA, 'guards', 'echo.log'))).toBe(true); + }); + + // Value: protects=without --run the id is unique per process and the secrets scan files are gone after the scan; fails_when=two + // runs in one second share an id, or added-line copies linger under the state root; why_new=new; seam=none; no_claim=none + test('an unnamed run gets a unique id, and the secrets scan leaves no added-line copies behind', () => { + const repo = makeFixtureRepo(); + repos.push(repo); + commit(repo, { 'lib/util.js': 'exports.add = (a, b) => b + a;\n' }); + pregate(repo, ['--only', 'secrets']); + pregate(repo, ['--only', 'secrets']); + const runs = fs.readdirSync(runsDir(repo)); + expect(runs.length).toBe(2); + for (const run of runs) expect(fs.existsSync(path.join(runsDir(repo), run, 'secrets'))).toBe(false); + expect(scratchRunId('weird id/with:chars')).toBe('weird-id-with-chars'); + expect(scratchRunId(undefined, 123)).toMatch(/^\d{8}T\d{6}Z-123-[0-9a-f]{4}$/); + expect(scratchRunId('...', 123)).toMatch(/^\d{8}T\d{6}Z-123-[0-9a-f]{4}$/); + expect(scratchRunId('a'.repeat(200))).toHaveLength(96); + }); + + // Value: protects=pruning removes only run directories older than the retention and never the current one; fails_when=a fresh + // run or the current run is deleted, or an old one is kept; why_new=new; seam=none; no_claim=none + test('prune drops runs older than 7 days and keeps fresh ones and the current run', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-pregate-prune-')); + try { + const old = path.join(root, 'old'); const fresh = path.join(root, 'fresh'); const current = path.join(root, 'current'); + for (const d of [old, fresh, current]) fs.mkdirSync(d); + const eightDays = (Date.now() - 8 * 24 * 3600 * 1000) / 1000; + fs.utimesSync(old, eightDays, eightDays); + fs.utimesSync(current, eightDays, eightDays); + pruneScratch(root, 'current'); + expect(fs.readdirSync(root).sort()).toEqual(['current', 'fresh']); + } finally { fs.rmSync(root, { recursive: true, force: true }); } + }); +}); + +describe('an unwritable state root', () => { + // Value: protects=a state root whose projects dir cannot be created still runs pregate, in a private per-run directory under + // os.tmpdir(), and says so under --explain; fails_when=the mkdir error escapes buildContext or the note is dropped; why_new=the + // fallback is new; seam=none; no_claim=none + test('falls back to a per-run tmpdir and says so', () => { + const repo = makeFixtureRepo(); + repos.push(repo); + fs.writeFileSync(path.join(repo.home, 'projects'), 'not a directory'); + commit(repo, { 'lib/util.js': 'exports.add = (a, b) => b + a;\n' }); + const r = pregate(repo, ['--only', 'secrets', '--explain']); + expect(r.status).toBe(EXIT.ok); + expect(readOut(repo).scratch.startsWith(os.tmpdir())).toBe(true); + expect(r.out + r.err).toContain('scratch: state root not writable'); + }); +}); diff --git a/test/pregate-selection.test.ts b/test/pregate-selection.test.ts index a5d5ec86a..d6eb6ef0c 100644 --- a/test/pregate-selection.test.ts +++ b/test/pregate-selection.test.ts @@ -11,9 +11,10 @@ */ import { afterEach, describe, expect, test } from 'bun:test'; import * as fs from 'node:fs'; +import * as os from 'node:os'; import * as path from 'node:path'; import { EXIT } from '../lib/headless-artifacts'; -import { changedLiterals, constantsIn } from '../lib/pregate/checks/literals'; +import { changedLiterals, constantsIn, consumersOf } from '../lib/pregate/checks/literals'; import { parseSummary, planRuns } from '../lib/pregate/checks/lanes'; import { DEFAULT_CONFIG, validateConfig } from '../lib/pregate/config'; import { isDocsEmptyLane } from '../lib/pregate/docs-lane'; @@ -200,6 +201,28 @@ describe('literals (pure)', () => { { name: 'MAX', old: '3', new: '4', file: 'f.ts' }, { name: 'timeout', old: '5000', new: '9000', file: 'f.ts' }, ]); }); + test('consumersOf: a module stem is a path fragment, never a bare word, and a one-character old value needs the constant name (GSTA-114 flood: install.sh 2 -> 3 flagged 575 consumers)', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-literals-')); + const nl = String.fromCharCode(10); + const files: Record = { + 'test/wrapper.test.ts': ["import { x } from '../capy/skill/gstack/scripts/install.sh';", 'expect(GSTACK_CAPY_WRAPPER).toBe(2);', ''].join(nl), + 'test/by-path.test.ts': ["const out = run('scripts/install.sh');", 'expect(out.wrapper).toBe(2);', ''].join(nl), + 'test/prose.test.ts': ['// run install first, then check 2 things', 'expect(count).toBe(2);', ''].join(nl), + 'test/other.test.ts': ['expect(install.steps.length).toBe(2);', ''].join(nl), + 'test/retry.test.ts': ["expect(require('../lib/retry.js').TIMEOUT_MS).toBe(5000);", ''].join(nl), + 'test/retry-path.test.ts': ["const mod = require('../lib/retry.js');", 'expect(mod.wait()).toBe(5000);', ''].join(nl), + 'test/retry-bare.test.ts': ["const mod = load('retry.js');", 'expect(mod.wait()).toBe(5000);', ''].join(nl), + 'test/reretry.test.ts': ["const mod = load('preretry.js');", 'expect(mod.wait()).toBe(5000);', ''].join(nl), + 'test/retry-words.test.ts': ['// the retry budget was 5000 once', 'expect(elapsed).toBeLessThan(5000);', ''].join(nl), + }; + for (const [rel, text] of Object.entries(files)) { fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); fs.writeFileSync(path.join(root, rel), text); } + const all = Object.keys(files); + // A one-character value: only the file that names the constant counts; the path fragment alone is not enough for "2". + expect(consumersOf(root, all, { name: 'GSTACK_CAPY_WRAPPER', old: '2', new: '3', file: 'capy/skill/gstack/scripts/install.sh' })).toEqual(['test/wrapper.test.ts:2']); + // A longer value: the constant's name, the module's full file name, or its stem as a path fragment; the bare word "retry" in prose and "preretry.js" do not count. + expect(consumersOf(root, all, { name: 'TIMEOUT_MS', old: '5000', new: '9000', file: 'lib/retry.js' })).toEqual(['test/retry.test.ts:1', 'test/retry-path.test.ts:2', 'test/retry-bare.test.ts:2']); + fs.rmSync(root, { recursive: true, force: true }); + }); test('runner summaries: bun, node --test, vitest, jest, pytest, zero-run', () => { expect(parseSummary(' 12 pass\n 1 fail\nRan 13 tests across 2 files.')).toEqual({ ran: 13, failed: 1 }); expect(parseSummary('# tests 4\n# pass 4\n# fail 0')).toEqual({ ran: 4, failed: 0 }); @@ -372,6 +395,7 @@ describe('gstack-pregate (bin)', () => { const ok = pregate(docs, ['--only', 'lanes', '--explain']); expect(ok.status).toBe(EXIT.ok); expect(ok.err).not.toContain('PREGATE_NO_LANE'); + expect(ok.out).not.toContain('PREGATE_NO_POLICY'); expect(ok.out).toMatch(/docs\/no-tests: 3 documentation file\(s\)/); expect(ok.out).toContain('.work/round-3-plan.md: docs/no-tests (empty lane)'); expect(ok.out).toContain('notes/redproof.md: docs/no-tests (empty lane)'); @@ -383,10 +407,12 @@ describe('gstack-pregate (bin)', () => { '.work/round-3-plan.md': '# plan\n', 'lib/orphan.js': 'exports.lonely = () => 2;\n', }); + // With no policy anywhere the unmapped source file is the PREGATE_NO_POLICY warning with a starter (Lane D), still never a pass, and the Markdown stays docs. const bad = pregate(mixed, ['--only', 'lanes']); - expect(bad.status).toBe(EXIT.fail); - expect(bad.err).toContain('(PREGATE_NO_LANE)'); - expect(bad.out).toContain('lanes: FAIL lanes — lib/orphan.js has no test lane'); + expect(bad.status).toBe(EXIT.ok); + expect(bad.err).not.toContain('PREGATE_NO_LANE'); + expect(bad.out).toContain('(PREGATE_NO_POLICY)'); + expect(bad.out).toContain('lanes: no policy: lib/orphan.js have no test lane'); expect(bad.out).toContain('docs/no-tests (empty lane): .work/round-3-plan.md'); expect(bad.out).not.toContain('.work/round-3-plan.md has no test lane'); diff --git a/test/pregate-starter.test.ts b/test/pregate-starter.test.ts new file mode 100644 index 000000000..fd911b866 --- /dev/null +++ b/test/pregate-starter.test.ts @@ -0,0 +1,139 @@ +/** + * Lane D (GSTA-112, E4): a repository with no pregate policy anywhere (no + * `.gstack/pregate.json` on the base, no user-owned file) gets a `warn` row + * with a printed starter policy instead of a hard PREGATE_NO_LANE, so the + * first /ship pass in a new repo is not blocked before anyone has written the + * lane policy (FOUN #292, harness-evals #81/#82). Nothing is written by `run`; + * `gstack-pregate init` is the explicit write. A repository that has a policy + * keeps the hard failure: missing coverage is never inferred silently. + */ +import { afterEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { EXIT } from '../lib/headless-artifacts'; +import { validateConfig } from '../lib/pregate/config'; +import { CHECK_IDS } from '../lib/pregate/index'; +import { nearestTestsFor } from '../lib/pregate/starter'; +import { BASE_PREGATE, cleanup, commit, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/pregate-fixture'; + +const repos: FixtureRepo[] = []; +const fixture = (opts?: Parameters[0]) => { const r = makeFixtureRepo(opts); repos.push(r); return r; }; +afterEach(() => { while (repos.length) cleanup(repos.pop()!); }); +/** The fixture's Windows workflow replaced by a lint job: with no policy its planner-computed matrix would otherwise make every selected test a requires-remote obligation. */ +const NO_PLATFORM_LANE = { '.github/workflows/windows.yml': 'name: Lint\non: [push]\njobs:\n lint:\n runs-on: ubuntu-24.04\n steps:\n - run: echo lint\n' }; +const lanes = (repo: FixtureRepo, extra: string[] = [], env: Record = {}) => runBin(repo, 'gstack-pregate', ['--base', 'main', '--allow-repo-commands', '--only', 'lanes', ...extra], env); +const readOut = (repo: FixtureRepo) => JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')); +const starterJson = (out: string): Record => { + const block = /PREGATE_STARTER: begin \.gstack\/pregate\.json\n([\s\S]*?)\n\s*lanes: PREGATE_STARTER: end/.exec(out); + if (!block) throw new Error(`no starter block in:\n${out}`); + return JSON.parse(block[1]!.split('\n').map(l => l.replace(/^\s*lanes: /, '')).join('\n')); +}; +/** An unmapped source file, a mapped one (its test runs) and a file whose only nearby test shares its stem without importing it. */ +const NO_POLICY_DIFF = { + 'lib/orphan.js': 'exports.lonely = () => 2;\n', + 'lib/util.js': 'exports.add = (a, b) => b + a;\n', + 'scripts/report.js': 'module.exports = () => "report";\n', + 'test/report.test.js': "const test = require('node:test');\ntest('report placeholder', () => {});\n", +}; + +describe('no policy anywhere: warn plus a printed starter (never PREGATE_NO_LANE)', () => { + // Value: protects=the first /ship in a repo with no .gstack/pregate.json is not blocked by PREGATE_NO_LANE: the unmapped files + // are a warn row (PREGATE_NO_POLICY) with a starter policy the owner can commit, and the selected tests still run; + // fails_when=the lanes check exits 1 with PREGATE_NO_LANE (the behaviour before this change) or the starter is missing or + // invalid; why_new=the no-policy branch is new; seam=none; no_claim=the starter's mappings are right for every repo + test('unmapped files are a warn row, the mapped test still runs, and the starter validates and maps by nearest test or stem', () => { + const repo = fixture({ pregate: null, baseFiles: NO_PLATFORM_LANE }); + commit(repo, NO_POLICY_DIFF); + const r = lanes(repo); + expect(r.err).not.toContain('PREGATE_NO_LANE'); + expect(r.status).toBe(EXIT.ok); + expect(r.out).toContain('(PREGATE_NO_POLICY)'); + expect(r.err).not.toContain('gstack-pregate: lanes'); + expect(r.out).toContain('no policy: lib/orphan.js scripts/report.js have no test lane'); + const row = readOut(repo).checks.find((c: any) => c.id === 'lanes'); + expect([row.status, row.code]).toEqual(['warn', 'PREGATE_NO_POLICY']); + expect(row.detail).toMatch(/\d+ tests ran in \d+ file\(s\)/); + expect(row.fix).toContain('gstack-pregate init'); + const starter = starterJson(r.out); + expect(validateConfig(starter, CHECK_IDS)).toEqual([]); + expect(starter.runner).toBe('node --test {files}'); + expect((starter.dependencies as Record)['scripts/report.js']).toEqual(['test/report.test.js']); + expect(starter.dependencies).not.toHaveProperty('lib/orphan.js'); + expect(r.out).toContain('TODO lib/orphan.js: no test lane and no nearby test'); + expect(r.out).toContain('PREGATE_STARTER: test directories: test'); + expect(r.out).toContain('PREGATE_STARTER: CI lanes: free-tests.yml/unit (linux)'); + expect(fs.existsSync(path.join(repo.work, '.gstack/pregate.json'))).toBe(false); + // With the planner-computed Windows lane and no selection manifest, the selected tests are a requires-remote obligation: that status stands, with the no-policy code and the starter beside it. + const platform = fixture({ pregate: null }); + commit(platform, NO_POLICY_DIFF); + const p = lanes(platform); + expect(p.status).toBe(EXIT.ok); + expect(p.err).not.toContain('PREGATE_NO_LANE'); + const prow = readOut(platform).checks.find((c: any) => c.id === 'lanes'); + expect([prow.status, prow.code]).toEqual(['requires-remote', 'PREGATE_NO_POLICY']); + expect(p.out).toContain('PREGATE_STARTER: CI lanes: free-tests.yml/unit (linux), windows.yml/windows-shard (windows, matrix computed; declare selection)'); + }); + + // Value: protects=warn never hides a failing selected test: a no-policy run whose lane fails is still fail; fails_when=the + // starter branch returns warn for a red lane; why_new=new branch; seam=none; no_claim=none + test('a failing selected test under no policy is still PREGATE_LANE_FAILED', () => { + const repo = fixture({ pregate: null, baseFiles: NO_PLATFORM_LANE }); + commit(repo, { ...NO_POLICY_DIFF, 'lib/util.js': 'exports.add = (a, b) => a - b;\n' }); + const r = lanes(repo); + expect(r.status).toBe(EXIT.fail); + expect(r.err).toContain('(PREGATE_LANE_FAILED)'); + expect(r.out).toContain('no policy: lib/orphan.js scripts/report.js have no test lane'); + }); + + // Value: protects=a repository with a policy keeps the hard PREGATE_NO_LANE (base copy, and the user-owned file); fails_when=the + // warn branch fires whenever an entry is unmapped; why_new=the branch condition is new; seam=none; no_claim=none + test('a base policy or a user-owned policy keeps PREGATE_NO_LANE as a hard failure', () => { + const withBase = fixture(); + commit(withBase, { 'lib/orphan.js': 'exports.lonely = () => 2;\n' }); + const b = lanes(withBase); + expect(b.status).toBe(EXIT.fail); + expect(b.err).toContain('(PREGATE_NO_LANE)'); + expect(b.out).not.toContain('PREGATE_STARTER'); + const withUser = fixture({ pregate: null }); + commit(withUser, { 'lib/orphan.js': 'exports.lonely = () => 2;\n' }); + const userPolicy = path.join(withUser.home, 'projects', 'starterslug', 'pregate.json'); + fs.mkdirSync(path.dirname(userPolicy), { recursive: true }); + fs.writeFileSync(userPolicy, JSON.stringify({ runner: 'node --test {files}' })); + const u = lanes(withUser, [], { GSTACK_PROJECT_SLUG: 'starterslug' }); + expect(u.status).toBe(EXIT.fail); + expect(u.err).toContain('(PREGATE_NO_LANE)'); + expect(u.out).not.toContain('PREGATE_STARTER'); + // --policy-from head with a head policy (no base copy) is strict too. + const withHead = fixture({ pregate: null }); + commit(withHead, { 'lib/orphan.js': 'exports.lonely = () => 2;\n', '.gstack/pregate.json': JSON.stringify({ runner: 'node --test {files}' }) + '\n' }); + const h = lanes(withHead, ['--policy-from', 'head']); + expect(h.status).toBe(EXIT.fail); + expect(h.err).toContain('(PREGATE_NO_LANE)'); + expect(h.out).not.toContain('PREGATE_STARTER'); + }); + + // Value: protects=`run` never writes the policy; `init` is the explicit write and the starter and the init draft agree on the + // runner; fails_when=run creates .gstack/pregate.json or init writes something validate rejects; why_new=the run/init split is + // the contract this lane documents; seam=none; no_claim=none + test('init is the explicit write: it produces a policy validate accepts, and run left none behind', () => { + const repo = fixture({ pregate: null }); + commit(repo, NO_POLICY_DIFF); + lanes(repo); + expect(fs.existsSync(path.join(repo.work, '.gstack/pregate.json'))).toBe(false); + const init = runBin(repo, 'gstack-pregate', ['init']); + expect(init.status).toBe(EXIT.ok); + expect(init.out).toContain('PREGATE_CONFIG_WRITTEN'); + expect(runBin(repo, 'gstack-pregate', ['validate']).out).toContain('PREGATE_CONFIG_VALID'); + expect(BASE_PREGATE.runner).toBe('node --test {files}'); + }); +}); + +describe('nearestTestsFor: the nearest directory with tests, else the tests sharing the stem', () => { + const tests = ['lib/render/a.test.ts', 'lib/render/b.test.ts', 'test/report.test.js', 'test/report-golden.test.js', 'test/reporter.test.js']; + test('a file under a directory with tests takes those tests; one with none takes its stem; a stem with no match is empty', () => { + expect(nearestTestsFor(tests, 'lib/render/draw.ts')).toEqual(['lib/render/a.test.ts', 'lib/render/b.test.ts']); + expect(nearestTestsFor(tests, 'src/report.js')).toEqual(['test/report.test.js', 'test/report-golden.test.js']); + expect(nearestTestsFor(tests, 'src/orphan.js')).toEqual([]); + expect(nearestTestsFor(tests, 'README')).toEqual([]); + }); +});