diff --git a/CHANGELOG.md b/CHANGELOG.md index b371c3a62..6bd1587d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,62 @@ # Changelog +## [1.94.19.0] - 2026-11-03 + +### Fixed + +- **A finished /ship no longer blocks `gstack-upgrade` for two hours (GST-1205).** Each `gstack-skill-start` writes a workflow lease that `gstack-upgrade` checks before moving a shared checkout. Nothing released it when the workflow ended, and on Capy every session's lease names the machine's one `capy-server`. So after a /ship finished, its own lease kept refusing the upgrade with UPGRADE_BUSY until it was 120 minutes old, unless you set `GSTACK_UPGRADE_FORCE=1`. Now `gstack-skill-start` records its `SESSION_ID` as an open run on the lease, and `gstack-skill-end` closes it with the new `gstack-lease release`. Once every run on a lease is closed, the lease is idle and blocks nothing (`gstack-lease list` shows `idle`). A run another session still has open keeps blocking. So does a holder that records no run (an older gstack, or `gstack-lease touch` without `--run`, which now leaves a `hold` entry that no release removes). Each run counts for 120 minutes from its own start, so one that never reaches `gstack-skill-end` stops blocking on time even while other sessions keep starting. A nested skill's own run covers it while it runs, but other sessions' starts no longer extend a run past its 120 minutes. + +## [1.94.18.0] - 2026-11-03 + +**The Windows-safe test subset is green again on current main.** `test/qa-report-dir.test.ts` expected `ENOTDIR` when `.gstack` is a file, but Windows reports `ENOENT` for the same mistake, so the test failed on every Windows run (windows-free-tests run 41607809007 at f6d6215). The test now accepts either code on Windows and still requires `ENOTDIR` everywhere else. With Windows checks now running by path, a red subset test would block any pull request that touches it. + +## [1.94.17.0] - 2026-11-03 + +- A pregate guard whose exact command already passed at this tree no longer runs again (GST-1186). When the base branch's `ship_lanes_from` maps the command, it is carried from the verified CI run through the same `gstack-evidence record-ci` path /ship lanes use (`GUARD: carried runs=…`); otherwise a counted local ledger pass of that command at the same working tree stands (CI rows count only through a live `record-ci` check) (`GUARD: reused …`, and the new read-only `gstack-evidence passed -- `). Repos whose suites are guards, like founderstar, stop rerunning about six minutes of suites per /ship. See docs/pregate.md#guard-carry. +- Known limit: `ship_lanes_from` is still read from `origin/` only, so the PR that adds a mapping cannot carry from it. +- A `ship_lanes_from` entry can name the CI jobs that run a lane instead of an inventory command: `{ "profiles": [...], "jobs": ["verify", "typecheck"] }`, exactly one of the two (GST-1189). It carries only when every named job succeeded in the newest attempt of the verified run at HEAD's tree, through the same `verifyRun` job check. A missing job refuses as `job_missing` and a skipped one as the new `job_skipped`. Lanes with no test files, like gbrain's verify and typecheck (run 41564913688) or harness-evals' `bunx tsc --noEmit`, can now carry, and they record `tests_ran=n/a`. Fake `check:` entries in `tree.json` can move to this form. A repo declaring `jobs` should raise its `min_gstack_version` to this release first, since older gstack rejects the key. See docs/pregate.md#ship-lanes-jobs. + +- A `ship_lanes_from` entry can declare `aliases`, other exact spellings of the same lane such as `env -u ANTHROPIC_API_KEY bun test tests/` (GST-1190). gstack never strips env prefixes on its own, since an `env -i` run is a different environment. +- A carried lane no longer reports the whole run's test count. An inventory lane reports how many of its files the run covered, labeled as a file count (alphaclaw's 46-test guards lane recorded 9677 before). +- `gstack-ship-preflight --tests-only --policy-from head --simulate` proves a new mapping before it lands: every line starts `SIMULATION:`, no lane runs, no evidence row is written and nothing clears. An inventory-mapped lane runs the branch's inventory command, so the dry run needs `--allow-repo-commands`. A branch's own map never carries, so the first pull request that adopts a mapping pays its full lanes. +- `gstack-evidence carried`, which the ship receipt reads, lists only rows the receipt base's policy mapped, so a run against another base never lands in a receipt. +- An unknown `.gstack/pregate.json` key under a `min_gstack_version` above the running gstack now says `needs gstack >= ; upgrade` (`PREGATE_GSTACK_TOO_OLD`). It still refuses. +- docs/pregate.md#prove-a-mapping documents the proof route, the version floors, and that `tree.json` writers using Bun's JUnit must take the root `testsuite` count. + +- A test file whose every test skipped on the CI runner no longer counts as carried (GST-1190, harness-evals: 75 tests). A `tree.json` that lists such files in `skipped_files` makes the pregate tests stage run them here. A /ship lane or guard whose inventory names one carries the covered files and runs only the skipped ones here, through the `per_file` runner its `ship_lanes_from` entry declares. The CI row is written only after that run passes with a test count above zero and the carried runs still stand (`skipped_failed`, `skipped_zero_run` or `run_changed` otherwise), and an entry without `per_file` refuses as `inventory_skipped`, naming the files. A partly skipped file still counts. An older `tree.json` without the list is read as before. gstack itself changes nothing yet: its own writer and a `per_file` entry for its `bun run test` lane switch on together in a follow-up pull request, after the pull requests now queued on older gstack have landed, because an older gstack rejects the new key in main's policy. + +- A run profile can declare `skippable: { jobs, gate, scripts }` for jobs a path filter skips (GST-1190, truther #1169). A skipped job counts only in a `pull_request` run whose gate job succeeded, with the workflow, the declared `scripts` (every file the gate reads), the paths the gate job names (also under its `working-directory`) and `package.json` when it runs a package script matching `origin/` byte for byte; the lane then prints `not owed: CI skipped at this tree (gate job , workflow unchanged)`. If the branch changed any of them, it controlled the skip, so the lane runs here. +- `gstack-ci-carry find --policy-from head --simulate` is the same dry run for one lane command, every line starting `SIMULATION:`. + +### Tests +- `test/ship-ci-carry.test.ts`: in founderstar's shape (guards only, no lanes), `check-quick` carries from CI, `play-through` reuses a same-tree ledger pass, `assets` runs, and a dirty tree carries nothing. It fails without the guard carry. +- `test/ship-ci-carry.test.ts`: a lane mapped by `jobs` carries when every named job passed at the same tree with no test files, and refuses for a missing job, a skipped job, a rerun whose newest attempt skipped the job, and another tree's run. Those five fail without the change. +- `test/ship-ci-carry.test.ts` (GST-1190): an alias carries and an undeclared env prefix does not; a narrow lane reports its covered file count, not 9677; a jobs-mapped `bunx tsc --noEmit` guard carries with `n/a`; the newer-gstack message; `--simulate` writes and clears nothing and the branch's own map never carries; `carried` drops a row another base mapped. Six of these fail without the change. +- `test/ship-ci-carry.test.ts` (GST-1190 skipped files): a fully skipped inventory file stops the lane carry, an empty list or older `tree.json` carries, and the tests stage runs only the skipped selected file. Both fail without the change. +- `test/ship-ci-carry.test.ts` (path-skipped jobs and `find --simulate`): a skip under an unchanged, green gate is not owed; a changed workflow, a changed gate script or a failed gate job runs the lane here; `find --policy-from head --simulate` prints only SIMULATION lines and writes nothing. All five fail without the change. +- `test/ship-ci-carry.test.ts` (review fixes): a gate script behind `${{ github.workspace }}`, under `working-directory` or behind a package script is compared; the gate is the job, not a same-named step; only a `pull_request` run's skip counts; a skippable profile must declare `scripts`; `passed` never counts a CI row; a path-skipped guard says not owed; `--simulate` runs a branch inventory only with `--allow-repo-commands`. Eight fail without the fixes; a newer-failure veto test covers existing behavior. +- `test/ship-ci-carry.test.ts` (per-file lane carry): the covered file carries and only the skipped file runs through `per_file`, a failing or zero-test remainder, or a run that moved to a new attempt meanwhile, records no CI row and runs the lane whole, and `per_file` needs `{files}` and an inventory entry. All five fail without the change. + +## [1.94.16.0] - 2026-11-03 + +**A declared placeholder restamped to the new version is a stamp, and a carried receipt names the head's version (GST-1026, GST-1030).** gbrain writes a PR's own BEHAVIOR_CHANGES row as `since: 'TBD'` and its `release:restamp` fills in the version. Finish refused that receipt ("behavior, tests or build inputs changed since review"), and the receipt carry and the tree receipt would have refused the same row (gbrain #8444). Separately, a receipt carried after a main merge and a restamp kept the version of the head it was written at (GGST-1). + +### Added +- **`stamp_placeholders` policy key** (default `[]`, so nothing changes for a repo that does not set it): literal placeholders such as `["TBD"]`, at most 16, one line and 32 characters each, never `{version}` and never a version. It needs `min_gstack_version` at the release that introduced it. + +### Fixed +- **`lib/stamp-only.ts`:** `stampOnlyLine` takes the policy's placeholders. On a line the branch added itself (named by its position), a placeholder inside a `stamp_patterns` template may become that template at the new version, alongside any stamp-form version move on the same line. The built-in forms, a bare `{version}` template, base lines (including one whose text duplicates a branch line) and undeclared placeholders are still refused. Finish's `stampedFiles`, the receipt carry and the tree receipt share it. +- **`bin/gstack-ship-receipt` carry:** the carried receipt's `version` is HEAD's, read the way `write` reads it, instead of the version copied from the old receipt. A head with no version source keeps the version `write --version` recorded. +- **`bin/gstack-ship-receipt` carry, pin-bump PRs:** a `package.json` the branch changed as product (a dependency pin) that both main and a restamp moved since review is compared with only its `version` field masked, at all four sides. The receipt carries when nothing else in the file moved, and still refuses when the pin or any other field changed (GBEV-3, gbrain-evals #306). + +### Tests +- `test/tree-receipt-stamp-sensitive.test.ts`, `test/gstack-ship-receipt-docs.test.ts`: the shared predicate and the carry classifier accept gbrain's `since: 'TBD'` → `since: '0.60.553.0'` row and refuse an undeclared placeholder, a base row, a duplicated base copy, a built-in form and a line no template matches. +- `test/ship-rereview-triggers.test.ts`: finish routes the declared row as metadata and stops on the same controls. +- `test/restamp-mentions.test.ts`: the tree receipt and the carry accept the declared row and refuse it undeclared or on a base row; the carried receipt records the head's version after a main merge and a restamp; a version-only restamp of an EVIDENCE.md declared in `stamp_files` carries, and an undeclared one stays product; a pin-bump PR's receipt carries after a main merge and a restamp, and refuses when the pin moved since review. +- `test/gstack-ship-policy.test.ts`: the key's default, floor and validation. + +The EVIDENCE.md refusal (GST-1030's second point) reproduces on main and is already fixed by the GST-980 own-row carry this branch builds on, so this change adds only the test for it. + ## [1.94.15.0] - 2026-11-03 - **A stacked PR keeps its /ship receipt after its parent squash-merges, even when the receipt was written against main (GST-1199).** `gstack-ship-receipt carry` used to refuse with `RECEIPT_CARRY_PRODUCT_CHANGED` once the parent's hunks moved into main, forcing a full re-review of unchanged code. The carry now finds the newest reviewed commit that main already holds (`git merge-tree` into main changes no product path) and compares from there: an unchanged child carries as `stack parent landed`, base edits next to the child's hunks go to `RECEIPT_CARRY_SCOPED_REVIEW`, and any other change still refuses. A parent that never landed never counts, even if the child's merge dropped its change. diff --git a/VERSION b/VERSION index f2224bc7b..807c34fb9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.94.15.0 +1.94.19.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index d3eba6498..9468d6bf6 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.94.15.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.94.19.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-ci-carry b/bin/gstack-ci-carry index c29156107..c5292f5eb 100755 --- a/bin/gstack-ci-carry +++ b/bin/gstack-ci-carry @@ -3,6 +3,7 @@ * gstack-ci-carry — CI as the one full test run (GSTA-258; docs/pregate.md#ci-carry). * * gstack-ci-carry find [--base ] [--profile ]... [--command --allow-repo-commands] [--json] + * [--policy-from head --simulate] * gstack-ci-carry wait [--base ] [--profile ] [--timeout ] [--interval ] * * find: the newest verified run of each profile (default: every profile the base branch's @@ -14,10 +15,9 @@ * carried, 3 still running or no run. Logic: lib/ci-carry.ts; the run checks are lib/run-evidence.ts. */ import { spawnSync } from 'node:child_process'; -import { findCarry, waitForRun } from '../lib/ci-carry'; -import { isConfigUnavailable, loadConfigFromBase } from '../lib/pregate/config'; +import { findCarry, laneMapping, mappingCoverage, waitForRun } from '../lib/ci-carry'; +import { isConfigUnavailable, loadConfigFromBase, loadConfigFromHead } from '../lib/pregate/config'; import { buildRunCheck } from '../lib/pregate/run-check'; -import { runInventory } from '../lib/run-evidence'; import { wtreeFingerprint } from '../lib/script-invocation'; import { resolveBaseBranch } from '../lib/ship-policy'; @@ -25,11 +25,13 @@ const HELP = `gstack-ci-carry — use a verified CI run at HEAD's exact tree ins find [--base ] [--profile ]... [--command --allow-repo-commands] [--json] CI_CARRY: accepted (exit 0) or CI_CARRY: none () (exit 1) + --policy-from head --simulate: the same check under the branch's own map and profiles, every line starting + SIMULATION:; a dry run that proves a new mapping before it lands (find never writes evidence; nothing clears) wait [--base ] [--profile ] [--timeout ] [--interval ] poll the newest run at HEAD (default 1800 s, every 20 s): CI_WAIT: green|red|running|none, then find on green; exit 0 carried, 1 red or not carried, 3 still running or no run -Profiles and ship_lanes_from come from origin/'s .gstack/pregate.json only. Only a clean, committed tree +Profiles and ship_lanes_from come from origin/'s .gstack/pregate.json only (a simulation reads the branch's). Only a clean, committed tree carries; fork runs, stale runs, a newer red or running run on HEAD and a tree.json that misses an inventory file never do. Exit 2 is usage.`; @@ -37,30 +39,35 @@ const argv = process.argv.slice(2); const cmd = argv[0]; const all = (n: string) => argv.flatMap((a, i) => (a === n && argv[i + 1] !== undefined ? [argv[i + 1]!] : [])); const flag = (n: string) => all(n).at(-1); -const out = (s: string) => process.stdout.write(`${s}\n`); +const simulate = argv.includes('--simulate'); +const out = (s: string) => process.stdout.write(`${simulate ? 'SIMULATION: ' : ''}${s}\n`); const usage = (m: string) => { process.stderr.write(`gstack-ci-carry: ${m}\n`); process.exit(2); }; if (!cmd || argv.includes('--help') || argv.includes('-h')) { out(HELP); process.exit(cmd ? 0 : 2); } if (cmd !== 'find' && cmd !== 'wait') usage(`unknown command ${cmd}; run gstack-ci-carry --help`); +const policyFrom = flag('--policy-from') ?? 'base'; +if (policyFrom !== 'base' && policyFrom !== 'head') usage('--policy-from needs base or head'); +if (simulate !== (policyFrom === 'head') || (simulate && cmd !== 'find')) usage('find --policy-from head --simulate go together: a branch\'s own map is only ever simulated, never carried'); const git = (args: string[]) => { const r = spawnSync('git', args, { encoding: 'utf8', timeout: 30_000 }); return r.status === 0 ? r.stdout.trim() : ''; }; const root = git(['rev-parse', '--show-toplevel']); if (!root) usage('not inside a git repository'); const base = resolveBaseBranch(root, flag('--base')); -const loaded = loadConfigFromBase(root, base); -if (isConfigUnavailable(loaded) || loaded.errors.length) { out(`CI_CARRY: none (profiles_unavailable: origin/${base}'s .gstack/pregate.json is unreadable or invalid)`); process.exit(1); } +const loaded = simulate ? loadConfigFromHead(root) : loadConfigFromBase(root, base); +if (simulate) out(`policy ${isConfigUnavailable(loaded) ? 'unavailable' : loaded.label}; a dry run: nothing is written and nothing clears (a carry reads origin/${base}'s map only)`); +if (isConfigUnavailable(loaded) || loaded.errors.length) { out(`CI_CARRY: none (profiles_unavailable: ${simulate ? 'the branch\'s' : `origin/${base}'s`} .gstack/pregate.json is unreadable or invalid)`); process.exit(1); } const command = flag('--command'); -const mapping = command !== undefined ? loaded.config.ship_lanes_from[command] : undefined; -if (command !== undefined && !mapping) { out(`CI_CARRY: none (not_mapped: ${JSON.stringify(command)} is not a key of origin/${base}'s ship_lanes_from)`); process.exit(1); } +const mapping = command !== undefined ? laneMapping(loaded.config.ship_lanes_from, command) : undefined; +if (command !== undefined && !mapping) { out(`CI_CARRY: none (not_mapped: ${JSON.stringify(command)} is neither a key nor a declared alias in origin/${base}'s ship_lanes_from)`); process.exit(1); } if (mapping && !argv.includes('--allow-repo-commands')) usage('--command runs the mapping\'s inventory, a pin-file command: pass --allow-repo-commands'); const profiles = all('--profile').length ? all('--profile') : mapping ? mapping.profiles : [...new Set(Object.values(loaded.config.ship_lanes_from).flatMap(m => m.profiles))]; if (!profiles.length) { out(`CI_CARRY: none (no_profile: origin/${base} maps no lane in ship_lanes_from; pass --profile )`); process.exit(1); } -const check = buildRunCheck(root, null, argv.includes('--allow-repo-commands'), base); +const check = buildRunCheck(root, null, argv.includes('--allow-repo-commands'), base, simulate ? 'head' : 'base'); if ('error' in check) { out(`CI_CARRY: none (profiles_unavailable: ${check.error})`); process.exit(1); } const head = git(['rev-parse', 'HEAD']); const tree = git(['rev-parse', 'HEAD^{tree}']); function find(): number { const wtree = () => { const fp = wtreeFingerprint({ cwd: root }); return 'tree' in fp ? fp.tree : null; }; - const v = findCarry(check as Exclude, { profiles, head, tree, wtree, ...(mapping ? { inventory: () => runInventory(mapping.inventory, root) } : {}) }); + const v = findCarry(check as Exclude, { profiles, head, tree, wtree, ...(mapping ? mappingCoverage(mapping, root) : {}) }); if (argv.includes('--json')) out(JSON.stringify(v)); else out(v.ok ? `CI_CARRY: accepted ${v.receipt}` : `CI_CARRY: none (${v.reason}: ${v.detail})`); return v.ok ? 0 : 1; diff --git a/bin/gstack-doctor-components.sh b/bin/gstack-doctor-components.sh index 23db8f80d..4ec7b9a61 100644 --- a/bin/gstack-doctor-components.sh +++ b/bin/gstack-doctor-components.sh @@ -258,6 +258,7 @@ gstack_result_anchor() { DX_CAPTURE_REDACTION_FAILED) printf '%s' 'dx-capture-redaction-failed' ;; DX_CAPTURE_INTERNAL) printf '%s' 'dx-capture-internal' ;; POLICY_GSTACK_TOO_OLD) printf '%s' 'policy-gstack-too-old' ;; + PREGATE_GSTACK_TOO_OLD) printf '%s' 'pregate-gstack-too-old' ;; QUEUE_NOT_READY) printf '%s' 'queue-not-ready' ;; QUEUE_PROOF_MISSING) printf '%s' 'queue-proof-missing' ;; QUEUE_STAMP_NOT_STAMP_ONLY) printf '%s' 'queue-stamp-not-stamp-only' ;; diff --git a/bin/gstack-evidence b/bin/gstack-evidence index 8fc3e692b..e3d2bbab1 100755 --- a/bin/gstack-evidence +++ b/bin/gstack-evidence @@ -5,7 +5,8 @@ * * gstack-evidence run [--reuse [--max-age ]] --label -- * gstack-evidence record-ci --label --base -- - * gstack-evidence carried [--json] + * gstack-evidence passed -- + * gstack-evidence carried [--base ] [--json] * gstack-evidence check [--label [--expect-cmd ]]... | --all * [--max-age ] [--allow-paths ] * gstack-evidence bundle --out [--label ]... [--require ] [--selection L=]... @@ -65,8 +66,11 @@ * `record-ci` writes the explicit CI-sourced row for a lane CI carried: it * finds and verifies the run itself (lib/ci-carry.ts) for a command the base's * ship_lanes_from maps, then records source "ci", the runs and profiles, the - * run's finish time as ts, bound to HEAD's clean tree; `carried` lists those - * rows at the current tree for the ship receipt. + * run's finish time as ts, bound to HEAD's clean tree, and the policy that + * mapped it (`origin/@:…`); `carried` lists those rows at the + * current tree for the ship receipt, only when that policy is origin/ (GST-1190: a row a run against another base wrote never + * counts). * * MACHINE-LOCAL by design: neither the ledger nor the logs are brain-synced. * A synced record citing an unsynced log would grade FRESH on a machine where @@ -90,10 +94,10 @@ import { resultLine } from "../lib/result-codes"; import { isUnavailable, loadPolicyFromBase, resolveBaseBranch } from "../lib/ship-policy"; import { versionAtRef } from "../lib/restamp"; import { bashCommandInvocation, runScriptSync, wtreeFingerprint } from "../lib/script-invocation"; -import { findCarry } from "../lib/ci-carry"; +import { findCarry, laneMapping, mappingCoverage, perFileCommand } from "../lib/ci-carry"; +import { runChanged } from "../lib/run-evidence"; import { isConfigUnavailable, loadConfigFromBase } from "../lib/pregate/config"; import { buildRunCheck } from "../lib/pregate/run-check"; -import { runInventory } from "../lib/run-evidence"; const BIN_DIR = dirname(Bun.fileURLToPath(import.meta.url)); const LOG_MAX_BYTES = 2 * 1024 * 1024; @@ -122,10 +126,17 @@ interface EvidenceRecord { source?: "ci"; runs?: string[]; profiles?: string[]; + /** The CI jobs a jobs-mapped lane carried from (record-ci, GST-1189); tests_ran is then "n/a". */ + jobs?: string[]; /** The base policy that mapped a carried lane (record-ci), as `origin/@:.gstack/pregate.json`. */ policy?: string; /** The runner's own test count (executed for bun/jest/vitest/pytest/rspec; ran with skips included for gstack's shard runner and the remote gate), 0 for ZERO-RUN, or "unknown" when no supported summary was seen. */ - tests_ran?: number | "unknown"; + tests_ran?: number | "unknown" | "n/a"; + /** "files" when a carried narrow lane's tests_ran counts its inventory files the run covered (record-ci), not tests. */ + tests_unit?: "files"; + /** Why a carried lane was not owed at all: every job it names was path-skipped by an unchanged gate (record-ci, GST-1190). */ + not_owed?: string; + skipped_here?: { label: string; files: string[]; tests_ran: number | "unknown" }; } /** Bytes of streamed output kept for the tests_ran parse (the 2MB log would truncate the summary). */ @@ -598,6 +609,36 @@ function identityDigest(): string { return sha256(JSON.stringify({ bun: version("bun"), node: version("node"), os: process.platform, arch: process.arch, image: process.env.GSTACK_RUNNER_IMAGE || null })); } +/** A local row that passed with a counted run within the max age, under this machine's identity. */ +function countedPass(r: EvidenceRecord, maxAgeHours: number): boolean { + if (r.exit !== 0 || typeof r.tests_ran !== "number" || r.tests_ran <= 0) return false; + const age = Date.now() - Date.parse(r.ts); + if (!(age >= 0 && age <= maxAgeHours * 3600_000)) return false; + return r.identity === identityDigest(); +} + +/** + * `passed -- ` (GST-1186, read-only): did this exact command pass here at the current working tree under any label? The + * newest local row for the command hash and tree decides, so a newer failure vetoes; it must be a counted pass (countedPass). + * CI rows never count: a CI pass counts only through record-ci's live check against the base. Prints PASSED: label= + * source=local ts= and exits 0, else NOT_PASSED: and exits 1. gstack-pregate's guards read it. + */ +function cmdPassed(argv: string[]): number { + const sep = argv.indexOf("--"); + const command = sep < 0 ? "" : argv.slice(sep + 1).join(" "); + if (!command || sep !== 0) { console.error("usage: gstack-evidence passed -- "); return 2; } + const wtree = currentWtree(); + if (!wtree) { console.log("NOT_PASSED: no working-tree fingerprint"); return 1; } + let records: EvidenceRecord[] = []; + try { records = readJsonl(ledgerPath().file); } catch { records = []; } + const cmdSha = sha256(command); + const newest = records.findLast((r) => r.cmd_sha256 === cmdSha && r.wtree === wtree && r.source === undefined); + if (!newest) { console.log(`NOT_PASSED: no local ledger row for this command at tree ${wtree.slice(0, 12)}`); return 1; } + if (!countedPass(newest, 24)) { console.log(`NOT_PASSED: the newest row (label=${newest.label} ts=${newest.ts}) is not a counted pass within 24 h under this runtime`); return 1; } + console.log(`PASSED: label=${newest.label} source=local ts=${newest.reused_from ?? newest.ts}`); + return 0; +} + /** * `run --reuse`: the newest local row for this label, command and working tree decides. A counted pass under the same * identity within the max age is copied (original ts kept) and printed as a run would be; anything else runs the command. @@ -612,10 +653,7 @@ function reused(file: string, label: string, cmdSha: string, maxAgeHours: number return false; } const newest = records.findLast((r) => r.label === label && r.cmd_sha256 === cmdSha && r.wtree === wtree && r.source === undefined); - if (!newest || newest.exit !== 0 || typeof newest.tests_ran !== "number" || newest.tests_ran <= 0) return false; - const age = Date.now() - Date.parse(newest.ts); - if (!(age >= 0 && age <= maxAgeHours * 3600_000)) return false; - if (newest.identity !== identityDigest()) return false; + if (!newest || !countedPass(newest, maxAgeHours)) return false; try { appendJsonl(file, { ...newest, reused_from: newest.reused_from ?? newest.ts }, { mode: 0o600 }); } catch (e: any) { @@ -629,9 +667,11 @@ function reused(file: string, label: string, cmdSha: string, maxAgeHours: number /** * `record-ci`: carry a lane from CI. Its command must be a key of the base branch's ship_lanes_from; the run is found and - * verified here (lib/ci-carry.ts) at HEAD's clean tree, so no caller supplies run ids, a test count or a tree. Never runs the lane. + * verified here (lib/ci-carry.ts) at HEAD's clean tree, so no caller supplies run ids, a test count or a tree. Never runs the lane; + * inventory files every run skipped entirely run here through the entry's per_file runner (label .skipped) first, and the CI + * row is written only when that run passes at the same tree. */ -function cmdRecordCi(argv: string[]): number { +async function cmdRecordCi(argv: string[]): Promise { const sep = argv.indexOf("--"); const opts = sep < 0 ? argv : argv.slice(0, sep); const get = (name: string) => { const i = opts.indexOf(name); return i >= 0 ? opts[i + 1] : undefined; }; @@ -651,17 +691,33 @@ function cmdRecordCi(argv: string[]): number { const loaded = loadConfigFromBase(cwd, base); if (isConfigUnavailable(loaded)) return none("config_unavailable", loaded.detail); if (loaded.errors.length) return none("config_invalid", loaded.errors.map((x) => `${x.path}: ${x.message}`).join("; ")); - const from = loaded.config.ship_lanes_from; - if (!Object.prototype.hasOwnProperty.call(from, command)) return none("not_mapped", `the command is not a key of origin/${base}'s ship_lanes_from`); - const m = from[command]!; + const m = laneMapping(loaded.config.ship_lanes_from, command); + if (!m) return none("not_mapped", `the command is neither a key nor a declared alias in origin/${base}'s ship_lanes_from`); const check = buildRunCheck(cwd, null, true, base); if ("error" in check) return none("profiles_unavailable", check.error); const head = git(["rev-parse", "HEAD"]); const tree = git(["rev-parse", "HEAD^{tree}"]); if (!head || !tree) return none("no_head", "HEAD is unreadable"); const wtree = () => { const fp = wtreeFingerprint({ cwd, timeoutMs: 30000 }); return "tree" in fp ? fp.tree : null; }; - const v = findCarry(check, { profiles: m.profiles, head, tree, wtree, inventory: () => runInventory(m.inventory, cwd) }); + const v = findCarry(check, { profiles: m.profiles, head, tree, wtree, ...mappingCoverage(m, cwd) }); if (!v.ok) return none(v.reason, v.detail); + let here: EvidenceRecord["skipped_here"]; + if (v.remainder?.length) { + const hereLabel = `${label}.skipped`; + console.error(`gstack-evidence record-ci: ${v.remainder.length} inventory file(s) skipped every test in CI; running them here (label=${hereLabel})`); + const hereCommand = perFileCommand(m.per_file!, v.remainder); + const exit = await cmdRun(["--label", hereLabel, "--", hereCommand]); + if (exit !== 0) return none("skipped_failed", `the per_file run of ${v.remainder.length} skipped file(s) exited ${exit} (label=${hereLabel})`); + let rows: EvidenceRecord[] = []; + try { rows = readJsonl(ledgerPath().file); } catch { rows = []; } + here = { label: hereLabel, files: v.remainder, tests_ran: rows.findLast((r) => r.label === hereLabel && r.source === undefined && r.cmd_sha256 === sha256(hereCommand) && r.wtree === tree)?.tests_ran ?? "unknown" }; + if (typeof here.tests_ran !== "number" || here.tests_ran <= 0) return none("skipped_zero_run", `the per_file run of ${v.remainder.length} skipped file(s) ran ${here.tests_ran} tests here (label=${hereLabel}); a lane passes on its runner count, never exit 0 alone`); + for (const e of v.runs) { + const changed = runChanged(check.deps, e.run_id, e.run_attempt); + if (changed) return none("run_changed", changed); + } + if (wtree() !== tree) return none("tree_changed", `the working tree changed while the per_file run of label=${hereLabel} ran`); + } let paths: ReturnType; try { paths = ledgerPath(); @@ -673,15 +729,22 @@ function cmdRecordCi(argv: string[]): number { const runs = v.runs.map((r) => `${r.run_id}/${r.run_attempt}`); const rec: EvidenceRecord = { ts: new Date(v.runs.map((r) => r.finished_at).sort()[0]!).toISOString(), label, command, cmd_sha256: sha256(command), exit: 0, duration_s: 0, - commit: head, tree, dirty: false, wtree: tree, tests_ran: v.tests, source: "ci", runs, profiles: m.profiles, policy: loaded.label, + commit: head, tree, dirty: false, wtree: tree, tests_ran: v.tests, source: "ci", runs, profiles: m.profiles, policy: loaded.label, ...(m.jobs ? { jobs: m.jobs } : {}), ...(v.unit ? { tests_unit: v.unit } : {}), ...(v.notOwed ? { not_owed: v.notOwed } : {}), ...(here ? { skipped_here: here } : {}), }; appendJsonl(paths.file, rec, { mode: 0o600 }); - console.error(`gstack-evidence: recorded label=${label} exit=0 tests_ran=${v.tests} log=- source=ci runs=${runs.join(",")} policy=${loaded.label}`); + console.error(`gstack-evidence: recorded label=${label} exit=0 tests_ran=${v.tests}${v.unit ? ` unit=${v.unit}` : ""} log=- source=ci runs=${runs.join(",")} policy=${loaded.label}${here ? ` skipped_here=${here.files.length} tests_ran_here=${here.tests_ran}` : ""}`); + if (v.notOwed) console.error(`gstack-evidence: not owed: ${v.notOwed}`); return 0; } -/** `carried`: each label's newest row when it is a CI-sourced row at the current working tree, as the receipt's ci_carried entries. */ +/** + * `carried`: each label's newest row when it is a CI-sourced row at the current working tree whose mapping policy is + * origin/ (the receipt's base; default the resolved base branch), as the receipt's ci_carried entries. + */ function cmdCarried(argv: string[]): number { + const at = argv.indexOf("--base"); + if (at >= 0 && !argv[at + 1]) { console.error("usage: gstack-evidence carried [--base ] [--json]"); return 2; } + const policyRef = `origin/${(at >= 0 ? argv[at + 1]! : resolveBaseBranch(git(["rev-parse", "--show-toplevel"]) ?? process.cwd())).replace(/^origin\//, "")}@`; let records: EvidenceRecord[] = []; try { records = readJsonl(ledgerPath().file); @@ -691,7 +754,7 @@ function cmdCarried(argv: string[]): number { const wtree = currentWtree(); const latest = new Map(); for (const r of records) latest.set(r.label, r); - const carried = [...latest.values()].filter((r) => r.source === "ci" && !!wtree && r.wtree === wtree && Array.isArray(r.runs)) + const carried = [...latest.values()].filter((r) => r.source === "ci" && !!wtree && r.wtree === wtree && Array.isArray(r.runs) && typeof r.policy === "string" && r.policy.startsWith(policyRef)) .flatMap((r) => r.runs!.map((ref) => { const [run, attempt] = ref.split("/").map(Number); return { lane: r.label, run, attempt, tree: r.wtree! }; })); if (argv.includes("--json")) console.log(JSON.stringify(carried)); else for (const c of carried) console.log(`CI_CARRIED: ${c.lane} run=${c.run}/${c.attempt} tree=${c.tree.slice(0, 12)}`); @@ -756,10 +819,10 @@ function cmdCheck(argv: string[]): number { allFresh = false; continue; } - const testsRan = typeof latest.tests_ran === "number" ? latest.tests_ran : "unknown"; + const testsRan = typeof latest.tests_ran === "number" || latest.tests_ran === "n/a" ? latest.tests_ran : "unknown"; const changedDuringRun = Array.isArray(latest.wtree_changed) ? latest.wtree_changed.filter((p): p is string => typeof p === "string" && p !== "" && !/[\u0000-\u001f\u007f]/.test(p)).slice(0, 5) : []; const ci = latest.source === "ci" && Array.isArray(latest.runs) ? ` source=ci runs=${latest.runs.filter((s) => /^\d+\/\d+$/.test(s)).join(",")}` : ""; - const detail = `label=${label} exit=${latest.exit} tests_ran=${testsRan} ts=${latest.ts}${latest.log_path ? ` log=${latest.log_path}` : ""}${ci}`; + const detail = `label=${label} exit=${latest.exit} tests_ran=${testsRan}${latest.tests_unit === "files" ? " unit=files" : ""} ts=${latest.ts}${latest.log_path ? ` log=${latest.log_path}` : ""}${ci}`; let verdict: "FRESH" | "STALE" | "ZERO-RUN" = "FRESH"; let reason = ""; if (latest.exit !== 0) { @@ -911,7 +974,9 @@ try { if (sub === "run") { process.exit(await cmdRun(rest)); } else if (sub === "record-ci") { - process.exit(cmdRecordCi(rest)); + process.exit(await cmdRecordCi(rest)); + } else if (sub === "passed") { + process.exit(cmdPassed(rest)); } else if (sub === "carried") { process.exit(cmdCarried(rest)); } else if (sub === "check") { @@ -923,7 +988,7 @@ try { } else if (sub === "verify") { process.exit(cmdVerify(rest)); } else { - console.error("usage: gstack-evidence run|record-ci|carried|check|bundle|ancestor|verify ..."); + console.error("usage: gstack-evidence run|record-ci|passed|carried|check|bundle|ancestor|verify ..."); process.exit(2); } } catch (e: any) { diff --git a/bin/gstack-lease b/bin/gstack-lease index 7a0aa3348..8fe9a0139 100755 --- a/bin/gstack-lease +++ b/bin/gstack-lease @@ -8,8 +8,11 @@ # shares them. # # Usage: -# gstack-lease touch [--root ] [--pid

| --from ] -# write /gstack-leases/ holding the pid's start identity; +# gstack-lease touch [--root ] [--pid

| --from ] [--run ] +# write /gstack-leases/ holding the pid's start identity +# and the open run (gstack-skill-start's SESSION_ID); without --run +# it records a hold no release closes; a run counts 120 min from its +# start and a hold 120 min from its last renewal; # pid defaults to the harness pid behind this call # (bin/gstack-harness-pid.sh, the same derivation the session marker # uses), or behind --from (a caller whose own parent is short-lived @@ -24,8 +27,11 @@ # --count-shared-harness (bin/gstack-upgrade only) still prints an # ancestor that is a shared harness (capy-server: every session on the # machine runs under it) while its lease is under 120 minutes old +# gstack-lease release --pid

--run [--root ] +# close run on 's lease (gstack-skill-end); once every run it +# recorded is closed the lease is idle and live skips it (GST-1205) # gstack-lease list [--root ] -# print ` m alive|dead|stale` for every lease +# print ` m alive|idle|dead|stale` for every lease # A pid alone is not a holder: pids are reused after a reboot or a machine # wake. A lease records its process's start identity (Linux: boot id and # /proc start time; elsewhere `ps -o lstart`), and a lease whose pid now @@ -47,12 +53,13 @@ BIN="$(cd "$(dirname "$0")" && pwd -P)" } VERB="${1:-}"; [ $# -gt 0 ] && shift -ROOT="" PID="" EXCLUDE="" FROM="" SHARED=0 +ROOT="" PID="" EXCLUDE="" FROM="" SHARED=0 RUN="" while [ $# -gt 0 ]; do case "$1" in --root) ROOT="${2:-}"; shift ;; --pid) PID="${2:-}"; shift ;; --from) FROM="${2:-}"; shift ;; + --run) RUN="${2:-}"; shift ;; --exclude-ancestry-of) EXCLUDE="${2:-}"; shift ;; --count-shared-harness) SHARED=1 ;; *) echo "gstack-lease: unknown argument '$1'" >&2; exit 2 ;; @@ -60,8 +67,8 @@ while [ $# -gt 0 ]; do shift done case "$VERB" in - touch|live|list) ;; - -h|--help|'') sed -n '2,28p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + touch|live|list|release) ;; + -h|--help|'') sed -n '2,34p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "gstack-lease: unknown verb '$VERB'" >&2; exit 2 ;; esac [ -n "$ROOT" ] || ROOT=$(gstack_source_root "$BIN") @@ -248,6 +255,15 @@ reclaim() { unlock return 1 } +# Run entries (GST-1205), lines after the identity: `run ` per +# open run, `hold ` for a holder that opened none, `idle` once every +# run closed. An entry older than 120 minutes no longer counts. +# fresh FILE [DROP] — the entries under 120 minutes old, minus the line whose +# first two fields are DROP ("run " or "hold"). +fresh() { sed -n '2,$p' "$1" 2>/dev/null | awk -v now="$(date +%s)" -v drop="${2:-}" '($1 == "run" && NF == 3 && now - $3 < 7200 && $1 " " $2 != drop) || ($1 == "hold" && NF == 2 && now - $2 < 7200 && $1 != drop)'; } +# idle FILE — the lease records entries and none of them still counts. A lease +# with no entries (written without --run, or by an older gstack) is never idle. +idle() { [ -n "$(sed -n '2,$p' "$1" 2>/dev/null | grep -v '^$')" ] && [ -z "$(fresh "$1")" ]; } age_min() { # FILE — minutes since its mtime _m=$(stat -c %Y "$1" 2>/dev/null || stat -f %m "$1" 2>/dev/null || echo 0) echo $(( ( $(date +%s) - _m ) / 60 )) @@ -264,7 +280,20 @@ case "$VERB" in exit 3 } _tmp="$DIR/.$PID.$$" - identity "$PID" > "$_tmp" && mv -f "$_tmp" "$DIR/$PID" || { rm -f "$_tmp"; unlock; exit 2; } + _id=$(identity "$PID") _keep="" _held=0 _now=$(date +%s) + if [ -f "$DIR/$PID" ] && [ "$(head -1 "$DIR/$PID" 2>/dev/null)" = "$_id" ]; then + if [ -n "$RUN" ]; then _keep=$(fresh "$DIR/$PID" "run $RUN"); else _keep=$(fresh "$DIR/$PID" hold); fi + [ -z "$(sed -n '2,$p' "$DIR/$PID" 2>/dev/null | grep -v '^$')" ] && _held=1 + fi + { + printf '%s\n' "$_id" + [ -z "$_keep" ] || printf '%s\n' "$_keep" + if [ -n "$RUN" ]; then + [ "$_held" -eq 1 ] && echo "hold $(stat -c %Y "$DIR/$PID" 2>/dev/null || stat -f %m "$DIR/$PID" 2>/dev/null || echo "$_now")" + echo "run $RUN $_now" + elif [ -n "$_keep" ]; then echo "hold $_now" + fi + } > "$_tmp" && mv -f "$_tmp" "$DIR/$PID" || { rm -f "$_tmp"; unlock; exit 2; } find "$DIR" -maxdepth 1 -mmin +120 -type f ! -name '.lock*' -exec rm -f {} + 2>/dev/null for _c in "$DIR"/.lock*.claim.*; do [ -f "$_c" ] || continue @@ -274,12 +303,29 @@ case "$VERB" in rm -rf "$DIR"/.lock*.gone.* 2>/dev/null unlock exit 0 ;; + release) + case "$PID" in ''|*[!0-9]*) echo "gstack-lease: release needs --pid and --run " >&2; exit 2 ;; esac + [ -n "$RUN" ] && [ "$RUN" != - ] || { echo "gstack-lease: release needs --pid and --run " >&2; exit 2; } + [ -f "$DIR/$PID" ] || exit 0 + lock || exit 3 + f="$DIR/$PID" _tmp="$DIR/.$PID.$$" + if sed -n '2,$p' "$f" | awk -v id="$RUN" '$1 == "run" && $2 == id { found = 1 } END { exit !found }'; then + _keep=$(fresh "$f" "run $RUN") + { + head -1 "$f" + if [ -n "$_keep" ]; then printf '%s\n' "$_keep"; else echo idle; fi + } > "$_tmp" && touch -r "$f" "$_tmp" && mv -f "$_tmp" "$f" || { rm -f "$_tmp"; unlock; exit 2; } + fi + unlock + exit 0 ;; list) for f in "$DIR"/*; do [ -f "$f" ] || continue p=${f##*/} _age=$(age_min "$f") - echo "$p ${_age}m $(state "$p" "$(head -1 "$f" 2>/dev/null)" "$_age")" + _st=$(state "$p" "$(head -1 "$f" 2>/dev/null)" "$_age") + [ "$_st" = alive ] && idle "$f" && _st=idle + echo "$p ${_age}m $_st" done exit 0 ;; esac @@ -296,6 +342,7 @@ done FOUND=1 for f in "$DIR"/*; do [ -f "$f" ] || continue + idle "$f" && continue p=${f##*/} _age=$(age_min "$f") case "$ANCESTRY" in *" $p "*) [ "$SHARED" -eq 1 ] && [ "$_age" -lt 120 ] && gstack_shared_harness "$p" || continue ;; esac diff --git a/bin/gstack-ship-receipt b/bin/gstack-ship-receipt index 41a4490e5..15f1a15f3 100755 --- a/bin/gstack-ship-receipt +++ b/bin/gstack-ship-receipt @@ -175,9 +175,9 @@ function reviewedProduct(repoRoot: string, base: string, head: string, pregateFi return typeof id === 'string' ? { head_tree: tree, product_patch_id: id || 'none' } : { head_tree: tree }; } -/** The lanes CI carried at the checked-out tree, from this machine's evidence ledger (gstack-evidence carried); undefined when none. */ -function ciCarried(): ShipReceipt['ci_carried'] { - const r = runScriptSync(path.join(path.dirname(Bun.fileURLToPath(import.meta.url)), 'gstack-evidence'), ['carried', '--json'], { encoding: 'utf8', timeout: 60_000 }); +/** The lanes CI carried at the checked-out tree under origin/'s mapping, from this machine's evidence ledger (gstack-evidence carried); undefined when none. */ +function ciCarried(base: string): ShipReceipt['ci_carried'] { + const r = runScriptSync(path.join(path.dirname(Bun.fileURLToPath(import.meta.url)), 'gstack-evidence'), ['carried', '--base', base, '--json'], { encoding: 'utf8', timeout: 60_000 }); try { const list = JSON.parse(r.stdout); return Array.isArray(list) && list.length ? list : undefined; } catch { return undefined; } } @@ -271,7 +271,7 @@ function cmdWrite(rest: string[], json: boolean): number { version: flag(rest, '--version') ?? receiptVersion(repoRoot, head, base), base, gated_tree: flag(rest, '--gated-tree') ?? tree?.gated_tree, predecessor: flag(rest, '--predecessor') ?? tree?.predecessor?.recorded_head ?? undefined, tree: tree?.tree, gate_reuse: tree ? `${tree.gate_reuse} (${tree.gate_reuse_reason})` : undefined, - gate: jsonArg(flag(rest, '--gate')), ci: jsonArg(flag(rest, '--ci')), ci_carried: ciCarried(), + gate: jsonArg(flag(rest, '--gate')), ci: jsonArg(flag(rest, '--ci')), ci_carried: ciCarried(base), spend_usd: flag(rest, '--spend-usd') !== undefined ? (Number.isFinite(Number(flag(rest, '--spend-usd'))) ? Number(flag(rest, '--spend-usd')) : flag(rest, '--spend-usd')) : undefined, session_kind: sessionKind, artifacts_consumed: flag(rest, '--artifacts-consumed'), run: flag(rest, '--run'), thread: flag(rest, '--thread'), policy: flag(rest, '--policy') ?? tree?.policy, queue_mode: flag(rest, '--queue-mode'), preregistration: flag(rest, '--preregistration'), history: flag(rest, '--history'), @@ -507,13 +507,13 @@ function verifiedOutputs(repoRoot: string, base: string, pregateFile: string | u return regen.proven.filter(p => !/[*?[\]\\]/.test(p) && matchesAny(owned, p)); } -interface ProductRules { extra: { metadata: string[]; outputs: readonly string[]; fragments?: string }; metadata: string[]; templates: readonly string[] } +interface ProductRules { extra: { metadata: string[]; outputs: readonly string[]; fragments?: string }; metadata: string[]; templates: readonly string[]; placeholders: readonly string[] } function productRules(repoRoot: string, base: string, outputs: readonly string[]): ProductRules { const load = loadPolicyFromBase(repoRoot, base); const policy: ShipPolicy | null = isUnavailable(load) ? null : load.policy; const fragments = policy?.changelog_fragments?.dir.replace(/^(\.\/)+/, '').replace(/\/+$/, ''); const extra = { metadata: [...(policy?.stamp_files ?? []), ...(policy?.stamp_paths ?? []), ...(policy?.release_outputs ?? []), ...(policy?.changelog ? [policy.changelog] : [])], outputs, fragments }; - return { extra, metadata: [...CARRY_METADATA_GLOBS, ...extra.metadata, ...(fragments ? [`${fragments}/*.md`] : []), ...outputs], templates: policy?.stamp_patterns ?? [] }; + return { extra, metadata: [...CARRY_METADATA_GLOBS, ...extra.metadata, ...(fragments ? [`${fragments}/*.md`] : []), ...outputs], templates: policy?.stamp_patterns ?? [], placeholders: policy?.stamp_placeholders ?? [] }; } /** @@ -540,7 +540,7 @@ function productSpec(repoRoot: string, rules: ProductRules, from: string, to: st const versions = new Map(); const version = (ref: string) => { if (!versions.has(ref)) versions.set(ref, versionAt(ref)); return versions.get(ref)!; }; const stamped = (p: string) => matchesAny(rules.metadata, p) && !matchesAny(rules.extra.outputs, p); - const classify = (p: string, a: string, b: string) => classifyCarryPath(p, rules.extra, stamped(p) ? { before: blobAt(a, p), after: blobAt(b, p), oldVersion: version(a), newVersion: version(b), templates: rules.templates, side: a === from && b === to ? branch?.own(p) : undefined } : undefined); + const classify = (p: string, a: string, b: string) => classifyCarryPath(p, rules.extra, stamped(p) ? { before: blobAt(a, p), after: blobAt(b, p), oldVersion: version(a), newVersion: version(b), templates: rules.templates, placeholders: rules.placeholders, side: a === from && b === to ? branch?.own(p) : undefined } : undefined); const changed = changedRaw.split('\n').filter(Boolean).map(p => ({ path: p, class: classify(p, from, to) })); const excludeGlobs = (globs: readonly string[]) => globs.map(g => `:(exclude,glob)${g}`); const demoted = !branch ? changed.filter(c => c.class === 'product' && matchesAny(rules.metadata, c.path)).map(c => c.path) @@ -556,7 +556,7 @@ function productSpec(repoRoot: string, rules: ProductRules, from: string, to: st const was = before.split('\n'); const now = after.split('\n'); const lines = was.map((_, i) => i).filter(i => was[i] !== now[i]); - return was.length === now.length && lines.length > 0 && lines.every(i => side.at.has(i + 1) && stampOnlyLine(was[i]!, now[i]!, oldVersion, newVersion, rules.templates, side, i + 1)); + return was.length === now.length && lines.length > 0 && lines.every(i => side.at.has(i + 1) && stampOnlyLine(was[i]!, now[i]!, oldVersion, newVersion, rules.templates, side, i + 1, rules.placeholders)); }); return { changed, product: ['.', ...excludeGlobs(CARRY_TEST_GLOBS), ...excludeGlobs(metadataGlobs)], tests: [...CARRY_TEST_GLOBS.map(g => `:(glob)${g}`), ...excludeGlobs(metadataGlobs)], demoted, ownRows }; } @@ -718,6 +718,16 @@ function cmdCarry(rest: string[], json: boolean): number { if (changedProduct.length >= CARRY_NAMED_PATHS) break; const cls = changed.find(c => c.path === p)?.class ?? classifyCarryPath(p, rules.extra); if (cls === 'metadata' && spec.ownRows.includes(p) && !movedBase.has(p)) stampOnly.push(p); + if (cls === 'metadata' && spec.demoted.includes(p) && movedBase.has(p) && /(^|\/)package\.json$/.test(p)) { + const masked = Object.values(sides).map(ref => { + const r = spawnSync('git', ['show', `${ref}:${p}`], { cwd: repoRoot, encoding: 'utf8', timeout: 30_000, maxBuffer: 16 * 1024 * 1024 }); + try { const { version: _, ...rest } = JSON.parse(r.status === 0 ? r.stdout : 'null') ?? {}; return JSON.stringify(rest); } catch { return null; } + }); + const [rb, rv, cb, cv] = masked; + if (masked.some(m => m === null) || rv !== cv || rb !== cb) changedProduct.push(p); + else stampOnly.push(p); + continue; + } if (cls !== 'product') continue; const verdict = comparePath(repoRoot, sides, p); if (verdict === 'changed') changedProduct.push(p); @@ -795,7 +805,10 @@ function cmdCarry(rest: string[], json: boolean): number { depthField = { review_depth: { ...now, depth: deeper(receipt.review_depth.depth, now.depth) }, review_exclude: receiptExclude(review.exclude) }; } // The rebound product_patch_id is HEAD's diff against the carry's base, so a retargeted receipt names that base from now on. - const rebound = { ...receipt, ...(retarget ? { base: base.replace(/^origin\//, '') } : {}), ...(pregateField ?? {}), ...depthField }; + // A receipt that records a version names HEAD's, read as write reads it: a restamp after a base merge moves it (GST-1030). + // A head with no version source keeps the recorded one (write --version named it). + const version = receipt.version === undefined ? undefined : (receiptVersion(repoRoot, head, base) ?? receipt.version); + const rebound = { ...receipt, ...(version !== undefined ? { version } : {}), ...(retarget ? { base: base.replace(/^origin\//, '') } : {}), ...(pregateField ?? {}), ...depthField }; const carried = carryReceipt(rebound, head, plan.carry, { tree: headTree, fromTree, productPatchId: headPatch }); const errors = validateReceipt(carried); if (errors.length) return refused('RECEIPT_CARRY_REFUSED', `the carried receipt does not validate: ${errors[0]!.message} at ${errors[0]!.path}`); diff --git a/bin/gstack-skill-end b/bin/gstack-skill-end index 865255a46..441495814 100755 --- a/bin/gstack-skill-end +++ b/bin/gstack-skill-end @@ -57,6 +57,9 @@ _TEL_END=$(date +%s) case "$TEL_START" in ''|*[!0-9]*) TEL_START="$_TEL_END" ;; esac _TEL_DUR=$(( _TEL_END - TEL_START )) [ -n "$SESSION_ID" ] && rm -f "$_GH/analytics/.pending-$SESSION_ID" 2>/dev/null || true +# Release the workflow lease run gstack-skill-start opened (SESSION_ID starts +# with its harness pid), so a finished run stops blocking gstack-upgrade (GST-1205). +[ -n "$SESSION_ID" ] && "$_BIN/gstack-lease" release --pid "${SESSION_ID%%-*}" --run "$SESSION_ID" >/dev/null 2>&1 || true # Record completion before the final drain, including its durable enqueue. # The normal sync gates still decide publication: full may sync timelines, diff --git a/bin/gstack-skill-start b/bin/gstack-skill-start index 16b1c9133..aca52a879 100755 --- a/bin/gstack-skill-start +++ b/bin/gstack-skill-start @@ -127,13 +127,6 @@ fi mkdir -p "$_GH/sessions" 2>/dev/null || true touch "$_GH/sessions/$PARENT_PID" 2>/dev/null || true find "$_GH/sessions" -mmin +120 -type f -exec rm {} + 2>/dev/null || true -# The install-scoped twin (bin/gstack-lease): what bin/gstack-upgrade reads -# before moving a checkout other sessions share. Fail-open, but a lease the -# held lock kept from being written is reported. -"$_BIN/gstack-lease" touch --pid "$PARENT_PID" >/dev/null 2>&1 || { - [ $? -eq 3 ] && echo "gstack-lease: the lease directory stayed locked by a live process; this session holds no workflow lease, so bin/gstack-upgrade cannot see it" >&2 - true -} _SESSIONS=$(find "$_GH/sessions" -mmin -120 -type f 2>/dev/null | wc -l | tr -d ' ') _PROACTIVE=$("$_BIN/gstack-config" get proactive 2>/dev/null || echo "core") @@ -241,9 +234,18 @@ _TEL_PROMPTED=$([ -f "$_GH/.telemetry-prompted" ] && echo "yes" || echo "no") _TEL_START=$(date +%s) # PID+epoch alone is guessable; a random suffix makes the block-binding token # unforgeable by content reflected into the same tool result. Falls back to -# the plain form when urandom is unavailable. +# this process's pid when urandom is unavailable, so two same-second starts +# never share a SESSION_ID (it also keys their lease runs, GST-1205). _SID_RAND=$(od -An -N4 -tx4 /dev/urandom 2>/dev/null | tr -d ' \n' || true) -_SESSION_ID="$PARENT_PID-$_TEL_START${_SID_RAND:+-$_SID_RAND}" +_SESSION_ID="$PARENT_PID-$_TEL_START-${_SID_RAND:-$$}" +# The install-scoped twin (bin/gstack-lease): what bin/gstack-upgrade reads +# before moving a checkout other sessions share. This run holds it until +# gstack-skill-end releases its SESSION_ID (GST-1205). Fail-open, but a lease +# the held lock kept from being written is reported. +"$_BIN/gstack-lease" touch --pid "$PARENT_PID" --run "$_SESSION_ID" >/dev/null 2>&1 || { + [ $? -eq 3 ] && echo "gstack-lease: the lease directory stayed locked by a live process; this session holds no workflow lease, so bin/gstack-upgrade cannot see it" >&2 + true +} echo "TELEMETRY: ${_TEL:-off}" echo "TEL_PROMPTED: $_TEL_PROMPTED" echo "SESSION_ID: $_SESSION_ID" diff --git a/docs/capy-install.md b/docs/capy-install.md index 53326df6d..6f0feef46 100644 --- a/docs/capy-install.md +++ b/docs/capy-install.md @@ -187,7 +187,7 @@ run from inside a workflow (/ship's docs step running `install.sh --for document-release`, #3571) prints `SELF_UPGRADE: skipped (a gstack run in this session holds a workflow lease on this installation: ... (UPGRADE_BUSY))`, leaves the checkout where the run started and stays ok with `behind=`. Each -`gstack-skill-start` renews the lease. On Capy the leased pid is the machine's +`gstack-skill-start` renews the lease and records its `SESSION_ID` as an open run; `gstack-skill-end` closes that run, and a lease whose runs are all closed is idle and blocks nothing, so a finished /ship no longer holds the upgrade back (GST-1205). Each run counts for 120 minutes from its own start, whatever other sessions renew. A lease with no recorded run (written without `--run`, or by an older gstack) counts as before. On Capy the leased pid is the machine's long-lived `capy-server`, so the deferral ends when the lease turns 120 minutes old, not when the process exits. Every session on the machine runs under that same `capy-server`, so `bin/gstack-upgrade` run by hand also @@ -231,7 +231,7 @@ identities, a taker stalled over 2 minutes between its `mkdir` and its owner rename can be broken and then publish into its successor's lock. While a live process holds the lock, or it cannot be taken at all, `touch` exits 3 without writing (`gstack-skill-start` reports that the session holds no lease) and a reclaim -keeps the lease. `bin/gstack-lease list` shows `alive|dead|stale`, and +keeps the lease. `bin/gstack-lease list` shows `alive|idle|dead|stale`, and `GSTACK_UPGRADE_FORCE=1` overrides. ## What the installer does diff --git a/docs/pregate.md b/docs/pregate.md index 69f1ce088..3bac81d5f 100644 --- a/docs/pregate.md +++ b/docs/pregate.md @@ -88,9 +88,9 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`). | `stall_timeout_s` | seconds a test command may print nothing while using under a tenth of one CPU before its process group is killed and the check is `incomplete` (`PREGATE_STALLED`); default 300, `0` disables; `--stall-timeout` overrides; the `--remote` gate is exempt (see [supervision](#supervision)) | | `timeout_s` | seconds each runner process may take (the unit-file group, each serial file, each guard without its own `timeout_s`, and the remote gate's deadline) when `--timeout` is not passed; 1 to 3600, default 600. `gstack-ship-preflight` and the `finish` fallback run pregate without `--timeout`, so this is how a repo with long lanes sets theirs | | `remote_gate` | the repo's remote test gate command; under 8 cores the local test tier becomes a `requires-remote` obligation naming it, and `--remote` runs it (see [remote gate](#remote-gate)) | -| `run_profiles` | a map from each profile name to `{ workflow, events, jobs, artifact, max_age_hours?, os? }`: the GitHub Actions runs `clear --run` verifies as evidence on the identical tree; a profile named after a lane (`free-tests.yml/macos-named-regressions`) clears that lane (see [run evidence](#run-evidence)). `--from-run` and the reuse job verify runs against them too ([from-run](#from-run), [CI reuse](#ci-reuse)) | +| `run_profiles` | a map from each profile name to `{ workflow, events, jobs, artifact, max_age_hours?, os?, skippable? }`: the GitHub Actions runs `clear --run` verifies as evidence on the identical tree; a profile named after a lane (`free-tests.yml/macos-named-regressions`) clears that lane (see [run evidence](#run-evidence)). `--from-run` and the reuse job verify runs against them too ([from-run](#from-run), [CI reuse](#ci-reuse)) | | `remote_gate_runs` | `{ profiles, inventory, skipped? }`: the run profiles that together clear `pregate.json/remote_gate`, the command that prints the remote gate's test files, one per line, which their runs must cover, and an optional command that prints which selected files the gate itself never runs (see [run evidence](#run-evidence)) | -| `ship_lanes_from` | maps each exact /ship lane command to `{ profiles, inventory }`: the run profiles whose verified runs at HEAD's clean tree carry that lane instead of running it, and the command listing its test files, which their runs must cover (see [CI carry](#ci-carry)); read from the base branch only | +| `ship_lanes_from` | maps each exact /ship lane command to `{ profiles, inventory }` or `{ profiles, jobs }`, with optional `aliases` (and, on an inventory entry, an optional `per_file` runner for files CI skipped entirely, see [skipped files](#skipped-files)): the run profiles whose verified runs at HEAD's clean tree carry that lane instead of running it, and either the command listing its test files, which their runs must cover, or the CI jobs that must have succeeded (see [CI carry](#ci-carry)); read from the base branch only | | `allowlist` | `[{ finding, paths?, pattern?, reason }]`: benign findings the `secrets` check accepts and names as allowlisted, by gstack-redact finding id or non-secret category, path globs and a pattern anchored at the finding; never `env.value` (see [allowlist](#allowlist)) | Only the `origin/` copy or the user-owned file below is honoured, so a @@ -259,7 +259,8 @@ Two stages with separate clocks, both printed: `--allow-repo-commands` the selected guards are listed and the check is `incomplete`. Guards are read from the base branch's copy (or the user-owned file when the base carries none) and cannot be - downgraded to warn. See [guards](#guards). + downgraded to warn. A guard whose exact command already passed at this tree + is not run again ([guard carry](#guard-carry)). See [guards](#guards). **Stage 2, test execution:** @@ -710,7 +711,22 @@ other gate change. Each profile names the workflow file, the events it accepts, every job that must have passed (each exactly once), and the artifact holding the run's `tree.json`. `max_age_hours` (1–168, default 24) bounds how old the run may be, and `os` (`Linux`, `macOS`, `Windows`) requires -`tree.json` to record that runner OS. Each workflow uploads `tree.json` from +`tree.json` to record that runner OS. `skippable` +(`{ jobs, gate, scripts }`, GST-1190) lets named jobs be skipped by a path +filter: a skipped job counts only in a `pull_request` run (whose gate diffs +against the pull request's base; a push or dispatch gate diffs against the last +push, which the branch's push history controls), when the `gate` job (the one +whose output decides the skip) succeeded in the same attempt, and the workflow +file plus the gate's inputs are byte-identical to `origin/`. Those inputs +are the declared `scripts`, which must list every repo file the gate reads, +plus every path the gate job's steps name in the base copy of the workflow +(also read under the job's `working-directory` and after a `${{ … }}` +prefix), plus `package.json` when the gate runs a package script. gstack +cannot see what a gate script imports or what a package script runs, so list +those files in `scripts`. Otherwise the pull request controlled the skip +decision, so the job refuses as `job_skipped` and the lane runs here. A lane +whose every named job was skipped this way prints `LANE:

(m /bin/gstack-upgrade once the lease is 120 minutes old, or GSTACK_UPGRADE_FORCE=1 /bin/gstack-upgrade when no other session uses this checkout (UPGRADE_BUSY))` and leaves the -checkout where the run started. Each `gstack-skill-start` renews the lease. +checkout where the run started. Each `gstack-skill-start` renews the lease +and opens a run that its `gstack-skill-end` closes; a lease whose runs are all +closed is idle and never counts (GST-1205). **Fix.** Wait for those sessions to finish (`bin/gstack-lease list` shows -every lease with its age and `alive|dead|stale`; a holder is named as +every lease with its age and `alive|idle|dead|stale`; a holder is named as `pid

(m )`), then re-run `bin/gstack-upgrade`. For the session's own run, or a shared-harness lease, re-run once the lease is 120 minutes old. When you know no other session uses the checkout, `GSTACK_UPGRADE_FORCE=1 bin/gstack-upgrade` @@ -3733,6 +3735,23 @@ the entry is reported as unverified, not `REGEN_STALE`. **Fix.** Run the printed install command (`bun install`, or the one matching the repository's lockfile), then rerun `gstack-regen check` or the pre-gate. + +### `PREGATE_GSTACK_TOO_OLD` + +**Meaning.** `.gstack/pregate.json` uses a key this gstack does not know, and +the same tree's `.gstack/ship-policy.json` sets `min_gstack_version` above this +gstack. The key belongs to a newer gstack (one a later release adds), so the +line reads `needs gstack >= ; upgrade`. The policy still refuses, +exactly as `PREGATE_CONFIG_INVALID` would. Only a gstack that has this check +prints it; one released before it (1.94.0.0 included) reports the same key as +`PREGATE_CONFIG_INVALID`. + +**Fix.** Upgrade gstack on this machine to at least the version named +(`bin/gstack-upgrade`, or the session install with `--for `), then +rerun. A repo adopting a new key raises `min_gstack_version` first, so an older +gstack that has this check prints this instead of a generic invalid-config +error. + ### `PREGATE_CONFIG_INVALID` diff --git a/lib/ci-carry.ts b/lib/ci-carry.ts index 16aec8629..c258f5ec4 100644 --- a/lib/ci-carry.ts +++ b/lib/ci-carry.ts @@ -14,7 +14,9 @@ * max age, the run's own tree.json naming HEAD's tree); * every run executed tests; * with an inventory command, the runs' tree.json files cover every file it - * prints (a lane carries only what CI provably ran); + * prints (a lane carries only what CI provably ran); with named jobs instead + * (a lane with no test files, GST-1189), each of those jobs succeeded in the + * run's newest attempt, read by verifyRun like the profile's own jobs; * every run still stands after the inventory ran (runChanged). * * Anything else returns a refusal with a reason code, and the caller runs the @@ -25,12 +27,18 @@ * bounded by a timeout, and returns green, red with the failing jobs, still * running, or none. */ -import { runChanged, runReceiptFields, uncoveredFiles, verifyRun, type ApiRun, type RunDeps, type RunEvidence, type RunProfile, type RunSource } from './run-evidence'; +import { normalPath, runChanged, runInventory, runReceiptFields, uncoveredFiles, verifyRun, type ApiRun, type RunDeps, type RunEvidence, type RunProfile, type RunSource } from './run-evidence'; export type CarryRefusal = 'dirty_tree' | 'no_profile' | 'no_run' | 'no_verified_run' | 'tree_mismatch' | 'no_tests' - | 'inventory_failed' | 'inventory_empty' | 'inventory_uncovered' | 'run_changed' | 'tree_changed' | 'api_error'; + | 'inventory_failed' | 'inventory_empty' | 'inventory_uncovered' | 'inventory_skipped' | 'run_changed' | 'tree_changed' | 'api_error'; export type CarryRefused = { ok: false; reason: CarryRefusal; detail: string }; -export type CarryVerdict = { ok: true; runs: RunEvidence[]; tests: number; receipt: string } | CarryRefused; +/** + * `tests` never claims a whole run's count for a narrow lane: with an inventory it is the number of the lane's inventory files + * the runs' tree.json covers (`unit: 'files'`; tree.json has no per-file test counts yet), with named jobs `n/a`, and only + * without either (the whole run carried) the runs' test count. `remainder`: inventory files every run skipped entirely + * (tree.json `skipped_files`), which the caller must run here with the lane's per_file runner before the lane counts. + */ +export type CarryVerdict = { ok: true; runs: RunEvidence[]; tests: number | 'n/a'; unit?: 'files'; notOwed?: string; remainder?: string[]; receipt: string } | CarryRefused; /** How many of HEAD's runs per workflow are tried, newest first. */ const CANDIDATES = 10; @@ -38,12 +46,13 @@ const no = (reason: CarryRefusal, detail: string): CarryRefused => ({ ok: false, const newestFirst = (a: ApiRun, b: ApiRun) => (b.created_at ?? '').localeCompare(a.created_at ?? '') || b.id - a.id; /** The newest run of `profile` on `head` that verifies; a refusal lists each candidate's reason, newest first. */ -function newestVerified(src: RunSource, name: string, profile: RunProfile, head: string): { ok: true; evidence: RunEvidence } | CarryRefused { +function newestVerified(src: RunSource, name: string, profile: RunProfile, head: string, jobs: string[]): { ok: true; evidence: RunEvidence } | CarryRefused { + if (jobs.length) profile = { ...profile, jobs: [...new Set([...profile.jobs, ...jobs])] }; const listed = src.deps.runsForHead(profile.workflow, head).filter(r => profile.events.includes(r.event)).sort(newestFirst).slice(0, CANDIDATES); if (!listed.length) return no('no_run', `no ${profile.workflow} run (${profile.events.join(', ')}) on ${head.slice(0, 12)} yet`); const refusals: string[] = []; for (const run of listed) { - const v = verifyRun(src.deps, { repo: src.repo, runRef: String(run.id), profileName: name, profile, vetoHeads: [head], now: src.now }); + const v = verifyRun(src.deps, { repo: src.repo, runRef: String(run.id), profileName: name, profile, vetoHeads: [head], now: src.now, unchanged: src.unchanged }); if (v.ok) return v; refusals.push(`run ${run.id}: ${v.reason}`); } @@ -52,9 +61,13 @@ function newestVerified(src: RunSource, name: string, profile: RunProfile, head: /** * The CI evidence for HEAD's tree under `profiles` (one verified run each), or why there is none. `wtree` reads the - * working-tree fingerprint (before the lookup and after it); `inventory` lists the files the runs must cover. + * working-tree fingerprint (before the lookup and after it); `inventory` lists the files the runs must cover, or `jobs` + * names the jobs that must have succeeded in each run's newest attempt (a lane with no test files; no test count needed). */ -export function findCarry(src: RunSource, o: { profiles: string[]; head: string; tree: string; wtree: () => string | null; inventory?: () => string[] | { error: string } }): CarryVerdict { +export function findCarry(src: RunSource, o: { profiles: string[]; head: string; tree: string; wtree: () => string | null; inventory?: () => string[] | { error: string }; jobs?: string[]; perFile?: boolean }): CarryVerdict { + const jobs = o.jobs ?? []; + let covered: number | undefined; + let remainder: string[] = []; const before = o.wtree(); if (before !== o.tree) return no('dirty_tree', `the working tree (${before ? before.slice(0, 12) : 'fingerprint unavailable'}) is not HEAD's committed tree ${o.tree.slice(0, 12)}; only a clean, committed tree carries`); if (!o.profiles.length) return no('no_profile', 'no run profile is named'); @@ -63,20 +76,24 @@ export function findCarry(src: RunSource, o: { profiles: string[]; head: string; for (const name of o.profiles) { const profile = src.profiles[name]; if (!profile) return no('no_profile', `${name} is not a run_profiles name on the base branch`); - const v = newestVerified(src, name, profile, o.head); + const v = newestVerified(src, name, profile, o.head, jobs); if (!v.ok) return v; runs.push(v.evidence); } const other = runs.find(e => e.tree !== o.tree); if (other) return no('tree_mismatch', `run ${other.run_id} tested tree ${other.tree.slice(0, 12)}, not ${o.tree.slice(0, 12)}`); - const empty = runs.find(e => !e.files.length || e.tests === 0); + const empty = jobs.length ? undefined : runs.find(e => !e.files.length || e.tests === 0); if (empty) return no('no_tests', `run ${empty.run_id}'s tree.json ${empty.files.length ? 'records 0 tests' : 'lists no test files'}`); if (o.inventory) { const listed = o.inventory(); if ('error' in listed) return no('inventory_failed', listed.error); if (!listed.length) return no('inventory_empty', 'the inventory command printed no files'); - const missing = uncoveredFiles(listed, runs); - if (missing.length) return no('inventory_uncovered', `${missing.length} of ${listed.length} inventory file(s) are in no run's tree.json: ${missing.slice(0, 5).join(', ')}${missing.length > 5 ? ', …' : ''}`); + const absent = uncoveredFiles(listed, runs, 'covered'); + const some = (fs: string[]) => `${fs.slice(0, 5).join(', ')}${fs.length > 5 ? ', …' : ''}`; + if (absent.length) return no('inventory_uncovered', `${absent.length} of ${listed.length} inventory file(s) are in no run's tree.json: ${some(absent)}`); + remainder = uncoveredFiles(listed, runs, 'uncovered'); + if (remainder.length && !o.perFile) return no('inventory_skipped', `${remainder.length} of ${listed.length} inventory file(s) skipped every test in CI and the lane declares no per_file runner: ${some(remainder)}`); + covered = new Set(listed.map(normalPath)).size - remainder.length; } for (const e of runs) { const changed = runChanged(src.deps, e.run_id, e.run_attempt); @@ -87,6 +104,13 @@ export function findCarry(src: RunSource, o: { profiles: string[]; head: string; } const after = o.wtree(); if (after !== o.tree) return no('tree_changed', `the working tree changed while the runs were verified (now ${after ? after.slice(0, 12) : 'unreadable'})`); + if (jobs.length) { + const gates = [...new Set(o.profiles.map(n => src.profiles[n]?.skippable?.gate).filter(Boolean))]; + const allSkipped = runs.every(e => jobs.every(j => e.skipped_jobs?.includes(j))); + const notOwed = allSkipped ? `CI skipped at this tree (gate job ${gates.join(', ')}, workflow unchanged)` : undefined; + return { ok: true, runs, tests: 'n/a', ...(notOwed ? { notOwed } : {}), receipt: `ci-carry tree=${o.tree.slice(0, 12)} jobs=${jobs.join(',')}${notOwed ? ` not_owed=${gates.join(',')}` : ''} ${runReceiptFields(runs, src.now)}` }; + } + if (covered !== undefined) return { ok: true, runs, tests: covered, unit: 'files', ...(remainder.length ? { remainder } : {}), receipt: `ci-carry tree=${o.tree.slice(0, 12)} files=${covered}${remainder.length ? ` skipped_here=${remainder.length}` : ''} ${runReceiptFields(runs, src.now)}` }; const tests = runs.reduce((n, e) => n + (e.tests ?? e.files.length), 0); return { ok: true, runs, tests, receipt: `ci-carry tree=${o.tree.slice(0, 12)} tests=${tests} ${runReceiptFields(runs, src.now)}` }; } @@ -109,24 +133,75 @@ export async function waitForRun(deps: RunDeps, o: { workflow: string; events: s } } -/** `ship_lanes_from` (base branch only): exact lane command → the profiles whose runs carry it and the command listing its test files. */ -export interface ShipLaneMapping { profiles: string[]; inventory: string } +/** + * `ship_lanes_from` (base branch only): exact lane command → the profiles whose runs carry it and either the command listing + * its test files (`inventory`) or the CI jobs that run it (`jobs`, for a lane with no test files such as a typecheck; GST-1189). + */ +export interface ShipLaneMapping { profiles: string[]; inventory?: string; jobs?: string[]; aliases?: string[]; per_file?: string } + +/** + * The `ship_lanes_from` entry for an exact lane command: the entry it keys, else the entry that declares it in `aliases` + * (GST-1190: the same lane run under another exact spelling, such as an env prefix). Never normalized: an alias is a + * second exact command the base's owner declared, not a pattern. + */ +export function laneMapping(from: Record, command: string): ShipLaneMapping | undefined { + if (Object.prototype.hasOwnProperty.call(from, command)) return from[command]; + return Object.values(from).find(m => m.aliases?.includes(command)); +} + +/** findCarry's coverage rule for one mapping: its inventory (a pin-file command run in `cwd`, with a per_file runner or not) or its named jobs. */ +export const mappingCoverage = (m: ShipLaneMapping, cwd: string): { inventory?: () => string[] | { error: string }; jobs?: string[]; perFile?: boolean } => + m.jobs ? { jobs: m.jobs } : { inventory: () => runInventory(m.inventory!, cwd), perFile: !!m.per_file }; + +/** + * What `gstack-evidence record-ci` reported, read from its own last record line onward (the per_file remainder's output comes + * first, so a test printing look-alike lines cannot change the reading): the carried runs, count, unit and remainder, any + * not-owed reason, or the last CARRY_NONE reason. + */ +export function recordCiResult(stderr: string): { runs?: string; tests?: string; files?: boolean; notOwed?: string; here?: string; none?: string } { + const lines = stderr.split('\n'); + const at = lines.findLastIndex(x => x.startsWith('gstack-evidence: recorded label=') && x.includes(' log=- source=ci runs=')); + const none = lines.findLast(x => x.startsWith('CARRY_NONE: '))?.slice('CARRY_NONE: '.length); + if (at < 0) return none ? { none } : {}; + const got = / tests_ran=(\d+|n\/a)( unit=files)? log=- source=ci runs=(\S+)/.exec(lines[at]!); + const here = / skipped_here=(\d+) tests_ran_here=(\S+)$/.exec(lines[at]!); + const notOwed = lines.slice(at + 1).find(x => x.startsWith('gstack-evidence: not owed: '))?.slice('gstack-evidence: not owed: '.length); + return { runs: got?.[3], tests: got?.[1], files: !!got?.[2], ...(notOwed ? { notOwed } : {}), ...(here ? { here: `skipped_here=${here[1]} file(s) tests_ran_here=${here[2]}` } : {}) }; +} + +/** The per_file runner command for `files`: `{files}` replaced by the paths, each single-quoted for bash. */ +export const perFileCommand = (template: string, files: string[]): string => + template.replaceAll('{files}', files.map(f => `'${f.replace(/'/g, `'\\''`)}'`).join(' ')); /** `: ` errors for `ship_lanes_from`; empty when valid. Shared with lib/pregate/config.ts. */ export function shipLanesErrors(raw: unknown, profileNames: readonly string[]): Array<{ path: string; message: string }> { if (raw === undefined) return []; - if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return [{ path: 'ship_lanes_from', message: 'must be an object of exact lane command → { profiles: [run_profiles names], inventory: command }' }]; + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return [{ path: 'ship_lanes_from', message: 'must be an object of exact lane command → { profiles: [run_profiles names], inventory: command } or { profiles, jobs: [job names] }' }]; const errors: Array<{ path: string; message: string }> = []; const oneLine = (s: unknown) => typeof s === 'string' && !!s.trim() && s.length <= 512 && !/[\0\r\n]/.test(s); + const declared = new Map(Object.keys(raw as object).map(k => [k, 'a key'])); for (const [command, m] of Object.entries(raw as Record)) { const at = `ship_lanes_from[${JSON.stringify(command)}]`; if (!oneLine(command)) errors.push({ path: at, message: 'the key must be the exact lane command: one line under 512 characters' }); - if (!m || typeof m !== 'object' || Array.isArray(m)) { errors.push({ path: at, message: 'must be { profiles, inventory }' }); continue; } + if (!m || typeof m !== 'object' || Array.isArray(m)) { errors.push({ path: at, message: 'must be { profiles, inventory } or { profiles, jobs }' }); continue; } const o = m as Record; - for (const k of Object.keys(o)) if (k !== 'profiles' && k !== 'inventory') errors.push({ path: `${at}.${k}`, message: 'unknown key; valid keys: profiles, inventory' }); + for (const k of Object.keys(o)) if (!['profiles', 'inventory', 'jobs', 'aliases', 'per_file'].includes(k)) errors.push({ path: `${at}.${k}`, message: 'unknown key; valid keys: profiles, inventory, jobs, aliases, per_file' }); + if ('per_file' in o && (!oneLine(o.per_file) || !(o.per_file as string).includes('{files}') || !('inventory' in o))) errors.push({ path: `${at}.per_file`, message: 'must be one command line under 512 characters containing {files}, on an entry with an inventory (it runs the inventory files CI skipped entirely)' }); + if ('aliases' in o) { + const aliases = o.aliases; + if (!Array.isArray(aliases) || !aliases.length || aliases.some(a => !oneLine(a)) || new Set(aliases).size !== aliases.length) errors.push({ path: `${at}.aliases`, message: 'must be a non-empty array of distinct exact lane commands, each one line under 512 characters' }); + else for (const a of aliases as string[]) { + if (declared.has(a)) errors.push({ path: `${at}.aliases`, message: `${JSON.stringify(a)} is already ${declared.get(a)}; each exact command maps to one entry` }); + else declared.set(a, `an alias of ${JSON.stringify(command)}`); + } + } if (!Array.isArray(o.profiles) || !o.profiles.length || o.profiles.some(p => typeof p !== 'string')) errors.push({ path: `${at}.profiles`, message: 'must be a non-empty array of run_profiles names' }); else for (const p of o.profiles as string[]) if (!profileNames.includes(p)) errors.push({ path: `${at}.profiles`, message: `${JSON.stringify(p)} is not a run_profiles name` }); - if (!oneLine(o.inventory)) errors.push({ path: `${at}.inventory`, message: 'must be one command line under 512 characters that prints the lane\'s test files, one per line' }); + if (('inventory' in o) === ('jobs' in o)) errors.push({ path: at, message: 'declare exactly one of inventory (a command listing the lane\'s test files) or jobs (the CI jobs that run it)' }); + else if ('jobs' in o) { + const jobs = o.jobs; + if (!Array.isArray(jobs) || !jobs.length || jobs.some(j => typeof j !== 'string' || !oneLine(j) || j.length > 128) || new Set(jobs).size !== jobs.length) errors.push({ path: `${at}.jobs`, message: 'must be a non-empty array of distinct CI job names, each one line under 128 characters' }); + } else if (!oneLine(o.inventory)) errors.push({ path: `${at}.inventory`, message: 'must be one command line under 512 characters that prints the lane\'s test files, one per line' }); } return errors; } diff --git a/lib/pregate/checks/guards.ts b/lib/pregate/checks/guards.ts index 1a67ff05a..c8d40feb0 100644 --- a/lib/pregate/checks/guards.ts +++ b/lib/pregate/checks/guards.ts @@ -11,9 +11,17 @@ * incident: a gbrain serial test landed without its manifest row and * passed both the isolation check and pregate; `test/cli-goldens.test.ts` * spawns the CLI as a subprocess, so no import graph selects it. + * + * GST-1186: a guard whose exact command already passed at this tree is not run + * again. #3901's carry (gstack-evidence record-ci) carries it from a verified + * CI run when the base's ship_lanes_from maps the command, else a counted + * ledger pass of the same command at the same working tree stands + * (gstack-evidence passed). ship_lanes_from is read from origin/ only, so + * the PR that adds a mapping cannot carry from it yet. */ import * as fs from 'node:fs'; import * as path from 'node:path'; +import { laneMapping, recordCiResult } from '../../ci-carry'; import { matchesAny } from '../../ship-policy'; import type { Guard } from '../config'; import { baseInputs, sha256 } from '../context'; @@ -23,8 +31,29 @@ import { checkScratch } from '../run-scratch'; import { readRange, STALL_IDLE, supervise } from '../supervise'; import type { EnvironmentCause } from '../types'; import type { CheckDef, CheckResult, PregateContext } from '../types'; +import { runScriptSync } from '../../script-invocation'; -interface GuardRun { guard: Guard; exit: number | null; timedOut: boolean; stalled: boolean; tail: string[]; log: string; environment: EnvironmentCause | null } +interface GuardRun { guard: Guard; exit: number | null; timedOut: boolean; stalled: boolean; tail: string[]; log: string; environment: EnvironmentCause | null; prior?: string; carryNone?: string } + +const EVIDENCE = path.resolve(import.meta.dir, '..', '..', '..', 'bin', 'gstack-evidence'); + +/** + * The pass that already covers this guard's exact command at this tree (GST-1186): a CI carry (or a CI skip the unchanged gate + * decided, printed as not owed), else a same-tree local ledger pass under the runtime the guard would launch with. + */ +function priorPass(ctx: PregateContext, guard: Guard, env: NodeJS.ProcessEnv): { prior?: string; carryNone?: string } { + const evidence = (args: string[]) => runScriptSync(EVIDENCE, args, { cwd: ctx.repoRoot, env, encoding: 'utf8', timeout: 600_000 }); + let carryNone: string | undefined; + if (laneMapping(ctx.config.ship_lanes_from, guard.command)) { + const ci = evidence(['record-ci', '--label', `guard-${guard.id}`, '--base', ctx.base, '--', guard.command]); + const got = recordCiResult(String(ci.stderr ?? '')); + if (ci.status === 0 && got.runs) return { prior: got.notOwed ? `not owed: ${got.notOwed} runs=${got.runs}` : `carried runs=${got.runs}` }; + carryNone = got.none ?? `record-ci exit ${ci.status}`; + } + const local = evidence(['passed', '--', guard.command]); + const hit = /^PASSED: (.+)$/m.exec(String(local.stdout ?? ''))?.[1]; + return local.status === 0 && hit ? { prior: `reused ${hit}`, carryNone } : { carryNone }; +} async function runGuard(ctx: PregateContext, guard: Guard, env: NodeJS.ProcessEnv, dir: string): Promise { const timeoutMs = guard.timeout_s ? guard.timeout_s * 1000 : ctx.timeoutMs; @@ -53,9 +82,14 @@ async function run(ctx: PregateContext): Promise { if (runtime.refusal) return { ...base, ...runtime.refusal, cause: 'environment', environment: launchCause('runtime', runtime.refusal.detail), lines: [`runtime: ${runtime.label}`, environmentLine(launchCause('runtime', runtime.refusal.detail))] }; const dir = checkScratch(ctx.scratch, 'guards'); const runs: GuardRun[] = []; - for (const g of selected) runs.push(await runGuard(ctx, g, runtime.env, dir)); + for (const g of selected) { + const p = priorPass(ctx, g, runtime.env); + runs.push(p.prior ? { guard: g, exit: 0, timedOut: false, stalled: false, tail: [], log: '-', environment: null, ...p } : { ...(await runGuard(ctx, g, runtime.env, dir)), carryNone: p.carryNone }); + } const lines = [`runtime: ${runtime.label}`]; for (const r of runs) { + if (r.carryNone) lines.push(`CI_CARRY: none guard=${r.guard.id} (${r.carryNone})`); + if (r.prior) { lines.push(`GUARD: ${r.guard.id} ${r.prior}`); continue; } lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`); if (r.exit !== 0) lines.push(...r.tail.map(l => ` ${l}`)); if (r.environment) lines.push(` ${r.guard.id} ${environmentLine(r.environment)}`); @@ -68,7 +102,8 @@ async function run(ctx: PregateContext): Promise { if (failed.length) return { ...withRuns, status: 'fail', cause: 'diff', code: 'PREGATE_GUARD_FAILED', detail: `${failed.length} guard(s) failed: ${failed.map(r => r.guard.id).join(', ')}${timedOut.length ? `; timed out: ${timedOut.map(r => r.guard.id).join(', ')}` : ''}${blocked.length ? `; blocked by the environment: ${blocked.map(r => r.guard.id).join(', ')}` : ''}`, fix: fixOf(failed) }; if (blocked.length) return environmentResult(withRuns, blocked[0]!.environment!, `${blocked.length} guard(s) could not run on this machine: ${blocked.map(r => `${r.guard.id} (${r.environment!.kind})`).join(', ')}${timedOut.length ? `; timed out: ${timedOut.map(r => r.guard.id).join(', ')}` : ''}`); if (timedOut.length) return { ...withRuns, status: 'incomplete', code: timedOut.some(r => r.stalled) ? 'PREGATE_STALLED' : 'PREGATE_INCOMPLETE', detail: `${timedOut.length} guard(s) did not finish: ${timedOut.map(r => `${r.guard.id} (${r.stalled ? 'stalled' : 'timed out'})`).join(', ')}`, fix: 'fix the hang, or raise the guard\'s timeout_s, --timeout or --stall-timeout; an unfinished guard never passes' }; - return { ...withRuns, detail: `${runs.length} guard(s) passed: ${runs.map(r => r.guard.id).join(', ')}` }; + const prior = runs.filter(r => r.prior).length; + return { ...withRuns, detail: `${runs.length} guard(s) passed: ${runs.map(r => r.guard.id).join(', ')}${prior ? ` (${prior} from an earlier pass at this tree)` : ''}` }; } export const guardsCheck: CheckDef = { id: 'guards', stage: 'preflight', tier: 2, downgradable: false, needsRepoCommands: true, run }; diff --git a/lib/pregate/checks/lanes.ts b/lib/pregate/checks/lanes.ts index a16f4b0ed..a0ff1f85a 100644 --- a/lib/pregate/checks/lanes.ts +++ b/lib/pregate/checks/lanes.ts @@ -40,7 +40,7 @@ import { selectLanes, type Selection } from '../lane-selection'; import { launchRuntime } from '../runtime'; import { cargoModel, type CargoModel } from '../rust'; import { adapterFor, readRange, STALL_IDLE, supervise, type Counter, type Supervised } from '../supervise'; -import { verifyFromRun, type RunProfile } from '../../run-evidence'; +import { ranFiles, verifyFromRun, type RunProfile } from '../../run-evidence'; import { matchesAny } from '../../ship-policy'; import type { CheckDef, CheckResult, PregateContext, RemoteObligation } from '../types'; @@ -643,9 +643,10 @@ function fromRun(ctx: PregateContext, sel: Selection): { line: string; passed: s const names = 'error' in check ? [] : fromRunProfiles(ctx, check.profiles); const v = 'error' in check ? { ok: false as const, reason: 'profiles_unavailable', detail: check.error } : verifyFromRun(check, ref, names, ctx.wtree, ctx.head); if (!v.ok) return { line: `REUSE: refused reason=${v.reason} — ${v.detail}; every selected test runs here`, passed: [] }; - const ran = new Set(v.evidence.files); + const ran = new Set(ranFiles(v.evidence)); const passed = sel.localTests.filter(t => ran.has(t)); - return { line: `REUSE: accepted ${passed.length} of ${sel.localTests.length} selected local test file(s) from ${v.receipt}; ${sel.localTests.length - passed.length} still run`, passed, receipt: v.receipt }; + const skipped = sel.localTests.filter(t => v.evidence.skipped?.includes(t)).length; + return { line: `REUSE: accepted ${passed.length} of ${sel.localTests.length} selected local test file(s) from ${v.receipt}; ${sel.localTests.length - passed.length} still run${skipped ? ` (${skipped} skipped every test in the run)` : ''}`, passed, receipt: v.receipt }; } /** diff --git a/lib/pregate/config.ts b/lib/pregate/config.ts index 9212f6acf..1797338a8 100644 --- a/lib/pregate/config.ts +++ b/lib/pregate/config.ts @@ -46,9 +46,13 @@ * tree, the command listing the gate's test files they must * cover, and the command naming the selected files the gate * itself never runs (gstack: paid files) - * ship_lanes_from exact /ship lane command → `{ profiles, inventory }`: the - * run profiles whose verified runs at HEAD's tree carry that - * lane, and the command listing its test files (lib/ci-carry.ts) + * ship_lanes_from exact /ship lane command → `{ profiles, inventory }` or + * `{ profiles, jobs }`: the run profiles whose verified runs + * at HEAD's tree carry that lane, and either the command + * listing its test files or the CI jobs that run it (a lane + * with no test files, GST-1189), plus optional `aliases`: + * other exact commands for the same lane (GST-1190) + * (lib/ci-carry.ts) * allowlist `{ finding, paths?, pattern?, reason }` entries the `secrets` * check accepts as benign: a gstack-redact finding id or a * non-secret category, scoped by path globs and/or a regex @@ -74,7 +78,8 @@ import { shipLanesErrors, type ShipLaneMapping } from '../ci-carry'; import type { ResultCodeName } from '../result-codes'; import { runProfileErrors, type RemoteGateRuns, type RunProfile } from '../run-evidence'; import { sealedErrors } from '../sealed-guard'; -import { containedRelative, globToRegExp } from '../ship-policy'; +import { containedRelative, globToRegExp, runningGstackVersion, SHIP_POLICY_PATH } from '../ship-policy'; +import { cmpVersion, parseVersion } from '../version-source'; export const PREGATE_CONFIG_PATH = '.gstack/pregate.json'; export const PREGATE_CONFIG_SCHEMA = 'https://github.com/garrytan/gstack/docs/pregate.md#pregate-v1'; @@ -291,6 +296,22 @@ function git(cwd: string, args: string[]): { ok: boolean; out: string } { return { ok: r.status === 0 && !r.error, out: (r.stdout ?? '').trim() }; } +/** + * GST-1190: an unknown key in a policy whose project floor (min_gstack_version in the same tree's ship-policy.json) is + * above this gstack belongs to a newer gstack. It still refuses, but as an upgrade, not as a typo. + */ +function newerGstackErrors(parsed: { config: PregateConfig; errors: ConfigError[]; declaresLanes: boolean }, policyText: string | null): typeof parsed { + const unknown = (e: ConfigError) => e.message.startsWith('unknown key'); + if (!policyText || !parsed.errors.some(unknown)) return parsed; + let min: unknown; + try { min = (JSON.parse(policyText) as Record | null)?.min_gstack_version; } catch { return parsed; } + const running = runningGstackVersion(); + const want = typeof min === 'string' ? parseVersion(min) : null; + const have = parseVersion(running); + if (!want || !have || cmpVersion(want, have) <= 0) return parsed; + return { ...parsed, errors: parsed.errors.map(e => unknown(e) ? { ...e, code: 'PREGATE_GSTACK_TOO_OLD' as const, message: `needs gstack >= ${min as string}; upgrade (this gstack is ${running} and does not know this key)` } : e) }; +} + export function loadConfigFromBase(repoRoot: string, base: string, knownChecks: readonly string[] = []): ConfigLoad { const ref = `origin/${base.replace(/^origin\//, '')}`; const sha = git(repoRoot, ['rev-parse', '--verify', '--quiet', `${ref}^{commit}`]); @@ -302,13 +323,15 @@ export function loadConfigFromBase(repoRoot: string, base: string, knownChecks: if (!entry.ok || entry.out) return { unavailable: true, base: base.replace(/^origin\//, ''), detail: `${ref}:${PREGATE_CONFIG_PATH} could not be read and git cannot confirm it is absent` }; return { config: DEFAULT_CONFIG, errors: [], present: false, label: `${ref}@${sha.out.slice(0, 12)} (no ${PREGATE_CONFIG_PATH})`, kind: 'none' }; } - return { ...parseConfig(show.out, knownChecks), present: true, label: `${ref}@${sha.out.slice(0, 12)}:${PREGATE_CONFIG_PATH}`, kind: 'base' }; + const policy = git(repoRoot, ['show', `${sha.out}:${SHIP_POLICY_PATH}`]); + return { ...newerGstackErrors(parseConfig(show.out, knownChecks), policy.ok ? policy.out : null), present: true, label: `${ref}@${sha.out.slice(0, 12)}:${PREGATE_CONFIG_PATH}`, kind: 'base' }; } export function loadConfigFromHead(repoRoot: string, knownChecks: readonly string[] = []): ConfigLoad { const file = path.join(repoRoot, PREGATE_CONFIG_PATH); if (!fs.existsSync(file)) return { config: DEFAULT_CONFIG, errors: [], present: false, label: `head (no ${PREGATE_CONFIG_PATH})`, kind: 'none' }; - return { ...parseConfig(fs.readFileSync(file, 'utf8'), knownChecks), present: true, label: `head (unreviewed) ${PREGATE_CONFIG_PATH}`, kind: 'head' }; + const policy = path.join(repoRoot, SHIP_POLICY_PATH); + return { ...newerGstackErrors(parseConfig(fs.readFileSync(file, 'utf8'), knownChecks), fs.existsSync(policy) ? fs.readFileSync(policy, 'utf8') : null), present: true, label: `head (unreviewed) ${PREGATE_CONFIG_PATH}`, kind: 'head' }; } /** A symlink to nothing on the way to `file` (a dotfiles link not yet cloned): a policy the user meant to have, not an absent one. */ @@ -365,7 +388,7 @@ export function explainConfig(): string { 'remote_gate: the repo\'s remote test gate command; under 8 cores the test tier becomes a requires-remote obligation naming it, and gstack-pregate --remote runs it (it must print PREGATE_REMOTE: tree= tests= failed=) (docs/pregate.md#remote-gate)', 'run_profiles: name → { workflow, events, jobs, artifact, max_age_hours?, os? }; the GitHub Actions runs gstack-pregate clear --run verifies as evidence on the identical tree (docs/pregate.md#run-evidence)', 'remote_gate_runs: { profiles, inventory, skipped? }; the run profiles that together clear the remote gate obligation, the command listing the gate\'s test files they must cover, and an optional command printing which selected files (PREGATE_FILES_FILE) the gate itself does not run, which then need no run (docs/pregate.md#run-evidence)', - 'ship_lanes_from: exact /ship lane command → { profiles, inventory }; a verified run of each profile at HEAD\'s clean tree whose tree.json covers the inventory carries the lane instead of running it again (docs/pregate.md#ci-carry)', + 'ship_lanes_from: exact /ship lane command → { profiles, inventory } or { profiles, jobs }, with optional aliases (other exact commands for the same lane); a verified run of each profile at HEAD\'s clean tree whose tree.json covers the inventory, or whose newest attempt ran every named job to success, carries the lane instead of running it again (docs/pregate.md#ci-carry)', `allowlist: [{ finding, paths?, pattern?, reason }]: benign findings the secrets check accepts and names as allowlisted. finding is a gstack-redact id (env.kv, google.api_key, pii.phone.e164, …) or a category (${ALLOWLISTABLE_CATEGORIES.join(', ')}); paths are globs; pattern is a regex matched at the finding's position in the added line; one of the two is required, reason always. Never ${NON_ALLOWLISTABLE.join(' or ')} or engine.* (docs/pregate.md#allowlist)`, 'no .gstack/pregate.json on the base: /projects//pregate.json is read instead (state root: GSTACK_STATE_ROOT, else GSTACK_HOME, else GSTACK_STATE_DIR, else a gstack plugin data dir, else ~/.gstack, per docs/state-root.md; run --explain prints the resolved path; outside the repo, so a PR cannot reach it; a path inside the checkout is ignored)', ].join('\n') + '\n'; diff --git a/lib/pregate/index.ts b/lib/pregate/index.ts index fda383c78..73ab1ba2f 100644 --- a/lib/pregate/index.ts +++ b/lib/pregate/index.ts @@ -267,7 +267,7 @@ function userPolicyChange(file: PregateFile): string | null { export interface ClearInput { runUrl?: string; bundleLane?: string; receiptPath?: string; runs?: string[]; evidence?: RunCheck; flaky?: FlakyInput } /** What a run clear needs: the base branch's run profiles, the verifier's dependencies, the clock, and the remote gate's inventory. */ -export interface RunCheck { repo: string; deps: RunDeps; profiles: Record; gateRuns: RemoteGateRuns | null; now: Date; inventory: (command: string, tree: string, selected?: string[]) => string[] | { error: string }; failures?: FailureDeps } +export interface RunCheck { repo: string; deps: RunDeps; profiles: Record; gateRuns: RemoteGateRuns | null; now: Date; unchanged?: (p: RunProfile) => string | null; inventory: (command: string, tree: string, selected?: string[]) => string[] | { error: string }; failures?: FailureDeps } export type ClearDecision = { receipt: string; attested: boolean; substitution?: PregateReceiptSubstitution; exception?: PregateException } | { refused: 'usage' | 'PREGATE_RECEIPT_INVALID'; message: string; fix?: string }; /** diff --git a/lib/pregate/run-check.ts b/lib/pregate/run-check.ts index 34db52967..b3c795fda 100644 --- a/lib/pregate/run-check.ts +++ b/lib/pregate/run-check.ts @@ -11,20 +11,58 @@ import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; -import { ghArtifactDeps, ghRunDeps, runInventory } from '../run-evidence'; +import { ghArtifactDeps, ghRunDeps, runInventory, type RunProfile } from '../run-evidence'; import { wtreeFingerprint } from '../script-invocation'; import { resolveBaseBranch } from '../ship-policy'; import { normalizeRemoteUrl } from '../ship-queue-train-record'; import { REMOTE_GATE_LANE } from './checks/lanes'; -import { PREGATE_CONFIG_PATH, isConfigUnavailable, loadConfigFromBase } from './config'; +import { PREGATE_CONFIG_PATH, isConfigUnavailable, loadConfigFromBase, loadConfigFromHead } from './config'; import type { RunCheck } from './index'; /** - * `profiled` says whether `lane` may have a run profile: an unreadable base counts as profiled, so a lane is - * recorded unverified only once the base shows it has none. + * Null when a skippable profile's workflow, its declared `scripts` and every repo path its gate job names are byte-identical + * at HEAD and origin/ (GST-1190); else the first that differs. Named paths are read from the gate job's block in the base + * workflow (after `${{ … }}` prefixes, and also under each working-directory), plus package.json when the job runs a package + * script. Files the gate reaches indirectly (imports, a package script's own files) count only when `scripts` declares them. */ -export function buildRunCheck(root: string, lane: string | null, allowRepoCommands: boolean, base = resolveBaseBranch(root)): RunCheck | { error: string; profiled?: boolean } { - const loaded = loadConfigFromBase(root, base); +export function gateInputsChanged(root: string, base: string, p: RunProfile): string | null { + const ref = `origin/${base.replace(/^origin\//, '')}`; + const wf = `.github/workflows/${p.workflow}`; + const gate = p.skippable?.gate; + if (!gate) return 'the profile names no gate job'; + const blob = (rev: string, f: string) => { const r = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${rev}:${f}`], { cwd: root, encoding: 'utf8', timeout: 30_000 }); return r.status === 0 ? r.stdout.trim() : null; }; + const show = spawnSync('git', ['show', `${ref}:${wf}`], { cwd: root, encoding: 'utf8', timeout: 30_000, maxBuffer: 8 * 1024 * 1024 }); + if (show.status !== 0) return `${wf} is not on ${ref}`; + const lines = show.stdout.split('\n'); + const jobsAt = lines.findIndex(l => /^jobs:\s*$/.test(l)); + const blocks: string[][] = []; + for (let i = jobsAt + 1; jobsAt >= 0 && i < lines.length && !/^\S/.test(lines[i]!); i++) { + if (/^ [\w-]+:\s*$/.test(lines[i]!)) blocks.push([lines[i]!]); + else blocks.at(-1)?.push(lines[i]!); + } + const block = blocks.find(b => b[0]!.trim() === `${gate}:` || b.some(l => /^ name:/.test(l) && l.trim() === `name: ${gate}`)); + if (!block) return `${wf} on ${ref} has no job ${gate}`; + const text = block.join('\n'); + const dirs = ['', ...[...text.matchAll(/working-directory:\s*['"]?([\w.\/-]+)/g)].map(m => `${m[1]!.replace(/^\.?\/+|\/+$/g, '')}/`)]; + const tokens = [...text.matchAll(/[\w.\/-]+/g)].map(m => m[0]).filter(t => t.includes('/') || /\.(sh|ts|js|mjs|cjs|py|rb)$/.test(t)); + const named = [...new Set(tokens.flatMap(t => dirs.map(d => path.posix.normalize(d + t.replace(/^\.?\/+/, ''))).filter(n => !n.startsWith('..'))))]; + const pkg = /\b(bun|npm|pnpm|yarn)\s+(run\s+)?[\w:-]+/.test(text) ? dirs.map(d => `${d}package.json`) : []; + for (const f of [wf, ...(p.skippable!.scripts ?? []), ...named, ...pkg]) { + const a = blob(ref, f); + const b = blob('HEAD', f); + if (a === null && b === null && f !== wf && !p.skippable!.scripts?.includes(f)) continue; + if (a === null || a !== b) return `${f} differs from ${ref}`; + } + return null; +} + +/** + * `profiled` says whether `lane` may have a run profile: an unreadable base counts as profiled, so a lane is + * recorded unverified only once the base shows it has none. `policy: 'head'` reads the branch's profiles instead, only + * for gstack-ship-preflight --simulate, a dry run that records and clears nothing. + */ +export function buildRunCheck(root: string, lane: string | null, allowRepoCommands: boolean, base = resolveBaseBranch(root), policy: 'base' | 'head' = 'base'): RunCheck | { error: string; profiled?: boolean } { + const loaded = policy === 'head' ? loadConfigFromHead(root) : loadConfigFromBase(root, base); if (isConfigUnavailable(loaded)) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is unreadable (${loaded.detail}); run profiles come only from the base`, profiled: true }; if (loaded.errors.length) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is invalid: ${loaded.errors.map(e => `${e.path}: ${e.message}`).join('; ')}`, profiled: true }; const origin = spawnSync('git', ['config', '--get', 'remote.origin.url'], { cwd: root, encoding: 'utf8', timeout: 30_000 }); @@ -32,6 +70,7 @@ export function buildRunCheck(root: string, lane: string | null, allowRepoComman if (!repo) return { error: 'origin is not a github.com repository; run evidence is read from GitHub Actions', profiled: lane === null ? false : lane === REMOTE_GATE_LANE ? !!loaded.config.remote_gate_runs : !!loaded.config.run_profiles[lane] }; return { repo, deps: ghRunDeps(repo, root), failures: ghArtifactDeps(repo), profiles: loaded.config.run_profiles, gateRuns: loaded.config.remote_gate_runs, now: new Date(), + unchanged: p => gateInputsChanged(root, base, p), inventory: (command, tree, selected) => { if (!allowRepoCommands) return { error: 'the inventory command is a pin-file command; pass --allow-repo-commands' }; const at = () => { const fp = wtreeFingerprint({ cwd: root }); return 'tree' in fp ? fp.tree : null; }; diff --git a/lib/restamp.ts b/lib/restamp.ts index be2c6fc23..9eba5f88e 100644 --- a/lib/restamp.ts +++ b/lib/restamp.ts @@ -313,7 +313,7 @@ export function branchAddedLines(cwd: string, since: string, files: string[], to return out; } -const escapeRegExp = (t: string) => t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +export const escapeRegExp = (t: string) => t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); const VERSION_RE = '(?\\d+(?:\\.\\d+){1,3})(?![0-9.])'; /** * The built-in stamp forms as scanners, the version in the `v` group: `(was X)`, `since: X` and `vX`. diff --git a/lib/result-codes-release.ts b/lib/result-codes-release.ts index 2d0a0dddb..6c209131a 100644 --- a/lib/result-codes-release.ts +++ b/lib/result-codes-release.ts @@ -16,6 +16,11 @@ export const RELEASE_RESULT_CODES = { summary: 'the ship policy sets min_gstack_version above this gstack, so this gstack refuses to read it', fix: 'upgrade gstack (bin/gstack-upgrade, or the session install with --for ) to the version named, then rerun', }, + PREGATE_GSTACK_TOO_OLD: { + anchor: 'pregate-gstack-too-old', + summary: 'the pregate policy uses a key this gstack does not know, and the project floor (min_gstack_version in .gstack/ship-policy.json) is above this gstack, so the key belongs to a newer gstack; the policy stays refused', + fix: 'upgrade gstack (bin/gstack-upgrade, or the session install with --for ) to at least the version named, then rerun', + }, QUEUE_NOT_READY: { anchor: 'queue-not-ready', summary: 'the lane is not ready to report: the working tree, HEAD or CI state does not match the pushed PR head', diff --git a/lib/run-evidence.ts b/lib/run-evidence.ts index 73ecbb215..135fec844 100644 --- a/lib/run-evidence.ts +++ b/lib/run-evidence.ts @@ -42,7 +42,11 @@ import * as path from 'node:path'; import { sha256Hex, writeReceipt } from './egress-receipt'; import { runBashSync } from './script-invocation'; -export interface RunProfile { workflow: string; events: string[]; jobs: string[]; artifact: string; max_age_hours?: number; os?: string } +/** + * `skippable` (GST-1190): named jobs the workflow may skip by path; a skip counts only in a pull_request run whose `gate` job + * succeeded, with the workflow, the declared `scripts` and any path the gate's steps name matching origin/ byte for byte. + */ +export interface RunProfile { workflow: string; events: string[]; jobs: string[]; artifact: string; max_age_hours?: number; os?: string; skippable?: { jobs: string[]; gate: string; scripts: string[] } } export interface RemoteGateRuns { profiles: string[]; inventory: string; skipped?: string } export interface ApiRun { id: number; run_attempt: number; status: string; conclusion: string | null; event: string; head_sha: string; head_branch?: string | null; @@ -50,7 +54,7 @@ export interface ApiRun { repository?: { full_name: string } | null; head_repository?: { full_name: string } | null; } export interface ApiJob { name: string; conclusion: string | null; completed_at?: string | null } -export interface TreeRecord { run_id?: unknown; run_attempt?: unknown; event?: unknown; sha?: unknown; head_sha?: unknown; tree?: unknown; reused?: unknown; reuse?: unknown; sizing_experiment?: unknown; os?: unknown; files?: unknown; tests?: unknown } +export interface TreeRecord { run_id?: unknown; run_attempt?: unknown; event?: unknown; sha?: unknown; head_sha?: unknown; tree?: unknown; reused?: unknown; reuse?: unknown; sizing_experiment?: unknown; os?: unknown; files?: unknown; tests?: unknown; skipped_files?: unknown } /** What a red run's reader needs beyond RunDeps (lib/pregate/flaky-exception.ts): every file of one artifact, and a pull request's head. */ export interface RunArtifactDeps { @@ -69,14 +73,15 @@ export interface RunDeps { } export type RunRefusal = 'run_ref_invalid' | 'other_repository' | 'fork_run' | 'workflow_mismatch' | 'event_mismatch' | 'run_in_progress' - | 'run_not_success' | 'job_missing' | 'job_ambiguous' | 'job_not_success' | 'artifact_missing' | 'artifact_mismatch' + | 'run_not_success' | 'job_missing' | 'job_ambiguous' | 'job_skipped' | 'job_not_success' | 'artifact_missing' | 'artifact_mismatch' | 'artifact_reused' | 'sizing_experiment' | 'tree_mismatch' | 'newer_run_vetoes' | 'run_changed' | 'run_stale' | 'os_mismatch' | 'api_error'; -export interface RunEvidence { run_id: number; run_attempt: number; head_sha: string; tree: string; profile: string; event: string; finished_at: string; files: string[]; tests: number | null } +/** `skipped`: the files whose every test skipped on the runner (tree.json `skipped_files`, GST-1190); undefined when an older tree.json does not say. */ +export interface RunEvidence { run_id: number; run_attempt: number; head_sha: string; tree: string; profile: string; event: string; finished_at: string; files: string[]; tests: number | null; skipped?: string[]; skipped_jobs?: string[] } export type RunVerdict = { ok: true; evidence: RunEvidence } | { ok: false; reason: RunRefusal; detail: string }; const SHA = /^[0-9a-f]{40}$/; const RUN_EVENTS = ['push', 'workflow_dispatch', 'pull_request']; -const PROFILE_KEYS = ['workflow', 'events', 'jobs', 'artifact', 'max_age_hours', 'os']; +const PROFILE_KEYS = ['workflow', 'events', 'jobs', 'artifact', 'max_age_hours', 'os', 'skippable']; const WORKFLOW_FILE = /^[A-Za-z0-9_.-]+\.ya?ml$/; const NAME = /^[A-Za-z0-9_.\/-]{1,128}$/; export const DEFAULT_MAX_AGE_HOURS = 24; @@ -113,6 +118,14 @@ export function runProfileErrors(profiles: unknown, gateRuns: unknown): Array<{ if (typeof p.artifact !== 'string' || !NAME.test(p.artifact)) errors.push({ path: `${at}.artifact`, message: 'must be the artifact name holding tree.json' }); if ('max_age_hours' in p && (!Number.isInteger(p.max_age_hours) || (p.max_age_hours as number) < 1 || (p.max_age_hours as number) > 168)) errors.push({ path: `${at}.max_age_hours`, message: 'must be an integer from 1 to 168' }); if ('os' in p && (typeof p.os !== 'string' || !/^(Linux|macOS|Windows)$/.test(p.os))) errors.push({ path: `${at}.os`, message: 'must be Linux, macOS or Windows' }); + if ('skippable' in p) { + const s = p.skippable as Record | null; + const names = (v: unknown) => Array.isArray(v) && v.length > 0 && v.length <= 32 && v.every(x => typeof x === 'string' && !!x.trim() && x.length <= 128); + const repoPath = (x: unknown) => typeof x === 'string' && !!x && !x.startsWith('/') && !x.split('/').includes('..') && !/[\0\r\n]/.test(x); + if (!s || typeof s !== 'object' || Array.isArray(s) || Object.keys(s).some(k => !['jobs', 'gate', 'scripts'].includes(k)) || !names(s.jobs) || typeof s.gate !== 'string' || !s.gate.trim() + || (s.jobs as string[]).includes(s.gate as string) || !Array.isArray(s.scripts) || !s.scripts.length || !s.scripts.every(repoPath)) + errors.push({ path: `${at}.skippable`, message: 'must be { jobs: [job names], gate: , scripts: [every repo file the gate reads] }, the gate not among the jobs' }); + } } } if (gateRuns !== undefined && gateRuns !== null) { @@ -135,13 +148,13 @@ export function runProfileErrors(profiles: unknown, gateRuns: unknown): Array<{ const fail = (reason: RunRefusal, detail: string): RunVerdict => ({ ok: false, reason, detail }); /** When a run's newest attempt started. A run vetoes the candidate when it started at or after the second the candidate was created: a red re-run of an older run counts, and re-running the candidate does not hide a failure in between. */ const started = (r: ApiRun): string => r.run_started_at ?? r.created_at ?? ''; -const normalPath = (f: string): string => f.replace(/^(\.\/)+/, ''); +export const normalPath = (f: string): string => f.replace(/^(\.\/)+/, ''); /** * The verdict for run `runRef` under `profile`. `vetoHeads` are further heads * whose newer runs of the same workflow can veto (the consumer's own head). */ -export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; profileName: string; profile: RunProfile; vetoHeads?: string[]; ignoreRun?: number; now: Date }): RunVerdict { +export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; profileName: string; profile: RunProfile; vetoHeads?: string[]; ignoreRun?: number; now: Date; unchanged?: (p: RunProfile) => string | null }): RunVerdict { const id = parseRunRef(o.runRef, o.repo); if (id === null) return fail('run_ref_invalid', `${JSON.stringify(o.runRef)} is not a run id or a run URL on ${o.repo}`); const p = o.profile; @@ -162,10 +175,22 @@ export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; prof const maxAge = p.max_age_hours ?? DEFAULT_MAX_AGE_HOURS; if (!Number.isFinite(finished)) return fail('api_error', `${at} has no completion time`); if (o.now.getTime() - finished > maxAge * 3600_000) return fail('run_stale', `${at} finished ${finishedAt}, more than ${maxAge} h before now`); + const skippedJobs: string[] = []; for (const name of p.jobs) { const hits = jobs.filter(j => j.name === name); if (hits.length === 0) return fail('job_missing', `${at} has no job named ${name}`); if (hits.length > 1) return fail('job_ambiguous', `${at} has ${hits.length} jobs named ${name}`); + if (hits[0]!.conclusion === 'skipped') { + const skip = p.skippable; + if (!skip?.jobs.includes(name)) return fail('job_skipped', `${at}'s ${name} job was skipped`); + if (run.event !== 'pull_request') return fail('job_skipped', `${at}'s ${name} job was skipped in a ${run.event} run; only a pull_request run's gate diffs against the PR base, so only its skip counts`); + const gate = jobs.filter(j => j.name === skip.gate); + if (gate.length !== 1 || gate[0]!.conclusion !== 'success') return fail('job_skipped', `${at}'s ${name} job was skipped and its gate job ${skip.gate} ${gate.length === 1 ? `concluded ${gate[0]!.conclusion}` : gate.length ? 'is ambiguous' : 'is missing'}`); + const changed = o.unchanged ? o.unchanged(p) : 'no base copy of the workflow was compared'; + if (changed) return fail('job_skipped', `${at}'s ${name} job was skipped, but ${changed}, so this branch decided the skip`); + skippedJobs.push(name); + continue; + } if (hits[0]!.conclusion !== 'success') return fail('job_not_success', `${at}'s ${name} job concluded ${hits[0]!.conclusion}`); } const heads = [...new Set([run.head_sha, ...(o.vetoHeads ?? []).filter(h => SHA.test(h))])]; @@ -196,11 +221,13 @@ export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; prof if (p.os && !oses.includes(p.os)) return fail('os_mismatch', `${at} ran on ${oses.filter(Boolean).map(String).join(', ') || 'an unrecorded OS'}; ${o.profileName} requires ${p.os}`); if (rec.files !== undefined && (!Array.isArray(rec.files) || rec.files.some(f => typeof f !== 'string'))) return fail('artifact_mismatch', `${at}'s ${p.artifact} files is not a list of paths`); if (rec.tests !== undefined && (!Number.isInteger(rec.tests) || (rec.tests as number) < 0)) return fail('artifact_mismatch', `${at}'s ${p.artifact} tests is not a non-negative integer`); + if (rec.skipped_files !== undefined && (!Array.isArray(rec.skipped_files) || rec.skipped_files.some(f => typeof f !== 'string'))) return fail('artifact_mismatch', `${at}'s ${p.artifact} skipped_files is not a list of paths`); const files = ((rec.files ?? []) as string[]).map(normalPath); const tests = rec.tests === undefined ? null : (rec.tests as number); const changed = runChanged(deps, id, run.run_attempt); if (changed) return fail('run_changed', changed); - return { ok: true, evidence: { run_id: id, run_attempt: run.run_attempt, head_sha: run.head_sha, tree: headTree, profile: o.profileName, event: run.event, finished_at: finishedAt!, files, tests } }; + const skipped = rec.skipped_files === undefined ? {} : { skipped: (rec.skipped_files as string[]).map(normalPath) }; + return { ok: true, evidence: { run_id: id, run_attempt: run.run_attempt, head_sha: run.head_sha, tree: headTree, profile: o.profileName, event: run.event, finished_at: finishedAt!, files, tests, ...skipped, ...(skippedJobs.length ? { skipped_jobs: skippedJobs } : {}) } }; } catch (e: any) { return fail('api_error', `reading run ${id} failed: ${String(e?.message ?? e).slice(0, 300)}`); } @@ -220,13 +247,21 @@ export function runInventory(command: string, cwd: string, env?: Record e.files.map(normalPath))); +/** A run's files minus the ones whose every test skipped there (GST-1190): only those ran anything. Older tree.json: all files. */ +export const ranFiles = (e: RunEvidence): string[] => { const skipped = new Set(e.skipped ?? []); return e.files.map(normalPath).filter(f => !skipped.has(f)); }; + +/** + * The inventory files no run covers. `skipped: 'uncovered'` (a carry, which drops tests on a machine where they might run) + * also counts a file every test of which skipped in the run; `'covered'` (the remote gate, a Linux runner like CI) does not. + */ +export function uncoveredFiles(inventory: string[], evidence: RunEvidence[], skipped: 'uncovered' | 'covered' = 'covered'): string[] { + const ran = new Set(evidence.flatMap(e => (skipped === 'uncovered' ? ranFiles(e) : e.files.map(normalPath)))); return inventory.filter(f => !ran.has(normalPath(f))); } /** What a run check reads: the repository, the verifier's dependencies, the base branch's profiles and the clock. */ -export interface RunSource { repo: string; deps: RunDeps; profiles: Record; now: Date } +/** `unchanged` (findCarry only): null when a profile's workflow and gate inputs match the base, else which file differs. */ +export interface RunSource { repo: string; deps: RunDeps; profiles: Record; now: Date; unchanged?: (p: RunProfile) => string | null } export type FromRunVerdict = { ok: true; evidence: RunEvidence; receipt: string } | { ok: false; reason: string; detail: string }; /** diff --git a/lib/ship-bin-common.ts b/lib/ship-bin-common.ts index 51bb7d167..c76e51887 100644 --- a/lib/ship-bin-common.ts +++ b/lib/ship-bin-common.ts @@ -10,6 +10,7 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { fragmentSlug } from './changelog-fragments'; +import { laneMapping, recordCiResult, type ShipLaneMapping } from './ci-carry'; import { isConfigUnavailable, loadConfigFromBase } from './pregate/config'; import { parseUnifiedDiff, versionAtRef } from './restamp'; import { branchSides, stampOnlyLine } from './stamp-only'; @@ -113,7 +114,8 @@ export function releasePaths(policy: Record | null, cwd: string, re * added since the merge base with origin/, and not added by the base since): after a lost slot and a merge of the * base its stale stamp is older than `ref`'s VERSION, so it must be at or newer than the fork point's version instead, * the bound gstack-restamp's stamp_paths stage uses (GST-895; the fork version itself since GST-963). A branch line is - * named by its position, and with a kept VERSION only such a line may move (GST-979, GST-980). + * named by its position, and with a kept VERSION only such a line may move (GST-979, GST-980); on it a declared + * `stamp_placeholders` value may also become the new version inside a `stamp_patterns` template (GST-1026). */ export function stampedFiles(policy: Record | null, cwd: string, ref: string, base: string): string[] { const globs = Array.isArray(policy?.stamp_files) ? policy.stamp_files.filter((s: unknown): s is string => typeof s === 'string') : []; @@ -123,11 +125,13 @@ export function stampedFiles(policy: Record | null, cwd: string, re let after = ''; try { after = extractVersion(fs.readFileSync(path.join(cwd, before.rel), 'utf8'), before.rel); } catch { return []; } if (!after) return []; - const templates = Array.isArray(policy?.stamp_patterns) ? policy.stamp_patterns.filter((s: unknown): s is string => typeof s === 'string') : []; + const strings = (v: unknown): string[] => (Array.isArray(v) ? v.filter((s): s is string => typeof s === 'string') : []); + const templates = strings(policy?.stamp_patterns); + const placeholders = strings(policy?.stamp_placeholders); const diff = git(cwd, ['diff', '--unified=0', '--no-color', '--no-ext-diff', '--no-textconv', '--no-renames', ref]); if (diff.status !== 0) return []; const side = branchSides(cwd, `origin/${base}`, 'HEAD', before.version, ref); - const stamp = (rel: string, removed: string, added: string, at: number): boolean => stampOnlyLine(removed, added, before.version, after, templates, side(rel), at); + const stamp = (rel: string, removed: string, added: string, at: number): boolean => stampOnlyLine(removed, added, before.version, after, templates, side(rel), at, placeholders); return parseUnifiedDiff(diff.stdout).filter(f => f.status === 'M' && !f.binary && !f.modeChanged && matchesAny(globs, f.path) && f.hunks.length > 0 && f.hunks.every(h => h.removed.length === h.added.length && h.removed.every((line, i) => stamp(f.path, line, h.added[i]!, h.removedAt[i]!)))).map(f => f.path); } @@ -171,15 +175,15 @@ export function laneConcurrency(mode: string | undefined, cpus = os.availablePar return cpus <= 4 ? 1 : Infinity; } -export interface LaneResult { label: string; exit: number | null; testsRan: string; log: string; carried?: string; reused?: string; carryNone?: string } +export interface LaneResult { label: string; exit: number | null; testsRan: string; log: string; carried?: string; reused?: string; carryNone?: string; notOwed?: string; here?: string } -/** The base branch's `ship_lanes_from` entry for each lane whose command is a key, byte for byte (the ledger binds its sha256). */ -export function mappedLanes(cwd: string, base: string, list: { label: string; command: string }[]): Map { +/** The base branch's `ship_lanes_from` entry for each lane whose command is a key or a declared alias, byte for byte (the ledger binds its sha256). */ +export function mappedLanes(cwd: string, base: string, list: { label: string; command: string }[]): Map { const loaded = loadConfigFromBase(cwd, base); - const map = new Map(); + const map = new Map(); if (isConfigUnavailable(loaded) || loaded.errors.length) return map; const from = loaded.config.ship_lanes_from; - for (const l of list) if (Object.prototype.hasOwnProperty.call(from, l.command)) map.set(l.label, from[l.command]!); + for (const l of list) { const m = laneMapping(from, l.command); if (m) map.set(l.label, m); } return map; } @@ -189,9 +193,9 @@ export function mappedLanes(cwd: string, base: string, list: { label: string; co */ export function carryLane(cwd: string, base: string, l: { label: string; command: string }): LaneResult { const rec = bin(cwd, 'gstack-evidence', ['record-ci', '--label', l.label, '--base', base, '--', l.command], 600_000); - const got = / tests_ran=(\d+) log=- source=ci runs=(\S+)/.exec(rec.stderr); - if (rec.status === 0 && got) return { label: l.label, exit: 0, testsRan: got[1]!, log: '-', carried: got[2]! }; - return { label: l.label, exit: null, testsRan: 'unknown', log: '-', carryNone: /^CARRY_NONE: (.+)$/m.exec(rec.stderr)?.[1] ?? `record_failed: ${firstLine(rec.stderr)}` }; + const got = recordCiResult(rec.stderr); + if (rec.status === 0 && got.runs) return { label: l.label, exit: 0, testsRan: got.files ? `${got.tests} (files)` : got.tests!, log: '-', carried: got.runs, ...(got.notOwed ? { notOwed: got.notOwed } : {}), ...(got.here ? { here: got.here } : {}) }; + return { label: l.label, exit: null, testsRan: 'unknown', log: '-', carryNone: got.none ?? `record_failed: ${firstLine(rec.stderr)}` }; } /** @@ -201,7 +205,7 @@ export function carryLane(cwd: string, base: string, l: { label: string; command */ export async function runLanes(cwd: string, list: { label: string; command: string }[], concurrency = Infinity, opts: { base?: string; reuse?: boolean } = {}): Promise { const out: LaneResult[] = []; - const mapped = opts.base ? mappedLanes(cwd, opts.base, list) : new Map(); + const mapped = opts.base ? mappedLanes(cwd, opts.base, list) : new Map(); let next = 0; const worker = async () => { while (next < list.length) { @@ -222,7 +226,8 @@ export async function runLanes(cwd: string, list: { label: string; command: stri /** Step 5's lane verdict: a lane passes on its runner count, never exit 0 alone; a carried lane names its CI runs. */ export function laneLine(l: LaneResult): { line: string; ok: boolean } { - if (l.carried) return { line: `LANE: ${l.label} carried run=${l.carried} tests_ran=${l.testsRan}`, ok: true }; + if (l.notOwed) return { line: `LANE: ${l.label} not owed: ${l.notOwed} run=${l.carried}`, ok: true }; + if (l.carried) return { line: `LANE: ${l.label} carried run=${l.carried} tests_ran=${l.testsRan}${l.here ? ` ${l.here}` : ''}`, ok: true }; const zero = l.exit === 0 && l.testsRan === '0'; const state = l.exit !== 0 ? 'FAIL' : zero ? 'ZERO-RUN' : l.testsRan === 'unknown' ? 'pass (count unavailable)' : 'pass'; return { line: `LANE: ${l.label} ${state} exit=${l.exit} tests_ran=${l.testsRan} log=${l.log}${l.reused ? ` reused=${l.reused}` : ''}`, ok: l.exit === 0 && !zero }; diff --git a/lib/ship-policy-stamps.ts b/lib/ship-policy-stamps.ts new file mode 100644 index 000000000..6e0b144da --- /dev/null +++ b/lib/ship-policy-stamps.ts @@ -0,0 +1,35 @@ +/** + * Validation of the stamp policy keys (stamp_patterns templates and stamp_placeholders), split out of ship-policy.ts + * to keep that module under the module-size ratchet. + */ +import type { PolicyError } from './ship-policy'; +import { parseVersion } from './version-source'; + +const MAX_TEMPLATES = 16; +const MAX_TEMPLATE = 128; +const MAX_PLACEHOLDERS = 16; +const MAX_PLACEHOLDER = 32; +export const VERSION_SLOT = '{version}'; + +export function templateErrors(value: unknown, out: PolicyError[]): void { + if (!Array.isArray(value)) { out.push({ code: 'POLICY_INVALID', path: 'stamp_patterns', message: 'must be an array of literal templates' }); return; } + if (value.length > MAX_TEMPLATES) out.push({ code: 'POLICY_CONTAINMENT', path: 'stamp_patterns', message: `more than ${MAX_TEMPLATES} templates` }); + value.forEach((t, i) => { + const at = `stamp_patterns[${i}]`; + if (typeof t !== 'string') { out.push({ code: 'POLICY_INVALID', path: at, message: 'must be a string' }); return; } + if (t.split(VERSION_SLOT).length !== 2) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(t)} must contain exactly one ${VERSION_SLOT}` }); + else if (t.length > MAX_TEMPLATE || /[\0\r\n]/.test(t)) out.push({ code: 'POLICY_CONTAINMENT', path: at, message: `must be one line of at most ${MAX_TEMPLATE} characters` }); + }); +} + +export function placeholderErrors(value: unknown, out: PolicyError[]): void { + if (!Array.isArray(value)) { out.push({ code: 'POLICY_INVALID', path: 'stamp_placeholders', message: 'must be an array of literal placeholders' }); return; } + if (value.length > MAX_PLACEHOLDERS) out.push({ code: 'POLICY_CONTAINMENT', path: 'stamp_placeholders', message: `more than ${MAX_PLACEHOLDERS} placeholders` }); + value.forEach((p, i) => { + const at = `stamp_placeholders[${i}]`; + if (typeof p !== 'string') { out.push({ code: 'POLICY_INVALID', path: at, message: 'must be a string' }); return; } + if (!p.trim() || p.length > MAX_PLACEHOLDER || /[\0\r\n]/.test(p)) out.push({ code: 'POLICY_CONTAINMENT', path: at, message: `must be one non-empty line of at most ${MAX_PLACEHOLDER} characters` }); + else if (p.includes(VERSION_SLOT)) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(p)} must not contain ${VERSION_SLOT}` }); + else if (parseVersion(p)) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(p)} is a version, not a placeholder` }); + }); +} diff --git a/lib/ship-policy.ts b/lib/ship-policy.ts index 0092b3be4..3fd85545c 100644 --- a/lib/ship-policy.ts +++ b/lib/ship-policy.ts @@ -18,6 +18,8 @@ import * as path from 'node:path'; import { defaultBaseBranch, noDefaultBaseLine } from './default-base'; import { RESULT_CODES, type ResultCodeName } from './result-codes'; import { cmpVersion, parseVersion } from './version-source'; +import { placeholderErrors, templateErrors } from './ship-policy-stamps'; +export { VERSION_SLOT } from './ship-policy-stamps'; export const SHIP_POLICY_PATH = '.gstack/ship-policy.json'; export const SHIP_POLICY_SCHEMA = 'https://github.com/garrytan/gstack/docs/ship-policy.md#schema-v1'; @@ -64,6 +66,8 @@ export interface ShipPolicy { stamp_files: string[]; /** Literal stamp templates with one `{version}` slot, beside the built-in `(was X)`, `since: X`, `vX`; writer and classifier share them. */ stamp_patterns: string[]; + /** Placeholders (`TBD`) a restamp may replace with the new version inside a `stamp_patterns` template, on the branch's own lines only. */ + stamp_placeholders: string[]; /** The repo's own restamp, run first in the staging worktree with the target version; gstack's version writer is skipped. */ restamp_command: string | null; /** Migration numbering: `{ dir, id }` where `id` is a regex with one capture group over the file name. */ @@ -161,6 +165,7 @@ export const DEFAULT_POLICY: ShipPolicy = { maintainers: [], stamp_files: [], stamp_patterns: [], + stamp_placeholders: [], restamp_command: null, migrations: null, gate_command: null, @@ -193,6 +198,7 @@ export const POLICY_EXPLANATIONS: Record = { maintainers: 'GitHub logins treated as maintainers by /review contributor mode; empty uses repository push permission.', stamp_files: 'The repo’s stamp rule: globs of the only files that may differ by a stamp (version file, manifests, plugin trees, the PR’s own since: rows). Empty keeps gstack’s per-file rules; set, it applies before every rule.', stamp_patterns: 'Literal templates with exactly one {version} slot (at most 16, 128 characters each), for example `"version": "{version}"` or `since: \'{version}\'`; the restamp writer and the tree receipt use them beside (was X), since: X and vX.', + stamp_placeholders: 'Literal placeholders (at most 16, one line, 32 characters each, never `{version}` or a version), for example `["TBD"]`: a line the branch added that reads a placeholder in a stamp_patterns template (`since: \'TBD\'`) may become that template at the new version, and finish, the receipt carry and the tree receipt read it as a stamp. Empty (default) accepts no placeholder.', restamp_command: 'The repo’s own restamp, run first in gstack-restamp’s staging worktree (a clean checkout of HEAD) with the target version as its argument and in GSTACK_RESTAMP_VERSION; gstack’s version writer is skipped, output confinement still applies. Needs a clean caller tree and --allow-repo-commands.', migrations: '{ dir, id }: the migration directory and a regex with one capture group for the number in a file name; `gstack-pregate regate` reports MIGRATION_COLLISION against the base branch and every open PR.', gate_command: 'The repo’s full pre-merge gate command. gstack-ship-queue gate-ahead and ready print it for the caller to run (in the background or in CI); gstack never executes it. null when the repo has none.', @@ -236,6 +242,7 @@ export const BUMP_NEVER_KEY_VERSION = '1.91.353.0'; export const BASE_RED_KEY_VERSION = '1.91.374.0'; /** `train.gates` and `train.land` are values of the older train key, so compatErrors checks them by value, as for bump: never. */ export const TRAIN_GATES_KEY_VERSION = '1.92.0.0'; +export const STAMP_PLACEHOLDERS_KEY_VERSION = '1.94.16.0'; export const REVIEW_EXCLUDE_KEY_VERSION = '1.93.33.0'; export const POLICY_KEY_INTRODUCED: Record = { bump: '1.91.73.0', queue_mode: '1.91.73.0', release_tool: '1.91.73.0', release_outputs: '1.91.73.0', stamp_paths: '1.91.73.0', @@ -245,6 +252,7 @@ export const POLICY_KEY_INTRODUCED: Record = { train: TRAIN_KEYS_VERSION, review_scope: REVIEW_SCOPE_KEY_VERSION, docs: DOCS_CAPS_KEY_VERSION, changelog_fragments: CHANGELOG_FRAGMENTS_KEY_VERSION, base_red: BASE_RED_KEY_VERSION, review_exclude_paths: REVIEW_EXCLUDE_KEY_VERSION, + stamp_placeholders: STAMP_PLACEHOLDERS_KEY_VERSION, }; const newer = (a: string, b: string): boolean => cmpVersion(parseVersion(a)!, parseVersion(b)!) > 0; @@ -274,9 +282,6 @@ const MAX_PATTERN = 256; const MAX_REGEX = 512; const MAX_COMMAND = 512; const SHA_RE = /^[0-9a-f]{7,64}$/; -const MAX_TEMPLATES = 16; -const MAX_TEMPLATE = 128; -export const VERSION_SLOT = '{version}'; function hasDotDot(p: string): boolean { return p.split(/[\\/]+/).some(seg => seg === '..'); @@ -343,17 +348,6 @@ function excludeErrors(value: unknown, out: PolicyError[]): void { }); } -function templateErrors(value: unknown, out: PolicyError[]): void { - if (!Array.isArray(value)) { out.push({ code: 'POLICY_INVALID', path: 'stamp_patterns', message: 'must be an array of literal templates' }); return; } - if (value.length > MAX_TEMPLATES) out.push({ code: 'POLICY_CONTAINMENT', path: 'stamp_patterns', message: `more than ${MAX_TEMPLATES} templates` }); - value.forEach((t, i) => { - const at = `stamp_patterns[${i}]`; - if (typeof t !== 'string') { out.push({ code: 'POLICY_INVALID', path: at, message: 'must be a string' }); return; } - if (t.split(VERSION_SLOT).length !== 2) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(t)} must contain exactly one ${VERSION_SLOT}` }); - else if (t.length > MAX_TEMPLATE || /[\0\r\n]/.test(t)) out.push({ code: 'POLICY_CONTAINMENT', path: at, message: `must be one line of at most ${MAX_TEMPLATE} characters` }); - }); -} - function migrationErrors(value: unknown, out: PolicyError[]): void { const m = value as Record | null; if (!m || typeof m !== 'object' || Array.isArray(m)) { out.push({ code: 'POLICY_INVALID', path: 'migrations', message: 'must be { dir, id } or null' }); return; } @@ -497,6 +491,7 @@ export function validatePolicy(raw: unknown, opts: { gstackVersion?: string } = if ('release_outputs' in obj) listErrors('release_outputs', obj.release_outputs, errors, globError); if ('stamp_files' in obj) listErrors('stamp_files', obj.stamp_files, errors, globError); if ('stamp_patterns' in obj) templateErrors(obj.stamp_patterns, errors); + if ('stamp_placeholders' in obj) placeholderErrors(obj.stamp_placeholders, errors); if ('restamp_command' in obj && obj.restamp_command !== null) { const t = obj.restamp_command; if (typeof t !== 'string' || !t.trim()) errors.push({ code: 'POLICY_INVALID', path: 'restamp_command', message: 'must be a non-empty string or null' }); diff --git a/lib/ship-preflight.ts b/lib/ship-preflight.ts index ed6591ce5..2b483b308 100644 --- a/lib/ship-preflight.ts +++ b/lib/ship-preflight.ts @@ -8,8 +8,12 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; +import { findCarry, laneMapping, mappingCoverage } from './ci-carry'; import { defaultBaseBranch } from './default-base'; +import { isConfigUnavailable, loadConfigFromHead } from './pregate/config'; +import { buildRunCheck } from './pregate/run-check'; import { classifyDiff } from './review-depth'; +import { wtreeFingerprint } from './script-invocation'; import { audit } from './ship-audit'; import { bin, carryNoneLine, firstLine, flagReader, gh, git, hostedOrigin, laneConcurrency, laneLine, lanes, Report, runLanes, shipPolicy, short } from './ship-bin-common'; @@ -20,6 +24,11 @@ export const PREFLIGHT_HELP = `gstack-ship-preflight — /ship Steps 1-3 and Ste --lanes serial|parallel how Step 5 runs its lanes; unset, one at a time on 4 or fewer cores, all at once above --policy-from head Step 5's pregate reads the branch's .gstack/pregate.json, which may only add test lanes (a branch that adds the lane for a file it touches); anything else is PREGATE_HEAD_POLICY_LOOSENS + gstack-ship-preflight --tests-only --base --policy-from head --simulate [--allow-repo-commands] --lane

the single reads and Steps 7-8's deterministic audits (CONTEXT, TODOS, COVERAGE_MAP, Coverage, PLAN_*, READER: coverage=needed|none plan=needed|none) @@ -164,6 +173,40 @@ async function tests(cwd: string, base: string, list: { label: string; command: r.next('Step 6: select eval suites for prompt-related changes with the project\'s selector, then Step 7'); } +/** + * --simulate (GST-1190): HEAD's ship_lanes_from and run_profiles, findCarry against this tree, nothing else. It never calls + * gstack-evidence or gstack-pregate, so it writes no row and clears no obligation: a PR controls both its own map and the + * CI that would satisfy it, so head policy never carries. The branch's inventory commands run only with `allow` + * (--allow-repo-commands), as gstack-ci-carry find --simulate requires. + */ +function simulate(cwd: string, base: string, list: { label: string; command: string }[], allow: boolean): { text: string; exit: number } { + const lines: string[] = []; + const say = (s: string) => lines.push(`SIMULATION: ${s}`); + const done = (exit: number) => ({ text: lines.join('\n') + '\n', exit }); + const loaded = loadConfigFromHead(cwd); + say(`policy ${isConfigUnavailable(loaded) ? 'unavailable' : loaded.label}; a dry run: only the branch's inventory commands run, and only with --allow-repo-commands; no evidence or ledger row is written, nothing clears (a carry reads origin/${base}'s map only)`); + if (isConfigUnavailable(loaded) || loaded.errors.length) { say(`refused: ${isConfigUnavailable(loaded) ? loaded.detail : loaded.errors.map(e => `${e.path}: ${e.message}`).join('; ')}`); return done(1); } + const check = buildRunCheck(cwd, null, allow, base, 'head'); + if ('error' in check) { say(`refused: ${check.error}`); return done(1); } + const head = git(cwd, ['rev-parse', 'HEAD']).stdout.trim(); + const tree = git(cwd, ['rev-parse', 'HEAD^{tree}']).stdout.trim(); + const wtree = () => { const fp = wtreeFingerprint({ cwd }); return 'tree' in fp ? fp.tree : null; }; + let mapped = 0; + let carried = 0; + for (const l of list) { + const m = laneMapping(loaded.config.ship_lanes_from, l.command); + if (!m) { say(`LANE: ${l.label} not mapped by the branch's ship_lanes_from; /ship would run it`); continue; } + mapped++; + if (m.inventory && !allow) { say(`CI_CARRY: none label=${l.label} (inventory_not_allowed: the branch's inventory command is a pin-file command; pass --allow-repo-commands)`); continue; } + const v = findCarry(check, { profiles: m.profiles, head, tree, wtree, ...mappingCoverage(m, cwd) }); + if (!v.ok) { say(`CI_CARRY: none label=${l.label} (${v.reason}: ${v.detail})`); continue; } + carried++; + say(`LANE: ${l.label} would carry run=${v.runs.map(e => `${e.run_id}/${e.run_attempt}`).join(',')} tests_ran=${v.tests}${v.unit ? ` unit=${v.unit}` : ''}${v.remainder ? ` and would run ${v.remainder.length} skipped file(s) here with per_file` : ''}`); + } + say(`${carried} of ${mapped} mapped lane(s) would carry once origin/${base} holds this map`); + return done(mapped > 0 && carried === mapped ? 0 : 1); +} + export async function preflight(argv: readonly string[], cwd: string): Promise<{ text: string; exit: number }> { const { flag, has } = flagReader(argv); const base = flag('--base'); @@ -178,6 +221,7 @@ export async function preflight(argv: readonly string[], cwd: string): Promise<{ if (!['base', 'head'].includes(policyFrom)) return { text: 'gstack-ship-preflight: --policy-from needs base|head\n', exit: 2 }; if (!base || typeof list === 'string') return { text: `gstack-ship-preflight: ${typeof list === 'string' ? list : '--base is required'}\n`, exit: 2 }; if (has('--tests-only') && !list.length) return { text: 'gstack-ship-preflight: --tests-only needs at least one --lane