diff --git a/CHANGELOG.md b/CHANGELOG.md index b371c3a62..6f411c10c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,111 @@ # Changelog +## [1.94.24.0] - 2026-11-03 + +### Fixed + +- **The OpenClaw and Hermes start hook no longer breaks a fresh harness.** When the harness has no config yet, gstack runs none of its commands: `openclaw config set` and `hermes plugins enable` would have created a config holding only gstack's plugin setting, with no model, provider or terminal settings. The hook reports `written` with the setup to run first (`openclaw onboard`, `hermes setup`), and the next `./setup` or `gstack-harness refresh` turns it on, changing only the plugin entry. Turning the hook off or unregistering in such a profile also leaves the config alone. If v1.92.23.0 or a later release before this fix already created a config holding only the plugin entry, move it aside, run the harness's own setup, then `gstack-harness refresh`. +- **`OPENCLAW_CONFIG_PATH` naming the profile's own `openclaw.json` keeps the start hook on.** gstack compares the path after resolving symlinks, so only a config outside the profile turns the hook off. When `GSTACK_HARNESS_START_HOOK` or `harness_start_hook` asked for the hook and it ends up off, `gstack-harness register` and `./setup --host` now exit 4 with the reason and the fix instead of exiting 0. +- **`gstack-harness status` reads the hook from the harness's config when the harness CLI is not on `PATH`**, as inside Hermes's or OpenClaw's own terminal tool. Hermes's `config.yaml` settles it (`active`, `disabled` or `written`); otherwise status says `unknown` instead of a wrong `written`. + +## [1.94.23.0] - 2026-11-03 + +**Two `gstack-pregate` preflight false positives fixed (GST-1211).** Both stopped /ship on real work in gbrain-evals. + +### Fixed +- **`strays`:** the scratch-name check matched three prefixes the docs never listed (`temp`, `debug-` and `probe`), so real files like `probe.py` and `temporal-graph-shipped.test.ts` failed preflight. It now matches only the documented names: `zz-*`, `tmp*`, `scratch*`, plus editor leftovers (`*.orig`, `*.rej`, `*.swp`, `*.swo`, `*.bak`, `*.tmp`, `*~`, `.DS_Store`, `Thumbs.db`), which docs/pregate.md and docs/troubleshooting.md now list too. +- **`literals`:** a named literal the branch only adds, such as `limit: 300` in a new object, was read as an existing `limit: 10` elsewhere in the file changing to 300, and every test that mentioned 10 was flagged. The check now compares every value under a name, so an added literal isn't a change. Editing an existing value still fails, and so do two cases the old last-value check could miss: changing one of two equal values under the same name, and deleting one site while editing another. + +### Tests +- `test/pregate-selection.test.ts`: `probe.py` and `temporal-*.test.ts` pass `strays` while `zz-notes.md`, `tmp-x.ts`, `scratch.py` and `util.js.orig` still fail; a new object literal beside `limit: 10` passes `literals` while editing that value still fails; the pure `changedLiterals` case keeps an exported constant change. + +## [1.94.22.0] - 2026-11-03 + +**/investigate, /review, /qa and /verify now say when to use them, and /investigate and /review declare what "done" means (GST-1088 lane B).** Untold, Opus 5.5 never opened /investigate in 252 trials (Run D, gstack-evals#140). A when-to-use description raised GPT-6.1 Sol's open rate from 9.5% to 27.0% in the same run. Claude Code's catalog shows only a description's first sentence, so each of the four now opens with a "Use when ..." sentence. The routing hook and the contract check that read the new frontmatter landed in v1.94.6.0 (lane A, off by default; see docs/skill-routing-and-contracts.md), so the "Done means" list now also names `gstack-skill-contract check` and `waive`. + +### Changed +- **Descriptions**: the first sentence of each is now a when-clause, for example /investigate's "Use when a test fails, code errors or output is wrong: find the root cause, then fix it." Hosts that show the full description also get the longer list of situations. The catalog stays within its existing budget. +- **Triggers**: /investigate, /review and /verify list the prompts people actually type, such as "test fails", "look over my diff" and "make it green". /qa keeps its three triggers because its size cap had 9 bytes of room. + +### Added +- **`contract:` frontmatter** (frozen v1 schema) on /investigate (the DEBUG REPORT labels, and a test or repro run after the last edit) and /review (one Agent dispatch per selected specialist, the adversarial pass, the Pre-Landing Review report). `{{CONTRACT_DONE_MEANS}}` (`scripts/resolvers/contract-done-means.ts`) renders each block as a short "Done means" list at the top of the skill, so the list and the block can't drift. + +### Tests +- `test/skill-descriptions.test.ts`: each of the four opens with a when-clause within the catalog, lead and host caps; triggers are short and don't overlap; ten sample prompts match exactly the intended skill; every template's contract block passes the v1 schema and names markers the skill text actually prints; the renderer covers each item kind; and malformed blocks fail with the file and the fix. + +## [1.94.21.0] - 2026-11-03 + +- **A stacked PR's /ship receipt now carries even when the child merged main before its receipt was written (GST-1214).** After the parent squash-landed and main was merged in again, `gstack-ship-receipt carry` compared the range from the parent's head, which still held every file the child's first main merge brought in, and refused with `RECEIPT_CARRY_PRODUCT_CHANGED` naming those files. The carry now leaves out paths whose content at the reviewed head is main's, unless HEAD changed them. A real edit to one of those files still refuses. + +## [1.94.20.0] - 2026-11-03 + +### Changed + +- **The review pins and the harness pins are one file per pin, so PRs that repin different files no longer conflict (GST-1084).** The reviewer-brief pins that lived in one table in `test/review-packets-removed.test.ts` now live under `test/fixtures/review-pins/`: `files/.json` holds a file's sha and the replay that admits it, `replays/.json` holds one replay's written no-loss justification, and `brief-bodies/.json` holds one brief body's sha. The harness pins in `scripts/harness-version.json` moved to `scripts/harness-pins/.json`; the manifest keeps only the version and the patterns, so `HARNESS_VERSION` and its readers are unchanged. Before, two PRs that repinned adjacent entries conflicted on the shared table (at least five main-merge conflicts on 2026-11-02, two of them on #3852). Every entry is ported unchanged, including the combined GST-508 entries from #3852. + +### For contributors + +- `/ship`'s post-review route still counts a non-behavioral harness re-pin as release metadata (GST-1031): it now reads the pin files under `scripts/harness-pins/` and refuses a manifest that regains a `files` table. +- The checks are at least as strict as before. Every pin still needs its sha and a replay whose written justification names a no-loss replay or re-score, and the brief-body check still guards the bodies. A malformed or misnamed pin file, a stray file in a pin directory, or a `files` block back in `scripts/harness-version.json` now fails too. New tests prove that a stale pin still fails, and that two branches repinning different, adjacent pins merge cleanly with a real `git merge-tree`, while two repins of one pin still conflict. +- `bun run scripts/bump-harness-version.ts` writes and deletes pin files itself. Repin a review file by editing its `files/` pin and adding the replay file it names. + +## [1.94.19.0] - 2026-11-03 + +### Fixed + +- **A finished /ship no longer blocks `gstack-upgrade` for two hours (GST-1205).** Each `gstack-skill-start` writes a workflow lease that `gstack-upgrade` checks before moving a shared checkout. Nothing released it when the workflow ended, and on Capy every session's lease names the machine's one `capy-server`. So after a /ship finished, its own lease kept refusing the upgrade with UPGRADE_BUSY until it was 120 minutes old, unless you set `GSTACK_UPGRADE_FORCE=1`. Now `gstack-skill-start` records its `SESSION_ID` as an open run on the lease, and `gstack-skill-end` closes it with the new `gstack-lease release`. Once every run on a lease is closed, the lease is idle and blocks nothing (`gstack-lease list` shows `idle`). A run another session still has open keeps blocking. So does a holder that records no run (an older gstack, or `gstack-lease touch` without `--run`, which now leaves a `hold` entry that no release removes). Each run counts for 120 minutes from its own start, so one that never reaches `gstack-skill-end` stops blocking on time even while other sessions keep starting. A nested skill's own run covers it while it runs, but other sessions' starts no longer extend a run past its 120 minutes. + +## [1.94.18.0] - 2026-11-03 + +**The Windows-safe test subset is green again on current main.** `test/qa-report-dir.test.ts` expected `ENOTDIR` when `.gstack` is a file, but Windows reports `ENOENT` for the same mistake, so the test failed on every Windows run (windows-free-tests run 41607809007 at f6d6215). The test now accepts either code on Windows and still requires `ENOTDIR` everywhere else. With Windows checks now running by path, a red subset test would block any pull request that touches it. + +## [1.94.17.0] - 2026-11-03 + +- A pregate guard whose exact command already passed at this tree no longer runs again (GST-1186). When the base branch's `ship_lanes_from` maps the command, it is carried from the verified CI run through the same `gstack-evidence record-ci` path /ship lanes use (`GUARD: carried runs=…`); otherwise a counted local ledger pass of that command at the same working tree stands (CI rows count only through a live `record-ci` check) (`GUARD: reused …`, and the new read-only `gstack-evidence passed -- `). Repos whose suites are guards, like founderstar, stop rerunning about six minutes of suites per /ship. See docs/pregate.md#guard-carry. +- Known limit: `ship_lanes_from` is still read from `origin/` only, so the PR that adds a mapping cannot carry from it. +- A `ship_lanes_from` entry can name the CI jobs that run a lane instead of an inventory command: `{ "profiles": [...], "jobs": ["verify", "typecheck"] }`, exactly one of the two (GST-1189). It carries only when every named job succeeded in the newest attempt of the verified run at HEAD's tree, through the same `verifyRun` job check. A missing job refuses as `job_missing` and a skipped one as the new `job_skipped`. Lanes with no test files, like gbrain's verify and typecheck (run 41564913688) or harness-evals' `bunx tsc --noEmit`, can now carry, and they record `tests_ran=n/a`. Fake `check:` entries in `tree.json` can move to this form. A repo declaring `jobs` should raise its `min_gstack_version` to this release first, since older gstack rejects the key. See docs/pregate.md#ship-lanes-jobs. + +- A `ship_lanes_from` entry can declare `aliases`, other exact spellings of the same lane such as `env -u ANTHROPIC_API_KEY bun test tests/` (GST-1190). gstack never strips env prefixes on its own, since an `env -i` run is a different environment. +- A carried lane no longer reports the whole run's test count. An inventory lane reports how many of its files the run covered, labeled as a file count (alphaclaw's 46-test guards lane recorded 9677 before). +- `gstack-ship-preflight --tests-only --policy-from head --simulate` proves a new mapping before it lands: every line starts `SIMULATION:`, no lane runs, no evidence row is written and nothing clears. An inventory-mapped lane runs the branch's inventory command, so the dry run needs `--allow-repo-commands`. A branch's own map never carries, so the first pull request that adopts a mapping pays its full lanes. +- `gstack-evidence carried`, which the ship receipt reads, lists only rows the receipt base's policy mapped, so a run against another base never lands in a receipt. +- An unknown `.gstack/pregate.json` key under a `min_gstack_version` above the running gstack now says `needs gstack >= ; upgrade` (`PREGATE_GSTACK_TOO_OLD`). It still refuses. +- docs/pregate.md#prove-a-mapping documents the proof route, the version floors, and that `tree.json` writers using Bun's JUnit must take the root `testsuite` count. + +- A test file whose every test skipped on the CI runner no longer counts as carried (GST-1190, harness-evals: 75 tests). A `tree.json` that lists such files in `skipped_files` makes the pregate tests stage run them here. A /ship lane or guard whose inventory names one carries the covered files and runs only the skipped ones here, through the `per_file` runner its `ship_lanes_from` entry declares. The CI row is written only after that run passes with a test count above zero and the carried runs still stand (`skipped_failed`, `skipped_zero_run` or `run_changed` otherwise), and an entry without `per_file` refuses as `inventory_skipped`, naming the files. A partly skipped file still counts. An older `tree.json` without the list is read as before. gstack itself changes nothing yet: its own writer and a `per_file` entry for its `bun run test` lane switch on together in a follow-up pull request, after the pull requests now queued on older gstack have landed, because an older gstack rejects the new key in main's policy. + +- A run profile can declare `skippable: { jobs, gate, scripts }` for jobs a path filter skips (GST-1190, truther #1169). A skipped job counts only in a `pull_request` run whose gate job succeeded, with the workflow, the declared `scripts` (every file the gate reads), the paths the gate job names (also under its `working-directory`) and `package.json` when it runs a package script matching `origin/` byte for byte; the lane then prints `not owed: CI skipped at this tree (gate job , workflow unchanged)`. If the branch changed any of them, it controlled the skip, so the lane runs here. +- `gstack-ci-carry find --policy-from head --simulate` is the same dry run for one lane command, every line starting `SIMULATION:`. + +### Tests +- `test/ship-ci-carry.test.ts`: in founderstar's shape (guards only, no lanes), `check-quick` carries from CI, `play-through` reuses a same-tree ledger pass, `assets` runs, and a dirty tree carries nothing. It fails without the guard carry. +- `test/ship-ci-carry.test.ts`: a lane mapped by `jobs` carries when every named job passed at the same tree with no test files, and refuses for a missing job, a skipped job, a rerun whose newest attempt skipped the job, and another tree's run. Those five fail without the change. +- `test/ship-ci-carry.test.ts` (GST-1190): an alias carries and an undeclared env prefix does not; a narrow lane reports its covered file count, not 9677; a jobs-mapped `bunx tsc --noEmit` guard carries with `n/a`; the newer-gstack message; `--simulate` writes and clears nothing and the branch's own map never carries; `carried` drops a row another base mapped. Six of these fail without the change. +- `test/ship-ci-carry.test.ts` (GST-1190 skipped files): a fully skipped inventory file stops the lane carry, an empty list or older `tree.json` carries, and the tests stage runs only the skipped selected file. Both fail without the change. +- `test/ship-ci-carry.test.ts` (path-skipped jobs and `find --simulate`): a skip under an unchanged, green gate is not owed; a changed workflow, a changed gate script or a failed gate job runs the lane here; `find --policy-from head --simulate` prints only SIMULATION lines and writes nothing. All five fail without the change. +- `test/ship-ci-carry.test.ts` (review fixes): a gate script behind `${{ github.workspace }}`, under `working-directory` or behind a package script is compared; the gate is the job, not a same-named step; only a `pull_request` run's skip counts; a skippable profile must declare `scripts`; `passed` never counts a CI row; a path-skipped guard says not owed; `--simulate` runs a branch inventory only with `--allow-repo-commands`. Eight fail without the fixes; a newer-failure veto test covers existing behavior. +- `test/ship-ci-carry.test.ts` (per-file lane carry): the covered file carries and only the skipped file runs through `per_file`, a failing or zero-test remainder, or a run that moved to a new attempt meanwhile, records no CI row and runs the lane whole, and `per_file` needs `{files}` and an inventory entry. All five fail without the change. + +## [1.94.16.0] - 2026-11-03 + +**A declared placeholder restamped to the new version is a stamp, and a carried receipt names the head's version (GST-1026, GST-1030).** gbrain writes a PR's own BEHAVIOR_CHANGES row as `since: 'TBD'` and its `release:restamp` fills in the version. Finish refused that receipt ("behavior, tests or build inputs changed since review"), and the receipt carry and the tree receipt would have refused the same row (gbrain #8444). Separately, a receipt carried after a main merge and a restamp kept the version of the head it was written at (GGST-1). + +### Added +- **`stamp_placeholders` policy key** (default `[]`, so nothing changes for a repo that does not set it): literal placeholders such as `["TBD"]`, at most 16, one line and 32 characters each, never `{version}` and never a version. It needs `min_gstack_version` at the release that introduced it. + +### Fixed +- **`lib/stamp-only.ts`:** `stampOnlyLine` takes the policy's placeholders. On a line the branch added itself (named by its position), a placeholder inside a `stamp_patterns` template may become that template at the new version, alongside any stamp-form version move on the same line. The built-in forms, a bare `{version}` template, base lines (including one whose text duplicates a branch line) and undeclared placeholders are still refused. Finish's `stampedFiles`, the receipt carry and the tree receipt share it. +- **`bin/gstack-ship-receipt` carry:** the carried receipt's `version` is HEAD's, read the way `write` reads it, instead of the version copied from the old receipt. A head with no version source keeps the version `write --version` recorded. +- **`bin/gstack-ship-receipt` carry, pin-bump PRs:** a `package.json` the branch changed as product (a dependency pin) that both main and a restamp moved since review is compared with only its `version` field masked, at all four sides. The receipt carries when nothing else in the file moved, and still refuses when the pin or any other field changed (GBEV-3, gbrain-evals #306). + +### Tests +- `test/tree-receipt-stamp-sensitive.test.ts`, `test/gstack-ship-receipt-docs.test.ts`: the shared predicate and the carry classifier accept gbrain's `since: 'TBD'` → `since: '0.60.553.0'` row and refuse an undeclared placeholder, a base row, a duplicated base copy, a built-in form and a line no template matches. +- `test/ship-rereview-triggers.test.ts`: finish routes the declared row as metadata and stops on the same controls. +- `test/restamp-mentions.test.ts`: the tree receipt and the carry accept the declared row and refuse it undeclared or on a base row; the carried receipt records the head's version after a main merge and a restamp; a version-only restamp of an EVIDENCE.md declared in `stamp_files` carries, and an undeclared one stays product; a pin-bump PR's receipt carries after a main merge and a restamp, and refuses when the pin moved since review. +- `test/gstack-ship-policy.test.ts`: the key's default, floor and validation. + +The EVIDENCE.md refusal (GST-1030's second point) reproduces on main and is already fixed by the GST-980 own-row carry this branch builds on, so this change adds only the test for it. + ## [1.94.15.0] - 2026-11-03 - **A stacked PR keeps its /ship receipt after its parent squash-merges, even when the receipt was written against main (GST-1199).** `gstack-ship-receipt carry` used to refuse with `RECEIPT_CARRY_PRODUCT_CHANGED` once the parent's hunks moved into main, forcing a full re-review of unchanged code. The carry now finds the newest reviewed commit that main already holds (`git merge-tree` into main changes no product path) and compares from there: an unchanged child carries as `stack parent landed`, base edits next to the child's hunks go to `RECEIPT_CARRY_SCOPED_REVIEW`, and any other change still refuses. A parent that never landed never counts, even if the child's merge dropped its change. diff --git a/VERSION b/VERSION index f2224bc7b..7a11af84d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.94.15.0 +1.94.24.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index d3eba6498..c74798cc8 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.94.15.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.94.24.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-ci-carry b/bin/gstack-ci-carry index c29156107..c5292f5eb 100755 --- a/bin/gstack-ci-carry +++ b/bin/gstack-ci-carry @@ -3,6 +3,7 @@ * gstack-ci-carry — CI as the one full test run (GSTA-258; docs/pregate.md#ci-carry). * * gstack-ci-carry find [--base ] [--profile ]... [--command --allow-repo-commands] [--json] + * [--policy-from head --simulate] * gstack-ci-carry wait [--base ] [--profile ] [--timeout ] [--interval ] * * find: the newest verified run of each profile (default: every profile the base branch's @@ -14,10 +15,9 @@ * carried, 3 still running or no run. Logic: lib/ci-carry.ts; the run checks are lib/run-evidence.ts. */ import { spawnSync } from 'node:child_process'; -import { findCarry, waitForRun } from '../lib/ci-carry'; -import { isConfigUnavailable, loadConfigFromBase } from '../lib/pregate/config'; +import { findCarry, laneMapping, mappingCoverage, waitForRun } from '../lib/ci-carry'; +import { isConfigUnavailable, loadConfigFromBase, loadConfigFromHead } from '../lib/pregate/config'; import { buildRunCheck } from '../lib/pregate/run-check'; -import { runInventory } from '../lib/run-evidence'; import { wtreeFingerprint } from '../lib/script-invocation'; import { resolveBaseBranch } from '../lib/ship-policy'; @@ -25,11 +25,13 @@ const HELP = `gstack-ci-carry — use a verified CI run at HEAD's exact tree ins find [--base ] [--profile ]... [--command --allow-repo-commands] [--json] CI_CARRY: accepted (exit 0) or CI_CARRY: none () (exit 1) + --policy-from head --simulate: the same check under the branch's own map and profiles, every line starting + SIMULATION:; a dry run that proves a new mapping before it lands (find never writes evidence; nothing clears) wait [--base ] [--profile ] [--timeout ] [--interval ] poll the newest run at HEAD (default 1800 s, every 20 s): CI_WAIT: green|red|running|none, then find on green; exit 0 carried, 1 red or not carried, 3 still running or no run -Profiles and ship_lanes_from come from origin/'s .gstack/pregate.json only. Only a clean, committed tree +Profiles and ship_lanes_from come from origin/'s .gstack/pregate.json only (a simulation reads the branch's). Only a clean, committed tree carries; fork runs, stale runs, a newer red or running run on HEAD and a tree.json that misses an inventory file never do. Exit 2 is usage.`; @@ -37,30 +39,35 @@ const argv = process.argv.slice(2); const cmd = argv[0]; const all = (n: string) => argv.flatMap((a, i) => (a === n && argv[i + 1] !== undefined ? [argv[i + 1]!] : [])); const flag = (n: string) => all(n).at(-1); -const out = (s: string) => process.stdout.write(`${s}\n`); +const simulate = argv.includes('--simulate'); +const out = (s: string) => process.stdout.write(`${simulate ? 'SIMULATION: ' : ''}${s}\n`); const usage = (m: string) => { process.stderr.write(`gstack-ci-carry: ${m}\n`); process.exit(2); }; if (!cmd || argv.includes('--help') || argv.includes('-h')) { out(HELP); process.exit(cmd ? 0 : 2); } if (cmd !== 'find' && cmd !== 'wait') usage(`unknown command ${cmd}; run gstack-ci-carry --help`); +const policyFrom = flag('--policy-from') ?? 'base'; +if (policyFrom !== 'base' && policyFrom !== 'head') usage('--policy-from needs base or head'); +if (simulate !== (policyFrom === 'head') || (simulate && cmd !== 'find')) usage('find --policy-from head --simulate go together: a branch\'s own map is only ever simulated, never carried'); const git = (args: string[]) => { const r = spawnSync('git', args, { encoding: 'utf8', timeout: 30_000 }); return r.status === 0 ? r.stdout.trim() : ''; }; const root = git(['rev-parse', '--show-toplevel']); if (!root) usage('not inside a git repository'); const base = resolveBaseBranch(root, flag('--base')); -const loaded = loadConfigFromBase(root, base); -if (isConfigUnavailable(loaded) || loaded.errors.length) { out(`CI_CARRY: none (profiles_unavailable: origin/${base}'s .gstack/pregate.json is unreadable or invalid)`); process.exit(1); } +const loaded = simulate ? loadConfigFromHead(root) : loadConfigFromBase(root, base); +if (simulate) out(`policy ${isConfigUnavailable(loaded) ? 'unavailable' : loaded.label}; a dry run: nothing is written and nothing clears (a carry reads origin/${base}'s map only)`); +if (isConfigUnavailable(loaded) || loaded.errors.length) { out(`CI_CARRY: none (profiles_unavailable: ${simulate ? 'the branch\'s' : `origin/${base}'s`} .gstack/pregate.json is unreadable or invalid)`); process.exit(1); } const command = flag('--command'); -const mapping = command !== undefined ? loaded.config.ship_lanes_from[command] : undefined; -if (command !== undefined && !mapping) { out(`CI_CARRY: none (not_mapped: ${JSON.stringify(command)} is not a key of origin/${base}'s ship_lanes_from)`); process.exit(1); } +const mapping = command !== undefined ? laneMapping(loaded.config.ship_lanes_from, command) : undefined; +if (command !== undefined && !mapping) { out(`CI_CARRY: none (not_mapped: ${JSON.stringify(command)} is neither a key nor a declared alias in origin/${base}'s ship_lanes_from)`); process.exit(1); } if (mapping && !argv.includes('--allow-repo-commands')) usage('--command runs the mapping\'s inventory, a pin-file command: pass --allow-repo-commands'); const profiles = all('--profile').length ? all('--profile') : mapping ? mapping.profiles : [...new Set(Object.values(loaded.config.ship_lanes_from).flatMap(m => m.profiles))]; if (!profiles.length) { out(`CI_CARRY: none (no_profile: origin/${base} maps no lane in ship_lanes_from; pass --profile )`); process.exit(1); } -const check = buildRunCheck(root, null, argv.includes('--allow-repo-commands'), base); +const check = buildRunCheck(root, null, argv.includes('--allow-repo-commands'), base, simulate ? 'head' : 'base'); if ('error' in check) { out(`CI_CARRY: none (profiles_unavailable: ${check.error})`); process.exit(1); } const head = git(['rev-parse', 'HEAD']); const tree = git(['rev-parse', 'HEAD^{tree}']); function find(): number { const wtree = () => { const fp = wtreeFingerprint({ cwd: root }); return 'tree' in fp ? fp.tree : null; }; - const v = findCarry(check as Exclude, { profiles, head, tree, wtree, ...(mapping ? { inventory: () => runInventory(mapping.inventory, root) } : {}) }); + const v = findCarry(check as Exclude, { profiles, head, tree, wtree, ...(mapping ? mappingCoverage(mapping, root) : {}) }); if (argv.includes('--json')) out(JSON.stringify(v)); else out(v.ok ? `CI_CARRY: accepted ${v.receipt}` : `CI_CARRY: none (${v.reason}: ${v.detail})`); return v.ok ? 0 : 1; diff --git a/bin/gstack-doctor-components.sh b/bin/gstack-doctor-components.sh index 23db8f80d..4ec7b9a61 100644 --- a/bin/gstack-doctor-components.sh +++ b/bin/gstack-doctor-components.sh @@ -258,6 +258,7 @@ gstack_result_anchor() { DX_CAPTURE_REDACTION_FAILED) printf '%s' 'dx-capture-redaction-failed' ;; DX_CAPTURE_INTERNAL) printf '%s' 'dx-capture-internal' ;; POLICY_GSTACK_TOO_OLD) printf '%s' 'policy-gstack-too-old' ;; + PREGATE_GSTACK_TOO_OLD) printf '%s' 'pregate-gstack-too-old' ;; QUEUE_NOT_READY) printf '%s' 'queue-not-ready' ;; QUEUE_PROOF_MISSING) printf '%s' 'queue-proof-missing' ;; QUEUE_STAMP_NOT_STAMP_ONLY) printf '%s' 'queue-stamp-not-stamp-only' ;; diff --git a/bin/gstack-evidence b/bin/gstack-evidence index 8fc3e692b..e3d2bbab1 100755 --- a/bin/gstack-evidence +++ b/bin/gstack-evidence @@ -5,7 +5,8 @@ * * gstack-evidence run [--reuse [--max-age ]] --label -- * gstack-evidence record-ci --label --base -- - * gstack-evidence carried [--json] + * gstack-evidence passed -- + * gstack-evidence carried [--base ] [--json] * gstack-evidence check [--label [--expect-cmd ]]... | --all * [--max-age ] [--allow-paths ] * gstack-evidence bundle --out [--label ]... [--require ] [--selection L=]... @@ -65,8 +66,11 @@ * `record-ci` writes the explicit CI-sourced row for a lane CI carried: it * finds and verifies the run itself (lib/ci-carry.ts) for a command the base's * ship_lanes_from maps, then records source "ci", the runs and profiles, the - * run's finish time as ts, bound to HEAD's clean tree; `carried` lists those - * rows at the current tree for the ship receipt. + * run's finish time as ts, bound to HEAD's clean tree, and the policy that + * mapped it (`origin/@:…`); `carried` lists those rows at the + * current tree for the ship receipt, only when that policy is origin/ (GST-1190: a row a run against another base wrote never + * counts). * * MACHINE-LOCAL by design: neither the ledger nor the logs are brain-synced. * A synced record citing an unsynced log would grade FRESH on a machine where @@ -90,10 +94,10 @@ import { resultLine } from "../lib/result-codes"; import { isUnavailable, loadPolicyFromBase, resolveBaseBranch } from "../lib/ship-policy"; import { versionAtRef } from "../lib/restamp"; import { bashCommandInvocation, runScriptSync, wtreeFingerprint } from "../lib/script-invocation"; -import { findCarry } from "../lib/ci-carry"; +import { findCarry, laneMapping, mappingCoverage, perFileCommand } from "../lib/ci-carry"; +import { runChanged } from "../lib/run-evidence"; import { isConfigUnavailable, loadConfigFromBase } from "../lib/pregate/config"; import { buildRunCheck } from "../lib/pregate/run-check"; -import { runInventory } from "../lib/run-evidence"; const BIN_DIR = dirname(Bun.fileURLToPath(import.meta.url)); const LOG_MAX_BYTES = 2 * 1024 * 1024; @@ -122,10 +126,17 @@ interface EvidenceRecord { source?: "ci"; runs?: string[]; profiles?: string[]; + /** The CI jobs a jobs-mapped lane carried from (record-ci, GST-1189); tests_ran is then "n/a". */ + jobs?: string[]; /** The base policy that mapped a carried lane (record-ci), as `origin/@:.gstack/pregate.json`. */ policy?: string; /** The runner's own test count (executed for bun/jest/vitest/pytest/rspec; ran with skips included for gstack's shard runner and the remote gate), 0 for ZERO-RUN, or "unknown" when no supported summary was seen. */ - tests_ran?: number | "unknown"; + tests_ran?: number | "unknown" | "n/a"; + /** "files" when a carried narrow lane's tests_ran counts its inventory files the run covered (record-ci), not tests. */ + tests_unit?: "files"; + /** Why a carried lane was not owed at all: every job it names was path-skipped by an unchanged gate (record-ci, GST-1190). */ + not_owed?: string; + skipped_here?: { label: string; files: string[]; tests_ran: number | "unknown" }; } /** Bytes of streamed output kept for the tests_ran parse (the 2MB log would truncate the summary). */ @@ -598,6 +609,36 @@ function identityDigest(): string { return sha256(JSON.stringify({ bun: version("bun"), node: version("node"), os: process.platform, arch: process.arch, image: process.env.GSTACK_RUNNER_IMAGE || null })); } +/** A local row that passed with a counted run within the max age, under this machine's identity. */ +function countedPass(r: EvidenceRecord, maxAgeHours: number): boolean { + if (r.exit !== 0 || typeof r.tests_ran !== "number" || r.tests_ran <= 0) return false; + const age = Date.now() - Date.parse(r.ts); + if (!(age >= 0 && age <= maxAgeHours * 3600_000)) return false; + return r.identity === identityDigest(); +} + +/** + * `passed -- ` (GST-1186, read-only): did this exact command pass here at the current working tree under any label? The + * newest local row for the command hash and tree decides, so a newer failure vetoes; it must be a counted pass (countedPass). + * CI rows never count: a CI pass counts only through record-ci's live check against the base. Prints PASSED: label= + * source=local ts= and exits 0, else NOT_PASSED: and exits 1. gstack-pregate's guards read it. + */ +function cmdPassed(argv: string[]): number { + const sep = argv.indexOf("--"); + const command = sep < 0 ? "" : argv.slice(sep + 1).join(" "); + if (!command || sep !== 0) { console.error("usage: gstack-evidence passed -- "); return 2; } + const wtree = currentWtree(); + if (!wtree) { console.log("NOT_PASSED: no working-tree fingerprint"); return 1; } + let records: EvidenceRecord[] = []; + try { records = readJsonl(ledgerPath().file); } catch { records = []; } + const cmdSha = sha256(command); + const newest = records.findLast((r) => r.cmd_sha256 === cmdSha && r.wtree === wtree && r.source === undefined); + if (!newest) { console.log(`NOT_PASSED: no local ledger row for this command at tree ${wtree.slice(0, 12)}`); return 1; } + if (!countedPass(newest, 24)) { console.log(`NOT_PASSED: the newest row (label=${newest.label} ts=${newest.ts}) is not a counted pass within 24 h under this runtime`); return 1; } + console.log(`PASSED: label=${newest.label} source=local ts=${newest.reused_from ?? newest.ts}`); + return 0; +} + /** * `run --reuse`: the newest local row for this label, command and working tree decides. A counted pass under the same * identity within the max age is copied (original ts kept) and printed as a run would be; anything else runs the command. @@ -612,10 +653,7 @@ function reused(file: string, label: string, cmdSha: string, maxAgeHours: number return false; } const newest = records.findLast((r) => r.label === label && r.cmd_sha256 === cmdSha && r.wtree === wtree && r.source === undefined); - if (!newest || newest.exit !== 0 || typeof newest.tests_ran !== "number" || newest.tests_ran <= 0) return false; - const age = Date.now() - Date.parse(newest.ts); - if (!(age >= 0 && age <= maxAgeHours * 3600_000)) return false; - if (newest.identity !== identityDigest()) return false; + if (!newest || !countedPass(newest, maxAgeHours)) return false; try { appendJsonl(file, { ...newest, reused_from: newest.reused_from ?? newest.ts }, { mode: 0o600 }); } catch (e: any) { @@ -629,9 +667,11 @@ function reused(file: string, label: string, cmdSha: string, maxAgeHours: number /** * `record-ci`: carry a lane from CI. Its command must be a key of the base branch's ship_lanes_from; the run is found and - * verified here (lib/ci-carry.ts) at HEAD's clean tree, so no caller supplies run ids, a test count or a tree. Never runs the lane. + * verified here (lib/ci-carry.ts) at HEAD's clean tree, so no caller supplies run ids, a test count or a tree. Never runs the lane; + * inventory files every run skipped entirely run here through the entry's per_file runner (label .skipped) first, and the CI + * row is written only when that run passes at the same tree. */ -function cmdRecordCi(argv: string[]): number { +async function cmdRecordCi(argv: string[]): Promise { const sep = argv.indexOf("--"); const opts = sep < 0 ? argv : argv.slice(0, sep); const get = (name: string) => { const i = opts.indexOf(name); return i >= 0 ? opts[i + 1] : undefined; }; @@ -651,17 +691,33 @@ function cmdRecordCi(argv: string[]): number { const loaded = loadConfigFromBase(cwd, base); if (isConfigUnavailable(loaded)) return none("config_unavailable", loaded.detail); if (loaded.errors.length) return none("config_invalid", loaded.errors.map((x) => `${x.path}: ${x.message}`).join("; ")); - const from = loaded.config.ship_lanes_from; - if (!Object.prototype.hasOwnProperty.call(from, command)) return none("not_mapped", `the command is not a key of origin/${base}'s ship_lanes_from`); - const m = from[command]!; + const m = laneMapping(loaded.config.ship_lanes_from, command); + if (!m) return none("not_mapped", `the command is neither a key nor a declared alias in origin/${base}'s ship_lanes_from`); const check = buildRunCheck(cwd, null, true, base); if ("error" in check) return none("profiles_unavailable", check.error); const head = git(["rev-parse", "HEAD"]); const tree = git(["rev-parse", "HEAD^{tree}"]); if (!head || !tree) return none("no_head", "HEAD is unreadable"); const wtree = () => { const fp = wtreeFingerprint({ cwd, timeoutMs: 30000 }); return "tree" in fp ? fp.tree : null; }; - const v = findCarry(check, { profiles: m.profiles, head, tree, wtree, inventory: () => runInventory(m.inventory, cwd) }); + const v = findCarry(check, { profiles: m.profiles, head, tree, wtree, ...mappingCoverage(m, cwd) }); if (!v.ok) return none(v.reason, v.detail); + let here: EvidenceRecord["skipped_here"]; + if (v.remainder?.length) { + const hereLabel = `${label}.skipped`; + console.error(`gstack-evidence record-ci: ${v.remainder.length} inventory file(s) skipped every test in CI; running them here (label=${hereLabel})`); + const hereCommand = perFileCommand(m.per_file!, v.remainder); + const exit = await cmdRun(["--label", hereLabel, "--", hereCommand]); + if (exit !== 0) return none("skipped_failed", `the per_file run of ${v.remainder.length} skipped file(s) exited ${exit} (label=${hereLabel})`); + let rows: EvidenceRecord[] = []; + try { rows = readJsonl(ledgerPath().file); } catch { rows = []; } + here = { label: hereLabel, files: v.remainder, tests_ran: rows.findLast((r) => r.label === hereLabel && r.source === undefined && r.cmd_sha256 === sha256(hereCommand) && r.wtree === tree)?.tests_ran ?? "unknown" }; + if (typeof here.tests_ran !== "number" || here.tests_ran <= 0) return none("skipped_zero_run", `the per_file run of ${v.remainder.length} skipped file(s) ran ${here.tests_ran} tests here (label=${hereLabel}); a lane passes on its runner count, never exit 0 alone`); + for (const e of v.runs) { + const changed = runChanged(check.deps, e.run_id, e.run_attempt); + if (changed) return none("run_changed", changed); + } + if (wtree() !== tree) return none("tree_changed", `the working tree changed while the per_file run of label=${hereLabel} ran`); + } let paths: ReturnType; try { paths = ledgerPath(); @@ -673,15 +729,22 @@ function cmdRecordCi(argv: string[]): number { const runs = v.runs.map((r) => `${r.run_id}/${r.run_attempt}`); const rec: EvidenceRecord = { ts: new Date(v.runs.map((r) => r.finished_at).sort()[0]!).toISOString(), label, command, cmd_sha256: sha256(command), exit: 0, duration_s: 0, - commit: head, tree, dirty: false, wtree: tree, tests_ran: v.tests, source: "ci", runs, profiles: m.profiles, policy: loaded.label, + commit: head, tree, dirty: false, wtree: tree, tests_ran: v.tests, source: "ci", runs, profiles: m.profiles, policy: loaded.label, ...(m.jobs ? { jobs: m.jobs } : {}), ...(v.unit ? { tests_unit: v.unit } : {}), ...(v.notOwed ? { not_owed: v.notOwed } : {}), ...(here ? { skipped_here: here } : {}), }; appendJsonl(paths.file, rec, { mode: 0o600 }); - console.error(`gstack-evidence: recorded label=${label} exit=0 tests_ran=${v.tests} log=- source=ci runs=${runs.join(",")} policy=${loaded.label}`); + console.error(`gstack-evidence: recorded label=${label} exit=0 tests_ran=${v.tests}${v.unit ? ` unit=${v.unit}` : ""} log=- source=ci runs=${runs.join(",")} policy=${loaded.label}${here ? ` skipped_here=${here.files.length} tests_ran_here=${here.tests_ran}` : ""}`); + if (v.notOwed) console.error(`gstack-evidence: not owed: ${v.notOwed}`); return 0; } -/** `carried`: each label's newest row when it is a CI-sourced row at the current working tree, as the receipt's ci_carried entries. */ +/** + * `carried`: each label's newest row when it is a CI-sourced row at the current working tree whose mapping policy is + * origin/ (the receipt's base; default the resolved base branch), as the receipt's ci_carried entries. + */ function cmdCarried(argv: string[]): number { + const at = argv.indexOf("--base"); + if (at >= 0 && !argv[at + 1]) { console.error("usage: gstack-evidence carried [--base ] [--json]"); return 2; } + const policyRef = `origin/${(at >= 0 ? argv[at + 1]! : resolveBaseBranch(git(["rev-parse", "--show-toplevel"]) ?? process.cwd())).replace(/^origin\//, "")}@`; let records: EvidenceRecord[] = []; try { records = readJsonl(ledgerPath().file); @@ -691,7 +754,7 @@ function cmdCarried(argv: string[]): number { const wtree = currentWtree(); const latest = new Map(); for (const r of records) latest.set(r.label, r); - const carried = [...latest.values()].filter((r) => r.source === "ci" && !!wtree && r.wtree === wtree && Array.isArray(r.runs)) + const carried = [...latest.values()].filter((r) => r.source === "ci" && !!wtree && r.wtree === wtree && Array.isArray(r.runs) && typeof r.policy === "string" && r.policy.startsWith(policyRef)) .flatMap((r) => r.runs!.map((ref) => { const [run, attempt] = ref.split("/").map(Number); return { lane: r.label, run, attempt, tree: r.wtree! }; })); if (argv.includes("--json")) console.log(JSON.stringify(carried)); else for (const c of carried) console.log(`CI_CARRIED: ${c.lane} run=${c.run}/${c.attempt} tree=${c.tree.slice(0, 12)}`); @@ -756,10 +819,10 @@ function cmdCheck(argv: string[]): number { allFresh = false; continue; } - const testsRan = typeof latest.tests_ran === "number" ? latest.tests_ran : "unknown"; + const testsRan = typeof latest.tests_ran === "number" || latest.tests_ran === "n/a" ? latest.tests_ran : "unknown"; const changedDuringRun = Array.isArray(latest.wtree_changed) ? latest.wtree_changed.filter((p): p is string => typeof p === "string" && p !== "" && !/[\u0000-\u001f\u007f]/.test(p)).slice(0, 5) : []; const ci = latest.source === "ci" && Array.isArray(latest.runs) ? ` source=ci runs=${latest.runs.filter((s) => /^\d+\/\d+$/.test(s)).join(",")}` : ""; - const detail = `label=${label} exit=${latest.exit} tests_ran=${testsRan} ts=${latest.ts}${latest.log_path ? ` log=${latest.log_path}` : ""}${ci}`; + const detail = `label=${label} exit=${latest.exit} tests_ran=${testsRan}${latest.tests_unit === "files" ? " unit=files" : ""} ts=${latest.ts}${latest.log_path ? ` log=${latest.log_path}` : ""}${ci}`; let verdict: "FRESH" | "STALE" | "ZERO-RUN" = "FRESH"; let reason = ""; if (latest.exit !== 0) { @@ -911,7 +974,9 @@ try { if (sub === "run") { process.exit(await cmdRun(rest)); } else if (sub === "record-ci") { - process.exit(cmdRecordCi(rest)); + process.exit(await cmdRecordCi(rest)); + } else if (sub === "passed") { + process.exit(cmdPassed(rest)); } else if (sub === "carried") { process.exit(cmdCarried(rest)); } else if (sub === "check") { @@ -923,7 +988,7 @@ try { } else if (sub === "verify") { process.exit(cmdVerify(rest)); } else { - console.error("usage: gstack-evidence run|record-ci|carried|check|bundle|ancestor|verify ..."); + console.error("usage: gstack-evidence run|record-ci|passed|carried|check|bundle|ancestor|verify ..."); process.exit(2); } } catch (e: any) { diff --git a/bin/gstack-harness-ledger.sh b/bin/gstack-harness-ledger.sh index 6d24acb4b..540a9b977 100644 --- a/bin/gstack-harness-ledger.sh +++ b/bin/gstack-harness-ledger.sh @@ -74,9 +74,10 @@ _gstack_harness_cut_block() { # plugins.entries.gstack-start; Hermes removes the plugin and its config entries when its # folder holds exactly the files gstack wrote, unedited, and is reached without a symlink # below the profile; otherwise it only disables it so an edit or a kept copy survives. The harness runs with PATH, HOME and the profile only. Without the harness on -# PATH it prints the command to run instead. +# PATH it prints the command to run instead. A profile with no harness config (openclaw.json, config.yaml) never had +# the plugin turned on, and the harness CLI would create a config holding only that change, so it runs no harness command. gstack_harness_hook_release() { - local ledger="$1" kind a b target="" on="" profile="" root="" edited=0 to="" sub cmd n=0 safe=1 total + local ledger="$1" kind a b target="" on="" profile="" root="" edited=0 to="" sub cmd cfg n=0 safe=1 total [ -f "$ledger" ] || return 0 while IFS=$'\t' read -r kind a b; do case "$kind" in @@ -91,6 +92,8 @@ gstack_harness_hook_release() { [ "${total:-0}" -gt "$n" ] && edited=1 fi command -v timeout >/dev/null 2>&1 && to="timeout 60" + case "$target" in openclaw) cfg=openclaw.json ;; hermes) cfg=config.yaml ;; *) cfg="" ;; esac + if [ -z "$cfg" ] || [ -f "$profile/$cfg" ]; then case "$target" in openclaw) cmd="config unset plugins.entries.gstack-start" if command -v openclaw >/dev/null 2>&1 && env -i PATH="$PATH" HOME="$HOME" OPENCLAW_STATE_DIR="$profile" $to openclaw $cmd /dev/null 2>&1; then : @@ -99,6 +102,7 @@ gstack_harness_hook_release() { if command -v hermes >/dev/null 2>&1 && env -i PATH="$PATH" HOME="$HOME" HERMES_HOME="$profile" $to hermes plugins "$sub" gstack-start /dev/null 2>&1; then : else echo " start hook: turn it off in Hermes yourself: HERMES_HOME=\"$profile\" hermes plugins disable gstack-start"; fi ;; esac + fi if [ "$safe" = 1 ] && [ -d "$root/__pycache__" ] && _gstack_harness_owned "$root/__pycache__" "$root"; then rm -f "$root"/__pycache__/*.pyc rmdir "$root/__pycache__" 2>/dev/null diff --git a/bin/gstack-lease b/bin/gstack-lease index 7a0aa3348..8fe9a0139 100755 --- a/bin/gstack-lease +++ b/bin/gstack-lease @@ -8,8 +8,11 @@ # shares them. # # Usage: -# gstack-lease touch [--root ] [--pid

| --from ] -# write /gstack-leases/ holding the pid's start identity; +# gstack-lease touch [--root ] [--pid

| --from ] [--run ] +# write /gstack-leases/ holding the pid's start identity +# and the open run (gstack-skill-start's SESSION_ID); without --run +# it records a hold no release closes; a run counts 120 min from its +# start and a hold 120 min from its last renewal; # pid defaults to the harness pid behind this call # (bin/gstack-harness-pid.sh, the same derivation the session marker # uses), or behind --from (a caller whose own parent is short-lived @@ -24,8 +27,11 @@ # --count-shared-harness (bin/gstack-upgrade only) still prints an # ancestor that is a shared harness (capy-server: every session on the # machine runs under it) while its lease is under 120 minutes old +# gstack-lease release --pid

--run [--root ] +# close run on 's lease (gstack-skill-end); once every run it +# recorded is closed the lease is idle and live skips it (GST-1205) # gstack-lease list [--root ] -# print ` m alive|dead|stale` for every lease +# print ` m alive|idle|dead|stale` for every lease # A pid alone is not a holder: pids are reused after a reboot or a machine # wake. A lease records its process's start identity (Linux: boot id and # /proc start time; elsewhere `ps -o lstart`), and a lease whose pid now @@ -47,12 +53,13 @@ BIN="$(cd "$(dirname "$0")" && pwd -P)" } VERB="${1:-}"; [ $# -gt 0 ] && shift -ROOT="" PID="" EXCLUDE="" FROM="" SHARED=0 +ROOT="" PID="" EXCLUDE="" FROM="" SHARED=0 RUN="" while [ $# -gt 0 ]; do case "$1" in --root) ROOT="${2:-}"; shift ;; --pid) PID="${2:-}"; shift ;; --from) FROM="${2:-}"; shift ;; + --run) RUN="${2:-}"; shift ;; --exclude-ancestry-of) EXCLUDE="${2:-}"; shift ;; --count-shared-harness) SHARED=1 ;; *) echo "gstack-lease: unknown argument '$1'" >&2; exit 2 ;; @@ -60,8 +67,8 @@ while [ $# -gt 0 ]; do shift done case "$VERB" in - touch|live|list) ;; - -h|--help|'') sed -n '2,28p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + touch|live|list|release) ;; + -h|--help|'') sed -n '2,34p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "gstack-lease: unknown verb '$VERB'" >&2; exit 2 ;; esac [ -n "$ROOT" ] || ROOT=$(gstack_source_root "$BIN") @@ -248,6 +255,15 @@ reclaim() { unlock return 1 } +# Run entries (GST-1205), lines after the identity: `run ` per +# open run, `hold ` for a holder that opened none, `idle` once every +# run closed. An entry older than 120 minutes no longer counts. +# fresh FILE [DROP] — the entries under 120 minutes old, minus the line whose +# first two fields are DROP ("run " or "hold"). +fresh() { sed -n '2,$p' "$1" 2>/dev/null | awk -v now="$(date +%s)" -v drop="${2:-}" '($1 == "run" && NF == 3 && now - $3 < 7200 && $1 " " $2 != drop) || ($1 == "hold" && NF == 2 && now - $2 < 7200 && $1 != drop)'; } +# idle FILE — the lease records entries and none of them still counts. A lease +# with no entries (written without --run, or by an older gstack) is never idle. +idle() { [ -n "$(sed -n '2,$p' "$1" 2>/dev/null | grep -v '^$')" ] && [ -z "$(fresh "$1")" ]; } age_min() { # FILE — minutes since its mtime _m=$(stat -c %Y "$1" 2>/dev/null || stat -f %m "$1" 2>/dev/null || echo 0) echo $(( ( $(date +%s) - _m ) / 60 )) @@ -264,7 +280,20 @@ case "$VERB" in exit 3 } _tmp="$DIR/.$PID.$$" - identity "$PID" > "$_tmp" && mv -f "$_tmp" "$DIR/$PID" || { rm -f "$_tmp"; unlock; exit 2; } + _id=$(identity "$PID") _keep="" _held=0 _now=$(date +%s) + if [ -f "$DIR/$PID" ] && [ "$(head -1 "$DIR/$PID" 2>/dev/null)" = "$_id" ]; then + if [ -n "$RUN" ]; then _keep=$(fresh "$DIR/$PID" "run $RUN"); else _keep=$(fresh "$DIR/$PID" hold); fi + [ -z "$(sed -n '2,$p' "$DIR/$PID" 2>/dev/null | grep -v '^$')" ] && _held=1 + fi + { + printf '%s\n' "$_id" + [ -z "$_keep" ] || printf '%s\n' "$_keep" + if [ -n "$RUN" ]; then + [ "$_held" -eq 1 ] && echo "hold $(stat -c %Y "$DIR/$PID" 2>/dev/null || stat -f %m "$DIR/$PID" 2>/dev/null || echo "$_now")" + echo "run $RUN $_now" + elif [ -n "$_keep" ]; then echo "hold $_now" + fi + } > "$_tmp" && mv -f "$_tmp" "$DIR/$PID" || { rm -f "$_tmp"; unlock; exit 2; } find "$DIR" -maxdepth 1 -mmin +120 -type f ! -name '.lock*' -exec rm -f {} + 2>/dev/null for _c in "$DIR"/.lock*.claim.*; do [ -f "$_c" ] || continue @@ -274,12 +303,29 @@ case "$VERB" in rm -rf "$DIR"/.lock*.gone.* 2>/dev/null unlock exit 0 ;; + release) + case "$PID" in ''|*[!0-9]*) echo "gstack-lease: release needs --pid and --run " >&2; exit 2 ;; esac + [ -n "$RUN" ] && [ "$RUN" != - ] || { echo "gstack-lease: release needs --pid and --run " >&2; exit 2; } + [ -f "$DIR/$PID" ] || exit 0 + lock || exit 3 + f="$DIR/$PID" _tmp="$DIR/.$PID.$$" + if sed -n '2,$p' "$f" | awk -v id="$RUN" '$1 == "run" && $2 == id { found = 1 } END { exit !found }'; then + _keep=$(fresh "$f" "run $RUN") + { + head -1 "$f" + if [ -n "$_keep" ]; then printf '%s\n' "$_keep"; else echo idle; fi + } > "$_tmp" && touch -r "$f" "$_tmp" && mv -f "$_tmp" "$f" || { rm -f "$_tmp"; unlock; exit 2; } + fi + unlock + exit 0 ;; list) for f in "$DIR"/*; do [ -f "$f" ] || continue p=${f##*/} _age=$(age_min "$f") - echo "$p ${_age}m $(state "$p" "$(head -1 "$f" 2>/dev/null)" "$_age")" + _st=$(state "$p" "$(head -1 "$f" 2>/dev/null)" "$_age") + [ "$_st" = alive ] && idle "$f" && _st=idle + echo "$p ${_age}m $_st" done exit 0 ;; esac @@ -296,6 +342,7 @@ done FOUND=1 for f in "$DIR"/*; do [ -f "$f" ] || continue + idle "$f" && continue p=${f##*/} _age=$(age_min "$f") case "$ANCESTRY" in *" $p "*) [ "$SHARED" -eq 1 ] && [ "$_age" -lt 120 ] && gstack_shared_harness "$p" || continue ;; esac diff --git a/bin/gstack-ship-receipt b/bin/gstack-ship-receipt index 873a00232..b9695b568 100755 --- a/bin/gstack-ship-receipt +++ b/bin/gstack-ship-receipt @@ -181,9 +181,9 @@ function reviewedProduct(repoRoot: string, base: string, head: string, pregateFi return typeof id === 'string' ? { head_tree: tree, product_patch_id: id || 'none' } : { head_tree: tree }; } -/** The lanes CI carried at the checked-out tree, from this machine's evidence ledger (gstack-evidence carried); undefined when none. */ -function ciCarried(): ShipReceipt['ci_carried'] { - const r = runScriptSync(path.join(path.dirname(Bun.fileURLToPath(import.meta.url)), 'gstack-evidence'), ['carried', '--json'], { encoding: 'utf8', timeout: 60_000 }); +/** The lanes CI carried at the checked-out tree under origin/'s mapping, from this machine's evidence ledger (gstack-evidence carried); undefined when none. */ +function ciCarried(base: string): ShipReceipt['ci_carried'] { + const r = runScriptSync(path.join(path.dirname(Bun.fileURLToPath(import.meta.url)), 'gstack-evidence'), ['carried', '--base', base, '--json'], { encoding: 'utf8', timeout: 60_000 }); try { const list = JSON.parse(r.stdout); return Array.isArray(list) && list.length ? list : undefined; } catch { return undefined; } } @@ -285,7 +285,7 @@ function cmdWrite(rest: string[], json: boolean): number { version: flag(rest, '--version') ?? receiptVersion(repoRoot, head, base), base, gated_tree: flag(rest, '--gated-tree') ?? tree?.gated_tree, predecessor: flag(rest, '--predecessor') ?? tree?.predecessor?.recorded_head ?? undefined, tree: tree?.tree, gate_reuse: tree ? `${tree.gate_reuse} (${tree.gate_reuse_reason})` : undefined, - gate: jsonArg(flag(rest, '--gate')), ci: jsonArg(flag(rest, '--ci')), ci_carried: ciCarried(), + gate: jsonArg(flag(rest, '--gate')), ci: jsonArg(flag(rest, '--ci')), ci_carried: ciCarried(base), spend_usd: flag(rest, '--spend-usd') !== undefined ? (Number.isFinite(Number(flag(rest, '--spend-usd'))) ? Number(flag(rest, '--spend-usd')) : flag(rest, '--spend-usd')) : undefined, session_kind: sessionKind, artifacts_consumed: flag(rest, '--artifacts-consumed'), run: flag(rest, '--run'), thread: flag(rest, '--thread'), policy: flag(rest, '--policy') ?? tree?.policy, queue_mode: flag(rest, '--queue-mode'), preregistration: flag(rest, '--preregistration'), history: flag(rest, '--history'), @@ -521,13 +521,13 @@ function verifiedOutputs(repoRoot: string, base: string, pregateFile: string | u return regen.proven.filter(p => !/[*?[\]\\]/.test(p) && matchesAny(owned, p)); } -interface ProductRules { extra: { metadata: string[]; outputs: readonly string[]; fragments?: string }; metadata: string[]; templates: readonly string[] } +interface ProductRules { extra: { metadata: string[]; outputs: readonly string[]; fragments?: string }; metadata: string[]; templates: readonly string[]; placeholders: readonly string[] } function productRules(repoRoot: string, base: string, outputs: readonly string[]): ProductRules { const load = loadPolicyFromBase(repoRoot, base); const policy: ShipPolicy | null = isUnavailable(load) ? null : load.policy; const fragments = policy?.changelog_fragments?.dir.replace(/^(\.\/)+/, '').replace(/\/+$/, ''); const extra = { metadata: [...(policy?.stamp_files ?? []), ...(policy?.stamp_paths ?? []), ...(policy?.release_outputs ?? []), ...(policy?.changelog ? [policy.changelog] : [])], outputs, fragments }; - return { extra, metadata: [...CARRY_METADATA_GLOBS, ...extra.metadata, ...(fragments ? [`${fragments}/*.md`] : []), ...outputs], templates: policy?.stamp_patterns ?? [] }; + return { extra, metadata: [...CARRY_METADATA_GLOBS, ...extra.metadata, ...(fragments ? [`${fragments}/*.md`] : []), ...outputs], templates: policy?.stamp_patterns ?? [], placeholders: policy?.stamp_placeholders ?? [] }; } /** @@ -554,7 +554,7 @@ function productSpec(repoRoot: string, rules: ProductRules, from: string, to: st const versions = new Map(); const version = (ref: string) => { if (!versions.has(ref)) versions.set(ref, versionAt(ref)); return versions.get(ref)!; }; const stamped = (p: string) => matchesAny(rules.metadata, p) && !matchesAny(rules.extra.outputs, p); - const classify = (p: string, a: string, b: string) => classifyCarryPath(p, rules.extra, stamped(p) ? { before: blobAt(a, p), after: blobAt(b, p), oldVersion: version(a), newVersion: version(b), templates: rules.templates, side: a === from && b === to ? branch?.own(p) : undefined } : undefined); + const classify = (p: string, a: string, b: string) => classifyCarryPath(p, rules.extra, stamped(p) ? { before: blobAt(a, p), after: blobAt(b, p), oldVersion: version(a), newVersion: version(b), templates: rules.templates, placeholders: rules.placeholders, side: a === from && b === to ? branch?.own(p) : undefined } : undefined); const changed = changedRaw.split('\n').filter(Boolean).map(p => ({ path: p, class: classify(p, from, to) })); const excludeGlobs = (globs: readonly string[]) => globs.map(g => `:(exclude,glob)${g}`); const demoted = !branch ? changed.filter(c => c.class === 'product' && matchesAny(rules.metadata, c.path)).map(c => c.path) @@ -570,7 +570,7 @@ function productSpec(repoRoot: string, rules: ProductRules, from: string, to: st const was = before.split('\n'); const now = after.split('\n'); const lines = was.map((_, i) => i).filter(i => was[i] !== now[i]); - return was.length === now.length && lines.length > 0 && lines.every(i => side.at.has(i + 1) && stampOnlyLine(was[i]!, now[i]!, oldVersion, newVersion, rules.templates, side, i + 1)); + return was.length === now.length && lines.length > 0 && lines.every(i => side.at.has(i + 1) && stampOnlyLine(was[i]!, now[i]!, oldVersion, newVersion, rules.templates, side, i + 1, rules.placeholders)); }); return { changed, product: ['.', ...excludeGlobs(CARRY_TEST_GLOBS), ...excludeGlobs(metadataGlobs)], tests: [...CARRY_TEST_GLOBS.map(g => `:(glob)${g}`), ...excludeGlobs(metadataGlobs)], demoted, ownRows }; } @@ -680,11 +680,19 @@ function cmdCarry(rest: string[], json: boolean): number { // receipt written against the base while stacked reviewed the parent's hunks too, and the squash moved them into the // base (GST-1199). A parent counts only when merging it into the current base changes no product path. const measure = (rangeBase: string) => { - const listed = [touchedPaths(repoRoot, rangeBase, from), touchedPaths(repoRoot, headBase, head)]; - if (!listed[0] || !listed[1]) return `git could not list the paths the branch touched against ${base}; run /ship on ${head.slice(0, 12)}`; - const touched = new Set([...listed[0], ...listed[1]]); - const spec = productSpec(repoRoot, rules, from, head, { touched, sides: [[rangeBase, from], [headBase, head]], own: branchSides(repoRoot, baseRef, from, '') }); - if (!spec) return `git diff ${from.slice(0, 12)} ${head.slice(0, 12)} failed`; + const listed = [touchedPaths(repoRoot, rangeBase, from), touchedPaths(repoRoot, headBase, head), rangeBase === fromBase ? new Set() : touchedPaths(repoRoot, fromBase, from)]; + if (!listed[0] || !listed[1] || !listed[2]) return `git could not list the paths the branch touched against ${base}; run /ship on ${head.slice(0, 12)}`; + // From a stack parent, a path whose reviewed content is the base's (an earlier base merge brought it in) is base + // context, not reviewed product, unless HEAD changed it against the base (GST-1214). + // A pathspec also names everything beneath a path, so a path with any compared path under it (a file the base turned + // into a directory) stays in. + const compared = [...listed[0], ...listed[1]]; + const baseMerged = rangeBase === fromBase ? [] : [...listed[0]].filter(p => !listed[2]!.has(p) && !listed[1]!.has(p) && !compared.some(q => q.startsWith(`${p}/`))); + const touched = new Set([...listed[0], ...listed[1]].filter(p => !baseMerged.includes(p))); + const found = productSpec(repoRoot, rules, from, head, { touched, sides: [[rangeBase, from], [headBase, head]], own: branchSides(repoRoot, baseRef, from, '') }); + if (!found) return `git diff ${from.slice(0, 12)} ${head.slice(0, 12)} failed`; + const skip = baseMerged.map(p => `:(exclude,literal)${p}`); + const spec = { ...found, product: [...found.product, ...skip], tests: [...found.tests, ...skip] }; const ids = [patchId(repoRoot, rangeBase, from, spec.product), patchId(repoRoot, headBase, head, spec.product)]; for (const r of ids) if (typeof r !== 'string') return `git could not compute the product patch-id at one of the heads: ${r.error}`; return { rangeBase, touched, spec, productFrom: ids[0] as string, productHead: ids[1] as string }; @@ -732,6 +740,16 @@ function cmdCarry(rest: string[], json: boolean): number { if (changedProduct.length >= CARRY_NAMED_PATHS) break; const cls = changed.find(c => c.path === p)?.class ?? classifyCarryPath(p, rules.extra); if (cls === 'metadata' && spec.ownRows.includes(p) && !movedBase.has(p)) stampOnly.push(p); + if (cls === 'metadata' && spec.demoted.includes(p) && movedBase.has(p) && /(^|\/)package\.json$/.test(p)) { + const masked = Object.values(sides).map(ref => { + const r = spawnSync('git', ['show', `${ref}:${p}`], { cwd: repoRoot, encoding: 'utf8', timeout: 30_000, maxBuffer: 16 * 1024 * 1024 }); + try { const { version: _, ...rest } = JSON.parse(r.status === 0 ? r.stdout : 'null') ?? {}; return JSON.stringify(rest); } catch { return null; } + }); + const [rb, rv, cb, cv] = masked; + if (masked.some(m => m === null) || rv !== cv || rb !== cb) changedProduct.push(p); + else stampOnly.push(p); + continue; + } if (cls !== 'product') continue; const verdict = comparePath(repoRoot, sides, p); if (verdict === 'changed') changedProduct.push(p); @@ -809,7 +827,10 @@ function cmdCarry(rest: string[], json: boolean): number { depthField = { review_depth: { ...now, depth: deeper(receipt.review_depth.depth, now.depth) }, review_exclude: receiptExclude(review.exclude) }; } // The rebound product_patch_id is HEAD's diff against the carry's base, so a retargeted receipt names that base from now on. - const rebound = { ...receipt, ...(retarget ? { base: base.replace(/^origin\//, '') } : {}), ...(pregateField ?? {}), ...depthField }; + // A receipt that records a version names HEAD's, read as write reads it: a restamp after a base merge moves it (GST-1030). + // A head with no version source keeps the recorded one (write --version named it). + const version = receipt.version === undefined ? undefined : (receiptVersion(repoRoot, head, base) ?? receipt.version); + const rebound = { ...receipt, ...(version !== undefined ? { version } : {}), ...(retarget ? { base: base.replace(/^origin\//, '') } : {}), ...(pregateField ?? {}), ...depthField }; const carried = carryReceipt(rebound, head, plan.carry, { tree: headTree, fromTree, productPatchId: headPatch }); const errors = validateReceipt(carried); if (errors.length) return refused('RECEIPT_CARRY_REFUSED', `the carried receipt does not validate: ${errors[0]!.message} at ${errors[0]!.path}`); diff --git a/bin/gstack-skill-end b/bin/gstack-skill-end index 865255a46..441495814 100755 --- a/bin/gstack-skill-end +++ b/bin/gstack-skill-end @@ -57,6 +57,9 @@ _TEL_END=$(date +%s) case "$TEL_START" in ''|*[!0-9]*) TEL_START="$_TEL_END" ;; esac _TEL_DUR=$(( _TEL_END - TEL_START )) [ -n "$SESSION_ID" ] && rm -f "$_GH/analytics/.pending-$SESSION_ID" 2>/dev/null || true +# Release the workflow lease run gstack-skill-start opened (SESSION_ID starts +# with its harness pid), so a finished run stops blocking gstack-upgrade (GST-1205). +[ -n "$SESSION_ID" ] && "$_BIN/gstack-lease" release --pid "${SESSION_ID%%-*}" --run "$SESSION_ID" >/dev/null 2>&1 || true # Record completion before the final drain, including its durable enqueue. # The normal sync gates still decide publication: full may sync timelines, diff --git a/bin/gstack-skill-start b/bin/gstack-skill-start index 16b1c9133..aca52a879 100755 --- a/bin/gstack-skill-start +++ b/bin/gstack-skill-start @@ -127,13 +127,6 @@ fi mkdir -p "$_GH/sessions" 2>/dev/null || true touch "$_GH/sessions/$PARENT_PID" 2>/dev/null || true find "$_GH/sessions" -mmin +120 -type f -exec rm {} + 2>/dev/null || true -# The install-scoped twin (bin/gstack-lease): what bin/gstack-upgrade reads -# before moving a checkout other sessions share. Fail-open, but a lease the -# held lock kept from being written is reported. -"$_BIN/gstack-lease" touch --pid "$PARENT_PID" >/dev/null 2>&1 || { - [ $? -eq 3 ] && echo "gstack-lease: the lease directory stayed locked by a live process; this session holds no workflow lease, so bin/gstack-upgrade cannot see it" >&2 - true -} _SESSIONS=$(find "$_GH/sessions" -mmin -120 -type f 2>/dev/null | wc -l | tr -d ' ') _PROACTIVE=$("$_BIN/gstack-config" get proactive 2>/dev/null || echo "core") @@ -241,9 +234,18 @@ _TEL_PROMPTED=$([ -f "$_GH/.telemetry-prompted" ] && echo "yes" || echo "no") _TEL_START=$(date +%s) # PID+epoch alone is guessable; a random suffix makes the block-binding token # unforgeable by content reflected into the same tool result. Falls back to -# the plain form when urandom is unavailable. +# this process's pid when urandom is unavailable, so two same-second starts +# never share a SESSION_ID (it also keys their lease runs, GST-1205). _SID_RAND=$(od -An -N4 -tx4 /dev/urandom 2>/dev/null | tr -d ' \n' || true) -_SESSION_ID="$PARENT_PID-$_TEL_START${_SID_RAND:+-$_SID_RAND}" +_SESSION_ID="$PARENT_PID-$_TEL_START-${_SID_RAND:-$$}" +# The install-scoped twin (bin/gstack-lease): what bin/gstack-upgrade reads +# before moving a checkout other sessions share. This run holds it until +# gstack-skill-end releases its SESSION_ID (GST-1205). Fail-open, but a lease +# the held lock kept from being written is reported. +"$_BIN/gstack-lease" touch --pid "$PARENT_PID" --run "$_SESSION_ID" >/dev/null 2>&1 || { + [ $? -eq 3 ] && echo "gstack-lease: the lease directory stayed locked by a live process; this session holds no workflow lease, so bin/gstack-upgrade cannot see it" >&2 + true +} echo "TELEMETRY: ${_TEL:-off}" echo "TEL_PROMPTED: $_TEL_PROMPTED" echo "SESSION_ID: $_SESSION_ID" diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 4216c5256..38ce1e467 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1001,8 +1001,10 @@ paid test file, its fixtures and its skills' prompt files, including the generated SKILL.md) plus `HARNESS_VERSION`, keyed with model, Claude CLI version and policy version (`scripts/eval-trial-series.ts`). Each trial also records the full consumed-input fingerprint as provenance. `scripts/harness-version.json` -pins every shared-harness file (`GLOBAL_TOUCHFILES` plus the session runners, the -PTY harness, the e2e gate and the LLM judge) by blob hash. When +holds the version and the harness patterns, and `scripts/harness-pins/.json` +pins each shared-harness file (`GLOBAL_TOUCHFILES` plus the session runners, the +PTY harness, the e2e gate and the LLM judge) by blob hash, one file per pin, so +two PRs that record decisions for different harness files never conflict. When `test/harness-version.test.ts` fails, run one of: ```bash @@ -1010,8 +1012,8 @@ bun run scripts/bump-harness-version.ts --non-behavioral "" # new series for every case ``` -On a merge conflict in the manifest, take the higher version, then rerun the -script. +On a merge conflict in the manifest or a pin file, take the higher version, then +rerun the script. ## Ubicloud VMs (`bun run test:ubicloud`) diff --git a/docs/capy-install.md b/docs/capy-install.md index 53326df6d..6f0feef46 100644 --- a/docs/capy-install.md +++ b/docs/capy-install.md @@ -187,7 +187,7 @@ run from inside a workflow (/ship's docs step running `install.sh --for document-release`, #3571) prints `SELF_UPGRADE: skipped (a gstack run in this session holds a workflow lease on this installation: ... (UPGRADE_BUSY))`, leaves the checkout where the run started and stays ok with `behind=`. Each -`gstack-skill-start` renews the lease. On Capy the leased pid is the machine's +`gstack-skill-start` renews the lease and records its `SESSION_ID` as an open run; `gstack-skill-end` closes that run, and a lease whose runs are all closed is idle and blocks nothing, so a finished /ship no longer holds the upgrade back (GST-1205). Each run counts for 120 minutes from its own start, whatever other sessions renew. A lease with no recorded run (written without `--run`, or by an older gstack) counts as before. On Capy the leased pid is the machine's long-lived `capy-server`, so the deferral ends when the lease turns 120 minutes old, not when the process exits. Every session on the machine runs under that same `capy-server`, so `bin/gstack-upgrade` run by hand also @@ -231,7 +231,7 @@ identities, a taker stalled over 2 minutes between its `mkdir` and its owner rename can be broken and then publish into its successor's lock. While a live process holds the lock, or it cannot be taken at all, `touch` exits 3 without writing (`gstack-skill-start` reports that the session holds no lease) and a reclaim -keeps the lease. `bin/gstack-lease list` shows `alive|dead|stale`, and +keeps the lease. `bin/gstack-lease list` shows `alive|idle|dead|stale`, and `GSTACK_UPGRADE_FORCE=1` overrides. ## What the installer does diff --git a/docs/hosts.md b/docs/hosts.md index ecabfbf76..4c465c7f1 100644 --- a/docs/hosts.md +++ b/docs/hosts.md @@ -80,14 +80,16 @@ It is off unless you turn it on, because turning it on changes the harness's own |---|---|---| | Plugin folder | `${OPENCLAW_STATE_DIR:-~/.openclaw}/extensions/gstack-start` (index.js, openclaw.plugin.json, package.json, note.txt) | `${HERMES_HOME:-~/.hermes}/plugins/gstack-start` (plugin.yaml, __init__.py, note.txt) | | Hook | `before_prompt_build`, returning the note as `prependContext` | `pre_llm_call`, returning the note as `context` | -| Settings it changes, through the harness's own CLI | `openclaw config set plugins.entries.gstack-start.hooks.allowConversationAccess true` and `...enabled true` in `openclaw.json`. OpenClaw runs `before_prompt_build` from a plugin it did not bundle only with that grant, which also lets the plugin's hooks see conversation content (the plugin reads only its own note.txt) | `hermes plugins enable gstack-start` adds it to `plugins.enabled` in `config.yaml` (and creates `config.yaml` when there is none) | +| Settings it changes, through the harness's own CLI | `openclaw config set plugins.entries.gstack-start.hooks.allowConversationAccess true` and `...enabled true` in `openclaw.json`. OpenClaw runs `before_prompt_build` from a plugin it did not bundle only with that grant, which also lets the plugin's hooks see conversation content (the plugin reads only its own note.txt) | `hermes plugins enable gstack-start` adds it to `plugins.enabled` in `config.yaml` | | Check | `openclaw plugins inspect gstack-start --runtime --json` lists `before_prompt_build` under `typedHooks` | `hermes plugins list` shows `gstack-start` enabled | -**Turn it on.** `gstack-harness register openclaw --start-hook` (or `hermes`), or `gstack-config set harness_start_hook on` followed by `./setup`, or `GSTACK_HARNESS_START_HOOK=1 ./setup --host openclaw`. The first of these that is set decides: the `--start-hook`/`--no-start-hook` flag, then `GSTACK_HARNESS_START_HOOK` (`1`/`on` or `0`/`off`), then `harness_start_hook`, then the choice saved in the registration's ledger. Whatever decides, the choice is saved, so refresh and upgrades keep it. The hook goes only into the harness profile's own plugin folder; a registration with a custom `--skills-dir`, or OpenClaw run with `OPENCLAW_CONFIG_PATH` pointing at another config, gets no hook (`HARNESS_HOOK_UNSUPPORTED` when you ask for one). A hook you already turned on stays on when a later run happens to have `OPENCLAW_CONFIG_PATH` set. +**Turn it on.** `gstack-harness register openclaw --start-hook` (or `hermes`), or `gstack-config set harness_start_hook on` followed by `./setup`, or `GSTACK_HARNESS_START_HOOK=1 ./setup --host openclaw`. The first of these that is set decides: the `--start-hook`/`--no-start-hook` flag, then `GSTACK_HARNESS_START_HOOK` (`1`/`on` or `0`/`off`), then `harness_start_hook`, then the choice saved in the registration's ledger. Whatever decides, the choice is saved, so refresh and upgrades keep it. The hook goes only into the harness profile's own plugin folder; a registration with a custom `--skills-dir`, or OpenClaw run with `OPENCLAW_CONFIG_PATH` pointing at another config, gets no hook (`HARNESS_HOOK_UNSUPPORTED` with `--start-hook`; with `GSTACK_HARNESS_START_HOOK` or `harness_start_hook`, `register` and `./setup --host` say why and exit 4). `OPENCLAW_CONFIG_PATH` naming the profile's own `openclaw.json`, by any path that resolves to it, is fine. A hook you already turned on stays on when a later run happens to have `OPENCLAW_CONFIG_PATH` set. -**Check it.** The receipt and `gstack-harness status` show `start hook: active` (configured for new sessions; a running OpenClaw gateway or Hermes gateway picks it up when it reloads or restarts), `written` (files in place but not active, with the reason and the command that fixes it), `disabled` (you turned it off in the harness), `unchecked` (a run that left a hook already on alone, or `status --no-check`; `gstack-harness status` checks it) or `off`. When the run that turns the hook on (or `--start-hook`) leaves it inactive, `gstack-harness register` and `./setup --host` exit 4 after registering the skills (`HARNESS_HOOK_INACTIVE`). Later refreshes and upgrades try once more each time, without failing, until the hook is active or you disable the plugin in the harness. +**Run the harness's own setup first.** gstack turns the hook on only in a harness that already has its config (`openclaw.json` from `openclaw onboard`, `config.yaml` from `hermes setup`). Without one, the harness CLI would create a config holding nothing but gstack's plugin setting, with no model, provider or terminal settings, so gstack runs no harness command: the plugin's files are written, the hook reports `written` with the setup to run, and `register` and `./setup --host` exit 4. Once the harness is set up, the next `./setup` or `gstack-harness refresh` turns the hook on, changing only the `gstack-start` plugin entry. -**Turn it off.** `gstack-harness register openclaw --no-start-hook` (or `hermes`), or `gstack-config set harness_start_hook off` followed by `./setup`. Either turns the plugin off through the harness's CLI and removes its folder. You can also switch it off in the harness (`openclaw config set plugins.entries.gstack-start.enabled false`, `hermes plugins disable gstack-start`); gstack then leaves it off and never turns it back on unless you pass `--start-hook` again. `gstack-harness unregister` and `gstack-uninstall` remove it with the rest (Hermes only disables it when its folder holds an edited file or a kept `.edited-*` copy, so your edit survives), and print the harness command to run when the harness CLI is not on `PATH`. +**Check it.** The receipt and `gstack-harness status` show `start hook: active` (configured for new sessions; a running OpenClaw gateway or Hermes gateway picks it up when it reloads or restarts), `written` (files in place but not active, with the reason and the command that fixes it), `disabled` (you turned it off in the harness), `unknown` (the harness CLI is not on `PATH`, as inside OpenClaw's or Hermes's own terminal tool, and the config file cannot settle it; gstack reads `plugins.enabled` and `plugins.disabled` from Hermes's `config.yaml` and the plugin entry from `openclaw.json` instead), `unchecked` (a run that left a hook already on alone, or `status --no-check`; `gstack-harness status` checks it) or `off`. When the run that turns the hook on (or `--start-hook`) leaves it inactive, `gstack-harness register` and `./setup --host` exit 4 after registering the skills (`HARNESS_HOOK_INACTIVE`). Later refreshes and upgrades try once more each time, without failing, until the hook is active or you disable the plugin in the harness. + +**Turn it off.** `gstack-harness register openclaw --no-start-hook` (or `hermes`), or `gstack-config set harness_start_hook off` followed by `./setup`. Either turns the plugin off through the harness's CLI and removes its folder. You can also switch it off in the harness (`openclaw config set plugins.entries.gstack-start.enabled false`, `hermes plugins disable gstack-start`); gstack then leaves it off and never turns it back on unless you pass `--start-hook` again. `gstack-harness unregister` and `gstack-uninstall` remove it with the rest (Hermes only disables it when its folder holds an edited file or a kept `.edited-*` copy, so your edit survives), and print the harness command to run when the harness CLI is not on `PATH`. In a profile with no harness config yet (no `openclaw.json` or `config.yaml`), the hook was never turned on, so turning it off or unregistering runs no harness command and leaves the config alone. **What it costs.** The note is about 50 tokens and lands on every user turn, and each copy stays in the session's history, so a long session pays more (mostly cache reads). HARN-18's panel measures the added cost per trial. OpenClaw also logs that it cannot verify where the plugin came from, because gstack writes it instead of installing it from npm or ClawHub. diff --git a/docs/pregate.md b/docs/pregate.md index a6b692927..66771f548 100644 --- a/docs/pregate.md +++ b/docs/pregate.md @@ -88,9 +88,9 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`). | `stall_timeout_s` | seconds a test command may print nothing while using under a tenth of one CPU before its process group is killed and the check is `incomplete` (`PREGATE_STALLED`); default 300, `0` disables; `--stall-timeout` overrides; the `--remote` gate is exempt (see [supervision](#supervision)) | | `timeout_s` | seconds each runner process may take (the unit-file group, each serial file, each guard without its own `timeout_s`, and the remote gate's deadline) when `--timeout` is not passed; 1 to 3600, default 600. `gstack-ship-preflight` and the `finish` fallback run pregate without `--timeout`, so this is how a repo with long lanes sets theirs | | `remote_gate` | the repo's remote test gate command; under 8 cores the local test tier becomes a `requires-remote` obligation naming it, and `--remote` runs it (see [remote gate](#remote-gate)) | -| `run_profiles` | name → `{ workflow, events, jobs, artifact, max_age_hours?, os? }`: the GitHub Actions runs `clear --run` verifies as evidence on the identical tree; a profile named after a lane (`free-tests.yml/macos-named-regressions`) clears that lane (see [run evidence](#run-evidence)). `--from-run` and the main-push reuse job verify runs against them too ([from-run](#from-run), [CI reuse](#ci-reuse)) | +| `run_profiles` | name → `{ workflow, events, jobs, artifact, max_age_hours?, os?, skippable? }`: the GitHub Actions runs `clear --run` verifies as evidence on the identical tree; a profile named after a lane (`free-tests.yml/macos-named-regressions`) clears that lane (see [run evidence](#run-evidence)). `--from-run` and the main-push reuse job verify runs against them too ([from-run](#from-run), [CI reuse](#ci-reuse)) | | `remote_gate_runs` | `{ profiles, inventory, skipped? }`: the run profiles that together clear `pregate.json/remote_gate`, the command that prints the remote gate's test files, one per line, which their runs must cover, and an optional command that prints which selected files the gate itself never runs (see [run evidence](#run-evidence)) | -| `ship_lanes_from` | maps each exact /ship lane command to `{ profiles, inventory }`: the run profiles whose verified runs at HEAD's clean tree carry that lane instead of running it, and the command listing its test files, which their runs must cover (see [CI carry](#ci-carry)); read from the base branch only | +| `ship_lanes_from` | maps each exact /ship lane command to `{ profiles, inventory }` or `{ profiles, jobs }`, with optional `aliases` (and, on an inventory entry, an optional `per_file` runner for files CI skipped entirely, see [skipped files](#skipped-files)): the run profiles whose verified runs at HEAD's clean tree carry that lane instead of running it, and either the command listing its test files, which their runs must cover, or the CI jobs that must have succeeded (see [CI carry](#ci-carry)); read from the base branch only | | `allowlist` | `[{ finding, paths?, pattern?, reason }]`: benign findings the `secrets` check accepts and names as allowlisted, by gstack-redact finding id or non-secret category, path globs and a pattern anchored at the finding; never `env.value` (see [allowlist](#allowlist)) | Only the `origin/` copy or the user-owned file below is honoured, so a @@ -201,9 +201,12 @@ Two stages with separate clocks, both printed: file is read whole from disk. A file over the size limit, or whose full diff does not parse, is scanned on its added lines alone, without that context. - `strays`: a new file at the repository root, a new top-level directory, or a - scratch name (`zz-*`, `tmp*`, `scratch*`) anywhere. + scratch name (`zz-*`, `tmp*`, `scratch*`) or editor leftover (`*.orig`, + `*.rej`, `*.swp`, `*.swo`, `*.bak`, `*.tmp`, `*~`, `.DS_Store`, `Thumbs.db`) + anywhere. - `literals`: for every changed exported constant or named timeout/retry - literal, the whole test tree (nightly- and Heavy-only files included) is + literal (a named literal the branch only adds, such as `limit: 300` in a new + object, is not a change), the whole test tree (nightly- and Heavy-only files included) is searched for the old value in files that name the constant or, when the old value is longer than one character, its module (the module's file name not followed by another extension, or a path ending @@ -259,7 +262,8 @@ Two stages with separate clocks, both printed: `--allow-repo-commands` the selected guards are listed and the check is `incomplete`. Guards are read from the base branch's copy (or the user-owned file when the base carries none) and cannot be - downgraded to warn. See [guards](#guards). + downgraded to warn. A guard whose exact command already passed at this tree + is not run again ([guard carry](#guard-carry)). See [guards](#guards). **Stage 2, test execution:** @@ -710,7 +714,22 @@ other gate change. Each profile names the workflow file, the events it accepts, every job that must have passed (each exactly once), and the artifact holding the run's `tree.json`. `max_age_hours` (1–168, default 24) bounds how old the run may be, and `os` (`Linux`, `macOS`, `Windows`) requires -`tree.json` to record that runner OS. Each workflow uploads `tree.json` from +`tree.json` to record that runner OS. `skippable` +(`{ jobs, gate, scripts }`, GST-1190) lets named jobs be skipped by a path +filter: a skipped job counts only in a `pull_request` run (whose gate diffs +against the pull request's base; a push or dispatch gate diffs against the last +push, which the branch's push history controls), when the `gate` job (the one +whose output decides the skip) succeeded in the same attempt, and the workflow +file plus the gate's inputs are byte-identical to `origin/`. Those inputs +are the declared `scripts`, which must list every repo file the gate reads, +plus every path the gate job's steps name in the base copy of the workflow +(also read under the job's `working-directory` and after a `${{ … }}` +prefix), plus `package.json` when the gate runs a package script. gstack +cannot see what a gate script imports or what a package script runs, so list +those files in `scripts`. Otherwise the pull request controlled the skip +decision, so the job refuses as `job_skipped` and the lane runs here. A lane +whose every named job was skipped this way prints `LANE:

(m /bin/gstack-upgrade once the lease is 120 minutes old, or GSTACK_UPGRADE_FORCE=1 /bin/gstack-upgrade when no other session uses this checkout (UPGRADE_BUSY))` and leaves the -checkout where the run started. Each `gstack-skill-start` renews the lease. +checkout where the run started. Each `gstack-skill-start` renews the lease +and opens a run that its `gstack-skill-end` closes; a lease whose runs are all +closed is idle and never counts (GST-1205). **Fix.** Wait for those sessions to finish (`bin/gstack-lease list` shows -every lease with its age and `alive|dead|stale`; a holder is named as +every lease with its age and `alive|idle|dead|stale`; a holder is named as `pid

(m )`), then re-run `bin/gstack-upgrade`. For the session's own run, or a shared-harness lease, re-run once the lease is 120 minutes old. When you know no other session uses the checkout, `GSTACK_UPGRADE_FORCE=1 bin/gstack-upgrade` @@ -3733,6 +3735,23 @@ the entry is reported as unverified, not `REGEN_STALE`. **Fix.** Run the printed install command (`bun install`, or the one matching the repository's lockfile), then rerun `gstack-regen check` or the pre-gate. + +### `PREGATE_GSTACK_TOO_OLD` + +**Meaning.** `.gstack/pregate.json` uses a key this gstack does not know, and +the same tree's `.gstack/ship-policy.json` sets `min_gstack_version` above this +gstack. The key belongs to a newer gstack (one a later release adds), so the +line reads `needs gstack >= ; upgrade`. The policy still refuses, +exactly as `PREGATE_CONFIG_INVALID` would. Only a gstack that has this check +prints it; one released before it (1.94.0.0 included) reports the same key as +`PREGATE_CONFIG_INVALID`. + +**Fix.** Upgrade gstack on this machine to at least the version named +(`bin/gstack-upgrade`, or the session install with `--for `), then +rerun. A repo adopting a new key raises `min_gstack_version` first, so an older +gstack that has this check prints this instead of a generic invalid-config +error. + ### `PREGATE_CONFIG_INVALID` @@ -3770,7 +3789,9 @@ anchored pattern and a reason ([docs/pregate.md](pregate.md#allowlist)); ### `PREGATE_STRAYS` **Meaning.** A file was added at the repository root, in a new top-level -directory, or under a scratch name (`zz-*`, `tmp*`, `scratch*`). +directory, or under a scratch name (`zz-*`, `tmp*`, `scratch*`) or an editor +leftover name (`*.orig`, `*.rej`, `*.swp`, `*.swo`, `*.bak`, `*.tmp`, `*~`, +`.DS_Store`, `Thumbs.db`). **Fix.** Delete or move it. An intentional root file goes in `strays.allow`. @@ -6451,7 +6472,7 @@ There is no ledger for that harness or id under `$GSTACK_STATE_ROOT/harness`. `g ### `gstack-harness: has no start hook` / `the start hook goes into the profile

, and this registration uses another skills folder` (HARNESS_HOOK_UNSUPPORTED) -The [start hook](hosts.md#start-hook) exists for OpenClaw and Hermes only, and it goes into the harness profile's own plugin folder, the one the harness scans. A registration with `--skills-dir` somewhere else has no profile gstack can trust, and with `OPENCLAW_CONFIG_PATH` set OpenClaw reads a config outside the profile, where gstack's settings change would not land, so `--start-hook` is refused and nothing is written. Register without `--skills-dir` (set `OPENCLAW_STATE_DIR` or `HERMES_HOME` to pick the profile), or drop `--start-hook`. When the choice comes from `GSTACK_HARNESS_START_HOOK` or `harness_start_hook` instead of the flag, such a registration just gets no hook, and its receipt says so. +The [start hook](hosts.md#start-hook) exists for OpenClaw and Hermes only, and it goes into the harness profile's own plugin folder, the one the harness scans. A registration with `--skills-dir` somewhere else has no profile gstack can trust, and with `OPENCLAW_CONFIG_PATH` naming another config OpenClaw reads settings outside the profile, where gstack's settings change would not land, so `--start-hook` is refused and nothing is written. `OPENCLAW_CONFIG_PATH` naming the profile's own `openclaw.json` (compared after resolving symlinks) is not refused. Register without `--skills-dir` (set `OPENCLAW_STATE_DIR` or `HERMES_HOME` to pick the profile), unset `OPENCLAW_CONFIG_PATH` or point it at the profile's own `openclaw.json`, or drop `--start-hook`. When the choice comes from `GSTACK_HARNESS_START_HOOK` or `harness_start_hook` instead of the flag, the registration gets no hook, and `gstack-harness register` and `./setup --host` exit 4 with the reason (`HARNESS_HOOK_INACTIVE`, `start hook is off`); refreshes only print it. ### `gstack-harness: GSTACK_HARNESS_START_HOOK="" is not on or off` (HARNESS_HOOK_CHOICE_INVALID) @@ -6466,7 +6487,9 @@ Each harness profile has one start hook, owned by the gstack registration that w ### `gstack-harness: start hook is written: . Turn it on: ` (HARNESS_HOOK_INACTIVE, exit 4) -The skills are registered and the start hook's files are in place, but the hook is not active in the harness. gstack turns the hook on through the harness's own CLI and then reads it back, so this means the CLI was not on `PATH`, one of its commands failed or timed out (the message quotes its last output), or the harness's settings block the plugin: OpenClaw's `plugins.allow` set without `gstack-start`, `gstack-start` in `plugins.deny`, or `hooks.allowPromptInjection: false`. gstack never edits your allow or deny lists. Run the printed command, then `gstack-harness status`; `--start-hook` again retries the whole step. `./setup --host` exits 4 for the same reason. Refreshes and upgrades retry on their own without failing, and stop once you disable the plugin in the harness. +The skills are registered and the start hook's files are in place, but the hook is not active in the harness. gstack turns the hook on through the harness's own CLI and then reads it back, so this means the harness has no config yet (`no OpenClaw config at yet` or `no Hermes config at yet`: run `openclaw onboard` or `hermes setup` first, then `gstack-harness refresh`; gstack runs no harness command until then, because the CLI would create a config holding only gstack's plugin setting), the CLI was not on `PATH`, one of its commands failed or timed out (the message quotes its last output), or the harness's settings block the plugin: OpenClaw's `plugins.allow` set without `gstack-start`, `gstack-start` in `plugins.deny`, or `hooks.allowPromptInjection: false`. gstack never edits your allow or deny lists. Run the printed command, then `gstack-harness status`; `--start-hook` again retries the whole step. `./setup --host` exits 4 for the same reason. Refreshes and upgrades retry on their own without failing, and stop once you disable the plugin in the harness. + +The same code with `start hook is off: ` means you asked for the hook through `GSTACK_HARNESS_START_HOOK` or `harness_start_hook` but this registration cannot take it (a custom `--skills-dir`, `OPENCLAW_CONFIG_PATH` naming another config, or an unsafe plugin folder); the message names the fix. ### `gstack-harness: no QM admin credential in the environment` / `... needs --yes` (HARNESS_QM_CREDENTIALS) diff --git a/gstack/llms.txt b/gstack/llms.txt index 3dbe42205..ec8a7721d 100644 --- a/gstack/llms.txt +++ b/gstack/llms.txt @@ -37,7 +37,7 @@ Conventions: - [/gstack-upgrade](gstack-upgrade/SKILL.md): Upgrade gstack to the latest version. - [/guard](guard/SKILL.md): Full safety mode: destructive command warnings + directory-scoped edits. - [/health](health/SKILL.md): Code quality dashboard. -- [/investigate](investigate/SKILL.md): Systematic debugging with root cause investigation. +- [/investigate](investigate/SKILL.md): Use when a test fails, code errors or output is wrong: find the root cause, then fix it. - [/ios-clean](ios-clean/SKILL.md): Remove the DebugBridge SPM package and all #if DEBUG wiring from an iOS app. - [/ios-design-review](ios-design-review/SKILL.md): Visual design audit for iOS apps on real hardware. - [/ios-fix](ios-fix/SKILL.md): Autonomous iOS bug fixer. @@ -56,10 +56,10 @@ Conventions: - [/plan-devex-review](plan-devex-review/SKILL.md): Interactive developer experience plan review. - [/plan-eng-review](plan-eng-review/SKILL.md): Eng manager-mode plan review. - [/plan-tune](plan-tune/SKILL.md): Self-tuning question sensitivity + developer psychographic for gstack (v1: observational). -- [/qa](qa/SKILL.md): Test the site, API or CLI and fix the bugs it finds. +- [/qa](qa/SKILL.md): Use when you test the site, API or CLI and fix bugs. - [/qa-only](qa-only/SKILL.md): Report browser/API/CLI/job/worker/webhook bugs. - [/retro](retro/SKILL.md): Weekly engineering retrospective. -- [/review](review/SKILL.md): Review this PR's diff. +- [/review](review/SKILL.md): Use when you review this PR's diff. - [/scrape](scrape/SKILL.md): Pull data from a web page through the Aside browser — your real, already signed-in sessions. - [/setup-browser-cookies](setup-browser-cookies/SKILL.md): Import cookies from your real Chromium browser into the headless browse session. - [/setup-deploy](setup-deploy/SKILL.md): Configure deployment settings for /land-and-deploy. @@ -71,7 +71,7 @@ Conventions: - [/test-audit](test-audit/SKILL.md): Find low-value or duplicate tests and the test-only code they keep alive. - [/ubicloud](ubicloud/SKILL.md): Run a heavy suite, build or eval on an ephemeral 16-vCPU Ubicloud VM: TTL-named VMs, a reaper, quota preflight and a hang watchdog. - [/unfreeze](unfreeze/SKILL.md): Clear the freeze boundary set by /freeze, allowing edits to all directories again. -- [/verify](verify/SKILL.md): Build loop while coding: run the repo's checks and the tests /ship would pick, fix, repeat until green. +- [/verify](verify/SKILL.md): Use when coding: run the checks and tests /ship picks until green. - [/write-skill](write-skill/SKILL.md): Write a new agent skill or improve one without bloat: the edit ladder, levels, drift and size checks, full diets. ## Browse Commands diff --git a/investigate/SKILL.md b/investigate/SKILL.md index cb6a50df2..191fe7e05 100644 --- a/investigate/SKILL.md +++ b/investigate/SKILL.md @@ -2,7 +2,7 @@ name: investigate preamble-tier: 2 version: 1.0.0 -description: Systematic debugging with root cause investigation. (gstack) +description: "Use when a test fails, code errors or output is wrong: find the root cause, then fix it. (gstack)" allowed-tools: - Bash - Read @@ -15,9 +15,29 @@ allowed-tools: triggers: - debug this - fix this bug + - test fails + - failing test - why is this broken - - root cause analysis + - this doesn't work + - build is broken + - stack trace - investigate this error + - root cause analysis +contract: + version: 1 + items: + - id: debug-report + kind: report_labels + heading: DEBUG REPORT + labels: + - Symptom + - Root cause + - Fix + - Evidence + - Regression test + - Status + - id: check-after-last-edit + kind: check_after_last_edit hooks: PreToolUse: - matcher: "Edit" @@ -61,6 +81,15 @@ gbrain: +## Done means + +Finish only when each of these exists (contract v1): + +- `debug-report`: a `DEBUG REPORT` with Symptom, Root cause, Fix, Evidence, Regression test and Status filled in (`none: ` where the skill's own stop rule leaves one empty). +- `check-after-last-edit`: a test or repro command run after your last edit. + +`gstack-skill-contract check` reads them from the transcript (docs/skill-routing-and-contracts.md). For an item that truly does not apply, run `gstack-skill-contract waive --reason ""` (this session only; it counts as a miss). + ## Shared preamble (load first) This skill follows gstack shared preamble `4e82fa4c839f` (AskUserQuestion format, voice, writing, completeness and completion rules). If your Read of that file, headed by this id, is still in context, do not read it again; otherwise Read `~/.claude/skills/gstack/investigate/sections/preamble.md` in full before the command below. A section or subsection here with a shared heading replaces the shared one. diff --git a/investigate/SKILL.md.tmpl b/investigate/SKILL.md.tmpl index c7469537d..a9f7b2ecb 100644 --- a/investigate/SKILL.md.tmpl +++ b/investigate/SKILL.md.tmpl @@ -3,10 +3,10 @@ name: investigate preamble-tier: 2 version: 1.0.0 description: | - Systematic debugging with root cause investigation. Iron Law: no fixes - without root cause. - Use when asked to "debug this", "fix this bug", "why is this broken", - "investigate this error", or "root cause analysis". (gstack) + Use when a test fails, code errors or output is wrong: find the root cause, then fix it. + Systematic debugging; Iron Law: no fixes without root cause. Fits a failing + build, test, script, service or config, an error message or stack trace, + wrong or missing output, or broken data, before any edit that fixes it. (gstack) allowed-tools: - Bash - Read @@ -19,9 +19,29 @@ allowed-tools: triggers: - debug this - fix this bug + - test fails + - failing test - why is this broken - - root cause analysis + - this doesn't work + - build is broken + - stack trace - investigate this error + - root cause analysis +contract: + version: 1 + items: + - id: debug-report + kind: report_labels + heading: DEBUG REPORT + labels: + - Symptom + - Root cause + - Fix + - Evidence + - Regression test + - Status + - id: check-after-last-edit + kind: check_after_last_edit hooks: PreToolUse: - matcher: "Edit" @@ -63,6 +83,8 @@ gbrain: render_as: "## Recent eureka moments (cross-project)" --- +{{CONTRACT_DONE_MEANS}} + {{PREAMBLE}} # Systematic Debugging diff --git a/lib/ci-carry.ts b/lib/ci-carry.ts index 16aec8629..c258f5ec4 100644 --- a/lib/ci-carry.ts +++ b/lib/ci-carry.ts @@ -14,7 +14,9 @@ * max age, the run's own tree.json naming HEAD's tree); * every run executed tests; * with an inventory command, the runs' tree.json files cover every file it - * prints (a lane carries only what CI provably ran); + * prints (a lane carries only what CI provably ran); with named jobs instead + * (a lane with no test files, GST-1189), each of those jobs succeeded in the + * run's newest attempt, read by verifyRun like the profile's own jobs; * every run still stands after the inventory ran (runChanged). * * Anything else returns a refusal with a reason code, and the caller runs the @@ -25,12 +27,18 @@ * bounded by a timeout, and returns green, red with the failing jobs, still * running, or none. */ -import { runChanged, runReceiptFields, uncoveredFiles, verifyRun, type ApiRun, type RunDeps, type RunEvidence, type RunProfile, type RunSource } from './run-evidence'; +import { normalPath, runChanged, runInventory, runReceiptFields, uncoveredFiles, verifyRun, type ApiRun, type RunDeps, type RunEvidence, type RunProfile, type RunSource } from './run-evidence'; export type CarryRefusal = 'dirty_tree' | 'no_profile' | 'no_run' | 'no_verified_run' | 'tree_mismatch' | 'no_tests' - | 'inventory_failed' | 'inventory_empty' | 'inventory_uncovered' | 'run_changed' | 'tree_changed' | 'api_error'; + | 'inventory_failed' | 'inventory_empty' | 'inventory_uncovered' | 'inventory_skipped' | 'run_changed' | 'tree_changed' | 'api_error'; export type CarryRefused = { ok: false; reason: CarryRefusal; detail: string }; -export type CarryVerdict = { ok: true; runs: RunEvidence[]; tests: number; receipt: string } | CarryRefused; +/** + * `tests` never claims a whole run's count for a narrow lane: with an inventory it is the number of the lane's inventory files + * the runs' tree.json covers (`unit: 'files'`; tree.json has no per-file test counts yet), with named jobs `n/a`, and only + * without either (the whole run carried) the runs' test count. `remainder`: inventory files every run skipped entirely + * (tree.json `skipped_files`), which the caller must run here with the lane's per_file runner before the lane counts. + */ +export type CarryVerdict = { ok: true; runs: RunEvidence[]; tests: number | 'n/a'; unit?: 'files'; notOwed?: string; remainder?: string[]; receipt: string } | CarryRefused; /** How many of HEAD's runs per workflow are tried, newest first. */ const CANDIDATES = 10; @@ -38,12 +46,13 @@ const no = (reason: CarryRefusal, detail: string): CarryRefused => ({ ok: false, const newestFirst = (a: ApiRun, b: ApiRun) => (b.created_at ?? '').localeCompare(a.created_at ?? '') || b.id - a.id; /** The newest run of `profile` on `head` that verifies; a refusal lists each candidate's reason, newest first. */ -function newestVerified(src: RunSource, name: string, profile: RunProfile, head: string): { ok: true; evidence: RunEvidence } | CarryRefused { +function newestVerified(src: RunSource, name: string, profile: RunProfile, head: string, jobs: string[]): { ok: true; evidence: RunEvidence } | CarryRefused { + if (jobs.length) profile = { ...profile, jobs: [...new Set([...profile.jobs, ...jobs])] }; const listed = src.deps.runsForHead(profile.workflow, head).filter(r => profile.events.includes(r.event)).sort(newestFirst).slice(0, CANDIDATES); if (!listed.length) return no('no_run', `no ${profile.workflow} run (${profile.events.join(', ')}) on ${head.slice(0, 12)} yet`); const refusals: string[] = []; for (const run of listed) { - const v = verifyRun(src.deps, { repo: src.repo, runRef: String(run.id), profileName: name, profile, vetoHeads: [head], now: src.now }); + const v = verifyRun(src.deps, { repo: src.repo, runRef: String(run.id), profileName: name, profile, vetoHeads: [head], now: src.now, unchanged: src.unchanged }); if (v.ok) return v; refusals.push(`run ${run.id}: ${v.reason}`); } @@ -52,9 +61,13 @@ function newestVerified(src: RunSource, name: string, profile: RunProfile, head: /** * The CI evidence for HEAD's tree under `profiles` (one verified run each), or why there is none. `wtree` reads the - * working-tree fingerprint (before the lookup and after it); `inventory` lists the files the runs must cover. + * working-tree fingerprint (before the lookup and after it); `inventory` lists the files the runs must cover, or `jobs` + * names the jobs that must have succeeded in each run's newest attempt (a lane with no test files; no test count needed). */ -export function findCarry(src: RunSource, o: { profiles: string[]; head: string; tree: string; wtree: () => string | null; inventory?: () => string[] | { error: string } }): CarryVerdict { +export function findCarry(src: RunSource, o: { profiles: string[]; head: string; tree: string; wtree: () => string | null; inventory?: () => string[] | { error: string }; jobs?: string[]; perFile?: boolean }): CarryVerdict { + const jobs = o.jobs ?? []; + let covered: number | undefined; + let remainder: string[] = []; const before = o.wtree(); if (before !== o.tree) return no('dirty_tree', `the working tree (${before ? before.slice(0, 12) : 'fingerprint unavailable'}) is not HEAD's committed tree ${o.tree.slice(0, 12)}; only a clean, committed tree carries`); if (!o.profiles.length) return no('no_profile', 'no run profile is named'); @@ -63,20 +76,24 @@ export function findCarry(src: RunSource, o: { profiles: string[]; head: string; for (const name of o.profiles) { const profile = src.profiles[name]; if (!profile) return no('no_profile', `${name} is not a run_profiles name on the base branch`); - const v = newestVerified(src, name, profile, o.head); + const v = newestVerified(src, name, profile, o.head, jobs); if (!v.ok) return v; runs.push(v.evidence); } const other = runs.find(e => e.tree !== o.tree); if (other) return no('tree_mismatch', `run ${other.run_id} tested tree ${other.tree.slice(0, 12)}, not ${o.tree.slice(0, 12)}`); - const empty = runs.find(e => !e.files.length || e.tests === 0); + const empty = jobs.length ? undefined : runs.find(e => !e.files.length || e.tests === 0); if (empty) return no('no_tests', `run ${empty.run_id}'s tree.json ${empty.files.length ? 'records 0 tests' : 'lists no test files'}`); if (o.inventory) { const listed = o.inventory(); if ('error' in listed) return no('inventory_failed', listed.error); if (!listed.length) return no('inventory_empty', 'the inventory command printed no files'); - const missing = uncoveredFiles(listed, runs); - if (missing.length) return no('inventory_uncovered', `${missing.length} of ${listed.length} inventory file(s) are in no run's tree.json: ${missing.slice(0, 5).join(', ')}${missing.length > 5 ? ', …' : ''}`); + const absent = uncoveredFiles(listed, runs, 'covered'); + const some = (fs: string[]) => `${fs.slice(0, 5).join(', ')}${fs.length > 5 ? ', …' : ''}`; + if (absent.length) return no('inventory_uncovered', `${absent.length} of ${listed.length} inventory file(s) are in no run's tree.json: ${some(absent)}`); + remainder = uncoveredFiles(listed, runs, 'uncovered'); + if (remainder.length && !o.perFile) return no('inventory_skipped', `${remainder.length} of ${listed.length} inventory file(s) skipped every test in CI and the lane declares no per_file runner: ${some(remainder)}`); + covered = new Set(listed.map(normalPath)).size - remainder.length; } for (const e of runs) { const changed = runChanged(src.deps, e.run_id, e.run_attempt); @@ -87,6 +104,13 @@ export function findCarry(src: RunSource, o: { profiles: string[]; head: string; } const after = o.wtree(); if (after !== o.tree) return no('tree_changed', `the working tree changed while the runs were verified (now ${after ? after.slice(0, 12) : 'unreadable'})`); + if (jobs.length) { + const gates = [...new Set(o.profiles.map(n => src.profiles[n]?.skippable?.gate).filter(Boolean))]; + const allSkipped = runs.every(e => jobs.every(j => e.skipped_jobs?.includes(j))); + const notOwed = allSkipped ? `CI skipped at this tree (gate job ${gates.join(', ')}, workflow unchanged)` : undefined; + return { ok: true, runs, tests: 'n/a', ...(notOwed ? { notOwed } : {}), receipt: `ci-carry tree=${o.tree.slice(0, 12)} jobs=${jobs.join(',')}${notOwed ? ` not_owed=${gates.join(',')}` : ''} ${runReceiptFields(runs, src.now)}` }; + } + if (covered !== undefined) return { ok: true, runs, tests: covered, unit: 'files', ...(remainder.length ? { remainder } : {}), receipt: `ci-carry tree=${o.tree.slice(0, 12)} files=${covered}${remainder.length ? ` skipped_here=${remainder.length}` : ''} ${runReceiptFields(runs, src.now)}` }; const tests = runs.reduce((n, e) => n + (e.tests ?? e.files.length), 0); return { ok: true, runs, tests, receipt: `ci-carry tree=${o.tree.slice(0, 12)} tests=${tests} ${runReceiptFields(runs, src.now)}` }; } @@ -109,24 +133,75 @@ export async function waitForRun(deps: RunDeps, o: { workflow: string; events: s } } -/** `ship_lanes_from` (base branch only): exact lane command → the profiles whose runs carry it and the command listing its test files. */ -export interface ShipLaneMapping { profiles: string[]; inventory: string } +/** + * `ship_lanes_from` (base branch only): exact lane command → the profiles whose runs carry it and either the command listing + * its test files (`inventory`) or the CI jobs that run it (`jobs`, for a lane with no test files such as a typecheck; GST-1189). + */ +export interface ShipLaneMapping { profiles: string[]; inventory?: string; jobs?: string[]; aliases?: string[]; per_file?: string } + +/** + * The `ship_lanes_from` entry for an exact lane command: the entry it keys, else the entry that declares it in `aliases` + * (GST-1190: the same lane run under another exact spelling, such as an env prefix). Never normalized: an alias is a + * second exact command the base's owner declared, not a pattern. + */ +export function laneMapping(from: Record, command: string): ShipLaneMapping | undefined { + if (Object.prototype.hasOwnProperty.call(from, command)) return from[command]; + return Object.values(from).find(m => m.aliases?.includes(command)); +} + +/** findCarry's coverage rule for one mapping: its inventory (a pin-file command run in `cwd`, with a per_file runner or not) or its named jobs. */ +export const mappingCoverage = (m: ShipLaneMapping, cwd: string): { inventory?: () => string[] | { error: string }; jobs?: string[]; perFile?: boolean } => + m.jobs ? { jobs: m.jobs } : { inventory: () => runInventory(m.inventory!, cwd), perFile: !!m.per_file }; + +/** + * What `gstack-evidence record-ci` reported, read from its own last record line onward (the per_file remainder's output comes + * first, so a test printing look-alike lines cannot change the reading): the carried runs, count, unit and remainder, any + * not-owed reason, or the last CARRY_NONE reason. + */ +export function recordCiResult(stderr: string): { runs?: string; tests?: string; files?: boolean; notOwed?: string; here?: string; none?: string } { + const lines = stderr.split('\n'); + const at = lines.findLastIndex(x => x.startsWith('gstack-evidence: recorded label=') && x.includes(' log=- source=ci runs=')); + const none = lines.findLast(x => x.startsWith('CARRY_NONE: '))?.slice('CARRY_NONE: '.length); + if (at < 0) return none ? { none } : {}; + const got = / tests_ran=(\d+|n\/a)( unit=files)? log=- source=ci runs=(\S+)/.exec(lines[at]!); + const here = / skipped_here=(\d+) tests_ran_here=(\S+)$/.exec(lines[at]!); + const notOwed = lines.slice(at + 1).find(x => x.startsWith('gstack-evidence: not owed: '))?.slice('gstack-evidence: not owed: '.length); + return { runs: got?.[3], tests: got?.[1], files: !!got?.[2], ...(notOwed ? { notOwed } : {}), ...(here ? { here: `skipped_here=${here[1]} file(s) tests_ran_here=${here[2]}` } : {}) }; +} + +/** The per_file runner command for `files`: `{files}` replaced by the paths, each single-quoted for bash. */ +export const perFileCommand = (template: string, files: string[]): string => + template.replaceAll('{files}', files.map(f => `'${f.replace(/'/g, `'\\''`)}'`).join(' ')); /** `: ` errors for `ship_lanes_from`; empty when valid. Shared with lib/pregate/config.ts. */ export function shipLanesErrors(raw: unknown, profileNames: readonly string[]): Array<{ path: string; message: string }> { if (raw === undefined) return []; - if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return [{ path: 'ship_lanes_from', message: 'must be an object of exact lane command → { profiles: [run_profiles names], inventory: command }' }]; + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return [{ path: 'ship_lanes_from', message: 'must be an object of exact lane command → { profiles: [run_profiles names], inventory: command } or { profiles, jobs: [job names] }' }]; const errors: Array<{ path: string; message: string }> = []; const oneLine = (s: unknown) => typeof s === 'string' && !!s.trim() && s.length <= 512 && !/[\0\r\n]/.test(s); + const declared = new Map(Object.keys(raw as object).map(k => [k, 'a key'])); for (const [command, m] of Object.entries(raw as Record)) { const at = `ship_lanes_from[${JSON.stringify(command)}]`; if (!oneLine(command)) errors.push({ path: at, message: 'the key must be the exact lane command: one line under 512 characters' }); - if (!m || typeof m !== 'object' || Array.isArray(m)) { errors.push({ path: at, message: 'must be { profiles, inventory }' }); continue; } + if (!m || typeof m !== 'object' || Array.isArray(m)) { errors.push({ path: at, message: 'must be { profiles, inventory } or { profiles, jobs }' }); continue; } const o = m as Record; - for (const k of Object.keys(o)) if (k !== 'profiles' && k !== 'inventory') errors.push({ path: `${at}.${k}`, message: 'unknown key; valid keys: profiles, inventory' }); + for (const k of Object.keys(o)) if (!['profiles', 'inventory', 'jobs', 'aliases', 'per_file'].includes(k)) errors.push({ path: `${at}.${k}`, message: 'unknown key; valid keys: profiles, inventory, jobs, aliases, per_file' }); + if ('per_file' in o && (!oneLine(o.per_file) || !(o.per_file as string).includes('{files}') || !('inventory' in o))) errors.push({ path: `${at}.per_file`, message: 'must be one command line under 512 characters containing {files}, on an entry with an inventory (it runs the inventory files CI skipped entirely)' }); + if ('aliases' in o) { + const aliases = o.aliases; + if (!Array.isArray(aliases) || !aliases.length || aliases.some(a => !oneLine(a)) || new Set(aliases).size !== aliases.length) errors.push({ path: `${at}.aliases`, message: 'must be a non-empty array of distinct exact lane commands, each one line under 512 characters' }); + else for (const a of aliases as string[]) { + if (declared.has(a)) errors.push({ path: `${at}.aliases`, message: `${JSON.stringify(a)} is already ${declared.get(a)}; each exact command maps to one entry` }); + else declared.set(a, `an alias of ${JSON.stringify(command)}`); + } + } if (!Array.isArray(o.profiles) || !o.profiles.length || o.profiles.some(p => typeof p !== 'string')) errors.push({ path: `${at}.profiles`, message: 'must be a non-empty array of run_profiles names' }); else for (const p of o.profiles as string[]) if (!profileNames.includes(p)) errors.push({ path: `${at}.profiles`, message: `${JSON.stringify(p)} is not a run_profiles name` }); - if (!oneLine(o.inventory)) errors.push({ path: `${at}.inventory`, message: 'must be one command line under 512 characters that prints the lane\'s test files, one per line' }); + if (('inventory' in o) === ('jobs' in o)) errors.push({ path: at, message: 'declare exactly one of inventory (a command listing the lane\'s test files) or jobs (the CI jobs that run it)' }); + else if ('jobs' in o) { + const jobs = o.jobs; + if (!Array.isArray(jobs) || !jobs.length || jobs.some(j => typeof j !== 'string' || !oneLine(j) || j.length > 128) || new Set(jobs).size !== jobs.length) errors.push({ path: `${at}.jobs`, message: 'must be a non-empty array of distinct CI job names, each one line under 128 characters' }); + } else if (!oneLine(o.inventory)) errors.push({ path: `${at}.inventory`, message: 'must be one command line under 512 characters that prints the lane\'s test files, one per line' }); } return errors; } diff --git a/lib/harness-register.ts b/lib/harness-register.ts index 9b00b665f..8744f0977 100644 --- a/lib/harness-register.ts +++ b/lib/harness-register.ts @@ -22,7 +22,7 @@ import { spawnSync } from 'node:child_process'; import { bunIsolationArgs } from './bun-isolation'; import { runBashSync } from './script-invocation'; import { frontmatterList, routerLine } from './skill-triggers'; -import { activateHook, assertHookRoot, hookChoice, hookFiles, hookProfile, hookRoot, inspectHook, isHookTarget, profileSkills, type HookResult } from './harness-start-hook'; +import { activateHook, assertHookRoot, foreignOpenclawConfig, hookChoice, hookConfig, hookFiles, hookProfile, hookRoot, inspectHook, isHookTarget, profileSkills, type HookResult } from './harness-start-hook'; export type SkillsMode = 'router' | 'hybrid' | 'all'; export type TargetName = 'openclaw' | 'hermes' | 'dots' | 'generic' | 'qm' | 'muse' | 'grok'; @@ -135,7 +135,7 @@ export const HARNESS_CODES = { HARNESS_RENDER_FAILED: { anchor: 'harness-render-failed', fix: 'run bun install in the gstack checkout, then register again; the render output above names the failing skill' }, HARNESS_BUDGET_EXCEEDED: { anchor: 'harness-budget-exceeded', fix: 'this is a gstack bug: report it with the byte count printed above' }, HARNESS_NOT_REGISTERED: { anchor: 'harness-not-registered', fix: 'run gstack-harness status to see the registrations on this machine' }, - HARNESS_HOOK_UNSUPPORTED: { anchor: 'harness-hook-unsupported', fix: 'the start hook is for OpenClaw and Hermes registered in their own profile: drop --skills-dir (set OPENCLAW_STATE_DIR or HERMES_HOME to pick the profile), or drop --start-hook' }, + HARNESS_HOOK_UNSUPPORTED: { anchor: 'harness-hook-unsupported', fix: 'the start hook is for OpenClaw and Hermes registered in their own profile: drop --skills-dir (set OPENCLAW_STATE_DIR or HERMES_HOME to pick the profile), unset OPENCLAW_CONFIG_PATH or point it at the profile\'s own openclaw.json, or drop --start-hook' }, HARNESS_HOOK_CHOICE_INVALID: { anchor: 'harness-hook-choice-invalid', fix: 'set it to on or off (GSTACK_HARNESS_START_HOOK also takes 1 or 0), or unset it' }, HARNESS_HOOK_OWNED_ELSEWHERE: { anchor: 'harness-hook-owned-elsewhere', fix: 'if another gstack install wrote it, run that install\'s gstack-harness register --no-start-hook; otherwise move the folder aside, then register again' }, HARNESS_HOOK_INACTIVE: { anchor: 'harness-hook-inactive', fix: 'run the command printed above, then gstack-harness status; the skills themselves are registered' }, @@ -525,7 +525,7 @@ function shellRegistry(opts: RegisterOptions, fn: string, args: string[]): boole return sourcedBash(opts.checkout, 'gstack-install-registry.sh', fn, args, opts.stateRoot, opts.env).status === 0; } -interface HookPlan { on: boolean; prevOn: boolean; explicit: boolean; retry: boolean; profile: string; root: string; files: Map; note?: string } +interface HookPlan { on: boolean; prevOn: boolean; explicit: boolean; retry: boolean; profile: string; root: string; files: Map; note?: string; fix?: string; requested?: boolean } /** The start hook for this run: whether it is on, where it goes, and its files (null for targets without a hook). */ function hookPlan(opts: RegisterOptions, dest: string, prev: Ledger | null): HookPlan | null { @@ -544,17 +544,19 @@ function hookPlan(opts: RegisterOptions, dest: string, prev: Ledger | null): Hoo const name = t.name; const off = (p: HookPlan) => { if (saved) assertHookRoot(name, saved.profile, saved.root, saved.root, (code, msg) => { throw new HarnessError(code, msg); }); return p; }; if (!choice.on) return off(plan); - const why = dest !== profileSkills(t.name, profile) ? 'this registration uses another skills folder (' + dest + ')' - : t.name === 'openclaw' && opts.env.OPENCLAW_CONFIG_PATH ? 'OPENCLAW_CONFIG_PATH points OpenClaw at a config outside that profile' : ''; + const requested = !opts.inherit; + const foreign = t.name === 'openclaw' ? foreignOpenclawConfig(opts.env, profile) : ''; + const [why, fix] = dest !== profileSkills(t.name, profile) ? ['this registration uses another skills folder (' + dest + ')', 'register without --skills-dir to get the hook, or pass --no-start-hook to keep this registration without it'] + : foreign ? ['OPENCLAW_CONFIG_PATH points OpenClaw at ' + foreign + ', outside that profile', 'unset OPENCLAW_CONFIG_PATH or set it to ' + hookConfig('openclaw', profile) + ', then register again'] : ['', '']; if (why && (plan.explicit || !prevOn)) { if (plan.explicit) throw new HarnessError('HARNESS_HOOK_UNSUPPORTED', 'the start hook goes into the ' + t.displayName + ' profile ' + profile + ', and ' + why); - return off({ ...plan, on: false, note: 'off: ' + why + ' (' + choice.source + ' asked for the hook)' }); + return off({ ...plan, on: false, requested, fix, note: 'off: ' + why + ' (' + choice.source + ' asked for the hook)' }); } try { assertHookRoot(t.name, profile, root, prevOn ? prev!.meta.hook_root : undefined, (code, msg) => { throw new HarnessError(code, msg); }); } catch (e) { if (plan.explicit || prevOn || !(e instanceof HarnessError)) throw e; - return off({ ...plan, on: false, note: 'off: ' + e.message + ' (' + choice.source + ' asked for the hook; ' + HARNESS_CODES[e.code].fix + ')' }); + return off({ ...plan, on: false, requested, fix: HARNESS_CODES[e.code].fix, note: 'off: ' + e.message + ' (' + choice.source + ' asked for the hook)' }); } return { ...plan, files: hookFiles(t.name, root) }; } @@ -562,7 +564,7 @@ function hookPlan(opts: RegisterOptions, dest: string, prev: Ledger | null): Hoo /** Turns the hook on after its files are written (when this run turns it on, was asked to, or the last try did not end active), and reads it back. */ function finishHook(opts: RegisterOptions, hook: HookPlan | null, released: string): RegisterResult['hook'] { if (!hook) return undefined; - if (!hook.on) return { state: 'off', ...(hook.note ? { detail: hook.note } : {}), ...(released ? { fix: released } : {}) }; + if (!hook.on) return { state: 'off', ...(hook.note ? { detail: hook.note } : {}), ...(released || hook.fix ? { fix: released || hook.fix } : {}), ...(hook.requested ? { required: true } : {}) }; const t = opts.target.name as 'openclaw' | 'hermes'; const required = !hook.prevOn || hook.explicit; if (!required && !hook.retry) return { state: 'unchecked', profile: hook.profile, root: hook.root, fix: 'gstack-harness status shows whether it is active' }; @@ -678,7 +680,7 @@ export function register(opts: RegisterOptions): RegisterResult { if (opts.registry && t.render === 'host') shellRegistry(opts, 'gstack_install_registry_upsert', [t.name, 'global', '-', dest, opts.checkout, opts.checkout, opts.version, '-', '-']); if (same && hook?.on && hook.explicit && prev.meta.hook_active !== 'no') writeLedger(lfile, { ...prev, meta: { ...prev.meta, hook_active: 'no' } }); const hookResult = finishHook(opts, hook, released); - const active = hookResult?.required || hookResult?.retried ? ({ active: 'yes', disabled: 'disabled' } as Record)[hookResult.state] ?? 'no' : ''; + const active = hook?.on && (hookResult?.required || hookResult?.retried) ? ({ active: 'yes', disabled: 'disabled' } as Record)[hookResult!.state] ?? 'no' : ''; const saved = active ? readLedger(lfile) : null; if (saved && saved.meta.hook_active !== active) writeLedger(lfile, { ...saved, meta: { ...saved.meta, hook_active: active } }); return { id, target: t.name, dest, mode, status: !prev ? 'installed' : same ? 'unchanged' : 'updated', written, skills, router, ledger: lfile, kept, ...(hookResult ? { hook: hookResult } : {}) }; diff --git a/lib/harness-start-hook.ts b/lib/harness-start-hook.ts index 7594d6865..7529d0fd2 100644 --- a/lib/harness-start-hook.ts +++ b/lib/harness-start-hook.ts @@ -22,14 +22,14 @@ export const START_NOTE_MAX = 300; export const START_NOTE = 'gstack: if you use a gstack skill this turn, first run the gstack-skill-start command in its Preamble section exactly as written, with no pipe, head, || true or ; echo. Read its output, then follow the skill.'; export type HookTarget = 'openclaw' | 'hermes'; -export type HookState = 'active' | 'written' | 'disabled' | 'unchecked' | 'off'; +export type HookState = 'active' | 'written' | 'disabled' | 'unknown' | 'unchecked' | 'off'; export interface HookResult { state: HookState; fix?: string; detail?: string; profile?: string; root?: string } type Env = Record; const MANAGED = "Managed by gstack-harness: it rewrites this file. Turn the hook off: gstack-harness register %s --no-start-hook"; const SPECS = { openclaw: { - profileVar: 'OPENCLAW_STATE_DIR', home: '.openclaw', sub: 'extensions', skills: ['skills'], + profileVar: 'OPENCLAW_STATE_DIR', home: '.openclaw', sub: 'extensions', skills: ['skills'], name: 'OpenClaw', config: 'openclaw.json', setup: 'openclaw onboard', activate: [['config', 'set', 'plugins.entries.' + HOOK_ID + '.hooks.allowConversationAccess', 'true'], ['config', 'set', 'plugins.entries.' + HOOK_ID + '.enabled', 'true']], files: (): Record => ({ 'index.js': [ @@ -55,7 +55,7 @@ const SPECS = { }), }, hermes: { - profileVar: 'HERMES_HOME', home: '.hermes', sub: 'plugins', skills: ['skills', 'gstack'], + profileVar: 'HERMES_HOME', home: '.hermes', sub: 'plugins', skills: ['skills', 'gstack'], name: 'Hermes', config: 'config.yaml', setup: 'hermes setup', activate: [['plugins', 'enable', HOOK_ID]], files: (): Record => ({ 'plugin.yaml': ['# ' + MANAGED.replace('%s', 'hermes'), 'name: ' + HOOK_ID, 'version: 1.0.0', "description: Puts gstack's start-command note on each user turn (managed by gstack-harness).", ''].join('\n'), @@ -91,6 +91,51 @@ export const hookRoot = (t: HookTarget, profile: string) => path.join(profile, S /** The skills folder a registration must use for the hook to go into that profile. */ export const profileSkills = (t: HookTarget, profile: string) => path.join(profile, ...SPECS[t].skills); const activateCommands = (t: HookTarget, profile: string) => SPECS[t].activate.map(a => SPECS[t].profileVar + '=' + JSON.stringify(profile) + ' ' + t + ' ' + a.join(' ')).join(' && '); +/** The harness's own config file in a profile: the file activation edits through the harness CLI. */ +export const hookConfig = (t: HookTarget, profile: string) => path.join(profile, SPECS[t].config); + +function canonical(p: string): string { + try { return fs.realpathSync(p); } catch {} + const parent = path.dirname(p); + return parent === p ? p : path.join(canonical(parent), path.basename(p)); +} +/** The config OPENCLAW_CONFIG_PATH names when it is not the profile's own openclaw.json (compared after realpath), else ''. */ +export function foreignOpenclawConfig(env: Env, profile: string): string { + const v = env.OPENCLAW_CONFIG_PATH; + if (!v) return ''; + const named = path.resolve(v); + return canonical(named) === canonical(hookConfig('openclaw', profile)) ? '' : named; +} + +/** A profile with no harness config yet: turning the plugin on through the harness CLI would create a config holding only that setting, so gstack waits for the harness's own setup. */ +function noConfig(t: HookTarget, profile: string): HookResult | null { + const cfg = hookConfig(t, profile); + if (fs.existsSync(cfg)) return null; + const s = SPECS[t]; + return { state: 'written', detail: 'no ' + s.name + ' config at ' + cfg + ' yet, and turning the plugin on would create one holding only that setting', fix: 'run ' + s.name + "'s own setup first (" + s.profileVar + '=' + JSON.stringify(profile) + ' ' + s.setup + '), then gstack-harness refresh' }; +} + +/** Reads the hook from the harness's config file when its CLI is not on PATH (a harness's own terminal tool often lacks it). */ +function inspectConfig(t: HookTarget, profile: string, fix: string): HookResult { + const why = t + ' is not on PATH'; + const written: HookResult = { state: 'written', detail: why + ', so gstack could not turn the plugin on or check it', fix }; + const cfg = hookConfig(t, profile); + const missing = noConfig(t, profile); + if (missing) return { ...missing, detail: why + ' and ' + missing.detail }; + let plugins: any; + try { plugins = ((t === 'hermes' ? Bun.YAML.parse(fs.readFileSync(cfg, 'utf8')) : JSON.parse(fs.readFileSync(cfg, 'utf8'))) as any)?.plugins ?? {}; } + catch { return { state: 'unknown', detail: why + ' and ' + cfg + ' could not be parsed, so gstack could not check the plugin', fix: 'gstack-harness status from a shell where ' + t + ' is on PATH' }; } + if (t === 'hermes') { + const listed = (k: string) => Array.isArray(plugins[k]) && plugins[k].includes(HOOK_ID); + if (listed('disabled')) return { state: 'disabled', detail: 'turned off in Hermes (plugins.disabled in ' + cfg + '); gstack leaves it off', fix }; + if (listed('enabled') && fs.existsSync(path.join(hookRoot(t, profile), 'plugin.yaml'))) return { state: 'active', detail: 'read from plugins.enabled in ' + cfg + ' (' + why + ')' }; + return written; + } + const entry = plugins.entries?.[HOOK_ID] ?? {}; + if (entry.enabled === false) return { state: 'disabled', detail: 'turned off in openclaw.json (plugins.entries.' + HOOK_ID + '.enabled false); gstack leaves it off', fix }; + if (entry.enabled === true && entry.hooks?.allowConversationAccess === true) return { state: 'unknown', detail: cfg + ' turns it on, but ' + why + ', so gstack could not check that OpenClaw loads it', fix: 'gstack-harness status from a shell where openclaw is on PATH' }; + return written; +} /** The plugin's files (absolute path -> bytes): fixed code plus note.txt, which the code reads at each turn. */ export function hookFiles(t: HookTarget, root: string): Map { @@ -124,6 +169,8 @@ const onPath = (bin: string, env: Env) => (env.PATH ?? '').split(path.delimiter) /** Turns the plugin on in the harness. Returns the first failure, or ''. */ export function activateHook(t: HookTarget, profile: string, env: Env, home: string): string { if (!onPath(t, env)) return t + ' is not on PATH'; + const missing = noConfig(t, profile); + if (missing) return missing.detail!; for (const a of SPECS[t].activate) { const r = runHarness(t, profile, a, env, home, 60_000); if (!r.ok) return t + ' ' + a.join(' ') + ' failed (' + r.why + ')'; @@ -136,12 +183,14 @@ function parseJson(s: string): any { return null; } -/** Reads the hook back from the harness: active (configured for new sessions), disabled by the user, or written (with the fix). */ +/** Reads the hook back from the harness: active (configured for new sessions), disabled by the user, written (with the fix), or unknown (no CLI and a config that cannot settle it). Runs no harness command while the profile has no harness config. */ const READ_TIMEOUT = 15_000; export function inspectHook(t: HookTarget, profile: string, env: Env, home: string): HookResult { const fix = activateCommands(t, profile); - if (!onPath(t, env)) return { state: 'written', detail: t + ' is not on PATH, so gstack could not turn the plugin on or check it', fix }; + if (!onPath(t, env)) return inspectConfig(t, profile, fix); + const missing = noConfig(t, profile); + if (missing) return missing; if (t === 'hermes') { const r = runHarness(t, profile, ['plugins', 'list', '--json'], env, home, READ_TIMEOUT); const row = (parseJson(r.stdout) ?? []).find?.((p: any) => p?.name === HOOK_ID); diff --git a/lib/pregate/checks/guards.ts b/lib/pregate/checks/guards.ts index 1a67ff05a..c8d40feb0 100644 --- a/lib/pregate/checks/guards.ts +++ b/lib/pregate/checks/guards.ts @@ -11,9 +11,17 @@ * incident: a gbrain serial test landed without its manifest row and * passed both the isolation check and pregate; `test/cli-goldens.test.ts` * spawns the CLI as a subprocess, so no import graph selects it. + * + * GST-1186: a guard whose exact command already passed at this tree is not run + * again. #3901's carry (gstack-evidence record-ci) carries it from a verified + * CI run when the base's ship_lanes_from maps the command, else a counted + * ledger pass of the same command at the same working tree stands + * (gstack-evidence passed). ship_lanes_from is read from origin/ only, so + * the PR that adds a mapping cannot carry from it yet. */ import * as fs from 'node:fs'; import * as path from 'node:path'; +import { laneMapping, recordCiResult } from '../../ci-carry'; import { matchesAny } from '../../ship-policy'; import type { Guard } from '../config'; import { baseInputs, sha256 } from '../context'; @@ -23,8 +31,29 @@ import { checkScratch } from '../run-scratch'; import { readRange, STALL_IDLE, supervise } from '../supervise'; import type { EnvironmentCause } from '../types'; import type { CheckDef, CheckResult, PregateContext } from '../types'; +import { runScriptSync } from '../../script-invocation'; -interface GuardRun { guard: Guard; exit: number | null; timedOut: boolean; stalled: boolean; tail: string[]; log: string; environment: EnvironmentCause | null } +interface GuardRun { guard: Guard; exit: number | null; timedOut: boolean; stalled: boolean; tail: string[]; log: string; environment: EnvironmentCause | null; prior?: string; carryNone?: string } + +const EVIDENCE = path.resolve(import.meta.dir, '..', '..', '..', 'bin', 'gstack-evidence'); + +/** + * The pass that already covers this guard's exact command at this tree (GST-1186): a CI carry (or a CI skip the unchanged gate + * decided, printed as not owed), else a same-tree local ledger pass under the runtime the guard would launch with. + */ +function priorPass(ctx: PregateContext, guard: Guard, env: NodeJS.ProcessEnv): { prior?: string; carryNone?: string } { + const evidence = (args: string[]) => runScriptSync(EVIDENCE, args, { cwd: ctx.repoRoot, env, encoding: 'utf8', timeout: 600_000 }); + let carryNone: string | undefined; + if (laneMapping(ctx.config.ship_lanes_from, guard.command)) { + const ci = evidence(['record-ci', '--label', `guard-${guard.id}`, '--base', ctx.base, '--', guard.command]); + const got = recordCiResult(String(ci.stderr ?? '')); + if (ci.status === 0 && got.runs) return { prior: got.notOwed ? `not owed: ${got.notOwed} runs=${got.runs}` : `carried runs=${got.runs}` }; + carryNone = got.none ?? `record-ci exit ${ci.status}`; + } + const local = evidence(['passed', '--', guard.command]); + const hit = /^PASSED: (.+)$/m.exec(String(local.stdout ?? ''))?.[1]; + return local.status === 0 && hit ? { prior: `reused ${hit}`, carryNone } : { carryNone }; +} async function runGuard(ctx: PregateContext, guard: Guard, env: NodeJS.ProcessEnv, dir: string): Promise { const timeoutMs = guard.timeout_s ? guard.timeout_s * 1000 : ctx.timeoutMs; @@ -53,9 +82,14 @@ async function run(ctx: PregateContext): Promise { if (runtime.refusal) return { ...base, ...runtime.refusal, cause: 'environment', environment: launchCause('runtime', runtime.refusal.detail), lines: [`runtime: ${runtime.label}`, environmentLine(launchCause('runtime', runtime.refusal.detail))] }; const dir = checkScratch(ctx.scratch, 'guards'); const runs: GuardRun[] = []; - for (const g of selected) runs.push(await runGuard(ctx, g, runtime.env, dir)); + for (const g of selected) { + const p = priorPass(ctx, g, runtime.env); + runs.push(p.prior ? { guard: g, exit: 0, timedOut: false, stalled: false, tail: [], log: '-', environment: null, ...p } : { ...(await runGuard(ctx, g, runtime.env, dir)), carryNone: p.carryNone }); + } const lines = [`runtime: ${runtime.label}`]; for (const r of runs) { + if (r.carryNone) lines.push(`CI_CARRY: none guard=${r.guard.id} (${r.carryNone})`); + if (r.prior) { lines.push(`GUARD: ${r.guard.id} ${r.prior}`); continue; } lines.push(r.stalled ? `GUARD: ${r.guard.id} stalled: no output for ${ctx.stallTimeoutMs / 1000} s${STALL_IDLE}; process group killed (log ${r.log})` : r.timedOut ? `GUARD: ${r.guard.id} timed out after ${r.guard.timeout_s ?? ctx.timeoutMs / 1000} s (log ${r.log})` : `GUARD: ${r.guard.id} exit=${r.exit} (log ${r.log})`); if (r.exit !== 0) lines.push(...r.tail.map(l => ` ${l}`)); if (r.environment) lines.push(` ${r.guard.id} ${environmentLine(r.environment)}`); @@ -68,7 +102,8 @@ async function run(ctx: PregateContext): Promise { if (failed.length) return { ...withRuns, status: 'fail', cause: 'diff', code: 'PREGATE_GUARD_FAILED', detail: `${failed.length} guard(s) failed: ${failed.map(r => r.guard.id).join(', ')}${timedOut.length ? `; timed out: ${timedOut.map(r => r.guard.id).join(', ')}` : ''}${blocked.length ? `; blocked by the environment: ${blocked.map(r => r.guard.id).join(', ')}` : ''}`, fix: fixOf(failed) }; if (blocked.length) return environmentResult(withRuns, blocked[0]!.environment!, `${blocked.length} guard(s) could not run on this machine: ${blocked.map(r => `${r.guard.id} (${r.environment!.kind})`).join(', ')}${timedOut.length ? `; timed out: ${timedOut.map(r => r.guard.id).join(', ')}` : ''}`); if (timedOut.length) return { ...withRuns, status: 'incomplete', code: timedOut.some(r => r.stalled) ? 'PREGATE_STALLED' : 'PREGATE_INCOMPLETE', detail: `${timedOut.length} guard(s) did not finish: ${timedOut.map(r => `${r.guard.id} (${r.stalled ? 'stalled' : 'timed out'})`).join(', ')}`, fix: 'fix the hang, or raise the guard\'s timeout_s, --timeout or --stall-timeout; an unfinished guard never passes' }; - return { ...withRuns, detail: `${runs.length} guard(s) passed: ${runs.map(r => r.guard.id).join(', ')}` }; + const prior = runs.filter(r => r.prior).length; + return { ...withRuns, detail: `${runs.length} guard(s) passed: ${runs.map(r => r.guard.id).join(', ')}${prior ? ` (${prior} from an earlier pass at this tree)` : ''}` }; } export const guardsCheck: CheckDef = { id: 'guards', stage: 'preflight', tier: 2, downgradable: false, needsRepoCommands: true, run }; diff --git a/lib/pregate/checks/lanes.ts b/lib/pregate/checks/lanes.ts index a16f4b0ed..a0ff1f85a 100644 --- a/lib/pregate/checks/lanes.ts +++ b/lib/pregate/checks/lanes.ts @@ -40,7 +40,7 @@ import { selectLanes, type Selection } from '../lane-selection'; import { launchRuntime } from '../runtime'; import { cargoModel, type CargoModel } from '../rust'; import { adapterFor, readRange, STALL_IDLE, supervise, type Counter, type Supervised } from '../supervise'; -import { verifyFromRun, type RunProfile } from '../../run-evidence'; +import { ranFiles, verifyFromRun, type RunProfile } from '../../run-evidence'; import { matchesAny } from '../../ship-policy'; import type { CheckDef, CheckResult, PregateContext, RemoteObligation } from '../types'; @@ -643,9 +643,10 @@ function fromRun(ctx: PregateContext, sel: Selection): { line: string; passed: s const names = 'error' in check ? [] : fromRunProfiles(ctx, check.profiles); const v = 'error' in check ? { ok: false as const, reason: 'profiles_unavailable', detail: check.error } : verifyFromRun(check, ref, names, ctx.wtree, ctx.head); if (!v.ok) return { line: `REUSE: refused reason=${v.reason} — ${v.detail}; every selected test runs here`, passed: [] }; - const ran = new Set(v.evidence.files); + const ran = new Set(ranFiles(v.evidence)); const passed = sel.localTests.filter(t => ran.has(t)); - return { line: `REUSE: accepted ${passed.length} of ${sel.localTests.length} selected local test file(s) from ${v.receipt}; ${sel.localTests.length - passed.length} still run`, passed, receipt: v.receipt }; + const skipped = sel.localTests.filter(t => v.evidence.skipped?.includes(t)).length; + return { line: `REUSE: accepted ${passed.length} of ${sel.localTests.length} selected local test file(s) from ${v.receipt}; ${sel.localTests.length - passed.length} still run${skipped ? ` (${skipped} skipped every test in the run)` : ''}`, passed, receipt: v.receipt }; } /** diff --git a/lib/pregate/checks/literals.ts b/lib/pregate/checks/literals.ts index b34311e09..a1db40bb7 100644 --- a/lib/pregate/checks/literals.ts +++ b/lib/pregate/checks/literals.ts @@ -57,10 +57,10 @@ export function constantsIn(text: string): Map { return out; } -/** Named numeric call sites (`timeout: 5000`) keyed by name; the last value per name wins, which is enough to spot a change. */ -export function callLiteralsIn(text: string): Map { - const out = new Map(); - for (const m of text.matchAll(CALL_PATTERN)) out.set(m[1]!, m[2]!); +/** Named numeric call sites (`timeout: 5000`) keyed by name, every value in source order. */ +export function callLiteralsIn(text: string): Map { + const out = new Map(); + for (const m of text.matchAll(CALL_PATTERN)) out.set(m[1]!, [...(out.get(m[1]!) ?? []), m[2]!]); return out; } @@ -69,7 +69,14 @@ export function changedLiterals(file: string, oldText: string, newText: string): const oldC = constantsIn(oldText); const newC = constantsIn(newText); for (const [name, old] of oldC) { const now = newC.get(name); if (now !== undefined && now !== old) out.push({ name, old, new: now, file }); } const oldCall = callLiteralsIn(oldText); const newCall = callLiteralsIn(newText); - for (const [name, old] of oldCall) { const now = newCall.get(name); if (now !== undefined && now !== old && !oldC.has(name)) out.push({ name, old, new: now, file }); } + for (const [name, olds] of oldCall) { + if (oldC.has(name)) continue; + const added = [...(newCall.get(name) ?? [])]; + const removed = olds.filter(v => { const i = added.indexOf(v); if (i >= 0) added.splice(i, 1); return i < 0; }); + if (!added.length) continue; + const seen = new Set(); + removed.forEach((old, i) => { if (!seen.has(old)) { seen.add(old); out.push({ name, old, new: added[Math.min(i, added.length - 1)]!, file }); } }); + } return out; } diff --git a/lib/pregate/checks/strays.ts b/lib/pregate/checks/strays.ts index 4b5d5b572..fed27ae1d 100644 --- a/lib/pregate/checks/strays.ts +++ b/lib/pregate/checks/strays.ts @@ -12,7 +12,7 @@ import type { CheckDef, CheckResult, PregateContext } from '../types'; /** Conventional root files a repo adds without a declaration. */ export const ROOT_OK = /^(?:README|LICENSE|LICENCE|NOTICE|CHANGELOG|CONTRIBUTING|CODE_OF_CONDUCT|SECURITY|AUTHORS|VERSION|Makefile|Dockerfile|Procfile|Gemfile|Rakefile|Cargo\.(?:toml|lock)|go\.(?:mod|sum)|package(?:-lock)?\.json|bun\.lockb?|pnpm-lock\.yaml|yarn\.lock|tsconfig[\w.-]*\.json|bunfig\.toml|pyproject\.toml|pubspec\.(?:yaml|lock)|analysis_options\.yaml|dart_test\.yaml|requirements[\w.-]*\.txt|setup\.(?:py|cfg)|\.[\w.-]+|[\w.-]+\.config\.[cm]?[jt]s|[\w.-]+\.md)$/i; -export const STRAY_NAME = /^(?:zz-|tmp|temp|scratch|debug-|probe)|\.(?:orig|rej|swp|swo|bak|tmp)$|~$|^\.DS_Store$|^Thumbs\.db$/i; +export const STRAY_NAME = /^(?:zz-|tmp|scratch)|\.(?:orig|rej|swp|swo|bak|tmp)$|~$|^\.DS_Store$|^Thumbs\.db$/i; function run(ctx: PregateContext): CheckResult { const inputs = baseInputs(ctx); diff --git a/lib/pregate/config.ts b/lib/pregate/config.ts index 9212f6acf..1797338a8 100644 --- a/lib/pregate/config.ts +++ b/lib/pregate/config.ts @@ -46,9 +46,13 @@ * tree, the command listing the gate's test files they must * cover, and the command naming the selected files the gate * itself never runs (gstack: paid files) - * ship_lanes_from exact /ship lane command → `{ profiles, inventory }`: the - * run profiles whose verified runs at HEAD's tree carry that - * lane, and the command listing its test files (lib/ci-carry.ts) + * ship_lanes_from exact /ship lane command → `{ profiles, inventory }` or + * `{ profiles, jobs }`: the run profiles whose verified runs + * at HEAD's tree carry that lane, and either the command + * listing its test files or the CI jobs that run it (a lane + * with no test files, GST-1189), plus optional `aliases`: + * other exact commands for the same lane (GST-1190) + * (lib/ci-carry.ts) * allowlist `{ finding, paths?, pattern?, reason }` entries the `secrets` * check accepts as benign: a gstack-redact finding id or a * non-secret category, scoped by path globs and/or a regex @@ -74,7 +78,8 @@ import { shipLanesErrors, type ShipLaneMapping } from '../ci-carry'; import type { ResultCodeName } from '../result-codes'; import { runProfileErrors, type RemoteGateRuns, type RunProfile } from '../run-evidence'; import { sealedErrors } from '../sealed-guard'; -import { containedRelative, globToRegExp } from '../ship-policy'; +import { containedRelative, globToRegExp, runningGstackVersion, SHIP_POLICY_PATH } from '../ship-policy'; +import { cmpVersion, parseVersion } from '../version-source'; export const PREGATE_CONFIG_PATH = '.gstack/pregate.json'; export const PREGATE_CONFIG_SCHEMA = 'https://github.com/garrytan/gstack/docs/pregate.md#pregate-v1'; @@ -291,6 +296,22 @@ function git(cwd: string, args: string[]): { ok: boolean; out: string } { return { ok: r.status === 0 && !r.error, out: (r.stdout ?? '').trim() }; } +/** + * GST-1190: an unknown key in a policy whose project floor (min_gstack_version in the same tree's ship-policy.json) is + * above this gstack belongs to a newer gstack. It still refuses, but as an upgrade, not as a typo. + */ +function newerGstackErrors(parsed: { config: PregateConfig; errors: ConfigError[]; declaresLanes: boolean }, policyText: string | null): typeof parsed { + const unknown = (e: ConfigError) => e.message.startsWith('unknown key'); + if (!policyText || !parsed.errors.some(unknown)) return parsed; + let min: unknown; + try { min = (JSON.parse(policyText) as Record | null)?.min_gstack_version; } catch { return parsed; } + const running = runningGstackVersion(); + const want = typeof min === 'string' ? parseVersion(min) : null; + const have = parseVersion(running); + if (!want || !have || cmpVersion(want, have) <= 0) return parsed; + return { ...parsed, errors: parsed.errors.map(e => unknown(e) ? { ...e, code: 'PREGATE_GSTACK_TOO_OLD' as const, message: `needs gstack >= ${min as string}; upgrade (this gstack is ${running} and does not know this key)` } : e) }; +} + export function loadConfigFromBase(repoRoot: string, base: string, knownChecks: readonly string[] = []): ConfigLoad { const ref = `origin/${base.replace(/^origin\//, '')}`; const sha = git(repoRoot, ['rev-parse', '--verify', '--quiet', `${ref}^{commit}`]); @@ -302,13 +323,15 @@ export function loadConfigFromBase(repoRoot: string, base: string, knownChecks: if (!entry.ok || entry.out) return { unavailable: true, base: base.replace(/^origin\//, ''), detail: `${ref}:${PREGATE_CONFIG_PATH} could not be read and git cannot confirm it is absent` }; return { config: DEFAULT_CONFIG, errors: [], present: false, label: `${ref}@${sha.out.slice(0, 12)} (no ${PREGATE_CONFIG_PATH})`, kind: 'none' }; } - return { ...parseConfig(show.out, knownChecks), present: true, label: `${ref}@${sha.out.slice(0, 12)}:${PREGATE_CONFIG_PATH}`, kind: 'base' }; + const policy = git(repoRoot, ['show', `${sha.out}:${SHIP_POLICY_PATH}`]); + return { ...newerGstackErrors(parseConfig(show.out, knownChecks), policy.ok ? policy.out : null), present: true, label: `${ref}@${sha.out.slice(0, 12)}:${PREGATE_CONFIG_PATH}`, kind: 'base' }; } export function loadConfigFromHead(repoRoot: string, knownChecks: readonly string[] = []): ConfigLoad { const file = path.join(repoRoot, PREGATE_CONFIG_PATH); if (!fs.existsSync(file)) return { config: DEFAULT_CONFIG, errors: [], present: false, label: `head (no ${PREGATE_CONFIG_PATH})`, kind: 'none' }; - return { ...parseConfig(fs.readFileSync(file, 'utf8'), knownChecks), present: true, label: `head (unreviewed) ${PREGATE_CONFIG_PATH}`, kind: 'head' }; + const policy = path.join(repoRoot, SHIP_POLICY_PATH); + return { ...newerGstackErrors(parseConfig(fs.readFileSync(file, 'utf8'), knownChecks), fs.existsSync(policy) ? fs.readFileSync(policy, 'utf8') : null), present: true, label: `head (unreviewed) ${PREGATE_CONFIG_PATH}`, kind: 'head' }; } /** A symlink to nothing on the way to `file` (a dotfiles link not yet cloned): a policy the user meant to have, not an absent one. */ @@ -365,7 +388,7 @@ export function explainConfig(): string { 'remote_gate: the repo\'s remote test gate command; under 8 cores the test tier becomes a requires-remote obligation naming it, and gstack-pregate --remote runs it (it must print PREGATE_REMOTE: tree= tests= failed=) (docs/pregate.md#remote-gate)', 'run_profiles: name → { workflow, events, jobs, artifact, max_age_hours?, os? }; the GitHub Actions runs gstack-pregate clear --run verifies as evidence on the identical tree (docs/pregate.md#run-evidence)', 'remote_gate_runs: { profiles, inventory, skipped? }; the run profiles that together clear the remote gate obligation, the command listing the gate\'s test files they must cover, and an optional command printing which selected files (PREGATE_FILES_FILE) the gate itself does not run, which then need no run (docs/pregate.md#run-evidence)', - 'ship_lanes_from: exact /ship lane command → { profiles, inventory }; a verified run of each profile at HEAD\'s clean tree whose tree.json covers the inventory carries the lane instead of running it again (docs/pregate.md#ci-carry)', + 'ship_lanes_from: exact /ship lane command → { profiles, inventory } or { profiles, jobs }, with optional aliases (other exact commands for the same lane); a verified run of each profile at HEAD\'s clean tree whose tree.json covers the inventory, or whose newest attempt ran every named job to success, carries the lane instead of running it again (docs/pregate.md#ci-carry)', `allowlist: [{ finding, paths?, pattern?, reason }]: benign findings the secrets check accepts and names as allowlisted. finding is a gstack-redact id (env.kv, google.api_key, pii.phone.e164, …) or a category (${ALLOWLISTABLE_CATEGORIES.join(', ')}); paths are globs; pattern is a regex matched at the finding's position in the added line; one of the two is required, reason always. Never ${NON_ALLOWLISTABLE.join(' or ')} or engine.* (docs/pregate.md#allowlist)`, 'no .gstack/pregate.json on the base: /projects//pregate.json is read instead (state root: GSTACK_STATE_ROOT, else GSTACK_HOME, else GSTACK_STATE_DIR, else a gstack plugin data dir, else ~/.gstack, per docs/state-root.md; run --explain prints the resolved path; outside the repo, so a PR cannot reach it; a path inside the checkout is ignored)', ].join('\n') + '\n'; diff --git a/lib/pregate/index.ts b/lib/pregate/index.ts index fda383c78..73ab1ba2f 100644 --- a/lib/pregate/index.ts +++ b/lib/pregate/index.ts @@ -267,7 +267,7 @@ function userPolicyChange(file: PregateFile): string | null { export interface ClearInput { runUrl?: string; bundleLane?: string; receiptPath?: string; runs?: string[]; evidence?: RunCheck; flaky?: FlakyInput } /** What a run clear needs: the base branch's run profiles, the verifier's dependencies, the clock, and the remote gate's inventory. */ -export interface RunCheck { repo: string; deps: RunDeps; profiles: Record; gateRuns: RemoteGateRuns | null; now: Date; inventory: (command: string, tree: string, selected?: string[]) => string[] | { error: string }; failures?: FailureDeps } +export interface RunCheck { repo: string; deps: RunDeps; profiles: Record; gateRuns: RemoteGateRuns | null; now: Date; unchanged?: (p: RunProfile) => string | null; inventory: (command: string, tree: string, selected?: string[]) => string[] | { error: string }; failures?: FailureDeps } export type ClearDecision = { receipt: string; attested: boolean; substitution?: PregateReceiptSubstitution; exception?: PregateException } | { refused: 'usage' | 'PREGATE_RECEIPT_INVALID'; message: string; fix?: string }; /** diff --git a/lib/pregate/run-check.ts b/lib/pregate/run-check.ts index 34db52967..b3c795fda 100644 --- a/lib/pregate/run-check.ts +++ b/lib/pregate/run-check.ts @@ -11,20 +11,58 @@ import { spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; -import { ghArtifactDeps, ghRunDeps, runInventory } from '../run-evidence'; +import { ghArtifactDeps, ghRunDeps, runInventory, type RunProfile } from '../run-evidence'; import { wtreeFingerprint } from '../script-invocation'; import { resolveBaseBranch } from '../ship-policy'; import { normalizeRemoteUrl } from '../ship-queue-train-record'; import { REMOTE_GATE_LANE } from './checks/lanes'; -import { PREGATE_CONFIG_PATH, isConfigUnavailable, loadConfigFromBase } from './config'; +import { PREGATE_CONFIG_PATH, isConfigUnavailable, loadConfigFromBase, loadConfigFromHead } from './config'; import type { RunCheck } from './index'; /** - * `profiled` says whether `lane` may have a run profile: an unreadable base counts as profiled, so a lane is - * recorded unverified only once the base shows it has none. + * Null when a skippable profile's workflow, its declared `scripts` and every repo path its gate job names are byte-identical + * at HEAD and origin/ (GST-1190); else the first that differs. Named paths are read from the gate job's block in the base + * workflow (after `${{ … }}` prefixes, and also under each working-directory), plus package.json when the job runs a package + * script. Files the gate reaches indirectly (imports, a package script's own files) count only when `scripts` declares them. */ -export function buildRunCheck(root: string, lane: string | null, allowRepoCommands: boolean, base = resolveBaseBranch(root)): RunCheck | { error: string; profiled?: boolean } { - const loaded = loadConfigFromBase(root, base); +export function gateInputsChanged(root: string, base: string, p: RunProfile): string | null { + const ref = `origin/${base.replace(/^origin\//, '')}`; + const wf = `.github/workflows/${p.workflow}`; + const gate = p.skippable?.gate; + if (!gate) return 'the profile names no gate job'; + const blob = (rev: string, f: string) => { const r = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${rev}:${f}`], { cwd: root, encoding: 'utf8', timeout: 30_000 }); return r.status === 0 ? r.stdout.trim() : null; }; + const show = spawnSync('git', ['show', `${ref}:${wf}`], { cwd: root, encoding: 'utf8', timeout: 30_000, maxBuffer: 8 * 1024 * 1024 }); + if (show.status !== 0) return `${wf} is not on ${ref}`; + const lines = show.stdout.split('\n'); + const jobsAt = lines.findIndex(l => /^jobs:\s*$/.test(l)); + const blocks: string[][] = []; + for (let i = jobsAt + 1; jobsAt >= 0 && i < lines.length && !/^\S/.test(lines[i]!); i++) { + if (/^ [\w-]+:\s*$/.test(lines[i]!)) blocks.push([lines[i]!]); + else blocks.at(-1)?.push(lines[i]!); + } + const block = blocks.find(b => b[0]!.trim() === `${gate}:` || b.some(l => /^ name:/.test(l) && l.trim() === `name: ${gate}`)); + if (!block) return `${wf} on ${ref} has no job ${gate}`; + const text = block.join('\n'); + const dirs = ['', ...[...text.matchAll(/working-directory:\s*['"]?([\w.\/-]+)/g)].map(m => `${m[1]!.replace(/^\.?\/+|\/+$/g, '')}/`)]; + const tokens = [...text.matchAll(/[\w.\/-]+/g)].map(m => m[0]).filter(t => t.includes('/') || /\.(sh|ts|js|mjs|cjs|py|rb)$/.test(t)); + const named = [...new Set(tokens.flatMap(t => dirs.map(d => path.posix.normalize(d + t.replace(/^\.?\/+/, ''))).filter(n => !n.startsWith('..'))))]; + const pkg = /\b(bun|npm|pnpm|yarn)\s+(run\s+)?[\w:-]+/.test(text) ? dirs.map(d => `${d}package.json`) : []; + for (const f of [wf, ...(p.skippable!.scripts ?? []), ...named, ...pkg]) { + const a = blob(ref, f); + const b = blob('HEAD', f); + if (a === null && b === null && f !== wf && !p.skippable!.scripts?.includes(f)) continue; + if (a === null || a !== b) return `${f} differs from ${ref}`; + } + return null; +} + +/** + * `profiled` says whether `lane` may have a run profile: an unreadable base counts as profiled, so a lane is + * recorded unverified only once the base shows it has none. `policy: 'head'` reads the branch's profiles instead, only + * for gstack-ship-preflight --simulate, a dry run that records and clears nothing. + */ +export function buildRunCheck(root: string, lane: string | null, allowRepoCommands: boolean, base = resolveBaseBranch(root), policy: 'base' | 'head' = 'base'): RunCheck | { error: string; profiled?: boolean } { + const loaded = policy === 'head' ? loadConfigFromHead(root) : loadConfigFromBase(root, base); if (isConfigUnavailable(loaded)) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is unreadable (${loaded.detail}); run profiles come only from the base`, profiled: true }; if (loaded.errors.length) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is invalid: ${loaded.errors.map(e => `${e.path}: ${e.message}`).join('; ')}`, profiled: true }; const origin = spawnSync('git', ['config', '--get', 'remote.origin.url'], { cwd: root, encoding: 'utf8', timeout: 30_000 }); @@ -32,6 +70,7 @@ export function buildRunCheck(root: string, lane: string | null, allowRepoComman if (!repo) return { error: 'origin is not a github.com repository; run evidence is read from GitHub Actions', profiled: lane === null ? false : lane === REMOTE_GATE_LANE ? !!loaded.config.remote_gate_runs : !!loaded.config.run_profiles[lane] }; return { repo, deps: ghRunDeps(repo, root), failures: ghArtifactDeps(repo), profiles: loaded.config.run_profiles, gateRuns: loaded.config.remote_gate_runs, now: new Date(), + unchanged: p => gateInputsChanged(root, base, p), inventory: (command, tree, selected) => { if (!allowRepoCommands) return { error: 'the inventory command is a pin-file command; pass --allow-repo-commands' }; const at = () => { const fp = wtreeFingerprint({ cwd: root }); return 'tree' in fp ? fp.tree : null; }; diff --git a/lib/restamp.ts b/lib/restamp.ts index be2c6fc23..9eba5f88e 100644 --- a/lib/restamp.ts +++ b/lib/restamp.ts @@ -313,7 +313,7 @@ export function branchAddedLines(cwd: string, since: string, files: string[], to return out; } -const escapeRegExp = (t: string) => t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +export const escapeRegExp = (t: string) => t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); const VERSION_RE = '(?\\d+(?:\\.\\d+){1,3})(?![0-9.])'; /** * The built-in stamp forms as scanners, the version in the `v` group: `(was X)`, `since: X` and `vX`. diff --git a/lib/result-codes-release.ts b/lib/result-codes-release.ts index 2d0a0dddb..6c209131a 100644 --- a/lib/result-codes-release.ts +++ b/lib/result-codes-release.ts @@ -16,6 +16,11 @@ export const RELEASE_RESULT_CODES = { summary: 'the ship policy sets min_gstack_version above this gstack, so this gstack refuses to read it', fix: 'upgrade gstack (bin/gstack-upgrade, or the session install with --for ) to the version named, then rerun', }, + PREGATE_GSTACK_TOO_OLD: { + anchor: 'pregate-gstack-too-old', + summary: 'the pregate policy uses a key this gstack does not know, and the project floor (min_gstack_version in .gstack/ship-policy.json) is above this gstack, so the key belongs to a newer gstack; the policy stays refused', + fix: 'upgrade gstack (bin/gstack-upgrade, or the session install with --for ) to at least the version named, then rerun', + }, QUEUE_NOT_READY: { anchor: 'queue-not-ready', summary: 'the lane is not ready to report: the working tree, HEAD or CI state does not match the pushed PR head', diff --git a/lib/run-evidence.ts b/lib/run-evidence.ts index bc4b576fe..513d8fa2d 100644 --- a/lib/run-evidence.ts +++ b/lib/run-evidence.ts @@ -42,7 +42,11 @@ import * as path from 'node:path'; import { sha256Hex, writeReceipt } from './egress-receipt'; import { runBashSync } from './script-invocation'; -export interface RunProfile { workflow: string; events: string[]; jobs: string[]; artifact: string; max_age_hours?: number; os?: string } +/** + * `skippable` (GST-1190): named jobs the workflow may skip by path; a skip counts only in a pull_request run whose `gate` job + * succeeded, with the workflow, the declared `scripts` and any path the gate's steps name matching origin/ byte for byte. + */ +export interface RunProfile { workflow: string; events: string[]; jobs: string[]; artifact: string; max_age_hours?: number; os?: string; skippable?: { jobs: string[]; gate: string; scripts: string[] } } export interface RemoteGateRuns { profiles: string[]; inventory: string; skipped?: string } export interface ApiRun { id: number; run_attempt: number; status: string; conclusion: string | null; event: string; head_sha: string; @@ -50,7 +54,7 @@ export interface ApiRun { repository?: { full_name: string } | null; head_repository?: { full_name: string } | null; } export interface ApiJob { name: string; conclusion: string | null; completed_at?: string | null } -export interface TreeRecord { run_id?: unknown; run_attempt?: unknown; event?: unknown; sha?: unknown; head_sha?: unknown; tree?: unknown; reused?: unknown; reuse?: unknown; os?: unknown; files?: unknown; tests?: unknown } +export interface TreeRecord { run_id?: unknown; run_attempt?: unknown; event?: unknown; sha?: unknown; head_sha?: unknown; tree?: unknown; reused?: unknown; reuse?: unknown; os?: unknown; files?: unknown; tests?: unknown; skipped_files?: unknown } /** What a red run's reader needs beyond RunDeps (lib/pregate/flaky-exception.ts): every file of one artifact, and a pull request's head. */ export interface RunArtifactDeps { @@ -69,14 +73,15 @@ export interface RunDeps { } export type RunRefusal = 'run_ref_invalid' | 'other_repository' | 'fork_run' | 'workflow_mismatch' | 'event_mismatch' | 'run_in_progress' - | 'run_not_success' | 'job_missing' | 'job_ambiguous' | 'job_not_success' | 'artifact_missing' | 'artifact_mismatch' + | 'run_not_success' | 'job_missing' | 'job_ambiguous' | 'job_skipped' | 'job_not_success' | 'artifact_missing' | 'artifact_mismatch' | 'artifact_reused' | 'tree_mismatch' | 'newer_run_vetoes' | 'run_changed' | 'run_stale' | 'os_mismatch' | 'api_error'; -export interface RunEvidence { run_id: number; run_attempt: number; head_sha: string; tree: string; profile: string; event: string; finished_at: string; files: string[]; tests: number | null } +/** `skipped`: the files whose every test skipped on the runner (tree.json `skipped_files`, GST-1190); undefined when an older tree.json does not say. */ +export interface RunEvidence { run_id: number; run_attempt: number; head_sha: string; tree: string; profile: string; event: string; finished_at: string; files: string[]; tests: number | null; skipped?: string[]; skipped_jobs?: string[] } export type RunVerdict = { ok: true; evidence: RunEvidence } | { ok: false; reason: RunRefusal; detail: string }; const SHA = /^[0-9a-f]{40}$/; const RUN_EVENTS = ['push', 'workflow_dispatch', 'pull_request']; -const PROFILE_KEYS = ['workflow', 'events', 'jobs', 'artifact', 'max_age_hours', 'os']; +const PROFILE_KEYS = ['workflow', 'events', 'jobs', 'artifact', 'max_age_hours', 'os', 'skippable']; const WORKFLOW_FILE = /^[A-Za-z0-9_.-]+\.ya?ml$/; const NAME = /^[A-Za-z0-9_.\/-]{1,128}$/; export const DEFAULT_MAX_AGE_HOURS = 24; @@ -113,6 +118,14 @@ export function runProfileErrors(profiles: unknown, gateRuns: unknown): Array<{ if (typeof p.artifact !== 'string' || !NAME.test(p.artifact)) errors.push({ path: `${at}.artifact`, message: 'must be the artifact name holding tree.json' }); if ('max_age_hours' in p && (!Number.isInteger(p.max_age_hours) || (p.max_age_hours as number) < 1 || (p.max_age_hours as number) > 168)) errors.push({ path: `${at}.max_age_hours`, message: 'must be an integer from 1 to 168' }); if ('os' in p && (typeof p.os !== 'string' || !/^(Linux|macOS|Windows)$/.test(p.os))) errors.push({ path: `${at}.os`, message: 'must be Linux, macOS or Windows' }); + if ('skippable' in p) { + const s = p.skippable as Record | null; + const names = (v: unknown) => Array.isArray(v) && v.length > 0 && v.length <= 32 && v.every(x => typeof x === 'string' && !!x.trim() && x.length <= 128); + const repoPath = (x: unknown) => typeof x === 'string' && !!x && !x.startsWith('/') && !x.split('/').includes('..') && !/[\0\r\n]/.test(x); + if (!s || typeof s !== 'object' || Array.isArray(s) || Object.keys(s).some(k => !['jobs', 'gate', 'scripts'].includes(k)) || !names(s.jobs) || typeof s.gate !== 'string' || !s.gate.trim() + || (s.jobs as string[]).includes(s.gate as string) || !Array.isArray(s.scripts) || !s.scripts.length || !s.scripts.every(repoPath)) + errors.push({ path: `${at}.skippable`, message: 'must be { jobs: [job names], gate: , scripts: [every repo file the gate reads] }, the gate not among the jobs' }); + } } } if (gateRuns !== undefined && gateRuns !== null) { @@ -135,13 +148,13 @@ export function runProfileErrors(profiles: unknown, gateRuns: unknown): Array<{ const fail = (reason: RunRefusal, detail: string): RunVerdict => ({ ok: false, reason, detail }); /** When a run's newest attempt started. A run vetoes the candidate when it started at or after the second the candidate was created: a red re-run of an older run counts, and re-running the candidate does not hide a failure in between. */ const started = (r: ApiRun): string => r.run_started_at ?? r.created_at ?? ''; -const normalPath = (f: string): string => f.replace(/^(\.\/)+/, ''); +export const normalPath = (f: string): string => f.replace(/^(\.\/)+/, ''); /** * The verdict for run `runRef` under `profile`. `vetoHeads` are further heads * whose newer runs of the same workflow can veto (the consumer's own head). */ -export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; profileName: string; profile: RunProfile; vetoHeads?: string[]; now: Date }): RunVerdict { +export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; profileName: string; profile: RunProfile; vetoHeads?: string[]; now: Date; unchanged?: (p: RunProfile) => string | null }): RunVerdict { const id = parseRunRef(o.runRef, o.repo); if (id === null) return fail('run_ref_invalid', `${JSON.stringify(o.runRef)} is not a run id or a run URL on ${o.repo}`); const p = o.profile; @@ -162,10 +175,22 @@ export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; prof const maxAge = p.max_age_hours ?? DEFAULT_MAX_AGE_HOURS; if (!Number.isFinite(finished)) return fail('api_error', `${at} has no completion time`); if (o.now.getTime() - finished > maxAge * 3600_000) return fail('run_stale', `${at} finished ${finishedAt}, more than ${maxAge} h before now`); + const skippedJobs: string[] = []; for (const name of p.jobs) { const hits = jobs.filter(j => j.name === name); if (hits.length === 0) return fail('job_missing', `${at} has no job named ${name}`); if (hits.length > 1) return fail('job_ambiguous', `${at} has ${hits.length} jobs named ${name}`); + if (hits[0]!.conclusion === 'skipped') { + const skip = p.skippable; + if (!skip?.jobs.includes(name)) return fail('job_skipped', `${at}'s ${name} job was skipped`); + if (run.event !== 'pull_request') return fail('job_skipped', `${at}'s ${name} job was skipped in a ${run.event} run; only a pull_request run's gate diffs against the PR base, so only its skip counts`); + const gate = jobs.filter(j => j.name === skip.gate); + if (gate.length !== 1 || gate[0]!.conclusion !== 'success') return fail('job_skipped', `${at}'s ${name} job was skipped and its gate job ${skip.gate} ${gate.length === 1 ? `concluded ${gate[0]!.conclusion}` : gate.length ? 'is ambiguous' : 'is missing'}`); + const changed = o.unchanged ? o.unchanged(p) : 'no base copy of the workflow was compared'; + if (changed) return fail('job_skipped', `${at}'s ${name} job was skipped, but ${changed}, so this branch decided the skip`); + skippedJobs.push(name); + continue; + } if (hits[0]!.conclusion !== 'success') return fail('job_not_success', `${at}'s ${name} job concluded ${hits[0]!.conclusion}`); } const heads = [...new Set([run.head_sha, ...(o.vetoHeads ?? []).filter(h => SHA.test(h))])]; @@ -195,11 +220,13 @@ export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; prof if (p.os && !oses.includes(p.os)) return fail('os_mismatch', `${at} ran on ${oses.filter(Boolean).map(String).join(', ') || 'an unrecorded OS'}; ${o.profileName} requires ${p.os}`); if (rec.files !== undefined && (!Array.isArray(rec.files) || rec.files.some(f => typeof f !== 'string'))) return fail('artifact_mismatch', `${at}'s ${p.artifact} files is not a list of paths`); if (rec.tests !== undefined && (!Number.isInteger(rec.tests) || (rec.tests as number) < 0)) return fail('artifact_mismatch', `${at}'s ${p.artifact} tests is not a non-negative integer`); + if (rec.skipped_files !== undefined && (!Array.isArray(rec.skipped_files) || rec.skipped_files.some(f => typeof f !== 'string'))) return fail('artifact_mismatch', `${at}'s ${p.artifact} skipped_files is not a list of paths`); const files = ((rec.files ?? []) as string[]).map(normalPath); const tests = rec.tests === undefined ? null : (rec.tests as number); const changed = runChanged(deps, id, run.run_attempt); if (changed) return fail('run_changed', changed); - return { ok: true, evidence: { run_id: id, run_attempt: run.run_attempt, head_sha: run.head_sha, tree: headTree, profile: o.profileName, event: run.event, finished_at: finishedAt!, files, tests } }; + const skipped = rec.skipped_files === undefined ? {} : { skipped: (rec.skipped_files as string[]).map(normalPath) }; + return { ok: true, evidence: { run_id: id, run_attempt: run.run_attempt, head_sha: run.head_sha, tree: headTree, profile: o.profileName, event: run.event, finished_at: finishedAt!, files, tests, ...skipped, ...(skippedJobs.length ? { skipped_jobs: skippedJobs } : {}) } }; } catch (e: any) { return fail('api_error', `reading run ${id} failed: ${String(e?.message ?? e).slice(0, 300)}`); } @@ -219,13 +246,21 @@ export function runInventory(command: string, cwd: string, env?: Record e.files.map(normalPath))); +/** A run's files minus the ones whose every test skipped there (GST-1190): only those ran anything. Older tree.json: all files. */ +export const ranFiles = (e: RunEvidence): string[] => { const skipped = new Set(e.skipped ?? []); return e.files.map(normalPath).filter(f => !skipped.has(f)); }; + +/** + * The inventory files no run covers. `skipped: 'uncovered'` (a carry, which drops tests on a machine where they might run) + * also counts a file every test of which skipped in the run; `'covered'` (the remote gate, a Linux runner like CI) does not. + */ +export function uncoveredFiles(inventory: string[], evidence: RunEvidence[], skipped: 'uncovered' | 'covered' = 'covered'): string[] { + const ran = new Set(evidence.flatMap(e => (skipped === 'uncovered' ? ranFiles(e) : e.files.map(normalPath)))); return inventory.filter(f => !ran.has(normalPath(f))); } /** What a run check reads: the repository, the verifier's dependencies, the base branch's profiles and the clock. */ -export interface RunSource { repo: string; deps: RunDeps; profiles: Record; now: Date } +/** `unchanged` (findCarry only): null when a profile's workflow and gate inputs match the base, else which file differs. */ +export interface RunSource { repo: string; deps: RunDeps; profiles: Record; now: Date; unchanged?: (p: RunProfile) => string | null } export type FromRunVerdict = { ok: true; evidence: RunEvidence; receipt: string } | { ok: false; reason: string; detail: string }; /** diff --git a/lib/ship-bin-common.ts b/lib/ship-bin-common.ts index 51bb7d167..c76e51887 100644 --- a/lib/ship-bin-common.ts +++ b/lib/ship-bin-common.ts @@ -10,6 +10,7 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { fragmentSlug } from './changelog-fragments'; +import { laneMapping, recordCiResult, type ShipLaneMapping } from './ci-carry'; import { isConfigUnavailable, loadConfigFromBase } from './pregate/config'; import { parseUnifiedDiff, versionAtRef } from './restamp'; import { branchSides, stampOnlyLine } from './stamp-only'; @@ -113,7 +114,8 @@ export function releasePaths(policy: Record | null, cwd: string, re * added since the merge base with origin/, and not added by the base since): after a lost slot and a merge of the * base its stale stamp is older than `ref`'s VERSION, so it must be at or newer than the fork point's version instead, * the bound gstack-restamp's stamp_paths stage uses (GST-895; the fork version itself since GST-963). A branch line is - * named by its position, and with a kept VERSION only such a line may move (GST-979, GST-980). + * named by its position, and with a kept VERSION only such a line may move (GST-979, GST-980); on it a declared + * `stamp_placeholders` value may also become the new version inside a `stamp_patterns` template (GST-1026). */ export function stampedFiles(policy: Record | null, cwd: string, ref: string, base: string): string[] { const globs = Array.isArray(policy?.stamp_files) ? policy.stamp_files.filter((s: unknown): s is string => typeof s === 'string') : []; @@ -123,11 +125,13 @@ export function stampedFiles(policy: Record | null, cwd: string, re let after = ''; try { after = extractVersion(fs.readFileSync(path.join(cwd, before.rel), 'utf8'), before.rel); } catch { return []; } if (!after) return []; - const templates = Array.isArray(policy?.stamp_patterns) ? policy.stamp_patterns.filter((s: unknown): s is string => typeof s === 'string') : []; + const strings = (v: unknown): string[] => (Array.isArray(v) ? v.filter((s): s is string => typeof s === 'string') : []); + const templates = strings(policy?.stamp_patterns); + const placeholders = strings(policy?.stamp_placeholders); const diff = git(cwd, ['diff', '--unified=0', '--no-color', '--no-ext-diff', '--no-textconv', '--no-renames', ref]); if (diff.status !== 0) return []; const side = branchSides(cwd, `origin/${base}`, 'HEAD', before.version, ref); - const stamp = (rel: string, removed: string, added: string, at: number): boolean => stampOnlyLine(removed, added, before.version, after, templates, side(rel), at); + const stamp = (rel: string, removed: string, added: string, at: number): boolean => stampOnlyLine(removed, added, before.version, after, templates, side(rel), at, placeholders); return parseUnifiedDiff(diff.stdout).filter(f => f.status === 'M' && !f.binary && !f.modeChanged && matchesAny(globs, f.path) && f.hunks.length > 0 && f.hunks.every(h => h.removed.length === h.added.length && h.removed.every((line, i) => stamp(f.path, line, h.added[i]!, h.removedAt[i]!)))).map(f => f.path); } @@ -171,15 +175,15 @@ export function laneConcurrency(mode: string | undefined, cpus = os.availablePar return cpus <= 4 ? 1 : Infinity; } -export interface LaneResult { label: string; exit: number | null; testsRan: string; log: string; carried?: string; reused?: string; carryNone?: string } +export interface LaneResult { label: string; exit: number | null; testsRan: string; log: string; carried?: string; reused?: string; carryNone?: string; notOwed?: string; here?: string } -/** The base branch's `ship_lanes_from` entry for each lane whose command is a key, byte for byte (the ledger binds its sha256). */ -export function mappedLanes(cwd: string, base: string, list: { label: string; command: string }[]): Map { +/** The base branch's `ship_lanes_from` entry for each lane whose command is a key or a declared alias, byte for byte (the ledger binds its sha256). */ +export function mappedLanes(cwd: string, base: string, list: { label: string; command: string }[]): Map { const loaded = loadConfigFromBase(cwd, base); - const map = new Map(); + const map = new Map(); if (isConfigUnavailable(loaded) || loaded.errors.length) return map; const from = loaded.config.ship_lanes_from; - for (const l of list) if (Object.prototype.hasOwnProperty.call(from, l.command)) map.set(l.label, from[l.command]!); + for (const l of list) { const m = laneMapping(from, l.command); if (m) map.set(l.label, m); } return map; } @@ -189,9 +193,9 @@ export function mappedLanes(cwd: string, base: string, list: { label: string; co */ export function carryLane(cwd: string, base: string, l: { label: string; command: string }): LaneResult { const rec = bin(cwd, 'gstack-evidence', ['record-ci', '--label', l.label, '--base', base, '--', l.command], 600_000); - const got = / tests_ran=(\d+) log=- source=ci runs=(\S+)/.exec(rec.stderr); - if (rec.status === 0 && got) return { label: l.label, exit: 0, testsRan: got[1]!, log: '-', carried: got[2]! }; - return { label: l.label, exit: null, testsRan: 'unknown', log: '-', carryNone: /^CARRY_NONE: (.+)$/m.exec(rec.stderr)?.[1] ?? `record_failed: ${firstLine(rec.stderr)}` }; + const got = recordCiResult(rec.stderr); + if (rec.status === 0 && got.runs) return { label: l.label, exit: 0, testsRan: got.files ? `${got.tests} (files)` : got.tests!, log: '-', carried: got.runs, ...(got.notOwed ? { notOwed: got.notOwed } : {}), ...(got.here ? { here: got.here } : {}) }; + return { label: l.label, exit: null, testsRan: 'unknown', log: '-', carryNone: got.none ?? `record_failed: ${firstLine(rec.stderr)}` }; } /** @@ -201,7 +205,7 @@ export function carryLane(cwd: string, base: string, l: { label: string; command */ export async function runLanes(cwd: string, list: { label: string; command: string }[], concurrency = Infinity, opts: { base?: string; reuse?: boolean } = {}): Promise { const out: LaneResult[] = []; - const mapped = opts.base ? mappedLanes(cwd, opts.base, list) : new Map(); + const mapped = opts.base ? mappedLanes(cwd, opts.base, list) : new Map(); let next = 0; const worker = async () => { while (next < list.length) { @@ -222,7 +226,8 @@ export async function runLanes(cwd: string, list: { label: string; command: stri /** Step 5's lane verdict: a lane passes on its runner count, never exit 0 alone; a carried lane names its CI runs. */ export function laneLine(l: LaneResult): { line: string; ok: boolean } { - if (l.carried) return { line: `LANE: ${l.label} carried run=${l.carried} tests_ran=${l.testsRan}`, ok: true }; + if (l.notOwed) return { line: `LANE: ${l.label} not owed: ${l.notOwed} run=${l.carried}`, ok: true }; + if (l.carried) return { line: `LANE: ${l.label} carried run=${l.carried} tests_ran=${l.testsRan}${l.here ? ` ${l.here}` : ''}`, ok: true }; const zero = l.exit === 0 && l.testsRan === '0'; const state = l.exit !== 0 ? 'FAIL' : zero ? 'ZERO-RUN' : l.testsRan === 'unknown' ? 'pass (count unavailable)' : 'pass'; return { line: `LANE: ${l.label} ${state} exit=${l.exit} tests_ran=${l.testsRan} log=${l.log}${l.reused ? ` reused=${l.reused}` : ''}`, ok: l.exit === 0 && !zero }; diff --git a/lib/ship-policy-stamps.ts b/lib/ship-policy-stamps.ts new file mode 100644 index 000000000..6e0b144da --- /dev/null +++ b/lib/ship-policy-stamps.ts @@ -0,0 +1,35 @@ +/** + * Validation of the stamp policy keys (stamp_patterns templates and stamp_placeholders), split out of ship-policy.ts + * to keep that module under the module-size ratchet. + */ +import type { PolicyError } from './ship-policy'; +import { parseVersion } from './version-source'; + +const MAX_TEMPLATES = 16; +const MAX_TEMPLATE = 128; +const MAX_PLACEHOLDERS = 16; +const MAX_PLACEHOLDER = 32; +export const VERSION_SLOT = '{version}'; + +export function templateErrors(value: unknown, out: PolicyError[]): void { + if (!Array.isArray(value)) { out.push({ code: 'POLICY_INVALID', path: 'stamp_patterns', message: 'must be an array of literal templates' }); return; } + if (value.length > MAX_TEMPLATES) out.push({ code: 'POLICY_CONTAINMENT', path: 'stamp_patterns', message: `more than ${MAX_TEMPLATES} templates` }); + value.forEach((t, i) => { + const at = `stamp_patterns[${i}]`; + if (typeof t !== 'string') { out.push({ code: 'POLICY_INVALID', path: at, message: 'must be a string' }); return; } + if (t.split(VERSION_SLOT).length !== 2) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(t)} must contain exactly one ${VERSION_SLOT}` }); + else if (t.length > MAX_TEMPLATE || /[\0\r\n]/.test(t)) out.push({ code: 'POLICY_CONTAINMENT', path: at, message: `must be one line of at most ${MAX_TEMPLATE} characters` }); + }); +} + +export function placeholderErrors(value: unknown, out: PolicyError[]): void { + if (!Array.isArray(value)) { out.push({ code: 'POLICY_INVALID', path: 'stamp_placeholders', message: 'must be an array of literal placeholders' }); return; } + if (value.length > MAX_PLACEHOLDERS) out.push({ code: 'POLICY_CONTAINMENT', path: 'stamp_placeholders', message: `more than ${MAX_PLACEHOLDERS} placeholders` }); + value.forEach((p, i) => { + const at = `stamp_placeholders[${i}]`; + if (typeof p !== 'string') { out.push({ code: 'POLICY_INVALID', path: at, message: 'must be a string' }); return; } + if (!p.trim() || p.length > MAX_PLACEHOLDER || /[\0\r\n]/.test(p)) out.push({ code: 'POLICY_CONTAINMENT', path: at, message: `must be one non-empty line of at most ${MAX_PLACEHOLDER} characters` }); + else if (p.includes(VERSION_SLOT)) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(p)} must not contain ${VERSION_SLOT}` }); + else if (parseVersion(p)) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(p)} is a version, not a placeholder` }); + }); +} diff --git a/lib/ship-policy.ts b/lib/ship-policy.ts index 3ce64bd8e..15e17895b 100644 --- a/lib/ship-policy.ts +++ b/lib/ship-policy.ts @@ -19,6 +19,8 @@ import { defaultBaseBranch, noDefaultBaseLine } from './default-base'; import { RESULT_CODES, type ResultCodeName } from './result-codes'; import { reviewErrors, type ReviewPolicy } from './ship-policy-review'; import { cmpVersion, parseVersion } from './version-source'; +import { placeholderErrors, templateErrors } from './ship-policy-stamps'; +export { VERSION_SLOT } from './ship-policy-stamps'; export const SHIP_POLICY_PATH = '.gstack/ship-policy.json'; export const SHIP_POLICY_SCHEMA = 'https://github.com/garrytan/gstack/docs/ship-policy.md#schema-v1'; @@ -66,6 +68,8 @@ export interface ShipPolicy { stamp_files: string[]; /** Literal stamp templates with one `{version}` slot, beside the built-in `(was X)`, `since: X`, `vX`; writer and classifier share them. */ stamp_patterns: string[]; + /** Placeholders (`TBD`) a restamp may replace with the new version inside a `stamp_patterns` template, on the branch's own lines only. */ + stamp_placeholders: string[]; /** The repo's own restamp, run first in the staging worktree with the target version; gstack's version writer is skipped. */ restamp_command: string | null; /** Migration numbering: `{ dir, id }` where `id` is a regex with one capture group over the file name. */ @@ -164,6 +168,7 @@ export const DEFAULT_POLICY: ShipPolicy = { maintainers: [], stamp_files: [], stamp_patterns: [], + stamp_placeholders: [], restamp_command: null, migrations: null, gate_command: null, @@ -197,6 +202,7 @@ export const POLICY_EXPLANATIONS: Record = { maintainers: 'GitHub logins treated as maintainers by /review contributor mode; empty uses repository push permission.', stamp_files: 'The repo’s stamp rule: globs of the only files that may differ by a stamp (version file, manifests, plugin trees, the PR’s own since: rows). Empty keeps gstack’s per-file rules; set, it applies before every rule.', stamp_patterns: 'Literal templates with exactly one {version} slot (at most 16, 128 characters each), for example `"version": "{version}"` or `since: \'{version}\'`; the restamp writer and the tree receipt use them beside (was X), since: X and vX.', + stamp_placeholders: 'Literal placeholders (at most 16, one line, 32 characters each, never `{version}` or a version), for example `["TBD"]`: a line the branch added that reads a placeholder in a stamp_patterns template (`since: \'TBD\'`) may become that template at the new version, and finish, the receipt carry and the tree receipt read it as a stamp. Empty (default) accepts no placeholder.', restamp_command: 'The repo’s own restamp, run first in gstack-restamp’s staging worktree (a clean checkout of HEAD) with the target version as its argument and in GSTACK_RESTAMP_VERSION; gstack’s version writer is skipped, output confinement still applies. Needs a clean caller tree and --allow-repo-commands.', migrations: '{ dir, id }: the migration directory and a regex with one capture group for the number in a file name; `gstack-pregate regate` reports MIGRATION_COLLISION against the base branch and every open PR.', gate_command: 'The repo’s full pre-merge gate command. gstack-ship-queue gate-ahead and ready print it for the caller to run (in the background or in CI); gstack never executes it. null when the repo has none.', @@ -240,6 +246,7 @@ export const BUMP_NEVER_KEY_VERSION = '1.91.353.0'; export const BASE_RED_KEY_VERSION = '1.91.374.0'; /** `train.gates` and `train.land` are values of the older train key, so compatErrors checks them by value, as for bump: never. */ export const TRAIN_GATES_KEY_VERSION = '1.92.0.0'; +export const STAMP_PLACEHOLDERS_KEY_VERSION = '1.94.16.0'; export const REVIEW_EXCLUDE_KEY_VERSION = '1.93.33.0'; export const REVIEW_POLICY_KEY_VERSION = '1.95.0.0'; export const POLICY_KEY_INTRODUCED: Record = { @@ -250,6 +257,7 @@ export const POLICY_KEY_INTRODUCED: Record = { train: TRAIN_KEYS_VERSION, review_scope: REVIEW_SCOPE_KEY_VERSION, docs: DOCS_CAPS_KEY_VERSION, changelog_fragments: CHANGELOG_FRAGMENTS_KEY_VERSION, base_red: BASE_RED_KEY_VERSION, review_exclude_paths: REVIEW_EXCLUDE_KEY_VERSION, review: REVIEW_POLICY_KEY_VERSION, + stamp_placeholders: STAMP_PLACEHOLDERS_KEY_VERSION, }; const newer = (a: string, b: string): boolean => cmpVersion(parseVersion(a)!, parseVersion(b)!) > 0; @@ -276,8 +284,6 @@ export interface PolicyError { code: ResultCodeName; path: string; message: stri const MAX_LIST = 64, MAX_PATTERN = 256, MAX_REGEX = 512, MAX_COMMAND = 512; const SHA_RE = /^[0-9a-f]{7,64}$/; -const MAX_TEMPLATES = 16, MAX_TEMPLATE = 128; -export const VERSION_SLOT = '{version}'; const hasDotDot = (p: string): boolean => p.split(/[\\/]+/).some(seg => seg === '..'); @@ -342,17 +348,6 @@ function excludeErrors(value: unknown, out: PolicyError[]): void { }); } -function templateErrors(value: unknown, out: PolicyError[]): void { - if (!Array.isArray(value)) { out.push({ code: 'POLICY_INVALID', path: 'stamp_patterns', message: 'must be an array of literal templates' }); return; } - if (value.length > MAX_TEMPLATES) out.push({ code: 'POLICY_CONTAINMENT', path: 'stamp_patterns', message: `more than ${MAX_TEMPLATES} templates` }); - value.forEach((t, i) => { - const at = `stamp_patterns[${i}]`; - if (typeof t !== 'string') { out.push({ code: 'POLICY_INVALID', path: at, message: 'must be a string' }); return; } - if (t.split(VERSION_SLOT).length !== 2) out.push({ code: 'POLICY_INVALID', path: at, message: `${JSON.stringify(t)} must contain exactly one ${VERSION_SLOT}` }); - else if (t.length > MAX_TEMPLATE || /[\0\r\n]/.test(t)) out.push({ code: 'POLICY_CONTAINMENT', path: at, message: `must be one line of at most ${MAX_TEMPLATE} characters` }); - }); -} - function migrationErrors(value: unknown, out: PolicyError[]): void { const m = value as Record | null; if (!m || typeof m !== 'object' || Array.isArray(m)) { out.push({ code: 'POLICY_INVALID', path: 'migrations', message: 'must be { dir, id } or null' }); return; } @@ -497,6 +492,7 @@ export function validatePolicy(raw: unknown, opts: { gstackVersion?: string } = if ('release_outputs' in obj) listErrors('release_outputs', obj.release_outputs, errors, globError); if ('stamp_files' in obj) listErrors('stamp_files', obj.stamp_files, errors, globError); if ('stamp_patterns' in obj) templateErrors(obj.stamp_patterns, errors); + if ('stamp_placeholders' in obj) placeholderErrors(obj.stamp_placeholders, errors); if ('restamp_command' in obj && obj.restamp_command !== null) { const t = obj.restamp_command; if (typeof t !== 'string' || !t.trim()) errors.push({ code: 'POLICY_INVALID', path: 'restamp_command', message: 'must be a non-empty string or null' }); diff --git a/lib/ship-preflight.ts b/lib/ship-preflight.ts index ed6591ce5..2b483b308 100644 --- a/lib/ship-preflight.ts +++ b/lib/ship-preflight.ts @@ -8,8 +8,12 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; +import { findCarry, laneMapping, mappingCoverage } from './ci-carry'; import { defaultBaseBranch } from './default-base'; +import { isConfigUnavailable, loadConfigFromHead } from './pregate/config'; +import { buildRunCheck } from './pregate/run-check'; import { classifyDiff } from './review-depth'; +import { wtreeFingerprint } from './script-invocation'; import { audit } from './ship-audit'; import { bin, carryNoneLine, firstLine, flagReader, gh, git, hostedOrigin, laneConcurrency, laneLine, lanes, Report, runLanes, shipPolicy, short } from './ship-bin-common'; @@ -20,6 +24,11 @@ export const PREFLIGHT_HELP = `gstack-ship-preflight — /ship Steps 1-3 and Ste --lanes serial|parallel how Step 5 runs its lanes; unset, one at a time on 4 or fewer cores, all at once above --policy-from head Step 5's pregate reads the branch's .gstack/pregate.json, which may only add test lanes (a branch that adds the lane for a file it touches); anything else is PREGATE_HEAD_POLICY_LOOSENS + gstack-ship-preflight --tests-only --base --policy-from head --simulate [--allow-repo-commands] --lane