Merge remote-tracking branch 'origin/main' into capy/ship-queue-train-land

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

# Conflicts:
#	lib/ship-queue-train-cli.ts
#	lib/ship-queue-train-gate.ts
#	lib/ship-queue-train-land.ts
#	lib/ship-queue-train-verify.ts
This commit is contained in:
garrytan committed 2026-10-31 21:48:44 +00:00
commit f7716cc47a
39 files changed
+2657 -183

No files matched your search

+82
View File
@@ -1,5 +1,87 @@
# Changelog
## [1.92.0.0] - 2026-10-31
**Merge trains work for any repository, not just gstack.** Until now a train needed gstack's own eval workflow, gated only its tip and needed a bisect to find a culprit, so peer crews like gbrain and Truther merged one PR at a time and folded main-red fixes together by hand (GSTA-157).
### Added
- **`gates` in the train block** (docs/ship-policy.md#train-gates, `TRAIN_GATES_KEY_VERSION`): `workflows` gates dispatch the repository's own CI on every prefix ref (`train-prefix/<nonce>/t<i>`) by nonce and need every named job green at that exact commit, with one rerun of failed jobs (a pass is GREEN with a FLAKY line); `command` gates run a test command on a Ubicloud VM per prefix under one spend cap per gate for the whole train (`train init` prices it for a full train and one rerun: $3 at 1800 s). A workflow is sent at most twice per prefix, and one build or verdict at a time writes a train. Every prefix has its own verdict, so the first red prefix after a green one names the culprit and the slots before it can still land.
- **`train land --train <nonce> --pr <n>` for these trains**: `direct` merges the pin itself (`gh pr merge --match-head-commit`) when branch protection is not strict and the pin's checks are green; `update` commits the gated tree on [pin, main] first. `train.land` sets the merge method and required checks; `rebase` always refuses, since a replayed tree is not the gated one. A workflows gate's `post_merge_checks` must succeed on main (`verify --post-merge <sha> --checks <names>`) before the next slot lands.
- **`train watch --land`** waits for every prefix run, reads the verdict and lands each landable slot in order.
- **`train init` and `train doctor`** print a train block from branch protection and the repository's workflows, plus the workflow snippet each gate needs, and check a live block, including a command cap too low for a full train. Neither writes anything.
- **Base-red PRs ride a train**: `ready --base-red --fixer <pr>` READYs a PR whose /ship paused on a red base, and `ready --fixes-base-red <cases>` READYs its fixer. The train seats the fixer at slot 1, allows main red on exactly those cases, gates every conditional PR on its own prefix and lands it only through an update commit.
- Result codes `QUEUE_TRAIN_BASE_RED`, `QUEUE_TRAIN_SPEND_CAP`, `QUEUE_TRAIN_LAND_METHOD`, `QUEUE_TRAIN_INIT_REFUSED`, `QUEUE_TRAIN_DOCTOR_FINDINGS`, `QUEUE_BASE_RED_NO_TRAIN` and `SHIP_BASE_RED_FIXER`, each with a docs/troubleshooting.md entry; captain/templates/merge-train-peer.md for a peer crew turning one on.
### Changed
- gstack's own train block (no `gates`) and every recorded train behave exactly as before. A train with gates refuses `train bisect` and `train rebuild`, since every prefix already has its verdict, and `verify --train` names `train land` instead. `train prune` keeps an attributed train while slots before its culprit are still unmerged.
### Tests
- `test/ship-queue-train-generic.test.ts`: policy validation (and gstack's own policy file unchanged), prefix verdicts including FIXES_BASE_RED, run classification, test records, base-red admission, workflow facts; on fixture repos: dispatch by nonce on every prefix, direct and update landing, rebase and fork refusals, an attributed culprit, the one-rerun FLAKY path, red on main, command gates through a fake ubi-runner with `UBI_REAP=0` and the spend cap, `ready --base-red`, a fixer train landed by `watch --land`, and `train init`/`train doctor`; plus the review fixes: unreadable protection, a dispatch no run adopts, a held lease, a run another process owns, output past the 1 MB log tail, rebase, post-merge checks and an INCOMPLETE train rebuilt on the same pins.
## [1.91.379.0] - 2026-10-31
**/ship's review readers now review the checkout /ship ships, even when it runs from a worktree on a shared machine.** Readers start as subagents on `machine: "shared"`, so each one opens in the machine's working directory. Their briefs never named a folder, so a /ship run from a worktree had its readers review whatever branch the primary checkout held. GBrain's crew saw a specialist return four findings in files outside the PR (GBRA-218, GST-482). A clean review of the wrong tree would have looked the same as a clean review of the right one.
### Fixed
- **Every reader's brief ends with a workspace paragraph** naming the checkout under review (the toplevel of the directory the bins run in). It tells the reader to start each command with `cd <checkout> && `, to give every file tool an absolute path under the checkout, and to make the first line of its final reply `REVIEWED_TREE: <HEAD^{tree}> <toplevel>`, measured there. The paragraph comes after the section text, so the rest of the brief is the same bytes as before. It covers the specialists, the batched specialist, the Red Team, the targeted reader, fix-check, Step 11's native and outside voices, and audit readers passed as extras (Steps 7, 8 and 14.5, and Step 9.2.1's QA reader). An audit reader's brief is now written to `briefs/` as its prompt file plus the paragraph, and its prompt file is not changed.
- **`gstack-review-dispatch collect` checks that line against the fingerprint recorded when the brief was rendered.** A missing line, or one naming another checkout, prints `COLLECT_WRONG_TREE`; a line from the right checkout at a later commit prints `COLLECT_TREE_MOVED`. Either way it exits 1, merges nothing, and `verify` reports the brief `NOT_COLLECTED`. With `--manifest`, the reader becomes `failed`, so a rerun stops on it instead of collecting the same reply again. An outside voice whose reply fails the check and holds no finding is recorded as unavailable coverage, never clean; one with findings is refused like any other reader, so none is lost. An audit reader gets the same check when the pass marks it collected, and that now needs `--output`.
- **Unavailable outside-voice output never counts as a review in `verify`**, even when its text carries a `Recommendation:` line.
### Tests
- `test/review-pass.test.ts`: a primary checkout and a worktree on another branch, with readers starting in the primary checkout. A reader that reviews the primary checkout, or names none, is refused and nothing it said is merged. One that follows its brief passes, and every brief names the worktree. An audit reader from the wrong checkout is refused. A rerun stops on the failed reader with one output event. A worktree at the same commit is told apart by its toplevel. A commit between render and reply is `COLLECT_TREE_MOVED`. A wrong-tree outside voice with no finding is `UNAVAILABLE`. All eight fail on the previous dispatch code.
- `test/review-dispatch.test.ts`: every role's brief is the old render byte for byte, followed by the paragraph. A list-item, heading, bold or backticked `REVIEWED_TREE` line counts, the path keeps its own characters, and the first line wins over one quoted later. An outside voice with findings and no line is refused. A render outside a readable checkout refuses, and a ledger rendered before this change collects as before.
## [1.91.378.0] - 2026-10-31
**The CSO runtime image workflow passes again after Docker moved its `scout-sbom-indexer:1` tag.** All 10 `validate-native` jobs in Stage CSO Runtime Images were failing with `SOURCE_TAG_MOVED: docker.io/docker/scout-sbom-indexer:1` on every PR that touched CSO paths. Docker pushed release 1.29.0 to the floating `:1` tag on 2026-10-29. The reviewed index `sha256:4b67f29e…` is Docker's 1.24.0 release, and the `1.24.0` tag still points at it. So the SBOM generator source now names `:1.24.0`. The pinned digests and the build-inputs revision are byte-identical, no requalification trigger changes, and the 10 qualified runtimes stay current.
### Changed
- **`lib/cso/images/build-inputs.json`**: `sbomGenerator.source` is `docker.io/docker/scout-sbom-indexer:1.24.0` instead of `:1`.
- **`scripts/cso-image-matrix.ts`**: every pinned family (SBOM generator, base images, uv) must name an exact release in its source tag: `<major>.<minor>.<patch>`, or `<major>.<minor>` for the PostgreSQL base, whose releases have two parts. A floating major or minor tag (`:1`, `:1.24`, `python:3.13-slim`), `latest` or another alias fails with `FLOATING_<FAMILY>_SOURCE` before any build. An exact tag can still be re-pushed upstream (Docker's official images rebuild them when their Debian base changes), and `SOURCE_TAG_MOVED` still reports that. Moving the generator to 1.29.0 is a separate change that needs a new build-inputs revision and a requalified catalog.
### Tests
- `test/cso-distribution.test.ts`: `:1`, `:1.24`, `:1.24-slim`, `:latest` and `:stable` are refused for the SBOM generator and for a base image; PostgreSQL accepts `17.2` but not `17`, Python refuses `3.13-slim-bookworm`; and the committed inputs build all 10 rows with the `1.24.0` source.
## [1.91.377.0] - 2026-10-31
**A passed GitHub Actions run can now clear the remote test gate for the same tree, so a repository stops running its full suite twice per slot.** gbrain paid a 30–45 minute Ubicloud gate and then an 18–20 minute full-ci dispatch on the same tree in every serial slot (GBRA-108: #7872's slot took 5.6 h). `gstack-pregate clear --remote pregate.json/remote_gate --run <id> --run <id>` accepts one verified run per profile the base branch declares. gstack checks each run against the runner's own record, never against a pasted claim. It reads the GitHub API's run and jobs, the tree git computes for the run's head, and the `tree.json` the run uploaded. Nothing is skipped: the tests still run once, in CI. A repository that declares no profiles is unchanged.
### Added
- **`gstack-pregate clear --run <id|url>`** (repeatable; `lib/run-evidence.ts`, docs/pregate.md#run-evidence). A run is accepted only when it belongs to this repository, ran a `push` or `workflow_dispatch` event of the profile's workflow, and its newest attempt succeeded with every named job passing exactly once. Its `tree.json` must name the run, attempt, event and head and record the head's tree, and it must be fresh (`max_age_hours`, default 24) and on the profile's OS. It is refused if any run of that workflow on the head that started at or after it failed, was cancelled or is still running, or if the run changed attempts while being checked. All runs must be on one tree: pregate.json's, or a stamp-only predecessor under the existing tree-receipt rule. Their `files` must cover the obligation's selected tests plus every file the declared `inventory` command lists for that tree. Refusals print `REUSE: refused reason=<code>`; an accepted clear prints `REUSE: accepted`, and the receipt records `runs=<id>/<attempt> profiles= heads= verified=`.
- **`run_profiles` and `remote_gate_runs`** in `.gstack/pregate.json`, read only from `origin/<base>`. A user-owned policy that sets them is refused, because nothing would read them there. `gstack-pregate clear` clears a lane named by a profile only through verified runs, so its `--run-url` is verified the same way, and `--bundle-lane` is refused. `gstack-ship-queue ready --dispatch-ci` still records a plain run URL for such a lane, and `verify` does not yet re-check it (docs/pregate.md#run-evidence). When the base config cannot be read, any lane's `--run-url` or `--bundle-lane` clear refuses, since the base may profile it. When the GitHub origin cannot be read, a profiled lane refuses rather than fall back to an unverified record.
### Changed
- **`gstack-pregate verify`** stays offline. It also refuses a run receipt recorded for another tree unless its stamp-only substitution is recorded.
### Tests
- `test/run-evidence.test.ts`: every refusal reason, including the re-run cases (a red re-run of an older run vetoes; re-running the candidate does not hide a failure in between), a run that changes attempts while being checked, malformed counts and paths, and profile validation.
- `test/pregate-clear-runs.test.ts`: on a fixture repository, clear on the same tree, verify offline, and bind the ship receipt. Also covers a stamp-only predecessor, a non-stamp difference, uncovered files, runs on different trees, base mismatch, repeated runs, the profile-to-run assignment, the profiled-lane matrix (`--run-url`, `--bundle-lane`, `--run`), a non-GitHub origin failing closed, an origin that `url.insteadOf` rewrites to a proxy (Capy's git.capy.ai) still read as the GitHub repository, and an unreadable base config refusing `--run-url` on every lane. The `--run-url` fallback test failed before the fix.
## [1.91.376.0] - 2026-10-31
bump: minor
---
**A remote verify now survives a dead local runner, and a long verify can resume from the tiers that already passed (GSTA-158 P3).** Before, a Capy machine that went to sleep or a runner that got killed lost the whole remote run, and the paid VM ran on until its TTL. Now `ubi-runner run` and `verify` run each phase detached on the VM, and running the same command again reattaches to the live VM. A verify can be split into tiers, and a rerun skips the tiers that passed.
### Added
- **Reattach by run key.** The key covers the checkout's content, the command, setup, env, a hash of each `--pass` value, size, image, origin/HEAD and the runner itself. A rerun prints `ubi-runner: reattached to <vm> created_at=<epoch> key=<k12>`, replays what the phase already printed and follows it from there. A finished setup is not rerun. `UBI_ATTACH_BUSY` means another live runner owns the VM. Exit 125 `UBI_ATTACH_LOST` means the remote session died without an exit status.
- **Reattach only while the TTL lasts.** A reattach needs the VM's remaining TTL to cover the run still to do (each unfinished phase's `UBI_RUN_TIMEOUT` bound) plus `UBI_REATTACH_MARGIN` (900 s). Otherwise the runner prints `UBI_REATTACH_TTL_SHORT` with the numbers, destroys the old VM and starts a fresh one. Tier checkpoints carry over to it.
- **Verify tiers.** `.gstack/ubicloud.json` can list `tiers: [{name, command}]` in place of `command`. Each tier writes a start, pass or fail checkpoint under `UBI_CHECKPOINT_DIR`. A rerun prints `UBI_VERIFY_RESUME:` per tier and reuses passes younger than `UBI_CHECKPOINT_MAX_AGE` whose logs still hash. It reruns a recorded fail only with `--rerun-failed`. The last line is `UBI_VERIFY: …`.
- **`attach NAME`, `run --probe` and `--key-input`.**
### Changed
- **HUP once a phase is live detaches** (`UBI_DETACHED`, exit 129, `UBICLOUD: left=1 … detached=1`): the VM keeps running under its TTL. INT and TERM still tear it down.
- **`gstack-pregate --stage tests --remote` reattaches** to the VM a killed run left. It bills `spent=` from the original create time, and a cap that is already spent refuses with `PREGATE_REMOTE_CAP_SPENT`. The `PREGATE_REMOTE:` receipt is unchanged.
## [1.91.375.0] - 2026-10-31
**A /ship subagent now stops once its READY is sent, so CI and review callbacks on a frozen head stop waking it.** Finished /ship subagents kept getting woken by checks and reviews on a PR they had already READY'd. Each wake cost tokens, and some acted on a head that was supposed to be frozen. The parent now subscribes to the PR first, so red checks and human review still reach an owner, and then stops the subagent.
### Changed
- **`capy/skill/gstack/SKILL.md`** (Plan, approve, ship): the parent subscribes to the PR when the /ship subagent opens it and keeps that subscription until the PR merges or closes. It stops the subagent at READY and owns the PR from then on. A stopped /ship subagent woken by a check, review or merge notice reports one line (what woke it, its pin, the PR head) and ends the turn, while a message from its parent is an instruction it carries out.
- **`docs/captain.md`** (the lane's /ship sequence): the host-neutral rule (the parent owns red checks, review, the slot stamp, re-READY and receipt carry after READY), with a pointer to the Capy mechanics. **`captain/templates/brief.md`**: one pointer line.
- **`test/fixtures/context-budget.json`**: the Capy skill's eager ceiling moves from 5136 to 5312 for the 176 tokens it grew (5136 to 5312, measured with `lib/context-bill`), set at the captured value as main's last raise was, rather than a fresh 10% headroom.
## [1.91.374.0] - 2026-10-31
**/ship pauses on a red base instead of failing.** When main is red for a reason outside a PR, /ship's Step 5 used to fail or ask the fix/TODO/blame/skip question, and unattended lanes either burned runs or held by hand. Truther's two main-red fixes (#815 and the cover-contract branch) each failed only on the other's broken cases, so neither could pass /ship alone until a Captain folded them into #818 by hand (GSTA-157, from the 2026-10-29 speed feedback).
+1 -1
View File
@@ -1 +1 @@
1.91.374.0
1.92.0.0
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.91.374.0 — regenerate/re-copy after upgrading gstack
# gstack digest v1.92.0.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
+46 -7
View File
@@ -6,7 +6,7 @@
* [--allow-repo-commands] [--policy-from base|head] [--out <pregate.json>]
* [--timeout <s>] [--stall-timeout <s>] [--remote --spend-cap <usd>] [--run <id>] [--explain] [--json]
* gstack-pregate verify [--out <pregate.json>] [--json] publication check: current tree, no fail/incomplete, obligations cleared
* gstack-pregate clear --remote <workflow.yml/job> --run-url <url> | --bundle-lane <label> | --receipt <file> [--out <pregate.json>]
* gstack-pregate clear --remote <workflow.yml/job> --run-url <url> | --run <id|url> [--run …] [--allow-repo-commands] | --bundle-lane <label> | --receipt <file> [--out <pregate.json>]
* gstack-pregate lanes [--explain] the lane list and platform tiers read from .github/workflows
* gstack-pregate validate | init [--explain] [--force] the working-tree .gstack/pregate.json
* gstack-pregate sealed [--range <a..b>] [--remote <name>] [--base <branch>] [--json]
@@ -27,14 +27,17 @@ import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { EXIT, exitTableHelp, formatResult, type PregateFile } from '../lib/headless-artifacts';
import { PREGATE_CONFIG_PATH, explainConfig, loadConfigFromHead } from '../lib/pregate/config';
import { PREGATE_CONFIG_PATH, explainConfig, isConfigUnavailable, loadConfigFromBase, loadConfigFromHead } from '../lib/pregate/config';
import { describeDraft, draftConfig } from '../lib/pregate/init-draft';
import { buildContext, defaultUserConfigPath, describeRefusal, loadedDotenvFiles } from '../lib/pregate/context';
import { CHECKS, CHECK_IDS, PREGATE_OUT_REL, PREGATE_TIER2, clearDecision, clearObligation, readPregateFile, renderTable, runPregate, verifyPregate, writePregateFile } from '../lib/pregate/index';
import { CHECKS, CHECK_IDS, PREGATE_OUT_REL, PREGATE_TIER2, clearDecision, clearObligation, readPregateFile, renderTable, runPregate, verifyPregate, writePregateFile, type RunCheck } from '../lib/pregate/index';
import { ghRunDeps, runInventory } from '../lib/run-evidence';
import { wtreeFingerprint } from '../lib/script-invocation';
import { describeLane, discoverLanes } from '../lib/pregate/workflows';
import { REMOTE_GATE_LANE } from '../lib/pregate/checks/lanes';
import { describeSubstitution } from '../lib/pregate/remote-receipt';
import { ensureTmpExcluded } from '../lib/restamp';
import { normalizeRemoteUrl } from '../lib/ship-queue-train-record';
import { RESULT_CODES, resultCodeSuffix, resultLine } from '../lib/result-codes';
import { DEFAULT_STALL_TIMEOUT_S } from '../lib/pregate/context';
import { cancelActive } from '../lib/pregate/supervise';
@@ -52,7 +55,11 @@ const HELP = `gstack-pregate — cheap checks before anything paid or remote (do
--receipt <file> (the gate's PREGATE_REMOTE: line, checked against the tested tree; a tree that
differs from pregate.json's only in the base branch's stamp rule is accepted and recorded as a
substitution, docs/pregate.md#remote-gate-stamp-only); a
test_backend/<backend> run URL is recorded as attested, not validated
test_backend/<backend> run URL is recorded as attested, not validated. --run <id|url> (repeatable)
verifies GitHub Actions runs against the base branch's run_profiles instead (docs/pregate.md#run-evidence):
one run of the profile named after the lane, or for pregate.json/remote_gate one run per
remote_gate_runs profile covering the gate's inventory (needs --allow-repo-commands); where a lane has a
profile, --run-url is verified the same way
lanes print every CI lane with its platform tier (from .github/workflows/*.yml)
validate | init the working-tree ${PREGATE_CONFIG_PATH}
sealed [--range <a..b>] [--remote <name>] [--base <branch>]
@@ -103,7 +110,7 @@ function flag(args: string[], name: string): string | undefined { const i = args
const VERB_FLAGS: Record<string, [string[], string[]]> = {
run: [['--stage', '--only', '--base', '--policy-from', '--out', '--timeout', '--stall-timeout', '--spend-cap', '--run'], ['--allow-repo-commands', '--explain', '--json', '--remote']],
verify: [['--out'], ['--json']],
clear: [['--remote', '--run-url', '--bundle-lane', '--receipt', '--out'], []],
clear: [['--remote', '--run-url', '--run', '--bundle-lane', '--receipt', '--out'], ['--allow-repo-commands']],
lanes: [[], ['--explain', '--json']],
validate: [[], ['--json']],
init: [[], ['--explain', '--force']],
@@ -218,19 +225,51 @@ function cmdClear(root: string, rest: string[]): number {
const out = outPath(root, rest);
const read = fs.existsSync(out) ? readPregateFile(out) : { file: null, errors: [] };
if (!read.file) { process.stderr.write(resultLine('gstack-pregate clear', 'PREGATE_STALE', `${out} is missing or invalid; run gstack-pregate first`) + '\n'); return EXIT.fail; }
const decision = clearDecision(read.file, lane, { runUrl: flag(rest, '--run-url'), bundleLane: flag(rest, '--bundle-lane'), receiptPath: flag(rest, '--receipt') }, root);
const runs = rest.flatMap((a, i) => a === '--run' ? [rest[i + 1]!] : []);
const runUrl = flag(rest, '--run-url');
const bundleLane = flag(rest, '--bundle-lane');
const checked = runs.length || runUrl || bundleLane ? runCheck(root, lane, rest.includes('--allow-repo-commands')) : undefined;
if (checked && 'error' in checked && (runs.length || checked.profiled)) { process.stderr.write(`gstack-pregate clear: ${checked.error}${checked.profiled ? `; ${lane} may have a run profile on the base branch, so it clears only with a verified --run` : ''}\n`); return EXIT.fail; }
const evidence = checked && !('error' in checked) ? checked : undefined;
const decision = clearDecision(read.file, lane, { runUrl, runs, bundleLane, receiptPath: flag(rest, '--receipt'), evidence }, root);
if ('refused' in decision) {
if (decision.refused === 'usage') { process.stderr.write(`gstack-pregate clear: ${decision.message}\n`); return EXIT.usage; }
process.stderr.write(`gstack-pregate clear: receipt refused — ${decision.message}; fix: ${decision.fix ?? RESULT_CODES.PREGATE_RECEIPT_INVALID.fix} ${resultCodeSuffix('PREGATE_RECEIPT_INVALID')}\n`);
return EXIT.fail;
}
const file = clearObligation(read.file, lane, decision.receipt, `gstack-pregate clear ${new Date().toISOString()}`, decision.substitution);
const current = readPregateFile(out).file;
if (!current || current.tree !== read.file.tree || current.generated_at !== read.file.generated_at) { process.stderr.write(resultLine('gstack-pregate clear', 'PREGATE_STALE', `${out} changed while ${lane} was being verified; rerun the clear against the new file`) + '\n'); return EXIT.fail; }
const file = clearObligation(current, lane, decision.receipt, `gstack-pregate clear ${new Date().toISOString()}`, decision.substitution);
writePregateFile(root, out, file);
if (/ runs=\S+ profiles=/.test(decision.receipt)) process.stdout.write(`REUSE: accepted ${lane} ${decision.receipt}\n`);
if (decision.substitution) process.stdout.write(`PREGATE_RECEIPT_SUBSTITUTED: ${describeSubstitution(decision.substitution)}\n`);
process.stdout.write(`PREGATE_CLEARED: ${lane} receipt=${decision.receipt} verdict=${file.verdict}${decision.attested ? ` (attested, not validated: ${lane} has no machine-checkable receipt)` : ''}\n`);
if (checked && 'error' in checked) process.stderr.write(`gstack-pregate clear: run profiles unavailable (${checked.error}); ${lane} was recorded without run verification\n`);
return EXIT.ok;
}
/** The run-evidence inputs for `clear --run`: the base branch's run profiles, this repository on GitHub, and the gate's inventory command (a pin-file command, so only with --allow-repo-commands). */
function runCheck(root: string, lane: string, allowRepoCommands: boolean): RunCheck | { error: string; profiled?: boolean } {
const base = resolveBaseBranch(root);
const loaded = loadConfigFromBase(root, base);
// Unknown profiles are treated as present: a lane is cleared unverified only once the base shows it has no profile.
if (isConfigUnavailable(loaded)) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is unreadable (${loaded.detail}); run profiles come only from the base`, profiled: true };
if (loaded.errors.length) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is invalid: ${loaded.errors.map(e => `${e.path}: ${e.message}`).join('; ')}`, profiled: true };
const repo = /^https:\/\/github\.com\/(.+)$/.exec(normalizeRemoteUrl(gitOut(root, ['config', '--get', 'remote.origin.url']) ?? ''))?.[1];
if (!repo) return { error: 'origin is not a github.com repository; run evidence is read from GitHub Actions', profiled: lane === REMOTE_GATE_LANE ? !!loaded.config.remote_gate_runs : !!loaded.config.run_profiles[lane] };
return {
repo, deps: ghRunDeps(repo, root), profiles: loaded.config.run_profiles, gateRuns: loaded.config.remote_gate_runs, now: new Date(),
inventory: (command, tree) => {
if (!allowRepoCommands) return { error: 'the inventory command is a pin-file command; pass --allow-repo-commands' };
const at = () => { const fp = wtreeFingerprint({ cwd: root }); return 'tree' in fp ? fp.tree : null; };
if (at() !== tree) return { error: `the working tree is not pregate.json's tree ${tree.slice(0, 12)}; the inventory must list that tree's gate files` };
const listed = runInventory(command, root);
if (at() !== tree) return { error: `the working tree changed while ${command} ran` };
return listed;
},
};
}
function gitOut(root: string, args: string[]): string | null {
const r = spawnSync('git', args, { cwd: root, encoding: 'utf8', timeout: 30_000 });
return r.status === 0 ? r.stdout.trim() : null;
+42 -7
View File
@@ -19,7 +19,7 @@ import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { EXIT, exitTableHelp } from '../lib/headless-artifacts';
import { BRIEF_ROLES, BriefError, briefFile, promptHash, renderBrief, sectionTexts, type BriefRole, type Slots } from '../lib/review-brief';
import { BRIEF_ROLES, BriefError, briefFile, checkReviewedTree, checkoutRoot, checkoutTree, promptHash, renderBrief, sectionTexts, withWorkspace, type BriefRole, type Slots, type TreeCheck } from '../lib/review-brief';
import { cleanMarkers, collect, mayBeUnavailable, parseFindings, readJsonl, readLedger, readerKey, recordJudgment, recordSent, appendEvent, verify, type JudgedLine, type MergedLine, type VerifyRow } from '../lib/review-ledger';
import { buildManifest, chunkCount, chunkOf, manifestPath, markReader, readManifest, renderStage, writeManifest, STAGES, type Extra, type Manifest, type PassReader, type ReaderState, type Stage } from '../lib/review-pass';
import type { Dispatch } from '../lib/review-depth';
@@ -37,15 +37,19 @@ const HELP = `gstack-review-dispatch — verbatim reviewer briefs and loss-free
--fix-list <f> --hunks <f> --tests <f> [--callers <f>] (fix-check)
--classes <money,gate,auth,secret> (targeted: the plan's TARGETED classes; the brief is review/specialists/targeted.md;
--diff-base must be the merge base with --base, default origin/HEAD's branch, on every pass)
writes <out>/<role>[-<name>].prompt.md, records it in the ledger (default <out>/review-ledger.jsonl) and prints
writes <out>/<role>[-<name>].prompt.md, the brief followed by the workspace paragraph naming the toplevel of the
checkout it runs in (the one its diff and merge base come from), records it in the ledger (default <out>/review-ledger.jsonl) and prints
BRIEF: reader=<role/name> sha256=<h> bytes=<n> file=<path>. Pass that file's content as the reader's prompt, unchanged.
sent --ledger <f> --role <role> [--name <n>] --sent <file>
the exact prompt a dispatch passed: BRIEF_SENT: … match, or BRIEF_MISMATCH: … (exit 1; the pass is not reviewed)
collect --ledger <f> --role <role> [--name <n>] --output <file> --merged <merged.jsonl>
collect --ledger <f> --role <role> [--name <n>] --output <file> --merged <merged.jsonl> [--manifest <f>]
the reader's raw output: hashes it and appends each finding to merged.jsonl with its label as written. A reply with
no finding must carry its brief's clean marker (NO FINDINGS; the Recommendation: line for the voices); without one it is
COLLECT_NOT_REVIEWED (exit 1): a refusal, an empty reply or a reader that never saw the diff, and verify reports NOT_COLLECTED
until the brief is re-sent and answered with a review, which voids it. --unavailable '<reason>' (the outside voice only; refused when the
until the brief is re-sent and answered with a review, which voids it. A reply whose REVIEWED_TREE line is missing or names
another checkout or tree than its brief was rendered on is COLLECT_WRONG_TREE (COLLECT_TREE_MOVED when only the tree
moved; exit 1, nothing merged, verify reports NOT_COLLECTED, and with --manifest <f> the reader becomes failed); the
outside voice's, when it holds no finding, is recorded as unavailable coverage instead --unavailable '<reason>' (the outside voice only; refused when the
output holds findings) records its failure text as missing coverage: COLLECT_UNAVAILABLE, and verify prints UNAVAILABLE and unavailable=N, never a review
judge --ledger <f> --merged <merged.jsonl> --judged <judged.jsonl>
judged lines {"id","severity","reason"?,"duplicate_of"?}: every merged id kept at its severity or above (a
@@ -68,6 +72,7 @@ const HELP = `gstack-review-dispatch — verbatim reviewer briefs and loss-free
chunk --manifest <f> --reader <r> --index <i>
BRIEF_CHUNK: reader=… index=i of=n sha256=…, then one JSON string line of at most 65,536 brief characters
mark --manifest <f> --reader <r> --state <intent|created|queued|collected|failed> [--path <p>] [--output <f>] [--error <e>]
an audit reader is collected with --output, whose REVIEWED_TREE line must match its render (else failed, COLLECT_WRONG_TREE)
mark --manifest <f> --before <tokens>,<credits>
mark --manifest <f> --log-start <key> --token <t> records a review-log start token once, so a rerun starts only what is missing
reader states move forward only; intent is recorded before the subagent is created, its path right after
@@ -113,18 +118,23 @@ function render(): number {
classes: flag('--classes'), brief,
fix_list: readFile(flag('--fix-list'), 'fix list'), hunks: readFile(flag('--hunks'), 'hunks'), tests: readFile(flag('--tests'), 'tests'), callers: readFile(flag('--callers'), 'callers'),
};
const text = renderBrief(role, slots, sectionTexts(ROOT));
const workdir = checkoutRoot();
const tree = checkoutTree(workdir);
if (!tree) throw new BriefError(`cannot read HEAD^{tree} and the toplevel of ${workdir}; a brief names the checkout its reply must prove`);
const text = withWorkspace(renderBrief(role, slots, sectionTexts(ROOT)), workdir);
fs.mkdirSync(out, { recursive: true, mode: 0o700 });
const file = path.join(path.resolve(out), briefFile(role, name));
fs.writeFileSync(file, text, { mode: 0o600 });
const ledger = flag('--ledger') ?? path.join(out, 'review-ledger.jsonl');
const sha = promptHash(text);
const targeted = brief === undefined ? {} : { classes: parseClassList(flag('--classes')!), rules: TARGETED_RULES, brief_sha: targetedBriefSha(checklist), diff_base: diffBase ?? null, patch_id: diffBase ? targetedPatchId(process.cwd(), diffBase) : 'none' };
appendEvent(ledger, { event: 'rendered', reader: readerKey(role, name), role, name: name ?? null, file, sha256: sha, bytes: Buffer.byteLength(text), ...targeted });
appendEvent(ledger, { event: 'rendered', reader: readerKey(role, name), role, name: name ?? null, file, sha256: sha, bytes: Buffer.byteLength(text), workdir, tree, ...targeted });
process.stdout.write(`BRIEF: reader=${readerKey(role, name)} sha256=${sha} bytes=${Buffer.byteLength(text)} file=${file}\n`);
return EXIT.ok;
}
const wrongTree = (reader: string, t: TreeCheck) => `${t.moved ? 'COLLECT_TREE_MOVED' : 'COLLECT_WRONG_TREE'}: reader=${reader} reported=${JSON.stringify(t.reported)} expected=${JSON.stringify(t.expected)} (${t.moved ? 'the checkout moved after this brief was rendered' : 'the reply did not review the checkout its brief named'}; nothing merged; restore it with a fresh pass under a new --pass and --out)\n`;
const PROGRAM_FILE = path.join(ROOT, 'lib', 'capy-review-pass.js');
const PROGRAM_VERSION = 1;
const git = (a: string[]) => { const r = spawnSync('git', a, { encoding: 'utf8', timeout: 60_000, maxBuffer: 256 * 1024 * 1024 }); return r.status === 0 ? r.stdout : null; };
@@ -215,6 +225,17 @@ function passStep(cmd: string): number {
process.stdout.write(`BEFORE: tokens=${m.before.tokens} credits=${m.before.credits}\n`);
return EXIT.ok;
}
const audit = m.readers.find(x => x.reader === flag('--reader'));
if (flag('--state') === 'collected' && audit?.prompt_file && audit.tree !== undefined) {
if (!flag('--output')) { process.stderr.write(`gstack-review-dispatch mark: ${audit.reader} is collected with --output <its reply>, which must prove the checkout it reviewed\n`); return EXIT.usage; }
const tree = checkReviewedTree(readFile(flag('--output'), 'reader output')!, audit.tree);
if (!tree.ok) {
markReader(m, audit.reader, 'failed', { output: flag('--output'), error: `reviewed another tree: ${tree.reported}` });
writeManifest(file, m);
process.stdout.write(wrongTree(audit.reader, tree));
return EXIT.fail;
}
}
const r = markReader(m, flag('--reader') ?? '', flag('--state') as ReaderState, { path: flag('--path'), output: flag('--output'), error: flag('--error') });
writeManifest(file, m);
process.stdout.write(readerLine(m, r) + '\n');
@@ -276,7 +297,21 @@ function main(): number {
if (unavailable !== undefined && !mayBeUnavailable(reader)) { process.stderr.write(`gstack-review-dispatch collect: --unavailable is for the outside voice only; ${reader}'s missing coverage leaves the review incomplete, never clean\n`); return EXIT.usage; }
if (unavailable !== undefined && !unavailable.trim()) { process.stderr.write('gstack-review-dispatch collect: --unavailable needs a reason\n'); return EXIT.usage; }
if (unavailable !== undefined && parseFindings(reader, output).length) { process.stderr.write(`gstack-review-dispatch collect: ${reader}'s output holds findings; collect it without --unavailable so none is lost\n`); return EXIT.usage; }
const r = collect(ledger, reader, output, merged, path.resolve(outFile!), unavailable);
const rendered = readLedger(ledger).filter(e => e.event === 'rendered' && e.reader === reader).at(-1);
const tree = unavailable === undefined && rendered && 'tree' in rendered ? checkReviewedTree(output, typeof rendered.tree === 'string' ? rendered.tree : null) : undefined;
if (tree && !tree.ok && mayBeUnavailable(reader) && !parseFindings(reader, output).length) {
const why = `reviewed another tree: reported ${tree.reported}, expected ${tree.expected}`;
const r = collect(ledger, reader, output, merged, path.resolve(outFile!), why);
process.stdout.write(`COLLECT_UNAVAILABLE: reader=${reader} sha256=${r.sha} reason=${JSON.stringify(why)} (missing outside coverage: unavailable, never clean)\n`);
return EXIT.ok;
}
const r = collect(ledger, reader, output, merged, path.resolve(outFile!), unavailable, tree);
if (tree && !tree.ok) {
const mf = flag('--manifest');
if (mf) { const m = readManifest(mf); markReader(m, reader, 'failed', { output: outFile, error: `reviewed another tree: ${tree.reported}` }); writeManifest(mf, m); }
process.stdout.write(wrongTree(reader, tree));
return EXIT.fail;
}
if (unavailable !== undefined) { process.stdout.write(`COLLECT_UNAVAILABLE: reader=${reader} sha256=${r.sha} reason=${JSON.stringify(unavailable)} (missing outside coverage: unavailable, never clean)\n`); return EXIT.ok; }
if (!r.review) {
process.stdout.write(`COLLECT_NOT_REVIEWED: reader=${reader} findings=0 sha256=${r.sha} (no finding and no ${cleanMarkers(reader).map(([m]) => m).join(' or ')} line: a refusal, an empty reply or a reader that never saw the diff is not a review; verify reports this brief NOT_COLLECTED)\n`);
+1
View File
@@ -34,6 +34,7 @@ Spend cap: <focused paid runs per PR, from SPEND>
3. Ship. CHANGELOG entry under [Unreleased]; GUARD_CHECKS, pregate and /ship at the final
head; one unstamped READY block (captain/templates/READY.md) to me; the PR body carries
gstack-approvals render. Never merge; stamp only on my slot call.
The lane subscribes to the PR, then stops its /ship subagent at READY (docs/captain.md, the lane's /ship sequence).
Done when: the READY block is sent at a frozen head, reports/<thread code>.md is written,
and its Approvals table carries the plan approval row (the owner's words, the date) or the
+1
View File
@@ -55,6 +55,7 @@ Every Capy thread that uses gstack follows these for new work, under a Captain o
- **Plan first.** Before building new functionality, run `/autoplan` on it as the first act after the install check, sized to the work (scope flags and `--max-rounds`; a one-file fix gets `--no-ui --no-dx --max-rounds 1` with `--estimate-usd` and a small `--spend-cap`, never no run).
- **ELI5, then the word.** Send the plan and gate to the person, or the Captain, in plain language: `gstack-owner-brief render --captain --out <run dir> --captain-name "<Captain name>"`, its choices as lettered lines in the ask format (`docs/captain.md`) and "do it" taking every recommendation at full scope. Build only after their word is answered into the run and it reports `APPROVED`; then `gstack-approvals import --autoplan <run dir> --by "<owner>"`.
- **/ship before ready.** Run /ship at the final pushed head before a PR is called ready; `SHIP_RECEIPT: pr=<n> head=<head>` must name that head, and a later push needs a new /ship. The review inside /ship, like the autoplan, is sized to the change.
- **Stop a finished /ship subagent.** When a /ship subagent opens its PR, the parent subscribes to that PR with Capy's subscribe tool and keeps that one subscription until the PR merges or closes, re-subscribing if the watch expires, then cancels it. Once the subscription is in place and READY arrives, the parent stops the subagent and owns the PR from then on, slot stamp, re-READY and receipt carry included. A stopped /ship subagent woken by a check, review or merge notice, not by its parent, reports one line naming what woke it, its pin and the PR head (`gh pr view <n> --json headRefOid --jq .headRefOid`) and ends the turn without acting, whether or not the head moved. A message from its parent is an instruction it carries out.
- **No merge without the owner's word.** A thread never merges on its own; under a Captain the controller merges on a recorded standing approval, and a thread with no Captain asks the person.
Exempt from the plan step: read-only work, the daily flaky pass's repairs, slot stamps, CHANGELOG-only resolves, cap re-pins under a written grant, S1 emergencies (a broken install or upgrade, a red main, a security bug) and fixes from an already-diagnosed, audited problem list. Every other fix gets a one-round, eng-only plan.
@@ -1,18 +0,0 @@
---
bump: minor
---
**Merge trains work for any repository, not just gstack.** Until now a train needed gstack's own eval workflow, gated only its tip and needed a bisect to find a culprit, so peer crews like gbrain and Truther merged one PR at a time and folded main-red fixes together by hand (GSTA-157).
### Added
- **`gates` in the train block** (docs/ship-policy.md#train-gates, `TRAIN_GATES_KEY_VERSION`): `workflows` gates dispatch the repository's own CI on every prefix ref (`train-prefix/<nonce>/t<i>`) by nonce and need every named job green at that exact commit, with one rerun of failed jobs (a pass is GREEN with a FLAKY line); `command` gates run a test command on a Ubicloud VM per prefix under one spend cap per gate for the whole train (`train init` prices it for a full train and one rerun: $3 at 1800 s). A workflow is sent at most twice per prefix, and one build or verdict at a time writes a train. Every prefix has its own verdict, so the first red prefix after a green one names the culprit and the slots before it can still land.
- **`train land --train <nonce> --pr <n>` for these trains**: `direct` merges the pin itself (`gh pr merge --match-head-commit`) when branch protection is not strict and the pin's checks are green; `update` commits the gated tree on [pin, main] first. `train.land` sets the merge method and required checks; `rebase` always refuses, since a replayed tree is not the gated one. A workflows gate's `post_merge_checks` must succeed on main (`verify --post-merge <sha> --checks <names>`) before the next slot lands.
- **`train watch --land`** waits for every prefix run, reads the verdict and lands each landable slot in order.
- **`train init` and `train doctor`** print a train block from branch protection and the repository's workflows, plus the workflow snippet each gate needs, and check a live block, including a command cap too low for a full train. Neither writes anything.
- **Base-red PRs ride a train**: `ready --base-red --fixer <pr>` READYs a PR whose /ship paused on a red base, and `ready --fixes-base-red <cases>` READYs its fixer. The train seats the fixer at slot 1, allows main red on exactly those cases, gates every conditional PR on its own prefix and lands it only through an update commit.
- Result codes `QUEUE_TRAIN_BASE_RED`, `QUEUE_TRAIN_SPEND_CAP`, `QUEUE_TRAIN_LAND_METHOD`, `QUEUE_TRAIN_INIT_REFUSED`, `QUEUE_TRAIN_DOCTOR_FINDINGS`, `QUEUE_BASE_RED_NO_TRAIN` and `SHIP_BASE_RED_FIXER`, each with a docs/troubleshooting.md entry; captain/templates/merge-train-peer.md for a peer crew turning one on.
### Changed
- gstack's own train block (no `gates`) and every recorded train behave exactly as before. A train with gates refuses `train bisect` and `train rebuild`, since every prefix already has its verdict, and `verify --train` names `train land` instead. `train prune` keeps an attributed train while slots before its culprit are still unmerged.
### Tests
- `test/ship-queue-train-generic.test.ts`: policy validation (and gstack's own policy file unchanged), prefix verdicts including FIXES_BASE_RED, run classification, test records, base-red admission, workflow facts; on fixture repos: dispatch by nonce on every prefix, direct and update landing, rebase and fork refusals, an attributed culprit, the one-rerun FLAKY path, red on main, command gates through a fake ubi-runner with `UBI_REAP=0` and the spend cap, `ready --base-red`, a fixer train landed by `watch --land`, and `train init`/`train doctor`; plus the review fixes: unreadable protection, a dispatch no run adopts, a held lease, a run another process owns, output past the 1 MB log tail, rebase, post-merge checks and an INCOMPLETE train rebuilt on the same pins.
+16
View File
@@ -1034,6 +1034,22 @@ never forwards `UBICLOUD_*`. All of this is pinned offline in
`test/ubicloud-runner.test.ts` against a fake API and a fake `ssh`, including
a fixture whose fake API creates, so `run` and `verify` execute end to end.
`run` and `verify` start their setup, command and tier phases detached on the
VM (output, leader pid and exit status in `~/.ubirun-<phase>.*`) and key each
run locally before any VM exists (content tree, command, setup content, env,
pass-value hashes, size, location, image, storage, origin/HEAD, the runner's
own sha256). A rerun of the same command after the runner died reattaches to
the live VM and continues from the first phase without an exit file
(`UBI_ATTACH_BUSY` while another runner owns it, exit 125 `UBI_ATTACH_LOST`
when the session left no status); HUP detaches (`UBI_DETACHED`, exit 129), INT
and TERM still tear down. A `.gstack/ubicloud.json` with `tiers` records
start/pass/fail per tier under `UBI_CHECKPOINT_DIR/<run key>/`, so a resume on
a fresh VM reruns setup and only the tiers without a pass; `pregate-remote.sh`
adds its tree and planned-set digest to the key, so a rerun of
`gstack-pregate --stage tests --remote` reattaches and bills from the original
`created_at`. The fake VM in `test/ubicloud-runner.test.ts` runs these phases
for real in a directory.
Under 8 cores (`gstack-doctor`'s `cores` row says so), run the full unit and
E2E tiers on a remote gate: `gstack-config set test_backend ubicloud` records
the choice and the row names the runner. Resume checklist after a wake or a
+7
View File
@@ -255,6 +255,13 @@ a push included. The sequence that works unattended on Capy:
10. After the slot call, the stamp commit keeps the pin's receipt; re-READY with the same
receipt, no second /ship.
When /ship runs in a subagent, the parent subscribes to the pull request's check and
review notices as soon as the subagent opens it, and stops the /ship subagent only once
that subscription is in place and its READY arrives. From then on the parent owns the pull
request: red checks, human review, the slot stamp and re-READY of step 10, and any receipt
carry. It does that work itself or sends it to the subagent as an explicit instruction. On
Capy the mechanics are in [the gstack skill](../capy/skill/gstack/SKILL.md#plan-approve-ship).
When the base moves under a reviewed branch, merge it in; never rebase. A merge that
leaves the product diff unchanged carries the receipt with `gstack-ship-receipt carry`
(pregate at HEAD, no review rerun), and a merge whose resolution changed product code
+5 -2
View File
@@ -145,8 +145,11 @@ archive bytes, not a verified response URL.
Image publication starts with the committed
[`build-inputs.json`](../lib/cso/images/build-inputs.json). Its reviewed revision
pins distinct native manifests for every stack and the SBOM generator. Each
profile records the inspected source tag and immutable index for provenance,
then `baseImages` keyed by `linux/amd64` and `linux/arm64`; Python additionally
profile records the inspected source tag and immutable index for provenance.
Every source tag (each base, Python's uv and the SBOM generator) must name an exact release (`<major>.<minor>.<patch>`, or
`<major>.<minor>` for PostgreSQL); a floating major or minor tag, `latest` or
another alias fails matrix generation with `FLOATING_<FAMILY>_SOURCE`. Each profile then records
`baseImages` keyed by `linux/amd64` and `linux/arm64`; Python additionally
records native uv manifests. CI fails if a source tag no longer resolves to the
reviewed index, a platform manifest is absent from that index, its image
configuration names another architecture, or an executable reports a different
+134 -4
View File
@@ -88,6 +88,8 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`).
| `stall_timeout_s` | seconds a test command may print nothing before its process group is killed and the check is `incomplete` (`PREGATE_STALLED`); default 300, `0` disables; `--stall-timeout` overrides; the `--remote` gate is exempt (see [supervision](#supervision)) |
| `timeout_s` | seconds each runner process may take (the unit-file group, each serial file, each guard without its own `timeout_s`, and the remote gate's deadline) when `--timeout` is not passed; 1 to 3600, default 600. `gstack-ship-preflight` and the `finish` fallback run pregate without `--timeout`, so this is how a repo with long lanes sets theirs |
| `remote_gate` | the repo's remote test gate command; under 8 cores the local test tier becomes a `requires-remote` obligation naming it, and `--remote` runs it (see [remote gate](#remote-gate)) |
| `run_profiles` | name → `{ workflow, events, jobs, artifact, max_age_hours?, os? }`: the GitHub Actions runs `clear --run` verifies as evidence on the identical tree; a profile named after a lane (`free-tests.yml/macos-named-regressions`) clears that lane (see [run evidence](#run-evidence)) |
| `remote_gate_runs` | `{ profiles, inventory }`: the run profiles that together clear `pregate.json/remote_gate`, and the command that prints the remote gate's test files, one per line, which their runs must cover (see [run evidence](#run-evidence)) |
| `allowlist` | `[{ finding, paths?, pattern?, reason }]`: benign findings the `secrets` check accepts and names as allowlisted, by gstack-redact finding id or non-secret category, path globs and a pattern anchored at the finding; never `env.value` (see [allowlist](#allowlist)) |
Only the `origin/<base>` copy or the user-owned file below is honoured, so a
@@ -112,9 +114,10 @@ the caller's environment, never the checkout's `.env`: the bin's shebang passes
`--no-env-file`, and a bare `bun gstack-pregate` launched beside any `.env` file
refuses (`POLICY_SOURCE_UNAVAILABLE`). `verify` reads both paths from the
record, not from its own environment.
It covers `gstack-pregate run` only: `sealed` and `price_file` are read from the
base branch's copy, so the user file refuses them (`PREGATE_CONFIG_INVALID`)
rather than holding a policy nothing reads. A user file that exists but cannot
It covers `gstack-pregate run` only: `sealed`, `price_file`, `run_profiles` and
`remote_gate_runs` are read from the base branch's copy, so the user file refuses
them (`PREGATE_CONFIG_INVALID`; an empty `run_profiles` is accepted) rather than
holding a policy nothing reads. A user file that exists but cannot
be read, or a symlink on its path that points nowhere, refuses the same way
instead of falling back to defaults. A base copy git cannot read (and
`git ls-tree` cannot confirm is absent) refuses with `POLICY_SOURCE_UNAVAILABLE`
@@ -435,7 +438,7 @@ the selected local files as a `requires-remote` obligation with the command to
run, `pregate.json/remote_gate` (the declared command) or
`test_backend/<backend>` (gstack's `scripts/ubicloud/ubi-runner.sh run --src
<repo> -- '<runner> <files>'`). Without either, the tier runs locally under
the supervision above. Close the obligation one of two ways:
the supervision above. Close the obligation one of three ways:
- after running it, `gstack-pregate clear --remote pregate.json/remote_gate
--receipt <file>` with the gate's output, validated as `--remote` validates it
@@ -444,6 +447,10 @@ the supervision above. Close the obligation one of two ways:
accepted and recorded, see [below](#remote-gate-stamp-only)), or
`gstack-pregate clear --remote test_backend/<backend> --run-url <url>`, which
has no machine-checkable receipt and is recorded as `attested:<url>`; or
- when CI already ran the full suite on this tree, `gstack-pregate clear --remote
pregate.json/remote_gate --run <id|url> [--run …] --allow-repo-commands`, one
verified GitHub Actions run per `remote_gate_runs` profile (see
[run evidence](#run-evidence)); or
- `gstack-pregate --stage tests --remote --spend-cap <usd> --allow-repo-commands`,
which runs the declared `remote_gate` now (under `--policy-from head`, still the trusted copy's gate;
a `test_backend` without `remote_gate` cannot be run this way). The gate runs
@@ -611,6 +618,129 @@ Every refusal line starts with a reason code:
| `PREGATE_REMOTE_INFRA` | 4 | create, setup, SSH or the stop deadline failed, or the runner's `UBICLOUD: left=` line is not `0`; the VM, its teardown line and `spent=` follow |
| `PREGATE_REMOTE_INTERRUPTED` | 130 | TERM or INT; ubi-runner tears the VM down on its own |
<a id="run-evidence"></a>
## Run evidence: one full suite per tree (`clear --run`)
A repository that runs its full suite in CI and again as its remote gate on the
same tree paid twice for one result (gbrain: a 30–45 min Ubicloud gate, then an
18–20 min full-ci dispatch, every serial slot). `clear --run` lets a passed
GitHub Actions run stand in for the gate, on the runner's own record and never
on prose: a pasted URL is only a pointer the verifier resolves.
**Declare it on the base branch.** In `.gstack/pregate.json`:
```json
{
"run_profiles": {
"test": { "workflow": "test.yml", "events": ["workflow_dispatch", "push"], "jobs": ["full-suite"], "artifact": "full-suite-tree" },
"e2e": { "workflow": "e2e.yml", "events": ["workflow_dispatch", "push"], "jobs": ["e2e-full-suite"], "artifact": "e2e-full-suite-tree" }
},
"remote_gate_runs": { "profiles": ["test", "e2e"], "inventory": "bash scripts/pregate-remote-gate.sh --list" }
}
```
Profiles are read only from `origin/<base>`, so a pull request can tighten
them but never relax them. That covers the profile, not the suite it names: the
workflow file a run executed and the inventory command are the branch's own, as
the remote gate's script already is, so a diff to either is reviewed like any
other gate change. Each profile names the workflow file, the events it
accepts, every job that must have passed (each exactly once), and the artifact
holding the run's `tree.json`. `max_age_hours` (1–168, default 24) bounds how
old the run may be, and `os` (`Linux`, `macOS`, `Windows`) requires
`tree.json` to record that runner OS. Each workflow uploads `tree.json` from
its own job: `run_id`, `run_attempt`, `event`, `sha` (or `head_sha`),
`tree` (`git rev-parse HEAD^{tree}` as tested), `files` (every test file the
run executed), `tests` (the count) and, with `os`, the runner OS.
**What `clear --run` checks** (`lib/run-evidence.ts`), for every run, online,
once:
- The run belongs to this repository and ran for a same-repository head. Its
workflow file is the profile's, its event is `push` or `workflow_dispatch`
(a `pull_request` run tests the merge commit, not the head tree, so no
profile may accept it), and it completed with success. The API answers with
the newest attempt, so an earlier green attempt under a newer red one is
refused, and the run is read again after the evidence is collected: a new
attempt meanwhile is `run_changed`.
- Every job the profile names appears exactly once in that attempt and
succeeded. Jobs and runs are read across every page.
- `tree.json` (an artifact of at most 8 MiB) names this run, attempt, event and
head (`sha` and `head_sha`, each checked when present). It is not a reuse
record (a reuse is never evidence). Its tree equals the tree git computes for
the run's head commit. `files` is a list of paths and `tests` a
non-negative integer when present; without `tests` the file count stands in.
- No run of the same workflow on that head, or on pregate.json's own head,
whose newest attempt started at or after the second this run was created
failed, was cancelled or is still running (`run_started_at`, so a red re-run
of an older run counts, a run started in the same second counts, and
re-running this run does not hide a failure in between).
- The run finished within `max_age_hours` (its jobs' latest `completed_at`,
else the run's `updated_at`), and on the profile's OS when it names one.
Re-running only the failed jobs refuses (`artifact_mismatch`) unless the job
that uploads `tree.json` re-runs too, because that file must name the
newest attempt. The 8 MiB cap is read from the artifact's zipped size before
download, so a workflow on the branch itself can still unpack a larger file
into the temporary directory.
Then, for the lane:
- pregate.json was recorded against this repository's base, whose profiles
apply (`base_mismatch` otherwise).
- Each profile gets its own run (a run id given twice is a usage error), and
every run verifies under its profile. The assignment is searched, so a run
valid for two profiles cannot starve the other. A lane with a profile needs
one run; `pregate.json/remote_gate` needs one per `remote_gate_runs`
profile.
- All runs tested one tree: pregate.json's tree, or a stamp-only predecessor of
it under exactly the rule [above](#remote-gate-stamp-only) (a policy whose
`stamp_sensitive_gates` names `tests` refuses that substitution).
- Every run executed something (`no_tests` names one that did not), and the
runs' `files` together cover the obligation's selected test files, plus, for
the remote gate, every file the `inventory` command prints. The inventory is
a pin-file command, so it runs only with `--allow-repo-commands`, and only
while the working tree is pregate.json's tree (checked before and after it
runs); an empty or failing inventory refuses, and it runs under `pipefail`,
so a pipeline whose first command fails refuses too.
- Every run is read once more after the inventory: a run that moved to a new
attempt or stopped being a success meanwhile refuses (`run_changed`).
Any API or git error refuses. Each refusal prints `REUSE: refused
reason=<code> — …` as `PREGATE_RECEIPT_INVALID`, and a run that fails
verification is named with its own reason (`fork_run`, `event_mismatch`,
`run_not_success`, `job_missing`, `artifact_mismatch`, `artifact_reused`,
`tree_mismatch`, `newer_run_vetoes`, `run_changed`, `run_stale`,
`os_mismatch`, `api_error`, …); the clear-level codes are `base_mismatch`,
`no_verified_run`, `tree_mismatch` (runs on different trees),
`tree_not_stamp_only`, `no_tests`, `inventory_failed`, `inventory_empty`,
`inventory_uncovered` (the remote gate) and `tests_uncovered` (a lane's own
selected tests). A wrong run count, a repeated run or a lane with no
profile is a usage error (exit 2), not a REUSE refusal.
Accepted, `clear` prints `REUSE: accepted <lane> <receipt>`, and the receipt
reads `remote-gate tree=<t12> tests=<n> runs=<id>/<attempt>,… profiles=<p>,…
heads=<h12> verified=<time>` (with `accepted-for=` and the substitution for a
stamp-only predecessor). A lane with its own profile gets `run tree=… runs=…`
the same way; for such a lane `--run-url` is verified as `--run` is, and
`--bundle-lane` is refused. A lane with no profile keeps today's `--run-url`
recording. When the base config or the GitHub origin cannot be read, `--run`
fails, and so do `--run-url` and `--bundle-lane` unless the base was read and
shows the lane has no profile; then the lane is recorded as before, with a
warning naming the cause. `gstack-ship-queue ready --dispatch-ci` still
records its dispatched runs by URL; routing it through the verifier is the next
change (P3b), so a repository that declares lane profiles clears those lanes
with `gstack-pregate clear --run`.
**`verify` stays offline.** The check ran at `clear`. `gstack-pregate verify`,
and `gstack-ship-queue ready` through it, refuse a run receipt whose `tree=`
is not pregate.json's tree unless a substitution for it is recorded. They make
no network call, so GitHub's availability and the run's age never sit on the
merge path.
**What is never reusable.** Security and advisory scans (osv, audit,
gitleaks) depend on the day, not on the tree, so they are not run profiles:
they run on every head as before. Pull-request required checks always run
fresh.
<a id="guards"></a>
## Guards and the lane-done rule
+11
View File
@@ -514,6 +514,17 @@ ones, and the arm lost blockers. `bin/gstack-review-dispatch` closes both ends:
missing or doubled title stops the pass. On Capy the pass runs as one call to the
checked-in program `lib/capy-review-pass.js`, served by `program` and loaded
unchanged by the gstack Capy skill.
7. Every brief, audit readers' included, ends with a workspace paragraph naming the
checkout the bins run in (GST-482): a reader on a shared machine starts in the
machine's working directory, so it is told to `cd` there on every command, to give
every file tool an absolute path under it, and to start its reply with
`REVIEWED_TREE: <HEAD^{tree}> <toplevel>` measured there. The paragraph follows the
section's render, which stays byte for byte. `collect` and `mark --state collected`
compare that line with the fingerprint recorded at render: a missing or foreign line is
`COLLECT_WRONG_TREE`, a later commit in the same checkout `COLLECT_TREE_MOVED`; nothing
is merged, the reader becomes `failed`, and `verify` reports `NOT_COLLECTED`. An
outside voice's tree failure is recorded `unavailable` only when its reply holds no finding. The line catches a reader that
worked elsewhere by accident; it is not proof of reading, since diff text can predict it.
## Where this is enforced
+17 -4
View File
@@ -3293,8 +3293,9 @@ repository's lockfile), then rerun `gstack-regen check` or the pre-gate.
**Meaning.** `.gstack/pregate.json` has an unknown key, a wrong type, a glob
that escapes the repository, a runner without `{files}`, or `secrets` in `warn`.
The user-owned `<state root>/projects/<slug>/pregate.json` also refuses when it
sets `sealed` or `price_file` (both are read only from the base branch's copy)
or exists but cannot be read (a dangling symlink on its path counts).
sets `sealed`, `price_file`, a non-empty `run_profiles` or `remote_gate_runs`
(all read only from the base branch's copy) or exists but cannot be read (a
dangling symlink on its path counts).
**Fix.** `gstack-pregate validate`. `secrets` and required-lane obligations
cannot be downgraded; every other check id in `warn` is honoured only from the
@@ -3490,6 +3491,12 @@ reports failures. The obligation stays uncleared. A run URL alone never clears
this obligation; a `test_backend/<backend>` obligation takes a run URL, recorded
as attested rather than validated.
`gstack-pregate clear --run <id|url>` refusals use this code too: the line
reads `gstack-pregate clear: receipt refused — REUSE: refused reason=<code> —
…`, and the GitHub Actions runs offered did not verify against the base
branch's `run_profiles`. Each reason code and
what it checks is listed in [run evidence](pregate.md#run-evidence).
Another tree is accepted in exactly one case: it differs from `pregate.json`'s
tree only in the repo's stamp rule (`.gstack/ship-policy.json` on the base
branch: `stamp_files`, `stamp_patterns`, the built-in version, manifest,
@@ -3585,7 +3592,9 @@ run, named by `workflow.yml/job`, or the small-machine `pregate.json/remote_gate
tier) has no receipt; or a `remote_gate` receipt accepted from a stamp-only
predecessor tree records `accepted_for` as a tree other than `pregate.json`'s,
or a `gated_tree` other than its receipt string's `tree=` (the file was edited
after `clear`), so it clears nothing here.
after `clear`), so it clears nothing here; or a `clear --run` receipt names
another tree than `pregate.json`'s with no substitution recorded (`run
receipt(s) verified for another tree`).
**Fix.** Dispatch that job, then
`gstack-pregate clear --remote <workflow.yml/job> --run-url <url>` (or
@@ -3593,7 +3602,11 @@ after `clear`), so it clears nothing here.
`clear --remote pregate.json/remote_gate --receipt <gate output>`: the gate's
own run on this tree, or a run on a predecessor tree that differs only by the
repo's stamp rule ([gate ahead of the stamp](pregate.md#remote-gate-stamp-only));
`verify` then prints the accepted substitution on its ok line. The ship receipt
`verify` then prints the accepted substitution on its ok line. When the base
branch declares run profiles, `clear --remote <lane> --run <id|url>` clears a
lane from verified GitHub Actions runs on this tree (for the remote gate, one
run per `remote_gate_runs` profile, with `--allow-repo-commands`); a profiled
lane refuses `--bundle-lane` ([run evidence](pregate.md#run-evidence)). The ship receipt
carries each obligation with its receipt.
<a id="issue-unreadable"></a>
+4 -3
View File
@@ -129,12 +129,13 @@ try {
}
const file = m.out + '/out/' + safe(r.reader) + '.txt';
await tools.capy.write({ path: file, content: g.summary });
const c = await sh(r.prompt_file ? bin + ' mark --manifest ' + q(mf) + ' --reader ' + q(r.reader) + ' --state collected --output ' + q(file)
: bin + ' collect --ledger ' + q(m.ledger) + roleArgs(r) + ' --output ' + q(file) + ' --merged ' + q(m.merged) + ' --manifest ' + q(mf));
if (c.code !== 0) return stop((r.prompt_file ? 'mark ' : 'collect ') + r.reader + ' exited ' + c.code + ': ' + c.out.slice(-500));
if (!r.prompt_file) {
const c = await sh(bin + ' collect --ledger ' + q(m.ledger) + roleArgs(r) + ' --output ' + q(file) + ' --merged ' + q(m.merged));
if (c.code !== 0) return stop('collect ' + r.reader + ' exited ' + c.code + ': ' + c.out.slice(-500));
lines.push(c.out.trim());
await mark(r.reader, 'collected', { output: file });
}
await mark(r.reader, 'collected', { output: file });
}
m = await load();
const stage = active.find((s) => m.readers.some((r) => r.stage === s && r.state !== 'collected'));
+1 -1
View File
@@ -6,7 +6,7 @@
"reviewedAt": "2026-09-10T00:00:00.000Z",
"reviewMethod": "Native OCI manifest inspection plus upstream runtime version metadata; CI rechecks architecture and executable versions before every build.",
"sbomGenerator": {
"source": "docker.io/docker/scout-sbom-indexer:1",
"source": "docker.io/docker/scout-sbom-indexer:1.24.0",
"indexImage": "docker.io/docker/scout-sbom-indexer@sha256:4b67f29eb0d1244ab0f62de867ac5dafd7262fcd7ebbdefa6ec8aacd6b15252d",
"images": {
"linux/amd64": "docker.io/docker/scout-sbom-indexer@sha256:dc9450ee50d985e1f5cc60069c91b5f45b4a8a7e9dbcdd33aaab259032a9cab9",
+17 -5
View File
@@ -36,6 +36,12 @@
* remote_gate the repo's remote test gate command: under 8 cores the test
* tier is a requires-remote obligation naming it, and
* `--remote` runs it (it prints `PREGATE_REMOTE: tree=… tests=… failed=…`)
* run_profiles name → `{ workflow, events, jobs, artifact, max_age_hours?, os? }`:
* the GitHub Actions runs `clear --run` accepts as evidence
* (lib/run-evidence.ts); a profile named after a lane clears it
* remote_gate_runs `{ profiles, inventory }`: the run profiles that together
* clear the remote gate obligation on the identical tree, and
* the command listing the gate's test files they must cover
* allowlist `{ finding, paths?, pattern?, reason }` entries the `secrets`
* check accepts as benign: a gstack-redact finding id or a
* non-secret category, scoped by path globs and/or a regex
@@ -53,6 +59,7 @@ import * as path from 'node:path';
import { spawnSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import type { ResultCodeName } from '../result-codes';
import { runProfileErrors, type RemoteGateRuns, type RunProfile } from '../run-evidence';
import { sealedErrors } from '../sealed-guard';
import { containedRelative, globToRegExp } from '../ship-policy';
@@ -84,15 +91,17 @@ export interface PregateConfig {
timeout_s: number | null;
remote_gate: string | null;
allowlist: AllowlistEntry[];
run_profiles: Record<string, RunProfile>;
remote_gate_runs: RemoteGateRuns | null;
}
export interface Guard { id: string; when: string[]; command: string; fix?: string; timeout_s?: number }
export interface AllowlistEntry { finding: string; paths?: string[]; pattern?: string; reason: string }
export const DEFAULT_CONFIG: PregateConfig = { runner: null, dependencies: {}, empty_lanes: [], selection: {}, warn: [], strays: { allow: [] }, local: [], guards: [], serial_patterns: DEFAULT_SERIAL_PATTERNS, price_file: null, stall_timeout_s: null, timeout_s: null, remote_gate: null, allowlist: [] };
export const DEFAULT_CONFIG: PregateConfig = { runner: null, dependencies: {}, empty_lanes: [], selection: {}, warn: [], strays: { allow: [] }, local: [], guards: [], serial_patterns: DEFAULT_SERIAL_PATTERNS, price_file: null, stall_timeout_s: null, timeout_s: null, remote_gate: null, allowlist: [], run_profiles: {}, remote_gate_runs: null };
export interface ConfigError { code: ResultCodeName; path: string; message: string }
export const MAX_LIST = 256;
const MAX_COMMAND = 512;
const KEYS = ['runner', 'dependencies', 'empty_lanes', 'selection', 'warn', 'strays', 'local', 'guards', 'serial_patterns', 'price_file', 'sealed', 'stall_timeout_s', 'timeout_s', 'remote_gate', 'allowlist'];
const KEYS = ['runner', 'dependencies', 'empty_lanes', 'selection', 'warn', 'strays', 'local', 'guards', 'serial_patterns', 'price_file', 'sealed', 'stall_timeout_s', 'timeout_s', 'remote_gate', 'allowlist', 'run_profiles', 'remote_gate_runs'];
const LANE_ID = /^[A-Za-z0-9_.-]+\.ya?ml\/[A-Za-z0-9_-]+$/;
const MAX_GUARDS = 32;
const GUARD_KEYS = ['id', 'when', 'command', 'fix', 'timeout_s'];
@@ -183,6 +192,7 @@ export function validateConfig(raw: unknown, knownChecks: readonly string[] = []
const why = commandError(obj.remote_gate);
if (why) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: 'remote_gate', message: why });
}
for (const e of runProfileErrors(obj.run_profiles, obj.remote_gate_runs)) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: e.path, message: e.message });
if ('guards' in obj) guardErrors(obj.guards, errors);
if ('allowlist' in obj) allowlistErrors(obj.allowlist, errors);
if ('sealed' in obj) for (const e of sealedErrors(obj.sealed)) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: e.path, message: e.message });
@@ -251,7 +261,7 @@ export function parseConfig(text: string, knownChecks: readonly string[] = []):
const errors = validateConfig(raw, knownChecks);
if (errors.length) return { config: DEFAULT_CONFIG, errors };
const o = raw as Partial<PregateConfig>;
return { config: { $schema: o.$schema, runner: o.runner ?? null, dependencies: o.dependencies ?? {}, empty_lanes: o.empty_lanes ?? [], selection: o.selection ?? {}, warn: o.warn ?? [], strays: { allow: o.strays?.allow ?? [] }, local: o.local ?? [], guards: o.guards ?? [], serial_patterns: o.serial_patterns ?? DEFAULT_SERIAL_PATTERNS, price_file: o.price_file ?? null, stall_timeout_s: o.stall_timeout_s ?? null, timeout_s: o.timeout_s ?? null, remote_gate: o.remote_gate ?? null, allowlist: o.allowlist ?? [] }, errors: [] };
return { config: { $schema: o.$schema, runner: o.runner ?? null, dependencies: o.dependencies ?? {}, empty_lanes: o.empty_lanes ?? [], selection: o.selection ?? {}, warn: o.warn ?? [], strays: { allow: o.strays?.allow ?? [] }, local: o.local ?? [], guards: o.guards ?? [], serial_patterns: o.serial_patterns ?? DEFAULT_SERIAL_PATTERNS, price_file: o.price_file ?? null, stall_timeout_s: o.stall_timeout_s ?? null, timeout_s: o.timeout_s ?? null, remote_gate: o.remote_gate ?? null, allowlist: o.allowlist ?? [], run_profiles: o.run_profiles ?? {}, remote_gate_runs: o.remote_gate_runs ?? null }, errors: [] };
}
export type ConfigLoad = { config: PregateConfig; errors: ConfigError[]; present: boolean; label: string; kind: 'base' | 'head' | 'user' | 'none'; /** sha256 of the user file's bytes (kind 'user' only). */ digest?: string } | { unavailable: true; base: string; detail: string };
@@ -304,10 +314,10 @@ export function loadConfigFromUser(file: string, knownChecks: readonly string[]
return { ...parseConfig('{}', knownChecks), errors: [{ code: 'PREGATE_CONFIG_INVALID', path: file, message: `cannot read the user-owned policy: ${(e as Error).message}` }], present: true, label: `user ${file}`, kind: 'user' };
}
const load = parseConfig(text, knownChecks);
// `sealed` (the pre-push guard) and `price_file` (gstack-eval-plan) read only the git copies, so a non-null value in the user file would look active and do nothing.
// `sealed` (the pre-push guard), `price_file` (gstack-eval-plan) and the run profiles (`clear --run`) read only the git copies, so a non-null value in the user file would look active and do nothing.
let raw: unknown = null;
try { raw = JSON.parse(text); } catch { /* parseConfig reported it */ }
const ignored = raw && typeof raw === 'object' && !Array.isArray(raw) ? ['sealed', 'price_file'].filter(k => (raw as Record<string, unknown>)[k] != null) : [];
const ignored = raw && typeof raw === 'object' && !Array.isArray(raw) ? ['sealed', 'price_file', 'run_profiles', 'remote_gate_runs'].filter(k => { const v = (raw as Record<string, unknown>)[k]; return v != null && !(k === 'run_profiles' && typeof v === 'object' && Object.keys(v).length === 0); }) : [];
// parseConfig names the repo copy for a JSON error; this file is not it.
const errors = [...load.errors.map(e => e.path === PREGATE_CONFIG_PATH ? { ...e, path: file } : e), ...ignored.map(k => ({ code: 'PREGATE_CONFIG_INVALID' as const, path: k, message: `the user-owned policy covers gstack-pregate run only; ${k} is read from ${PREGATE_CONFIG_PATH} on the base branch` }))];
return { ...load, errors, present: true, label: `user ${file}`, kind: 'user', digest: createHash('sha256').update(text).digest('hex') };
@@ -333,6 +343,8 @@ export function explainConfig(): string {
'stall_timeout_s: seconds without test output before the lanes check kills the runner\'s process group and records it incomplete (default 300; 0 disables; --stall-timeout overrides)',
'timeout_s: seconds each runner process may take (the unit-file group, each serial file, each guard without its own timeout_s, and the remote gate\'s deadline) when --timeout is not passed; 1 to 3600, default 600',
'remote_gate: the repo\'s remote test gate command; under 8 cores the test tier becomes a requires-remote obligation naming it, and gstack-pregate --remote runs it (it must print PREGATE_REMOTE: tree=<tree> tests=<n> failed=<n>) (docs/pregate.md#remote-gate)',
'run_profiles: name → { workflow, events, jobs, artifact, max_age_hours?, os? }; the GitHub Actions runs gstack-pregate clear --run verifies as evidence on the identical tree (docs/pregate.md#run-evidence)',
'remote_gate_runs: { profiles, inventory }; the run profiles that together clear the remote gate obligation, and the command listing the gate\'s test files they must cover (docs/pregate.md#run-evidence)',
`allowlist: [{ finding, paths?, pattern?, reason }]: benign findings the secrets check accepts and names as allowlisted. finding is a gstack-redact id (env.kv, google.api_key, pii.phone.e164, …) or a category (${ALLOWLISTABLE_CATEGORIES.join(', ')}); paths are globs; pattern is a regex matched at the finding's position in the added line; one of the two is required, reason always. Never ${NON_ALLOWLISTABLE.join(' or ')} or engine.* (docs/pregate.md#allowlist)`,
'no .gstack/pregate.json on the base: <state root>/projects/<slug>/pregate.json is read instead (state root: GSTACK_STATE_ROOT, else GSTACK_HOME, else GSTACK_STATE_DIR, else a gstack plugin data dir, else ~/.gstack, per docs/state-root.md; run --explain prints the resolved path; outside the repo, so a PR cannot reach it; a path inside the checkout is ignored)',
].join('\n') + '\n';
+100 -12
View File
@@ -23,10 +23,11 @@ import { regenCheck } from './checks/regen';
import { secretsCheck } from './checks/secrets';
import { straysCheck } from './checks/strays';
import { timingCheck } from './checks/timing';
import { NON_DOWNGRADABLE, userPolicyAbsent } from './config';
import { NON_DOWNGRADABLE, PREGATE_CONFIG_PATH, userPolicyAbsent } from './config';
import { sha256, withFingerprint } from './context';
import { baseBranchOf, baseMismatchFix, describeSubstitution, substituteStampOnlyTree } from './remote-receipt';
import { resolveBaseBranch } from '../ship-policy';
import { parseRunReceipt, parseRunRef, runChanged, runReceiptFields, uncoveredFiles, verifyRun, type RemoteGateRuns, type RunDeps, type RunEvidence, type RunProfile } from '../run-evidence';
import type { CheckDef, CheckResult, PregateContext, Stage } from './types';
export const CHECKS: readonly CheckDef[] = [regenCheck, secretsCheck, straysCheck, literalsCheck, timingCheck, guardsCheck, lanesCheck];
@@ -173,6 +174,9 @@ export function verifyPregate(repoRoot: string, file: PregateFile, treeNow: Wtre
// A receipt accepted from a stamp-only predecessor tree was accepted for one head tree; recorded for another, it clears nothing here.
const misapplied = (file.requires_remote ?? []).filter(r => r.substitution && (r.substitution.accepted_for !== file.tree || !(r.receipt ?? '').includes(`tree=${r.substitution.gated_tree.slice(0, 12)} `)));
if (misapplied.length) return { ok: false, code: 'PREGATE_REMOTE_UNCLEARED', message: `${misapplied.length} remote receipt(s) accepted for another tree or recorded against another gated tree: ${misapplied.map(r => `${r.lane} (accepted for ${r.substitution!.accepted_for.slice(0, 12)} from gated ${r.substitution!.gated_tree.slice(0, 12)}, pregate.json is ${file.tree.slice(0, 12)})`).join(', ')}; clear again`, current, uncleared: misapplied };
// A run receipt was verified online at clear for its tree; offline it must still name this tree (or carry its substitution).
const foreignRuns = (file.requires_remote ?? []).filter(r => { const run = parseRunReceipt(r.receipt ?? ''); return run && !r.substitution && run.tree !== file.tree.slice(0, 12); });
if (foreignRuns.length) return { ok: false, code: 'PREGATE_REMOTE_UNCLEARED', message: `${foreignRuns.length} run receipt(s) verified for another tree: ${foreignRuns.map(r => `${r.lane} (${r.receipt})`).join(', ')}; clear again with runs on tree ${file.tree.slice(0, 12)}`, current, uncleared: foreignRuns };
const substituted = (file.requires_remote ?? []).filter(r => r.substitution).map(r => `${r.lane}: ${describeSubstitution(r.substitution!)}`);
return { ok: true, message: `pregate ${file.verdict} is current for tree ${file.tree.slice(0, 12)}${substituted.length ? `; ${substituted.join('; ')}` : ''}`, current, uncleared };
}
@@ -198,7 +202,9 @@ function userPolicyChange(file: PregateFile): string | null {
return `${absent} now holds a user-owned policy, and pregate.json was recorded under the defaults; rerun gstack-pregate`;
}
export interface ClearInput { runUrl?: string; bundleLane?: string; receiptPath?: string }
export interface ClearInput { runUrl?: string; bundleLane?: string; receiptPath?: string; runs?: string[]; evidence?: RunCheck }
/** What a run clear needs: the base branch's run profiles, the verifier's dependencies, the clock, and the remote gate's inventory. */
export interface RunCheck { repo: string; deps: RunDeps; profiles: Record<string, RunProfile>; gateRuns: RemoteGateRuns | null; now: Date; inventory: (command: string, tree: string) => string[] | { error: string } }
export type ClearDecision = { receipt: string; attested: boolean; substitution?: PregateReceiptSubstitution } | { refused: 'usage' | 'PREGATE_RECEIPT_INVALID'; message: string; fix?: string };
/**
@@ -212,11 +218,19 @@ export type ClearDecision = { receipt: string; attested: boolean; substitution?:
* the tree mismatch refuses as before. A `test_backend/<backend>` obligation
* has no machine-checkable receipt, so its run URL is recorded as attested.
* Platform lanes take a run URL or an evidence-bundle lane label.
*
* `--run` (and `--run-url` on a lane the base branch profiles) goes to
* clearByRuns, which accepts only verified Actions runs; a profiled lane
* refuses `--bundle-lane`, and the remote gate clears by runs only when the
* base declares remote_gate_runs.
*/
export function clearDecision(file: PregateFile, lane: string, input: ClearInput, repoRoot: string): ClearDecision {
const list = file.requires_remote ?? [];
const target = list.find(r => r.lane === lane);
if (!target) return { refused: 'usage', message: `no requires-remote obligation named ${lane}; obligations: ${list.map(r => r.lane).join(', ') || '(none)'}` };
const profiled = input.evidence && (lane === REMOTE_GATE_LANE ? input.evidence.gateRuns : input.evidence.profiles[lane]);
if (input.runs?.length || (input.runUrl && profiled)) return clearByRuns(file, lane, [...(input.runs ?? []), ...(input.runUrl ? [input.runUrl] : [])], input.evidence, repoRoot);
if (input.bundleLane && profiled && lane !== REMOTE_GATE_LANE) return { refused: 'usage', message: `${lane} has a run profile on the base branch; clear it with --run <id|url>, which verifies the run` };
if (target.lane === REMOTE_GATE_LANE) {
if (!input.receiptPath) return { refused: 'usage', message: `${lane} clears only with --receipt <file> holding the remote gate's PREGATE_REMOTE: tree=… tests=… failed=… line (a run URL is not checkable); or rerun gstack-pregate --remote` };
let text: string;
@@ -226,16 +240,9 @@ export function clearDecision(file: PregateFile, lane: string, input: ClearInput
if (!report) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${input.receiptPath} has no PREGATE_REMOTE: line; ${lane} stays uncleared` };
const own = remoteReportProblem(report, report.tree);
if (own) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${own}; ${lane} stays uncleared` };
let substitution: PregateReceiptSubstitution | undefined;
if (report.tree !== file.tree) {
// The stamp rule is the ship's own base branch's policy (origin/HEAD, as regate reads it), never a branch the artifact names.
const base = resolveBaseBranch(repoRoot);
const recorded = baseBranchOf(file.base);
if (recorded && recorded !== base) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${treeMismatch(report.tree, file.tree)}; pregate.json was recorded against origin/${recorded}, not this repository's base origin/${base}, so the stamp rule that would class the difference is not the base's; ${lane} stays uncleared`, fix: baseMismatchFix(repoRoot, base, recorded) };
const s = substituteStampOnlyTree(repoRoot, base, report.tree, file.tree);
if ('problem' in s) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${s.problem}; ${lane} stays uncleared`, fix: s.fix };
substitution = s.substitution;
}
const sub = treeSubstitution(file, report.tree, lane, repoRoot);
if ('refused' in sub) return sub;
const substitution = sub.substitution;
const problem = remoteReportProblem(report, substitution ? report.tree : file.tree);
if (problem) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${problem}; ${lane} stays uncleared` };
const accepted = substitution ? ` accepted-for=${file.tree.slice(0, 12)} stamp-only=${substitution.files.length}` : '';
@@ -251,6 +258,87 @@ export function clearDecision(file: PregateFile, lane: string, input: ClearInput
return { receipt, attested: false };
}
/** The stamp-only substitution that lets a receipt for `gatedTree` clear pregate.json's tree, none when the trees are equal, or the refusal. */
function treeSubstitution(file: PregateFile, gatedTree: string, lane: string, repoRoot: string): { substitution?: PregateReceiptSubstitution } | Extract<ClearDecision, { refused: string }> {
if (gatedTree === file.tree) return {};
// The stamp rule is the ship's own base branch's policy (origin/HEAD, as regate reads it), never a branch the artifact names.
const base = resolveBaseBranch(repoRoot);
const recorded = baseBranchOf(file.base);
if (recorded && recorded !== base) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${treeMismatch(gatedTree, file.tree)}; pregate.json was recorded against origin/${recorded}, not this repository's base origin/${base}, so the stamp rule that would class the difference is not the base's; ${lane} stays uncleared`, fix: baseMismatchFix(repoRoot, base, recorded) };
const s = substituteStampOnlyTree(repoRoot, base, gatedTree, file.tree);
if ('problem' in s) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${s.problem}; ${lane} stays uncleared`, fix: s.fix };
return { substitution: s.substitution };
}
/**
* `clear --run`: GitHub Actions runs as the obligation's evidence (lib/run-evidence.ts). The remote gate needs one
* verified run per `remote_gate_runs` profile, all on one tree, whose tree.json file lists together cover every file
* the gate's inventory command prints; any other lane needs one verified run of the profile named after it. The run
* tree must be pregate.json's tree or a stamp-only predecessor of it, exactly as a gate report's tree must.
*/
function clearByRuns(file: PregateFile, lane: string, refs: string[], check: RunCheck | undefined, repoRoot: string): ClearDecision {
if (!check) return { refused: 'usage', message: `--run needs the base branch's run_profiles; none could be read` };
const isGate = lane === REMOTE_GATE_LANE;
const names = isGate ? check.gateRuns?.profiles ?? [] : check.profiles[lane] ? [lane] : [];
if (!names.length) return { refused: 'usage', message: isGate ? `--run clears ${lane} only when the base branch's ${PREGATE_CONFIG_PATH} declares remote_gate_runs` : `--run clears ${lane} only when the base branch's ${PREGATE_CONFIG_PATH} declares a run_profiles entry named ${lane}` };
if (refs.length !== names.length) return { refused: 'usage', message: `${lane} clears with exactly ${names.length} run(s), one per profile (${names.join(', ')}); got ${refs.length}` };
const ids = refs.map(ref => parseRunRef(ref, check.repo) ?? ref);
if (new Set(ids).size !== ids.length) return { refused: 'usage', message: `${lane} needs a different run for each profile; got ${refs.join(', ')}` };
// The profiles are the base branch's; a pregate.json recorded against another base names another authority.
const base = resolveBaseBranch(repoRoot);
const recorded = baseBranchOf(file.base);
if (recorded && recorded !== base) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=base_mismatch — pregate.json was recorded against origin/${recorded}, but run profiles come from this repository's base origin/${base}; ${lane} stays uncleared`, fix: baseMismatchFix(repoRoot, base, recorded) };
const verdicts = new Map<string, ReturnType<typeof verifyRun>>();
const verdict = (name: string, ref: string) => {
const key = `${name}\0${ref}`;
if (!verdicts.has(key)) verdicts.set(key, verifyRun(check.deps, { repo: check.repo, runRef: ref, profileName: name, profile: check.profiles[name]!, vetoHeads: file.head ? [file.head] : [], now: check.now }));
return verdicts.get(key)!;
};
const missingProfile = names.find(name => !check.profiles[name]);
if (missingProfile) return { refused: 'usage', message: `run profile ${missingProfile} is not declared` };
// Every profile gets its own verified run; search the whole assignment so a run valid for two profiles cannot starve one.
const assign = (i: number, used: Set<string>): RunEvidence[] | null => {
if (i === names.length) return [];
for (const ref of refs) {
if (used.has(ref)) continue;
const v = verdict(names[i]!, ref);
if (!v.ok) continue;
const rest = assign(i + 1, new Set([...used, ref]));
if (rest) return [v.evidence, ...rest];
}
return null;
};
const evidence = assign(0, new Set());
if (!evidence) {
const failed = names.map(name => `${name}: ${refs.map(ref => { const v = verdict(name, ref); return v.ok ? `${ref} ok` : `${ref} reason=${v.reason} ${v.detail}`; }).join('; ')}`);
return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=no_verified_run — no run per profile verifies (${failed.join(' | ')}); ${lane} stays uncleared` };
}
const trees = [...new Set(evidence.map(e => e.tree))];
if (trees.length > 1) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=tree_mismatch — the runs tested different trees (${trees.map(t => t.slice(0, 12)).join(', ')}); ${lane} stays uncleared` };
const sub = treeSubstitution(file, trees[0]!, lane, repoRoot);
if ('refused' in sub) return { ...sub, message: `REUSE: refused reason=tree_not_stamp_only — ${sub.message}` };
const idle = evidence.filter(e => (e.tests ?? e.files.length) === 0);
if (idle.length) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=no_tests — ${idle.map(e => `run ${e.run_id} (${e.profile})`).join(', ')} records no tests; ${lane} stays uncleared` };
const required = new Set(file.requires_remote?.find(r => r.lane === lane)?.tests ?? []);
if (isGate) {
const listed = check.inventory(check.gateRuns!.inventory, file.tree);
if ('error' in listed) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=inventory_failed — the remote gate's inventory command failed: ${listed.error}; ${lane} stays uncleared` };
if (!listed.length) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=inventory_empty — the remote gate's inventory command listed no test files; ${lane} stays uncleared` };
for (const f of listed) required.add(f);
}
const missing = uncoveredFiles([...required], evidence);
if (missing.length) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=${isGate ? 'inventory_uncovered' : 'tests_uncovered'} — ${missing.length} of the ${required.size} test file(s) ${lane} must cover appear in no run's tree.json (${missing.slice(0, 5).join(', ')}${missing.length > 5 ? ', …' : ''}); ${lane} stays uncleared` };
for (const e of evidence) {
let changed: string | null;
try { changed = runChanged(check.deps, e.run_id, e.run_attempt); }
catch (err: any) { changed = `reading run ${e.run_id} again failed: ${String(err?.message ?? err).slice(0, 200)}`; }
if (changed) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=run_changed — ${changed}; ${lane} stays uncleared` };
}
const tests = evidence.reduce((n, e) => n + (e.tests ?? e.files.length), 0);
const accepted = sub.substitution ? ` accepted-for=${file.tree.slice(0, 12)} stamp-only=${sub.substitution.files.length}` : '';
return { receipt: `${isGate ? 'remote-gate' : 'run'} tree=${trees[0]!.slice(0, 12)} tests=${tests}${accepted} ${runReceiptFields(evidence, check.now)}`, attested: false, substitution: sub.substitution };
}
/** Record the receipt that clears one obligation (a run URL, an evidence-bundle lane label, or a remote gate report, with the stamp-only substitution when one was accepted). */
export function clearObligation(file: PregateFile, lane: string, receipt: string, by: string, substitution?: PregateReceiptSubstitution): PregateFile {
const list = file.requires_remote ?? [];
+50
View File
@@ -14,6 +14,7 @@
* fix-check fix_list, hunks, tests, callers (scripts/resolvers/fix-check.ts)
* targeted classes, brief (review/specialists/targeted.md's text), checklist (its path), diff_base, stack, test_fw, learnings
*/
import { spawnSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import * as fs from 'node:fs';
import * as path from 'node:path';
@@ -134,6 +135,55 @@ function renderRole(role: BriefRole, slots: Slots, t: SectionTexts): string {
}
}
/**
* The closing paragraph every dispatched brief carries (GST-482): a reader on a shared machine starts in that machine's working
* directory, and every command runs in a fresh shell, so the brief names the checkout under review and asks for a fingerprint
* measured there. It is appended after the section's render, so the brief before it stays byte for byte what renderBrief returns.
*/
export function workspaceParagraph(dir: string): string {
const q = `'${dir.replace(/'/g, `'\\''`)}'`;
return `Workspace: the checkout under review is ${dir}. Your shell and file tools may start in another checkout, and every command runs in a fresh shell: begin each command with \`cd ${q} && \`, and give every file read, search, edit or write an absolute path under ${dir}. Make the first line of your final reply the line \`cd ${q} && echo "REVIEWED_TREE: $(git rev-parse 'HEAD^{tree}') $(git rev-parse --show-toplevel)"\` prints, even where this brief limits your output to findings; a reply without it, or from another checkout, does not count.`;
}
export const withWorkspace = (text: string, dir: string) => `${text}\n\n${workspaceParagraph(dir)}`;
/** The checkout a brief names: the toplevel of the directory the bins run in, where its diff and merge base come from. */
export function checkoutRoot(): string {
const r = spawnSync('git', ['rev-parse', '--show-toplevel'], { encoding: 'utf8', timeout: 30_000 });
return realDir(r.status === 0 ? r.stdout.trim() : process.cwd());
}
/**
* One REVIEWED_TREE line as "<tree> <toplevel>", or null. A list marker or heading mark may lead it, and a whole-line `**…**` or
* backtick wrapper, a bold label and backticks around the sha or the whole path are removed; the path's own characters are kept.
*/
function parseTreeLine(line: string): string | null {
let t = line.trim().replace(/^(?:[-*>]\s+|\d+[.)]\s+|#+\s+)/, '');
for (const w of ['**', '`']) if (t.length > 2 * w.length && t.startsWith(w) && t.endsWith(w)) t = t.slice(w.length, -w.length).trim();
const m = /^(?:\*\*REVIEWED_TREE:\*\*|REVIEWED_TREE:)\s+`?([0-9a-f]{40,64})`?\s+(.+)$/.exec(t);
if (!m) return null;
const p = m[2]!.length > 2 && m[2]!.startsWith('`') && m[2]!.endsWith('`') ? m[2]!.slice(1, -1) : m[2]!;
return `${m[1]} ${realDir(p)}`;
}
export const isReviewedTreeLine = (line: string) => parseTreeLine(line) !== null;
const realDir = (p: string) => { try { return fs.realpathSync(p); } catch { return path.resolve(p); } };
/** The checkout a reply says it reviewed: its first REVIEWED_TREE line (a later one may be quoted inside a finding); null with none. */
const reportedTree = (output: string): string | null => output.replace(/\r\n/g, '\n').split('\n').map(parseTreeLine).find(t => t !== null) ?? null;
/** The fingerprint a reply must report, read from the checkout itself when its brief is rendered: "<HEAD^{tree}> <toplevel>". */
export function checkoutTree(dir: string): string | null {
const git = (a: string[]) => spawnSync('git', ['-C', dir, ...a], { encoding: 'utf8', timeout: 30_000 });
const tree = git(['rev-parse', 'HEAD^{tree}']), top = git(['rev-parse', '--show-toplevel']);
return tree.status === 0 && top.status === 0 ? `${tree.stdout.trim()} ${realDir(top.stdout.trim())}` : null;
}
/** moved: the reply reviewed the named checkout, but at another tree than the one its brief was rendered on. */
export type TreeCheck = { ok: boolean; moved: boolean; expected: string; reported: string };
/** Whether a reply's REVIEWED_TREE line is the fingerprint recorded when its brief was rendered (GST-482). */
export function checkReviewedTree(output: string, expected: string | null): TreeCheck {
const reported = reportedTree(output);
const ok = expected !== null && reported === expected;
const top = (t: string) => t.slice(t.indexOf(' ') + 1);
return { ok, moved: !ok && !!reported && expected !== null && top(reported) === top(expected),
expected: expected ?? 'none (the checkout was unreadable at render)', reported: reported ?? 'none' };
}
/** The file a rendered brief is written to under the run's briefs directory. */
export function briefFile(role: BriefRole, name?: string): string {
return `${role}${name ? `-${name.replace(/[^a-zA-Z0-9._-]/g, '_')}` : ''}.prompt.md`;
+13 -6
View File
@@ -16,7 +16,7 @@
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { promptHash } from './review-brief';
import { isReviewedTreeLine, promptHash, type TreeCheck } from './review-brief';
export interface LedgerEvent { event: 'rendered' | 'sent' | 'output' | 'judgment' | 'judge' | 'program'; [k: string]: unknown }
export const readerKey = (role: string, name?: string) => (name ? `${role}/${name}` : role);
@@ -83,6 +83,7 @@ export function parseFindings(reader: string, output: string): Array<{ label: st
let current: Finding | null = null;
let head = '';
let gap = false;
let treeSeen = false;
const role = reader.split('/')[0];
const named = (l: string) => (role === 'adversarial-native' && l.toUpperCase() === 'FIXABLE' ? 'CRITICAL' : l);
const strongest = (t: string) => [...t.matchAll(ALL_LABELS)].map(m => named(m[1]!)).reduce<string | undefined>((a, b) => (a === undefined ? b : stronger(a, b)), undefined);
@@ -90,6 +91,7 @@ export function parseFindings(reader: string, output: string): Array<{ label: st
for (const line of output.replace(/\r\n/g, '\n').split('\n')) {
const t = line.trim();
if (!t) { gap = true; continue; }
if (!treeSeen && isReviewedTreeLine(t)) { treeSeen = true; continue; }
const afterGap = gap;
gap = false;
if (/^NO FINDINGS\.?$/.test(t) || /^Recommendation:/i.test(t)) { current = null; continue; }
@@ -155,15 +157,18 @@ const hasCleanMarker = (reader: string, output: string) => output.replace(/\r\n/
/**
* collect: hash the raw output, append its findings to merged.jsonl and record both; a reply with no finding and no clean marker records
* review: false. With unavailable (the outside voice only) the output is its failure text, recorded as missing coverage.
* review: false. With unavailable (the outside voice only) the output is its failure text, recorded as missing coverage. With a
* failed tree check (GST-482: the reply reviewed another checkout, or names none) nothing is merged and the output is no review.
*/
export function collect(ledger: string, reader: string, output: string, mergedFile: string, outputFile?: string, unavailable?: string): { sha: string; lines: MergedLine[]; review: boolean } {
export function collect(ledger: string, reader: string, output: string, mergedFile: string, outputFile?: string, unavailable?: string, tree?: TreeCheck): { sha: string; lines: MergedLine[]; review: boolean } {
const sha = promptHash(output);
const prior = readLedger(ledger).filter(e => e.event === 'output' && e.reader === reader).length;
const lines = unavailable ? [] : parseFindings(reader, output).map((f, i) => ({ id: `${reader}#${prior ? `${prior + 1}.` : ''}${i + 1}`, reader, label: f.label, raw: f.raw, output_sha256: sha }));
const review = !unavailable && (lines.length > 0 || hasCleanMarker(reader, output));
const wrongTree = !unavailable && tree !== undefined && !tree.ok;
const lines = unavailable || wrongTree ? [] : parseFindings(reader, output).map((f, i) => ({ id: `${reader}#${prior ? `${prior + 1}.` : ''}${i + 1}`, reader, label: f.label, raw: f.raw, output_sha256: sha }));
const review = !unavailable && !wrongTree && (lines.length > 0 || hasCleanMarker(reader, output));
fs.appendFileSync(mergedFile, lines.map(l => JSON.stringify(l) + '\n').join(''), { mode: 0o600 });
appendEvent(ledger, { event: 'output', reader, sha256: sha, file: outputFile ? path.resolve(outputFile) : null, merged: path.resolve(mergedFile), ids: lines.map(l => l.id), review, ...(unavailable ? { unavailable } : {}) });
appendEvent(ledger, { event: 'output', reader, sha256: sha, file: outputFile ? path.resolve(outputFile) : null, merged: path.resolve(mergedFile), ids: lines.map(l => l.id), review, ...(unavailable ? { unavailable } : {}),
...(tree && !unavailable ? { tree: tree.expected, reported_tree: tree.reported, tree_ok: tree.ok } : {}) });
return { sha, lines, review };
}
/** The outside voice may fail without blocking (ship/sections/adversarial.md): its failure is recorded as unavailable coverage, never as a review. */
@@ -222,6 +227,7 @@ export function recordJudgment(ledger: string, merged: MergedLine[], judged: Jud
* a ledger written before collect recorded the field is judged by the same rule; with no readable file, collect's own review field.
*/
function reviewed(o: LedgerEvent): boolean {
if (o.tree_ok === false || unavailableOutput(o)) return false;
if (Array.isArray(o.ids) && o.ids.length) return true;
let text: string | undefined;
try { text = typeof o.file === 'string' ? fs.readFileSync(o.file, 'utf8') : undefined; } catch { /* unreadable */ }
@@ -292,6 +298,7 @@ export function verify(ledger: string): { rows: VerifyRow[]; clean: boolean; bri
const replaced = review > 0 && later.slice(0, review).some(e => e.event === 'sent' && e.reader === reader && e.sha256 === events[p!.at]!.sha256);
if (unavailableOutput(o)) { status = 'UNAVAILABLE'; notes.push(`reason=${JSON.stringify(o.unavailable)}`); }
else if (replaced) { status = 'VOIDED'; voided++; notes.push('replaced_by_a_review'); }
else if (o.tree_ok === false) { status = 'NOT_REVIEWED'; notes.push(`wrong_tree reported=${JSON.stringify(o.reported_tree)} expected=${JSON.stringify(o.tree)}`); }
else { status = 'NOT_REVIEWED'; notes.push('no_finding_no_clean_marker'); }
}
rows.push({ kind: 'OUTPUT', reader: p?.label ?? reader, status, detail: `sha=${short(o.sha256)} findings=${ids.length}${notes.length ? ' ' + notes.join(' ') : ''}` });
+13 -6
View File
@@ -13,7 +13,7 @@
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { BriefError, briefFile, promptHash, renderBrief, type BriefRole, type SectionTexts, type Slots } from './review-brief';
import { BriefError, briefFile, checkoutRoot, checkoutTree, promptHash, renderBrief, withWorkspace, type BriefRole, type SectionTexts, type Slots } from './review-brief';
import { appendEvent, readerKey } from './review-ledger';
import { diffCommand, type Dispatch } from './review-depth';
import { TARGETED_RULES, parseClassList, targetedBriefSha, targetedPatchId } from './review-targeted';
@@ -31,6 +31,8 @@ export interface PassReader {
/** An audit or QA reader: its prompt is a file the parent wrote from its section; its output is returned, never collected. */
prompt_file?: string; model?: PassModel;
file?: string; sha256?: string; bytes?: number; chunks?: number; findings_sha256?: string;
/** The fingerprint the reader's reply must report: the checkout's HEAD^{tree} and toplevel when its brief was rendered (GST-482). */
tree?: string | null;
path?: string; output?: string; error?: string; state_at?: string;
}
export interface Manifest {
@@ -126,11 +128,16 @@ export function renderStage(m: Manifest, stage: Stage, t: SectionTexts, diffFor:
const briefs = path.join(m.out, 'briefs');
fs.mkdirSync(briefs, { recursive: true, mode: 0o700 });
const done: PassReader[] = [];
const workdir = checkoutRoot();
const tree = checkoutTree(workdir);
if (!tree) throw new BriefError(`cannot read HEAD^{tree} and the toplevel of ${workdir}; a brief names the checkout its reply must prove`);
for (const r of m.readers.filter(x => x.stage === stage && x.state === 'planned')) {
if (r.prompt_file) {
let text: string;
try { text = fs.readFileSync(r.prompt_file, 'utf8'); } catch { throw new BriefError(`cannot read ${r.reader}'s prompt file ${r.prompt_file}`); }
Object.assign(r, { file: r.prompt_file, sha256: promptHash(text), bytes: Buffer.byteLength(text), chunks: chunkCount(text), state: 'rendered' });
try { text = withWorkspace(fs.readFileSync(r.prompt_file, 'utf8'), workdir); } catch { throw new BriefError(`cannot read ${r.reader}'s prompt file ${r.prompt_file}`); }
const file = path.join(briefs, `${safe(r.reader)}.prompt.md`);
fs.writeFileSync(file, text, { mode: 0o600 });
Object.assign(r, { file, sha256: promptHash(text), bytes: Buffer.byteLength(text), chunks: chunkCount(text), state: 'rendered', tree });
done.push(r);
continue;
}
@@ -155,13 +162,13 @@ export function renderStage(m: Manifest, stage: Stage, t: SectionTexts, diffFor:
Object.assign(slots, { checklist, brief, classes: (r.classes ?? []).join(','), diff_base: r.diff_base ?? m.diff_base });
targeted = { classes: parseClassList(slots.classes!), rules: TARGETED_RULES, brief_sha: targetedBriefSha(checklist), diff_base: slots.diff_base, patch_id: targetedPatchId(process.cwd(), slots.diff_base!) };
}
const text = renderBrief(role, slots, t);
const text = withWorkspace(renderBrief(role, slots, t), workdir);
const file = path.join(briefs, briefFile(role, r.name));
fs.writeFileSync(file, text, { mode: 0o600 });
const sha = promptHash(text);
const bytes = Buffer.byteLength(text);
appendEvent(m.ledger, { event: 'rendered', reader: r.reader, role, name: r.name ?? null, file, sha256: sha, bytes, pass: m.pass, stage, ...(findingsSha ? { findings_sha256: findingsSha } : {}), ...targeted });
Object.assign(r, { file, sha256: sha, bytes, chunks: chunkCount(text), state: 'rendered', ...(findingsSha ? { findings_sha256: findingsSha } : {}) });
appendEvent(m.ledger, { event: 'rendered', reader: r.reader, role, name: r.name ?? null, file, sha256: sha, bytes, pass: m.pass, stage, workdir, tree, ...(findingsSha ? { findings_sha256: findingsSha } : {}), ...targeted });
Object.assign(r, { file, sha256: sha, bytes, chunks: chunkCount(text), state: 'rendered', tree, ...(findingsSha ? { findings_sha256: findingsSha } : {}) });
done.push(r);
}
return done;
+261
View File
@@ -0,0 +1,261 @@
/**
* run-evidence — a GitHub Actions run as evidence for one tree (GSTA-156).
*
* A consumer that would otherwise run a suite again (pregate's remote gate
* obligation, a platform lane) may accept a run that already passed on the
* identical tree, but only on the runner's own record, never on prose: the
* GitHub API's record of the run, the tree git computes from the run's head
* commit, and the profile's artifact (its tree.json) the run uploaded itself.
* `verifyRun` refuses unless every check holds:
*
* the run belongs to this repository and ran for a same-repository head;
* its workflow file and event are the profile's; it is the newest attempt,
* completed with success; every job the profile names appears exactly once
* in that attempt and succeeded; its artifact names this run, attempt,
* event and head (the API always answers with the newest attempt, so an
* older green attempt under a newer red one is refused), is not a reuse
* record, records the tree git computes for the head, and holds well-formed
* files (a list of paths) and tests (a non-negative integer); no run of the
* same workflow on that head, or on any other head the consumer names, that
* started at or after this run was created failed, was cancelled or is still
* running (a red re-run of an older run counts, and re-running this run does
* not hide a failure in between); it finished within the profile's
* max_age_hours (default 24); its runner OS matches when the profile names
* one; and a second read of the run still shows the same successful attempt
* (run_changed otherwise).
*
* Profiles are the base branch's `.gstack/pregate.json` `run_profiles`, so a
* pull request can tighten them but never relax them. The workflow a run
* executed and the tree.json it uploaded are the run head's own, so a diff to
* that workflow is reviewed like any other gate change. Every refusal names a
* reason code; an API or git error refuses (fail closed). Verification runs
* once, online, at `gstack-pregate clear`; the receipt it writes carries the
* evidence, and `gstack-pregate verify` checks that receipt offline.
*/
import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { sha256Hex, writeReceipt } from './egress-receipt';
import { runBashSync } from './script-invocation';
export interface RunProfile { workflow: string; events: string[]; jobs: string[]; artifact: string; max_age_hours?: number; os?: string }
export interface RemoteGateRuns { profiles: string[]; inventory: string }
export interface ApiRun {
id: number; run_attempt: number; status: string; conclusion: string | null; event: string; head_sha: string;
path?: string; created_at?: string; run_started_at?: string; updated_at?: string;
repository?: { full_name: string } | null; head_repository?: { full_name: string } | null;
}
export interface ApiJob { name: string; conclusion: string | null; completed_at?: string | null }
export interface TreeRecord { run_id?: unknown; run_attempt?: unknown; event?: unknown; sha?: unknown; head_sha?: unknown; tree?: unknown; reused?: unknown; reuse?: unknown; os?: unknown; files?: unknown; tests?: unknown }
/** Everything verifyRun reads, injectable so every refusal has a fixture. */
export interface RunDeps {
run(id: number): ApiRun;
runsForHead(workflow: string, sha: string): ApiRun[];
jobs(id: number, attempt: number): ApiJob[];
artifact(id: number, name: string): TreeRecord;
commitTree(sha: string): string | null;
}
export type RunRefusal = 'run_ref_invalid' | 'other_repository' | 'fork_run' | 'workflow_mismatch' | 'event_mismatch' | 'run_in_progress'
| 'run_not_success' | 'job_missing' | 'job_ambiguous' | 'job_not_success' | 'artifact_missing' | 'artifact_mismatch'
| 'artifact_reused' | 'tree_mismatch' | 'newer_run_vetoes' | 'run_changed' | 'run_stale' | 'os_mismatch' | 'api_error';
export interface RunEvidence { run_id: number; run_attempt: number; head_sha: string; tree: string; profile: string; event: string; finished_at: string; files: string[]; tests: number | null }
export type RunVerdict = { ok: true; evidence: RunEvidence } | { ok: false; reason: RunRefusal; detail: string };
const SHA = /^[0-9a-f]{40}$/;
const PROFILE_KEYS = ['workflow', 'events', 'jobs', 'artifact', 'max_age_hours', 'os'];
const WORKFLOW_FILE = /^[A-Za-z0-9_.-]+\.ya?ml$/;
const NAME = /^[A-Za-z0-9_.\/-]{1,128}$/;
export const DEFAULT_MAX_AGE_HOURS = 24;
const MAX_ARTIFACT_BYTES = 8 * 1024 * 1024;
/** A numeric run id, or a run URL on this repository (`…/actions/runs/<id>[/attempts/<n>][/job/<j>]`). */
export function parseRunRef(ref: string, repo: string): number | null {
if (/^\d{1,15}$/.test(ref)) return Number(ref);
const m = /^https:\/\/github\.com\/([\w.-]+\/[\w.-]+)\/actions\/runs\/(\d{1,15})(?:\/(?:attempts\/\d+|job\/\d+))?\/?$/.exec(ref);
return m && m[1]!.toLowerCase() === repo.toLowerCase() ? Number(m[2]) : null;
}
/** `<key>: <message>` errors for `run_profiles` and `remote_gate_runs`; empty when valid. Shared with lib/pregate/config.ts. */
export function runProfileErrors(profiles: unknown, gateRuns: unknown): Array<{ path: string; message: string }> {
const errors: Array<{ path: string; message: string }> = [];
const names = new Set<string>();
if (profiles !== undefined) {
if (!profiles || typeof profiles !== 'object' || Array.isArray(profiles)) errors.push({ path: 'run_profiles', message: 'must be an object of name → { workflow, events, jobs, artifact, max_age_hours?, os? }' });
else for (const [name, raw] of Object.entries(profiles as Record<string, unknown>)) {
const at = `run_profiles[${JSON.stringify(name)}]`;
names.add(name);
if (!NAME.test(name)) errors.push({ path: at, message: 'name must be 1-128 letters, digits, . _ / -' });
const p = raw as Record<string, unknown>;
if (!p || typeof p !== 'object' || Array.isArray(p)) { errors.push({ path: at, message: 'must be an object' }); continue; }
for (const k of Object.keys(p)) if (!PROFILE_KEYS.includes(k)) errors.push({ path: `${at}.${k}`, message: `unknown key; valid keys: ${PROFILE_KEYS.join(', ')}` });
if (typeof p.workflow !== 'string' || !WORKFLOW_FILE.test(p.workflow)) errors.push({ path: `${at}.workflow`, message: 'must be a workflow file name like test.yml' });
for (const k of ['events', 'jobs'] as const) {
const v = p[k];
if (!Array.isArray(v) || v.length === 0 || v.length > 32 || v.some(s => typeof s !== 'string' || !s.trim())) errors.push({ path: `${at}.${k}`, message: 'must be a non-empty array of up to 32 names' });
}
if (Array.isArray(p.events) && p.events.some(e => e !== 'push' && e !== 'workflow_dispatch')) errors.push({ path: `${at}.events`, message: 'events are push or workflow_dispatch (a pull_request run tests the merge commit, not the head tree)' });
if (typeof p.artifact !== 'string' || !NAME.test(p.artifact)) errors.push({ path: `${at}.artifact`, message: 'must be the artifact name holding tree.json' });
if ('max_age_hours' in p && (!Number.isInteger(p.max_age_hours) || (p.max_age_hours as number) < 1 || (p.max_age_hours as number) > 168)) errors.push({ path: `${at}.max_age_hours`, message: 'must be an integer from 1 to 168' });
if ('os' in p && (typeof p.os !== 'string' || !/^(Linux|macOS|Windows)$/.test(p.os))) errors.push({ path: `${at}.os`, message: 'must be Linux, macOS or Windows' });
}
}
if (gateRuns !== undefined && gateRuns !== null) {
const g = gateRuns as Record<string, unknown>;
if (!g || typeof g !== 'object' || Array.isArray(g)) errors.push({ path: 'remote_gate_runs', message: 'must be { profiles: [names], inventory: command }' });
else {
for (const k of Object.keys(g)) if (k !== 'profiles' && k !== 'inventory') errors.push({ path: `remote_gate_runs.${k}`, message: 'unknown key; valid keys: profiles, inventory' });
if (!Array.isArray(g.profiles) || g.profiles.length === 0 || g.profiles.some(p => typeof p !== 'string')) errors.push({ path: 'remote_gate_runs.profiles', message: 'must be a non-empty array of run_profiles names' });
else {
for (const p of g.profiles as string[]) if (!names.has(p)) errors.push({ path: 'remote_gate_runs.profiles', message: `${JSON.stringify(p)} is not a run_profiles name` });
if (new Set(g.profiles).size !== g.profiles.length) errors.push({ path: 'remote_gate_runs.profiles', message: 'names each profile once' });
}
if (typeof g.inventory !== 'string' || !g.inventory.trim() || g.inventory.length > 512 || /[\0\r\n]/.test(g.inventory)) errors.push({ path: 'remote_gate_runs.inventory', message: 'must be one command line under 512 characters that prints the remote gate\'s test files, one per line' });
}
}
return errors;
}
const fail = (reason: RunRefusal, detail: string): RunVerdict => ({ ok: false, reason, detail });
/** When a run's newest attempt started. A run vetoes the candidate when it started at or after the second the candidate was created: a red re-run of an older run counts, and re-running the candidate does not hide a failure in between. */
const started = (r: ApiRun): string => r.run_started_at ?? r.created_at ?? '';
const normalPath = (f: string): string => f.replace(/^(\.\/)+/, '');
/**
* The verdict for run `runRef` under `profile`. `vetoHeads` are further heads
* whose newer runs of the same workflow can veto (the consumer's own head).
*/
export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; profileName: string; profile: RunProfile; vetoHeads?: string[]; now: Date }): RunVerdict {
const id = parseRunRef(o.runRef, o.repo);
if (id === null) return fail('run_ref_invalid', `${JSON.stringify(o.runRef)} is not a run id or a run URL on ${o.repo}`);
const p = o.profile;
try {
const run = deps.run(id);
const at = `run ${id} attempt ${run.run_attempt}`;
if ((run.repository?.full_name ?? '').toLowerCase() !== o.repo.toLowerCase()) return fail('other_repository', `${at} belongs to ${run.repository?.full_name ?? 'an unknown repository'}, not ${o.repo}`);
if ((run.head_repository?.full_name ?? '').toLowerCase() !== o.repo.toLowerCase()) return fail('fork_run', `${at} ran for ${run.head_repository?.full_name ?? 'an unknown repository'}, not ${o.repo}`);
if (run.path !== `.github/workflows/${p.workflow}`) return fail('workflow_mismatch', `${at} ran ${run.path ?? 'an unknown workflow'}, not .github/workflows/${p.workflow}`);
if (!p.events.includes(run.event)) return fail('event_mismatch', `${at} is a ${run.event} run; ${o.profileName} accepts ${p.events.join(', ')}`);
if (run.status !== 'completed') return fail('run_in_progress', `${at} is ${run.status}`);
if (run.conclusion !== 'success') return fail('run_not_success', `${at} concluded ${run.conclusion}`);
if (!SHA.test(run.head_sha)) return fail('api_error', `${at} has no head commit`);
const jobs = deps.jobs(id, run.run_attempt);
const completed = jobs.map(j => j.completed_at).filter((t): t is string => !!t).sort().pop();
const finishedAt = completed ?? run.updated_at;
const finished = Date.parse(finishedAt ?? '');
const maxAge = p.max_age_hours ?? DEFAULT_MAX_AGE_HOURS;
if (!Number.isFinite(finished)) return fail('api_error', `${at} has no completion time`);
if (o.now.getTime() - finished > maxAge * 3600_000) return fail('run_stale', `${at} finished ${finishedAt}, more than ${maxAge} h before now`);
for (const name of p.jobs) {
const hits = jobs.filter(j => j.name === name);
if (hits.length === 0) return fail('job_missing', `${at} has no job named ${name}`);
if (hits.length > 1) return fail('job_ambiguous', `${at} has ${hits.length} jobs named ${name}`);
if (hits[0]!.conclusion !== 'success') return fail('job_not_success', `${at}'s ${name} job concluded ${hits[0]!.conclusion}`);
}
const heads = [...new Set([run.head_sha, ...(o.vetoHeads ?? []).filter(h => SHA.test(h))])];
for (const head of heads) {
const newer = deps.runsForHead(p.workflow, head).filter(r => !(r.id === id && r.run_attempt === run.run_attempt) && started(r) >= (run.created_at ?? '') && !(r.status === 'completed' && r.conclusion === 'success'));
if (newer.length) return fail('newer_run_vetoes', `newer run ${newer[0]!.id} of ${p.workflow} on ${head.slice(0, 12)} is ${newer[0]!.status === 'completed' ? newer[0]!.conclusion : newer[0]!.status}`);
}
let rec: TreeRecord;
try { rec = deps.artifact(id, p.artifact); }
catch (e: any) { return fail('artifact_missing', `${at} has no readable ${p.artifact} tree.json: ${String(e?.message ?? e).slice(0, 200)}`); }
if (!rec || typeof rec !== 'object') return fail('artifact_mismatch', `${at}'s ${p.artifact} tree.json is not a JSON object`);
if (rec.reused === true || rec.reuse === true) return fail('artifact_reused', `${at} reused another run; a reuse is never evidence`);
const recHeads = rec.head_sha === undefined && rec.sha === undefined ? [['head', undefined, run.head_sha] as const] : ([['sha', rec.sha, run.head_sha], ['head_sha', rec.head_sha, run.head_sha]] as const).filter(([, got]) => got !== undefined);
const wrong = ([['run_id', rec.run_id, id], ['run_attempt', rec.run_attempt, run.run_attempt], ['event', rec.event, run.event], ...recHeads] as const).filter(([, got, want]) => got !== want);
if (wrong.length) return fail('artifact_mismatch', `${at}'s ${p.artifact} records ${wrong.map(([k, got, want]) => `${k} ${String(got)} (expected ${want})`).join(', ')}`);
const headTree = deps.commitTree(run.head_sha);
if (!headTree) return fail('api_error', `git cannot read the tree of ${run.head_sha}`);
if (rec.tree !== headTree) return fail('tree_mismatch', `${at} recorded tree ${String(rec.tree)}, but its head ${run.head_sha.slice(0, 12)} has tree ${headTree}`);
if (p.os && rec.os !== p.os) return fail('os_mismatch', `${at} ran on ${String(rec.os ?? 'an unrecorded OS')}; ${o.profileName} requires ${p.os}`);
if (rec.files !== undefined && (!Array.isArray(rec.files) || rec.files.some(f => typeof f !== 'string'))) return fail('artifact_mismatch', `${at}'s ${p.artifact} files is not a list of paths`);
if (rec.tests !== undefined && (!Number.isInteger(rec.tests) || (rec.tests as number) < 0)) return fail('artifact_mismatch', `${at}'s ${p.artifact} tests is not a non-negative integer`);
const files = ((rec.files ?? []) as string[]).map(normalPath);
const tests = rec.tests === undefined ? null : (rec.tests as number);
const changed = runChanged(deps, id, run.run_attempt);
if (changed) return fail('run_changed', changed);
return { ok: true, evidence: { run_id: id, run_attempt: run.run_attempt, head_sha: run.head_sha, tree: headTree, profile: o.profileName, event: run.event, finished_at: finishedAt!, files, tests } };
} catch (e: any) {
return fail('api_error', `reading run ${id} failed: ${String(e?.message ?? e).slice(0, 300)}`);
}
}
/** Why run `id` no longer stands as attempt `attempt`, a success; null while it does. Read again after the evidence, and for every run once a whole clear is assembled. */
export function runChanged(deps: RunDeps, id: number, attempt: number): string | null {
const again = deps.run(id);
return again.run_attempt === attempt && again.status === 'completed' && again.conclusion === 'success' ? null : `run ${id} moved to attempt ${again.run_attempt} (${again.status === 'completed' ? again.conclusion : again.status}) while it was being verified`;
}
/** The remote gate's inventory: `command`'s non-empty output lines under pipefail, so a pipeline whose producer fails refuses instead of listing part of the gate. */
export function runInventory(command: string, cwd: string): string[] | { error: string } {
const r = runBashSync(`set -o pipefail\n${command}`, { cwd, encoding: 'utf8', timeout: 300_000, maxBuffer: 64 * 1024 * 1024 });
if (r.status !== 0 || r.error) return { error: `${command} exited ${r.status ?? r.error}: ${(r.stderr ?? '').trim().slice(0, 200)}` };
return [...new Set(r.stdout.split('\n').map(l => l.trim()).filter(Boolean))];
}
/** The files in `inventory` that no verified run's tree.json lists. */
export function uncoveredFiles(inventory: string[], evidence: RunEvidence[]): string[] {
const ran = new Set(evidence.flatMap(e => e.files.map(normalPath)));
return inventory.filter(f => !ran.has(normalPath(f)));
}
/** The receipt suffix a run clear records: runs, profiles and the verification time, read back by `parseRunReceipt`. */
export function runReceiptFields(evidence: RunEvidence[], now: Date): string {
return `runs=${evidence.map(e => `${e.run_id}/${e.run_attempt}`).join(',')} profiles=${evidence.map(e => e.profile).join(',')} heads=${[...new Set(evidence.map(e => e.head_sha.slice(0, 12)))].join(',')} verified=${now.toISOString()}`;
}
export function parseRunReceipt(receipt: string): { runs: string[]; profiles: string[]; tree: string | null } | null {
const runs = / runs=(\S+)/.exec(receipt)?.[1];
const profiles = / profiles=(\S+)/.exec(receipt)?.[1];
if (!runs || !profiles || !/ verified=\S+/.test(receipt)) return null;
return { runs: runs.split(','), profiles: profiles.split(','), tree: /\btree=([0-9a-f]{12})\b/.exec(receipt)?.[1] ?? null };
}
/** Egress receipts for the reads below (fail-open, like contributor-mode): only this repository's name, run ids and a commit sha leave the machine. */
const receiptFor = (host: string, payloadClass: string, request: string) => ({ sink: 'run-evidence', host, payloadClass, bytes: 0, sha256: sha256Hex(request), consent: 'user ran gstack-pregate clear --run (or --run-url on a profiled lane); reads its own repository\'s Actions runs and fetches a run head by sha' });
const receiptFailed = (err: unknown) => process.stderr.write(`[run-evidence] egress receipt could not be written (${(err as Error).message}); proceeding (fail-open)\n`);
function gh(args: string[], timeoutMs = 60_000): string {
try { writeReceipt(receiptFor('api.github.com', 'actions-run-read', args.join(' '))); } catch (err) { receiptFailed(err); }
const r = spawnSync('gh', args, { encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
if (r.status !== 0 || r.error) throw new Error(`gh ${args.slice(0, 2).join(' ')} failed: ${(r.stderr || String(r.error ?? '')).trim().slice(0, 300)}`);
return r.stdout;
}
/** The gh- and git-backed dependencies for `repo`, reading trees in `repoRoot` (fetching a missing head from origin once). */
export function ghRunDeps(repo: string, repoRoot: string): RunDeps {
const json = <T>(p: string): T => JSON.parse(gh(['api', p]));
// Every page, one JSON value per line: a veto or a required job past the first 100 entries must still be seen.
const all = <T>(p: string, key: string): T[] => gh(['api', '--paginate', p, '--jq', `.${key}[]`]).split('\n').filter(Boolean).map(l => JSON.parse(l) as T);
return {
run: id => json<ApiRun>(`repos/${repo}/actions/runs/${id}`),
runsForHead: (workflow, sha) => all<ApiRun>(`repos/${repo}/actions/workflows/${workflow}/runs?head_sha=${sha}&per_page=100`, 'workflow_runs'),
jobs: (id, attempt) => all<ApiJob>(`repos/${repo}/actions/runs/${id}/attempts/${attempt}/jobs?per_page=100`, 'jobs'),
artifact: (id, name) => {
const listed = all<{ name: string; size_in_bytes: number; expired: boolean }>(`repos/${repo}/actions/runs/${id}/artifacts?name=${encodeURIComponent(name)}&per_page=100`, 'artifacts').filter(a => a.name === name && !a.expired);
if (listed.length !== 1) throw new Error(`run ${id} has ${listed.length} unexpired artifacts named ${name}`);
if (!(listed[0]!.size_in_bytes <= MAX_ARTIFACT_BYTES)) throw new Error(`artifact ${name} is ${listed[0]!.size_in_bytes} bytes, over ${MAX_ARTIFACT_BYTES}`);
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-run-evidence-'));
try {
gh(['run', 'download', String(id), '--repo', repo, '--name', name, '--dir', dir], 120_000);
const file = path.join(dir, 'tree.json');
if (fs.statSync(file).size > MAX_ARTIFACT_BYTES) throw new Error(`tree.json is over ${MAX_ARTIFACT_BYTES} bytes`);
return JSON.parse(fs.readFileSync(file, 'utf8')) as TreeRecord;
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
},
commitTree: sha => {
const read = () => spawnSync('git', ['rev-parse', '--verify', '--quiet', `${sha}^{tree}`], { cwd: repoRoot, encoding: 'utf8', timeout: 30_000 });
let r = read();
if (r.status !== 0) {
try { writeReceipt(receiptFor('origin', 'git-fetch-by-sha', sha)); } catch (err) { receiptFailed(err); }
spawnSync('git', ['fetch', '--quiet', '--no-tags', 'origin', sha], { cwd: repoRoot, timeout: 120_000 });
r = read();
}
const tree = (r.stdout ?? '').trim();
return r.status === 0 && /^[0-9a-f]{40,64}$/.test(tree) ? tree : null;
},
};
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gstack",
"version": "1.91.374",
"version": "1.92.0",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
+5 -2
View File
@@ -10,6 +10,8 @@ export type CsoRuntimeBuildStack = typeof CSO_RUNTIME_STACKS[number];
const IMAGE = /^[a-z0-9.-]+(?::[0-9]+)?\/[a-z0-9][a-z0-9._/-]*@sha256:[a-f0-9]{64}$/;
const TAGGED_IMAGE = /^[a-z0-9.-]+(?::[0-9]+)?\/[a-z0-9][a-z0-9._/-]*:[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/;
const RELEASE_TAG = /:v?\d+\.\d+\.\d+(?:[._-][A-Za-z0-9._-]*)?$/;
const TWO_PART_RELEASE_TAG = /:v?\d+\.\d+(?:\.\d+)?(?:[._-][A-Za-z0-9._-]*)?$/;
const REVISION = /^[a-z0-9][a-z0-9._-]{0,100}$/;
const REQUIRED_VERSIONS: Record<CsoRuntimeBuildStack, string[]> = {
node: ['node', 'npm', 'cso-preparation'],
@@ -56,8 +58,9 @@ function repositoryOfTag(image: string): string {
return image.slice(0, colon);
}
function assertPinnedFamily(source: unknown, indexImage: unknown, images: unknown, label: string): Record<CsoRuntimeBuildPlatform, string> {
function assertPinnedFamily(source: unknown, indexImage: unknown, images: unknown, label: string, releaseTag = RELEASE_TAG): Record<CsoRuntimeBuildPlatform, string> {
if (typeof source !== 'string' || !TAGGED_IMAGE.test(source)) throw new Error(`INVALID_${label}_SOURCE`);
if (!releaseTag.test(source)) throw new Error(`FLOATING_${label}_SOURCE: ${source} must name an exact release version, not a moving major, minor or alias tag`);
if (typeof indexImage !== 'string' || !IMAGE.test(indexImage)) throw new Error(`UNPINNED_${label}_INDEX`);
if (!record(images)) throw new Error(`MISSING_${label}_IMAGES`);
const repository = repositoryOfTag(source);
@@ -101,7 +104,7 @@ export function imageBuildMatrix(input: unknown): { include: ImageBuildRow[] } {
if (typeof stack !== 'string' || !CSO_RUNTIME_STACKS.includes(stack as CsoRuntimeBuildStack) || seenStacks.has(stack)) throw new Error('INVALID_STACK');
seenStacks.add(stack);
const typedStack = stack as CsoRuntimeBuildStack;
const bases = assertPinnedFamily(source, indexImage, baseImages, 'BASE');
const bases = assertPinnedFamily(source, indexImage, baseImages, 'BASE', typedStack === 'postgresql' ? TWO_PART_RELEASE_TAG : RELEASE_TAG);
if (!record(versions)) throw new Error('MISSING_TOOL_VERSIONS');
const exactVersion = (value: unknown) => typeof value === 'string' &&
(typedStack === 'postgresql'
+36 -5
View File
@@ -27,10 +27,18 @@
# teardown) times the size's price. Without a key it runs the same files
# locally in parallel shards and reports spent=0.
#
# Reattach: the tree and the planned set's digest go into ubi-runner's run key
# (--key-input), so a rerun after the local runner died (a machine stop, a
# SIGKILL) reattaches to the VM still running this gate instead of paying for a
# second one. Its minutes since created_at count against the cap: the wall
# budget shrinks by them (PREGATE_REMOTE_CAP_SPENT when none is left), and
# spent= bills from the original create.
#
# Exit codes: 0 pass (or a paid-only selection, tests=0); 1 test failures (or
# a rejected file list); 2 missing input; 3 tree or plan mismatch (or the tree
# moved during the run); 4 infrastructure failure (create, setup, SSH, stop
# deadline); 5 the spend cap or pregate's deadline refuses the run. Every refusal line starts with a
# deadline, another gate on this machine owning the VM to reattach to); 5 the
# spend cap or pregate's deadline refuses the run. Every refusal line starts with a
# PREGATE_REMOTE_* reason code (docs/pregate.md#remote-gate). Nothing is
# uploaded or spent before the local tree check.
set -euo pipefail
@@ -147,8 +155,6 @@ if [ -n "$outer_s" ]; then
fi
if [ "$outer_stop" -lt "$stop_after" ]; then stop_after=$outer_stop stop_why="$outer_from ${outer_s}s minus 120 s teardown reserve and 15 s margin"; fi
fi
say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)"
digest=$(tr -d '\r' <"$files" | grep . | LC_ALL=C sort | sha256sum | cut -d' ' -f1)
prelude="$logs/vm-setup.sh"
{
@@ -183,6 +189,28 @@ args=(run -s "$size" --src "$root" --setup "$prelude"
--pull "/tmp/gstack-free-test-*:$logs"
--pull "work/$(basename "$root")/.context/free-test-logs:$logs")
[ -z "${GSTACK_FREE_JOBS:-}" ] || args+=(--env "GSTACK_FREE_JOBS=$GSTACK_FREE_JOBS")
args+=(--key-input "pregate-remote tree=$tree plan=$digest")
# A rerun of this gate for the same tree and planned set has the same
# ubi-runner run key, so it reattaches to the VM a killed run left behind
# (`ubi-runner: reattached to <vm> created_at=<epoch>`). The minutes that VM
# has billed since created_at come off the cap's wall budget first.
prior_vm=""
attach=$(bash "$here/ubi-runner.sh" "${args[@]}" --probe -- "bash ~/.pregate-run.sh" 2>/dev/null \
| sed -n 's/^UBI_ATTACH_INFO: vm=\([^ ]*\) created_at=\([0-9]*\) key=[0-9a-f]* busy=\([01]\)$/\1 \2 \3/p' | tail -1) || attach=""
if [ -n "$attach" ]; then
read -r prior_vm prior_created prior_busy <<<"$attach"
[ "$prior_busy" = 0 ] || stop 4 PREGATE_REMOTE_ATTACH_BUSY "another gate on this machine still runs VM $prior_vm for this tree and planned set" "wait for it; this run created nothing and spent nothing"
prior_min=$(( ($(date +%s) - prior_created + 59) / 60 ))
left_s=$(( (wall_min - 3 - prior_min) * 60 ))
if [ "$left_s" -le 0 ]; then
stop 5 PREGATE_REMOTE_CAP_SPENT "VM $prior_vm, created ${prior_min} min ago for this tree and planned set, has already used the cap's $wall_min min wall (\$$(usd "$prior_min") of \$$cap)" \
"nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down $prior_vm" "or rerun with a higher --spend-cap to let it finish: $rerun"
fi
if [ "$left_s" -lt "$stop_after" ]; then stop_after=$left_s stop_why="the cap's $wall_min min wall minus 3 min for teardown and $prior_min min already billed since $prior_vm was created"; fi
say "reattaching to VM $prior_vm, created ${prior_min} min ago by an earlier run of this gate for the same tree and planned set"
fi
say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)"
# The VM run gets its own session (python3, which ubi-runner already needs, for
# setsid on Linux and macOS): pregate's --timeout kill signals this
@@ -220,7 +248,7 @@ while :; do
rm -f "$log.pid" "$log.timedout" "$log.left"
t0=$(date +%s)
vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$(( (stop_after + 900 + 3599) / 3600 ))"
say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
[ -n "$prior_vm" ] || say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
UBI_LEFT_FILE="$log.left" python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
bash -c "$vm_session" pregate-remote-vm "$stop_after" "$log" "$stream" "$here/ubi-runner.sh" "${args[@]}" -n "$vm_name" &
vmrun=$!
@@ -236,6 +264,9 @@ while :; do
tailer=""
t_end=$(date +%s)
t_create=$(at 'ubi-runner: creating ')
# A reattached VM is billed from its original create.
reattached=$(sed -n 's/^[0-9]* ubi-runner: reattached to [^ ]* created_at=\([0-9][0-9]*\) .*/\1/p' "$log" | tail -1)
[ -z "$reattached" ] || t_create=$reattached
if [ -n "$t_create" ] && ! grep -q 'ubi-runner: create failed' "$log"; then minutes=$(( (t_end - t_create + 59) / 60 )); billed=$(( billed + (minutes > 0 ? minutes : 1) )); fi
if [ "$rc" -ne 0 ] && [ "$attempt" -eq 1 ] && grep -q 'vCPU quota exhausted' "$log"; then
say "vCPU quota refused the VM; one retry in ${PREGATE_REMOTE_QUOTA_WAIT_S:-60} s"
@@ -279,7 +310,7 @@ if [ -n "$result" ] && [ "$rc" -ne 124 ]; then
echo "PREGATE_REMOTE: tree=$vm_tree tests=$(field tests "$result") failed=$(field failed "$result") spent=$spent"
exit $(( rc == 0 ? 0 : 1 ))
fi
vm=$(sed -n 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' "$log" | tail -1)
vm=$(sed -n -e 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' -e 's/^[0-9]* ubi-runner: reattached to \([^ ]*\) .*/\1/p' "$log" | tail -1)
down=$(grep -E 'ubi-runner: (destroyed|gone:|WARNING|FAILED to destroy)' "$log" | tail -1 | cut -d' ' -f2- || true)
cause=$(grep -E 'ubi-runner: ' "$log" | grep -vE 'ubi-runner: (destroyed|gone:|destroy requested|waiting for an in-flight)' | tail -1 | cut -d' ' -f2- || true)
[ "$rc" -ne 124 ] || cause="the VM was told to stop after ${stop_after}s ($stop_why) and torn down"
+628 -40
View File
@@ -33,8 +33,9 @@
# Teardown: `up` records each VM's state (location, create status) before it
# sends the create request, and `down` destroys and polls until the VM is
# confirmed gone or provably never existed, waiting out a create that may still
# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, HUP, QUIT) runs
# that `down`; further signals are ignored until it finishes. A create refused
# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, QUIT, and HUP
# until a remote phase is live) runs that `down`; further signals are ignored
# until it finishes. A create refused
# for vCPU quota prints the project's usage per owner.
#
# The owner-age sweep is off by default (the TTL reap below always runs). With UBI_GC_HOURS set to a positive
@@ -69,6 +70,33 @@
# UBI_RUN_TIMEOUT seconds (0, unlimited), exiting 124 with the last 40 lines,
# so a hung verify fails fast instead of holding the VM until its TTL.
#
# Detached phases and reattach: `run` and `verify` start their setup, command
# and tier phases on the VM detached from the ssh connection (setsid; stdout to
# ~/.ubirun-<phase>.out, stderr to .err, the session leader's pid in .pid, and
# the exit status in .exit, written by rename once the streams are closed) and
# stream both files back. Before creating a VM they compute a run key locally
# (the checkout's content tree, the command, the setup script's content, every
# --env entry, each --pass name with a sha256 of its value, size, location,
# image, storage, origin/HEAD, tiers and this script's own sha256) and look for
# a run record (<state dir>/<vm>/run) with the same key whose VM still exists:
# its runner alive (lease pid and start time) refuses with UBI_ATTACH_BUSY,
# otherwise the run reattaches, printing `reattached to <vm> created_at=<epoch>
# key=<k12>`, and continues from the first phase without an exit file. After
# unpack the VM recomputes the tree and refuses a mismatch (UBI_VM_TREE_MISMATCH).
# A leader gone with no exit file exits 125 (UBI_ATTACH_LOST). Signals: INT
# and TERM tear the VM down as before; HUP once a remote phase is live detaches
# (UBI_DETACHED, exit 129) and leaves the VM under its TTL; SIGKILL or a machine
# stop leaves the record and the VM, and the next run of the same command
# reattaches. `attach NAME` resumes a recorded run explicitly. Reattach never
# extends a TTL, and the reaper still applies.
#
# Tier checkpoints: a .gstack/ubicloud.json with `tiers` runs each tier as its
# own detached phase after setup, and records start/pass/fail per tier under
# UBI_CHECKPOINT_DIR/<run key>/ (write-once JSON plus the tier's log, published
# under a lock with a monotonic generation). A rerun reuses a pass younger than
# UBI_CHECKPOINT_MAX_AGE whose log re-hashes, so a resume on a fresh VM reruns
# only setup and the tiers without a pass.
#
# `verify` is the one entry point for a repository's full remote verify: it
# reads .gstack/ubicloud.json (setup, command, size, location, env, pass, pull)
# and runs it through `run`, so lanes share one committed definition instead of
@@ -100,6 +128,11 @@ PREFIX="ubirun"
if [ -n "${UBI_VM_RECORD:-}" ]; then VM_RECORD="$UBI_VM_RECORD"
elif [ -d "$HOME/.capy/work" ]; then VM_RECORD="$HOME/.capy/work/ubi-runner/vms.tsv"
else VM_RECORD="$STATE_ROOT/vms.tsv"; fi
if [ -n "${UBI_CHECKPOINT_DIR:-}" ]; then CHECKPOINT_DIR="$UBI_CHECKPOINT_DIR"
elif [ -d "$HOME/.capy/work" ]; then CHECKPOINT_DIR="$HOME/.capy/work/ubi-runner/checkpoints"
else CHECKPOINT_DIR="$STATE_ROOT/checkpoints"; fi
CHECKPOINT_MAX_AGE="${UBI_CHECKPOINT_MAX_AGE:-86400}"
REATTACH_MARGIN="${UBI_REATTACH_MARGIN:-900}"
die() { echo "ubi-runner: $*" >&2; exit 1; }
log() { echo "ubi-runner: $*" >&2; }
@@ -109,7 +142,7 @@ for bin in curl python3 ssh ssh-keygen tar; do
done
UBICLOUD_API_KEY="${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}"
[[ "$GC_HOURS" =~ ^[0-9]+$ ]] || die "UBI_GC_HOURS must be a whole number of hours (unset or 0 disables the sweep)"
for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT; do
for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT CHECKPOINT_MAX_AGE REATTACH_MARGIN; do
[[ "${!v}" =~ ^[0-9]+$ ]] || die "UBI_$v must be a whole number of seconds (0 disables it)"
done
[ -z "$QUOTA_VCPUS" ] || [[ "$QUOTA_VCPUS" =~ ^[0-9]+$ ]] || die "UBI_QUOTA_VCPUS must be a whole number of vCPUs"
@@ -296,6 +329,410 @@ watched() {
return "$rc"
}
proc_start() { ps -o lstart= -p "$1" 2>/dev/null | tr -s ' ' || true; }
fsize() { if [ -f "$1" ]; then wc -c <"$1" | tr -d ' '; else echo 0; fi; }
sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum | cut -d' ' -f1; else shasum -a 256 | cut -d' ' -f1; fi; }
# TREE_SH: print the content tree of the current directory, the same way here
# and on the VM: at a checkout's root, `git add -A` into a copy of its index
# (tracked plus untracked files that are not ignored, the files pack uploads);
# elsewhere, the same into a scratch repository over the directory.
TREE_SH='idx=$(mktemp -d "${TMPDIR:-/tmp}/ubirun-tree.XXXXXX") || exit 1
if [ "$(git rev-parse --show-toplevel 2>/dev/null)" = "$(pwd -P)" ]; then
gi=$(git rev-parse --git-path index)
if [ "$(git config --bool core.splitIndex)" != true ] && [ -f "$gi" ]; then cp "$gi" "$idx/index"; fi
else
git init -q --bare "$idx/git" && export GIT_DIR="$idx/git" GIT_WORK_TREE=.
fi
GIT_INDEX_FILE="$idx/index" git add -A >/dev/null 2>&1 && GIT_INDEX_FILE="$idx/index" git write-tree
rc=$?; rm -rf "$idx"; exit $rc'
# vm_tree NAME QDEST: the content tree of QDEST on the VM.
vm_tree() {
remote "$1" "command -v git >/dev/null || { sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git; } >/dev/null 2>&1; cd $2 && bash -c $(sq "$TREE_SH")" </dev/null
}
base_sha() { git -C "$1" rev-parse -q --verify 'refs/remotes/origin/HEAD^{commit}' 2>/dev/null || echo none; }
# phase_status NAME PHASE: "exit <rc>" once the phase wrote its exit file,
# "alive" while its session leader runs, "lost" when the leader is gone with no
# exit file, "none" when it never started.
phase_status() {
local p="~/.ubirun-$2"
remote "$1" "if [ -f $p.exit ]; then echo exit \$(cat $p.exit); elif [ -s $p.pid ] && kill -0 \$(cat $p.pid) 2>/dev/null; then echo alive; elif [ -s $p.pid ]; then echo lost; else echo none; fi" </dev/null
}
# phase_start NAME PHASE INNER: start INNER on the VM in its own session,
# detached from this ssh connection. The leader records its pid, runs INNER with
# stdout and stderr in files, and writes the exit status by rename only after
# INNER's streams are closed.
phase_start() {
local p="~/.ubirun-$2" inner
inner=$(sq "echo \$\$ >$p.pid.tmp && mv $p.pid.tmp $p.pid; ( $3 ) >$p.out 2>$p.err </dev/null; echo \$? >$p.exit.tmp && mv $p.exit.tmp $p.exit")
remote "$1" "rm -f $p.exit $p.pid; : >$p.out; : >$p.err; if command -v setsid >/dev/null 2>&1; then setsid -f bash -c $inner </dev/null >/dev/null 2>&1; else nohup bash -c $inner </dev/null >/dev/null 2>&1 & fi" </dev/null >/dev/null
}
# stream_cmd PHASE OUT_OFFSET ERR_OFFSET: follow both output files from the
# given byte offsets until the session leader exits.
stream_cmd() {
local p="~/.ubirun-$1"
printf '%s' "i=0; while [ ! -s $p.pid ] && [ ! -f $p.exit ] && [ \$i -lt 600 ]; do sleep 0.1; i=\$((i + 1)); done; pid=\$(cat $p.pid 2>/dev/null); f=; [ -z \"\$pid\" ] || f=\"-f --pid=\$pid\"; tail -c +$(( $3 + 1 )) \$f $p.err >&2 & tail -c +$(( $2 + 1 )) \$f $p.out; wait"
}
PHASE_LIVE="" STREAM_PID="" RUN_KEY="" TREE=""
# on_hup: once a remote phase is live, HUP detaches: the VM keeps running under
# its TTL and the next run of the same command reattaches. Before that, HUP
# tears down like INT and TERM.
on_hup() {
[ -n "$PHASE_LIVE" ] || exit 130
[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null || true
trap - EXIT
rm -f "$(state_dir "$RUN_VM")/lease"
leftover "left=1 owner=$OWNER vm=$RUN_VM detached=1"
echo "UBI_DETACHED: vm=$RUN_VM key=${RUN_KEY:0:12} reattach: rerun the same command" >&2
exit 129
}
# detached NAME PHASE INNER: run INNER as a detached phase (or reattach to it),
# streaming its output to this process and to the VM's state dir
# (<phase>.out, <phase>.err and the combined <phase>.log) under the same
# heartbeat and watchdog as watched. RUN_TIMEOUT counts from the phase's
# recorded start, not from a reattach. Returns the phase's exit status, 124
# after a watchdog kill, 125 when the session is gone without an exit status.
detached() {
local name=$1 phase=$2 inner=$3 dir st rc=0 tries=0 start now idle elapsed last_beat why old_traps spid sst oo oe
load "$name"
dir=$(state_dir "$name")
local lout="$dir/$phase.out" lerr="$dir/$phase.err" out="$dir/$phase.log" p="~/.ubirun-$phase"
st=none
if [ -f "$dir/phase.$phase.start" ]; then
st=$(phase_status "$name" "$phase") || st=""
fi
case $st in
none)
: >"$lout"; : >"$lerr"; : >"$out"
date +%s >"$dir/phase.$phase.start"
phase_start "$name" "$phase" "$inner" || { log "could not start $phase on $name"; return 255; } ;;
alive) log "$phase is still running on $name; following it" ;;
"exit "*) ;;
lost) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; return 125 ;;
*) log "could not read the state of $phase on $name"; return 255 ;;
esac
PHASE_LIVE=1
start=$(cat "$dir/phase.$phase.start")
# A runner killed on this machine leaves its stream client behind; it would
# append to the same local copies.
if [ -f "$dir/stream" ] && read -r spid sst <"$dir/stream" && [ -n "$spid" ] && [ "$(proc_start "$spid")" = "$sst" ]; then
kill -TERM -- "-$spid" 2>/dev/null || true
fi
oo=$(fsize "$lout") oe=$(fsize "$lerr")
if [ "$st" != none ]; then head -c "$oo" "$lout"; head -c "$oe" "$lerr" >&2; fi
old_traps=$(trap -p INT TERM)
trap '[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null; exit 130' INT TERM
last_beat=$(date +%s)
while :; do
touch "$out"
# shellcheck disable=SC2046
( exec python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
ssh $(ssh_opts) "ubi@$IP" "umask 022; $(stream_cmd "$phase" "$oo" "$oe")" ) \
</dev/null > >(tee -a "$lout" | tee -a "$out") 2> >(tee -a "$lerr" | tee -a "$out" >&2) &
STREAM_PID=$!
echo "$STREAM_PID $(proc_start "$STREAM_PID")" >"$dir/stream"
why=""
while :; do
sleep 1
kill -0 "$STREAM_PID" 2>/dev/null || break
now=$(date +%s)
elapsed=$(( now - start ))
idle=$(( now - $(mtime "$out") ))
if [ "$IDLE_TIMEOUT" -gt 0 ] && [ "$idle" -ge "$IDLE_TIMEOUT" ]; then
why="printed nothing for ${idle}s (UBI_IDLE_TIMEOUT=$IDLE_TIMEOUT)"
elif [ "$RUN_TIMEOUT" -gt 0 ] && [ "$elapsed" -ge "$RUN_TIMEOUT" ]; then
why="ran ${elapsed}s (UBI_RUN_TIMEOUT=$RUN_TIMEOUT)"
fi
[ -z "$why" ] || break
if [ "$HEARTBEAT" -gt 0 ] && [ $(( now - last_beat )) -ge "$HEARTBEAT" ]; then
log "heartbeat: $phase ${elapsed}s elapsed, last output ${idle}s ago"
last_beat=$now
fi
done
if [ -n "$why" ]; then
log "watchdog: $phase $why; killing it. Last output:"
tail -n 40 "$out" >&2
kill -TERM -- "-$STREAM_PID" 2>/dev/null || kill "$STREAM_PID" 2>/dev/null || true
wait "$STREAM_PID" 2>/dev/null || true
STREAM_PID=""
# A phase that finished in the instant before the kill keeps its exit 0.
if [ "$(phase_status "$name" "$phase" 2>/dev/null)" = "exit 0" ]; then
log "watchdog: $phase finished before it could be killed; its exit 0 stands"
rc=0
else
remote_cancel "$name" "$p.pid"
rc=124
fi
break
fi
wait "$STREAM_PID" 2>/dev/null || true
STREAM_PID=""
st=$(phase_status "$name" "$phase") || st=""
case $st in
"exit "*) rc=${st#exit }; break ;;
lost|none) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; rc=125; break ;;
esac
tries=$(( tries + 1 ))
if [ "$tries" -ge 5 ]; then log "lost the connection to $phase on $name 5 times; giving up (the VM keeps it until teardown)"; rc=255; break; fi
log "the stream of $phase on $name dropped; following it again"
sleep "$POLL"
oo=$(fsize "$lout") oe=$(fsize "$lerr")
done
rm -f "$dir/stream"
trap - INT TERM; eval "$old_traps"
return "$rc"
}
# write_record DIR: the run record a later run of the same command reattaches
# by (key, tree, and this invocation for `attach NAME`). Remote files of each
# phase: ~/.ubirun-<phase>.{out,err,exit,pid} (setup, command, tier-<name>).
qwords() { local x; for x in "$@"; do printf '%q ' "$x"; done; }
write_record() {
{
printf 'RUN_KEY=%q\nTREE=%q\nPULL_ROOT=%q\nRESUME=%q\nRERUN_FAILED=%q\n' "$RUN_KEY" "$TREE" "$PULL_ROOT" "$RESUME" "$RERUN_FAILED"
printf 'RUN_ARGS=(%s)\n' "$(qwords "${RUN_ARGS[@]}")"
printf 'TIER_NAMES=(%s)\n' "$(qwords ${TIER_NAMES[@]+"${TIER_NAMES[@]}"})"
printf 'TIER_CMDS=(%s)\n' "$(qwords ${TIER_CMDS[@]+"${TIER_CMDS[@]}"})"
echo "# remote files: ~/.ubirun-<phase>.{out,err,exit,pid}; phase starts: phase.<phase>.start; lease: lease (pid start time)"
} >"$1/run.$$" && mv "$1/run.$$" "$1/run"
}
lease_take() { echo "$$ $(proc_start $$)" >"$1/lease.$$" && mv "$1/lease.$$" "$1/lease"; }
lease_busy() {
local pid st
[ -f "$1/lease" ] || return 1
read -r pid st <"$1/lease" || true
[ -n "$pid" ] && [ "$pid" != "$$" ] && kill -0 "$pid" 2>/dev/null && [ "$(proc_start "$pid")" = "$st" ]
}
# vm_alive NAME: the recorded VM still answers show. One confirmed gone is
# cleaned up (state dir and scratch record), so its record is never tried again.
vm_alive() {
local n=$1 out code
( load "$n"; [ "$CREATED" = yes ] && [ -n "$IP" ] ) 2>/dev/null || return 1
out=$(mktemp)
code=$(load "$n"; api "$out" vm "$LOCATION/$n" show)
rm -f "$out"
case $code in
200) return 0 ;;
404) log "run record for $n: the VM is gone; starting a new one"; cmd_down "$n" >/dev/null 2>&1 || true; return 1 ;;
*) return 1 ;;
esac
}
# ttl_short NAME SETUP: true when NAME's TTL (its created_at plus the name's
# -t<hours>, minus now) does not cover the run still to do plus
# REATTACH_MARGIN, with the numbers in TTL_WHY. The run still to do is the
# UBI_RUN_TIMEOUT bound of each phase that has not finished (setup when SETUP is
# set, then the command or every tier the plan does not reuse), counted from a
# started phase's recorded start; it is 0 when UBI_RUN_TIMEOUT is 0.
TTL_WHY=""
ttl_short() {
local name=$1 dir created left now need=0 p i phases=()
[[ "$name" =~ -t([0-9]{1,3})$ ]] || return 1
dir=$(state_dir "$name")
created=$(load "$name"; echo "$CREATE_AT")
now=$(date +%s)
left=$(( created + 10#${BASH_REMATCH[1]} * 3600 - now ))
[ -z "$2" ] || phases+=(setup)
if [ ${#TIER_NAMES[@]} -gt 0 ]; then
for i in "${!TIER_NAMES[@]}"; do [ "${TIER_ACT[i]}" = reuse ] || phases+=("tier-${TIER_NAMES[i]}"); done
else
phases+=(command)
fi
if [ "$RUN_TIMEOUT" -gt 0 ]; then
for p in "${phases[@]}"; do
if [ -f "$dir/phase.$p.start" ]; then i=$(( RUN_TIMEOUT - (now - $(cat "$dir/phase.$p.start")) )); [ "$i" -gt 0 ] || i=0
else i=$RUN_TIMEOUT; fi
need=$(( need + i ))
done
fi
TTL_WHY="ttl_left=${left}s needs=$(( need + REATTACH_MARGIN ))s (run still to do ${need}s + UBI_REATTACH_MARGIN=${REATTACH_MARGIN}s)"
[ "$left" -lt $(( need + REATTACH_MARGIN )) ]
}
# find_record: the VM of a run record with this RUN_KEY whose VM still exists.
find_record() {
local f n
for f in "$STATE_ROOT"/*/run; do
[ -f "$f" ] || continue
grep -qx "RUN_KEY=$RUN_KEY" "$f" || continue
n=$(basename "$(dirname "$f")")
if vm_alive "$n"; then echo "$n"; return 0; fi
done
}
# Tier checkpoints: write-once records <tier>.<epoch>.<pid>.json beside a copy
# of the tier's log, under CHECKPOINT_DIR/<run key>/, published under a lock
# with a monotonic generation; the newest generation per tier decides.
CKPT_PY='
import fcntl, hashlib, json, os, shutil, sys, time
op, d = sys.argv[1], sys.argv[2]
def records():
out = []
for f in (os.listdir(d) if os.path.isdir(d) else []):
if not f.endswith(".json") or f.startswith("."):
continue
try:
r = json.load(open(os.path.join(d, f)))
except (OSError, ValueError):
continue
if isinstance(r, dict) and isinstance(r.get("generation"), int):
r["_file"] = f
out.append(r)
return out
def newest(rs, tier):
rs = [r for r in rs if r.get("tier") == tier]
return max(rs, key=lambda r: r["generation"]) if rs else None
def sha(p):
h = hashlib.sha256()
with open(p, "rb") as fh:
for b in iter(lambda: fh.read(65536), b""):
h.update(b)
return h.hexdigest()
if op == "decide":
now, maxage, resume, rerun = int(sys.argv[3]), int(sys.argv[4]), sys.argv[5] == "1", sys.argv[6] == "1"
rs = records()
for tier in sys.argv[7:]:
r = newest(rs, tier)
f = r["_file"] if r else "-"
age = now - int(r.get("finished_at") or r.get("started_at") or now) if r else 0
dur = int(r.get("duration_s") or 0) if r else 0
code = str(r.get("exit", "-")) if r else "-"
log = os.path.join(d, str(r.get("log"))) if r and r.get("log") else ""
if not resume:
act, why = "run", "--no-resume"
elif r is None:
act, why = "run", "no record"
elif r.get("status") == "start":
act, why = "run", "interrupted (only a start record, %s)" % f
elif r.get("status") == "pass":
if age >= maxage:
act, why = "run", "pass %s is %ds old (UBI_CHECKPOINT_MAX_AGE=%d)" % (f, age, maxage)
elif not log or not os.path.isfile(log) or sha(log) != r.get("log_sha256"):
act, why = "run", "pass %s: its log does not re-hash" % f
else:
act, why = "reuse", "-"
elif r.get("status") == "fail":
act, why = ("rerun", "prior fail %s (exit %s; --rerun-failed)" % (f, code)) if rerun else ("blocked", "-")
else:
act, why = "run", "unreadable record %s" % f
print("\t".join([tier, act, f, str(age), str(dur), code, why]))
elif op == "publish":
tier, status, vm, started, finished, code, log, startgen, key, tree, pid = sys.argv[3:14]
os.makedirs(d, exist_ok=True)
with open(os.path.join(d, ".lock"), "a") as lk:
fcntl.flock(lk, fcntl.LOCK_EX)
rs = records()
gen = max([r["generation"] for r in rs] + [0]) + 1
if status != "start":
r = newest(rs, tier)
if r is None or str(r["generation"]) != startgen:
print("stale writer: its start record is generation %s, the newest for %s is %s" % (startgen, tier, r["generation"] if r else "none"), file=sys.stderr)
sys.exit(2)
rec = {"tier": tier, "status": status, "generation": gen, "vm": vm, "key": key, "tree": tree, "started_at": int(started)}
if status != "start":
rec.update(finished_at=int(finished), duration_s=int(finished) - int(started), exit=int(code))
epoch = int(time.time())
while os.path.exists(os.path.join(d, "%s.%d.%s.json" % (tier, epoch, pid))) or os.path.exists(os.path.join(d, "%s.%d.%s.log" % (tier, epoch, pid))):
epoch += 1
stem = "%s.%d.%s" % (tier, epoch, pid)
if status != "start":
shutil.copyfile(log, os.path.join(d, stem + ".log"))
rec["log"], rec["log_sha256"] = stem + ".log", sha(os.path.join(d, stem + ".log"))
tmp = os.path.join(d, ".%s.tmp" % stem)
with open(tmp, "w") as fh:
json.dump(rec, fh, sort_keys=True)
fh.write("\n")
os.link(tmp, os.path.join(d, stem + ".json"))
os.unlink(tmp)
print(gen)
'
# ckpt_publish TIER STATUS VM STARTED FINISHED EXIT LOG STARTGEN: print the new
# record's generation; fails (nothing published) for a stale writer.
ckpt_publish() { python3 -c "$CKPT_PY" publish "$CHECKPOINT_DIR/$RUN_KEY" "$@" "$RUN_KEY" "$TREE" "$$"; }
TIER_ACT=() TIER_TO_RUN=0 TIER_PASSED=0 TIER_REUSED="" TIER_FAILED="" TIER_PRIOR="" TIER_SAVED=0 TIER_BLOCKED="" TIER_BLOCKED_RC=1
# tier_plan: decide each tier from its newest checkpoint and print one
# UBI_VERIFY_RESUME line per tier before anything runs.
tier_plan() {
local plan t act f age dur code why
plan=$(python3 -c "$CKPT_PY" decide "$CHECKPOINT_DIR/$RUN_KEY" "$(date +%s)" "$CHECKPOINT_MAX_AGE" "$RESUME" "$RERUN_FAILED" "${TIER_NAMES[@]}") \
|| die "verify: could not read the tier checkpoints in $CHECKPOINT_DIR/$RUN_KEY"
while IFS=$'\t' read -r t act f age dur code why; do
case $act in
reuse)
echo "UBI_VERIFY_RESUME: tier=$t reuse $f (${age}s)" >&2
TIER_REUSED=${TIER_REUSED:+$TIER_REUSED,}$t TIER_SAVED=$(( TIER_SAVED + dur )) TIER_PASSED=$(( TIER_PASSED + 1 )) ;;
blocked)
echo "UBI_VERIFY_RESUME: tier=$t fail $f (${age}s, exit $code): not rerun without --rerun-failed" >&2
if [ -z "$TIER_BLOCKED" ]; then TIER_BLOCKED=$t; if [[ "$code" =~ ^[0-9]+$ ]]; then TIER_BLOCKED_RC=$code; fi; fi ;;
rerun)
echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
TIER_PRIOR=${TIER_PRIOR:+$TIER_PRIOR,}$t:$code TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
*)
echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
esac
TIER_ACT+=("$act")
done <<<"$plan"
}
verify_line() {
echo "UBI_VERIFY: tree=${TREE:0:12} key=${RUN_KEY:0:12} tiers=${#TIER_NAMES[@]} passed=$TIER_PASSED reused=${TIER_REUSED:-none} failed=${TIER_FAILED:-none} prior_failures=${TIER_PRIOR:-none} saved_s=$TIER_SAVED"
}
# run_tiers NAME QDEST: run every tier the plan did not reuse, in order, each
# as its own detached phase; stop at the first failure and return its status.
# A pass is recorded only when the VM's tree after the tier equals the tree
# before it.
run_tiers() {
local name=$1 qdest=$2 dir i t phase before after trc started finished gen
dir=$(state_dir "$name")
for i in "${!TIER_NAMES[@]}"; do
[ "${TIER_ACT[i]}" != reuse ] || continue
t=${TIER_NAMES[i]} phase="tier-${TIER_NAMES[i]}"
if [ ! -f "$dir/phase.$phase.start" ]; then
before=$(vm_tree "$name" "$qdest") || before=unavailable
echo "$before" >"$dir/phase.$phase.tree"
if gen=$(ckpt_publish "$t" start "$name" "$(date +%s)" 0 0 "" 0); then echo "$gen" >"$dir/phase.$phase.gen"
else echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t start record (the tier still runs)" >&2; fi
fi
log "running tier $t on $name: ${TIER_CMDS[i]}"
trc=0
detached "$name" "$phase" "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "${TIER_CMDS[i]}")" || trc=$?
started=$(cat "$dir/phase.$phase.start" 2>/dev/null || date +%s) finished=$(date +%s)
log "tier $t exited $trc after $(( finished - started ))s"
case $trc in 125|255) TIER_FAILED=$t; return "$trc" ;; esac
gen=$(cat "$dir/phase.$phase.gen" 2>/dev/null || echo 0)
if [ "$trc" -ne 0 ]; then
ckpt_publish "$t" fail "$name" "$started" "$finished" "$trc" "$dir/$phase.log" "$gen" >/dev/null \
|| echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t fail record (the result stands)" >&2
TIER_FAILED=$t
return "$trc"
fi
TIER_PASSED=$(( TIER_PASSED + 1 ))
after=$(vm_tree "$name" "$qdest") || after=unavailable
before=$(cat "$dir/phase.$phase.tree" 2>/dev/null || echo unavailable)
if [ "$after" != "$before" ] || [ "$after" = unavailable ]; then
echo "UBI_VERIFY_TREE_MOVED: tier=$t before=${before:0:12} after=${after:0:12}: a tier must leave the source tree as it found it; no pass record" >&2
continue
fi
ckpt_publish "$t" pass "$name" "$started" "$finished" 0 "$dir/$phase.log" "$gen" >/dev/null \
|| echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t pass record (the result stands)" >&2
done
}
# vm_rows: "location name owner expires created" for every VM in the project;
# owner is "-" for untagged ubirun-<epoch>-* names and "." for VMs this runner
# did not name; expires is the epoch the name's -t<hours> TTL ends at, or "-";
@@ -770,9 +1207,11 @@ run_teardown() {
[ "$rc" -ne 0 ] || exit 1
}
TIER_NAMES=() TIER_CMDS=() RESUME=1 RERUN_FAILED=0 FORCE_ATTACH="" RUN_ARGS=()
cmd_run() {
require_owner
local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=()
local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=() passes=() key_inputs=() probe=0
RUN_ARGS=("$@")
while [ $# -gt 0 ]; do
case $1 in
-S|--storage) check_storage "$2"; up_args+=("$1" "$2"); shift 2 ;;
@@ -781,60 +1220,161 @@ cmd_run() {
--dest) dest=$2; shift 2 ;;
--setup) setup=$2; shift 2 ;;
--env) envs+=("$2"); shift 2 ;;
--pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; envs+=("$2=${!2}"); shift 2 ;;
--pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; passes+=("$2"); shift 2 ;;
--pull) pulls+=("$2"); shift 2 ;;
--keep) keep=1; shift ;;
--key-input) key_inputs+=("$2"); shift 2 ;;
--probe) probe=1; shift ;;
--resume) RESUME=1; shift ;;
--no-resume) RESUME=0; shift ;;
--rerun-failed) RERUN_FAILED=1; shift ;;
--) shift; break ;;
*) die "run: unknown option $1" ;;
esac
done
[ $# -gt 0 ] || die "run: missing command after --"
[ $# -gt 0 ] || [ ${#TIER_NAMES[@]} -gt 0 ] || die "run: missing command after --"
[ -z "$setup" ] || [ -f "$setup" ] || die "setup script not found: $setup"
command -v git >/dev/null || die "git is required: the run key fingerprints the checkout with it"
src=$(cd "$src" && pwd)
[ -n "$dest" ] || dest="work/$(basename "$src")"
local i name=""
local i name="" size=$DEFAULT_SIZE location=$DEFAULT_LOCATION image=ubuntu-noble storage="" e n
for (( i = 0; i < ${#up_args[@]}; i += 2 )); do
case ${up_args[i]} in -n|--name) name=${up_args[i+1]} ;; esac
case ${up_args[i]} in
-n|--name) name=${up_args[i+1]} ;;
-s|--size) size=${up_args[i+1]} ;;
-l|--location) location=${up_args[i+1]} ;;
-b|--image) image=${up_args[i+1]} ;;
-S|--storage) storage=${up_args[i+1]} ;;
esac
done
# The run key: everything that decides what the VM runs, computed before any
# VM exists. Pass values enter only as their sha256.
if [ -z "$FORCE_ATTACH" ]; then
TREE=$(cd "$src" && bash -c "$TREE_SH") || die "could not fingerprint $src (git add -A into a temporary index failed)"
RUN_KEY=$({
echo "ubi-runner run key v1"
echo "tree $TREE"
printf 'command %q\n' "$*"
if [ -n "$setup" ]; then echo "setup $(sha256 <"$setup")"; else echo "setup none"; fi
for e in ${envs[@]+"${envs[@]}"}; do printf 'env %q\n' "$e"; done
for n in ${passes[@]+"${passes[@]}"}; do printf 'pass %s %s\n' "$n" "$(printf %s "${!n}" | sha256)"; done
printf 'vm %q %q %q %q\n' "$size" "$location" "$image" "$storage"
echo "base $(base_sha "$src")"
echo "runner $(sha256 <"${BASH_SOURCE[0]}")"
for e in ${key_inputs[@]+"${key_inputs[@]}"}; do printf 'input %q\n' "$e"; done
for i in "${!TIER_NAMES[@]}"; do printf 'tier %q %q\n' "${TIER_NAMES[i]}" "${TIER_CMDS[i]}"; done
} | sha256)
fi
[ ${#TIER_NAMES[@]} -eq 0 ] || tier_plan
local attach=""
if [ -n "$FORCE_ATTACH" ]; then
vm_alive "$FORCE_ATTACH" || die "attach: $FORCE_ATTACH no longer exists; rerun the original command to start fresh"
attach=$FORCE_ATTACH
else
attach=$(find_record)
fi
if [ "$probe" = 1 ]; then
if [ -z "$attach" ]; then echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12}"
else
local busy=0
! lease_busy "$(state_dir "$attach")" || busy=1
if [ "$busy" = 0 ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12} ttl_short vm=$attach $TTL_WHY"
else
echo "UBI_ATTACH_INFO: vm=$attach created_at=$(load "$attach"; echo "$CREATE_AT") key=${RUN_KEY:0:12} busy=$busy"
fi
fi
return 0
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && { [ -n "$TIER_BLOCKED" ] || [ "$TIER_TO_RUN" -eq 0 ]; } && [ -n "$attach" ] && ! lease_busy "$(state_dir "$attach")"; then
log "verify: no tier needs $attach, which a dead runner left behind; destroying it"
cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && [ -n "$TIER_BLOCKED" ]; then
TIER_FAILED=$TIER_BLOCKED
verify_line
return "$TIER_BLOCKED_RC"
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && [ "$TIER_TO_RUN" -eq 0 ]; then
log "verify: every tier passed under this key; no VM needed"
verify_line
return 0
fi
if [ -n "$attach" ] && lease_busy "$(state_dir "$attach")"; then
die "UBI_ATTACH_BUSY: vm=$attach key=${RUN_KEY:0:12}: runner pid $(cut -d' ' -f1 <"$(state_dir "$attach")/lease") on this machine still owns it; wait for it or stop it, then rerun"
fi
if [ -n "$attach" ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
log "UBI_REATTACH_TTL_SHORT: vm=$attach key=${RUN_KEY:0:12} $TTL_WHY: starting a fresh VM (tier checkpoints carry over) and destroying $attach"
cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
attach=""
fi
[ -z "$attach" ] || name=$attach
[ -n "$name" ] || { name=$(new_name); up_args+=(-n "$name"); }
RUN_VM=$name
# Armed before the create request: an interrupted `up` still destroys its VM.
# A second signal must not cut teardown short, so the EXIT handler ignores them.
trap 'run_teardown "$?"' EXIT
trap 'exit 130' INT TERM HUP QUIT
cmd_up "${up_args[@]}" >/dev/null
if [[ "$name" =~ -t([0-9]+)$ ]]; then
log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
trap 'exit 130' INT TERM QUIT
trap on_hup HUP
if [ -n "$attach" ]; then
load "$name"
log "reattached to $name created_at=$CREATE_AT key=${RUN_KEY:0:12}"
else
log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
cmd_up "${up_args[@]}" >/dev/null
trap 'exit 130' INT TERM QUIT
trap on_hup HUP
if [[ "$name" =~ -t([0-9]+)$ ]]; then
log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
else
log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
fi
fi
local dir
dir=$(state_dir "$name")
write_record "$dir"
lease_take "$dir"
if [ "$keep" = 1 ]; then
trap - EXIT
# A kept VM's finished run is not reattached by the next run; a detach (HUP) keeps the record.
trap 'rm -f "$(state_dir "$RUN_VM")/run"' EXIT
leftover "left=1 owner=$OWNER vm=$name kept=1"
log "--keep: leaving $name running; destroy with: $0 down $name"
fi
local e
for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done \
| remote "$name" 'umask 077 && cat > ~/.ubirun-env'
cmd_sync "$name" "$src" "$dest"
local qdest rc=0
local qdest rc=0 vmt
qdest=$(printf %q "$dest")
if [ ! -f "$dir/synced" ]; then
{ for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done
for n in ${passes[@]+"${passes[@]}"}; do printf 'export %s=%q\n' "$n" "${!n}"; done; } \
| remote "$name" 'umask 077 && cat > ~/.ubirun-env'
cmd_sync "$name" "$src" "$dest"
# A subdirectory of a checkout uploads the whole .git with only its own
# files, so its tree is compared VM to VM (tiers) but not to this side.
if [ -z "$(git -C "$src" rev-parse --show-prefix 2>/dev/null)" ]; then
vmt=$(vm_tree "$name" "$qdest") || vmt=unavailable
if [ "$vmt" != "$TREE" ]; then
log "UBI_VM_TREE_MISMATCH: vm=$name tree=${vmt:0:12} expected=${TREE:0:12}: the unpacked checkout is not the one this run was keyed on"
exit 3
fi
fi
[ -z "$setup" ] || remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup"
: >"$dir/synced"
fi
if [ -n "$setup" ]; then
remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup"
log "running setup $(basename "$setup")"
watched "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; }
detached "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; }
fi
local start
start=$(date +%s)
log "running on $name: $*"
watched "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$?
log "command exited $rc after $(( $(date +%s) - start ))s"
if [ ${#TIER_NAMES[@]} -gt 0 ]; then
run_tiers "$name" "$qdest" || rc=$?
else
log "running on $name: $*"
detached "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$?
log "command exited $rc after $(( $(date +%s) - $(cat "$dir/phase.command.start" 2>/dev/null || date +%s) ))s"
fi
local p
for p in ${pulls[@]+"${pulls[@]}"}; do
@@ -845,9 +1385,23 @@ cmd_run() {
fi
cmd_pull "$name" "${p%%:*}" "${p#*:}" || log "pull failed: $p"
done
[ ${#TIER_NAMES[@]} -eq 0 ] || verify_line
return "$rc"
}
# attach NAME: resume the run recorded for NAME on this machine (the same
# invocation, key and tiers), as a rerun of the same command would.
cmd_attach() {
local name=${1:-}
[ -n "$name" ] || die "attach: missing NAME"
valid_name "$name" || die "invalid VM name '$name'"
[ -f "$(state_dir "$name")/run" ] || die "attach: no run record for $name on this machine ($(state_dir "$name")/run); a machine replacement loses it"
# shellcheck disable=SC1091
. "$(state_dir "$name")/run"
FORCE_ATTACH=$name
cmd_run "${RUN_ARGS[@]}"
}
# verify [run opts] [--trust-config] [-- COMMAND]: run the repository's
# committed remote verify. .gstack/ubicloud.json under --src (default .)
# supplies setup, command, size, location, storage, image, env, pass and pull;
@@ -860,7 +1414,7 @@ cmd_run() {
# local variable it lists under pass is forwarded only when this command line
# also names it (--pass NAME) or grants every listed name (--trust-config).
cmd_verify() {
local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=()
local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=() cfg_tier_names=() cfg_tier_cmds=()
for (( i = 1; i <= $#; i++ )); do
[ "${!i}" = -- ] && break
if [ "${!i}" = --src ]; then j=$(( i + 1 )); src=${!j}; fi
@@ -869,7 +1423,7 @@ cmd_verify() {
[ -f "$cfg" ] || die "verify: no $cfg; commit one ({\"setup\": \"scripts/ubicloud/bootstrap.sh\", \"command\": \"bash scripts/ubicloud/verify.sh\"}) so every lane runs the same verify instead of its own wrapper"
err=$(mktemp)
parsed=$(python3 -c '
import json, os, shlex, sys
import json, os, re, shlex, sys
cfg, src = sys.argv[1], sys.argv[2]
try:
c = json.load(open(cfg))
@@ -895,7 +1449,25 @@ for item in c.get("pull", []):
if not sep or not remote:
print("pull entry %r must be REMOTE_GLOB:LOCAL_DIR" % (item,), file=sys.stderr); sys.exit(2)
args += ["--pull", q(remote + ":" + inside(local, "pull destination"))]
tiers = c.get("tiers")
names = []
if tiers is not None:
if "command" in c:
print("UBI_CONFIG_TIERS_AND_COMMAND: give either command or tiers, not both", file=sys.stderr); sys.exit(2)
if not isinstance(tiers, list) or not tiers:
print("UBI_CONFIG_TIER_NAME: tiers must be a non-empty list of {\"name\", \"command\"}", file=sys.stderr); sys.exit(2)
for t in tiers:
name = t.get("name") if isinstance(t, dict) else None
if not isinstance(name, str) or not re.fullmatch(r"[a-z0-9][a-z0-9_-]{0,31}", name):
print("UBI_CONFIG_TIER_NAME: tier name %r must match ^[a-z0-9][a-z0-9_-]{0,31}$" % (name,), file=sys.stderr); sys.exit(2)
if name in names:
print("UBI_CONFIG_TIER_NAME: tier name %r appears twice" % name, file=sys.stderr); sys.exit(2)
if not isinstance(t.get("command"), str) or not t["command"].strip():
print("UBI_CONFIG_TIER_NAME: tier %r needs a command" % name, file=sys.stderr); sys.exit(2)
names.append(name)
setup = c.get("setup", "")
print("cfg_tier_names=(" + " ".join(q(n) for n in names) + ")")
print("cfg_tier_cmds=(" + " ".join(q(t["command"]) for t in (tiers or [])) + ")")
print("cfg_args=(" + " ".join(args) + ")")
print("cfg_pass=(" + " ".join(q(x) for x in c.get("pass", [])) + ")")
print("cfg_setup=" + q(inside(setup, "setup") if setup else ""))
@@ -912,7 +1484,7 @@ print("cfg_cmd=" + q(c.get("command", "")))
case $1 in
--) shift; user_cmd=("$@"); break ;;
--trust-config) trust=1; shift ;;
--keep) user_pre+=("$1"); shift ;;
--keep|--resume|--no-resume|--rerun-failed|--probe) user_pre+=("$1"); shift ;;
--pass|-S|--storage|-s|--size|-l|--location|-n|--name|-b|--image|--src|--dest|--setup|--env|--pull)
[ $# -ge 2 ] || die "verify: $1 needs a value"
[ "$1" != --pass ] || granted+=("$2")
@@ -927,11 +1499,16 @@ print("cfg_cmd=" + q(c.get("command", "")))
missing+=("$n")
done
[ ${#missing[@]} -eq 0 ] || die "verify: $cfg asks to forward local ${missing[*]}; grant each on this command line with --pass NAME, or every name it lists with --trust-config"
[ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --"
[ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd")
# A trailing -- COMMAND replaces the configured tiers as it replaces command.
if [ ${#user_cmd[@]} -eq 0 ] && [ ${#cfg_tier_names[@]} -gt 0 ]; then
TIER_NAMES=("${cfg_tier_names[@]}") TIER_CMDS=("${cfg_tier_cmds[@]}")
else
[ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --"
[ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd")
fi
log "verify: $cfg"
PULL_ROOT=$(cd "$src" && pwd)
cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- "${user_cmd[@]}"
cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- ${user_cmd[@]+"${user_cmd[@]}"}
}
usage() {
@@ -939,13 +1516,17 @@ usage() {
usage: ubi-runner.sh <command> [args]
run [up opts] [--src DIR] [--dest PATH] [--setup FILE] [--env K=V]...
[--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] -- COMMAND
up + sync + setup + command + pull + destroy; exits with COMMAND's status
verify [run opts] [--trust-config] [-- COMMAND]
[--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] [--key-input S]... [--probe] -- COMMAND
up + sync + setup + command + pull + destroy; exits with COMMAND's status.
A rerun of the same command (same run key) reattaches to its live VM;
--probe prints UBI_ATTACH_INFO (and the tier plan) and exits
verify [run opts] [--trust-config] [--resume|--no-resume] [--rerun-failed] [-- COMMAND]
run the repository's committed verify (.gstack/ubicloud.json under --src);
its setup script and every pull destination stay inside --src (checked
again at pull time), and its pass names are forwarded only when granted
here (--pass NAME, or all: --trust-config)
here (--pass NAME, or all: --trust-config). With tiers, a pass checkpoint
under the same run key is reused (resume is the default)
attach NAME resume the run recorded for NAME on this machine, as a rerun would
up [-s SIZE] [-l LOCATION] [-n NAME] [-S STORAGE_GIB] [-b IMAGE]
create a VM and wait for SSH; prints its name
(STORAGE_GIB: one of $ALLOWED_STORAGE)
@@ -965,6 +1546,11 @@ usage: ubi-runner.sh <command> [args]
owner's TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS
cli ARGS... raw Ubicloud CLI passthrough (e.g. cli vm list)
signals: INT and TERM tear the VM down. HUP, once a remote phase is live, detaches
(UBI_DETACHED, exit 129; the VM runs on under its TTL). SIGKILL or a machine stop leaves
the run record and the VM; rerun the same command to reattach (UBI_ATTACH_BUSY while
another runner owns it; exit 125 UBI_ATTACH_LOST when the remote session left no status).
VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; up, run and verify need
UBI_OWNER (e.g. your thread code); run and verify end with a UBICLOUD: left=<n> line.
defaults: size=$DEFAULT_SIZE location=$DEFAULT_LOCATION (env UBI_SIZE, UBI_LOCATION)
@@ -976,7 +1562,8 @@ env: UBI_TTL_HOURS (${TTL_HOURS}h in the name, 1-999), UBI_REAP (0: up skips the
UBI_QUOTA_WAIT (${QUOTA_WAIT}s to wait for quota room instead of refusing),
UBI_GC_HOURS (unset/0: no sweep on up), UBI_CREATE_GRACE (${CREATE_GRACE}s),
UBI_DOWN_TIMEOUT (${DOWN_TIMEOUT}s), UBI_API_TIMEOUT (${API_TIMEOUT}s), UBI_VM_RECORD (scratch VM-id file),
UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640")
UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640"),
UBI_CHECKPOINT_DIR (tier checkpoints; $CHECKPOINT_DIR), UBI_CHECKPOINT_MAX_AGE (${CHECKPOINT_MAX_AGE}s a pass is reused)
EOF
}
@@ -989,6 +1576,7 @@ esac
case $cmd in
run) cmd_run "$@" ;;
verify) cmd_verify "$@" ;;
attach) cmd_attach "$@" ;;
up) trap 'trap "" INT TERM HUP QUIT; [ -z "$UP_NAME" ] || [ -n "$UP_READY" ] || cmd_down "$UP_NAME" || log "WARNING: failed to destroy $UP_NAME; run: $0 down $UP_NAME"' EXIT
trap 'exit 130' INT TERM HUP QUIT
cmd_up "$@" ;;
+20
View File
@@ -368,6 +368,26 @@ describe('CSO runtime staging gates', () => {
expect(() => imageBuildMatrix(versions)).toThrow('UNPINNED_TOOL_VERSION');
});
test.each(['docker.io/tool/sbom:1', 'docker.io/tool/sbom:1.24', 'docker.io/tool/sbom:1.24-slim', 'docker.io/tool/sbom:latest', 'docker.io/tool/sbom:stable'])('floating source tag %s cannot pin a reviewed family', source => {
const sbom = reviewedInputs(); sbom.sbomGenerator.source = source;
expect(() => imageBuildMatrix(sbom)).toThrow('FLOATING_SBOM_GENERATOR_SOURCE');
const base = reviewedInputs(); base.profiles[0].source = source.replace('docker.io/tool/sbom', 'docker.io/library/node');
expect(() => imageBuildMatrix(base)).toThrow('FLOATING_BASE_SOURCE');
});
test('PostgreSQL alone may name a two-part release, which is exact for it', () => {
const postgres = reviewedInputs(); postgres.profiles[4].source = 'docker.io/library/postgres:17-bookworm';
expect(() => imageBuildMatrix(postgres)).toThrow('FLOATING_BASE_SOURCE');
const python = reviewedInputs(); python.profiles[2].source = 'docker.io/library/python:3.13-slim-bookworm';
expect(() => imageBuildMatrix(python)).toThrow('FLOATING_BASE_SOURCE');
expect(imageBuildMatrix(reviewedInputs()).include.find(row => row.stack === 'postgresql')?.baseSource).toBe('docker.io/library/postgres:17.2');
});
test('the committed build inputs name release tags for every source', () => {
expect(buildInputs.sbomGenerator.source).toBe('docker.io/docker/scout-sbom-indexer:1.24.0');
expect(committedImageBuildMatrix().include).toHaveLength(10);
});
test('duplicate stacks and missing Python uv images fail precisely', () => {
const duplicate = reviewedInputs(); duplicate.profiles[1].stack = 'node';
expect(() => imageBuildMatrix(duplicate)).toThrow('INVALID_STACK');
+3
View File
@@ -98,6 +98,9 @@ const MODULE_SINKS = [
// repository permission (`gh api .../collaborators/<user>/permission`); the
// author login and repo slug leave the machine, sink 'contributor-mode'.
'lib/contributor-mode.ts',
// GSTA-156: gstack-pregate clear --run reads the repository's own Actions runs
// through gh and fetches a missing run head from origin by sha; sink 'run-evidence'.
'lib/run-evidence.ts',
// The Memorable bridge hook: gstack-owned code that hands each prompt to a
// vendor CLI. hosts/ has no curl/fetch for the scanner to see, so the
// receipt wiring is pinned here explicitly.
+1 -1
View File
@@ -9,7 +9,7 @@
"browser-skills/hackernews-frontpage": 371,
"canary": 7738,
"captain": 6551,
"capy/skill/gstack": 5136,
"capy/skill/gstack": 5312,
"careful": 1239,
"codex": 9890,
"context-restore": 6287,
+246
View File
@@ -0,0 +1,246 @@
/**
* gstack-pregate clear --run (GSTA-156): GitHub Actions runs clear the remote gate obligation, or a lane with a run
* profile, on the identical tree or a stamp-only predecessor, covering the gate's inventory; verify then checks the
* recorded receipt offline. The run API is a fake; trees come from the fixture repository's git.
*/
import { afterEach, describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { EXIT, type PregateFile } from '../lib/headless-artifacts';
import { buildContext } from '../lib/pregate/context';
import { clearDecision, clearObligation, runPregate, verifyPregate, type RunCheck } from '../lib/pregate/index';
import type { ApiRun, RunDeps, RunProfile } from '../lib/run-evidence';
import { BASE_PREGATE, cleanup, commit, makeFixtureRepo, readFile, runBin, type FixtureRepo } from './helpers/pregate-fixture';
const repos: FixtureRepo[] = [];
afterEach(() => { while (repos.length) cleanup(repos.pop()!); });
const LANE = 'pregate.json/remote_gate';
const NOW = new Date('2026-10-29T12:00:00Z');
const POLICY = { bump: 'patch', queue_mode: 'stamp-at-merge', changelog: 'CHANGELOG.md', stamp_files: ['VERSION', 'package.json', 'CHANGELOG.md'] };
const PROFILES: Record<string, RunProfile> = {
test: { workflow: 'test.yml', events: ['workflow_dispatch'], jobs: ['full-suite'], artifact: 'full-suite-tree' },
e2e: { workflow: 'e2e.yml', events: ['workflow_dispatch'], jobs: ['e2e-full-suite'], artifact: 'e2e-full-suite-tree' },
};
const GATE_RUNS = { profiles: ['test', 'e2e'], inventory: 'printf "test/a.test.ts\\ntest/b.test.ts\\n"' };
function fixture(extra: Record<string, unknown> = {}): FixtureRepo {
const repo = makeFixtureRepo({
pregate: { ...BASE_PREGATE, ...extra, remote_gate: 'node scripts/remote-gate.js', dependencies: { ...BASE_PREGATE.dependencies, 'CHANGELOG.md': [], VERSION: [] } },
baseFiles: { '.gstack/ship-policy.json': JSON.stringify(POLICY, null, 2) + '\n', VERSION: '1.0.0\n', 'CHANGELOG.md': '# Changelog\n\n## [Unreleased]\n\n- add\n' },
});
repos.push(repo);
repo.git('checkout', '-q', 'main');
commit(repo, { 'scripts/remote-gate.js': "console.log('PREGATE_REMOTE: tree=' + process.env.PREGATE_TREE + ' tests=3 failed=0');\n" }, 'remote gate');
repo.git('push', '-q', 'origin', 'main');
repo.git('checkout', '-q', 'feature');
repo.git('merge', '-q', '--ff-only', 'main');
commit(repo, { 'lib/util.js': 'exports.add = (a, b) => a + b + 0;\n' }, 'the real change');
return repo;
}
function stamp(repo: FixtureRepo): string {
const pkg = JSON.parse(readFile(repo, 'package.json'));
pkg.version = '1.0.1';
return commit(repo, { VERSION: '1.0.1\n', 'package.json': JSON.stringify(pkg, null, 2) + '\n', 'CHANGELOG.md': '# Changelog\n\n## [1.0.1] - 2026-10-29\n\n- add\n' }, 'v1.0.1 stamp');
}
async function pregateAt(repo: FixtureRepo): Promise<PregateFile> {
const loaded = buildContext({ repoRoot: repo.work, base: 'main', policyFrom: 'base', allowRepoCommands: true, env: { ...process.env, ...repo.env }, progress: () => {}, cores: 4, testBackend: 'local' });
if (!('ctx' in loaded)) throw new Error('fixture context refused');
const file = await runPregate(loaded.ctx, { stage: 'tests' });
expect(file.requires_remote).toEqual([expect.objectContaining({ lane: LANE, receipt: null })]);
return file;
}
/** A run of `profile` on `head` with a tree.json listing `files`, as GitHub would report it. */
interface FakeRun { id: number; profile: string; head: string; files: string[]; conclusion?: string; updated?: string }
function check(repo: FixtureRepo, runs: FakeRun[], over: Partial<RunCheck> = {}): RunCheck {
const byId = new Map(runs.map(r => [r.id, r]));
const api = (r: FakeRun): ApiRun => ({
id: r.id, run_attempt: 1, status: 'completed', conclusion: r.conclusion ?? 'success', event: 'workflow_dispatch', head_sha: r.head,
path: `.github/workflows/${PROFILES[r.profile]!.workflow}`, created_at: '2026-10-29T10:00:00Z', updated_at: r.updated ?? '2026-10-29T10:30:00Z',
repository: { full_name: 'fx/fx' }, head_repository: { full_name: 'fx/fx' },
});
const deps: RunDeps = {
run: id => { const r = byId.get(id); if (!r) throw new Error('HTTP 404'); return api(r); },
runsForHead: (wf, sha) => runs.filter(r => r.head === sha && PROFILES[r.profile]!.workflow === wf).map(api),
jobs: id => [{ name: PROFILES[byId.get(id)!.profile]!.jobs[0]!, conclusion: 'success' }],
artifact: id => { const r = byId.get(id)!; return { run_id: id, run_attempt: 1, event: 'workflow_dispatch', sha: r.head, tree: repo.git('rev-parse', `${r.head}^{tree}`), files: r.files, tests: r.files.length * 5 }; },
commitTree: sha => repo.git('rev-parse', `${sha}^{tree}`),
};
return { repo: 'fx/fx', deps, profiles: PROFILES, gateRuns: GATE_RUNS, now: NOW, inventory: () => ['test/a.test.ts', 'test/b.test.ts'], ...over };
}
describe.skipIf(process.platform === 'win32')('clear --run: Actions runs as remote gate evidence', () => {
test('one verified run per profile on the identical tree clears the gate, and verify accepts the receipt offline', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const d = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 201, profile: 'e2e', head, files: ['test/b.test.ts'] }, { id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/util.test.js'] }]) }, repo.work);
if ('refused' in d) throw new Error(d.message);
expect(d.receipt).toBe(`remote-gate tree=${file.tree.slice(0, 12)} tests=15 runs=200/1,201/1 profiles=test,e2e heads=${head.slice(0, 12)} verified=2026-10-29T12:00:00.000Z`);
expect(d.substitution).toBeUndefined();
const cleared = clearObligation(file, LANE, d.receipt, 'test');
expect(verifyPregate(repo.work, cleared, { tree: file.tree } as any)).toMatchObject({ ok: true });
const out = path.join(repo.work, '.gstack/tmp/pregate.json');
fs.mkdirSync(path.dirname(out), { recursive: true });
fs.writeFileSync(out, JSON.stringify(cleared));
const bound = runBin(repo, 'gstack-ship-receipt', ['write', '--pr', '7', '--base', 'main', '--version', '1.0.0', '--session-kind', 'interactive', '--pregate', out, '--json']);
expect([bound.status, bound.err]).toEqual([EXIT.ok, expect.any(String)]);
expect(bound.out).toContain(d.receipt);
}, 120_000);
test('runs on a stamp-only predecessor are accepted for the stamped head and recorded as a substitution', async () => {
const repo = fixture();
const gated = repo.git('rev-parse', 'HEAD');
stamp(repo);
const file = await pregateAt(repo);
const d = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 200, profile: 'test', head: gated, files: ['test/a.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'e2e', head: gated, files: ['test/b.test.ts'] }]) }, repo.work);
if ('refused' in d) throw new Error(d.message);
expect(d.receipt).toContain(`accepted-for=${file.tree.slice(0, 12)} stamp-only=3 runs=200/1,201/1`);
expect(d.substitution?.accepted_for).toBe(file.tree);
expect(verifyPregate(repo.work, clearObligation(file, LANE, d.receipt, 'test', d.substitution), { tree: file.tree } as any)).toMatchObject({ ok: true });
}, 120_000);
test('refusals: a non-stamp difference, an uncovered inventory file, a failed run, a missing profile, no remote_gate_runs', async () => {
const repo = fixture();
const older = repo.git('rev-parse', 'HEAD~1');
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const both = (h: string, files: string[][] = [['test/a.test.ts', 'test/util.test.js'], ['test/b.test.ts']], conclusion?: string): FakeRun[] => [{ id: 200, profile: 'test', head: h, files: files[0]!, conclusion }, { id: 201, profile: 'e2e', head: h, files: files[1]! }];
const msg = (d: ReturnType<typeof clearDecision>) => ('refused' in d ? `${d.refused}: ${d.message}` : `cleared ${d.receipt}`);
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(older)) }, repo.work))).toMatch(/^PREGATE_RECEIPT_INVALID: .*lib\/util\.js/);
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, [['test/a.test.ts', 'test/util.test.js'], ['test/c.test.ts']])) }, repo.work))).toContain('1 of the 3 test file(s) pregate.json/remote_gate must cover appear in no run\'s tree.json (test/b.test.ts)');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, [['test/a.test.ts', 'test/b.test.ts'], ['test/b.test.ts']])) }, repo.work))).toContain('(test/util.test.js)');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, [['test/a.test.ts', 'test/util.test.js'], []])) }, repo.work))).toContain('reason=no_tests — run 201 (e2e) records no tests');
expect(msg(clearDecision(file, LANE, { runs: ['200', 'https://github.com/fx/fx/actions/runs/200'], evidence: check(repo, both(head)) }, repo.work))).toBe('usage: pregate.json/remote_gate needs a different run for each profile; got 200, https://github.com/fx/fx/actions/runs/200');
expect(msg(clearDecision({ ...file, base: 'origin/release@0123456789ab' }, LANE, { runs: ['200', '201'], evidence: check(repo, both(head)) }, repo.work))).toMatch(/^PREGATE_RECEIPT_INVALID: REUSE: refused reason=base_mismatch/);
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, undefined, 'failure')) }, repo.work))).toContain('reason=no_verified_run');
expect(msg(clearDecision(file, LANE, { runs: ['200'], evidence: check(repo, both(head)) }, repo.work))).toBe('usage: pregate.json/remote_gate clears with exactly 2 run(s), one per profile (test, e2e); got 1');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head), { gateRuns: null }) }, repo.work))).toContain('only when the base branch\'s .gstack/pregate.json declares remote_gate_runs');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head), { inventory: () => ({ error: 'boom' }) }) }, repo.work))).toContain('inventory command failed: boom');
}, 120_000);
test('runs that tested different trees, or that record no tests, never clear the gate', async () => {
const repo = fixture();
const older = repo.git('rev-parse', 'HEAD~1');
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const split = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/b.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'e2e', head: older, files: ['test/b.test.ts'] }]) }, repo.work);
expect('refused' in split && split.message).toContain('the runs tested different trees');
const empty = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/util.test.js'] }, { id: 201, profile: 'e2e', head, files: ['test/b.test.ts'] }], { inventory: () => [] }) }, repo.work);
expect('refused' in empty && empty.message).toContain('inventory command listed no test files');
}, 120_000);
test('a run valid for two profiles does not starve the other: the profile-to-run assignment is searched, not taken greedily', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const profiles = { ...PROFILES, wide: PROFILES.test!, narrow: { ...PROFILES.test!, max_age_hours: 1 } };
const runs: FakeRun[] = [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'test', head, files: ['test/b.test.ts'], updated: '2026-10-29T11:30:00Z' }];
const d = clearDecision(file, LANE, { runs: ['201', '200'], evidence: check(repo, runs, { profiles, gateRuns: { profiles: ['wide', 'narrow'], inventory: 'true' } }) }, repo.work);
if ('refused' in d) throw new Error(d.message);
expect(d.receipt).toContain('runs=200/1,201/1 profiles=wide,narrow ');
}, 120_000);
test('a lane with its own profile clears from --run-url through the verifier; a lane without one keeps the plain run-URL record, and so does a repository whose run profiles cannot be read (a non-GitHub origin), with a warning', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const base = await pregateAt(repo);
const PLATFORM = 'test.yml/full-suite';
const file: PregateFile = { ...base, requires_remote: [...base.requires_remote!, { lane: PLATFORM, platform: 'macos', workflow: 'test.yml', job: 'full-suite', tests: ['test/a.test.ts'], receipt: null }] };
const url = 'https://github.com/fx/fx/actions/runs/200';
const profiled = clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts'] }], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
if ('refused' in profiled) throw new Error(profiled.message);
expect(profiled).toMatchObject({ attested: false });
expect(profiled.receipt).toStartWith(`run tree=${file.tree.slice(0, 12)} tests=5 runs=200/1 profiles=${PLATFORM} `);
const failed = clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts'], conclusion: 'failure' }], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
expect('refused' in failed && failed.refused).toBe('PREGATE_RECEIPT_INVALID');
expect(clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, []) }, repo.work)).toEqual({ receipt: url, attested: false });
const bundled = clearDecision(file, PLATFORM, { bundleLane: 'macos', evidence: check(repo, [], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
expect('refused' in bundled && `${bundled.refused}: ${bundled.message}`).toBe(`usage: ${PLATFORM} has a run profile on the base branch; clear it with --run <id|url>, which verifies the run`);
expect(clearDecision(file, PLATFORM, { bundleLane: 'macos', evidence: check(repo, []) }, repo.work)).toEqual({ receipt: 'bundle-lane:macos', attested: false });
const unprofiled = clearDecision(file, PLATFORM, { runs: ['200'], evidence: check(repo, []) }, repo.work);
expect('refused' in unprofiled && `${unprofiled.refused}: ${unprofiled.message}`).toContain(`usage: --run clears ${PLATFORM} only when the base branch's .gstack/pregate.json declares a run_profiles entry named`);
const gateFromUrl = clearDecision(file, LANE, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/b.test.ts', 'test/util.test.js'] }]) }, repo.work);
expect('refused' in gateFromUrl && `${gateFromUrl.refused}: ${gateFromUrl.message}`).toBe('usage: pregate.json/remote_gate clears with exactly 2 run(s), one per profile (test, e2e); got 1');
const oneProfile = clearDecision(file, LANE, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/b.test.ts', 'test/util.test.js'] }], { gateRuns: { profiles: ['test'], inventory: 'true' } }) }, repo.work);
if ('refused' in oneProfile) throw new Error(oneProfile.message);
expect(oneProfile.receipt).toStartWith(`remote-gate tree=${file.tree.slice(0, 12)} tests=15 runs=200/1 profiles=test `);
const uncovered = clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/smoke.test.ts'] }], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
expect('refused' in uncovered && uncovered.message).toContain(`reason=tests_uncovered — 1 of the 1 test file(s) ${PLATFORM} must cover appear in no run's tree.json (test/a.test.ts)`);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', PLATFORM, '--run-url', url]);
expect([r.status, r.err]).toEqual([EXIT.ok, expect.stringContaining('origin is not a github.com repository')]);
expect(r.err).toContain(`${PLATFORM} was recorded without run verification`);
const viaRun = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run', '200', '--run', '201', '--allow-repo-commands']);
expect([viaRun.status, viaRun.err]).toEqual([EXIT.fail, expect.stringContaining('origin is not a github.com repository')]);
}, 120_000);
test('a lane the base profiles never falls back to an unverified record when the GitHub origin cannot be read', async () => {
const repo = fixture({ run_profiles: PROFILES, remote_gate_runs: GATE_RUNS });
const file = await pregateAt(repo);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run-url', 'https://github.com/fx/fx/actions/runs/200']);
expect([r.status, r.err]).toEqual([EXIT.fail, expect.stringContaining(`${LANE} may have a run profile on the base branch, so it clears only with a verified --run`)]);
expect(JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')).requires_remote[0].receipt).toBeNull();
}, 120_000);
test('an unreadable base config refuses --run-url even on a lane with no profile, since the base may profile it', async () => {
const repo = fixture();
const base = await pregateAt(repo);
const PLATFORM = 'test.yml/full-suite';
const file: PregateFile = { ...base, requires_remote: [...base.requires_remote!, { lane: PLATFORM, platform: 'macos', workflow: 'test.yml', job: 'full-suite', tests: ['test/a.test.ts'], receipt: null }] };
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
repo.git('update-ref', '-d', 'refs/remotes/origin/main');
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', PLATFORM, '--run-url', 'https://github.com/fx/fx/actions/runs/200']);
expect([r.status, r.err]).toEqual([EXIT.fail, expect.stringContaining(`${PLATFORM} may have a run profile on the base branch, so it clears only with a verified --run`)]);
expect(JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')).requires_remote.find((o: { lane: string }) => o.lane === PLATFORM).receipt).toBeNull();
}, 120_000);
test('offline verify refuses a run receipt recorded for another tree', async () => {
const repo = fixture();
const file = await pregateAt(repo);
const forged = clearObligation(file, LANE, `remote-gate tree=${'0'.repeat(12)} tests=10 runs=200/1 profiles=test heads=${'1'.repeat(12)} verified=2026-10-29T12:00:00.000Z`, 'test');
expect(verifyPregate(repo.work, forged, { tree: file.tree } as any)).toMatchObject({ ok: false, code: 'PREGATE_REMOTE_UNCLEARED' });
}, 120_000);
test('every run is read again once the clear is assembled: a run that turns red while another profile or the inventory is read refuses', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const base = check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'e2e', head, files: ['test/b.test.ts'] }]);
let flipped = false;
const evidence: RunCheck = { ...base, deps: { ...base.deps, run: id => flipped && id === 200 ? { ...base.deps.run(id), run_attempt: 2, conclusion: 'failure' } : base.deps.run(id) }, inventory: () => { flipped = true; return ['test/a.test.ts', 'test/b.test.ts']; } };
const d = clearDecision(file, LANE, { runs: ['200', '201'], evidence }, repo.work);
expect('refused' in d && d.message).toContain('REUSE: refused reason=run_changed — run 200 moved to attempt 2 (failure)');
}, 120_000);
test('origin is read before url.insteadOf, so a Capy git.capy.ai proxy still names the GitHub repository', async () => {
const repo = fixture();
const file = await pregateAt(repo);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
repo.git('remote', 'set-url', 'origin', 'https://github.com/fx/fx.git');
repo.git('config', 'url.https://git.capy.ai/.insteadOf', 'https://github.com/');
expect(repo.git('remote', 'get-url', 'origin')).toBe('https://git.capy.ai/fx/fx.git');
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run', '200', '--run', '201', '--allow-repo-commands']);
expect(r.err).not.toContain('origin is not a github.com repository');
expect([r.status, r.err]).toEqual([EXIT.usage, expect.stringContaining('declares remote_gate_runs')]);
}, 120_000);
test('the bin refuses --run when the base declares no run profiles, and keeps the --receipt route unchanged', async () => {
const repo = fixture();
const file = await pregateAt(repo);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
repo.git('remote', 'set-url', 'origin', 'https://github.com/fx/fx.git');
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run', '200', '--run', '201', '--allow-repo-commands']);
expect(r.status).toBe(EXIT.usage);
expect(r.err).toContain('--run clears pregate.json/remote_gate only when the base branch\'s .gstack/pregate.json declares remote_gate_runs');
}, 120_000);
});
+5 -1
View File
@@ -157,12 +157,16 @@ describe('gstack-pregate with a user-owned policy', () => {
expect(invalid.err).toContain(`fix: edit ${userPolicy} directly`);
expect(invalid.err).not.toContain('.gstack/pregate.json has');
for (const [key, value] of [['sealed', { paths: ['evals/**'] }], ['price_file', 'evals/calls.json']] as const) {
const profile = { workflow: 'test.yml', events: ['push'], jobs: ['full-suite'], artifact: 'full-suite-tree' };
for (const [key, value] of [['sealed', { paths: ['evals/**'] }], ['price_file', 'evals/calls.json'], ['run_profiles', { test: profile }], ['remote_gate_runs', { profiles: ['test'], inventory: 'true' }]] as const) {
writePolicy(userPolicy, { [key]: value });
const r = run();
expect(r.status).toBe(EXIT.refused);
expect(r.err).toContain(`covers gstack-pregate run only; ${key}`);
}
// An empty run_profiles is the documented default, not a policy, so it is accepted.
writePolicy(userPolicy, { run_profiles: {}, local: LOCAL_ALL });
expect(run().err).not.toContain('PREGATE_CONFIG_INVALID');
// A null price_file is the documented default, not a policy, so it is accepted.
writePolicy(userPolicy, { price_file: null, local: LOCAL_ALL });
expect(run().err).not.toContain('PREGATE_CONFIG_INVALID');
+101 -5
View File
@@ -3,7 +3,7 @@ import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { BriefError, promptHash, renderBrief, sectionTexts } from '../lib/review-brief';
import { BriefError, checkReviewedTree, promptHash, renderBrief, sectionTexts, workspaceParagraph } from '../lib/review-brief';
import { collect, judge, parseFindings, type MergedLine } from '../lib/review-ledger';
const ROOT = path.resolve(import.meta.dir, '..');
@@ -11,7 +11,19 @@ const BIN = path.join(ROOT, 'bin', 'gstack-review-dispatch');
const dirs: string[] = [];
afterEach(() => { while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); });
const tmp = () => { const d = fs.mkdtempSync(path.join(os.tmpdir(), 'review-dispatch-')); dirs.push(d); return d; };
const run = (args: string[], cwd = ROOT) => spawnSync(BIN, args, { cwd, encoding: 'utf8', timeout: 60_000 });
/** A reader that follows its brief's workspace paragraph starts its reply with the REVIEWED_TREE line of that checkout (GST-482). */
function treeLineOf(cwd: string): string {
const git = (a: string[]) => spawnSync('git', ['-C', cwd, ...a], { encoding: 'utf8', timeout: 30_000 }).stdout.trim();
return `REVIEWED_TREE: ${git(['rev-parse', 'HEAD^{tree}'])} ${git(['rev-parse', '--show-toplevel'])}\n`;
}
function followBrief(output: string, cwd: string) {
const text = fs.readFileSync(output, 'utf8');
if (!text.includes('REVIEWED_TREE:')) fs.writeFileSync(output, treeLineOf(cwd) + text);
}
const run = (args: string[], cwd = ROOT) => {
if (args[0] === 'collect' && !args.includes('--unavailable') && args.includes('--output')) followBrief(args[args.indexOf('--output') + 1]!, cwd);
return spawnSync(BIN, args, { cwd, encoding: 'utf8', timeout: 60_000 });
};
const SHA = 'a'.repeat(40);
const section = (rel: string) => fs.readFileSync(path.join(ROOT, rel), 'utf8');
const field = (out: string, k: string) => new RegExp(`${k}=(\\S+)`).exec(out)?.[1] ?? '';
@@ -338,10 +350,10 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
const b = tmp();
const batched = field(run(['render', '--role', 'specialist-batched', '--checklists', 'testing,security,state-machines', '--out', b, '--diff-base', SHA]).stdout, 'file');
const cases = [
{ reader: 'specialist-batched', dir: b, brief: batched, output: refusal, args: ['--role', 'specialist-batched'] },
{ reader: 'specialist-batched', dir: b, brief: batched, output: treeLineOf(ROOT) + refusal, args: ['--role', 'specialist-batched'] },
...['', 'I could not access the diff: git diff failed with "bad object", so there is nothing here to review.\n'].map(output => {
const p = pass();
return { reader: 'specialist/testing', dir: p.out, brief: p.file, output, args: ['--role', 'specialist', '--name', 'testing'] };
return { reader: 'specialist/testing', dir: p.out, brief: p.file, output: treeLineOf(ROOT) + output, args: ['--role', 'specialist', '--name', 'testing'] };
}),
];
for (const c of cases) {
@@ -365,7 +377,7 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
// the non-review rule; fails_when=the rule makes a real NO FINDINGS specialist or a findings reply dirty or exit 1;
// why_new=the non-review rule above could over-reach; seam=collect and verify; no_claim=other roles (the matrix below covers them)
test('a NO FINDINGS reply and a findings reply still collect with exit 0 and verify clean', () => {
for (const [output, findings] of [['NO FINDINGS\n', 0], ['{"severity":"CRITICAL","path":"a.ts","line":3,"category":"logic","summary":"off by one","specialist":"testing"}\n', 1]] as const) {
for (const [output, findings] of [[treeLineOf(ROOT) + 'NO FINDINGS\n', 0], [treeLineOf(ROOT) + '{"severity":"CRITICAL","path":"a.ts","line":3,"category":"logic","summary":"off by one","specialist":"testing"}\n', 1]] as const) {
const p = pass();
run(['sent', '--ledger', p.ledger, '--role', 'specialist', '--name', 'testing', '--sent', p.file]);
const col = run(['collect', '--ledger', p.ledger, '--role', 'specialist', '--name', 'testing', '--output', p.w('reply.out', output), '--merged', p.merged]);
@@ -476,3 +488,87 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
expect([v.status, v.stdout.trim()]).toEqual([3, "VERIFY: no brief records (this run's ledger kept no prompts; dispatched-vs-rendered cannot be checked)"]);
});
});
describe('the workspace paragraph (GST-482)', () => {
// Value: protects=GSTA-167's no-loss pin: every role's brief is renderBrief's bytes unchanged, then only the workspace paragraph naming the checkout; fails_when=the paragraph is inserted, a section byte moves, or the CLI names another folder than the checkout it ran in;
// why_new=GST-482 appends a paragraph to every brief; seam=gstack-review-dispatch render; no_claim=that the reader obeys it
test('every role renders main\'s brief byte for byte, then the paragraph naming the checkout the CLI ran in', () => {
const t = sectionTexts(ROOT);
const top = fs.realpathSync(spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: ROOT, encoding: 'utf8', timeout: 30_000 }).stdout.trim());
const w = (n: string, s: string) => { const f = path.join(tmp(), n); fs.writeFileSync(f, s); return f; };
const cases: Array<[string, string[], Record<string, string>]> = [
['specialist', ['--name', 'testing'], { name: 'testing', checklist: path.join(ROOT, 'review/specialists/testing.md') }],
['specialist', ['--name', 'security'], { name: 'security', checklist: path.join(ROOT, 'review/specialists/security.md') }],
['specialist-batched', ['--checklists', 'testing,security'], { checklists: ['testing', 'security'].map(c => path.join(ROOT, 'review/specialists', c + '.md')).join(',') }],
['red-team', ['--findings-file', w('f.jsonl', 'x\n'), '--specialists', '2'], { checklist: path.join(ROOT, 'review/specialists/red-team.md'), findings: 'x\n', specialists: '2' }],
['adversarial-native', [], {}],
['fix-check', ['--fix-list', w('fl', 'a'), '--hunks', w('h', 'b'), '--tests', w('t', 'c')], { fix_list: 'a', hunks: 'b', tests: 'c' }],
];
for (const [role, args, slots] of cases) {
const out = tmp();
const r = run(['render', '--role', role, '--out', out, ...(role === 'fix-check' ? [] : ['--diff-base', SHA]), ...args]);
expect(r.status, role + ': ' + r.stderr).toBe(0);
const main = renderBrief(role as Parameters<typeof renderBrief>[0], { diff_base: role === 'fix-check' ? undefined : SHA, ...slots }, t);
expect(fs.readFileSync(field(r.stdout, 'file'), 'utf8'), role).toBe(main + '\n\n' + workspaceParagraph(top));
}
expect(workspaceParagraph(top)).toContain(`begin each command with \`cd '${top}' && \``);
expect(workspaceParagraph(top)).toContain(`absolute path under ${top}`);
});
test('a list-item or bold REVIEWED_TREE line counts; a ledger rendered before the paragraph existed is collected as before', () => {
const out = tmp();
const p = { ledger: path.join(out, 'review-ledger.jsonl'), merged: path.join(out, 'merged.jsonl'), file: field(run(['render', '--role', 'specialist', '--name', 'testing', '--out', out, '--diff-base', SHA]).stdout, 'file'),
w: (n: string, t: string) => { const f = path.join(out, n); fs.writeFileSync(f, t); return f; } };
run(['sent', '--ledger', p.ledger, '--role', 'specialist', '--name', 'testing', '--sent', p.file]);
const line = treeLineOf(ROOT).trim();
const col = run(['collect', '--ledger', p.ledger, '--role', 'specialist', '--name', 'testing', '--output', p.w('reply.out', '- **' + line + '**\nNO FINDINGS\n'), '--merged', p.merged]);
expect(col.stdout).toStartWith('COLLECT: reader=specialist/testing findings=0 ');
const legacy = tmp();
const ledger = path.join(legacy, 'review-ledger.jsonl');
const brief = path.join(legacy, 'b.md');
fs.writeFileSync(brief, 'old brief');
fs.writeFileSync(ledger, JSON.stringify({ event: 'rendered', reader: 'specialist/testing', role: 'specialist', name: 'testing', file: brief, sha256: promptHash('old brief'), bytes: 9 }) + '\n');
run(['sent', '--ledger', ledger, '--role', 'specialist', '--name', 'testing', '--sent', brief]);
const reply = path.join(legacy, 'r.out');
fs.writeFileSync(reply, 'NO FINDINGS\n');
const old = spawnSync(BIN, ['collect', '--ledger', ledger, '--role', 'specialist', '--name', 'testing', '--output', reply, '--merged', path.join(legacy, 'm.jsonl')], { cwd: ROOT, encoding: 'utf8', timeout: 60_000 });
expect(old.stdout).toStartWith('COLLECT: reader=specialist/testing findings=0 ');
});
});
describe('reading the REVIEWED_TREE line (GST-482 review fixes)', () => {
const top = fs.realpathSync(spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: ROOT, encoding: 'utf8', timeout: 30_000 }).stdout.trim());
const tree = spawnSync('git', ['rev-parse', 'HEAD^{tree}'], { cwd: ROOT, encoding: 'utf8', timeout: 30_000 }).stdout.trim();
const expected = `${tree} ${top}`;
test('Markdown around the line, label, sha or whole path is read; the path keeps its own characters; the first line wins', () => {
for (const line of [`REVIEWED_TREE: ${tree} ${top}`, `**REVIEWED_TREE:** ${tree} ${top}`, `REVIEWED_TREE: \`${tree}\` \`${top}\``, `## REVIEWED_TREE: ${tree} ${top}`, `- **REVIEWED_TREE: ${tree} ${top}**`])
expect(checkReviewedTree(line + '\nNO FINDINGS', expected).ok, line).toBe(true);
expect(checkReviewedTree(`REVIEWED_TREE: ${tree} ${top}*`, expected).ok).toBe(false);
expect(checkReviewedTree(`REVIEWED_TREE: ${tree} ${top}\n\`\`\`\nREVIEWED_TREE: ${'b'.repeat(40)} /elsewhere\n\`\`\``, expected).ok).toBe(true);
expect(checkReviewedTree(`REVIEWED_TREE: ${'b'.repeat(40)} /elsewhere\nREVIEWED_TREE: ${tree} ${top}`, expected)).toMatchObject({ ok: false, reported: `${'b'.repeat(40)} /elsewhere` });
});
test('an outside voice with findings and no tree line is refused, never turned into unavailable coverage that drops them', () => {
const out = tmp();
const ledger = path.join(out, 'review-ledger.jsonl');
const merged = path.join(out, 'merged.jsonl');
const head = spawnSync('git', ['rev-parse', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 30_000 }).stdout.trim();
const file = field(run(['render', '--role', 'outside-voice', '--out', out, '--diff-base', head]).stdout, 'file');
run(['sent', '--ledger', ledger, '--role', 'outside-voice', '--sent', file]);
const reply = path.join(out, 'r.out');
fs.writeFileSync(reply, '1. **CRITICAL**: auth bypass in a.ts:3\nBlocks: yes\n\nRecommendation: request changes\n');
const c = spawnSync(BIN, ['collect', '--ledger', ledger, '--role', 'outside-voice', '--output', reply, '--merged', merged], { cwd: ROOT, encoding: 'utf8', timeout: 60_000 });
expect([c.status, c.stdout.startsWith('COLLECT_WRONG_TREE: reader=outside-voice reported="none"')]).toEqual([1, true]);
expect(run(['verify', '--ledger', ledger]).stdout).not.toContain('VERIFY: clean');
fs.writeFileSync(reply, 'Codex timed out before it reviewed anything.\n');
const u = spawnSync(BIN, ['collect', '--ledger', ledger, '--role', 'outside-voice', '--output', reply, '--merged', merged], { cwd: ROOT, encoding: 'utf8', timeout: 60_000 });
expect([u.status, u.stdout.startsWith('COLLECT_UNAVAILABLE: reader=outside-voice')]).toEqual([0, true]);
});
test('render outside a readable checkout refuses instead of recording a fingerprint no reply can match', () => {
const dir = tmp();
const r = run(['render', '--role', 'adversarial-native', '--out', path.join(dir, 'o'), '--diff-base', SHA], dir);
expect(r.status).not.toBe(0);
expect(r.stderr).toContain('BRIEF_RENDER_FAILED: cannot read HEAD^{tree} and the toplevel of');
});
});
+134 -6
View File
@@ -7,7 +7,7 @@ import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { promptHash, sectionTexts } from '../lib/review-brief';
import { promptHash, sectionTexts, withWorkspace } from '../lib/review-brief';
import { CHUNK_CHARS, buildManifest, chunkCount, chunkOf, markReader, renderStage, specialistFindings, type Manifest } from '../lib/review-pass';
import type { Dispatch } from '../lib/review-depth';
@@ -129,19 +129,28 @@ function repo(): { dir: string; env: NodeJS.ProcessEnv } {
interface Sub { title: string; prompt: string; status: string; polls: number; model?: unknown; summary?: string; late?: boolean }
/** A fake Capy host: the shell is real (the repository's bins), subagents are scripted. */
const cleanReply = (title: string) => (title.endsWith('outside-voice') ? 'Recommendation: ship as-is because no finding survived' : 'NO FINDINGS');
function host(dir: string, env: NodeJS.ProcessEnv, o: { output?: (title: string) => string; fail?: (title: string) => boolean; reject?: (title: string) => boolean; createThrows?: (title: string) => 'before' | 'after' | undefined; pollsToFinish?: number; lateSummary?: boolean } = {}) {
/** The REVIEWED_TREE line a reader prints from dir (its brief's checkout when it follows the workspace paragraph); '' outside a checkout. */
function treeLine(dir: string): string {
if (!dir) return '';
const git = (a: string[]) => spawnSync('git', ['-C', dir, ...a], { encoding: 'utf8', timeout: 30_000 });
const tree = git(['rev-parse', 'HEAD^{tree}']), top = git(['rev-parse', '--show-toplevel']);
return tree.status === 0 && top.status === 0 ? `REVIEWED_TREE: ${tree.stdout.trim()} ${top.stdout.trim()}\n` : '';
}
const briefDir = (prompt: string) => /Workspace: the checkout under review is (.+?)\. Your shell and file tools/.exec(prompt)?.[1] ?? '';
function host(dir: string, env: NodeJS.ProcessEnv, o: { output?: (title: string) => string; reportFrom?: (title: string) => string; fail?: (title: string) => boolean; reject?: (title: string) => boolean; createThrows?: (title: string) => 'before' | 'after' | undefined; pollsToFinish?: number; lateSummary?: boolean } = {}) {
let clock = 0;
const subs = new Map<string, Sub>();
const creates: Sub[] = [];
const starts: number[][] = [];
const reply = (s: Sub) => treeLine(o.reportFrom?.(s.title) ?? briefDir(s.prompt)) + (o.output?.(s.title) ?? cleanReply(s.title));
const advance = () => {
for (const s of subs.values()) {
if (s.late && !s.summary) { s.summary = o.output?.(s.title) ?? cleanReply(s.title); continue; }
if (s.late && !s.summary) { s.summary = reply(s); continue; }
if (s.status !== 'working') continue;
if (++s.polls < (o.pollsToFinish ?? 1)) continue;
if (o.fail?.(s.title)) s.status = 'failed';
else if (o.lateSummary && !s.late) { s.status = 'done'; s.late = true; }
else { s.status = 'done'; s.summary = o.output?.(s.title) ?? cleanReply(s.title); }
else { s.status = 'done'; s.summary = reply(s); }
}
};
const tools = {
@@ -320,7 +329,7 @@ describe('Capy pass program (fake host, real bins)', () => {
const h = host(dir, env, { output: t => (t.endsWith('audit/coverage') ? 'Coverage: 80% (one test added)' : 'NO FINDINGS') });
const res = await runPass(h, { base: 'main', out, pass: 'x8p1', planFlags: ['--depth-floor', 'full'], extras: path.join(out, 'extras.json') });
expect(h.creates[0]!.title).toBe('ship x8p1 audit/coverage');
expect(h.creates[0]!.prompt).toBe('Run Step 7 as written.\n');
expect(h.creates[0]!.prompt).toBe(withWorkspace('Run Step 7 as written.\n', fs.realpathSync(dir)));
expect(res).toContain('Coverage: 80% (one test added)');
expect(fs.readFileSync(path.join(out, 'merged.jsonl'), 'utf8')).not.toContain('Coverage');
}, 240_000);
@@ -422,7 +431,7 @@ describe('Capy pass program (fake host, real bins)', () => {
extras: [{ name: 'qa', prompt_file: path.join(out, 'qa.prompt.md'), model: { model_id: 'anthropic/claude-haiku-5-5', reasoning_mode: 'low' } }] });
expect(res).not.toContain('STOP:');
expect(res).toContain('NEXT: judge (Phase 4)');
expect(h.creates.find(c => c.title === 'ship x13p1 audit/qa')!.prompt).toBe('QA brief');
expect(h.creates.find(c => c.title === 'ship x13p1 audit/qa')!.prompt).toBe(withWorkspace('QA brief', fs.realpathSync(dir)));
}, 180_000);
test('a plan with no reviewer pass dispatches nothing and passes NEXT through', async () => {
@@ -460,3 +469,122 @@ describe('the Capy skill loader', () => {
}, 180_000);
});
describe('readers on a shared machine review the checkout the pass ships (GST-482)', () => {
/** The primary checkout on feat, and a worktree of it on another branch with one more commit: the GBRA-218 layout. */
function worktree() {
const { dir, env } = repo();
const wt = path.join(tmp('rp-wt-'), 'wt');
const sh = (c: string, cwd = dir) => { const r = spawnSync('bash', ['-c', c], { cwd, env, encoding: 'utf8', timeout: 60_000 }); if (r.status !== 0) throw new Error(c + ': ' + r.stderr); return r.stdout.trim(); };
sh(`git worktree add -q -b allow ${JSON.stringify(wt)} feat`);
sh('for i in $(seq 1 20); do echo "export const w$i = $i;" >> src/a.ts; done && git commit -qam allow', wt);
return { dir, env, wt: fs.realpathSync(wt) };
}
// Value: protects=a reader that never left the machine's default checkout cannot hand /ship a review of another tree; fails_when=collect merges a reply whose REVIEWED_TREE names the primary checkout while the pass ships a worktree, or a reply with no such line;
// why_new=GBRA-218: pass 1's specialist reviewed /workspace/gbrain while /ship shipped ~/.capy/work/wt-allow; seam=gstack-review-dispatch collect; no_claim=a reader that fingerprints the right checkout but reads another
test('a reader that reviews the primary checkout instead of the worktree is refused and nothing it said is merged', async () => {
const { dir, env, wt } = worktree();
const out = path.join(wt, '.state/run/p1');
const h = host(dir, env, { reportFrom: () => dir, output: t => (t.includes('specialist') ? '[CRITICAL] (confidence: 9/10) src/other.ts:3 a file outside this PR' : cleanReply(t)) });
const res = await runPass(h, { base: 'main', out, pass: 'w1p1', cwd: wt, planFlags: ['--depth-floor', 'full'] });
expect(res).toContain('COLLECT_WRONG_TREE: reader=');
expect(res).toContain(`reported="${treeLine(dir).slice(15).trim()}"`);
expect(res).toMatch(/STOP: collect \S+ exited 1/);
expect(res).not.toContain('NEXT: judge');
expect(fs.existsSync(path.join(out, 'merged.jsonl')) ? fs.readFileSync(path.join(out, 'merged.jsonl'), 'utf8') : '').toBe('');
const v = bin(wt, env, ['verify', '--ledger', path.join(out, 'review-ledger.jsonl')]).stdout;
expect(v).toMatch(/^OUTPUT \S+ NOT_REVIEWED .*wrong_tree/m);
expect(v).toContain('VERIFY: dirty');
}, 240_000);
test('a reply that names no checkout is refused the same way', async () => {
const { dir, env, wt } = worktree();
const out = path.join(wt, '.state/run/p1');
const res = await runPass(host(dir, env, { reportFrom: () => '' }), { base: 'main', out, pass: 'w2p1', cwd: wt, planFlags: ['--depth-floor', 'full'] });
expect(res).toContain('COLLECT_WRONG_TREE: reader=');
expect(res).toContain('reported="none"');
expect(res).not.toContain('NEXT: judge');
}, 240_000);
// Value: protects=/ship from a worktree on a shared machine works when readers follow their brief, though they start in another checkout; fails_when=a brief does not name the shipped checkout, or a correct reply is refused;
// why_new=GBRA-218 (no brief named a directory); seam=lib/review-pass.ts renderStage and the Capy pass program; no_claim=model compliance with the paragraph
test('a reader that works in the checkout its brief names passes, and every brief names the worktree', async () => {
const { dir, env, wt } = worktree();
const out = path.join(wt, '.state/run/p1');
fs.mkdirSync(out, { recursive: true });
fs.writeFileSync(path.join(out, 'coverage.prompt.md'), 'Run Step 7 as written.\n');
const h = host(dir, env);
const res = await runPass(h, { base: 'main', out, pass: 'w3p1', cwd: wt, planFlags: ['--depth-floor', 'full'],
extras: [{ name: 'coverage', prompt_file: path.join(out, 'coverage.prompt.md'), first: true }] });
expect(res).not.toContain('STOP:');
expect(res).toContain('NEXT: judge (Phase 4)');
expect(h.creates.length).toBeGreaterThan(2);
for (const c of h.creates) expect(c.prompt.endsWith('\n\n' + withWorkspaceTail(wt)), c.title).toBe(true);
expect(bin(wt, env, ['verify', '--ledger', path.join(out, 'review-ledger.jsonl')]).stdout).not.toContain('wrong_tree');
}, 240_000);
test('an audit reader that worked in the primary checkout is refused when it is collected', async () => {
const { dir, env, wt } = worktree();
const out = path.join(wt, '.state/run/p1');
fs.mkdirSync(out, { recursive: true });
fs.writeFileSync(path.join(out, 'docs.prompt.md'), 'Run the Step 14.5 audit as written.\n');
const h = host(dir, env, { reportFrom: t => (t.endsWith('audit/docs') ? dir : briefDirOf(h, t)) });
const res = await runPass(h, { base: 'main', out, pass: 'w4p1', cwd: wt, planFlags: ['--depth-floor', 'full'],
extras: [{ name: 'docs', prompt_file: path.join(out, 'docs.prompt.md'), first: true }] });
expect(res).toContain('COLLECT_WRONG_TREE: reader=audit/docs');
expect(res).not.toContain('NEXT: judge');
expect(h.creates.map(c => c.title)).toEqual(['ship w4p1 audit/docs']);
}, 240_000);
test('a rerun after a wrong-tree STOP stops on the failed reader and never collects its reply twice', async () => {
const { dir, env, wt } = worktree();
const out = path.join(wt, '.state/run/p1');
const h = host(dir, env, { reportFrom: () => dir });
const input = { base: 'main', out, pass: 'w5p1', cwd: wt, planFlags: ['--depth-floor', 'full'] };
expect(await runPass(h, input)).toContain('COLLECT_WRONG_TREE: reader=');
const again = await runPass(h, input);
expect(again).toMatch(/STOP: \S+ \(subagent \d+\) reviewed another tree: [0-9a-f]{40} /);
const outputs = fs.readFileSync(path.join(out, 'review-ledger.jsonl'), 'utf8').trim().split('\n').map(l => JSON.parse(l)).filter(e => e.event === 'output');
expect(outputs.length).toBe(1);
}, 240_000);
test('a worktree at the same commit as the primary checkout is still told apart by its toplevel', async () => {
const { dir, env } = repo();
const wt = path.join(tmp('rp-wt-'), 'same');
expect(spawnSync('git', ['worktree', 'add', '-q', '--detach', wt, 'feat'], { cwd: dir, env, encoding: 'utf8', timeout: 60_000 }).status).toBe(0);
const out = path.join(wt, '.state/run/p1');
const res = await runPass(host(dir, env, { reportFrom: () => dir }), { base: 'main', out, pass: 'w6p1', cwd: fs.realpathSync(wt), planFlags: ['--depth-floor', 'full'] });
expect(res).toContain('COLLECT_WRONG_TREE: reader=');
expect(res).not.toContain('COLLECT_TREE_MOVED');
}, 240_000);
test('a commit between render and reply is COLLECT_TREE_MOVED, not a pass', async () => {
const { dir, env, wt } = worktree();
const out = path.join(wt, '.state/run/p1');
let moved = false;
const h = host(dir, env, { reportFrom: t => {
if (!moved) { moved = true; spawnSync('bash', ['-c', 'echo "export const late = 1;" >> src/a.ts && git commit -qam late'], { cwd: wt, env, encoding: 'utf8', timeout: 60_000 }); }
return briefDirOf(h, t);
} });
const res = await runPass(h, { base: 'main', out, pass: 'w7p1', cwd: wt, planFlags: ['--depth-floor', 'full'] });
expect(res).toContain('COLLECT_TREE_MOVED: reader=');
expect(res).not.toContain('NEXT: judge');
}, 240_000);
test('an outside voice that reviewed another tree is recorded as unavailable coverage, never clean, and the voices still finish', async () => {
const { dir, env, wt } = worktree();
const out = path.join(wt, '.state/run/p1');
const h = host(dir, env, { reportFrom: t => (t.endsWith('outside-voice') ? dir : briefDirOf(h, t)) });
const base = { base: 'main', out, pass: 'w8p1', cwd: wt, planFlags: ['--depth-floor', 'full'] };
expect(await runPass(h, base)).toContain('NEXT: judge (Phase 4)');
const models = { 'outside-voice': { model_id: 'openai/gpt-6.1-sol' }, 'adversarial-native': { model_id: 'anthropic/claude-opus-5-5', reasoning_mode: 'high' } };
const res = await runPass(h, { ...base, stage: 'voices', models });
expect(res).toContain('COLLECT_UNAVAILABLE: reader=outside-voice');
expect(res).toContain('reviewed another tree');
expect(res).toContain('NEXT: judge the voices');
const v = bin(wt, env, ['verify', '--ledger', path.join(out, 'review-ledger.jsonl')]).stdout;
expect(v).toMatch(/^BRIEF outside-voice UNAVAILABLE /m);
}, 240_000);
});
const withWorkspaceTail = (dir: string) => withWorkspace('', dir).slice(2);
const briefDirOf = (h: ReturnType<typeof host>, title: string) => briefDir(h.creates.find(c => c.title === title)?.prompt ?? '');
+2 -1
View File
@@ -18,6 +18,7 @@ import { carryReceipt, extractReceipt, renderReceiptBlock } from '../lib/ship-re
import { bindShippedReceipt, parseReviewTargeted, receiptTargetedLine } from '../lib/ship-receipt-depth';
import type { ShipReceipt } from '../lib/headless-artifacts';
import { cleanup, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/restamp-fixture';
import { checkoutTree } from '../lib/review-brief';
const ROOT = path.resolve(import.meta.dir, '..');
const PLAN = path.join(ROOT, 'bin', 'gstack-review-plan');
@@ -268,7 +269,7 @@ describe('the targeted role end to end on a fixture repository', () => {
expect(runBin(repo, 'gstack-review-dispatch', ['sent', '--ledger', ledger, '--role', 'targeted', '--sent', file]).out).toContain('BRIEF_SENT: reader=targeted');
const output = path.join(repo.root, 'targeted.out');
const finding = { severity: 'CRITICAL', confidence: 8, path: 'src/charge.ts', line: 1, category: 'money', summary: 'cap is never enforced', fingerprint: 'src/charge.ts:1:money', specialist: 'targeted' };
fs.writeFileSync(output, JSON.stringify(finding) + '\n');
fs.writeFileSync(output, `REVIEWED_TREE: ${checkoutTree(repo.work)}\n` + JSON.stringify(finding) + '\n');
const merged = path.join(repo.root, 'merged.jsonl');
expect(runBin(repo, 'gstack-review-dispatch', ['collect', '--ledger', ledger, '--role', 'targeted', '--output', output, '--merged', merged]).out).toContain('COLLECT: reader=targeted findings=1');
const judged = path.join(repo.root, 'judged.jsonl');
+183
View File
@@ -0,0 +1,183 @@
/**
* lib/run-evidence.ts: a GitHub Actions run as evidence for one tree (GSTA-156). Every refusal reason has a fixture;
* the dependencies are fakes, so nothing here reaches GitHub.
*/
import { describe, expect, test } from 'bun:test';
import { parseRunReceipt, parseRunRef, runInventory, runProfileErrors, runReceiptFields, uncoveredFiles, verifyRun, type ApiJob, type ApiRun, type RunDeps, type RunProfile, type TreeRecord } from '../lib/run-evidence';
const REPO = 'garrytan/gbrain';
const HEAD = 'a'.repeat(40);
const TREE = 'b'.repeat(40);
const NOW = new Date('2026-10-29T12:00:00Z');
const PROFILE: RunProfile = { workflow: 'test.yml', events: ['workflow_dispatch', 'push'], jobs: ['full-suite'], artifact: 'full-suite-tree' };
interface World { run: ApiRun; others: ApiRun[]; jobs: ApiJob[]; artifact: TreeRecord | Error; trees: Record<string, string | null>; throwOn?: 'run'; later?: ApiRun }
function world(over: Partial<World> = {}): World {
const run: ApiRun = {
id: 100, run_attempt: 1, status: 'completed', conclusion: 'success', event: 'workflow_dispatch', head_sha: HEAD,
path: '.github/workflows/test.yml', created_at: '2026-10-29T10:00:00Z', updated_at: '2026-10-29T10:20:00Z',
repository: { full_name: REPO }, head_repository: { full_name: REPO },
};
return {
run, others: [], jobs: [{ name: 'full-suite', conclusion: 'success' }, { name: 'lint', conclusion: 'success' }],
artifact: { run_id: 100, run_attempt: 1, event: 'workflow_dispatch', sha: HEAD, tree: TREE, files: ['test/a.test.ts', 'test/b.test.ts'], tests: 12 },
trees: { [HEAD]: TREE }, ...over,
};
}
function deps(w: World): RunDeps {
let reads = 0;
return {
run: id => { if (w.throwOn === 'run') throw new Error('HTTP 502'); if (id !== w.run.id) throw new Error('HTTP 404'); return reads++ && w.later ? w.later : w.run; },
runsForHead: (_wf, sha) => [w.run, ...w.others].filter(r => r.head_sha === sha),
jobs: () => w.jobs,
artifact: () => { if (w.artifact instanceof Error) throw w.artifact; return w.artifact; },
commitTree: sha => w.trees[sha] ?? null,
};
}
const check = (w: World, extra: { runRef?: string; profile?: RunProfile; vetoHeads?: string[] } = {}) =>
verifyRun(deps(w), { repo: REPO, runRef: extra.runRef ?? '100', profileName: 'test', profile: extra.profile ?? PROFILE, vetoHeads: extra.vetoHeads, now: NOW });
const reason = (w: World, extra: Parameters<typeof check>[1] = {}) => { const v = check(w, extra); return v.ok ? 'ok' : v.reason; };
describe('verifyRun', () => {
test('a completed same-repository run with its job, artifact and head tree agreeing is evidence for that tree', () => {
const v = check(world());
expect(v).toEqual({ ok: true, evidence: { run_id: 100, run_attempt: 1, head_sha: HEAD, tree: TREE, profile: 'test', event: 'workflow_dispatch', finished_at: '2026-10-29T10:20:00Z', files: ['test/a.test.ts', 'test/b.test.ts'], tests: 12 } });
});
test('a run URL on this repository is accepted; another repository, a malformed ref and an unknown host are not', () => {
expect(reason(world(), { runRef: `https://github.com/${REPO}/actions/runs/100/attempts/1` })).toBe('ok');
expect(reason(world(), { runRef: 'https://github.com/someone/else/actions/runs/100' })).toBe('run_ref_invalid');
expect(reason(world(), { runRef: 'https://example.com/garrytan/gbrain/actions/runs/100' })).toBe('run_ref_invalid');
expect(reason(world(), { runRef: '100; rm -rf /' })).toBe('run_ref_invalid');
});
test('each refusal names its reason', () => {
const base = world();
const cases: Array<[string, World]> = [
['other_repository', world({ run: { ...base.run, repository: { full_name: 'x/y' } } })],
['fork_run', world({ run: { ...base.run, head_repository: { full_name: 'fork/gbrain' } } })],
['workflow_mismatch', world({ run: { ...base.run, path: '.github/workflows/e2e.yml' } })],
['event_mismatch', world({ run: { ...base.run, event: 'pull_request' } })],
['run_in_progress', world({ run: { ...base.run, status: 'in_progress', conclusion: null } })],
['run_not_success', world({ run: { ...base.run, conclusion: 'failure' } })],
['run_not_success', world({ run: { ...base.run, conclusion: 'cancelled' } })],
['run_stale', world({ run: { ...base.run, updated_at: '2026-10-28T10:00:00Z' } })],
['job_missing', world({ jobs: [{ name: 'lint', conclusion: 'success' }] })],
['job_ambiguous', world({ jobs: [{ name: 'full-suite', conclusion: 'success' }, { name: 'full-suite', conclusion: 'success' }] })],
['job_not_success', world({ jobs: [{ name: 'full-suite', conclusion: 'skipped' }] })],
['artifact_missing', world({ artifact: new Error('no artifact named full-suite-tree') })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), run_id: 99 } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), run_attempt: 2 } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), sha: 'c'.repeat(40) } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), head_sha: 'c'.repeat(40) } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), sha: undefined } })],
['artifact_reused', world({ artifact: { ...(base.artifact as TreeRecord), reused: true } })],
['tree_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), tree: 'd'.repeat(40) } })],
['api_error', world({ trees: {} })],
['api_error', world({ throwOn: 'run' })],
];
for (const [want, w] of cases) expect([want, reason(w)]).toEqual([want, want]);
});
test('the newest attempt decides: a run whose latest attempt failed is refused even if an earlier attempt passed', () => {
const w = world();
expect(reason(world({ run: { ...w.run, run_attempt: 2, conclusion: 'failure' }, artifact: { ...(w.artifact as TreeRecord), run_attempt: 2 } }))).toBe('run_not_success');
});
test('a newer failed, cancelled or running run of the workflow on the head vetoes; an older one or a newer green does not', () => {
const w = world();
const newer = (status: string, conclusion: string | null): ApiRun => ({ ...w.run, id: 101, created_at: '2026-10-29T11:00:00Z', status, conclusion });
expect(reason(world({ others: [newer('completed', 'failure')] }))).toBe('newer_run_vetoes');
expect(reason(world({ others: [newer('completed', 'cancelled')] }))).toBe('newer_run_vetoes');
expect(reason(world({ others: [newer('in_progress', null)] }))).toBe('newer_run_vetoes');
expect(reason(world({ others: [newer('completed', 'success')] }))).toBe('ok');
expect(reason(world({ others: [{ ...newer('completed', 'failure'), created_at: '2026-10-29T09:00:00Z' }] }))).toBe('ok');
// A re-run keeps its run's created_at; its newest attempt started after the candidate, so its red result still vetoes.
expect(reason(world({ others: [{ ...newer('completed', 'failure'), created_at: '2026-10-29T09:00:00Z', run_started_at: '2026-10-29T11:00:00Z' }] }))).toBe('newer_run_vetoes');
// Re-running the candidate itself does not hide a failure that started after it was first created.
expect(reason(world({ run: { ...w.run, run_started_at: '2026-10-29T12:00:00Z' }, others: [newer('completed', 'failure')] }))).toBe('newer_run_vetoes');
});
test('a newer failure on another head the consumer names (same tree, the lane\'s own head) vetoes too', () => {
const w = world();
const other = 'e'.repeat(40);
const red: ApiRun = { ...w.run, id: 102, head_sha: other, created_at: '2026-10-29T11:30:00Z', conclusion: 'failure' };
expect(reason(world({ others: [red] }))).toBe('ok');
expect(reason(world({ others: [red] }), { vetoHeads: [other] })).toBe('newer_run_vetoes');
});
test('the run is read again after the evidence: a newer attempt of the same run, seen or started meanwhile, refuses', () => {
const w = world();
expect(reason(world({ later: { ...w.run, run_attempt: 2, status: 'in_progress', conclusion: null } }))).toBe('run_changed');
expect(reason(world({ others: [{ ...w.run, run_attempt: 2, run_started_at: '2026-10-29T11:00:00Z', status: 'completed', conclusion: 'failure' }] }))).toBe('newer_run_vetoes');
});
test('tree.json counts and file lists must be well formed', () => {
const art = world().artifact as TreeRecord;
expect(reason(world({ artifact: { ...art, tests: -1 } }))).toBe('artifact_mismatch');
expect(reason(world({ artifact: { ...art, tests: 1.5 } }))).toBe('artifact_mismatch');
expect(reason(world({ artifact: { ...art, files: 'test/a.test.ts' } }))).toBe('artifact_mismatch');
const v = check(world({ artifact: { ...art, files: ['./test/a.test.ts'], tests: undefined } }));
expect(v.ok && [v.evidence.files, v.evidence.tests]).toEqual([['test/a.test.ts'], null]);
});
test('a profile that names an OS needs the artifact to record it', () => {
const w = world();
expect(reason(w, { profile: { ...PROFILE, os: 'Linux' } })).toBe('os_mismatch');
expect(reason(world({ artifact: { ...(w.artifact as TreeRecord), os: 'Linux' } }), { profile: { ...PROFILE, os: 'Linux' } })).toBe('ok');
});
test('a failed run created in the same second as the candidate vetoes it: ties count as newer', () => {
const w = world();
expect(reason(world({ others: [{ ...w.run, id: 101, status: 'completed', conclusion: 'failure' }] }))).toBe('newer_run_vetoes');
});
test('freshness reads the jobs\' latest completed_at when the API gives it, not a later updated_at', () => {
const w = world();
const jobs: ApiJob[] = [{ name: 'full-suite', conclusion: 'success', completed_at: '2026-10-28T10:00:00Z' }, { name: 'lint', conclusion: 'success', completed_at: '2026-10-28T09:00:00Z' }];
expect(reason(world({ jobs }))).toBe('run_stale');
const v = check(world({ jobs: jobs.map(j => ({ ...j, completed_at: '2026-10-29T10:10:00Z' })) }));
expect(v.ok && v.evidence.finished_at).toBe('2026-10-29T10:10:00Z');
expect(w.run.updated_at).toBe('2026-10-29T10:20:00Z');
});
test('max_age_hours widens the freshness window', () => {
const stale = world({ run: { ...world().run, updated_at: '2026-10-28T10:00:00Z' } });
expect(reason(stale)).toBe('run_stale');
expect(reason(stale, { profile: { ...PROFILE, max_age_hours: 48 } })).toBe('ok');
});
});
describe('runInventory', () => {
test('lists unique non-empty lines, and a pipeline whose producer fails refuses instead of listing part of the gate', () => {
const cwd = process.cwd();
expect(runInventory("printf 'test/b.test.ts\\n\\ntest/a.test.ts\\ntest/b.test.ts\\n'", cwd)).toEqual(['test/b.test.ts', 'test/a.test.ts']);
expect(runInventory("{ printf 'test/a.test.ts\\n'; exit 17; } | sort", cwd)).toMatchObject({ error: expect.stringContaining('exited 17') });
});
});
describe('profiles, coverage and receipts', () => {
test('runProfileErrors accepts a well-formed pair and names every malformed field', () => {
expect(runProfileErrors({ test: PROFILE, e2e: { ...PROFILE, workflow: 'e2e.yml', jobs: ['e2e-full-suite'], artifact: 'e2e-full-suite-tree', os: 'Linux', max_age_hours: 12 } }, { profiles: ['test', 'e2e'], inventory: 'bash scripts/pregate-remote-gate.sh --list' })).toEqual([]);
const errors = runProfileErrors({ bad: { workflow: '../x.yml', events: ['pull_request'], jobs: [], artifact: '', max_age_hours: 0, os: 'BeOS', extra: 1 } }, { profiles: ['missing'], inventory: 'a\nb' }).map(e => e.path);
expect(errors).toEqual(expect.arrayContaining(['run_profiles["bad"].extra', 'run_profiles["bad"].workflow', 'run_profiles["bad"].events', 'run_profiles["bad"].jobs', 'run_profiles["bad"].artifact', 'run_profiles["bad"].max_age_hours', 'run_profiles["bad"].os', 'remote_gate_runs.profiles', 'remote_gate_runs.inventory']));
expect(runProfileErrors({ test: PROFILE }, { profiles: ['test', 'test'], inventory: 'true' })).toEqual([{ path: 'remote_gate_runs.profiles', message: 'names each profile once' }]);
});
test('uncoveredFiles lists the inventory files no run recorded', () => {
const ev = (files: string[]) => ({ run_id: 1, run_attempt: 1, head_sha: HEAD, tree: TREE, profile: 'p', event: 'push', finished_at: '', files, tests: null });
expect(uncoveredFiles(['a', 'b', 'c'], [ev(['a']), ev(['b'])])).toEqual(['c']);
expect(uncoveredFiles(['a'], [ev(['a', 'z'])])).toEqual([]);
expect(uncoveredFiles(['./test/x.test.ts'], [ev(['test/x.test.ts'])])).toEqual([]);
});
test('the receipt fields round-trip and keep the tree prefix verify reads', () => {
const v = check(world());
if (!v.ok) throw new Error(v.detail);
const receipt = `remote-gate tree=${TREE.slice(0, 12)} tests=12 ${runReceiptFields([v.evidence], NOW)}`;
expect(receipt).toBe(`remote-gate tree=${TREE.slice(0, 12)} tests=12 runs=100/1 profiles=test heads=${HEAD.slice(0, 12)} verified=2026-10-29T12:00:00.000Z`);
expect(parseRunReceipt(receipt)).toEqual({ runs: ['100/1'], profiles: ['test'], tree: TREE.slice(0, 12) });
expect(parseRunReceipt(`remote-gate tree=${TREE.slice(0, 12)} tests=3 receipt=/x sha256=abc`)).toBeNull();
expect(parseRunRef('123', REPO)).toBe(123);
});
});
+48 -3
View File
@@ -37,12 +37,22 @@ const UBI_RUNNER_STUB = `#!/usr/bin/env bash
# VM directory, runs setup and the command there, and prints ubi-runner's log
# lines. STUB_FAIL picks a failure: create, quota, quota-once, ssh. STUB_HANG
# sleeps before the command; TERM tears down as ubi-runner's trap does.
# Every teardown is appended to $STUB_CALLS.down.
# Every teardown is appended to $STUB_CALLS.down. STUB_REATTACH=<vm>:<created_at>
# plays a live run record for this key: --probe reports it (busy per
# STUB_REATTACH_BUSY) and run prints ubi-runner's reattached line instead of
# creating a VM.
set -uo pipefail
[ "$1" != owner ] || { echo stub; exit 0; }
left() { [ "\${STUB_LEFT:-0}" = none ] || echo "UBICLOUD: left=\${STUB_LEFT:-0} owner=stub vm=$name others=0" | tee -a "\${UBI_LEFT_FILE:-/dev/null}" >&2; }
[ "$1" = run ] || exit 2
shift
case " $* " in
*" --probe "*)
printf '%s\\n' "$*" >>"$STUB_CALLS.probe"
if [ -n "\${STUB_REATTACH:-}" ]; then echo "UBI_ATTACH_INFO: vm=\${STUB_REATTACH%%:*} created_at=\${STUB_REATTACH#*:} key=0123456789ab busy=\${STUB_REATTACH_BUSY:-0}"
else echo "UBI_ATTACH_INFO: none key=0123456789ab"; fi
exit 0 ;;
esac
printf '%s\\n' "$*" >>"$STUB_CALLS"
src=. setup="" size="" name=unnamed envs=()
while [ $# -gt 0 ]; do
@@ -52,16 +62,20 @@ while [ $# -gt 0 ]; do
--src) src=$2; shift 2 ;;
--setup) setup=$2; shift 2 ;;
--env) envs+=("$2"); shift 2 ;;
--pull) shift 2 ;;
--pull|--key-input) shift 2 ;;
--) shift; break ;;
*) echo "stub: unknown option $1" >&2; exit 2 ;;
esac
done
cmd=$*
log() { echo "ubi-runner: $*" >&2; }
log "creating eu-central-h1/$name ($size)"
calls=$(wc -l <"$STUB_CALLS")
fail=\${STUB_FAIL:-}
if [ -n "\${STUB_REATTACH:-}" ]; then
name=\${STUB_REATTACH%%:*}
log "reattached to $name created_at=\${STUB_REATTACH#*:} key=0123456789ab"
else
log "creating eu-central-h1/$name ($size)"
if [ "$fail" = quota ] || { [ "$fail" = quota-once ] && [ "$calls" -eq 1 ]; }; then
log "quota refused eu-central-h1/$name ($size): it needs 16 vCPUs and the project already uses 64 of 64"
log "create failed: vCPU quota exhausted; wait for running VMs to finish, or use fewer or smaller VMs (ci:ubicloud --vms N)"
@@ -77,6 +91,7 @@ if [ "$fail" = create ]; then
exit 1
fi
log "ready: $name (192.0.2.1)"
fi
trap 'log "destroyed eu-central-h1/$name"; left; echo "term $name" >>"$STUB_CALLS.down"; exit 130' TERM
vm=$STUB_VM_HOME
dest="$vm/work/$(basename "$src")"
@@ -282,6 +297,36 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
expect(twice.out).toContain('create failed: vCPU quota exhausted');
}, 90_000);
test('a rerun for the same tree and planned set reattaches: the receipt still parses, spent= bills from the original created_at, and a used-up wall budget refuses', () => {
const created = Math.floor(Date.now() / 1000) - 5 * 60 - 10;
const f = fixture();
const r = gate(f, ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}` });
expect(r.exit, r.out).toBe(0);
expect(r.out).toContain('reattaching to VM ubirun-stub-old, created 6 min ago by an earlier run of this gate for the same tree and planned set');
expect(r.out).toContain("the VM is told to stop after 2160s (the cap's 45 min wall minus 3 min for teardown and 6 min already billed since ubirun-stub-old was created)");
expect(r.out).not.toContain('if this run is killed before its teardown line');
expect(r.out.trim().split('\n').at(-1)).toMatch(/^PREGATE_REMOTE: /);
expect(r.report).toMatchObject({ tree: f.tree, tests: 1, failed: 0 });
// Billed from created_at (5 min 10 s before the gate started), not from the reattach: 6 min at $0.00831.
expect(r.report!.spent).toBe('0.05');
expect(remoteReportProblem(r.report!, f.tree)).toBeNull();
const plan = new RegExp(`--key-input pregate-remote tree=${f.tree} plan=[0-9a-f]{64}`);
expect(r.calls[0]).toMatch(plan);
expect(fs.readFileSync(f.calls + '.probe', 'utf8')).toMatch(plan);
const spent = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created - 40 * 60}` });
expect(spent.exit, spent.out).toBe(5);
expect(spent.out).toContain("PREGATE_REMOTE_CAP_SPENT: VM ubirun-stub-old, created 46 min ago for this tree and planned set, has already used the cap's 45 min wall ($0.39 of $2)");
expect(spent.out).toContain('nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down ubirun-stub-old');
expect(spent.report).toBeNull();
expect(spent.calls).toEqual([]);
const busy = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}`, STUB_REATTACH_BUSY: '1' });
expect(busy.exit, busy.out).toBe(4);
expect(busy.out).toContain('PREGATE_REMOTE_ATTACH_BUSY: another gate on this machine still runs VM ubirun-stub-old');
expect(busy.calls).toEqual([]);
}, 90_000);
test('the VM refuses a tree that differs after unpack, a planned set that differs from the list, and a tree that moved during the run', () => {
const edited = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_VM_EDIT: 'test/alpha.test.ts' });
expect(edited.exit, edited.out).toBe(3);
+391 -31
View File
@@ -389,18 +389,23 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
/**
* Fake Ubicloud API that can create: create adds the name to the listed file
* (or answers the quota refusal in refuse with HTTP 400), show is running at
* 192.0.2.1 while listed, destroy unlists. The fake ssh records every remote
* command in ssh.log and plays the role the command asks for: it stores
* ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, drains tar
* uploads, runs the setup phase per setup-mode (ok or hang) and echoes the
* command phase. No network, no VM.
* 192.0.2.1 while listed, destroy unlists. The fake VM is the directory vm/:
* the fake ssh records every remote command in ssh.log, keeps copies of
* ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, serves pulls
* from remote-fs/, and runs every other command there for real (HOME=vm/,
* vmbin/ first on PATH), so uploads unpack, detached phases write their pid,
* output and exit files, tail --pid follows them and pkill ends them. The stub
* make in vmbin/ echoes `ran: make ARGS`, or runs command-mode or
* make-<target>.sh from the fixture root. No network, no VM.
*/
function fakeVm(opts: { listed?: string; refuse?: string } = {}) {
const root = mkdtempSync(join(tmpdir(), 'ubi-run-'));
const bin = join(root, 'bin'), state = join(root, 'state'), log = join(root, 'calls.log'), sshLog = join(root, 'ssh.log'), record = join(root, 'vms.tsv');
const vm = join(root, 'vm'), vmbin = join(root, 'vmbin'), srcDir = join(root, 'src'), checkpoints = join(root, 'checkpoints');
mkdirSync(vm); mkdirSync(vmbin); mkdirSync(srcDir);
writeFileSync(join(srcDir, 'README'), 'fixture checkout\n');
mkdirSync(bin); mkdirSync(state, { recursive: true });
writeFileSync(join(root, 'listed'), opts.listed ?? '');
writeFileSync(join(root, 'setup-mode'), 'ok');
writeFileSync(join(bin, 'curl'), '#!/usr/bin/env bash\n' +
'out=""; while [ $# -gt 0 ]; do [ "$1" = -o ] && out=$2; shift; done\n' +
'body=$(cat); echo "$body" >> ' + JSON.stringify(log) + '\n' +
@@ -415,26 +420,33 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
' *) : > "$out"; printf 200 ;;\n' +
'esac\n');
writeFileSync(join(bin, 'ssh'), '#!/usr/bin/env bash\n' +
'root=' + JSON.stringify(root) + '\n' +
'root=' + JSON.stringify(root) + '; vm="$root/vm"\n' +
'cmd=${*: -1}\n' +
'printf \'%s\\n\' "$cmd" >> "$root/ssh.log"\n' +
'export HOME="$vm" PATH="$root/vmbin:$PATH"\n' +
'cd "$vm" || exit 255\n' +
'case "$cmd" in\n' +
' *pkill*) echo "$cmd" >> "$root/cancel.log" ;;\n' +
' *pkill*) echo "$cmd" >> "$root/cancel.log"; exec bash -c "$cmd" ;;\n' +
' *captured*) cat > "$root/captured" ;;\n' +
' *"cat > ~/.ubirun-env"*) cat > "$root/ubirun-env" ;;\n' +
' *"cat > ~/.ubirun-setup.sh"*) cat > "$root/ubirun-setup.sh" ;;\n' +
' *"tar -xzf -"*) cat > /dev/null ;;\n' +
' *"cat > ~/.ubirun-env"*) tee "$root/ubirun-env" > "$vm/.ubirun-env" ;;\n' +
' *"cat > ~/.ubirun-setup.sh"*) tee "$root/ubirun-setup.sh" > "$vm/.ubirun-setup.sh" ;;\n' +
' *"tar -czf -"*) base=${cmd##* }; tar -czf - -C "$root/remote-fs" "$base" ;;\n' +
' *"bash -l ~/.ubirun-setup.sh"*) echo "setup start"; if [ "$(cat "$root/setup-mode")" = hang ]; then sleep 20; fi; echo "setup done" ;;\n' +
' *"bash -lc "*) if [ -f "$root/command-mode" ]; then bash -c "$(cat "$root/command-mode")"; else echo "ran: $cmd"; fi ;;\n' +
' *) : ;;\n' +
' *"ls -d -- "*) : ;;\n' +
' *) exec bash -c "$cmd" ;;\n' +
'esac\n');
chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755);
const env: Record<string, string> = { ...(process.env as Record<string, string>), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0' };
writeFileSync(join(vmbin, 'make'), '#!/usr/bin/env bash\n' +
'root=' + JSON.stringify(root) + '\n' +
'if [ -f "$root/command-mode" ]; then exec bash -c "$(cat "$root/command-mode")"; fi\n' +
'echo "ran: make $*"\n' +
'[ ! -f "$root/make-$1.sh" ] || exec bash "$root/make-$1.sh"\n');
chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755); chmodSync(join(vmbin, 'make'), 0o755);
const env: Record<string, string> = { ...(process.env as Record<string, string>), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0', UBI_CHECKPOINT_DIR: checkpoints };
const run = (args: string[], extra: Record<string, string> = {}, stdin?: string) =>
Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: stdin === undefined ? undefined : Buffer.from(stdin), timeout: 90_000 });
const spawn = (args: string[], extra: Record<string, string> = {}) =>
Bun.spawn(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: 'ignore', stdout: 'pipe', stderr: 'pipe', timeout: 120_000 });
return {
root, state, record, run,
root, state, record, run, spawn, vm, src: srcDir, checkpoints,
calls: () => (existsSync(log) ? readFileSync(log, 'utf8') : ''),
ssh: () => (existsSync(sshLog) ? readFileSync(sshLog, 'utf8') : ''),
listed: () => readFileSync(join(root, 'listed'), 'utf8'),
@@ -455,17 +467,18 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
test('run: a hung setup is killed by the watchdog, exits 124, never reaches the command, and the VM is destroyed', () => {
const f = fakeVm();
try {
writeFileSync(join(f.root, 'setup-mode'), 'hang');
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo hi\n');
const r = run(f, ['run', '--src', f.root, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' });
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo "setup start"; sleep 20\n');
const r = run(f, ['run', '--src', f.src, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' });
const err = r.stderr.toString();
expect(r.exitCode, err).toBe(124);
expect(err).toMatch(/watchdog: setup printed nothing for \d+s \(UBI_IDLE_TIMEOUT=2\)/);
expect(err).toContain('setup failed (exit 124)');
expect(f.ssh()).not.toContain('bash -lc');
expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/cat ~\/\.ubirun-setup-\d+\.sid/);
expect(f.ssh()).toMatch(/setsid -w bash -c 'printf %s \$\$ >~\/\.ubirun-setup-\d+\.sid; cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh'; else bash -c '[^']*'; fi$/m);
expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/rm -f ~\/\.ubirun-setup-\d+\.sid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/);
expect(err).toContain('setup start');
// The setup runs detached in its own session: its leader's pid, both streams and the exit status are files on the VM.
expect(f.ssh()).toMatch(/setsid -f bash -c 'echo \$\$ >~\/\.ubirun-setup\.pid\.tmp && mv ~\/\.ubirun-setup\.pid\.tmp ~\/\.ubirun-setup\.pid; \( cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh \) >~\/\.ubirun-setup\.out 2>~\/\.ubirun-setup\.err <\/dev\/null; echo \$\? >~\/\.ubirun-setup\.exit\.tmp && mv ~\/\.ubirun-setup\.exit\.tmp ~\/\.ubirun-setup\.exit' <\/dev\/null/m);
expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/sid=\$\(cat ~\/\.ubirun-setup\.pid 2>\/dev\/null\); rm -f ~\/\.ubirun-setup\.pid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/);
expect(existsSync(join(f.vm, '.ubirun-setup.exit'))).toBe(false);
expect(f.calls()).toContain('destroy');
expect(err).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3/);
expect(f.listed()).toBe('');
@@ -477,7 +490,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const f = fakeVm();
try {
writeFileSync(join(f.root, 'command-mode'), 'for i in 1 2 3 4 5 6 7 8 9 10; do echo "tick $i"; sleep 1; done');
const r = run(f, ['run', '--src', f.root, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' });
const r = run(f, ['run', '--src', f.src, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' });
expect(r.exitCode).toBe(124);
expect(r.stderr.toString()).toMatch(/watchdog: command ran \d+s \(UBI_RUN_TIMEOUT=3\); killing it/);
expect(r.stderr.toString()).toContain('command exited 124 after');
@@ -489,7 +502,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const f = fakeVm({ listed: 'eu-central-h1 ubirun-thrd1-9999999999-sib1-t3\n' });
try {
const leftFile = join(f.root, 'left');
const r = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile });
const r = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile });
const err = r.stderr.toString();
expect(r.exitCode, err).toBe(0);
expect(err).toMatch(/UBICLOUD: left=0 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
@@ -499,7 +512,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const parser = read('scripts/ubicloud/pregate-remote.sh').match(/left=\$\(sed -n '([^']+)' "\$log\.left"/)![1];
expect(Bun.spawnSync(['sed', '-n', parser, leftFile], { timeout: 30_000 }).stdout.toString()).toBe('0\n');
// Value: protects=run --keep reports its live VM as UBICLOUD: left=1 ... kept=1 and never a clean left=0; fails_when=the kept line is dropped or --keep still prints the teardown line; why_new=no test runs run --keep; seam=none; no_claim=real VM lifetime (fake API and ssh)
const kept = f.run(['run', '--src', f.root, '--keep', '--', 'true'], { UBI_REAP: '0' });
const kept = f.run(['run', '--src', f.src, '--keep', '--', 'true'], { UBI_REAP: '0' });
const keptErr = kept.stderr.toString();
expect(kept.exitCode, keptErr).toBe(0);
expect(keptErr).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 kept=1\n/);
@@ -512,14 +525,14 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const f = fakeVm();
try {
writeFileSync(join(f.root, 'listed.destroy-fails'), '');
const leaked = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
const leaked = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
const err = leaked.stderr.toString();
expect(leaked.exitCode, err).toBe(1);
expect(err).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=(ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3) others=0\n[^]*VM cleanup not confirmed for \1 \(left=1\); destroy it with: \S+ down \1/);
expect(f.listed()).toContain('ubirun-thrd1-');
// Value: protects=a failing command whose VM also leaks keeps its own exit status, not 1; fails_when=run_teardown exits 1 whenever left is not 0; why_new=the leak case above only runs a passing command; seam=none; no_claim=real destroy failure modes (fake API returns 500)
writeFileSync(join(f.root, 'command-mode'), 'exit 7');
const failedLeak = f.run(['run', '--src', f.root, '--', 'false'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
const failedLeak = f.run(['run', '--src', f.src, '--', 'make test'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
expect(failedLeak.exitCode, failedLeak.stderr.toString()).toBe(7);
expect(failedLeak.stderr.toString()).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
const blind = fakeVm();
@@ -529,7 +542,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
writeFileSync(join(blind.root, 'bin/curl'), '#!/usr/bin/env bash\nif [ -f ' + JSON.stringify(join(blind.root, 'armed')) + ' ]; then touch ' + JSON.stringify(join(blind.root, 'listed.list-fails')) + '; fi\nexec ' + JSON.stringify(join(blind.root, 'bin/curl.real')) + ' "$@"\n');
chmodSync(join(blind.root, 'bin/curl'), 0o755);
writeFileSync(join(blind.root, 'command-mode'), 'touch ' + JSON.stringify(join(blind.root, 'armed')));
const r = blind.run(['run', '--src', blind.root, '--', 'true'], { UBI_REAP: '0' });
const r = blind.run(['run', '--src', blind.src, '--', 'make test'], { UBI_REAP: '0' });
expect(r.exitCode, r.stderr.toString()).toBe(1);
expect(r.stderr.toString()).toMatch(/UBICLOUD: left=unknown owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=unknown\n/);
} finally { rmSync(blind.root, { recursive: true, force: true }); }
@@ -551,7 +564,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
test('run: --pass forwards a named local variable only; UBICLOUD_* and non-names are refused before any API call', () => {
const f = fakeVm();
try {
const ok = run(f, ['run', '--src', f.root, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' });
const ok = run(f, ['run', '--src', f.src, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' });
expect(ok.exitCode).toBe(0);
expect(ok.stderr.toString()).toContain('command exited 0 after');
const envFile = readFileSync(join(f.root, 'ubirun-env'), 'utf8');
@@ -561,7 +574,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
expect(ok.stdout.toString() + ok.stderr.toString()).not.toContain('sekret');
for (const bad of ['UBICLOUD_API_KEY', 'UBICLOUD_API_TOKEN', 'X;Y', 'HOME/..']) {
const before = f.calls();
const r = run(f, ['run', '--src', f.root, '--pass', bad, '--', 'true']);
const r = run(f, ['run', '--src', f.src, '--pass', bad, '--', 'true']);
expect(r.exitCode).toBe(1);
expect(r.stderr.toString()).toMatch(bad.startsWith('UBICLOUD_') ? /never forwards its own Ubicloud credentials/ : /not a variable name/);
expect(f.calls()).toBe(before);
@@ -753,4 +766,351 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
}
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 30_000);
// GSTA-158 P3: detached phases, reattach and tier checkpoints.
const until = async (cond: () => boolean, ms: number) => {
const deadline = Date.now() + ms;
while (!cond()) { if (Date.now() > deadline) throw new Error('condition not met in time'); await Bun.sleep(100); }
};
const vmDirs = (f: ReturnType<typeof fakeVm>) => (existsSync(f.state) ? readdirSync(f.state).filter(d => d.startsWith('ubirun-')) : []);
const phaseStarted = (f: ReturnType<typeof fakeVm>, phase: string) => vmDirs(f).some(d => existsSync(join(f.state, d, 'phase.' + phase + '.start')));
const creates = (f: ReturnType<typeof fakeVm>) => f.calls().split('\n').filter(l => l.includes('"create"')).length;
const killSession = async (f: ReturnType<typeof fakeVm>, phase: string) => {
const pidFile = join(f.vm, '.ubirun-' + phase + '.pid');
await until(() => existsSync(pidFile) && readFileSync(pidFile, 'utf8').trim() !== '', 10_000);
const pid = Number(readFileSync(pidFile, 'utf8').trim());
try { process.kill(-pid, 'SIGKILL'); } catch { /* already gone */ }
await until(() => { try { process.kill(pid, 0); return false; } catch { return true; } }, 10_000);
};
const gitSrc = (f: ReturnType<typeof fakeVm>, cfg?: unknown) => {
const dir = cfg === undefined ? join(f.root, 'repo') : src(f, cfg);
mkdirSync(dir, { recursive: true });
writeFileSync(join(dir, 'README'), 'fixture repo\n');
for (const args of [['init', '-q', '-b', 'main'], ['add', '-A'], ['commit', '-qm', 'fixture']]) {
const r = Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', ...args], { cwd: dir, timeout: 30_000 });
if (r.exitCode !== 0) throw new Error('git ' + args.join(' ') + ': ' + r.stderr.toString());
}
return dir;
};
const TIERS = { setup: 'scripts/boot.sh', tiers: [{ name: 't1', command: 'make t1' }, { name: 't2', command: 'make t2' }, { name: 't3', command: 'make t3' }] };
const bootCounts = (f: ReturnType<typeof fakeVm>, dir: string) => writeFileSync(join(dir, 'scripts/boot.sh'), 'echo boot >> ~/setup-count\n');
const verifyLine = (out: string) => out.split('\n').find(l => l.startsWith('UBI_VERIFY: ')) ?? '';
test('run: a SIGKILLed runner leaves its VM; the same command reattaches, skips the finished setup and reports the command\'s exit', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo setup >> ~/setup-count\n');
writeFileSync(join(f.root, 'make-test.sh'), 'echo begin; sleep 4; echo end; exit 3\n');
const args = ['run', '--src', repo, '--setup', setup, '--', 'make test'];
const a = f.spawn(args);
await until(() => vmDirs(f).some(d => existsSync(join(f.state, d, 'command.out')) && readFileSync(join(f.state, d, 'command.out'), 'utf8').includes('begin')), 30_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
const name = vmDirs(f)[0]!;
expect(f.listed()).toContain(name);
const b = f.run(args);
const err = b.stderr.toString();
expect(b.exitCode, err).toBe(3);
expect(err).toMatch(new RegExp('ubi-runner: reattached to ' + name + ' created_at=\\d{10} key=[0-9a-f]{12}\n'));
expect(b.stdout.toString().split('\n').filter(l => l === 'begin' || l === 'end')).toEqual(['begin', 'end']);
expect(err).toMatch(/command exited 3 after \d+s/);
expect(creates(f)).toBe(1);
expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('setup\n');
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: a second live runner of the same command refuses with UBI_ATTACH_BUSY and creates nothing', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 4; echo done\n');
const args = ['run', '--src', repo, '--', 'make test'];
const a = f.spawn(args);
await until(() => phaseStarted(f, 'command'), 30_000);
const b = f.run(args);
expect(b.exitCode).toBe(1);
expect(b.stderr.toString()).toMatch(new RegExp('UBI_ATTACH_BUSY: vm=' + vmDirs(f)[0] + ' key=[0-9a-f]{12}: runner pid ' + a.pid + ' '));
expect(await a.exited).toBe(0);
expect(await new Response(a.stdout).text()).toContain('done');
expect(creates(f)).toBe(1);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: HUP once the command is live detaches (UBI_DETACHED, exit 129, VM kept, no pass value on disk); a rerun reattaches and finishes', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3; echo finished\n');
const args = ['run', '--src', repo, '--pass', 'UBI_TEST_TOKEN', '--', 'make test'];
const a = f.spawn(args, { UBI_TEST_TOKEN: 'hup-secret-value' });
await until(() => phaseStarted(f, 'command'), 30_000);
await Bun.sleep(500);
process.kill(a.pid, 'SIGHUP');
expect(await a.exited).toBe(129);
const name = vmDirs(f)[0]!;
const err = await new Response(a.stderr).text();
expect(err).toMatch(new RegExp('UBI_DETACHED: vm=' + name + ' key=[0-9a-f]{12} reattach: rerun the same command'));
expect(f.listed()).toContain(name);
expect(f.calls()).not.toContain('destroy');
for (const file of readdirSync(join(f.state, name))) {
const p = join(f.state, name, file);
if (lstatSync(p).isFile()) expect(readFileSync(p, 'utf8')).not.toContain('hup-secret-value');
}
const b = f.run(args, { UBI_TEST_TOKEN: 'hup-secret-value' });
expect(b.exitCode, b.stderr.toString()).toBe(0);
expect(b.stderr.toString()).toContain('reattached to ' + name);
expect(b.stdout.toString()).toContain('finished');
expect(creates(f)).toBe(1);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: INT and TERM during a live command still tear the VM down', async () => {
for (const sig of ['SIGTERM', 'SIGINT'] as const) {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3\n');
const a = f.spawn(['run', '--src', repo, '--', 'make test']);
await until(() => phaseStarted(f, 'command'), 30_000);
await Bun.sleep(500);
process.kill(a.pid, sig);
expect(await a.exited).toBe(130);
expect(await new Response(a.stderr).text()).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-/);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}
}, 90_000);
test('run: a remote session gone without an exit status exits 125 with UBI_ATTACH_LOST and the VM name', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 30\n');
const args = ['run', '--src', repo, '--', 'make test'];
const a = f.spawn(args);
await until(() => existsSync(join(f.vm, '.ubirun-command.pid')) && readFileSync(join(f.vm, '.ubirun-command.pid'), 'utf8').trim() !== '', 30_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
await killSession(f, 'command');
const name = vmDirs(f)[0]!;
const b = f.run(args);
expect(b.exitCode).toBe(125);
expect(b.stderr.toString()).toContain('UBI_ATTACH_LOST: vm=' + name + ' phase=command');
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: the run key covers tree, command, setup content, env, pass value, image, origin/HEAD and the runner itself; a pass value never appears in it', () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo one\n');
const runner = join(f.root, 'runner-copy.sh'); writeFileSync(runner, readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8'));
const key = (opts: { args?: string[]; env?: Record<string, string>; cmd?: string; script?: string } = {}) => {
const r = Bun.spawnSync(['bash', opts.script ?? join(DIR, 'ubi-runner.sh'), 'run', '--probe', '--src', repo, '--setup', setup, '--env', 'MODE=a', '--pass', 'UBI_TEST_TOKEN', ...(opts.args ?? []), '--', opts.cmd ?? 'make test'],
{ env: { ...process.env, PATH: join(f.root, 'bin') + ':' + process.env.PATH, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: f.state, UBI_OWNER: 'thrd1', UBI_TEST_TOKEN: 'v1', ...(opts.env ?? {}) }, timeout: 30_000 });
const out = r.stdout.toString() + r.stderr.toString();
expect(out).not.toContain('v1');
const m = /^UBI_ATTACH_INFO: none key=([0-9a-f]{12})$/m.exec(out);
if (!m) throw new Error(out);
return m[1]!;
};
const base = key();
expect(key()).toBe(base);
const variants: Record<string, string> = {};
variants.command = key({ cmd: 'make check' });
variants.env = key({ args: ['--env', 'MODE=b'] });
variants.pass = key({ env: { UBI_TEST_TOKEN: 'v2' } });
variants.image = key({ args: ['-b', 'ubuntu-jammy'] });
variants.size = key({ args: ['-s', 'standard-8'] });
writeFileSync(runner, readFileSync(runner, 'utf8') + '# edited\n');
variants.runner = key({ script: runner });
writeFileSync(setup, 'echo two\n');
variants.setup = key();
writeFileSync(setup, 'echo one\n');
expect(key()).toBe(base);
Bun.spawnSync(['git', 'update-ref', 'refs/remotes/origin/HEAD', 'HEAD'], { cwd: repo, timeout: 10_000 });
variants.base = key();
writeFileSync(join(repo, 'new-file.txt'), 'untracked source\n');
variants.tree = key();
for (const [what, k] of Object.entries(variants)) expect({ what, same: k === base }).toEqual({ what, same: false });
expect(new Set(Object.values(variants)).size).toBe(Object.keys(variants).length);
expect(f.calls()).not.toContain('"create"');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 60_000);
test('verify tiers: config refuses command with tiers, bad and duplicate tier names, before any API call', () => {
const f = fakeVm();
try {
const cases: Array<[unknown, RegExp]> = [
[{ command: 'make', tiers: [{ name: 'a', command: 'x' }] }, /UBI_CONFIG_TIERS_AND_COMMAND/],
[{ tiers: [{ name: 'Unit', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name 'Unit' must match/],
[{ tiers: [{ name: '-x', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name '-x'/],
[{ tiers: [{ name: 'a'.repeat(33), command: 'x' }] }, /UBI_CONFIG_TIER_NAME/],
[{ tiers: [{ name: 'a.b', command: 'x' }] }, /UBI_CONFIG_TIER_NAME/],
[{ tiers: [{ name: 'a', command: 'x' }, { name: 'a', command: 'y' }] }, /UBI_CONFIG_TIER_NAME: tier name 'a' appears twice/],
[{ tiers: [{ name: 'a' }] }, /UBI_CONFIG_TIER_NAME: tier 'a' needs a command/],
[{ tiers: [] }, /UBI_CONFIG_TIER_NAME: tiers must be a non-empty list/],
];
for (const [cfg, want] of cases) {
const dir = src(f, cfg);
const r = run(f, ['verify', '--src', dir]);
expect(r.exitCode).toBe(1);
expect(r.stderr.toString()).toMatch(want);
}
expect(f.calls()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 30_000);
test('verify tiers: a run killed during tier 3 with its VM gone resumes on a new VM: setup reruns, t1 and t2 are reused, t3 reruns', async () => {
const f = fakeVm();
try {
const dir = gitSrc(f, TIERS);
bootCounts(f, dir);
Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 });
writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n');
const a = f.spawn(['verify', '--src', dir]);
await until(() => existsSync(join(f.root, 't3-started')), 60_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
await killSession(f, 'tier-t3');
const first = vmDirs(f)[0]!;
writeFileSync(join(f.root, 'listed'), '');
writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n');
const b = f.run(['verify', '--src', dir]);
const err = b.stderr.toString(), out = b.stdout.toString();
expect(b.exitCode, err).toBe(0);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse t1\.\d{10}\.\d+\.json \(\d+s\)\n/);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse t2\.\d{10}\.\d+\.json \(\d+s\)\n/);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t3 run: interrupted \(only a start record, t3\.\d{10}\.\d+\.json\)\n/);
expect(err).toContain('run record for ' + first + ': the VM is gone; starting a new one');
expect(err).not.toContain('reattached to');
expect(creates(f)).toBe(2);
expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('boot\nboot\n');
expect(out).not.toContain('ran: make t1');
expect(out).toContain('t3 ok');
expect(verifyLine(out)).toMatch(/^UBI_VERIFY: tree=[0-9a-f]{12} key=[0-9a-f]{12} tiers=3 passed=3 reused=t1,t2 failed=none prior_failures=none saved_s=\d+$/);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 120_000);
test('verify tiers: a fail record blocks a rerun until --rerun-failed (then prior_failures); passes are reused unless stale, tampered, --no-resume or the tree changed', () => {
const f = fakeVm();
try {
const dir = gitSrc(f, TIERS);
writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 broke; exit 7\n');
const first = run(f, ['verify', '--src', dir]);
expect(first.exitCode, first.stderr.toString()).toBe(7);
expect(first.stdout.toString()).not.toContain('ran: make t3');
expect(verifyLine(first.stdout.toString())).toMatch(/ tiers=3 passed=1 reused=none failed=t2 prior_failures=none saved_s=0$/);
expect(creates(f)).toBe(1);
const blocked = run(f, ['verify', '--src', dir]);
expect(blocked.exitCode).toBe(7);
expect(blocked.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 fail t2\.\d{10}\.\d+\.json \(\d+s, exit 7\): not rerun without --rerun-failed/);
expect(verifyLine(blocked.stdout.toString())).toMatch(/ failed=t2 prior_failures=none /);
expect(creates(f)).toBe(1);
writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 fixed\n');
const rerun = run(f, ['verify', '--src', dir, '--rerun-failed']);
expect(rerun.exitCode, rerun.stderr.toString()).toBe(0);
expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /);
expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 run: prior fail t2\.\d{10}\.\d+\.json \(exit 7; --rerun-failed\)/);
expect(verifyLine(rerun.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1 failed=none prior_failures=t2:7 saved_s=\d+$/);
expect(creates(f)).toBe(2);
const all = run(f, ['verify', '--src', dir]);
expect(all.exitCode).toBe(0);
expect(verifyLine(all.stdout.toString())).toMatch(/ passed=3 reused=t1,t2,t3 failed=none prior_failures=none saved_s=\d+$/);
expect(all.stderr.toString()).toContain('every tier passed under this key; no VM needed');
expect(creates(f)).toBe(2);
const probe = (args: string[] = [], extra: Record<string, string> = {}) => run(f, ['verify', '--src', dir, '--probe', ...args], extra).stderr.toString();
expect(probe(['--no-resume'])).toMatch(/tier=t1 run: --no-resume\n[^]*tier=t2 run: --no-resume\n[^]*tier=t3 run: --no-resume\n/);
expect(probe([], { UBI_CHECKPOINT_MAX_AGE: '0' })).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json is \d+s old \(UBI_CHECKPOINT_MAX_AGE=0\)/);
const keyDir = join(f.checkpoints, readdirSync(f.checkpoints)[0]!);
const t1Log = readdirSync(keyDir).filter(n => n.startsWith('t1.') && n.endsWith('.log')).sort().at(-1)!;
writeFileSync(join(keyDir, t1Log), 'tampered\n');
const tampered = probe();
expect(tampered).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json: its log does not re-hash/);
expect(tampered).toMatch(/tier=t2 reuse /);
writeFileSync(join(dir, 'new-source.ts'), 'export {}\n');
const moved = probe();
for (const t of ['t1', 't2', 't3']) expect(moved).toContain('UBI_VERIFY_RESUME: tier=' + t + ' run: no record');
expect(creates(f)).toBe(2);
// Records are write-once JSON; every one carries its generation.
const gens = readdirSync(keyDir).filter(n => n.endsWith('.json')).map(n => JSON.parse(readFileSync(join(keyDir, n), 'utf8')).generation as number).sort((x, y) => x - y);
expect(gens).toEqual(gens.map((_, i) => i + 1));
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 120_000);
test('verify tiers: a reattach whose VM TTL no longer covers the run left plus UBI_REATTACH_MARGIN starts a fresh VM, destroys the old one and reuses the passed tiers', async () => {
const f = fakeVm();
try {
const dir = gitSrc(f, TIERS);
bootCounts(f, dir);
Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 });
writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n');
const env = { UBI_REAP: '0', UBI_TTL_HOURS: '1' };
const a = f.spawn(['verify', '--src', dir], env);
await until(() => existsSync(join(f.root, 't3-started')), 60_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
const first = vmDirs(f)[0]!;
expect(first).toMatch(/-t1$/);
const probe = (extra: Record<string, string>) => run(f, ['verify', '--src', dir, '--probe'], { ...env, ...extra }).stdout.toString();
expect(probe({})).toMatch(new RegExp('UBI_ATTACH_INFO: vm=' + first + ' created_at=\\d{10} '));
// A 1 h TTL leaves about 3600 s: t3's 1200 s run timeout plus a 2700 s margin does not fit.
const short = { UBI_RUN_TIMEOUT: '1200', UBI_REATTACH_MARGIN: '2700' };
expect(probe(short)).toMatch(new RegExp('UBI_ATTACH_INFO: none key=[0-9a-f]{12} ttl_short vm=' + first + ' ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\)'));
expect(creates(f)).toBe(1);
writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n');
const b = run(f, ['verify', '--src', dir], { ...env, ...short });
const err = b.stderr.toString();
expect(b.exitCode, err).toBe(0);
expect(err).toMatch(new RegExp('UBI_REATTACH_TTL_SHORT: vm=' + first + ' key=[0-9a-f]{12} ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\): starting a fresh VM \\(tier checkpoints carry over\\) and destroying ' + first));
expect(err).not.toContain('reattached to');
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse /);
expect(verifyLine(b.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1,t2 failed=none /);
expect(creates(f)).toBe(2);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 120_000);
test('verify tiers: a writer whose start record is no longer the newest publishes nothing (stale writer)', () => {
const py = /\nCKPT_PY='\n([^']*)'\n/.exec(readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8'))![1]!;
const root = mkdtempSync(join(tmpdir(), 'ubi-ckpt-'));
try {
const d = join(root, 'key');
const log = join(root, 'tier.log'); writeFileSync(log, 'ok\n');
const publish = (status: string, startgen: string, pid: string) => Bun.spawnSync(['python3', '-c', py, 'publish', d, 't1', status, 'vm', '100', '200', '0', status === 'start' ? '' : log, startgen, 'k', 'tree', pid], { timeout: 30_000 });
expect(publish('start', '0', '1').stdout.toString().trim()).toBe('1');
expect(publish('start', '0', '2').stdout.toString().trim()).toBe('2');
const stale = publish('pass', '1', '1');
expect(stale.exitCode).toBe(2);
expect(stale.stderr.toString()).toContain('stale writer: its start record is generation 1, the newest for t1 is 2');
expect(readdirSync(d).filter(n => n.endsWith('.json')).length).toBe(2);
expect(publish('pass', '2', '2').stdout.toString().trim()).toBe('3');
} finally { rmSync(root, { recursive: true, force: true }); }
});
test('verify tiers: a tier that edits the source gets no pass record (UBI_VERIFY_TREE_MOVED) and reruns next time', () => {
const f = fakeVm();
try {
const dir = gitSrc(f, { tiers: [{ name: 'mover', command: 'make mover' }, { name: 'clean', command: 'make clean' }] });
writeFileSync(join(f.root, 'make-mover.sh'), 'echo edited > moved.txt\n');
const r = run(f, ['verify', '--src', dir]);
expect(r.exitCode, r.stderr.toString()).toBe(0);
expect(r.stderr.toString()).toMatch(/UBI_VERIFY_TREE_MOVED: tier=mover before=[0-9a-f]{12} after=[0-9a-f]{12}: a tier must leave the source tree as it found it; no pass record/);
expect(verifyLine(r.stdout.toString())).toMatch(/ tiers=2 passed=2 reused=none failed=none /);
const next = run(f, ['verify', '--src', dir, '--probe']).stderr.toString();
expect(next).toMatch(/tier=mover run: interrupted \(only a start record/);
expect(next).toMatch(/tier=clean reuse /);
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
});
+15 -5
View File
@@ -237,6 +237,10 @@ and for browser tests `xvfb` plus
`bun run test:ubicloud` (the free suite with the CI lane's environment) and
`gstack-pregate --stage tests --remote` under 8 cores.
A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]`
for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a
fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`.
---
## Step 4: Read the result
@@ -266,9 +270,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown.
## Resume checklist after a wake or a machine swap
1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is
still running without a live runner.
2. A background log without its exit sentinel after the VM is gone is an
interrupted run, not a pass; rerun it.
3. Measure before/after pairs on the same size and location, never one side
1. Rerun the same command first: a run whose runner died or detached
(`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a
finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125
`UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over).
2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no
rerun will reattach to.
3. After a machine swap nothing reattaches; tier checkpoints on a Drive
(`UBI_CHECKPOINT_DIR`) still count.
4. A background log with no exit sentinel and no VM is an interrupted run, not
a pass; rerun it.
5. Measure before/after pairs on the same size and location, never one side
local and the other remote.
+15 -5
View File
@@ -163,6 +163,10 @@ and for browser tests `xvfb` plus
`bun run test:ubicloud` (the free suite with the CI lane's environment) and
`gstack-pregate --stage tests --remote` under 8 cores.
A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]`
for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a
fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`.
---
## Step 4: Read the result
@@ -192,9 +196,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown.
## Resume checklist after a wake or a machine swap
1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is
still running without a live runner.
2. A background log without its exit sentinel after the VM is gone is an
interrupted run, not a pass; rerun it.
3. Measure before/after pairs on the same size and location, never one side
1. Rerun the same command first: a run whose runner died or detached
(`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a
finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125
`UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over).
2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no
rerun will reattach to.
3. After a machine swap nothing reattaches; tier checkpoints on a Drive
(`UBI_CHECKPOINT_DIR`) still count.
4. A background log with no exit sentinel and no VM is an interrupted run, not
a pass; rerun it.
5. Measure before/after pairs on the same size and location, never one side
local and the other remote.