mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 22:45:02 +01:00
v1.91.132.0 feat(pregate): a remote gate for gstack's own pregate (Ubicloud or keyless local shards); concurrent next-version fetches (#3258)
* feat(test-free): --files-from runs exactly the listed planned files Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(pregate): declare scripts/ubicloud/pregate-remote.sh as gstack's remote gate (keyed Ubicloud VM, keyless local shards) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * perf(next-version): fetch PR VERSIONs concurrently; a timed-out fetch is an unknown claim Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(pregate): gstack remote-gate example and the small-machine pregate row Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): remote gate leaves paid test files to the paid gate, adds reason codes and the keyless backend line Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(next-version): unknown claims make the pick provisional (complete: false) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(pregate): remote-gate reason codes, paid-file handling and measured keyed/keyless timing Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): remote gate finishes before pregate's deadline pregate kills the remote gate at --timeout with a 5 s grace, which could cut a VM teardown short. pregate-remote.sh now reads the enclosing deadline (PREGATE_TIMEOUT_S, else the gstack-pregate --timeout it runs under, else 600), reserves 120 s for teardown plus 15 s margin, and tells the VM to stop at the earlier of that point and the cap's wall; a deadline that leaves under 300 s refuses before any VM (PREGATE_REMOTE_TIMEOUT_TOO_SHORT). It names the VM up front and prints the exact ubi-runner.sh down line, and the body is one braced block so an edit to the script mid-run cannot change what the run executes (keyed run 5 died after teardown with '$2: unbound variable' for that reason). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(restamp): an incomplete claim queue is never reported as read gstack-next-version now returns complete: false with unknown_claims when a PR's VERSION fetch timed out or failed. restamp's claim-at-open path took any usable JSON as 'read'; it now says 'incomplete', names the unknown PRs and calls the version provisional, the same warn-not-refuse stance as the existing 'unverified' fallback. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): only a gstack-pregate process sets the remote gate's deadline The ancestor walk matched any command line that mentioned gstack-pregate, so a shell line running pregate beside the tests (bash -c '...; bin/gstack-pregate ...') gave the stub gate a 600 s deadline. It now matches only a process whose program, or the script its interpreter runs, is gstack-pregate. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): A3 remote gate, --files-from, concurrent next-version Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.132.0 chore: stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
8f103baa71
commit
a59734fed6
16 files changed
+1186
-23
No files matched your search
@@ -1,6 +1,7 @@
|
||||
{
|
||||
"$schema": "https://github.com/garrytan/gstack/docs/pregate.md#pregate-v1",
|
||||
"runner": "bun test {files}",
|
||||
"remote_gate": "bash scripts/ubicloud/pregate-remote.sh",
|
||||
"dependencies": {
|
||||
".github/workflows/**": ["test/pregate-selection.test.ts", "test/free-tests-workflow-wiring.test.ts"],
|
||||
"scripts/gen-ship-goldens.ts": ["test/host-config.test.ts"],
|
||||
|
||||
@@ -1,5 +1,26 @@
|
||||
# Changelog
|
||||
|
||||
## [1.91.132.0] - 2026-10-18
|
||||
|
||||
**On a machine under 8 cores, `gstack-pregate` clears its test tier on one Ubicloud VM in about 7 minutes for about $0.06, instead of running locally for an hour.** gstack's own `.gstack/pregate.json` now declares a remote gate, so a small machine records the local test tier as a `requires-remote` obligation and `gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands` clears it. All 1,280 free test files ran in 404 s (create 24 s, setup 72 s, tests 250 s) for $0.06; the same 798-file selection took 3,550 s locally on 4 cores. `gstack-next-version` also fetches open PRs' VERSION claims in parallel (8 PRs: 4,160 ms before, about 0.5 s now) and says so when it could not read one.
|
||||
|
||||
### Added
|
||||
- **`scripts/ubicloud/pregate-remote.sh`, gstack's remote gate.** With `UBICLOUD_API_KEY` it runs the selected files on one VM with `bun run test:ubicloud`'s environment, checks the tree and the planned file set on the VM before any test runs, and reports `PREGATE_REMOTE: tree=… tests=… failed=… spent=…` from billed minutes. Without a key it runs the same files locally in parallel shards and prints `backend=local (N cores; not the CI environment)`.
|
||||
- The spend cap buys the wall time at the size's list price (at most 45 minutes); a cap that cannot buy 20 minutes refuses before any VM exists.
|
||||
- It finishes before pregate's own `--timeout`: 120 s are reserved for teardown, and a deadline too short to be useful refuses up front.
|
||||
- At create time it prints the VM name and the exact `bash scripts/ubicloud/ubi-runner.sh down <name>` line.
|
||||
- Paid test files in a selection are named and left to the paid PR gate; a paid-only selection reports `tests=0`, which never clears.
|
||||
- Every refusal starts with a `PREGATE_REMOTE_*` reason code; the table is in docs/pregate.md#remote-gate.
|
||||
- **`test-free-shards.ts --files-from <path>`** plans exactly the listed files. Unknown paths, an empty list and malformed or repeated lines are rejected before planning.
|
||||
|
||||
### Changed
|
||||
- **`gstack-next-version` fetches claims concurrently** (up to 10, 15 s each) and sorts its output by PR number. A fetch that times out or fails is listed in `unknown_claims`, the JSON says `complete: false`, and the pick is marked provisional on stdout and stderr. The fork/path hint appears only for HTTP 404.
|
||||
- **`gstack-restamp` in claim-at-open mode reports an incomplete claim queue as `claims: incomplete`**, naming the unknown PRs, instead of `read`.
|
||||
|
||||
### If you…
|
||||
- **…run `gstack-pregate` on a machine under 8 cores in this repo:** clear the test tier with `gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands`. Keyless works too but is practical only for small selections (98 files took 103 s on 4 cores).
|
||||
- **…see `complete: false` from `gstack-next-version`:** rerun it before stamping; the slot may collide.
|
||||
|
||||
## [1.91.131.0] - 2026-10-18
|
||||
|
||||
**`gstack-pregate` finds the tests for Python files, and a missing `node_modules` no longer reads as "stale generated files".** In a Python repo, the `lanes` check failed almost every touched `.py` file with `PREGATE_NO_LANE`, because the import graph only parsed JavaScript and TypeScript. In a FastAPI service with about 1,500 non-test modules, Scott Gigante's change (#3201) cut the unmapped count from 1,473 to 780, and most of the rest are migrations and scripts no import reaches. Separately, a fresh checkout without `node_modules` made the pre-gate report `REGEN_STALE` and suggest rerunning the generator, which sent people the wrong way.
|
||||
|
||||
@@ -187,6 +187,7 @@ the new defaults.
|
||||
| Run one free test file while repairing | `bun test <file>` | Free, seconds | Never bare `bun test` for the suite |
|
||||
| Run full free acceptance before publishing | `bun run test` | Free, a few minutes | Bun 1.4.2 |
|
||||
| Run the full free suite from a small machine | `bun run test:ubicloud` | Free suite on a billed 16-vCPU VM, about 5 minutes | `UBICLOUD_API_KEY` |
|
||||
| Clear pregate's test tier on a machine under 8 cores ([remote gate](docs/pregate.md#remote-gate)) | `gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands` (same command keyed or keyless) | Keyed: one billed 16-vCPU VM, about 7 minutes and $0.06 for the whole free tier, worst case $0.38; keyless: free, local parallel shards, practical for small selections only (98 files took 103 s on 4 cores) | Keyed: `UBICLOUD_API_KEY`; keyless: nothing |
|
||||
| Run the curated Windows-safe subset | `bun run test:windows` (CI: six `windows-latest` jobs) | Free | Windows, Git Bash |
|
||||
| Refresh the Windows duration seed | `gh workflow run windows-free-tests.yml --ref <branch> -f record_durations=true`, then commit the `free-test-durations-windows` artifact | Free CI runners | Pushed branch, `gh` with workflow rights |
|
||||
| Run a native Windows or Dia qualification campaign | `gh workflow run native-qualification.yml --ref <branch> -f mode=<mode>` ([modes](docs/TESTING_INTERNALS.md#windows-free-lane)) | Free CI runners | Pushed branch, `gh` with workflow rights |
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.91.131.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.91.132.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
+76
-15
@@ -6,7 +6,10 @@
|
||||
// slot at the requested bump level, and emits the whole picture as JSON.
|
||||
//
|
||||
// Contract: util NEVER writes files or mutates state. Pure reader + reporter.
|
||||
// /ship consumes the JSON and decides what to do.
|
||||
// /ship consumes the JSON and decides what to do. `unknown_claims` lists the
|
||||
// open PRs whose VERSION could not be read (a 15 s timeout or a non-404
|
||||
// failure): their claims are unknown, so the picked slot may collide, and
|
||||
// `complete` is false (a provisional pick, also said on stderr and in warnings).
|
||||
//
|
||||
// Usage:
|
||||
// gstack-next-version --base <branch> --bump <major|minor|patch|micro> \
|
||||
@@ -34,7 +37,7 @@
|
||||
// empty, unreadable or malformed
|
||||
// 3 — util bug (unexpected exception)
|
||||
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import { execFileSync, spawn, spawnSync } from "node:child_process";
|
||||
import { existsSync, readFileSync, readdirSync, statSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
@@ -84,10 +87,13 @@ type Output = {
|
||||
active_siblings: Sibling[];
|
||||
reason: string;
|
||||
warnings: string[];
|
||||
unknown_claims: number[];
|
||||
complete: boolean;
|
||||
};
|
||||
|
||||
const ACTIVE_SIBLING_MAX_AGE_S = 24 * 60 * 60;
|
||||
const GH_API_CONCURRENCY = 10;
|
||||
const VERSION_FETCH_TIMEOUT_MS = 15000;
|
||||
|
||||
// Collision resolution: bump past the highest claimed within the same level.
|
||||
// Semantics: if my bump is MINOR and the queue claims 1.7.0.0, I advance to
|
||||
@@ -117,6 +123,33 @@ function runCommand(cmd: string, args: string[], timeoutMs = 15000): { ok: boole
|
||||
};
|
||||
}
|
||||
|
||||
// Async twin of runCommand for the per-PR VERSION fetches, so GH_API_CONCURRENCY
|
||||
// workers really overlap. A fetch past timeoutMs is killed and settles at once
|
||||
// as timedOut, without waiting on output pipes a descendant may still hold.
|
||||
function runCommandAsync(cmd: string, args: string[], timeoutMs: number): Promise<{ ok: boolean; stdout: string; stderr: string; timedOut: boolean }> {
|
||||
return new Promise((done) => {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const child = spawn(cmd, args, { stdio: ["ignore", "pipe", "pipe"] });
|
||||
const timer = setTimeout(() => {
|
||||
child.kill("SIGKILL");
|
||||
child.stdout.destroy();
|
||||
child.stderr.destroy();
|
||||
done({ ok: false, stdout, stderr, timedOut: true });
|
||||
}, timeoutMs);
|
||||
child.stdout.on("data", (chunk) => { stdout += chunk; });
|
||||
child.stderr.on("data", (chunk) => { stderr += chunk; });
|
||||
child.on("error", (error) => {
|
||||
clearTimeout(timer);
|
||||
done({ ok: false, stdout, stderr: stderr || String(error), timedOut: false });
|
||||
});
|
||||
child.on("close", (code) => {
|
||||
clearTimeout(timer);
|
||||
done({ ok: code === 0, stdout, stderr, timedOut: false });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// VERSION-path resolution for monorepos. Priority: CLI flag > .gstack/version-path
|
||||
// at repo root > "VERSION". Pure function; takes the repo root as an argument so
|
||||
// tests can drive it with a fixture dir without mocking git.
|
||||
@@ -181,7 +214,13 @@ function readBaseVersion(base: string, versionPath: string, repoRoot: string, wa
|
||||
return v;
|
||||
}
|
||||
|
||||
async function fetchGithubClaimed(base: string, versionPath: string, excludePR: number | null, warnings: string[]): Promise<{ claimed: ClaimedPR[]; offline: boolean }> {
|
||||
async function fetchGithubClaimed(
|
||||
base: string,
|
||||
versionPath: string,
|
||||
excludePR: number | null,
|
||||
warnings: string[],
|
||||
fetchTimeoutMs = VERSION_FETCH_TIMEOUT_MS,
|
||||
): Promise<{ claimed: ClaimedPR[]; offline: boolean; unknownClaims: number[] }> {
|
||||
const list = runCommand("gh", [
|
||||
"pr",
|
||||
"list",
|
||||
@@ -196,7 +235,7 @@ async function fetchGithubClaimed(base: string, versionPath: string, excludePR:
|
||||
]);
|
||||
if (!list.ok) {
|
||||
warnings.push(`gh pr list failed: ${list.stderr.trim().slice(0, 200)}`);
|
||||
return { claimed: [], offline: true };
|
||||
return { claimed: [], offline: true, unknownClaims: [] };
|
||||
}
|
||||
let prs: {
|
||||
number: number;
|
||||
@@ -209,7 +248,7 @@ async function fetchGithubClaimed(base: string, versionPath: string, excludePR:
|
||||
prs = JSON.parse(list.stdout);
|
||||
} catch (e) {
|
||||
warnings.push(`gh pr list returned invalid JSON`);
|
||||
return { claimed: [], offline: true };
|
||||
return { claimed: [], offline: true, unknownClaims: [] };
|
||||
}
|
||||
// Determine our repo owner to filter out fork PRs. `gh api contents?ref=<branch>`
|
||||
// resolves to OUR repo regardless of where the PR originated, so fork PRs would
|
||||
@@ -221,43 +260,56 @@ async function fetchGithubClaimed(base: string, versionPath: string, excludePR:
|
||||
: prs
|
||||
).filter((p) => excludePR === null || p.number !== excludePR);
|
||||
// Fetch each PR's VERSION at its head in parallel (bounded concurrency).
|
||||
// Workers finish in any order, so claims and warnings are sorted by PR number.
|
||||
const results: ClaimedPR[] = [];
|
||||
const prWarnings: { pr: number; text: string }[] = [];
|
||||
const unknownClaims: number[] = [];
|
||||
const queue = [...sameRepoPRs];
|
||||
const workers = Array.from({ length: Math.min(GH_API_CONCURRENCY, sameRepoPRs.length) }, async () => {
|
||||
while (queue.length) {
|
||||
const pr = queue.shift();
|
||||
if (!pr) return;
|
||||
const warn = (text: string) => prWarnings.push({ pr: pr.number, text: `PR #${pr.number}: ${text}` });
|
||||
// gh passes branch name via argv, not shell — safe.
|
||||
// encodeURI handles spaces in subproject paths (e.g. "Tinas Second Brain/...")
|
||||
// while leaving "/" untouched so the GitHub Contents API gets the path intact.
|
||||
const content = runCommand("gh", [
|
||||
const content = await runCommandAsync("gh", [
|
||||
"api",
|
||||
`repos/{owner}/{repo}/contents/${encodeURI(versionPath)}?ref=${encodeURIComponent(pr.headRefName)}`,
|
||||
"-q",
|
||||
".content",
|
||||
]);
|
||||
], fetchTimeoutMs);
|
||||
if (content.timedOut) {
|
||||
unknownClaims.push(pr.number);
|
||||
warn(`VERSION fetch timed out after ${fetchTimeoutMs / 1000} s; claim unknown, the slot may collide; rerun, then ask the coordinator`);
|
||||
continue;
|
||||
}
|
||||
if (!content.ok) {
|
||||
warnings.push(
|
||||
`PR #${pr.number}: could not fetch ${versionPath} (fork, private, or wrong path — try --version-path or .gstack/version-path)`,
|
||||
);
|
||||
if (/HTTP 404|Not Found/i.test(content.stderr)) {
|
||||
warn(`could not fetch ${versionPath} (fork, private, or wrong path — try --version-path or .gstack/version-path)`);
|
||||
} else {
|
||||
unknownClaims.push(pr.number);
|
||||
warn(`VERSION fetch failed (${content.stderr.trim().split("\n")[0]?.slice(0, 200) || "no error output"}); claim unknown, the slot may collide; rerun, then ask the coordinator`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let versionStr: string;
|
||||
try {
|
||||
versionStr = extractVersion(Buffer.from(content.stdout.trim(), "base64").toString("utf8"), versionPath);
|
||||
} catch {
|
||||
warnings.push(`PR #${pr.number}: VERSION is not valid base64`);
|
||||
warn("VERSION is not valid base64");
|
||||
continue;
|
||||
}
|
||||
if (!parseVersion(versionStr)) {
|
||||
warnings.push(`PR #${pr.number}: VERSION is malformed (${versionStr})`);
|
||||
warn(`VERSION is malformed (${versionStr})`);
|
||||
continue;
|
||||
}
|
||||
results.push({ pr: pr.number, branch: pr.headRefName, version: versionStr, url: pr.url });
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
return { claimed: results, offline: false };
|
||||
warnings.push(...prWarnings.sort((a, b) => a.pr - b.pr).map((w) => w.text));
|
||||
return { claimed: results.sort((a, b) => a.pr - b.pr), offline: false, unknownClaims: unknownClaims.sort((a, b) => a - b) };
|
||||
}
|
||||
|
||||
async function fetchGitlabClaimed(base: string, versionPath: string, excludePR: number | null, warnings: string[]): Promise<{ claimed: ClaimedPR[]; offline: boolean }> {
|
||||
@@ -730,8 +782,9 @@ async function main() {
|
||||
|
||||
let claimed: ClaimedPR[] = [];
|
||||
let offline = false;
|
||||
let unknownClaims: number[] = [];
|
||||
if (host === "github") {
|
||||
({ claimed, offline } = await fetchGithubClaimed(args.base, versionPath, excludePR, warnings));
|
||||
({ claimed, offline, unknownClaims } = await fetchGithubClaimed(args.base, versionPath, excludePR, warnings));
|
||||
} else if (host === "gitlab") {
|
||||
({ claimed, offline } = await fetchGitlabClaimed(args.base, versionPath, excludePR, warnings));
|
||||
} else {
|
||||
@@ -789,6 +842,12 @@ async function main() {
|
||||
finalReason = `bumped past active sibling ${fmtVersion(highest, width)}`;
|
||||
}
|
||||
|
||||
const complete = unknownClaims.length === 0;
|
||||
if (!complete) {
|
||||
const provisional = `provisional pick: the VERSION claims of PR ${unknownClaims.map((n) => `#${n}`).join(", ")} are unknown, so ${fmtVersion(finalVersion, width)} is not a verified slot and may collide; rerun, then ask the coordinator before stamping`;
|
||||
warnings.push(provisional);
|
||||
console.error(`gstack-next-version: ${provisional}`);
|
||||
}
|
||||
const out: Output = {
|
||||
version: fmtVersion(finalVersion, width),
|
||||
current_version: args.current || baseVersion,
|
||||
@@ -803,6 +862,8 @@ async function main() {
|
||||
active_siblings: activeSiblings,
|
||||
reason: finalReason,
|
||||
warnings,
|
||||
unknown_claims: unknownClaims,
|
||||
complete,
|
||||
};
|
||||
process.stdout.write(JSON.stringify(out, null, 2) + "\n");
|
||||
}
|
||||
@@ -811,7 +872,7 @@ async function main() {
|
||||
// from lib/version-source so existing importers of this module keep working
|
||||
// unchanged.
|
||||
export { parseVersion, fmtVersion, bumpVersion, cmpVersion, versionWidth, extractVersion };
|
||||
export { pickNextSlot, markActiveSiblings, resolveVersionPath, fetchGitClaimed };
|
||||
export { pickNextSlot, markActiveSiblings, resolveVersionPath, fetchGitClaimed, fetchGithubClaimed, VERSION_FETCH_TIMEOUT_MS };
|
||||
|
||||
// Only run main() when invoked as a script, not when imported by tests.
|
||||
if (import.meta.main) {
|
||||
|
||||
@@ -240,6 +240,62 @@ the supervision above. Close the obligation one of two ways:
|
||||
`spent, reserved, unknown`: a gate that prints no `spent=` is `unknown`,
|
||||
counted at the cap.
|
||||
|
||||
gstack's own example: `.gstack/pregate.json` declares
|
||||
`"remote_gate": "bash scripts/ubicloud/pregate-remote.sh"`. With
|
||||
`UBICLOUD_API_KEY` set the script runs the selected files on one Ubicloud VM
|
||||
with `bun run test:ubicloud`'s environment, refuses before creating the VM when
|
||||
the cap cannot buy a 20-minute run at the size's list price (`UBI_SIZE`,
|
||||
default `standard-16`), checks the tree and the planned file set on the VM
|
||||
before any test runs, and reports `spent=` from billed minutes. Without a key
|
||||
it runs the same files locally in parallel shards
|
||||
(`test-free-shards.ts --files-from`) and reports `spent=0`, after a
|
||||
`backend=local (N cores; not the CI environment)` line. Paid test files in the
|
||||
selection (skill-e2e and the other paid globs in `test/helpers/paid-test-set.ts`)
|
||||
are named and left to the paid PR gate; a selection of only paid files reports
|
||||
`tests=0`, which never clears the obligation. The command is the same either way
|
||||
(`--remote` always needs a cap; the keyless path spends nothing).
|
||||
|
||||
pregate kills the gate at its `--timeout` (default 600 s) with a 5 s grace, so
|
||||
the script finishes before that deadline: it reads the deadline from
|
||||
`PREGATE_TIMEOUT_S`, else from the `gstack-pregate` command it runs under (its
|
||||
`--timeout`, else 600), reserves 120 s for teardown plus 15 s margin, and tells
|
||||
the VM to stop at whichever comes first, that point or the cap's wall (at most
|
||||
45 minutes, minus 3 for teardown). A deadline that leaves the VM under 300 s
|
||||
refuses before any VM (`PREGATE_REMOTE_TIMEOUT_TOO_SHORT`). At create time it
|
||||
prints the VM's name and the exact
|
||||
`bash scripts/ubicloud/ubi-runner.sh down <name>` line to destroy it if the
|
||||
run is killed before its teardown line. The whole script is parsed before it
|
||||
runs, so editing it during a run changes nothing. `--timeout 2700` gives a
|
||||
whole-tier run room inside the 45-minute wall:
|
||||
|
||||
```bash
|
||||
gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands
|
||||
```
|
||||
|
||||
Measured (October 2026): keyed, all 1,280 free files on standard-16 took 404 s
|
||||
from start to teardown (create 24 s, setup 72 s, tests 250 s) for $0.06.
|
||||
Keyless on 4 cores, 98 files took 103 s, and GSTA-35's 798-file selection took
|
||||
3,550 s, so keyless is practical for small selections only. A large selection
|
||||
on a small machine needs a key, or a `--timeout` longer than the run. Keyless also runs in
|
||||
your environment, not CI's: a file that needs what setup-free-suite.sh installs
|
||||
(Playwright's Chromium, Xvfb) fails locally until you install it.
|
||||
|
||||
Every refusal line starts with a reason code:
|
||||
|
||||
| code | exit | meaning |
|
||||
|---|---|---|
|
||||
| `PREGATE_REMOTE_USAGE` | 2 | an input is missing or malformed; run it through `gstack-pregate --stage tests --remote` |
|
||||
| `PREGATE_REMOTE_TREE_MISMATCH` | 3 | the working tree is not `PREGATE_TREE`; nothing uploaded, nothing spent |
|
||||
| `PREGATE_REMOTE_TREE_MOVED` | 3 | the tree changed during the run (the moved paths are listed); no result reported |
|
||||
| `PREGATE_REMOTE_VM_TREE` | 3 | the VM's unpacked tree or planned file set differs from what was checked locally |
|
||||
| `PREGATE_REMOTE_NO_RESULT` | 1 | test-free-shards printed no result (a rejected file list) |
|
||||
| `PREGATE_REMOTE_PAID_ONLY` | 0 | every selected file is a paid test file; `tests=0` cannot clear |
|
||||
| `PREGATE_REMOTE_CAP_TOO_LOW` | 5 | the cap cannot buy a 20-minute run at the size's price; nothing created |
|
||||
| `PREGATE_REMOTE_NO_PRICE` | 5 | `UBI_SIZE` has no list price in the script |
|
||||
| `PREGATE_REMOTE_TIMEOUT_TOO_SHORT` | 5 | pregate's deadline leaves the VM under 300 s after the 135 s teardown reserve; nothing created |
|
||||
| `PREGATE_REMOTE_INFRA` | 4 | create, setup, SSH or the stop deadline failed; the VM, its teardown line and `spent=` follow |
|
||||
| `PREGATE_REMOTE_INTERRUPTED` | 130 | TERM or INT; ubi-runner tears the VM down on its own |
|
||||
|
||||
<a id="guards"></a>
|
||||
## Guards and the lane-done rule
|
||||
|
||||
|
||||
+5
-1
@@ -183,7 +183,11 @@ function planVersions(o: RestampOptions, loaded: LoadedPolicy, policyLabel: stri
|
||||
try { q = JSON.parse(r.stdout ?? ''); } catch {}
|
||||
const usable = r.status === 0 && q && typeof q.version === 'string' && parseVersion(q.version) && (q.offline === false || q.fallback === 'git');
|
||||
newVersion = usable ? q.version : bumped(atBase.version, level);
|
||||
claims = usable ? `read (claim-at-open; queue ${JSON.stringify(q.claimed ?? q.queue ?? [])})` : 'unverified (claim-at-open; gstack-next-version unusable, local arithmetic)';
|
||||
const queue = JSON.stringify(q?.claimed ?? q?.queue ?? []);
|
||||
const unknown = (Array.isArray(q?.unknown_claims) ? q.unknown_claims : []).map((n: unknown) => `#${n}`).join(', ') || 'some open PRs';
|
||||
claims = !usable ? 'unverified (claim-at-open; gstack-next-version unusable, local arithmetic)'
|
||||
: q.complete === false ? `incomplete (claim-at-open; the VERSION claims of PR ${unknown} are unknown, so ${q.version} is provisional and may collide; rerun, then ask the coordinator before stamping; queue ${queue})`
|
||||
: `read (claim-at-open; queue ${queue})`;
|
||||
}
|
||||
// The branch's own tentative stamp: its VERSION when it moved past base; else nothing (mentions come from the lines it added).
|
||||
const branchStamp = oldVersion !== atBase.version ? oldVersion : null;
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.91.131",
|
||||
"version": "1.91.132",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -73,6 +73,7 @@
|
||||
* bun run scripts/test-free-shards.ts --wall-timeout 600 # override the kill deadline
|
||||
* bun run scripts/test-free-shards.ts --verbose # forward the full child stream
|
||||
* bun run scripts/test-free-shards.ts --quick # explicit fast subset, not acceptance
|
||||
* bun run scripts/test-free-shards.ts --files-from list.txt # exactly these planned files (one per line)
|
||||
* bun run scripts/test-free-shards.ts --attribute-home # name real-home writers: each file alone, private HOME
|
||||
* bun run scripts/test-free-shards.ts --ci-plan plan.json --shards 20
|
||||
* bun run scripts/test-free-shards.ts --ci-run plan.json --shard 1 --result result.json
|
||||
@@ -540,6 +541,7 @@ type CliOptions = {
|
||||
ciVerify: string | null;
|
||||
result: string | null;
|
||||
results: string | null;
|
||||
filesFrom: string | null;
|
||||
};
|
||||
|
||||
export function parseCliOptions(argv: string[]): CliOptions {
|
||||
@@ -554,8 +556,8 @@ export function parseCliOptions(argv: string[]): CliOptions {
|
||||
let wallTimeoutExplicit = false;
|
||||
let quick = false;
|
||||
let attributeHome = false;
|
||||
const paths: Record<'ciPlan' | 'ciRun' | 'ciVerify' | 'result' | 'results', string | null> = {
|
||||
ciPlan: null, ciRun: null, ciVerify: null, result: null, results: null,
|
||||
const paths: Record<'ciPlan' | 'ciRun' | 'ciVerify' | 'result' | 'results' | 'filesFrom', string | null> = {
|
||||
ciPlan: null, ciRun: null, ciVerify: null, result: null, results: null, filesFrom: null,
|
||||
};
|
||||
const pathFlag = (flag: string, key: keyof typeof paths) => (next: () => string | undefined) => {
|
||||
const value = next();
|
||||
@@ -576,6 +578,7 @@ export function parseCliOptions(argv: string[]): CliOptions {
|
||||
'--ci-verify': pathFlag('--ci-verify', 'ciVerify'),
|
||||
'--result': pathFlag('--result', 'result'),
|
||||
'--results': pathFlag('--results', 'results'),
|
||||
'--files-from': pathFlag('--files-from', 'filesFrom'),
|
||||
'--shards': (next) => {
|
||||
const value = next();
|
||||
if (!value) throw new Error('Missing value for --shards');
|
||||
@@ -595,6 +598,7 @@ export function parseCliOptions(argv: string[]): CliOptions {
|
||||
});
|
||||
|
||||
const ciModes = [paths.ciPlan, paths.ciRun, paths.ciVerify].filter(Boolean).length;
|
||||
if (paths.filesFrom && (quick || windowsOnly || listOnly || ciModes || recordDurations || attributeHome)) throw new Error('--files-from cannot be combined with --quick, --windows-only, --list, --record-durations, --attribute-home or the --ci-* modes');
|
||||
if (ciModes > 1 || (ciModes && (quick || listOnly || dryRun || recordDurations || attributeHome))) throw new Error('CI modes cannot be combined with other selection modes');
|
||||
if (paths.ciRun && (shardIndex === null || !paths.result)) throw new Error('--ci-run requires --shard and --result');
|
||||
if (paths.ciVerify && !paths.results) throw new Error('--ci-verify requires --results');
|
||||
@@ -602,6 +606,47 @@ export function parseCliOptions(argv: string[]): CliOptions {
|
||||
return { dryRun, listOnly, recordDurations, windowsOnly, verbose, shardCount, shardIndex, wallTimeoutMs, wallTimeoutExplicit, quick, attributeHome, ...paths };
|
||||
}
|
||||
|
||||
const describeLines = (lines: string[]) => `${lines.slice(0, 5).join(', ')}${lines.length > 5 ? ` (+${lines.length - 5} more)` : ''}`;
|
||||
|
||||
/**
|
||||
* `--files-from <path>`: one repo-relative path per line (LF or CRLF), blank
|
||||
* lines ignored. Rejected before any planning: an unreadable file, a malformed
|
||||
* or repeated line, an empty list, or a path outside the free planned set.
|
||||
*/
|
||||
export function readFilesFrom(file: string, planned: readonly string[]): string[] {
|
||||
let text: string;
|
||||
try {
|
||||
text = fs.readFileSync(file, 'utf8');
|
||||
} catch (error) {
|
||||
throw new Error(`--files-from ${file}: cannot read it (${(error as NodeJS.ErrnoException).code ?? String(error)})`);
|
||||
}
|
||||
const files: string[] = [];
|
||||
const malformed: string[] = [];
|
||||
text.split(/\r?\n/).forEach((line, index) => {
|
||||
if (line === '') return;
|
||||
const segments = line.split('/');
|
||||
if (line !== line.trim() || line.includes('\\') || line.startsWith('/') || /^[A-Za-z]:/.test(line)
|
||||
|| segments.some(segment => segment === '' || segment === '.' || segment === '..')) {
|
||||
malformed.push(`line ${index + 1} ${JSON.stringify(line)}`);
|
||||
} else if (files.includes(line)) {
|
||||
malformed.push(`line ${index + 1} repeats ${line}`);
|
||||
} else {
|
||||
files.push(line);
|
||||
}
|
||||
});
|
||||
if (malformed.length) throw new Error(`--files-from ${file}: ${malformed.length} malformed line(s): ${describeLines(malformed)}; fix: one repo-relative path per line (forward slashes, no surrounding spaces, no ./ or ../, no repeats)`);
|
||||
if (!files.length) throw new Error(`--files-from ${file}: the list is empty (blank lines are ignored); nothing to run`);
|
||||
const plannedSet = new Set(planned);
|
||||
const unknown = files.filter(f => !plannedSet.has(f));
|
||||
if (unknown.length) throw new Error(`--files-from ${file}: ${unknown.length} path(s) not in the free planned set: ${describeLines(unknown)}; fix: use paths from \`bun run scripts/test-free-shards.ts --list\``);
|
||||
return files;
|
||||
}
|
||||
|
||||
/** sha256 of the sorted list, one path per line: the digest a remote gate compares against its upload. */
|
||||
export function filesDigest(files: readonly string[]): string {
|
||||
return createHash('sha256').update([...files].sort().map(f => `${f}\n`).join('')).digest('hex');
|
||||
}
|
||||
|
||||
function formatShardSummary(shards: string[][]): string[] {
|
||||
return shards.map((files, index) => {
|
||||
const preview = files.slice(0, 3).join(', ');
|
||||
@@ -1887,6 +1932,10 @@ async function main(): Promise<number> {
|
||||
}
|
||||
|
||||
let files = allFiles;
|
||||
if (options.filesFrom) {
|
||||
files = readFilesFrom(options.filesFrom, allFiles);
|
||||
console.log(`[test:free] files-from ${options.filesFrom}: ${files.length} files, sha256=${filesDigest(files)}`);
|
||||
}
|
||||
if (options.quick) {
|
||||
const missing = QUICK_CORE.filter(file => !allFiles.includes(file));
|
||||
if (missing.length) throw new Error(`Quick core files missing: ${missing.join(', ')}`);
|
||||
@@ -2015,7 +2064,15 @@ async function main(): Promise<number> {
|
||||
outcomes.push(exclusiveOutcome);
|
||||
}
|
||||
|
||||
return Math.max(homeChange ? 1 : 0, (await retryFailedFreeFiles(outcomes, totalShards, options)).exitCode);
|
||||
const exitCode = Math.max(homeChange ? 1 : 0, (await retryFailedFreeFiles(outcomes, totalShards, options)).exitCode);
|
||||
if (options.filesFrom) {
|
||||
// The machine-read result for a remote gate: tests from each shard's own summary; failed counts failing files, plus one per failed shard with none attributed.
|
||||
const tests = outcomes.reduce((sum, outcome) => sum + (outcome.summary?.testsRan ?? 0), 0);
|
||||
const failing = outcomes.filter(outcome => outcome.status !== 'passed');
|
||||
const failed = exitCode === 0 ? 0 : Math.max(1, new Set(failing.flatMap(outcome => outcome.failingFiles)).size + failing.filter(outcome => !outcome.failingFiles.length).length);
|
||||
console.log(`[test:free] FILES-FROM RESULT: files=${files.length} tests=${tests} failed=${failed} exit=${exitCode}`);
|
||||
}
|
||||
return exitCode;
|
||||
}
|
||||
|
||||
/** Dirty generated artifacts (SKILL.md / host outputs) after a failed exclusive shard. */
|
||||
|
||||
Executable
+279
@@ -0,0 +1,279 @@
|
||||
#!/usr/bin/env bash
|
||||
# pregate-remote.sh — gstack's declared remote_gate (.gstack/pregate.json).
|
||||
#
|
||||
# gstack-pregate runs this under 8 cores for --stage tests --remote:
|
||||
#
|
||||
# gstack-pregate --stage tests --remote --spend-cap 2 --allow-repo-commands
|
||||
#
|
||||
# with PREGATE_FILES_FILE (the selected test files, one per line),
|
||||
# PREGATE_TREE (the working-tree fingerprint being tested) and
|
||||
# PREGATE_SPEND_CAP_USD. It prints the contract line pregate reads
|
||||
# (docs/pregate.md#remote-gate):
|
||||
#
|
||||
# PREGATE_REMOTE: tree=<tree tested> tests=<n> failed=<n> spent=<usd>
|
||||
#
|
||||
# With UBICLOUD_API_KEY set it runs the files on one Ubicloud VM with the
|
||||
# environment of bun run test:ubicloud (scripts/ubicloud/test-free.sh:
|
||||
# setup-free-suite.sh, xvfb-run, GSTACK_EXPECT_BINARIES=1, the free retry
|
||||
# policy), through test-free-shards.ts --files-from. The wall timeout is the
|
||||
# whole minutes the cap buys at the size's list price (UBI_SIZE, default
|
||||
# standard-16), at most 45; under 20 (the shortest useful run) it refuses
|
||||
# before the VM is created. On the VM it recomputes bin/gstack-wtree
|
||||
# after unpack (the tree reported) and checks that the planned set's digest
|
||||
# equals the list's before any test runs. Setup rebuilds committed build
|
||||
# output exactly as the CI free lane does (lib/diagram-render/dist is not
|
||||
# byte-reproducible there); the script prints what setup rewrote, and the
|
||||
# tests must leave the post-setup tree unchanged. spent= is billed minutes (create to confirmed
|
||||
# teardown) times the size's price. Without a key it runs the same files
|
||||
# locally in parallel shards and reports spent=0.
|
||||
#
|
||||
# Exit codes: 0 pass (or a paid-only selection, tests=0); 1 test failures (or
|
||||
# a rejected file list); 2 missing input; 3 tree or plan mismatch (or the tree
|
||||
# moved during the run); 4 infrastructure failure (create, setup, SSH, stop
|
||||
# deadline); 5 the spend cap or pregate's deadline refuses the run. Every refusal line starts with a
|
||||
# PREGATE_REMOTE_* reason code (docs/pregate.md#remote-gate). Nothing is
|
||||
# uploaded or spent before the local tree check.
|
||||
set -euo pipefail
|
||||
# Keep heredoc bodies on temp files, not the pipe window (test/heredoc-pipe-deadlock.test.ts).
|
||||
BASH_COMPAT=50
|
||||
|
||||
# The whole script is one braced block, so bash parses it before running and an
|
||||
# edit to this file during a run cannot change what the run executes.
|
||||
{
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
root=$(cd "$here/../.." && pwd)
|
||||
rerun='gstack-pregate --stage tests --remote --spend-cap <usd> --allow-repo-commands'
|
||||
|
||||
say() { printf 'pregate-remote: %s\n' "$@"; }
|
||||
stop() { local code=$1 reason=$2; shift 2; say "$reason: $1"; shift; [ $# -eq 0 ] || say "$@"; say "see docs/pregate.md#remote-gate"; exit "$code"; }
|
||||
|
||||
[ -n "${PREGATE_FILES_FILE:-}" ] || stop 2 PREGATE_REMOTE_USAGE "PREGATE_FILES_FILE is not set; run: $rerun"
|
||||
[ -n "${PREGATE_TREE:-}" ] || stop 2 PREGATE_REMOTE_USAGE "PREGATE_TREE is not set; run: $rerun"
|
||||
[ -f "$PREGATE_FILES_FILE" ] || stop 2 PREGATE_REMOTE_USAGE "PREGATE_FILES_FILE=$PREGATE_FILES_FILE does not exist; run: $rerun"
|
||||
|
||||
started=$(date +%s)
|
||||
wtree() { (cd "$root" && bin/gstack-wtree); }
|
||||
tree=$(wtree) || stop 3 PREGATE_REMOTE_TREE_MISMATCH "bin/gstack-wtree could not fingerprint $root" "nothing uploaded, nothing spent" "rerun: $rerun"
|
||||
if [ "$tree" != "$PREGATE_TREE" ]; then
|
||||
stop 3 PREGATE_REMOTE_TREE_MISMATCH "the working tree is ${tree:0:12}, not the planned tree ${PREGATE_TREE:0:12}" "nothing uploaded, nothing spent" "rerun: $rerun"
|
||||
fi
|
||||
|
||||
logs="$root/.context/pregate-remote/$(date -u +%Y%m%dT%H%M%SZ)-$$"
|
||||
mkdir -p "$logs"
|
||||
log="$logs/run.log"
|
||||
|
||||
# pregate's selection can name paid test files (skill-e2e and other paid
|
||||
# evals); the paid PR gate covers those, and test-free-shards --files-from
|
||||
# rejects them, so only the free files go on.
|
||||
files="$logs/free-files.txt"
|
||||
paid=$(cd "$root" && bun -e 'import { isPaidTestFile } from "./test/helpers/paid-test-set.ts"; import * as fs from "node:fs";
|
||||
const lines = fs.readFileSync(process.argv[1], "utf8").split(/\r?\n/).filter(Boolean);
|
||||
fs.writeFileSync(process.argv[2], lines.filter(f => !isPaidTestFile(f)).map(f => f + "\n").join(""));
|
||||
console.log(lines.filter(f => isPaidTestFile(f)).join("\n"));' "$PREGATE_FILES_FILE" "$files") || stop 2 PREGATE_REMOTE_USAGE "could not split $PREGATE_FILES_FILE into free and paid test files"
|
||||
if [ -n "$paid" ]; then
|
||||
say "$(grep -c . <<<"$paid") paid test file(s) are covered by the paid PR gate, not the free tier: $(head -5 <<<"$paid" | paste -sd, - | sed 's/,/, /g')$( [ "$(grep -c . <<<"$paid")" -le 5 ] || echo " ...")"
|
||||
fi
|
||||
count=$(grep -c . "$files" || true)
|
||||
if [ "$count" -eq 0 ]; then
|
||||
say "PREGATE_REMOTE_PAID_ONLY: every selected file is a paid test file; no free test ran, so this receipt cannot clear the obligation (tests=0)" "see docs/pregate.md#remote-gate"
|
||||
echo "PREGATE_REMOTE: tree=$tree tests=0 failed=0 spent=0"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
result_line() { grep '^\[test:free\] FILES-FROM RESULT: ' "$1" | tail -1 || true; }
|
||||
field() { sed -n "s/.* $1=\([0-9][0-9]*\).*/\1/p" <<<"$2"; }
|
||||
|
||||
if [ -z "${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}" ]; then
|
||||
say "no UBICLOUD_API_KEY: running $count files locally in parallel shards"
|
||||
set +e
|
||||
(cd "$root" && bun run scripts/test-free-shards.ts --files-from "$files") 2>&1 | tee "$log"
|
||||
rc=${PIPESTATUS[0]}
|
||||
set -e
|
||||
after=$(wtree) || after=unavailable
|
||||
if [ "$after" != "$tree" ]; then
|
||||
(cd "$root" && git diff --name-status "$tree" "$after" 2>&1 | head -20 | sed 's/^/ moved: /') || true
|
||||
stop 3 PREGATE_REMOTE_TREE_MOVED "the working tree moved during the run (${tree:0:12} before, ${after:0:12} after); no result reported" "log: $log" "rerun on a tree nothing else edits: $rerun"
|
||||
fi
|
||||
result=$(result_line "$log")
|
||||
[ -n "$result" ] || stop "$(( rc == 0 ? 1 : rc ))" PREGATE_REMOTE_NO_RESULT "test-free-shards exited $rc without a FILES-FROM RESULT line" "log: $log"
|
||||
say "log: $log"
|
||||
echo "backend=local ($(getconf _NPROCESSORS_ONLN 2>/dev/null || sysctl -n hw.ncpu) cores; not the CI environment)"
|
||||
echo "PREGATE_REMOTE: tree=$tree tests=$(field tests "$result") failed=$(field failed "$result") spent=0"
|
||||
exit "$rc"
|
||||
fi
|
||||
|
||||
size=${UBI_SIZE:-standard-16}
|
||||
case $size in
|
||||
standard-2) ppm=0.00104 ;; standard-4) ppm=0.00208 ;; standard-8) ppm=0.00416 ;;
|
||||
standard-16) ppm=0.00831 ;; standard-30) ppm=0.01559 ;; standard-60) ppm=0.03117 ;;
|
||||
*) stop 5 PREGATE_REMOTE_NO_PRICE "no price for UBI_SIZE=$size (priced: standard-2, -4, -8, -16, -30, -60; Ubicloud list price per minute)" "nothing created, nothing spent" ;;
|
||||
esac
|
||||
cap=${PREGATE_SPEND_CAP_USD:-}
|
||||
[[ "$cap" =~ ^[0-9]+(\.[0-9]+)?$ ]] || stop 2 PREGATE_REMOTE_USAGE "PREGATE_SPEND_CAP_USD is not a dollar amount ('$cap'); run: $rerun"
|
||||
max_min=45 min_min=20
|
||||
wall_min=$(awk -v c="$cap" -v p="$ppm" -v m="$max_min" 'BEGIN { w = int(c / p); print (w < m ? w : m) }')
|
||||
usd() { awk -v m="$1" -v p="$ppm" 'BEGIN { s = m * p; c = int(s * 100); if (c < s * 100) c++; printf "%.2f", c / 100 }'; }
|
||||
if [ "$wall_min" -lt "$min_min" ]; then
|
||||
stop 5 PREGATE_REMOTE_CAP_TOO_LOW "worst case \$$(usd "$min_min") ($min_min min x \$$ppm/min on $size) exceeds the spend cap \$$cap" "nothing created, nothing spent" "rerun with --spend-cap $(usd "$min_min") or more"
|
||||
fi
|
||||
# The enclosing gstack-pregate kills this gate at its --timeout (default 600 s)
|
||||
# with a 5 s grace, so the VM must be told to stop early enough for teardown to
|
||||
# be confirmed first: 120 s reserved for teardown plus 15 s margin. The deadline
|
||||
# comes from PREGATE_TIMEOUT_S, else from the gstack-pregate process this gate
|
||||
# runs under (its --timeout, else 600); run by hand there is none.
|
||||
outer_s=${PREGATE_TIMEOUT_S:-} outer_from=PREGATE_TIMEOUT_S
|
||||
if [ -z "$outer_s" ]; then
|
||||
outer_from="" pid=$PPID
|
||||
for _ in 1 2 3 4 5 6 7 8; do
|
||||
{ [ -n "$pid" ] && [ "$pid" -gt 1 ]; } || break
|
||||
cmdline=$(ps -o args= -p "$pid" 2>/dev/null) || break
|
||||
read -r arg0 arg1 _ <<<"$cmdline" || true
|
||||
case "${arg0##*/} ${arg1##*/}" in
|
||||
"gstack-pregate "*|*" gstack-pregate")
|
||||
outer_s=$(sed -n 's/.*--timeout[ =]\([0-9][0-9]*\).*/\1/p' <<<"$cmdline")
|
||||
outer_s=${outer_s:-600} outer_from="gstack-pregate --timeout"
|
||||
break ;;
|
||||
esac
|
||||
pid=$(ps -o ppid= -p "$pid" 2>/dev/null | tr -d ' ')
|
||||
done
|
||||
fi
|
||||
stop_after=$(( (wall_min - 3) * 60 )) stop_why="the cap's $wall_min min wall minus 3 min for teardown"
|
||||
if [ -n "$outer_s" ]; then
|
||||
outer_stop=$(( outer_s - ($(date +%s) - started) - 135 ))
|
||||
if [ "$outer_stop" -lt 300 ]; then
|
||||
stop 5 PREGATE_REMOTE_TIMEOUT_TOO_SHORT "$outer_from ${outer_s}s leaves ${outer_stop}s for the VM after 120 s reserved for teardown and 15 s margin (at least 300 s needed)" "nothing created, nothing spent" "rerun with --timeout 2700: $rerun"
|
||||
fi
|
||||
if [ "$outer_stop" -lt "$stop_after" ]; then stop_after=$outer_stop stop_why="$outer_from ${outer_s}s minus 120 s teardown reserve and 15 s margin"; fi
|
||||
fi
|
||||
say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)"
|
||||
|
||||
digest=$(tr -d '\r' <"$files" | grep . | LC_ALL=C sort | sha256sum | cut -d' ' -f1)
|
||||
prelude="$logs/vm-setup.sh"
|
||||
{
|
||||
printf '%s\n' 'set -euo pipefail' 'BASH_COMPAT=50' \
|
||||
'command -v git >/dev/null || { sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git >/dev/null; }' \
|
||||
'vm_tree=$(bin/gstack-wtree) || vm_tree=unavailable' \
|
||||
"if [ \"\$vm_tree\" != $tree ]; then echo \"PREGATE_VM_TREE_MISMATCH vm=\$vm_tree\"; exit 3; fi" \
|
||||
'echo "PREGATE_VM_TREE $vm_tree"' \
|
||||
"cat > ~/.pregate-files.txt <<'PREGATE_FILES'"
|
||||
tr -d '\r' <"$files" | grep .
|
||||
printf '%s\n' 'PREGATE_FILES' "cat > ~/.pregate-run.sh <<'PREGATE_RUN'" \
|
||||
'planned=$(bun run scripts/test-free-shards.ts --files-from ~/.pregate-files.txt --dry-run | sed -n "s/^\[test:free\] files-from .*: [0-9]* files, sha256=\([0-9a-f]*\)$/\1/p")' \
|
||||
"if [ \"\$planned\" != $digest ]; then echo \"PREGATE_VM_PLAN_MISMATCH planned=\${planned:-none} expected=$digest\"; exit 3; fi" \
|
||||
'xvfb-run -a bun run test:free --files-from ~/.pregate-files.txt' \
|
||||
'rc=$?' \
|
||||
'after=$(bin/gstack-wtree || echo unavailable)' \
|
||||
'built=$(cat ~/.pregate-built-tree)' \
|
||||
'echo "PREGATE_VM_TREE_AFTER $after"' \
|
||||
'[ "$after" = "$built" ] || git diff --name-status "$built" "$after" 2>&1 | head -20 | sed "s/^/PREGATE_VM_MOVED /"' \
|
||||
'exit $rc' \
|
||||
'PREGATE_RUN' \
|
||||
'bash scripts/ubicloud/setup-free-suite.sh' \
|
||||
'built=$(bin/gstack-wtree || echo unavailable)' \
|
||||
'echo "$built" > ~/.pregate-built-tree' \
|
||||
'echo "PREGATE_VM_TREE_BUILT $built"' \
|
||||
'[ "$built" = "$vm_tree" ] || git diff --name-status "$vm_tree" "$built" 2>&1 | head -20 | sed "s/^/PREGATE_VM_REBUILT /"'
|
||||
} >"$prelude"
|
||||
|
||||
args=(run -s "$size" --src "$root" --setup "$prelude"
|
||||
--env GSTACK_EXPECT_BINARIES=1 --env GSTACK_FREE_RETRY_FLAKY=1
|
||||
--env GSTACK_FLAKE_LEDGER=/tmp/gstack-free-test-flake-ledger.jsonl
|
||||
--pull "/tmp/gstack-free-test-*:$logs"
|
||||
--pull "work/$(basename "$root")/.context/free-test-logs:$logs")
|
||||
[ -z "${GSTACK_FREE_JOBS:-}" ] || args+=(--env "GSTACK_FREE_JOBS=$GSTACK_FREE_JOBS")
|
||||
|
||||
# The VM run gets its own session (python3, which ubi-runner already needs, for
|
||||
# setsid on Linux and macOS): pregate's --timeout and stall kill signal this
|
||||
# script's process group and then SIGKILL it, which must never cut ubi-runner's
|
||||
# teardown short. Output goes to files (a dead reader would SIGPIPE ubi-runner),
|
||||
# stamped for the phase timings; this script streams it. A watchdog sends
|
||||
# ubi-runner TERM at the cap-derived wall timeout, and so does a TERM or INT
|
||||
# to this script; ubi-runner then destroys the VM.
|
||||
vm_session='
|
||||
exec </dev/null > >(exec >/dev/null 2>&1; trap "" INT TERM HUP; while IFS= read -r line; do printf "%s %s\n" "$(date +%s)" "$line" >>"$2"; printf "%s\n" "$line" >>"$3"; done) 2>&1
|
||||
bash "$4" "${@:5}" -- "bash ~/.pregate-run.sh" &
|
||||
runner=$!
|
||||
echo "$runner" >"$2.pid"
|
||||
( sleep "$1"; touch "$2.timedout"; kill -TERM "$runner" ) </dev/null >/dev/null 2>&1 &
|
||||
watchdog=$!
|
||||
wait "$runner"
|
||||
rc=$?
|
||||
kill "$watchdog" 2>/dev/null
|
||||
[ ! -f "$2.timedout" ] || rc=124
|
||||
exit "$rc"
|
||||
'
|
||||
stream="$logs/stream.log"
|
||||
at() { grep -m1 -E "^[0-9]+ $1" "$log" | cut -d' ' -f1 || true; }
|
||||
( while sleep 60 </dev/null >/dev/null 2>&1; do say "still running ($(( $(date +%s) - started )) s)"; done ) &
|
||||
heartbeat=$!
|
||||
vmrun="" tailer=""
|
||||
trap 'kill "$heartbeat" $tailer 2>/dev/null || true' EXIT
|
||||
trap '[ -z "$vmrun" ] || kill -TERM "$(cat "$log.pid" 2>/dev/null)" 2>/dev/null; stop 130 PREGATE_REMOTE_INTERRUPTED "interrupted: ubi-runner is tearing the VM down on its own (log: $log)"' INT TERM
|
||||
|
||||
billed=0 attempt=0
|
||||
while :; do
|
||||
attempt=$(( attempt + 1 ))
|
||||
: >"$log"
|
||||
: >"$stream"
|
||||
rm -f "$log.pid" "$log.timedout"
|
||||
t0=$(date +%s)
|
||||
vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')"
|
||||
say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
|
||||
python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
|
||||
bash -c "$vm_session" pregate-remote-vm "$stop_after" "$log" "$stream" "$here/ubi-runner.sh" "${args[@]}" -n "$vm_name" &
|
||||
vmrun=$!
|
||||
tail -n +1 -f "$stream" &
|
||||
tailer=$!
|
||||
set +e
|
||||
wait "$vmrun"
|
||||
rc=$?
|
||||
set -e
|
||||
vmrun=""
|
||||
sleep 1
|
||||
kill "$tailer" 2>/dev/null || true
|
||||
tailer=""
|
||||
t_end=$(date +%s)
|
||||
t_create=$(at 'ubi-runner: creating ')
|
||||
if [ -n "$t_create" ] && ! grep -q 'ubi-runner: create failed' "$log"; then minutes=$(( (t_end - t_create + 59) / 60 )); billed=$(( billed + (minutes > 0 ? minutes : 1) )); fi
|
||||
if [ "$rc" -ne 0 ] && [ "$attempt" -eq 1 ] && grep -q 'vCPU quota exhausted' "$log"; then
|
||||
say "vCPU quota refused the VM; one retry in ${PREGATE_REMOTE_QUOTA_WAIT_S:-60} s"
|
||||
cp "$log" "$logs/attempt-1.log"
|
||||
sleep "${PREGATE_REMOTE_QUOTA_WAIT_S:-60}"
|
||||
continue
|
||||
fi
|
||||
break
|
||||
done
|
||||
spent=$(usd "$billed")
|
||||
|
||||
t_ready=$(at 'ubi-runner: ready: ')
|
||||
t_setup=$(at 'ubi-runner: running setup')
|
||||
t_run=$(at 'ubi-runner: running on ')
|
||||
t_exit=$(at 'ubi-runner: command exited')
|
||||
span() { if [ -n "$1" ] && [ -n "$2" ]; then echo "$(( $2 - $1 ))s"; else echo "-"; fi; }
|
||||
say "phases: create $(span "$t0" "$t_ready"), setup $(span "$t_setup" "$t_run"), run $(span "$t_run" "$t_exit"), teardown $(span "$t_exit" "$t_end"); total $(( t_end - started ))s (target 600s)"
|
||||
say "log: $log"
|
||||
|
||||
vm_tree=$(sed -n 's/^[0-9]* PREGATE_VM_TREE \([0-9a-f]*\)$/\1/p' "$log" | tail -1)
|
||||
mismatch=$(grep -m1 -oE 'PREGATE_VM_(TREE|PLAN)_MISMATCH .*' "$log" || true)
|
||||
if [ -n "$mismatch" ]; then
|
||||
stop 3 PREGATE_REMOTE_VM_TREE "the VM refused before running tests: $mismatch (expected tree ${tree:0:12})" "no tests ran; spent=$spent ($billed billed min on $size)" "rerun: $rerun"
|
||||
fi
|
||||
result=$(result_line <(cut -d' ' -f2- "$log"))
|
||||
if [ -n "$result" ] && [ "$rc" -ne 124 ]; then
|
||||
after=$(sed -n 's/^[0-9]* PREGATE_VM_TREE_AFTER \(.*\)$/\1/p' "$log" | tail -1)
|
||||
built=$(sed -n 's/^[0-9]* PREGATE_VM_TREE_BUILT \(.*\)$/\1/p' "$log" | tail -1)
|
||||
sed -n 's/^[0-9]* PREGATE_VM_REBUILT / setup rebuilt (committed build output, as the CI free lane rebuilds it): /p' "$log"
|
||||
if [ "$after" != "$built" ]; then
|
||||
sed -n 's/^[0-9]* PREGATE_VM_MOVED / moved: /p' "$log"
|
||||
stop 3 PREGATE_REMOTE_TREE_MOVED "the VM's tree moved during the run (${built:0:12} after setup, ${after:0:12} after tests); no result reported" "spent=$spent ($billed billed min on $size)" "rerun: $rerun"
|
||||
fi
|
||||
echo "PREGATE_REMOTE: tree=$vm_tree tests=$(field tests "$result") failed=$(field failed "$result") spent=$spent"
|
||||
exit $(( rc == 0 ? 0 : 1 ))
|
||||
fi
|
||||
vm=$(sed -n 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' "$log" | tail -1)
|
||||
down=$(grep -E 'ubi-runner: (destroyed|gone:|WARNING|FAILED to destroy)' "$log" | tail -1 | cut -d' ' -f2- || true)
|
||||
cause=$(grep -E 'ubi-runner: ' "$log" | grep -vE 'ubi-runner: (destroyed|gone:|destroy requested|waiting for an in-flight)' | tail -1 | cut -d' ' -f2- || true)
|
||||
[ "$rc" -ne 124 ] || cause="the VM was told to stop after ${stop_after}s ($stop_why) and torn down"
|
||||
stop 4 PREGATE_REMOTE_INFRA "infrastructure failure (ubi-runner exit $rc): ${cause:-no ubi-runner message}" \
|
||||
"VM: ${vm:-none created}${down:+; $down}" "log: $log" "spent=$spent ($billed billed min on $size)"
|
||||
}
|
||||
+12
-1
@@ -1,11 +1,12 @@
|
||||
{
|
||||
"recordedFrom": "96764e80 (pre-engine base runners); paid lane re-recorded at v1.91.12.0 for the marathon tier and the --slice-budget, --max-parallel, --case and --trials flags (PR 2999); free lane re-recorded for the --attribute-home flag (PR 3014)",
|
||||
"recordedFrom": "96764e80 (pre-engine base runners); paid lane re-recorded at v1.91.12.0 for the marathon tier and the --slice-budget, --max-parallel, --case and --trials flags (PR 2999); free lane re-recorded for the --attribute-home flag (PR 3014) and the --files-from flag (fix wave 3, W2-A3)",
|
||||
"free": [
|
||||
{
|
||||
"argv": [],
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -33,6 +34,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": true,
|
||||
"filesFrom": null,
|
||||
"listOnly": true,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -59,6 +61,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": true,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": true,
|
||||
@@ -84,6 +87,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": true,
|
||||
"windowsOnly": false,
|
||||
@@ -109,6 +113,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -137,6 +142,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -163,6 +169,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -191,6 +198,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -221,6 +229,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -249,6 +258,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
@@ -399,6 +409,7 @@
|
||||
"result": {
|
||||
"ok": {
|
||||
"dryRun": false,
|
||||
"filesFrom": null,
|
||||
"listOnly": false,
|
||||
"recordDurations": false,
|
||||
"windowsOnly": false,
|
||||
|
||||
@@ -19,6 +19,8 @@ import {
|
||||
markActiveSiblings,
|
||||
resolveVersionPath,
|
||||
fetchGitClaimed,
|
||||
fetchGithubClaimed,
|
||||
VERSION_FETCH_TIMEOUT_MS,
|
||||
} from "../bin/gstack-next-version";
|
||||
|
||||
describe("parseVersion", () => {
|
||||
@@ -1012,6 +1014,8 @@ describe("integration (smoke)", () => {
|
||||
expect(parsed.fallback).toBeNull();
|
||||
expect(parsed.warnings).toEqual([]);
|
||||
expect(parsed.claimed).toEqual([CLAIM]); // the canned PR's owner matches the canned repo owner
|
||||
expect(parsed.complete).toBe(true);
|
||||
expect(parsed.unknown_claims).toEqual([]);
|
||||
expect(parsed.version).toBe("1.6.5.0"); // the canned claim on 1.6.4.0 is skipped
|
||||
} finally {
|
||||
rmSync(stubDir, { recursive: true, force: true });
|
||||
@@ -1217,3 +1221,160 @@ describe("no version source: one signal, never an invented slot (G1)", () => {
|
||||
expect(r.stderr).not.toContain("assuming");
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
// W2-A3 (fix wave 3): VERSION fetches run through async workers bounded at
|
||||
// GH_API_CONCURRENCY, each with a 15 s timeout; claims and warnings are sorted
|
||||
// by PR number, and a timed-out or failed fetch is an unknown claim. A named
|
||||
// fake `gh` first on PATH answers every call with a per-PR delay; the live
|
||||
// GitHub API is never called.
|
||||
describe("concurrent VERSION fetches (W2-A3)", () => {
|
||||
const OWNER = "fixture-owner";
|
||||
type FakePR = { pr: number; version?: string; delay: number; reply?: "404" | "502" | "malformed" };
|
||||
const prBranch = (pr: number) => `fixture/pr-${pr}`;
|
||||
const prUrl = (pr: number) => `https://github.com/${OWNER}/repo/pull/${pr}`;
|
||||
const claim = (pr: number, version: string) => ({ pr, branch: prBranch(pr), version, url: prUrl(pr) });
|
||||
|
||||
function delayedGh(dir: string, prs: FakePR[]): string {
|
||||
const flight = join(dir, "flight.log");
|
||||
const list = JSON.stringify(prs.map((p) => ({ number: p.pr, headRefName: prBranch(p.pr), headRepositoryOwner: { login: OWNER }, url: prUrl(p.pr), isDraft: false })));
|
||||
const reply = (p: FakePR) => {
|
||||
if (p.reply === "404") return `echo 'gh: Not Found (HTTP 404)' >&2; exit 1`;
|
||||
if (p.reply === "502") return `echo 'gh: Server Error (HTTP 502)' >&2; exit 1`;
|
||||
const body = p.reply === "malformed" ? "not-a-version\n" : `${p.version}\n`;
|
||||
return `echo '${Buffer.from(body).toString("base64")}'; exit 0`;
|
||||
};
|
||||
writeFileSync(join(dir, "gh"), [
|
||||
"#!/bin/sh",
|
||||
'case "$*" in',
|
||||
` "pr list --state open --base main --limit 200 --json number,headRefName,headRepositoryOwner,url,isDraft") printf '%s\\n' '${list}'; exit 0 ;;`,
|
||||
` "repo view --json owner -q .owner.login") echo '${OWNER}'; exit 0 ;;`,
|
||||
...prs.map((p) => ` "api repos/{owner}/{repo}/contents/VERSION?ref=${encodeURIComponent(prBranch(p.pr))} -q .content") echo + >> '${flight}'; sleep ${p.delay}; echo - >> '${flight}'; ${reply(p)} ;;`),
|
||||
"esac",
|
||||
'echo "FAKE_GH_UNHANDLED $*" >&2',
|
||||
"exit 2",
|
||||
"",
|
||||
].join("\n"), { mode: 0o755 });
|
||||
return flight;
|
||||
}
|
||||
|
||||
async function fetchWith(prs: FakePR[], fetchTimeoutMs?: number) {
|
||||
const dir = mkdtempSync(join(tmpdir(), "nextver-concurrency-"));
|
||||
const flight = delayedGh(dir, prs);
|
||||
const savedPath = process.env.PATH;
|
||||
process.env.PATH = `${dir}:${savedPath ?? ""}`;
|
||||
const warnings: string[] = [];
|
||||
const started = performance.now();
|
||||
try {
|
||||
const result: { claimed: { pr: number }[]; offline: boolean; unknownClaims: number[] } = await fetchGithubClaimed("main", "VERSION", null, warnings, fetchTimeoutMs);
|
||||
let running = 0;
|
||||
let maxInFlight = 0;
|
||||
for (const event of readFileSync(flight, "utf8").trim().split("\n")) {
|
||||
running += event === "+" ? 1 : -1;
|
||||
maxInFlight = Math.max(maxInFlight, running);
|
||||
}
|
||||
return { ...result, warnings, ms: performance.now() - started, maxInFlight };
|
||||
} finally {
|
||||
process.env.PATH = savedPath;
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
test("8 PRs at 0.5 s each finish in under 2 s with every claim, sorted by PR number", async () => {
|
||||
const r = await fetchWith([48, 41, 47, 42, 46, 43, 45, 44].map((pr) => ({ pr, version: `1.6.${pr}.0`, delay: 0.5 })));
|
||||
expect(r.ms).toBeLessThan(2000); // gstack-timing-ok: the plan's contract (8 x 0.5 s serial took 4,160 ms)
|
||||
expect(r.offline).toBe(false);
|
||||
expect(r.claimed).toEqual([41, 42, 43, 44, 45, 46, 47, 48].map((pr) => claim(pr, `1.6.${pr}.0`)));
|
||||
expect(r.warnings).toEqual([]);
|
||||
expect(r.unknownClaims).toEqual([]);
|
||||
expect(r.maxInFlight).toBeGreaterThan(1);
|
||||
}, 30_000);
|
||||
|
||||
test("randomized delays give the same claims, warnings and unknown claims every run, within the concurrency bound", async () => {
|
||||
let seed = 7;
|
||||
const random = () => { seed = (seed * 1103515245 + 12345) % 2 ** 31; return seed / 2 ** 31; };
|
||||
const base: Omit<FakePR, "delay">[] = Array.from({ length: 14 }, (_, i) => ({ pr: 60 + i, version: `2.0.${i}.0` }));
|
||||
base[2]!.reply = "404";
|
||||
base[5]!.reply = "malformed";
|
||||
base[9]!.reply = "502";
|
||||
const runs = [];
|
||||
for (let run = 0; run < 3; run++) runs.push(await fetchWith(base.map((p) => ({ ...p, delay: Number((random() * 0.3).toFixed(2)) }))));
|
||||
for (const r of runs) {
|
||||
expect(r.claimed).toEqual(runs[0]!.claimed);
|
||||
expect(r.warnings).toEqual(runs[0]!.warnings);
|
||||
expect(r.unknownClaims).toEqual([69]);
|
||||
expect(r.maxInFlight).toBeLessThanOrEqual(10);
|
||||
}
|
||||
expect(runs[0]!.claimed.map((c) => c.pr)).toEqual([60, 61, 63, 64, 66, 67, 68, 70, 71, 72, 73]);
|
||||
expect(runs[0]!.warnings).toEqual([
|
||||
"PR #62: could not fetch VERSION (fork, private, or wrong path — try --version-path or .gstack/version-path)",
|
||||
"PR #65: VERSION is malformed (not-a-version)",
|
||||
"PR #69: VERSION fetch failed (gh: Server Error (HTTP 502)); claim unknown, the slot may collide; rerun, then ask the coordinator",
|
||||
]);
|
||||
}, 60_000);
|
||||
|
||||
test("a timed-out fetch is killed, keeps its cause and counts as an unknown claim", async () => {
|
||||
expect(VERSION_FETCH_TIMEOUT_MS).toBe(15000);
|
||||
const r = await fetchWith([{ pr: 51, version: "1.7.1.0", delay: 0.1 }, { pr: 52, version: "1.7.2.0", delay: 10 }], 500);
|
||||
expect(r.ms).toBeLessThan(5000); // gstack-timing-ok: the killed fetch must not wait out its 10 s sleeper
|
||||
expect(r.claimed).toEqual([claim(51, "1.7.1.0")]);
|
||||
expect(r.unknownClaims).toEqual([52]);
|
||||
expect(r.warnings).toEqual(["PR #52: VERSION fetch timed out after 0.5 s; claim unknown, the slot may collide; rerun, then ask the coordinator"]);
|
||||
}, 30_000);
|
||||
|
||||
function cliRepo(dir: string): string {
|
||||
// A local origin whose path contains github.com pins host:"github" by the URL sniff, so no auth probe or network call runs.
|
||||
const bare = join(dir, "github.com", "origin.git");
|
||||
const work = join(dir, "work");
|
||||
const git = (cwd: string, ...args: string[]) => Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd, timeout: 30_000 });
|
||||
mkdirSync(bare, { recursive: true });
|
||||
mkdirSync(work);
|
||||
git(bare, "init", "-q", "--bare", "-b", "main");
|
||||
git(work, "init", "-q", "-b", "main");
|
||||
writeFileSync(join(work, "VERSION"), "1.6.3.0\n");
|
||||
git(work, "add", "-A");
|
||||
git(work, "commit", "-qm", "v1.6.3.0 chore: base");
|
||||
git(work, "remote", "add", "origin", bare);
|
||||
git(work, "push", "-q", "origin", "main");
|
||||
return work;
|
||||
}
|
||||
|
||||
test("a timed-out worker beside successful ones makes the pick provisional: complete false, on stderr and in warnings", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "nextver-provisional-cli-"));
|
||||
try {
|
||||
delayedGh(dir, [{ pr: 91, version: "1.6.4.0", delay: 0 }, { pr: 92, version: "1.6.9.0", delay: 25 }, { pr: 93, version: "1.6.5.0", delay: 0.2 }]);
|
||||
const proc = Bun.spawnSync(["bun", "run", join(import.meta.dir, "..", "bin", "gstack-next-version"), "--base", "main", "--bump", "patch",
|
||||
"--current-version", "1.6.3.0", "--workspace-root", "null", "--exclude-pr", "999"],
|
||||
{ cwd: cliRepo(dir), timeout: 40_000, env: { ...process.env, PATH: `${dir}:${process.env.PATH ?? ""}` } });
|
||||
const out = JSON.parse(proc.stdout.toString());
|
||||
expect(out.claimed.map((c: { pr: number }) => c.pr)).toEqual([91, 93]);
|
||||
expect(out.unknown_claims).toEqual([92]);
|
||||
expect(out.complete).toBe(false);
|
||||
expect(out.version).toBe("1.6.6.0");
|
||||
const provisional = "provisional pick: the VERSION claims of PR #92 are unknown, so 1.6.6.0 is not a verified slot and may collide; rerun, then ask the coordinator before stamping";
|
||||
expect(out.warnings).toEqual(["PR #92: VERSION fetch timed out after 15 s; claim unknown, the slot may collide; rerun, then ask the coordinator", provisional]);
|
||||
expect(proc.stderr.toString()).toContain(`gstack-next-version: ${provisional}`);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}, 45_000);
|
||||
|
||||
test("the CLI JSON carries unknown_claims", () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), "nextver-unknown-cli-"));
|
||||
try {
|
||||
delayedGh(dir, [{ pr: 81, version: "1.6.4.0", delay: 0 }, { pr: 82, delay: 0, reply: "502" }]);
|
||||
const work = cliRepo(dir);
|
||||
const proc = Bun.spawnSync(["bun", "run", join(import.meta.dir, "..", "bin", "gstack-next-version"), "--base", "main", "--bump", "patch",
|
||||
"--current-version", "1.6.3.0", "--workspace-root", "null", "--exclude-pr", "999"],
|
||||
{ cwd: work, timeout: 30_000, env: { ...process.env, PATH: `${dir}:${process.env.PATH ?? ""}` } });
|
||||
const out = JSON.parse(proc.stdout.toString());
|
||||
expect(out.host, proc.stderr.toString()).toBe("github");
|
||||
expect(out.unknown_claims).toEqual([82]);
|
||||
expect(out.complete).toBe(false);
|
||||
expect(out.claimed.map((c: { pr: number }) => c.pr)).toEqual([81]);
|
||||
expect(out.version).toBe("1.6.5.0");
|
||||
expect(out.warnings).toContain("PR #82: VERSION fetch failed (gh: Server Error (HTTP 502)); claim unknown, the slot may collide; rerun, then ask the coordinator");
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -162,6 +162,30 @@ describe('three PRs, reordered twice, zero manual edits', () => {
|
||||
const forced = restamp(repo, ['--next', '--dry-run', '--queue-mode', 'stamp-at-merge']);
|
||||
expect(forced.out).toContain('claims: ignored (stamp-at-merge)');
|
||||
});
|
||||
|
||||
test('claim-at-open with an unknown claim says incomplete, names the PR and never says read', () => {
|
||||
const repo = fixture({ policy: { ...BASE_POLICY, queue_mode: 'claim-at-open' }, prefix: 'gstack-restamp-github.com-' });
|
||||
makePr(repo, 'pr-a');
|
||||
const shim = path.join(repo.root, 'gh-unknown-claim');
|
||||
fs.mkdirSync(shim);
|
||||
const list = JSON.stringify([41, 42].map(n => ({ number: n, headRefName: `pr-${n}`, headRepositoryOwner: { login: 'o' }, url: `https://github.com/o/r/pull/${n}`, isDraft: false })));
|
||||
fs.writeFileSync(path.join(shim, 'gh'), [
|
||||
'#!/bin/sh',
|
||||
'case "$*" in',
|
||||
` "pr list --state open --base main --limit 200 --json number,headRefName,headRepositoryOwner,url,isDraft") printf '%s\\n' '${list}'; exit 0 ;;`,
|
||||
" \"repo view --json owner -q .owner.login\") echo o; exit 0 ;;",
|
||||
` "api repos/{owner}/{repo}/contents/VERSION?ref=pr-41 -q .content") echo '${Buffer.from('1.2.5.0\n').toString('base64')}'; exit 0 ;;`,
|
||||
" \"api repos/{owner}/{repo}/contents/VERSION?ref=pr-42 -q .content\") echo 'gh: Server Error (HTTP 502)' >&2; exit 1 ;;",
|
||||
'esac',
|
||||
'exit 2',
|
||||
'',
|
||||
].join('\n'), { mode: 0o755 });
|
||||
const r = restamp(repo, ['--next', '--dry-run'], { PATH: `${shim}:${process.env.PATH ?? ''}` });
|
||||
expect(r.status, r.err).toBe(EXIT.ok);
|
||||
expect(r.out).toContain('STAMP_VERSION: 1.2.6.0');
|
||||
expect(r.out).toContain('claims: incomplete (claim-at-open; the VERSION claims of PR #42 are unknown, so 1.2.6.0 is provisional and may collide; rerun, then ask the coordinator before stamping; queue [');
|
||||
expect(r.out).not.toMatch(/^claims: read/m);
|
||||
});
|
||||
});
|
||||
|
||||
describe('journal and injected failures', () => {
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* W2-A3 (fix wave 3): test-free-shards.ts --files-from <path> plans exactly
|
||||
* the listed files from the free planned set and rejects, before planning, an
|
||||
* unknown path, an empty list or a malformed line. The remote gate
|
||||
* (scripts/ubicloud/pregate-remote.sh) compares the printed digest with its
|
||||
* upload; the full run's FILES-FROM RESULT line is covered there.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { collectFreeTestFiles, filesDigest, parseCliOptions, readFilesFrom } from '../scripts/test-free-shards';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const DIR = fs.mkdtempSync(path.join(os.tmpdir(), 'free-files-from-'));
|
||||
afterAll(() => fs.rmSync(DIR, { recursive: true, force: true }));
|
||||
let n = 0;
|
||||
const list = (text: string) => {
|
||||
const file = path.join(DIR, `list-${++n}.txt`);
|
||||
fs.writeFileSync(file, text);
|
||||
return file;
|
||||
};
|
||||
const PLANNED = ['browse/test/a.test.ts', 'test/b.test.ts', 'test/c.test.ts', 'test/d.test.ts', 'test/e.test.ts', 'test/f.test.ts', 'test/g.test.ts'];
|
||||
|
||||
describe('--files-from parsing', () => {
|
||||
test('one path per line; blank lines and CRLF endings are accepted; order is kept', () => {
|
||||
expect(readFilesFrom(list('test/c.test.ts\r\n\r\nbrowse/test/a.test.ts\n\n'), PLANNED)).toEqual(['test/c.test.ts', 'browse/test/a.test.ts']);
|
||||
});
|
||||
|
||||
test('paths outside the free planned set are rejected with the first five, a count and the --list fix', () => {
|
||||
const file = list(['test/b.test.ts', ...['x1', 'x2', 'x3', 'x4', 'x5', 'x6', 'x7'].map(name => `test/${name}.test.ts`)].join('\n'));
|
||||
expect(() => readFilesFrom(file, PLANNED)).toThrow(
|
||||
`--files-from ${file}: 7 path(s) not in the free planned set: test/x1.test.ts, test/x2.test.ts, test/x3.test.ts, test/x4.test.ts, test/x5.test.ts (+2 more); fix: use paths from \`bun run scripts/test-free-shards.ts --list\``);
|
||||
});
|
||||
|
||||
test('an empty list, an unreadable file and every malformed line shape are rejected', () => {
|
||||
const empty = list('\n\r\n\n');
|
||||
expect(() => readFilesFrom(empty, PLANNED)).toThrow(`--files-from ${empty}: the list is empty (blank lines are ignored); nothing to run`);
|
||||
expect(() => readFilesFrom(path.join(DIR, 'absent.txt'), PLANNED)).toThrow('cannot read it (ENOENT)');
|
||||
const bad = [' test/b.test.ts', 'test/c.test.ts ', './test/d.test.ts', 'test/../test/e.test.ts', '/abs/test/f.test.ts', 'test\\g.test.ts', 'C:/x.test.ts', 'test//b.test.ts'];
|
||||
for (const line of bad) {
|
||||
expect(() => readFilesFrom(list(`test/b.test.ts\n${line}\n`), PLANNED), line).toThrow(`1 malformed line(s): line 2 ${JSON.stringify(line)}; fix: one repo-relative path per line`);
|
||||
}
|
||||
expect(() => readFilesFrom(list('test/b.test.ts\ntest/c.test.ts\ntest/b.test.ts\n'), PLANNED)).toThrow('line 3 repeats test/b.test.ts');
|
||||
});
|
||||
|
||||
test('the digest is the sorted list, so a reordered upload still matches', () => {
|
||||
expect(filesDigest(['test/c.test.ts', 'browse/test/a.test.ts'])).toBe(filesDigest(['browse/test/a.test.ts', 'test/c.test.ts']));
|
||||
expect(filesDigest(['test/c.test.ts'])).not.toBe(filesDigest(['test/c.test.ts', 'test/b.test.ts']));
|
||||
});
|
||||
|
||||
test('it conflicts with --quick, --windows-only, --list, --record-durations, --attribute-home and every --ci-* mode', () => {
|
||||
expect(parseCliOptions(['--files-from', 'x.txt', '--dry-run']).filesFrom).toBe('x.txt');
|
||||
expect(() => parseCliOptions(['--files-from'])).toThrow('Missing path for --files-from');
|
||||
for (const extra of [['--quick'], ['--windows-only'], ['--list'], ['--record-durations'], ['--attribute-home'], ['--ci-plan', 'p.json'], ['--ci-run', 'p.json', '--shard', '1', '--result', 'r.json'], ['--ci-verify', 'p.json', '--results', 'r']]) {
|
||||
expect(() => parseCliOptions(['--files-from', 'x.txt', ...extra]), extra[0]).toThrow('--files-from cannot be combined with --quick, --windows-only, --list, --record-durations, --attribute-home or the --ci-* modes');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('--files-from planning through the CLI', () => {
|
||||
const cli = (...args: string[]) => {
|
||||
const r = Bun.spawnSync([process.execPath, path.join(ROOT, 'scripts/test-free-shards.ts'), ...args], { cwd: ROOT, timeout: 60_000 });
|
||||
return { exit: r.exitCode, out: r.stdout.toString() + r.stderr.toString() };
|
||||
};
|
||||
|
||||
test('a dry run plans exactly the listed files and prints their digest', () => {
|
||||
const planned = collectFreeTestFiles(ROOT);
|
||||
const chosen = [planned[3]!, planned[0]!];
|
||||
const file = list(chosen.join('\n') + '\n');
|
||||
const r = cli('--files-from', file, '--dry-run');
|
||||
expect(r.exit, r.out).toBe(0);
|
||||
expect(r.out).toContain(`[test:free] files-from ${file}: 2 files, sha256=${filesDigest(chosen)}`);
|
||||
expect(r.out).toContain('Would run 2 files across');
|
||||
const listed = r.out.split('\n').filter(line => line.startsWith('Shard ') && line.includes('->')).flatMap(line => line.split('-> ')[1]!.split(', '));
|
||||
expect(listed.sort()).toEqual([...chosen].sort());
|
||||
}, 60_000);
|
||||
|
||||
test('a rejected list exits 1 before planning', () => {
|
||||
const r = cli('--files-from', list('test/not-a-real-file.test.ts\n'), '--dry-run');
|
||||
expect(r.exit).toBe(1);
|
||||
expect(r.out).toContain('1 path(s) not in the free planned set: test/not-a-real-file.test.ts');
|
||||
expect(r.out).not.toContain('Would run');
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -0,0 +1,402 @@
|
||||
/**
|
||||
* W2-A3 (fix wave 3): scripts/ubicloud/pregate-remote.sh, gstack's declared
|
||||
* remote_gate. Each case runs the real script in a fixture checkout whose
|
||||
* scripts/ubicloud/ubi-runner.sh is a stub that "creates" a local VM
|
||||
* directory, so the keyed path runs its real VM-side setup prelude, the real
|
||||
* test-free-shards.ts --files-from and the real bin/gstack-wtree, offline and
|
||||
* at no cost. Reports are judged through the exported parseRemoteReport /
|
||||
* remoteReportProblem, exactly as gstack-pregate --remote judges them.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { parseConfig } from '../lib/pregate/config';
|
||||
import { parseRemoteReport, remoteReportProblem } from '../lib/pregate/checks/lanes';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const SCRATCH = fs.mkdtempSync(path.join(os.tmpdir(), 'pregate-remote-gate-'));
|
||||
afterAll(() => fs.rmSync(SCRATCH, { recursive: true, force: true }));
|
||||
|
||||
const COPIED = ['scripts/ubicloud/pregate-remote.sh', 'bin/gstack-wtree', 'bin/gstack-path-spelling.sh',
|
||||
'scripts/test-free-shards.ts', 'scripts/test-strict-output.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/free-home-guard.ts',
|
||||
'scripts/lib/free-ci-health.ts', 'scripts/lib/windows-curation.ts', 'lib/state-root.ts', 'test/helpers/paid-test-set.ts',
|
||||
'test/helpers/touchfiles.ts', 'test/helpers/touchfiles-data.ts', 'test/helpers/test-selection.ts'];
|
||||
const TESTS: Record<string, string> = {
|
||||
'test/alpha.test.ts': "import { expect, test } from 'bun:test';\ntest('alpha', () => { expect(1).toBe(1); });\n",
|
||||
'test/beta.test.ts': "import { expect, test } from 'bun:test';\ntest('beta', () => { expect(2).toBe(2); });\n",
|
||||
'test/fails.test.ts': "import { expect, test } from 'bun:test';\ntest('fails', () => { expect(1).toBe(2); });\n",
|
||||
'test/empty.test.ts': "import 'bun:test';\n",
|
||||
'test/mover.test.ts': "import { test } from 'bun:test';\nimport * as fs from 'node:fs';\nimport * as path from 'node:path';\ntest('moves the tree', () => { fs.writeFileSync(path.join(import.meta.dir, '..', 'moved.txt'), 'x'); });\n",
|
||||
};
|
||||
|
||||
/** Stub of scripts/ubicloud/ubi-runner.sh (written into each fixture as that file). */
|
||||
const UBI_RUNNER_STUB = `#!/usr/bin/env bash
|
||||
# Stub of scripts/ubicloud/ubi-runner.sh for test/ubicloud-pregate-remote.test.ts:
|
||||
# "run" copies the checkout (tracked + untracked-unignored + .git) into a local
|
||||
# VM directory, runs setup and the command there, and prints ubi-runner's log
|
||||
# lines. STUB_FAIL picks a failure: create, quota, quota-once, ssh. STUB_HANG
|
||||
# sleeps before the command; TERM tears down as ubi-runner's trap does.
|
||||
# Every teardown is appended to $STUB_CALLS.down.
|
||||
set -uo pipefail
|
||||
[ "$1" != owner ] || { echo stub; exit 0; }
|
||||
[ "$1" = run ] || exit 2
|
||||
shift
|
||||
printf '%s\\n' "$*" >>"$STUB_CALLS"
|
||||
src=. setup="" size="" name=unnamed envs=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
-s) size=$2; shift 2 ;;
|
||||
-n) name=$2; shift 2 ;;
|
||||
--src) src=$2; shift 2 ;;
|
||||
--setup) setup=$2; shift 2 ;;
|
||||
--env) envs+=("$2"); shift 2 ;;
|
||||
--pull) shift 2 ;;
|
||||
--) shift; break ;;
|
||||
*) echo "stub: unknown option $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
cmd=$*
|
||||
log() { echo "ubi-runner: $*" >&2; }
|
||||
log "creating eu-central-h1/$name ($size)"
|
||||
calls=$(wc -l <"$STUB_CALLS")
|
||||
fail=\${STUB_FAIL:-}
|
||||
if [ "$fail" = quota ] || { [ "$fail" = quota-once ] && [ "$calls" -eq 1 ]; }; then
|
||||
log "quota refused eu-central-h1/$name ($size): it needs 16 vCPUs and the project already uses 64 of 64"
|
||||
log "create failed: vCPU quota exhausted; wait for running VMs to finish, or use fewer or smaller VMs (ci:ubicloud --vms N)"
|
||||
log "gone: eu-central-h1/$name (never existed or already destroyed)"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$fail" = create ]; then
|
||||
echo "Validation failed for following fields: size" >&2
|
||||
log "create failed"
|
||||
log "gone: eu-central-h1/$name (never existed or already destroyed)"
|
||||
exit 1
|
||||
fi
|
||||
log "ready: $name (192.0.2.1)"
|
||||
trap 'log "destroyed eu-central-h1/$name"; echo "term $name" >>"$STUB_CALLS.down"; exit 130' TERM
|
||||
vm=$STUB_VM_HOME
|
||||
dest="$vm/work/$(basename "$src")"
|
||||
rm -rf "$dest" && mkdir -p "$dest"
|
||||
(cd "$src" && { git ls-files -z -co --exclude-standard; printf '.git\\0'; } | tar --null -T - -cf -) | tar -xf - -C "$dest"
|
||||
[ -z "\${STUB_VM_EDIT:-}" ] || echo "// edited on the VM" >>"$dest/$STUB_VM_EDIT"
|
||||
for e in \${envs[@]+"\${envs[@]}"}; do printf 'export %s\\n' "$e"; done >"$vm/.ubirun-env"
|
||||
log "synced $src -> $name:$dest"
|
||||
if [ -n "$setup" ]; then
|
||||
cp "$setup" "$vm/.ubirun-setup.sh"
|
||||
log "running setup $(basename "$setup")"
|
||||
if ! (cd "$dest" && HOME="$vm" bash -c '. ~/.ubirun-env && bash ~/.ubirun-setup.sh'); then
|
||||
log "setup failed"
|
||||
log "destroyed eu-central-h1/$name"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
[ -z "\${STUB_VM_LIST_ADD:-}" ] || echo "$STUB_VM_LIST_ADD" >>"$vm/.pregate-files.txt"
|
||||
log "running on $name: $cmd"
|
||||
if [ -n "\${STUB_HANG:-}" ]; then sleep "$STUB_HANG" & wait $!; fi
|
||||
rc=0
|
||||
if [ "$fail" = ssh ]; then
|
||||
rc=255
|
||||
else
|
||||
(cd "$dest" && HOME="$vm" PATH="$STUB_BIN:$PATH" bash -c ". ~/.ubirun-env && $cmd") || rc=$?
|
||||
fi
|
||||
log "command exited $rc after 1s"
|
||||
log "destroyed eu-central-h1/$name"
|
||||
echo "exit $name" >>"$STUB_CALLS.down"
|
||||
exit "$rc"
|
||||
`;
|
||||
|
||||
interface Fixture { repo: string; vm: string; calls: string; bin: string; tree: string }
|
||||
|
||||
let counter = 0;
|
||||
function fixture(): Fixture {
|
||||
const base = path.join(SCRATCH, String(++counter));
|
||||
const repo = path.join(base, 'gstack');
|
||||
const vm = path.join(base, 'vm');
|
||||
const bin = path.join(base, 'bin');
|
||||
for (const dir of [repo, vm, bin]) fs.mkdirSync(dir, { recursive: true });
|
||||
const put = (rel: string, body: string, mode = 0o644) => {
|
||||
fs.mkdirSync(path.dirname(path.join(repo, rel)), { recursive: true });
|
||||
fs.writeFileSync(path.join(repo, rel), body, { mode });
|
||||
};
|
||||
for (const rel of COPIED) put(rel, fs.readFileSync(path.join(ROOT, rel), 'utf8'), fs.statSync(path.join(ROOT, rel)).mode & 0o777);
|
||||
for (const [rel, body] of Object.entries(TESTS)) put(rel, body);
|
||||
put('scripts/ubicloud/ubi-runner.sh', UBI_RUNNER_STUB, 0o755);
|
||||
put('scripts/ubicloud/setup-free-suite.sh', '#!/usr/bin/env bash\necho "setup-free-suite (stub)"\n[ -z "${STUB_SETUP_FAIL:-}" ] || exit 1\n[ -z "${STUB_SETUP_REBUILD:-}" ] || echo rebuilt >>dist/bundle.txt\n', 0o755);
|
||||
put('dist/bundle.txt', 'committed build output\n');
|
||||
put('.gitignore', '.context/\n');
|
||||
put('package.json', JSON.stringify({ scripts: { 'test:free': 'bun run scripts/test-free-shards.ts' } }) + '\n');
|
||||
fs.writeFileSync(path.join(bin, 'xvfb-run'), '#!/bin/sh\n[ "$1" = -a ] && shift\nexec "$@"\n', { mode: 0o755 });
|
||||
const git = (...args: string[]) => {
|
||||
const r = Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', ...args], { cwd: repo, timeout: 30_000 });
|
||||
if (r.exitCode !== 0) throw new Error(`git ${args.join(' ')}: ${r.stderr.toString()}`);
|
||||
return r.stdout.toString().trim();
|
||||
};
|
||||
git('init', '-q', '-b', 'main');
|
||||
git('add', '-A');
|
||||
git('commit', '-qm', 'fixture');
|
||||
const tree = Bun.spawnSync(['bin/gstack-wtree'], { cwd: repo, timeout: 30_000 }).stdout.toString().trim();
|
||||
return { repo, vm, calls: path.join(base, 'calls.log'), bin, tree };
|
||||
}
|
||||
|
||||
function gate(f: Fixture, files: string[] | null, env: Record<string, string | undefined> = {}) {
|
||||
const list = path.join(path.dirname(f.repo), 'files.txt');
|
||||
if (files) fs.writeFileSync(list, files.join('\n') + '\n');
|
||||
const base: Record<string, string | undefined> = { ...process.env, UBICLOUD_API_KEY: undefined, UBICLOUD_API_TOKEN: undefined, UBI_SIZE: undefined,
|
||||
GSTACK_FREE_JOBS: '2', GSTACK_FLAKE_LEDGER: path.join(path.dirname(f.repo), 'flakes.jsonl'), STUB_CALLS: f.calls, STUB_VM_HOME: f.vm, STUB_BIN: f.bin };
|
||||
if (files) Object.assign(base, { PREGATE_FILES_FILE: list, PREGATE_TREE: f.tree, PREGATE_SPEND_CAP_USD: '2' });
|
||||
const merged = Object.fromEntries(Object.entries({ ...base, ...env }).filter(([, v]) => v !== undefined)) as Record<string, string>;
|
||||
const r = Bun.spawnSync(['bash', 'scripts/ubicloud/pregate-remote.sh'], { cwd: f.repo, env: merged, timeout: 90_000 });
|
||||
const out = r.stdout.toString() + r.stderr.toString();
|
||||
return { exit: r.exitCode, out, report: parseRemoteReport(out), calls: fs.existsSync(f.calls) ? fs.readFileSync(f.calls, 'utf8').trim().split('\n') : [] };
|
||||
}
|
||||
const KEY = { UBICLOUD_API_KEY: 'offline-stub' };
|
||||
|
||||
describe('pregate-remote.sh inputs and the local tree check', () => {
|
||||
test('run bare it exits 2 before any work, naming the pregate command', () => {
|
||||
const f = fixture();
|
||||
const r = gate(f, null, KEY);
|
||||
expect(r.exit).toBe(2);
|
||||
expect(r.out).toContain('pregate-remote: PREGATE_REMOTE_USAGE: PREGATE_FILES_FILE is not set; run: gstack-pregate --stage tests --remote --spend-cap <usd> --allow-repo-commands');
|
||||
expect(r.out).toContain('see docs/pregate.md#remote-gate');
|
||||
expect(r.calls).toEqual([]);
|
||||
expect(fs.existsSync(path.join(f.repo, '.context'))).toBe(false);
|
||||
}, 60_000);
|
||||
|
||||
test('a working tree that is not PREGATE_TREE exits 3 before any upload, with both prefixes and the rerun command', () => {
|
||||
const f = fixture();
|
||||
const r = gate(f, ['test/alpha.test.ts'], { ...KEY, PREGATE_TREE: 'f'.repeat(40) });
|
||||
expect(r.exit).toBe(3);
|
||||
expect(r.out).toContain(`PREGATE_REMOTE_TREE_MISMATCH: the working tree is ${f.tree.slice(0, 12)}, not the planned tree ffffffffffff`);
|
||||
expect(r.out).toContain('nothing uploaded, nothing spent');
|
||||
expect(r.out).toContain('rerun: gstack-pregate --stage tests --remote --spend-cap <usd> --allow-repo-commands');
|
||||
expect(r.calls).toEqual([]);
|
||||
expect(r.report).toBeNull();
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
|
||||
test('a passing run prints the VM tree, the counts and billed spend; pregate clears it', () => {
|
||||
const f = fixture();
|
||||
const r = gate(f, ['test/alpha.test.ts', 'test/beta.test.ts'], KEY);
|
||||
expect(r.exit, r.out).toBe(0);
|
||||
expect(r.report).toEqual({ tree: f.tree, tests: 2, failed: 0, spent: '0.01' });
|
||||
expect(remoteReportProblem(r.report!, f.tree)).toBeNull();
|
||||
expect(r.out).toContain("2 files on one standard-16 VM; worst case $0.38 under the cap $2; the VM is told to stop after 2520s (the cap's 45 min wall minus 3 min for teardown)");
|
||||
expect(r.out).toMatch(/phases: create \d+s, setup \d+s, run \d+s, teardown \d+s; total \d+s \(target 600s\)/);
|
||||
expect(r.out.trim().split('\n').at(-1)).toMatch(/^PREGATE_REMOTE: /);
|
||||
expect(r.calls).toHaveLength(1);
|
||||
for (const arg of ['-s standard-16 ', '--env GSTACK_EXPECT_BINARIES=1', '--env GSTACK_FREE_RETRY_FLAKY=1', '--env GSTACK_FREE_JOBS=2', '-- bash ~/.pregate-run.sh']) expect(r.calls[0]).toContain(arg);
|
||||
expect(fs.readFileSync(path.join(f.vm, '.pregate-files.txt'), 'utf8')).toBe('test/alpha.test.ts\ntest/beta.test.ts\n');
|
||||
}, 90_000);
|
||||
|
||||
test('setup rebuilding committed build output (as CI does) is printed; the reported tree stays the uploaded one', () => {
|
||||
const f = fixture();
|
||||
const r = gate(f, ['test/alpha.test.ts'], { ...KEY, STUB_SETUP_REBUILD: '1' });
|
||||
expect(r.exit, r.out).toBe(0);
|
||||
expect(r.out).toContain(' setup rebuilt (committed build output, as the CI free lane rebuilds it): M\tdist/bundle.txt');
|
||||
expect(r.report).toEqual({ tree: f.tree, tests: 1, failed: 0, spent: '0.01' });
|
||||
}, 90_000);
|
||||
|
||||
test('a failing test reports failed>0 and exits 1; a file with zero tests reports tests=0; neither clears', () => {
|
||||
const failing = gate(fixture(), ['test/alpha.test.ts', 'test/fails.test.ts'], KEY);
|
||||
expect(failing.exit, failing.out).toBe(1);
|
||||
expect(failing.report).toMatchObject({ tests: 2, failed: 1 });
|
||||
expect(remoteReportProblem(failing.report!, failing.report!.tree)).toBe('the remote gate reported 1 failed of 2');
|
||||
const empty = gate(fixture(), ['test/empty.test.ts'], KEY);
|
||||
expect(empty.report, empty.out).toMatchObject({ tests: 0, failed: 0 });
|
||||
expect(remoteReportProblem(empty.report!, empty.report!.tree)).toBe('the remote gate reported 0 tests');
|
||||
}, 90_000);
|
||||
|
||||
test('the cap refuses before any VM is created, at the price of the size actually used', () => {
|
||||
const f = fixture();
|
||||
const low = gate(f, ['test/alpha.test.ts'], { ...KEY, PREGATE_SPEND_CAP_USD: '0.1' });
|
||||
expect(low.exit).toBe(5);
|
||||
expect(low.out).toContain('PREGATE_REMOTE_CAP_TOO_LOW: worst case $0.17 (20 min x $0.00831/min on standard-16) exceeds the spend cap $0.1');
|
||||
expect(low.out).toContain('nothing created, nothing spent');
|
||||
expect(low.out).toContain('rerun with --spend-cap 0.17 or more');
|
||||
const big = gate(f, ['test/alpha.test.ts'], { ...KEY, PREGATE_SPEND_CAP_USD: '0.5', UBI_SIZE: 'standard-60' });
|
||||
expect(big.exit).toBe(5);
|
||||
expect(big.out).toContain('worst case $0.63 (20 min x $0.03117/min on standard-60) exceeds the spend cap $0.5');
|
||||
const unpriced = gate(f, ['test/alpha.test.ts'], { ...KEY, UBI_SIZE: 'burstable-1' });
|
||||
expect(unpriced.exit).toBe(5);
|
||||
expect(unpriced.out).toContain('PREGATE_REMOTE_NO_PRICE: no price for UBI_SIZE=burstable-1');
|
||||
const noCap = gate(f, ['test/alpha.test.ts'], { ...KEY, PREGATE_SPEND_CAP_USD: '' });
|
||||
expect(noCap.exit).toBe(2);
|
||||
expect([low, big, unpriced, noCap].every(r => r.report === null)).toBe(true);
|
||||
expect(fs.existsSync(f.calls)).toBe(false);
|
||||
}, 90_000);
|
||||
|
||||
test('create, setup and SSH failures exit 4 with the ubi-runner message, the VM, its teardown line, the log and spent=', () => {
|
||||
const create = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_FAIL: 'create' });
|
||||
expect(create.exit, create.out).toBe(4);
|
||||
expect(create.out).toContain('PREGATE_REMOTE_INFRA: infrastructure failure (ubi-runner exit 1): ubi-runner: create failed');
|
||||
expect(create.out).toMatch(/VM: eu-central-h1\/(ubirun-stub-\d+-[0-9a-f]{8}); ubi-runner: gone: eu-central-h1\/\1 \(never existed or already destroyed\)/);
|
||||
expect(create.out).toMatch(/VM (ubirun-stub-\d+-[0-9a-f]{8}); if this run is killed before its teardown line, destroy it with: bash scripts\/ubicloud\/ubi-runner\.sh down \1\n/);
|
||||
expect(create.out).toMatch(/log: \S+\/\.context\/pregate-remote\/\S+\/run\.log/);
|
||||
expect(create.out).toContain('spent=0.00 (0 billed min on standard-16)');
|
||||
const setup = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_SETUP_FAIL: '1' });
|
||||
expect(setup.exit, setup.out).toBe(4);
|
||||
expect(setup.out).toContain('ubi-runner: setup failed');
|
||||
expect(setup.out).toMatch(/; ubi-runner: destroyed eu-central-h1\/ubirun-stub-\d+-[0-9a-f]{8}/);
|
||||
expect(setup.out).toContain('spent=0.01 (1 billed min on standard-16)');
|
||||
const ssh = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_FAIL: 'ssh' });
|
||||
expect(ssh.exit, ssh.out).toBe(4);
|
||||
expect(ssh.out).toContain('infrastructure failure (ubi-runner exit 255): ubi-runner: command exited 255 after 1s');
|
||||
expect([create, setup, ssh].every(r => r.report === null)).toBe(true);
|
||||
}, 90_000);
|
||||
|
||||
test('a quota refusal is retried once, then the run stops', () => {
|
||||
const once = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_FAIL: 'quota-once', PREGATE_REMOTE_QUOTA_WAIT_S: '0' });
|
||||
expect(once.exit, once.out).toBe(0);
|
||||
expect(once.out).toContain('vCPU quota refused the VM; one retry in 0 s');
|
||||
expect(once.calls).toHaveLength(2);
|
||||
const twice = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_FAIL: 'quota', PREGATE_REMOTE_QUOTA_WAIT_S: '0' });
|
||||
expect(twice.exit, twice.out).toBe(4);
|
||||
expect(twice.calls).toHaveLength(2);
|
||||
expect(twice.out).toContain('create failed: vCPU quota exhausted');
|
||||
}, 90_000);
|
||||
|
||||
test('the VM refuses a tree that differs after unpack, a planned set that differs from the list, and a tree that moved during the run', () => {
|
||||
const edited = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_VM_EDIT: 'test/alpha.test.ts' });
|
||||
expect(edited.exit, edited.out).toBe(3);
|
||||
expect(edited.out).toContain('PREGATE_REMOTE_VM_TREE: the VM refused before running tests: PREGATE_VM_TREE_MISMATCH vm=');
|
||||
expect(edited.out).toContain('no tests ran; spent=0.01');
|
||||
const listed = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_VM_LIST_ADD: 'test/beta.test.ts' });
|
||||
expect(listed.exit, listed.out).toBe(3);
|
||||
expect(listed.out).toContain('PREGATE_VM_PLAN_MISMATCH planned=');
|
||||
const moved = gate(fixture(), ['test/alpha.test.ts', 'test/mover.test.ts'], KEY);
|
||||
expect(moved.exit, moved.out).toBe(3);
|
||||
expect(moved.out).toContain("PREGATE_REMOTE_TREE_MOVED: the VM's tree moved during the run");
|
||||
expect(moved.out).toContain(' moved: A\tmoved.txt');
|
||||
expect([edited, listed, moved].every(r => r.report === null)).toBe(true);
|
||||
}, 90_000);
|
||||
|
||||
test('killing the gate\'s process group (pregate\'s timeout) never cuts teardown short; TERM under an outer deadline tears the VM down; an edit to the script mid-run changes nothing', async () => {
|
||||
const run = (f: Fixture) => {
|
||||
const list = path.join(path.dirname(f.repo), 'files.txt');
|
||||
fs.writeFileSync(list, 'test/alpha.test.ts\n');
|
||||
return Bun.spawn(['bash', 'scripts/ubicloud/pregate-remote.sh'], { cwd: f.repo, detached: true, stdout: 'pipe', stderr: 'pipe',
|
||||
env: { ...process.env, ...KEY, UBI_SIZE: '', GSTACK_FREE_JOBS: '2', STUB_CALLS: f.calls, STUB_VM_HOME: f.vm, STUB_BIN: f.bin, STUB_HANG: '4',
|
||||
PREGATE_FILES_FILE: list, PREGATE_TREE: f.tree, PREGATE_SPEND_CAP_USD: '2', PREGATE_TIMEOUT_S: '900' } as Record<string, string> });
|
||||
};
|
||||
const until = async (cond: () => boolean, ms: number) => {
|
||||
const deadline = Date.now() + ms;
|
||||
while (!cond()) { if (Date.now() > deadline) throw new Error('condition not met in time'); await Bun.sleep(50); }
|
||||
};
|
||||
const vmLog = (f: Fixture) => { try { return fs.readFileSync(fs.readdirSync(path.join(f.repo, '.context/pregate-remote')).map(d => path.join(f.repo, '.context/pregate-remote', d, 'run.log'))[0]!, 'utf8'); } catch { return ''; } };
|
||||
const killed = fixture();
|
||||
const a = run(killed);
|
||||
await until(() => vmLog(killed).includes('ubi-runner: running on'), 30_000);
|
||||
process.kill(-a.pid, 'SIGKILL');
|
||||
await a.exited;
|
||||
await until(() => fs.existsSync(killed.calls + '.down'), 30_000);
|
||||
expect(fs.readFileSync(killed.calls + '.down', 'utf8')).toMatch(/^exit ubirun-stub-\d+-[0-9a-f]{8}\n$/);
|
||||
const termed = fixture();
|
||||
const b = run(termed);
|
||||
await until(() => vmLog(termed).includes('ubi-runner: running on'), 30_000);
|
||||
process.kill(b.pid, 'SIGTERM');
|
||||
expect(await b.exited).toBe(130);
|
||||
expect(await new Response(b.stdout).text()).toContain('interrupted: ubi-runner is tearing the VM down on its own');
|
||||
await until(() => fs.existsSync(termed.calls + '.down'), 10_000);
|
||||
expect(fs.readFileSync(termed.calls + '.down', 'utf8')).toMatch(/^term ubirun-stub-\d+-[0-9a-f]{8}\n$/);
|
||||
const edited = fixture();
|
||||
const c = run(edited);
|
||||
await until(() => vmLog(edited).includes('ubi-runner: running on'), 30_000);
|
||||
const script = path.join(edited.repo, 'scripts/ubicloud/pregate-remote.sh');
|
||||
fs.writeFileSync(script, '#!/usr/bin/env bash\n# an edit that shifts every byte offset\n' + fs.readFileSync(script, 'utf8').split('\n').slice(1).join('\n'));
|
||||
expect(await c.exited).toBe(0);
|
||||
expect((await new Response(c.stdout).text()).trim().split('\n').at(-1)).toMatch(new RegExp(`^PREGATE_REMOTE: tree=${edited.tree} tests=1 failed=0 spent=`));
|
||||
}, 90_000);
|
||||
|
||||
test('an enclosing pregate deadline shorter than the wall sets when the VM is told to stop; one too short refuses before any VM', () => {
|
||||
const f = fixture();
|
||||
const viaEnv = gate(f, ['test/alpha.test.ts'], { ...KEY, PREGATE_TIMEOUT_S: '900' });
|
||||
expect(viaEnv.exit, viaEnv.out).toBe(0);
|
||||
expect(viaEnv.out).toMatch(/the VM is told to stop after (76[0-5])s \(PREGATE_TIMEOUT_S 900s minus 120 s teardown reserve and 15 s margin\)/);
|
||||
const short = gate(f, ['test/alpha.test.ts'], { ...KEY, PREGATE_TIMEOUT_S: '400' });
|
||||
expect(short.exit).toBe(5);
|
||||
expect(short.out).toMatch(/PREGATE_REMOTE_TIMEOUT_TOO_SHORT: PREGATE_TIMEOUT_S 400s leaves 26\ds for the VM after 120 s reserved for teardown and 15 s margin \(at least 300 s needed\)/);
|
||||
expect(short.out).toContain('nothing created, nothing spent');
|
||||
expect(fs.readFileSync(f.calls, 'utf8').trim().split('\n')).toHaveLength(1);
|
||||
// Run under a process named gstack-pregate, the deadline is read from its --timeout (600 s when it passes none).
|
||||
const shim = path.join(path.dirname(f.repo), 'gstack-pregate');
|
||||
fs.writeFileSync(shim, '#!/usr/bin/env bash\nbash scripts/ubicloud/pregate-remote.sh\n', { mode: 0o755 });
|
||||
const run = (args: string[], via = '"$0" "$@"; exit $?') => {
|
||||
const r = Bun.spawnSync(['bash', '-c', via, shim, ...args], { cwd: f.repo, timeout: 90_000, env: { ...process.env, ...KEY, UBI_SIZE: '', GSTACK_FREE_JOBS: '2',
|
||||
STUB_CALLS: f.calls, STUB_VM_HOME: f.vm, STUB_BIN: f.bin, PREGATE_FILES_FILE: path.join(path.dirname(f.repo), 'files.txt'), PREGATE_TREE: f.tree, PREGATE_SPEND_CAP_USD: '2' } as Record<string, string> });
|
||||
return r.stdout.toString() + r.stderr.toString();
|
||||
};
|
||||
expect(run(['--stage', 'tests', '--remote', '--timeout', '1200'])).toMatch(/told to stop after 106[0-5]s \(gstack-pregate --timeout 1200s minus/);
|
||||
expect(run(['--stage', 'tests', '--remote'])).toMatch(/told to stop after 46[0-5]s \(gstack-pregate --timeout 600s minus/);
|
||||
// An ancestor that only mentions gstack-pregate (a shell line running it beside the gate) is not the enclosing pregate.
|
||||
expect(run([], ': bin/gstack-pregate --timeout 400; bash scripts/ubicloud/pregate-remote.sh; exit $?')).toContain("told to stop after 2520s (the cap's 45 min wall minus 3 min for teardown)");
|
||||
}, 120_000);
|
||||
|
||||
test('the VM environment is bun run test:ubicloud\'s (scripts/ubicloud/test-free.sh), so the two cannot drift', () => {
|
||||
const script = fs.readFileSync(path.join(ROOT, 'scripts/ubicloud/pregate-remote.sh'), 'utf8');
|
||||
const wrapper = fs.readFileSync(path.join(ROOT, 'scripts/ubicloud/test-free.sh'), 'utf8');
|
||||
const flags = [...wrapper.matchAll(/--(?:env|pull) "?[^\s"]+(?: [^\s"]+)?/g)].map(m => m[0]).filter(flag => !flag.includes('GSTACK_FREE_JOBS'));
|
||||
expect(flags.length).toBeGreaterThanOrEqual(5);
|
||||
for (const flag of flags) expect(script).toContain(flag);
|
||||
expect(wrapper).toContain('--setup "$here/setup-free-suite.sh"');
|
||||
expect(script).toContain("'bash scripts/ubicloud/setup-free-suite.sh'");
|
||||
expect(wrapper).toContain('xvfb-run -a bun run test:free');
|
||||
expect(script).toContain("'xvfb-run -a bun run test:free --files-from ~/.pregate-files.txt'");
|
||||
});
|
||||
});
|
||||
|
||||
describe('pregate-remote.sh keyless path', () => {
|
||||
test('without a key it runs the list locally in parallel shards and reports the tree it tested with spent=0', () => {
|
||||
const f = fixture();
|
||||
const r = gate(f, ['test/alpha.test.ts', 'test/beta.test.ts']);
|
||||
expect(r.exit, r.out).toBe(0);
|
||||
expect(r.out).toContain('no UBICLOUD_API_KEY: running 2 files locally in parallel shards');
|
||||
expect(r.out).toMatch(/\nbackend=local \(\d+ cores; not the CI environment\)\nPREGATE_REMOTE: /);
|
||||
expect(r.report).toEqual({ tree: f.tree, tests: 2, failed: 0, spent: '0' });
|
||||
expect(remoteReportProblem(r.report!, f.tree)).toBeNull();
|
||||
expect(fs.existsSync(f.calls)).toBe(false);
|
||||
}, 90_000);
|
||||
|
||||
test('a tree that moves during the local run fails without a report', () => {
|
||||
const r = gate(fixture(), ['test/alpha.test.ts', 'test/mover.test.ts']);
|
||||
expect(r.exit, r.out).toBe(3);
|
||||
expect(r.out).toContain('PREGATE_REMOTE_TREE_MOVED: the working tree moved during the run');
|
||||
expect(r.out).toContain(' moved: A\tmoved.txt');
|
||||
expect(r.report).toBeNull();
|
||||
}, 90_000);
|
||||
|
||||
test('paid test files in the selection are named and left to the paid PR gate; a paid-only selection reports tests=0 and cannot clear', () => {
|
||||
const f = fixture();
|
||||
const paid = ['test/skill-e2e-a.test.ts', 'test/skill-e2e-b.test.ts', 'test/skill-llm-eval.test.ts', 'test/codex-e2e-x.test.ts', 'test/carve-section-loading-y.test.ts', 'test/skill-routing-e2e.test.ts'];
|
||||
const mixed = gate(f, ['test/alpha.test.ts', ...paid]);
|
||||
expect(mixed.exit, mixed.out).toBe(0);
|
||||
expect(mixed.out).toContain('6 paid test file(s) are covered by the paid PR gate, not the free tier: test/skill-e2e-a.test.ts, test/skill-e2e-b.test.ts, test/skill-llm-eval.test.ts, test/codex-e2e-x.test.ts, test/carve-section-loading-y.test.ts ...');
|
||||
expect(mixed.out).toContain('no UBICLOUD_API_KEY: running 1 files locally in parallel shards');
|
||||
expect(mixed.report).toEqual({ tree: f.tree, tests: 1, failed: 0, spent: '0' });
|
||||
const keyed = gate(f, ['test/beta.test.ts', 'test/skill-e2e-a.test.ts'], KEY);
|
||||
expect(keyed.exit, keyed.out).toBe(0);
|
||||
expect(fs.readFileSync(path.join(f.vm, '.pregate-files.txt'), 'utf8')).toBe('test/beta.test.ts\n');
|
||||
const only = gate(f, paid.slice(0, 2));
|
||||
expect(only.exit, only.out).toBe(0);
|
||||
expect(only.out).toContain('PREGATE_REMOTE_PAID_ONLY: every selected file is a paid test file');
|
||||
expect(only.report).toEqual({ tree: f.tree, tests: 0, failed: 0, spent: '0' });
|
||||
expect(remoteReportProblem(only.report!, f.tree)).toBe('the remote gate reported 0 tests');
|
||||
}, 90_000);
|
||||
|
||||
test('a list test-free-shards rejects exits 1 without a report', () => {
|
||||
const r = gate(fixture(), ['test/alpha.test.ts', 'test/missing.test.ts']);
|
||||
expect(r.exit, r.out).toBe(1);
|
||||
expect(r.out).toContain('1 path(s) not in the free planned set: test/missing.test.ts');
|
||||
expect(r.out).toContain('PREGATE_REMOTE_NO_RESULT: test-free-shards exited 1 without a FILES-FROM RESULT line');
|
||||
expect(r.report).toBeNull();
|
||||
}, 90_000);
|
||||
});
|
||||
|
||||
describe('.gstack/pregate.json declares the remote gate', () => {
|
||||
test('remote_gate is this script, and the config still validates', () => {
|
||||
const { config, errors } = parseConfig(fs.readFileSync(path.join(ROOT, '.gstack/pregate.json'), 'utf8'));
|
||||
expect(errors).toEqual([]);
|
||||
expect(config.remote_gate).toBe('bash scripts/ubicloud/pregate-remote.sh');
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user