CalvinBackup/gstack
1
0 Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-11-02 06:54:50 +01:00
Code Issues Packages Projects Releases Wiki Activity

v1.91.378.0 stamp: train gate.train-5b1b9be84164-5ba425af4b9a-556785 S2 (#3699)

Browse Source
This commit is contained in:
garrytan committed 2000-01-01 00:00:00 +00:00
2 parents 291dee91a7 c830c9b8ba
commit f44c631cd0
68 files changed
+5457 -201
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CHANGELOG.md
+90
View File
@@ -1,5 +1,95 @@
# Changelog
## [1.91.378.0] - 2026-10-31
**The CSO runtime image workflow passes again after Docker moved its `scout-sbom-indexer:1` tag.** All 10 `validate-native` jobs in Stage CSO Runtime Images were failing with `SOURCE_TAG_MOVED: docker.io/docker/scout-sbom-indexer:1` on every PR that touched CSO paths. Docker pushed release 1.29.0 to the floating `:1` tag on 2026-10-29. The reviewed index `sha256:4b67f29e…` is Docker's 1.24.0 release, and the `1.24.0` tag still points at it. So the SBOM generator source now names `:1.24.0`. The pinned digests and the build-inputs revision are byte-identical, no requalification trigger changes, and the 10 qualified runtimes stay current.
### Changed
- **`lib/cso/images/build-inputs.json`**: `sbomGenerator.source` is `docker.io/docker/scout-sbom-indexer:1.24.0` instead of `:1`.
- **`scripts/cso-image-matrix.ts`**: every pinned family (SBOM generator, base images, uv) must name an exact release in its source tag: `<major>.<minor>.<patch>`, or `<major>.<minor>` for the PostgreSQL base, whose releases have two parts. A floating major or minor tag (`:1`, `:1.24`, `python:3.13-slim`), `latest` or another alias fails with `FLOATING_<FAMILY>_SOURCE` before any build. An exact tag can still be re-pushed upstream (Docker's official images rebuild them when their Debian base changes), and `SOURCE_TAG_MOVED` still reports that. Moving the generator to 1.29.0 is a separate change that needs a new build-inputs revision and a requalified catalog.
### Tests
- `test/cso-distribution.test.ts`: `:1`, `:1.24`, `:1.24-slim`, `:latest` and `:stable` are refused for the SBOM generator and for a base image; PostgreSQL accepts `17.2` but not `17`, Python refuses `3.13-slim-bookworm`; and the committed inputs build all 10 rows with the `1.24.0` source.
## [1.91.377.0] - 2026-10-31
**A passed GitHub Actions run can now clear the remote test gate for the same tree, so a repository stops running its full suite twice per slot.** gbrain paid a 30–45 minute Ubicloud gate and then an 18–20 minute full-ci dispatch on the same tree in every serial slot (GBRA-108: #7872's slot took 5.6 h). `gstack-pregate clear --remote pregate.json/remote_gate --run <id> --run <id>` accepts one verified run per profile the base branch declares. gstack checks each run against the runner's own record, never against a pasted claim. It reads the GitHub API's run and jobs, the tree git computes for the run's head, and the `tree.json` the run uploaded. Nothing is skipped: the tests still run once, in CI. A repository that declares no profiles is unchanged.
### Added
- **`gstack-pregate clear --run <id|url>`** (repeatable; `lib/run-evidence.ts`, docs/pregate.md#run-evidence). A run is accepted only when it belongs to this repository, ran a `push` or `workflow_dispatch` event of the profile's workflow, and its newest attempt succeeded with every named job passing exactly once. Its `tree.json` must name the run, attempt, event and head and record the head's tree, and it must be fresh (`max_age_hours`, default 24) and on the profile's OS. It is refused if any run of that workflow on the head that started at or after it failed, was cancelled or is still running, or if the run changed attempts while being checked. All runs must be on one tree: pregate.json's, or a stamp-only predecessor under the existing tree-receipt rule. Their `files` must cover the obligation's selected tests plus every file the declared `inventory` command lists for that tree. Refusals print `REUSE: refused reason=<code>`; an accepted clear prints `REUSE: accepted`, and the receipt records `runs=<id>/<attempt> profiles= heads= verified=`.
- **`run_profiles` and `remote_gate_runs`** in `.gstack/pregate.json`, read only from `origin/<base>`. A user-owned policy that sets them is refused, because nothing would read them there. `gstack-pregate clear` clears a lane named by a profile only through verified runs, so its `--run-url` is verified the same way, and `--bundle-lane` is refused. `gstack-ship-queue ready --dispatch-ci` still records a plain run URL for such a lane, and `verify` does not yet re-check it (docs/pregate.md#run-evidence). When the base config cannot be read, any lane's `--run-url` or `--bundle-lane` clear refuses, since the base may profile it. When the GitHub origin cannot be read, a profiled lane refuses rather than fall back to an unverified record.
### Changed
- **`gstack-pregate verify`** stays offline. It also refuses a run receipt recorded for another tree unless its stamp-only substitution is recorded.
### Tests
- `test/run-evidence.test.ts`: every refusal reason, including the re-run cases (a red re-run of an older run vetoes; re-running the candidate does not hide a failure in between), a run that changes attempts while being checked, malformed counts and paths, and profile validation.
- `test/pregate-clear-runs.test.ts`: on a fixture repository, clear on the same tree, verify offline, and bind the ship receipt. Also covers a stamp-only predecessor, a non-stamp difference, uncovered files, runs on different trees, base mismatch, repeated runs, the profile-to-run assignment, the profiled-lane matrix (`--run-url`, `--bundle-lane`, `--run`), a non-GitHub origin failing closed, an origin that `url.insteadOf` rewrites to a proxy (Capy's git.capy.ai) still read as the GitHub repository, and an unreadable base config refusing `--run-url` on every lane. The `--run-url` fallback test failed before the fix.
## [1.91.376.0] - 2026-10-31
bump: minor
---
**A remote verify now survives a dead local runner, and a long verify can resume from the tiers that already passed (GSTA-158 P3).** Before, a Capy machine that went to sleep or a runner that got killed lost the whole remote run, and the paid VM ran on until its TTL. Now `ubi-runner run` and `verify` run each phase detached on the VM, and running the same command again reattaches to the live VM. A verify can be split into tiers, and a rerun skips the tiers that passed.
### Added
- **Reattach by run key.** The key covers the checkout's content, the command, setup, env, a hash of each `--pass` value, size, image, origin/HEAD and the runner itself. A rerun prints `ubi-runner: reattached to <vm> created_at=<epoch> key=<k12>`, replays what the phase already printed and follows it from there. A finished setup is not rerun. `UBI_ATTACH_BUSY` means another live runner owns the VM. Exit 125 `UBI_ATTACH_LOST` means the remote session died without an exit status.
- **Reattach only while the TTL lasts.** A reattach needs the VM's remaining TTL to cover the run still to do (each unfinished phase's `UBI_RUN_TIMEOUT` bound) plus `UBI_REATTACH_MARGIN` (900 s). Otherwise the runner prints `UBI_REATTACH_TTL_SHORT` with the numbers, destroys the old VM and starts a fresh one. Tier checkpoints carry over to it.
- **Verify tiers.** `.gstack/ubicloud.json` can list `tiers: [{name, command}]` in place of `command`. Each tier writes a start, pass or fail checkpoint under `UBI_CHECKPOINT_DIR`. A rerun prints `UBI_VERIFY_RESUME:` per tier and reuses passes younger than `UBI_CHECKPOINT_MAX_AGE` whose logs still hash. It reruns a recorded fail only with `--rerun-failed`. The last line is `UBI_VERIFY: …`.
- **`attach NAME`, `run --probe` and `--key-input`.**
### Changed
- **HUP once a phase is live detaches** (`UBI_DETACHED`, exit 129, `UBICLOUD: left=1 … detached=1`): the VM keeps running under its TTL. INT and TERM still tear it down.
- **`gstack-pregate --stage tests --remote` reattaches** to the VM a killed run left. It bills `spent=` from the original create time, and a cap that is already spent refuses with `PREGATE_REMOTE_CAP_SPENT`. The `PREGATE_REMOTE:` receipt is unchanged.
## [1.91.375.0] - 2026-10-31
**A /ship subagent now stops once its READY is sent, so CI and review callbacks on a frozen head stop waking it.** Finished /ship subagents kept getting woken by checks and reviews on a PR they had already READY'd. Each wake cost tokens, and some acted on a head that was supposed to be frozen. The parent now subscribes to the PR first, so red checks and human review still reach an owner, and then stops the subagent.
### Changed
- **`capy/skill/gstack/SKILL.md`** (Plan, approve, ship): the parent subscribes to the PR when the /ship subagent opens it and keeps that subscription until the PR merges or closes. It stops the subagent at READY and owns the PR from then on. A stopped /ship subagent woken by a check, review or merge notice reports one line (what woke it, its pin, the PR head) and ends the turn, while a message from its parent is an instruction it carries out.
- **`docs/captain.md`** (the lane's /ship sequence): the host-neutral rule (the parent owns red checks, review, the slot stamp, re-READY and receipt carry after READY), with a pointer to the Capy mechanics. **`captain/templates/brief.md`**: one pointer line.
- **`test/fixtures/context-budget.json`**: the Capy skill's eager ceiling moves from 5136 to 5312 for the 176 tokens it grew (5136 to 5312, measured with `lib/context-bill`), set at the captured value as main's last raise was, rather than a fresh 10% headroom.
## [1.91.374.0] - 2026-10-31
**/ship pauses on a red base instead of failing.** When main is red for a reason outside a PR, /ship's Step 5 used to fail or ask the fix/TODO/blame/skip question, and unattended lanes either burned runs or held by hand. Truther's two main-red fixes (#815 and the cover-contract branch) each failed only on the other's broken cases, so neither could pass /ship alone until a Captain folded them into #818 by hand (GSTA-157, from the 2026-10-29 speed feedback).
### Added
- **`gstack-base-red classify`** (bin/gstack-base-red, lib/base-red.ts): reads a new `base_red: { setup, select, timeout_sec }` block from the base branch's ship policy, then calls a failure base-red only when every failure the lane reported is named, the PR touches neither the failing test file nor any path its signature names, and `setup` then `select {files}` fails with the same test id and first error line at the base head and at HEAD in scratch worktrees under the state root. Anything else stays in-branch and /ship triages exactly as before. `--pause` writes a pause record (repository and branch hashed, 24-hour expiry).
- **`gstack-base-red wait --pause <file>`**: on each new base head it reruns the paused files there and exits 0 (`BASE_GREEN`) on the first head that passes; a head that still fails keeps it waiting. Exits 3 on `BASE_RED_WAIT_TIMEOUT` or `BASE_RED_PAUSE_EXPIRED`.
- **Conditional receipts**: `gstack-ship-receipt write --base-red <pause file>` adds a `base_red` block; `result` then prints `SHIP_PAUSED: base-red …` with `status=incomplete`. `bindReceipt` refuses a conditional receipt by default (`QUEUE_RECEIPT_CONDITIONAL`), so `ready`, `train build`, `verify --train` and `carry` never accept one.
- **Policy key `base_red`** (`BASE_RED_KEY_VERSION`) with validation: `select` needs a `{files}` slot, commands are one line under 512 characters, `timeout_sec` is 10–7200.
### Changed
- **/ship Step 5** (ship/sections/tests.md.tmpl) runs `classify` before the T3 question when T1 calls every failure pre-existing, carries base-red cases as that run's only waived failures into Step 16, and documents `/ship --resume <record>` after `BASE_GREEN` (Step 5 reruns on HEAD merged with the new base in a scratch worktree; a green run rewrites the receipt without `base_red`). docs/captain.md names the paused state.
### Tests
- `test/base-red.test.ts`: base-red on a real fixture with a bare origin and no leftover worktree; in-branch without running anything when the PR touches the test or a path its signature names; in-branch when the test passes on the base or fails there differently; unconfigured, unnamed and failing-setup cases; the hashed pause path; conditional receipts refused by default; `result` reports `SHIP_PAUSED` and `status=incomplete`; `carry` refuses; `wait` skips a still-red base move and returns on the first green head, and expires or times out; policy validation.
## [1.91.373.0] - 2026-10-31
- **No-gstack arm in every head-to-head eval.** `/eval-plan` preregistrations now say in a required `## Comparison` section whether any arm loads gstack or another skill stack (`stack: loaded`) or none does (`stack: none` with a `why:` line). A loaded comparison must add a `## No-gstack arm` section and one `<model>#no-gstack` arm per model. That arm uses the same model and inputs, a plain prompt with no skill text, and the same panel and judge, and its result is reported beside the variants. `gstack-eval-plan prereg --check` refuses a loaded comparison without it (`PREREG_NO_GSTACK_ARM`). Arms are now identities `<model>[#<loadout>]`, priced by the model, so the pilot reserves a variant and its no-gstack arm on the same model separately; a repeated identity is refused (`EVAL_ARM_DUPLICATE`). The rule is written up in `docs/TESTING_INTERNALS.md`.
## [1.91.372.0] - 2026-10-31
**A test that fails once in CI and passes when run again no longer turns the job red, and every such flake is recorded for the daily flaky-fix pass.** `gstack-flaky run` wraps any test command that writes a JUnit report (Playwright, `bun test`, jest, vitest, pytest). When a few tests fail, it runs the command once more. A test that failed and then passes is a flaky pass, so the job stays green, an Actions `::warning` and a step-summary row name it, and one line goes on a ledger. A rerun counts only when the runner itself reports a full green run, so a real failure, a coverage threshold, a test the first run never reached, a crash or a cancelled job stays red. Before this, each one-off flake meant rerunning the whole CI job by hand (five times in Truther's last evidence window, three of them for one test).
### Added
- **`bin/gstack-flaky run --junit <file.xml> -- <command>`**: reruns the whole command once when at most `--max-reruns` (3) tests failed and the runner made no retries of its own. `--rerun-scope test` opts in to rerunning only the failed tests (Playwright `--grep`/`--project`, bun `--test-name-pattern`), for suites with no coverage gate and no fail-on-flaky policy; it still falls back to the whole command when it sees an early stop, a coverage table or the runner's own retries.
- **Ledger** at `GSTACK_FLAKE_LEDGER_JUNIT` (default `<state root>/flake-ledger-junit.jsonl`), kept apart from the free suite's file-level ledger: one redacted line per flaky pass and per test that failed its rerun, with tree, commit, branch, run URL, attempts, scope and the first lines of the error.
- **`gstack-flaky ids --junit <file>`** prints the test ids a report holds (`<file>::<project>::<describe > title>`).
- Exit codes: 0 pass (flaky passes included), 1 a test failed, 2 usage, 3 no usable report, a crash or a killed runner, each with a `FLAKY_<CODE>` and a `fix:` line. A missing, stale, empty or malformed report never passes, and an existing `--junit` file that is not a JUnit report is refused rather than deleted.
### Docs
- `docs/flaky.md`: the GitHub Actions setup, runners, rerun scope and its preconditions, rules, output and exit codes. `docs/TESTING_INTERNALS.md#flake-ledger` points other repositories to it.
### Tests
- `test/flaky-run.test.ts` with captured Playwright 1.64 and bun 1.4 JUnit reports under `test/fixtures/junit/`: flaky pass, real red, rerun scopes, coverage-threshold and early-stop runs, unreported failures, cancelled runs, report deletion and ledger redaction.
### Gate integrity
- `test/helpers/free-fixtures-data.ts`: registers `test/fixtures/junit/**` to `test/flaky-run.test.ts`. No assertion, threshold or skip changed.
## [1.91.371.0] - 2026-10-31
**A PR push in a repository with a merge train no longer waits ten minutes for a check that does no work.** Under a train policy every PR push runs E2E Evals in `pr-summary` mode, which plans no paid slices, yet its `readiness` job still waited out the 600-second quiet window and green Free Tests before the run could finish. Now `readiness` waits for `route` and skips in `pr-summary` mode, so E2E Evals on a PR push finishes in about a minute (58 s on #3609) instead of after the 10-minute wait.
VERSION
+1 -1
View File
@@ -1 +1 @@
1.91.371.0
1.91.378.0
agents-digest/gstack-AGENTS.md
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.91.371.0 — regenerate/re-copy after upgrading gstack
# gstack digest v1.91.378.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
bin/gstack-base-red
Executable
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env -S bun --no-env-file --config=/dev/null
/**
* gstack-base-red — /ship Step 5's red-base proof and pause (docs/ship-policy.md#base-red).
*
* gstack-base-red classify --failures <evidence.json> [--base <branch>] [--head <sha>] [--pause] [--json]
* gstack-base-red wait --pause <file> [--timeout <s>] [--poll <s>] [--json]
*
* `classify` reads the base branch's `base_red` policy (never the working tree),
* prints one BASE_RED: or IN_BRANCH: line per failure and a BASE_RED_VERDICT:
* line, and with --pause writes the pause record when every failure is
* base-red (PAUSE: <file>). Exit 0 only for base-red; 1 in-branch.
* `wait` returns 0 (BASE_GREEN: <sha>) once a new base head passes the
* paused cases, 3 on timeout or an expired pause. Logic: lib/base-red.ts.
*/
import * as fs from 'node:fs';
import { classifyBaseRed, classifyLines, parseEvidence, readPause, waitBaseGreen, writePause } from '../lib/base-red';
import { EXIT, exitTableHelp } from '../lib/headless-artifacts';
import { resultLineWithDocs } from '../lib/result-codes';
import { isUnavailable, loadPolicyFromBase, policyUnavailableLine, resolveBaseBranch } from '../lib/ship-policy';
import { git } from '../lib/ship-queue-gate';
const HELP = `gstack-base-red — prove a Step 5 failure is the base branch's own, and pause /ship on it
classify --failures <evidence.json> [--base <branch>] [--head <sha>] [--pause] [--json]
evidence: { "lane": "<Step 5 label>", "total": <failures the lane reported>,
"failures": [{ "id": "<test name>", "file": "<test file>", "signature": "<first error line>" }] }
wait --pause <file> [--timeout <s>] [--poll <s>] [--json]
${exitTableHelp()}`;
const flag = (args: string[], name: string) => { const i = args.indexOf(name); return i >= 0 ? args[i + 1] : undefined; };
const say = (json: boolean, lines: string[], value: unknown) => process.stdout.write(json ? JSON.stringify(value) + '\n' : lines.join('\n') + '\n');
function policyFor(repoRoot: string, base: string, cmd: string) {
git(repoRoot, ['fetch', '--no-tags', '-q', 'origin', `+refs/heads/${base}:refs/remotes/origin/${base}`]);
const load = loadPolicyFromBase(repoRoot, base);
if (isUnavailable(load)) { process.stderr.write(policyUnavailableLine(cmd, base, load.detail) + '\n'); return 'unavailable' as const; }
if (load.errors.length) { process.stderr.write(`${cmd}: ${load.source.label} is invalid (${load.errors[0]!.message} at ${load.errors[0]!.path}); every failure stays in-branch\n`); return null; }
return load.policy.base_red;
}
function classify(args: string[], json: boolean, repoRoot: string): number {
const file = flag(args, '--failures');
if (!file) { process.stderr.write(`gstack-base-red classify: --failures <evidence.json> is required\n${HELP}\n`); return EXIT.usage; }
let raw: unknown;
try { raw = JSON.parse(fs.readFileSync(file, 'utf8')); } catch (e: any) { process.stderr.write(`gstack-base-red classify: cannot read ${file}: ${e.message}\n`); return EXIT.usage; }
const evidence = parseEvidence(raw);
if (typeof evidence === 'string') { process.stderr.write(`gstack-base-red classify: ${file}: ${evidence}\n`); return EXIT.usage; }
const base = resolveBaseBranch(repoRoot, flag(args, '--base'));
const policy = policyFor(repoRoot, base, 'gstack-base-red classify');
if (policy === 'unavailable') return EXIT.fail;
const r = classifyBaseRed({ repoRoot, base, head: flag(args, '--head') ?? 'HEAD', evidence, policy });
const lines = classifyLines(r);
if (r.code !== 'BASE_RED' && r.code !== 'IN_BRANCH') process.stderr.write(resultLineWithDocs('gstack-base-red classify', r.code, r.detail) + '\n');
let pause: string | undefined;
if (r.code === 'BASE_RED' && args.includes('--pause')) {
const branch = git(repoRoot, ['rev-parse', '--abbrev-ref', 'HEAD']).out || r.head;
pause = writePause(repoRoot, branch, r, evidence).file;
lines.push(`PAUSE: ${pause}`);
}
say(json, lines, { ...r, pause });
return r.code === 'BASE_RED' ? EXIT.ok : EXIT.fail;
}
function wait(args: string[], json: boolean, repoRoot: string): number {
const file = flag(args, '--pause');
if (!file) { process.stderr.write(`gstack-base-red wait: --pause <file> is required\n${HELP}\n`); return EXIT.usage; }
const pause = readPause(file);
if (typeof pause === 'string') { process.stderr.write(`gstack-base-red wait: ${pause}\n`); return EXIT.usage; }
const policy = policyFor(repoRoot, pause.base, 'gstack-base-red wait');
if (!policy || policy === 'unavailable') { process.stderr.write(resultLineWithDocs('gstack-base-red wait', 'BASE_RED_UNCONFIGURED', `origin/${pause.base} no longer declares a usable base_red`) + '\n'); return EXIT.refused; }
const timeoutSec = Number(flag(args, '--timeout') ?? 21600);
const pollSec = Number(flag(args, '--poll') ?? 120);
if (!(timeoutSec > 0) || !(pollSec >= 5)) { process.stderr.write('gstack-base-red wait: --timeout must be > 0 and --poll >= 5 seconds\n'); return EXIT.usage; }
const r = waitBaseGreen({ repoRoot, pause, policy, timeoutSec, pollSec });
if (r.code === 'BASE_GREEN') { say(json, [`BASE_GREEN: origin/${pause.base} ${r.sha} passes ${pause.cases.length} paused case(s) after ${r.checked} base head(s); resume with /ship --resume <record>`], r); return EXIT.ok; }
process.stderr.write(resultLineWithDocs('gstack-base-red wait', r.code, r.detail) + '\n');
say(json, [`BASE_RED_WAIT: ${r.code} last base ${r.sha.slice(0, 12)} (${r.checked} head(s) checked)`], r);
return EXIT.refused;
}
const [cmd, ...rest] = process.argv.slice(2);
const json = rest.includes('--json');
const repoRoot = git(process.cwd(), ['rev-parse', '--show-toplevel']).out || process.cwd();
if (cmd === 'classify') process.exit(classify(rest, json, repoRoot));
if (cmd === 'wait') process.exit(wait(rest, json, repoRoot));
process.stdout.write(HELP + '\n');
process.exit(cmd === '--help' || cmd === '-h' ? EXIT.ok : EXIT.usage);
bin/gstack-doctor-components.sh
+10 -2
View File
@@ -194,7 +194,6 @@ gstack_result_anchor() {
ISSUE_UNREADABLE) printf '%s' 'issue-unreadable' ;;
ISSUE_LOOKUP_FAILED) printf '%s' 'issue-lookup-failed' ;;
ISSUE_LINKS_MISSING) printf '%s' 'issue-links-missing' ;;
PREREG_INCOMPLETE) printf '%s' 'prereg-incomplete' ;;
EVAL_PRICE_MISSING) printf '%s' 'eval-price-missing' ;;
EVAL_DRY_RUN_FAILED) printf '%s' 'eval-dry-run-failed' ;;
EVAL_ARMS_UNRESOLVED) printf '%s' 'eval-arms-unresolved' ;;
@@ -240,7 +239,6 @@ gstack_result_anchor() {
OUTSIDE_VOICE_UNAVAILABLE) printf '%s' 'outside-voice-unavailable' ;;
ARTIFACT_MACHINE_PATH) printf '%s' 'artifact-machine-path' ;;
ARTIFACT_PUBLIC_TARGET_EXISTS) printf '%s' 'artifact-public-target-exists' ;;
PREREG_DIAGNOSTIC_UNTESTED) printf '%s' 'prereg-diagnostic-untested' ;;
EVAL_MODELS_INVALID) printf '%s' 'eval-models-invalid' ;;
EVAL_MODEL_DENIED) printf '%s' 'eval-model-denied' ;;
EVAL_MODEL_OLD_GENERATION) printf '%s' 'eval-model-old-generation' ;;
@@ -287,6 +285,12 @@ gstack_result_anchor() {
QUEUE_TRAIN_FROZEN) printf '%s' 'queue-train-frozen' ;;
QUEUE_RECEIPT_MISSING) printf '%s' 'queue-receipt-missing' ;;
QUEUE_RECEIPT_STALE) printf '%s' 'queue-receipt-stale' ;;
QUEUE_RECEIPT_CONDITIONAL) printf '%s' 'queue-receipt-conditional' ;;
BASE_RED_UNCONFIGURED) printf '%s' 'base-red-unconfigured' ;;
BASE_RED_UNNAMED) printf '%s' 'base-red-unnamed' ;;
BASE_RED_REPRO_ERROR) printf '%s' 'base-red-repro-error' ;;
BASE_RED_PAUSE_EXPIRED) printf '%s' 'base-red-pause-expired' ;;
BASE_RED_WAIT_TIMEOUT) printf '%s' 'base-red-wait-timeout' ;;
QUEUE_RECEIPT_DEPTH_RAISED) printf '%s' 'queue-receipt-depth-raised' ;;
QUEUE_RECEIPT_EXCEPTION_INVALID) printf '%s' 'queue-receipt-exception-invalid' ;;
QUEUE_SCREEN_STALE_REF) printf '%s' 'queue-screen-stale-ref' ;;
@@ -346,6 +350,10 @@ gstack_result_anchor() {
BUDGET_LEDGER_INVALID) printf '%s' 'budget-ledger-invalid' ;;
BUDGET_CHECK_SKIPPED) printf '%s' 'budget-check-skipped' ;;
HANDOFF_PIN_MOVED) printf '%s' 'handoff-pin-moved' ;;
PREREG_INCOMPLETE) printf '%s' 'prereg-incomplete' ;;
PREREG_DIAGNOSTIC_UNTESTED) printf '%s' 'prereg-diagnostic-untested' ;;
PREREG_NO_GSTACK_ARM) printf '%s' 'prereg-no-gstack-arm' ;;
EVAL_ARM_DUPLICATE) printf '%s' 'eval-arm-duplicate' ;;
*) return 1 ;;
esac
}
bin/gstack-eval-plan
+8 -6
View File
@@ -6,7 +6,7 @@
* gstack-eval-plan arms [--arms <model,model>] [--policy-doc <file>] [--input-tokens N] [--output-tokens N] [--json]
* gstack-eval-plan dry-run --input <rows.jsonl> --arms <model,model> [--fields a,b] [--output-budget N]
* [--retry-prompt <file>] [--max-rows N] [--out <dir>] [--json]
* gstack-eval-plan pilot start --out <dir> --cap <usd> --arms <model,model> --dry-run <dry-run.json> [--n 3]
* gstack-eval-plan pilot start --out <dir> --cap <usd> --arms <model[#loadout],...> --dry-run <dry-run.json> [--n 3]
* gstack-eval-plan pilot settle --out <dir> --attempt <id> --usd <n|unknown>
* gstack-eval-plan pilot release --out <dir> --attempt <id>
* gstack-eval-plan pilot report --out <dir> --total <items> [--workers W] [--json]
@@ -31,7 +31,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import { spawnSync } from 'node:child_process';
import { checkPrereg, DEFAULT_TOKENS, dryRun, pilotReport, preregTemplate, priceGate, releasePilot, renderArms, renderDryRun, renderGatePrice, renderPilotReport, renderPreregDiagnostics, resolveArms, resolvePlanSource, runPreregDiagnostics, settlePilot, startPilot } from '../lib/eval-plan';
import { armModel, checkPrereg, DEFAULT_TOKENS, PREREG_SECTIONS, dryRun, pilotReport, preregTemplate, priceGate, releasePilot, renderArms, renderDryRun, renderGatePrice, renderPilotReport, renderPreregDiagnostics, resolveArms, resolvePlanSource, runPreregDiagnostics, settlePilot, startPilot } from '../lib/eval-plan';
import { checkModelRules, EVAL_SCOPES, type EvalScope, type ModelRulesCheck } from '../lib/eval-models';
import { EXIT, exitTableHelp } from '../lib/headless-artifacts';
import { resultLine } from '../lib/result-codes';
@@ -50,7 +50,7 @@ const HELP = `gstack-eval-plan — preregistration, priced arms, $0 dry run, pri
dry-run --input rows.jsonl --arms m,m [--fields a,b] [--output-budget N] [--retry-prompt f] [--max-rows N] [--cap usd] [--out dir] [--json]
stub model over the real rows; checklist: price table, output budget, retry prompt text,
and with --cap reservation-fits-cap (one call: largest row + output budget, priced)
pilot start --out dir --cap usd --arms m,m --dry-run dry-run.json [--n 3]
pilot start --out dir --cap usd --arms m,m#no-gstack --dry-run dry-run.json [--n 3]
pilot settle --out dir --attempt id --usd <n|unknown> pilot release --out dir --attempt id
pilot report --out dir --total items [--workers W] [--json] spend table + extrapolation with a 95% band, then the approval ask
price-gate --gate-file <file> | --gate "<exact gate command>" [--plan-file f] [--base b] [--cap-usd N] [--smoke-cap-usd N] [--allow-unpriced "<reason>"] [--out dir] [--json]
@@ -90,7 +90,7 @@ function lintModels(args: string[], uses: Array<{ model: string; scope: EvalScop
if (from !== 'base' && from !== 'head') throw new Error('--models-from needs base or head');
const top = spawnSync('git', ['rev-parse', '--show-toplevel'], { encoding: 'utf8', timeout: 30_000 });
const repoRoot = top.status === 0 ? top.stdout.trim() : null;
const check = checkModelRules(repoRoot, uses, { from, base: repoRoot ? resolveBaseBranch(repoRoot, flag(args, '--base')) : 'main' });
const check = checkModelRules(repoRoot, uses.map(u => (armModel(u.model) === u.model ? u : { ...u, model: armModel(u.model), where: `${u.where} (${u.model})` })), { from, base: repoRoot ? resolveBaseBranch(repoRoot, flag(args, '--base')) : 'main' });
if (!args.includes('--json')) process.stdout.write(check.lines.join('\n') + '\n');
for (const e of check.errors) process.stderr.write(resultLine(`gstack-eval-plan ${command}`, e.code, e.message) + '\n');
return check;
@@ -116,11 +116,13 @@ function doPrereg(rest: string[]): number {
const run = runnable && allowed ? runPreregDiagnostics(d, repoRoot) : null;
if (rest.includes('--json')) process.stdout.write(JSON.stringify({ file, ...r, diagnostics_run: run }) + '\n');
else {
process.stdout.write(`PREREG: ${r.ok ? 'ok' : 'incomplete'} present=${r.present.length}/5${r.missing.length ? ` missing=${r.missing.join(',')}` : ''}${r.empty.length ? ` empty=${r.empty.join(',')}` : ''}${d ? ` diagnostics=${d.errors.length ? 'invalid' : d.expected.length}` : ''} file=${file}\n`);
process.stdout.write(`PREREG: ${r.ok ? 'ok' : 'incomplete'} present=${r.present.length}/${PREREG_SECTIONS.length}${r.missing.length ? ` missing=${r.missing.join(',')}` : ''}${r.empty.length ? ` empty=${r.empty.join(',')}` : ''}${r.comparison.stack ? ` stack=${r.comparison.stack}` : ''}${r.no_gstack.required ? ` no_gstack_arm=${r.no_gstack.ok ? 'ok' : r.no_gstack.problems[0] === 'missing section' ? 'missing' : 'incomplete'}` : ''}${d ? ` diagnostics=${d.errors.length ? 'invalid' : d.expected.length}` : ''} file=${file}\n`);
if (runnable && !allowed) process.stdout.write(`PREREG_COMMAND: summarizer=${JSON.stringify(d.summarizer)} fixture=${d.fixture} cwd=${repoRoot}\n`);
if (run) process.stdout.write(renderPreregDiagnostics(run));
}
if (!r.ok) process.stderr.write(resultLine('gstack-eval-plan prereg', 'PREREG_INCOMPLETE', [...r.missing.map(m => `missing ${m}`), ...r.empty.map(e => `empty ${e}`), ...(d?.errors ?? []).map(e => `diagnostics: ${e}`)].join('; ')) + '\n');
const incomplete = [...r.missing.map(m => `missing ${m}`), ...r.empty.map(e => `empty ${e}`), ...r.comparison.errors.map(e => `comparison: ${e}`), ...(d?.errors ?? []).map(e => `diagnostics: ${e}`)];
if (incomplete.length) process.stderr.write(resultLine('gstack-eval-plan prereg', 'PREREG_INCOMPLETE', incomplete.join('; ')) + '\n');
if (!r.no_gstack.ok) process.stderr.write(resultLine('gstack-eval-plan prereg', 'PREREG_NO_GSTACK_ARM', `no-gstack arm: ${r.no_gstack.problems.join('; ')}`) + '\n');
if (run && !run.ok) process.stderr.write(resultLine('gstack-eval-plan prereg', 'PREREG_DIAGNOSTIC_UNTESTED', run.error ?? run.rows.filter(row => row.status !== 'ok').map(row => `${row.name} ${row.status} (expected ${row.expected ?? 'undeclared'}, got ${row.got ?? 'none'})`).join('; ')) + '\n');
if (!r.ok || (run && !run.ok)) return EXIT.fail;
if (runnable && !allowed) { process.stderr.write(resultLine('gstack-eval-plan prereg', 'REPO_COMMANDS_NOT_ALLOWED', `the Diagnostics section declares a summarizer; it runs only with --allow-repo-commands, and the preregistration is not checked until it does`) + '\n'); return EXIT.refused; }
bin/gstack-flaky
Executable
+125
View File
@@ -0,0 +1,125 @@
#!/usr/bin/env -S bun --no-env-file --config=/dev/null
/**
* gstack-flaky (GSTA-158): rerun failed tests once and record the flake, for
* any runner that writes JUnit (Playwright, bun, jest, pytest, vitest). CREW
* rule 16 for every repository: "you can just run the failed test again and
* note that it is flaky". The rules live in lib/flaky-run.ts; docs/flaky.md is
* the how-to and the error-code table.
*/
import * as fs from 'fs';
import { DEFAULT_MAX_RERUNS, appendLedger, ledgerPath, renderResult, runFlaky, safeText, stepSummary, workflowEscape } from '../lib/flaky-run';
import { JunitError, readJunit, type JunitRunner } from '../lib/junit';
const USAGE = `Usage:
gstack-flaky run --junit <file.xml> [--runner playwright|bun|generic] [--max-reruns <n>]
[--rerun-scope command|test] [--ledger <file>] -- <test command...>
gstack-flaky ids --junit <file> [--runner playwright|bun|generic]
gstack-flaky --help
run: runs the command (an argv, no shell), reads the JUnit report it writes at
--junit, and reruns each failed test once when the runner made no retries and
at most --max-reruns (${DEFAULT_MAX_RERUNS}) tests failed. The rerun repeats the whole
command and counts only on the runner's full green verdict (exit 0, no failed
or missing test). --rerun-scope test reruns the failed test alone (playwright:
<file> --grep <title> --project <p>; bun: <file> --test-name-pattern <title>)
when the runner's summary proves the first run complete and nothing but the
failed tests can fail it (no coverage, no runner-flaky test); otherwise, and
always for generic runners, the whole command reruns. A test that passes its
rerun once, with the rerun exiting 0, is a flaky pass: green, and one line on the ledger (--ledger, else
GSTACK_FLAKE_LEDGER_JUNIT, else <state root>/flake-ledger-junit.jsonl). --junit
holds the first run's report afterwards; reruns' reports sit beside it as
<name>.rerun-<n>.xml. The command runs with PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1
so Playwright's own retries are reported.
ids: prints the id of every test in a report (<file>::<project>::<title>).
Output: one FLAKY: line per test that failed at least once, FLAKY_RERUN:
scope=command (<why>) when the rerun repeated the whole command, then
FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>
Exit: 0 pass (flaky passes included), 1 a test failed, 2 usage,
3 no usable report, a crash or a killed runner (FLAKY_<CODE> and a fix: line on stderr).
`;
const FIX: Record<string, string> = {
FLAKY_NO_RESULTS: 'make the command write JUnit to the --junit path (playwright: PLAYWRIGHT_JUNIT_OUTPUT_NAME=<file> with --reporter=junit; bun: --reporter=junit --reporter-outfile=<file>)',
FLAKY_JUNIT_INVALID: 'the report is not well-formed XML; check that one process writes it and nothing else appends to it',
FLAKY_RETRIES_HIDDEN: 'set PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1 where the report is written',
FLAKY_COMMAND_FAILED: 'read the command output above: the suite crashed or a non-test step failed; that is a real red',
FLAKY_COMMAND_KILLED: 'the runner was killed by a signal (a cancelled job, an OOM kill); nothing was rerun, rerun the job',
FLAKY_COMMAND_NOT_STARTED: 'check the command after -- (it runs without a shell; wrap it as bash -c \'<script> "$@"\' _ when it needs one)',
FLAKY_UNREPORTED_FAILURES: 'the runner counted failures its JUnit omits (a test file that failed to load, an error outside any test); fix that first, it is a real red',
};
function die(msg: string, code = 2): never {
process.stderr.write(`gstack-flaky: ${msg}\n`);
process.exit(code);
}
function coded(e: JunitError): never {
process.stderr.write(`gstack-flaky: ${e.message} (${e.code})\nfix: ${FIX[e.code] ?? 'see docs/flaky.md'}\n`);
process.exit(3);
}
function parse(argv: string[]) {
const o: { junit?: string; runner?: JunitRunner; maxReruns: number; rerunScope: 'command' | 'test'; ledger?: string; command: string[] } = { maxReruns: DEFAULT_MAX_RERUNS, rerunScope: 'command', command: [] };
for (let i = 0; i < argv.length; i++) {
const a = argv[i]!;
const val = () => { const v = argv[++i]; if (v === undefined) die(`${a} needs a value`); return v; };
if (a === '--') { o.command = argv.slice(i + 1); break; }
if (a === '--junit') o.junit = val();
else if (a === '--runner') {
const r = val();
if (r !== 'playwright' && r !== 'bun' && r !== 'generic') die(`--runner must be playwright, bun or generic, got ${JSON.stringify(r)}`);
o.runner = r;
} else if (a === '--max-reruns') {
const n = Number(val());
if (!Number.isInteger(n) || n < 0) die('--max-reruns must be a whole number');
o.maxReruns = n;
} else if (a === '--rerun-scope') {
const v = val();
if (v !== 'command' && v !== 'test') die(`--rerun-scope must be command or test, got ${JSON.stringify(v)}`);
o.rerunScope = v;
} else if (a === '--ledger') o.ledger = val();
else die(`unknown option ${a}\n${USAGE}`);
}
return o;
}
const [sub, ...rest] = process.argv.slice(2);
if (!sub || sub === '--help' || sub === '-h') { process.stdout.write(USAGE); process.exit(sub ? 0 : 2); }
if (sub === 'ids') {
const o = parse(rest);
if (!o.junit) die('ids needs --junit <file>');
try {
for (const t of readJunit(o.junit, o.runner).tests) process.stdout.write(`${safeText(t.id)}\n`);
} catch (e) { if (e instanceof JunitError) coded(e); throw e; }
process.exit(0);
}
if (sub !== 'run') die(`unknown command ${sub}\n${USAGE}`);
const o = parse(rest);
if (!o.junit) die('run needs --junit <file.xml>');
if (!/\.xml$/i.test(o.junit) || fs.statSync(o.junit, { throwIfNoEntry: false })?.isDirectory()) die(`--junit must name a .xml file (it is deleted before every attempt), got ${JSON.stringify(o.junit)}`);
let existing: string | false | undefined;
try { existing = fs.statSync(o.junit, { throwIfNoEntry: false }) && fs.readFileSync(o.junit, 'utf-8').slice(0, 4096); } catch (e) {
die(`--junit ${JSON.stringify(o.junit)} exists but cannot be read (${(e as Error).message}); gstack-flaky deletes that path before every attempt, so it refuses to touch it`);
}
if (existing && existing.trim() && !/<testsuites?[\s>/]/.test(existing)) die(`--junit ${JSON.stringify(o.junit)} exists and is not a JUnit report; gstack-flaky deletes that path before every attempt, so it refuses to touch it`);
if (!o.command.length) die('run needs the test command after --');
let result;
try {
result = await runFlaky({ command: o.command, junit: o.junit, runner: o.runner, maxReruns: o.maxReruns, rerunScope: o.rerunScope });
} catch (e) { if (e instanceof JunitError) coded(e); throw e; }
const ledger = ledgerPath(o.ledger);
appendLedger(result.ledger, ledger, s => process.stderr.write(`gstack-flaky: ${s}\n`));
for (const l of renderResult(result, ledger)) process.stdout.write(`${l}\n`);
const summary = process.env.GITHUB_STEP_SUMMARY && stepSummary(result);
if (summary) { try { fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY!, summary); } catch { /* a summary is a convenience, never the verdict */ } }
if (process.env.GITHUB_ACTIONS === 'true') {
for (const f of result.outcomes.filter(x => x.status === 'flaky-pass')) process.stdout.write(`::warning title=flaky test::${workflowEscape(safeText(f.test.id))} passed on attempt ${f.attempts.length}\n`);
}
if (result.code) {
process.stderr.write(`gstack-flaky: ${result.message} (${result.code})\nfix: ${FIX[result.code] ?? 'see docs/flaky.md'}\n`);
}
process.exit(result.exitCode);
bin/gstack-pregate
+46 -7
View File
@@ -6,7 +6,7 @@
* [--allow-repo-commands] [--policy-from base|head] [--out <pregate.json>]
* [--timeout <s>] [--stall-timeout <s>] [--remote --spend-cap <usd>] [--run <id>] [--explain] [--json]
* gstack-pregate verify [--out <pregate.json>] [--json] publication check: current tree, no fail/incomplete, obligations cleared
* gstack-pregate clear --remote <workflow.yml/job> --run-url <url> | --bundle-lane <label> | --receipt <file> [--out <pregate.json>]
* gstack-pregate clear --remote <workflow.yml/job> --run-url <url> | --run <id|url> [--run …] [--allow-repo-commands] | --bundle-lane <label> | --receipt <file> [--out <pregate.json>]
* gstack-pregate lanes [--explain] the lane list and platform tiers read from .github/workflows
* gstack-pregate validate | init [--explain] [--force] the working-tree .gstack/pregate.json
* gstack-pregate sealed [--range <a..b>] [--remote <name>] [--base <branch>] [--json]
@@ -27,14 +27,17 @@ import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { EXIT, exitTableHelp, formatResult, type PregateFile } from '../lib/headless-artifacts';
import { PREGATE_CONFIG_PATH, explainConfig, loadConfigFromHead } from '../lib/pregate/config';
import { PREGATE_CONFIG_PATH, explainConfig, isConfigUnavailable, loadConfigFromBase, loadConfigFromHead } from '../lib/pregate/config';
import { describeDraft, draftConfig } from '../lib/pregate/init-draft';
import { buildContext, defaultUserConfigPath, describeRefusal, loadedDotenvFiles } from '../lib/pregate/context';
import { CHECKS, CHECK_IDS, PREGATE_OUT_REL, PREGATE_TIER2, clearDecision, clearObligation, readPregateFile, renderTable, runPregate, verifyPregate, writePregateFile } from '../lib/pregate/index';
import { CHECKS, CHECK_IDS, PREGATE_OUT_REL, PREGATE_TIER2, clearDecision, clearObligation, readPregateFile, renderTable, runPregate, verifyPregate, writePregateFile, type RunCheck } from '../lib/pregate/index';
import { ghRunDeps, runInventory } from '../lib/run-evidence';
import { wtreeFingerprint } from '../lib/script-invocation';
import { describeLane, discoverLanes } from '../lib/pregate/workflows';
import { REMOTE_GATE_LANE } from '../lib/pregate/checks/lanes';
import { describeSubstitution } from '../lib/pregate/remote-receipt';
import { ensureTmpExcluded } from '../lib/restamp';
import { normalizeRemoteUrl } from '../lib/ship-queue-train-record';
import { RESULT_CODES, resultCodeSuffix, resultLine } from '../lib/result-codes';
import { DEFAULT_STALL_TIMEOUT_S } from '../lib/pregate/context';
import { cancelActive } from '../lib/pregate/supervise';
@@ -52,7 +55,11 @@ const HELP = `gstack-pregate — cheap checks before anything paid or remote (do
--receipt <file> (the gate's PREGATE_REMOTE: line, checked against the tested tree; a tree that
differs from pregate.json's only in the base branch's stamp rule is accepted and recorded as a
substitution, docs/pregate.md#remote-gate-stamp-only); a
test_backend/<backend> run URL is recorded as attested, not validated
test_backend/<backend> run URL is recorded as attested, not validated. --run <id|url> (repeatable)
verifies GitHub Actions runs against the base branch's run_profiles instead (docs/pregate.md#run-evidence):
one run of the profile named after the lane, or for pregate.json/remote_gate one run per
remote_gate_runs profile covering the gate's inventory (needs --allow-repo-commands); where a lane has a
profile, --run-url is verified the same way
lanes print every CI lane with its platform tier (from .github/workflows/*.yml)
validate | init the working-tree ${PREGATE_CONFIG_PATH}
sealed [--range <a..b>] [--remote <name>] [--base <branch>]
@@ -103,7 +110,7 @@ function flag(args: string[], name: string): string | undefined { const i = args
const VERB_FLAGS: Record<string, [string[], string[]]> = {
run: [['--stage', '--only', '--base', '--policy-from', '--out', '--timeout', '--stall-timeout', '--spend-cap', '--run'], ['--allow-repo-commands', '--explain', '--json', '--remote']],
verify: [['--out'], ['--json']],
clear: [['--remote', '--run-url', '--bundle-lane', '--receipt', '--out'], []],
clear: [['--remote', '--run-url', '--run', '--bundle-lane', '--receipt', '--out'], ['--allow-repo-commands']],
lanes: [[], ['--explain', '--json']],
validate: [[], ['--json']],
init: [[], ['--explain', '--force']],
@@ -218,19 +225,51 @@ function cmdClear(root: string, rest: string[]): number {
const out = outPath(root, rest);
const read = fs.existsSync(out) ? readPregateFile(out) : { file: null, errors: [] };
if (!read.file) { process.stderr.write(resultLine('gstack-pregate clear', 'PREGATE_STALE', `${out} is missing or invalid; run gstack-pregate first`) + '\n'); return EXIT.fail; }
const decision = clearDecision(read.file, lane, { runUrl: flag(rest, '--run-url'), bundleLane: flag(rest, '--bundle-lane'), receiptPath: flag(rest, '--receipt') }, root);
const runs = rest.flatMap((a, i) => a === '--run' ? [rest[i + 1]!] : []);
const runUrl = flag(rest, '--run-url');
const bundleLane = flag(rest, '--bundle-lane');
const checked = runs.length || runUrl || bundleLane ? runCheck(root, lane, rest.includes('--allow-repo-commands')) : undefined;
if (checked && 'error' in checked && (runs.length || checked.profiled)) { process.stderr.write(`gstack-pregate clear: ${checked.error}${checked.profiled ? `; ${lane} may have a run profile on the base branch, so it clears only with a verified --run` : ''}\n`); return EXIT.fail; }
const evidence = checked && !('error' in checked) ? checked : undefined;
const decision = clearDecision(read.file, lane, { runUrl, runs, bundleLane, receiptPath: flag(rest, '--receipt'), evidence }, root);
if ('refused' in decision) {
if (decision.refused === 'usage') { process.stderr.write(`gstack-pregate clear: ${decision.message}\n`); return EXIT.usage; }
process.stderr.write(`gstack-pregate clear: receipt refused — ${decision.message}; fix: ${decision.fix ?? RESULT_CODES.PREGATE_RECEIPT_INVALID.fix} ${resultCodeSuffix('PREGATE_RECEIPT_INVALID')}\n`);
return EXIT.fail;
}
const file = clearObligation(read.file, lane, decision.receipt, `gstack-pregate clear ${new Date().toISOString()}`, decision.substitution);
const current = readPregateFile(out).file;
if (!current || current.tree !== read.file.tree || current.generated_at !== read.file.generated_at) { process.stderr.write(resultLine('gstack-pregate clear', 'PREGATE_STALE', `${out} changed while ${lane} was being verified; rerun the clear against the new file`) + '\n'); return EXIT.fail; }
const file = clearObligation(current, lane, decision.receipt, `gstack-pregate clear ${new Date().toISOString()}`, decision.substitution);
writePregateFile(root, out, file);
if (/ runs=\S+ profiles=/.test(decision.receipt)) process.stdout.write(`REUSE: accepted ${lane} ${decision.receipt}\n`);
if (decision.substitution) process.stdout.write(`PREGATE_RECEIPT_SUBSTITUTED: ${describeSubstitution(decision.substitution)}\n`);
process.stdout.write(`PREGATE_CLEARED: ${lane} receipt=${decision.receipt} verdict=${file.verdict}${decision.attested ? ` (attested, not validated: ${lane} has no machine-checkable receipt)` : ''}\n`);
if (checked && 'error' in checked) process.stderr.write(`gstack-pregate clear: run profiles unavailable (${checked.error}); ${lane} was recorded without run verification\n`);
return EXIT.ok;
}
/** The run-evidence inputs for `clear --run`: the base branch's run profiles, this repository on GitHub, and the gate's inventory command (a pin-file command, so only with --allow-repo-commands). */
function runCheck(root: string, lane: string, allowRepoCommands: boolean): RunCheck | { error: string; profiled?: boolean } {
const base = resolveBaseBranch(root);
const loaded = loadConfigFromBase(root, base);
// Unknown profiles are treated as present: a lane is cleared unverified only once the base shows it has no profile.
if (isConfigUnavailable(loaded)) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is unreadable (${loaded.detail}); run profiles come only from the base`, profiled: true };
if (loaded.errors.length) return { error: `the base branch's ${PREGATE_CONFIG_PATH} is invalid: ${loaded.errors.map(e => `${e.path}: ${e.message}`).join('; ')}`, profiled: true };
const repo = /^https:\/\/github\.com\/(.+)$/.exec(normalizeRemoteUrl(gitOut(root, ['config', '--get', 'remote.origin.url']) ?? ''))?.[1];
if (!repo) return { error: 'origin is not a github.com repository; run evidence is read from GitHub Actions', profiled: lane === REMOTE_GATE_LANE ? !!loaded.config.remote_gate_runs : !!loaded.config.run_profiles[lane] };
return {
repo, deps: ghRunDeps(repo, root), profiles: loaded.config.run_profiles, gateRuns: loaded.config.remote_gate_runs, now: new Date(),
inventory: (command, tree) => {
if (!allowRepoCommands) return { error: 'the inventory command is a pin-file command; pass --allow-repo-commands' };
const at = () => { const fp = wtreeFingerprint({ cwd: root }); return 'tree' in fp ? fp.tree : null; };
if (at() !== tree) return { error: `the working tree is not pregate.json's tree ${tree.slice(0, 12)}; the inventory must list that tree's gate files` };
const listed = runInventory(command, root);
if (at() !== tree) return { error: `the working tree changed while ${command} ran` };
return listed;
},
};
}
function gitOut(root: string, args: string[]): string | null {
const r = spawnSync('git', args, { cwd: root, encoding: 'utf8', timeout: 30_000 });
return r.status === 0 ? r.stdout.trim() : null;
bin/gstack-ship-receipt
+18 -8
View File
@@ -7,6 +7,7 @@
* [--session-kind <k>] [--artifacts-consumed <n>/<m>] [--run <id>] [--thread <id>]
* [--receipt <tree-receipt.json>] [--policy <label>] [--preregistration <text>] [--history <text>]
* [--pregate <pregate.json>] [--issues <json|@file>] [--approval <line>]... [--approval-run <dir>]...
* [--base-red <pause.json>]
* [--out <file>] [--body <pr-body.md>] [--json]
* gstack-ship-receipt read --pr <n> [--repo <owner/repo>] [--check-prose] [--json]
* gstack-ship-receipt read --body <file> [--json]
@@ -47,11 +48,13 @@ import { checkDocsCaps, docsCapLine, docsExceptionLine, parseReceiptDocs } from
import { deeper, depthBelow, exceptionProblems, parseReviewLoopException, parseReviewTargeted, receiptDepthAtHead, reviewLoopExceptionLine } from '../lib/ship-receipt-depth';
import { checkReceiptProse, proseLine } from '../lib/ship-receipt-prose';
import { resolveStateRoot } from '../lib/state-root';
import { readPause, receiptBaseRed } from '../lib/base-red';
import { isRegistryUnavailable, loadRegistryFromBase } from '../lib/regen';
const HELP = `gstack-ship-receipt — the \`\`\`gstack-ship-receipt\`\`\` block (docs/ship-policy.md#ship-receipt)
write [flags] compose, validate and print the block; --body <file> puts it first in that PR body
write [flags] compose, validate and print the block; --body <file> puts it first in that PR body;
--base-red <pause.json> makes it conditional (gstack-base-red classify --pause)
read --pr <n> [--repo <r>] parse the block from the PR body through gh (RECEIPT_MISSING / RECEIPT_INVALID)
read --body <file> parse the block from a file
read … --check-prose also compare every counter the body quotes (passes=2, 1 deferred, passed: 41)
@@ -184,6 +187,8 @@ function cmdWrite(rest: string[], json: boolean): number {
if (!read.ok) { for (const e of read.errors) process.stderr.write(resultLine('gstack-ship-receipt write', e.code, `${e.message}; nothing written`) + '\n'); return EXIT.fail; }
pregate = read.field;
}
const pause = flag(rest, '--base-red') ? readPause(flag(rest, '--base-red')!) : undefined;
if (typeof pause === 'string') { process.stderr.write(`gstack-ship-receipt write: --base-red: ${pause}; nothing written\n`); return EXIT.usage; }
const base = flag(rest, '--base') ?? resolveBaseBranch(repoRoot);
const bodyFile = flag(rest, '--body');
const bodyText = bodyFile && fs.existsSync(bodyFile) ? fs.readFileSync(bodyFile, 'utf8').replace(/```gstack-ship-receipt[\s\S]*?\n```/g, '') : '';
@@ -247,6 +252,7 @@ function cmdWrite(rest: string[], json: boolean): number {
...pregate, issues: issuesFile ? receiptIssuesFrom(issuesFile) : undefined, approval: approval.kind === 'ok' ? approval.approval : undefined,
review_loop: reviewLoop, ...(docs?.ok ? { docs: docs.docs } : {}), ...product,
gstack_version: runningGstackVersion(), review_depth: reviewDepth, review_loop_exception: loopException?.ok ? loopException.exception : undefined, review_targeted: targeted?.ok ? targeted.targeted : undefined,
base_red: pause ? receiptBaseRed(pause) : undefined,
};
} catch (e: any) { process.stderr.write(`gstack-ship-receipt write: ${e.message}\n`); return EXIT.usage; }
if (input.pr !== undefined && !Number.isInteger(input.pr)) { process.stderr.write('gstack-ship-receipt write: --pr must be an integer\n'); return EXIT.usage; }
@@ -355,13 +361,16 @@ function cmdResult(rest: string[], json: boolean): number {
}
source = `PR #${pr}${repo ? ` (${repo})` : ''}`;
} else { process.stderr.write(`gstack-ship-receipt result: --pr <n> or --body <file> is required\n${HELP}\n`); return EXIT.usage; }
const bound = bindReceipt(body, pr, [head]);
const bound = bindReceipt(body, pr, [head], { conditional: true });
if (!bound.ok) return refused(bound.code, bound.message);
return writeRunRecord({ receipt: bound.receipt, head, pr, source, repoRoot, out: flag(rest, '--out'), json, line: bound.line, incomplete });
const red = bound.receipt.base_red;
if (red) process.stderr.write(`SHIP_PAUSED: base-red cases=${red.cases.length} base=${red.base.slice(0, 12)} lane=${red.lane} expires=${red.expires_at}; gstack-base-red wait --pause <file>, then /ship --resume <record>\n`);
return writeRunRecord({ receipt: bound.receipt, head, pr, source, repoRoot, out: flag(rest, '--out'), json, line: bound.line, incomplete, status: red ? 'incomplete' : 'complete' });
}
/** The run record under <state>/ship/runs/ and the terminal line, shared by `result` and `carry`. */
function writeRunRecord(a: { receipt: ShipReceipt; head: string; pr: string | undefined; source: string; repoRoot: string | undefined; out: string | undefined; json: boolean; line: string; incomplete: (why: string) => number; extra?: Record<string, unknown> }): number {
function writeRunRecord(a: { receipt: ShipReceipt; head: string; pr: string | undefined; source: string; repoRoot: string | undefined; out: string | undefined; json: boolean; line: string; incomplete: (why: string) => number; extra?: Record<string, unknown>; status?: 'complete' | 'incomplete' }): number {
const status = a.status ?? 'complete';
const stamp = new Date().toISOString().replace(/[:.]/g, '-');
const dir = a.out ? path.resolve(a.out) : path.join(resolveStateRoot(), 'ship', 'runs', `${stamp}-${a.head.slice(0, 12)}`);
const receiptPath = path.join(dir, 'ship-receipt.md');
@@ -370,15 +379,15 @@ function writeRunRecord(a: { receipt: ShipReceipt; head: string; pr: string | un
fs.writeFileSync(receiptPath, renderReceiptBlock(a.receipt));
} catch (e) { return a.incomplete(`cannot write the run record under ${dir}: ${e instanceof Error ? e.message : String(e)}`); }
const run = {
schema_version: 1, skill: 'ship', status: 'complete', head: a.head, pr: a.receipt.pr ?? (a.pr ? Number(a.pr) : null), base: a.receipt.base, version: a.receipt.version ?? null,
schema_version: 1, skill: 'ship', status, head: a.head, pr: a.receipt.pr ?? (a.pr ? Number(a.pr) : null), base: a.receipt.base, version: a.receipt.version ?? null,
queue_mode: a.receipt.queue_mode ?? null, session_kind: a.receipt.session_kind ?? process.env.GSTACK_SESSION_KIND ?? null,
receipt: receiptPath, receipt_source: a.source, branch: a.repoRoot ? git(['rev-parse', '--abbrev-ref', 'HEAD'], a.repoRoot) ?? null : null, created_at: new Date().toISOString(),
...(a.extra ?? {}),
};
try { fs.writeFileSync(path.join(dir, 'ship-run.json'), JSON.stringify(run, null, 2) + '\n'); } catch (e) { return a.incomplete(`cannot write ${path.join(dir, 'ship-run.json')}: ${e instanceof Error ? e.message : String(e)}`); }
process.stderr.write(a.line + '\n');
process.stdout.write(a.json ? JSON.stringify({ ...run, run: dir }) + '\n' : `GSTACK_RESULT: skill=ship status=complete run=${dir}\n`);
return EXIT.ok;
process.stdout.write(a.json ? JSON.stringify({ ...run, run: dir }) + '\n' : `GSTACK_RESULT: skill=ship status=${status} run=${dir}\n`);
return status === 'complete' ? EXIT.ok : EXIT.fail;
}
/**
@@ -497,7 +506,7 @@ function cmdCarry(rest: string[], json: boolean): number {
if (!repoRoot) { process.stderr.write('gstack-ship-receipt carry: run it inside the git checkout whose head carries the receipt\n'); return EXIT.usage; }
const head = (flag(rest, '--head') ?? git(['rev-parse', 'HEAD'], repoRoot))?.toLowerCase();
if (!head || !/^[0-9a-f]{40,64}$/.test(head)) { process.stderr.write('gstack-ship-receipt carry: cannot resolve HEAD\n'); return EXIT.usage; }
const refused = (code: 'QUEUE_RECEIPT_MISSING' | 'RECEIPT_CARRY_REFUSED' | 'RECEIPT_CARRY_PRODUCT_CHANGED' | 'RECEIPT_CARRY_TREE_MISMATCH' | 'RECEIPT_GIT_TOO_OLD' | 'RECEIPT_CARRY_PREGATE_STALE', message: string) => {
const refused = (code: 'QUEUE_RECEIPT_MISSING' | 'QUEUE_RECEIPT_CONDITIONAL' | 'RECEIPT_CARRY_REFUSED' | 'RECEIPT_CARRY_PRODUCT_CHANGED' | 'RECEIPT_CARRY_TREE_MISMATCH' | 'RECEIPT_GIT_TOO_OLD' | 'RECEIPT_CARRY_PREGATE_STALE', message: string) => {
process.stderr.write(resultLineWithDocs('gstack-ship-receipt carry', code, message) + '\n');
process.stdout.write(json ? JSON.stringify({ skill: 'ship', status: 'refused', run: '-', code, message }) + '\n' : `CARRY: refused (${code})\nGSTACK_RESULT: skill=ship status=refused run=-\n`);
return EXIT.refused;
@@ -529,6 +538,7 @@ function cmdCarry(rest: string[], json: boolean): number {
if (!ex.receipt || ex.errors.length) return refused('QUEUE_RECEIPT_MISSING', `${source}'s ship receipt is invalid (${ex.errors[0]?.message ?? 'invalid'}); run /ship`);
const receipt = ex.receipt;
if (pr !== undefined && receipt.pr !== undefined && String(receipt.pr) !== pr) return refused('RECEIPT_CARRY_REFUSED', `the receipt names pr=${receipt.pr}, not this PR`);
if (receipt.base_red) return refused('QUEUE_RECEIPT_CONDITIONAL', 'the receipt is paused on a red base, so there is nothing passing to carry; /ship --resume <record> re-runs Step 5 and writes a full receipt');
const from = receipt.head.toLowerCase();
if (!/^[0-9a-f]{40,64}$/.test(from)) return refused('RECEIPT_CARRY_REFUSED', `the receipt's head ${receipt.head.slice(0, 12)} is not a full commit id`);
if (from === head) {
captain/templates/brief.md
+1
View File
@@ -34,6 +34,7 @@ Spend cap: <focused paid runs per PR, from SPEND>
3. Ship. CHANGELOG entry under [Unreleased]; GUARD_CHECKS, pregate and /ship at the final
head; one unstamped READY block (captain/templates/READY.md) to me; the PR body carries
gstack-approvals render. Never merge; stamp only on my slot call.
The lane subscribes to the PR, then stops its /ship subagent at READY (docs/captain.md, the lane's /ship sequence).
Done when: the READY block is sent at a frozen head, reports/<thread code>.md is written,
and its Approvals table carries the plan approval row (the owner's words, the date) or the
capy/skill/gstack/SKILL.md
+1
View File
@@ -55,6 +55,7 @@ Every Capy thread that uses gstack follows these for new work, under a Captain o
- **Plan first.** Before building new functionality, run `/autoplan` on it as the first act after the install check, sized to the work (scope flags and `--max-rounds`; a one-file fix gets `--no-ui --no-dx --max-rounds 1` with `--estimate-usd` and a small `--spend-cap`, never no run).
- **ELI5, then the word.** Send the plan and gate to the person, or the Captain, in plain language: `gstack-owner-brief render --captain --out <run dir> --captain-name "<Captain name>"`, its choices as lettered lines in the ask format (`docs/captain.md`) and "do it" taking every recommendation at full scope. Build only after their word is answered into the run and it reports `APPROVED`; then `gstack-approvals import --autoplan <run dir> --by "<owner>"`.
- **/ship before ready.** Run /ship at the final pushed head before a PR is called ready; `SHIP_RECEIPT: pr=<n> head=<head>` must name that head, and a later push needs a new /ship. The review inside /ship, like the autoplan, is sized to the change.
- **Stop a finished /ship subagent.** When a /ship subagent opens its PR, the parent subscribes to that PR with Capy's subscribe tool and keeps that one subscription until the PR merges or closes, re-subscribing if the watch expires, then cancels it. Once the subscription is in place and READY arrives, the parent stops the subagent and owns the PR from then on, slot stamp, re-READY and receipt carry included. A stopped /ship subagent woken by a check, review or merge notice, not by its parent, reports one line naming what woke it, its pin and the PR head (`gh pr view <n> --json headRefOid --jq .headRefOid`) and ends the turn without acting, whether or not the head moved. A message from its parent is an instruction it carries out.
- **No merge without the owner's word.** A thread never merges on its own; under a Captain the controller merges on a recorded standing approval, and a thread with no Captain asks the person.
Exempt from the plan step: read-only work, the daily flaky pass's repairs, slot stamps, CHANGELOG-only resolves, cap re-pins under a written grant, S1 emergencies (a broken install or upgrade, a red main, a security bug) and fixes from an already-diagnosed, audited problem list. Every other fix gets a one-round, eng-only plan.
changelog.d/capy-gst-495-cso-sbom-release-tag.md
-8
View File
@@ -1,8 +0,0 @@
**The CSO runtime image workflow passes again after Docker moved its `scout-sbom-indexer:1` tag.** All 10 `validate-native` jobs in Stage CSO Runtime Images were failing with `SOURCE_TAG_MOVED: docker.io/docker/scout-sbom-indexer:1` on every PR that touched CSO paths. Docker pushed release 1.29.0 to the floating `:1` tag on 2026-10-29. The reviewed index `sha256:4b67f29e…` is Docker's 1.24.0 release, and the `1.24.0` tag still points at it. So the SBOM generator source now names `:1.24.0`. The pinned digests and the build-inputs revision are byte-identical, no requalification trigger changes, and the 10 qualified runtimes stay current.
### Changed
- **`lib/cso/images/build-inputs.json`**: `sbomGenerator.source` is `docker.io/docker/scout-sbom-indexer:1.24.0` instead of `:1`.
- **`scripts/cso-image-matrix.ts`**: every pinned family (SBOM generator, base images, uv) must name an exact release in its source tag: `<major>.<minor>.<patch>`, or `<major>.<minor>` for the PostgreSQL base, whose releases have two parts. A floating major or minor tag (`:1`, `:1.24`, `python:3.13-slim`), `latest` or another alias fails with `FLOATING_<FAMILY>_SOURCE` before any build. An exact tag can still be re-pushed upstream (Docker's official images rebuild them when their Debian base changes), and `SOURCE_TAG_MOVED` still reports that. Moving the generator to 1.29.0 is a separate change that needs a new build-inputs revision and a requalified catalog.
### Tests
- `test/cso-distribution.test.ts`: `:1`, `:1.24`, `:1.24-slim`, `:latest` and `:stable` are refused for the SBOM generator and for a base image; PostgreSQL accepts `17.2` but not `17`, Python refuses `3.13-slim-bookworm`; and the committed inputs build all 10 rows with the `1.24.0` source.
docs/TESTING_INTERNALS.md
+68
View File
@@ -1034,6 +1034,22 @@ never forwards `UBICLOUD_*`. All of this is pinned offline in
`test/ubicloud-runner.test.ts` against a fake API and a fake `ssh`, including
a fixture whose fake API creates, so `run` and `verify` execute end to end.
`run` and `verify` start their setup, command and tier phases detached on the
VM (output, leader pid and exit status in `~/.ubirun-<phase>.*`) and key each
run locally before any VM exists (content tree, command, setup content, env,
pass-value hashes, size, location, image, storage, origin/HEAD, the runner's
own sha256). A rerun of the same command after the runner died reattaches to
the live VM and continues from the first phase without an exit file
(`UBI_ATTACH_BUSY` while another runner owns it, exit 125 `UBI_ATTACH_LOST`
when the session left no status); HUP detaches (`UBI_DETACHED`, exit 129), INT
and TERM still tear down. A `.gstack/ubicloud.json` with `tiers` records
start/pass/fail per tier under `UBI_CHECKPOINT_DIR/<run key>/`, so a resume on
a fresh VM reruns setup and only the tiers without a pass; `pregate-remote.sh`
adds its tree and planned-set digest to the key, so a rerun of
`gstack-pregate --stage tests --remote` reattaches and bills from the original
`created_at`. The fake VM in `test/ubicloud-runner.test.ts` runs these phases
for real in a directory.
Under 8 cores (`gstack-doctor`'s `cores` row says so), run the full unit and
E2E tiers on a remote gate: `gstack-config set test_backend ubicloud` records
the choice and the row names the runner. Resume checklist after a wake or a
@@ -1435,6 +1451,13 @@ prints a run's ledgers as a table. A flaky file is fixed at source; the retry
never masks it. `bun run eval:pass-rates` lists only the local ledger
(`flakeLedgerPath()`, overridden by `GSTACK_FLAKE_LEDGER`).
Other repositories get the same rule per test with `bin/gstack-flaky run`
([docs/flaky.md](flaky.md)): it reads a JUnit report from any runner, reruns
the command once when tests failed (counting only the runner's full green
verdict), and writes `flaky-pass` and `rerun-fail` lines to its
own ledger (`flake-ledger-junit.jsonl`, or `GSTACK_FLAKE_LEDGER_JUNIT`), kept apart
from this file-level ledger.
The aggregate job of `free-tests.yml` downloads every `flake-ledger-*` artifact
of the run and appends `bun run scripts/test-health-report.ts flake-summary <dir>`
to its job summary. The Windows lane uploads `flake-ledger-windows-<shard>`.
@@ -1637,6 +1660,51 @@ lower-case words in path components under `test/fixtures/`; a substring match
separate change with its own touchfile updates; the list is here so that change
can be scoped.
## No-gstack arm
Garry, 2026-10-31: "all of our head to head tests have to test no gstack from
here so that that case is tested obviously Do it for science". A variant that
beats another variant can still lose to the plain model, and only a no-gstack
arm shows that.
**Scope.** Every preregistered comparison where any arm loads gstack or another
skill stack: review variants, preamble loadouts, skill diets scored by behavior
evals, and model panels run under gstack or under another stack. The
preregistration says which kind it is in `## Comparison`: `stack: loaded`, or
`stack: none` with a `why:` line (a scorer calibration, for example, where no
arm loads any skill text). `gstack-eval-plan prereg --check` refuses a loaded
comparison without the arm (`PREREG_NO_GSTACK_ARM`).
**How the arm is built.** One arm per model in the comparison, named
`<model>#no-gstack` in the Arms section and listed on the `arms:` line of
`## No-gstack arm`. Arm identities are `<model>[#<loadout>]` and are priced by
the model, so the pilot reserves the no-gstack arm beside its variants under its
own attempts. The arm gets:
- the same model, settings, inputs, fixture and tools as the variants;
- a plain prompt: the task text only, with no SKILL.md, preamble, hooks,
installed skills or routing lines from gstack or any other stack;
- the same panel: the same `E2E_KINDS` kind and trial count, the same judge and
rubric, and the same blinding labels (the judge reads it as `A` or `B` like
any other arm, per [Candidate-side blinding](#candidate-side-blinding));
- the same budget treatment: it is priced and reserved in the pilot like every
other arm.
**Reporting.** The no-gstack result is one row beside the variants in every
results table and every verdict, including when it wins. A comparison that
drops the row has not reported its result. The prereg check cannot see the
report, so the `report:` line states where the row goes and the reviewer of
the result holds the author to it.
**Precedent and its gaps.** `test/helpers/arm-benchmark-harness.ts` already has
a `without-skill` arm beside `with-skill`, and it is the closest existing
control. It falls short of this rule in three ways: each arm and task runs
once rather than as a panel; the judge (`armJudge` in
`test/helpers/llm-judge.ts`) scores each diff on its own, without `A`/`B`
labels; and the candidate's temporary folder is named after the arm
(`arm-<fixture>-<arm>-*`). Existing paid cases are not retrofitted by the
change that added this rule; an audit of them is tracked separately.
## Running evals as an agent: detach
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
docs/captain.md
+13 -1
View File
@@ -243,12 +243,24 @@ a push included. The sequence that works unattended on Capy:
7. Body and receipt: compose the PR body (What changed, Why, Blast radius, Approvals first on Capy),
write the ship receipt into it, post body and title.
8. Terminal line: `gstack-ship-receipt result --pr <n>` prints
`GSTACK_RESULT: skill=ship status=complete`.
`GSTACK_RESULT: skill=ship status=complete`. `SHIP_PAUSED: base-red …` with
`status=incomplete` means Step 5 proved its failures are the base's own
([base_red](ship-policy.md#base-red)): the lane is waiting, not failed. It
registers `gstack-base-red wait --pause <file>` as a background operation,
tells the controller which cases hold it, and on `BASE_GREEN` runs
`/ship --resume <record>`; no READY goes out on the conditional receipt.
9. READY: `gstack-ship-queue ready --pr <n> --pregate <pregate.json> --screen|--no-screen`;
send the printed block.
10. After the slot call, the stamp commit keeps the pin's receipt; re-READY with the same
receipt, no second /ship.
When /ship runs in a subagent, the parent subscribes to the pull request's check and
review notices as soon as the subagent opens it, and stops the /ship subagent only once
that subscription is in place and its READY arrives. From then on the parent owns the pull
request: red checks, human review, the slot stamp and re-READY of step 10, and any receipt
carry. It does that work itself or sends it to the subagent as an explicit instruction. On
Capy the mechanics are in [the gstack skill](../capy/skill/gstack/SKILL.md#plan-approve-ship).
When the base moves under a reviewed branch, merge it in; never rebase. A merge that
leaves the product diff unchanged carries the receipt with `gstack-ship-receipt carry`
(pregate at HEAD, no review rerun), and a merge whose resolution changed product code
docs/flaky.md
+137
View File
@@ -0,0 +1,137 @@
# Flaky tests: rerun once and record it (`gstack-flaky`)
`gstack-flaky run` runs your test command, reads the JUnit report it writes,
and when tests failed, runs the command once more. A test that passes on its
rerun is a **flaky pass**: the job stays green and the flake is recorded on a ledger, so
the daily flaky pass can fix it at the source. A test that fails twice is red.
This is CREW rule 16 ("run the failed test again and note that it is flaky")
for any runner that writes JUnit: Playwright, `bun test`, jest, vitest and
pytest. gstack's own free suite keeps its file-level retry
(`GSTACK_FREE_RETRY_FLAKY`, [Flake ledger](TESTING_INTERNALS.md#flake-ledger)).
## Get started in GitHub Actions
```yaml
- uses: oven-sh/setup-bun@v2
with: { bun-version: 1.4.4 }
- name: Install gstack-flaky (pinned)
env: { GSTACK_REF: <a gstack commit sha you have tested> }
run: |
git clone --quiet --filter=blob:none https://github.com/garrytan/gstack.git "$RUNNER_TEMP/gstack"
git -C "$RUNNER_TEMP/gstack" checkout --quiet "$GSTACK_REF"
- name: E2E
env:
PLAYWRIGHT_JUNIT_OUTPUT_NAME: results/junit.xml
GSTACK_FLAKE_LEDGER_JUNIT: ${{ runner.temp }}/flaky-junit-ledger.jsonl
run: |
"$RUNNER_TEMP/gstack/bin/gstack-flaky" run --junit results/junit.xml -- npx playwright test --reporter=line,junit
- if: always()
uses: actions/upload-artifact@v4
with: { name: flaky-junit-ledger-e2e, path: "${{ runner.temp }}/flaky-junit-ledger.jsonl", if-no-files-found: ignore, retention-days: 90 }
```
Replace `GSTACK_REF` with a gstack commit you have tested; the bin needs only
Bun and git. In Actions it also writes a step-summary table of the flaky
passes and one `::warning` annotation per flaky test, so they are visible on
the run before anyone reads the ledger.
## Runners
| Runner | Command after `--` | Notes |
|---|---|---|
| Playwright | `npx playwright test --reporter=line,junit` with `PLAYWRIGHT_JUNIT_OUTPUT_NAME=<file>` | `gstack-flaky` sets `PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1` so Playwright's own `retries` are reported; with retries configured, those attempts stand and nothing is rerun again. |
| bun | `bun test --reporter=junit --reporter-outfile=<file>` | |
| jest, vitest, pytest (`--runner generic`) | your command, writing JUnit | Always reruns the whole command, even with `--rerun-scope test`. |
The command after `--` runs without a shell. When it needs one, wrap it as
`bash -c '<script> "$@"' _` so a targeted rerun's selection arguments reach
the runner.
## Rerun scope
**Default: the whole command.** The rerun repeats your command exactly, and
counts only on the runner's own full green verdict: it exits 0, no test fails,
every test of the first run is in its report, and the runner counts no failure
outside the report. That verdict covers every reason a run can fail, including
the ones a narrowed rerun cannot see: a test the first run never reached
(`--bail`, `--max-failures`, `-x`), a coverage threshold (however coverage is
configured, `bunfig.toml` and lcov included), a fail-on-flaky policy, a
global setup error. A test that failed in the first run and passes once in this
rerun is a flaky pass. It costs one more run of the suite, which is still far
cheaper than rerunning the CI job by hand.
**Opt-in: `--rerun-scope test`.** Reruns the failed test alone (Playwright:
`<file> --grep <title> --project <project>`; bun: `<file> --test-name-pattern <title>`)
and clears it when that rerun exits 0. Use it only when the first run's exit
status can come from failed tests alone: no coverage threshold in any form, no
`--fail-on-flaky-tests`, no other step inside the command that can fail. As a
safety net, `gstack-flaky` still reruns the whole command
(`FLAKY_RERUN: scope=command (<why>)`) when the runner exited anything but
1, its summary is missing, it stopped early (bun `Bailed out`, Playwright
`did not run`, `interrupted`, `Testing stopped early`, a global timeout), it
printed a coverage table or got `--coverage`, or it retried flaky tests itself.
It cannot see a coverage threshold configured without console output (for
example `coverageReporter = ["lcov"]` in `bunfig.toml`), which is why this
scope is opt-in.
## Rules
- **One rerun, small failures only.** A failed test gets one rerun when the
runner made no retries and at most `--max-reruns` (3) tests failed. More
failures than that is a real red, and nothing is rerun. When the run is
already red for a test that cannot be rerun, a whole-command rerun is
skipped.
- **A rerun clears a test only cleanly.** The rerun's command must exit 0 and
the test must run once and pass on its first attempt; a rerun the runner
retried again does not count.
- **Never green by accident.** The report is deleted before every attempt, and
a report that attempt did not write counts as missing. A missing, empty,
zero-test or malformed report fails, as does a non-zero exit with no failed
test (a crash or a non-test step) and a runner killed by a signal. Cancelling
`gstack-flaky` (SIGINT, SIGTERM) is passed to the runner and latched: nothing
is rerun after it, whatever exit the runner chooses. When the runner's
summary counts failures its JUnit omits (a bun file that fails to load, a
Playwright error outside any test), the run is red and nothing is rerun.
- **Reports.** After the run, `--junit` holds the first run's report, so a
later upload or test reporter sees what failed. Each rerun's report is kept
beside it as `<name>.rerun-<n>.xml`. Other reporters' outputs (Playwright's
`test-results/` and HTML report) are rewritten by a rerun; upload the JUnit
report as the record. An existing `--junit` file that is not a JUnit report,
or cannot be read, is refused (usage, exit 2), never deleted.
- **Ids.** A test is `<file>::<project>::<describe > title>`, with the project
empty outside Playwright, so a test in two browser projects is two tests.
`gstack-flaky ids --junit <file>` prints them.
- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`) and per test
that failed its rerun (`rerun-fail`), with the tree, commit, branch, run URL,
attempts, the rerun's scope (`test` or `command`) and the first lines of
the error. Ids, paths and failure text go through gstack's redaction engine
first; when the scan cannot run, they are withheld. The path is `--ledger`,
else `GSTACK_FLAKE_LEDGER_JUNIT`, else
`<state root>/flake-ledger-junit.jsonl`, and `FLAKY_RESULT` names it. It is
kept apart from the free suite's file-level ledger (`GSTACK_FLAKE_LEDGER`).
A ledger write error is a warning and never changes the verdict.
## Output and exit codes
One `FLAKY: <flaky-pass|fail|rerun-fail> <id> attempts=<…> (<reason>)` line per
test that failed at least once, `FLAKY_RERUN: scope=command (<why>)` when the
rerun repeated the whole command, then
`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
| Exit | Meaning |
|---|---|
| 0 | Every test passed, flaky passes included |
| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES` |
| 2 | Usage error, including a `--junit` path that is not a `.xml` file, or an existing file there that is not a readable JUnit report |
| 3 | No usable report, a crash, or a killed runner (a code below on stderr) |
| Code | Exit | Cause | Fix |
|---|---|---|---|
| `FLAKY_NO_RESULTS` | 3 | No report at `--junit`, an empty one, or one with no testcase | Make the command write JUnit to that path |
| `FLAKY_JUNIT_INVALID` | 3 | The report is not well-formed XML | Make sure one process writes it and nothing appends to it |
| `FLAKY_RETRIES_HIDDEN` | 3 | Playwright retried but its report omits the retries | Set `PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1` where the report is written |
| `FLAKY_COMMAND_FAILED` | 3 | Non-zero exit with no failed test | Read the command output: a real red |
| `FLAKY_COMMAND_KILLED` | 3 | The runner was killed by a signal (a cancelled job, an OOM kill) | Nothing was rerun; rerun the job |
| `FLAKY_COMMAND_NOT_STARTED` | 3 | The command could not be started | Check the argv after `--` |
| `FLAKY_UNREPORTED_FAILURES` | 1 | The runner counted failures or errors its JUnit omits | Fix the load error or the error outside any test: a real red |
| `FLAKY_LEDGER_WRITE_FAILED` | — | The ledger could not be appended (warning) | Check the path; the verdict stands |
docs/pregate.md
+134 -4
View File
@@ -88,6 +88,8 @@ the byte-pinned `/ship` renders in `test/fixtures/golden/`).
| `stall_timeout_s` | seconds a test command may print nothing before its process group is killed and the check is `incomplete` (`PREGATE_STALLED`); default 300, `0` disables; `--stall-timeout` overrides; the `--remote` gate is exempt (see [supervision](#supervision)) |
| `timeout_s` | seconds each runner process may take (the unit-file group, each serial file, each guard without its own `timeout_s`, and the remote gate's deadline) when `--timeout` is not passed; 1 to 3600, default 600. `gstack-ship-preflight` and the `finish` fallback run pregate without `--timeout`, so this is how a repo with long lanes sets theirs |
| `remote_gate` | the repo's remote test gate command; under 8 cores the local test tier becomes a `requires-remote` obligation naming it, and `--remote` runs it (see [remote gate](#remote-gate)) |
| `run_profiles` | name → `{ workflow, events, jobs, artifact, max_age_hours?, os? }`: the GitHub Actions runs `clear --run` verifies as evidence on the identical tree; a profile named after a lane (`free-tests.yml/macos-named-regressions`) clears that lane (see [run evidence](#run-evidence)) |
| `remote_gate_runs` | `{ profiles, inventory }`: the run profiles that together clear `pregate.json/remote_gate`, and the command that prints the remote gate's test files, one per line, which their runs must cover (see [run evidence](#run-evidence)) |
| `allowlist` | `[{ finding, paths?, pattern?, reason }]`: benign findings the `secrets` check accepts and names as allowlisted, by gstack-redact finding id or non-secret category, path globs and a pattern anchored at the finding; never `env.value` (see [allowlist](#allowlist)) |
Only the `origin/<base>` copy or the user-owned file below is honoured, so a
@@ -112,9 +114,10 @@ the caller's environment, never the checkout's `.env`: the bin's shebang passes
`--no-env-file`, and a bare `bun gstack-pregate` launched beside any `.env` file
refuses (`POLICY_SOURCE_UNAVAILABLE`). `verify` reads both paths from the
record, not from its own environment.
It covers `gstack-pregate run` only: `sealed` and `price_file` are read from the
base branch's copy, so the user file refuses them (`PREGATE_CONFIG_INVALID`)
rather than holding a policy nothing reads. A user file that exists but cannot
It covers `gstack-pregate run` only: `sealed`, `price_file`, `run_profiles` and
`remote_gate_runs` are read from the base branch's copy, so the user file refuses
them (`PREGATE_CONFIG_INVALID`; an empty `run_profiles` is accepted) rather than
holding a policy nothing reads. A user file that exists but cannot
be read, or a symlink on its path that points nowhere, refuses the same way
instead of falling back to defaults. A base copy git cannot read (and
`git ls-tree` cannot confirm is absent) refuses with `POLICY_SOURCE_UNAVAILABLE`
@@ -435,7 +438,7 @@ the selected local files as a `requires-remote` obligation with the command to
run, `pregate.json/remote_gate` (the declared command) or
`test_backend/<backend>` (gstack's `scripts/ubicloud/ubi-runner.sh run --src
<repo> -- '<runner> <files>'`). Without either, the tier runs locally under
the supervision above. Close the obligation one of two ways:
the supervision above. Close the obligation one of three ways:
- after running it, `gstack-pregate clear --remote pregate.json/remote_gate
--receipt <file>` with the gate's output, validated as `--remote` validates it
@@ -444,6 +447,10 @@ the supervision above. Close the obligation one of two ways:
accepted and recorded, see [below](#remote-gate-stamp-only)), or
`gstack-pregate clear --remote test_backend/<backend> --run-url <url>`, which
has no machine-checkable receipt and is recorded as `attested:<url>`; or
- when CI already ran the full suite on this tree, `gstack-pregate clear --remote
pregate.json/remote_gate --run <id|url> [--run …] --allow-repo-commands`, one
verified GitHub Actions run per `remote_gate_runs` profile (see
[run evidence](#run-evidence)); or
- `gstack-pregate --stage tests --remote --spend-cap <usd> --allow-repo-commands`,
which runs the declared `remote_gate` now (under `--policy-from head`, still the trusted copy's gate;
a `test_backend` without `remote_gate` cannot be run this way). The gate runs
@@ -611,6 +618,129 @@ Every refusal line starts with a reason code:
| `PREGATE_REMOTE_INFRA` | 4 | create, setup, SSH or the stop deadline failed, or the runner's `UBICLOUD: left=` line is not `0`; the VM, its teardown line and `spent=` follow |
| `PREGATE_REMOTE_INTERRUPTED` | 130 | TERM or INT; ubi-runner tears the VM down on its own |
<a id="run-evidence"></a>
## Run evidence: one full suite per tree (`clear --run`)
A repository that runs its full suite in CI and again as its remote gate on the
same tree paid twice for one result (gbrain: a 30–45 min Ubicloud gate, then an
18–20 min full-ci dispatch, every serial slot). `clear --run` lets a passed
GitHub Actions run stand in for the gate, on the runner's own record and never
on prose: a pasted URL is only a pointer the verifier resolves.
**Declare it on the base branch.** In `.gstack/pregate.json`:
```json
{
"run_profiles": {
"test": { "workflow": "test.yml", "events": ["workflow_dispatch", "push"], "jobs": ["full-suite"], "artifact": "full-suite-tree" },
"e2e": { "workflow": "e2e.yml", "events": ["workflow_dispatch", "push"], "jobs": ["e2e-full-suite"], "artifact": "e2e-full-suite-tree" }
},
"remote_gate_runs": { "profiles": ["test", "e2e"], "inventory": "bash scripts/pregate-remote-gate.sh --list" }
}
```
Profiles are read only from `origin/<base>`, so a pull request can tighten
them but never relax them. That covers the profile, not the suite it names: the
workflow file a run executed and the inventory command are the branch's own, as
the remote gate's script already is, so a diff to either is reviewed like any
other gate change. Each profile names the workflow file, the events it
accepts, every job that must have passed (each exactly once), and the artifact
holding the run's `tree.json`. `max_age_hours` (1–168, default 24) bounds how
old the run may be, and `os` (`Linux`, `macOS`, `Windows`) requires
`tree.json` to record that runner OS. Each workflow uploads `tree.json` from
its own job: `run_id`, `run_attempt`, `event`, `sha` (or `head_sha`),
`tree` (`git rev-parse HEAD^{tree}` as tested), `files` (every test file the
run executed), `tests` (the count) and, with `os`, the runner OS.
**What `clear --run` checks** (`lib/run-evidence.ts`), for every run, online,
once:
- The run belongs to this repository and ran for a same-repository head. Its
workflow file is the profile's, its event is `push` or `workflow_dispatch`
(a `pull_request` run tests the merge commit, not the head tree, so no
profile may accept it), and it completed with success. The API answers with
the newest attempt, so an earlier green attempt under a newer red one is
refused, and the run is read again after the evidence is collected: a new
attempt meanwhile is `run_changed`.
- Every job the profile names appears exactly once in that attempt and
succeeded. Jobs and runs are read across every page.
- `tree.json` (an artifact of at most 8 MiB) names this run, attempt, event and
head (`sha` and `head_sha`, each checked when present). It is not a reuse
record (a reuse is never evidence). Its tree equals the tree git computes for
the run's head commit. `files` is a list of paths and `tests` a
non-negative integer when present; without `tests` the file count stands in.
- No run of the same workflow on that head, or on pregate.json's own head,
whose newest attempt started at or after the second this run was created
failed, was cancelled or is still running (`run_started_at`, so a red re-run
of an older run counts, a run started in the same second counts, and
re-running this run does not hide a failure in between).
- The run finished within `max_age_hours` (its jobs' latest `completed_at`,
else the run's `updated_at`), and on the profile's OS when it names one.
Re-running only the failed jobs refuses (`artifact_mismatch`) unless the job
that uploads `tree.json` re-runs too, because that file must name the
newest attempt. The 8 MiB cap is read from the artifact's zipped size before
download, so a workflow on the branch itself can still unpack a larger file
into the temporary directory.
Then, for the lane:
- pregate.json was recorded against this repository's base, whose profiles
apply (`base_mismatch` otherwise).
- Each profile gets its own run (a run id given twice is a usage error), and
every run verifies under its profile. The assignment is searched, so a run
valid for two profiles cannot starve the other. A lane with a profile needs
one run; `pregate.json/remote_gate` needs one per `remote_gate_runs`
profile.
- All runs tested one tree: pregate.json's tree, or a stamp-only predecessor of
it under exactly the rule [above](#remote-gate-stamp-only) (a policy whose
`stamp_sensitive_gates` names `tests` refuses that substitution).
- Every run executed something (`no_tests` names one that did not), and the
runs' `files` together cover the obligation's selected test files, plus, for
the remote gate, every file the `inventory` command prints. The inventory is
a pin-file command, so it runs only with `--allow-repo-commands`, and only
while the working tree is pregate.json's tree (checked before and after it
runs); an empty or failing inventory refuses, and it runs under `pipefail`,
so a pipeline whose first command fails refuses too.
- Every run is read once more after the inventory: a run that moved to a new
attempt or stopped being a success meanwhile refuses (`run_changed`).
Any API or git error refuses. Each refusal prints `REUSE: refused
reason=<code> — …` as `PREGATE_RECEIPT_INVALID`, and a run that fails
verification is named with its own reason (`fork_run`, `event_mismatch`,
`run_not_success`, `job_missing`, `artifact_mismatch`, `artifact_reused`,
`tree_mismatch`, `newer_run_vetoes`, `run_changed`, `run_stale`,
`os_mismatch`, `api_error`, …); the clear-level codes are `base_mismatch`,
`no_verified_run`, `tree_mismatch` (runs on different trees),
`tree_not_stamp_only`, `no_tests`, `inventory_failed`, `inventory_empty`,
`inventory_uncovered` (the remote gate) and `tests_uncovered` (a lane's own
selected tests). A wrong run count, a repeated run or a lane with no
profile is a usage error (exit 2), not a REUSE refusal.
Accepted, `clear` prints `REUSE: accepted <lane> <receipt>`, and the receipt
reads `remote-gate tree=<t12> tests=<n> runs=<id>/<attempt>,… profiles=<p>,…
heads=<h12> verified=<time>` (with `accepted-for=` and the substitution for a
stamp-only predecessor). A lane with its own profile gets `run tree=… runs=…`
the same way; for such a lane `--run-url` is verified as `--run` is, and
`--bundle-lane` is refused. A lane with no profile keeps today's `--run-url`
recording. When the base config or the GitHub origin cannot be read, `--run`
fails, and so do `--run-url` and `--bundle-lane` unless the base was read and
shows the lane has no profile; then the lane is recorded as before, with a
warning naming the cause. `gstack-ship-queue ready --dispatch-ci` still
records its dispatched runs by URL; routing it through the verifier is the next
change (P3b), so a repository that declares lane profiles clears those lanes
with `gstack-pregate clear --run`.
**`verify` stays offline.** The check ran at `clear`. `gstack-pregate verify`,
and `gstack-ship-queue ready` through it, refuse a run receipt whose `tree=`
is not pregate.json's tree unless a substitution for it is recorded. They make
no network call, so GitHub's availability and the run's age never sit on the
merge path.
**What is never reusable.** Security and advisory scans (osv, audit,
gitleaks) depend on the day, not on the tree, so they are not run profiles:
they run on every head as before. Pull-request required checks always run
fresh.
<a id="guards"></a>
## Guards and the lane-done rule
docs/ship-policy.md
+49
View File
@@ -61,6 +61,7 @@ refuses the file (`POLICY_GSTACK_TOO_OLD`). `validate` ends with
| `min_gstack_version` | version or `null` | `null` | The oldest gstack that may read the file; see the compatibility rule above. |
| `docs` | `{ caps: [{ path, max_bytes }] }` or `null` | `null` | Documentation files with a byte cap. `/ship` Step 14.5 runs `gstack-ship-receipt docs-caps --base <base>` (the caps come from `origin/<base>`'s policy, never the branch's copy; `--policy <file>` is for a repository bootstrapping its policy) before the verification gate and prints one `DOCS_CAP: <path> <bytes>/<max> ok\|over` line per cap; any `over` (an absent file counts) exits 1 and blocks the ship until the file is trimmed or the cap raised on the base branch (`DOCS_AUDIT_BLOCKED`). At most 64 caps, each path contained. See [the ship receipt](#ship-receipt). Needs `min_gstack_version`. |
| `train` | `{ gate_workflow, free_workflow, full_gate_check, record_artifact, max_size, fill_hours, screen_selection, trial_reservation_usd }` or `null` | `null` | Turns on [merge trains](#run-a-merge-train), read from the base revision only. Every field is required and an unknown field is `POLICY_INVALID`: `max_size` 1 to 6, `fill_hours` above 0 and at most 24, `screen_selection` `gstack-paid` \| `none`, `trial_reservation_usd` above 0 and at most 100. `null` keeps the per-PR queue exactly as it is. Needs `min_gstack_version`. |
| `base_red` | `{ setup, select, timeout_sec }` or `null` | `null` | How `/ship` Step 5 proves a failure is the base branch's own before pausing on it; see [base_red](#base-red). Read from the base revision only. Needs `min_gstack_version`. |
| `changelog_fragments` | `{ dir, required_checks? }` or `null` | `null` | PRs carry unversioned [CHANGELOG fragments](#changelog-fragments) in `dir` (one flat, repo-relative directory, no `..`) and the merge step ([`land`](#land)) assigns the version. `required_checks` (up to 50 distinct names) are the checks [`train land`](#stamped-trains-under-changelog_fragments-train-land) requires present and green on a stamp commit. `null` keeps `[Unreleased]` sections and per-PR stamps. Needs `min_gstack_version`. |
Containment, checked by `validate` and again at write time: paths and globs are
@@ -1042,6 +1043,54 @@ line, and the codes a refusal ends with are
[`REVIEW_SCOPE_INPUT`](troubleshooting.md#review-scope-input) and
[`REVIEW_SCOPE_RANGE`](troubleshooting.md#review-scope-range).
<a id="base-red"></a>
## `base_red`: pausing /ship on a red base
When the base branch is red for a reason outside the PR (Truther's #815/#818
pair each failed only on the other's broken cases), /ship's Step 5 used to fail
or ask the T3 triage question. With a `base_red` block on the base branch it can
prove the failure is the base's and pause instead:
```json
"base_red": { "setup": "bun install --frozen-lockfile", "select": "bun test {files}", "timeout_sec": 900 },
"min_gstack_version": "<the release that introduced base_red>"
```
`setup` installs dependencies in a scratch checkout (`null` when none are
needed); `select` is the lane command with a `{files}` slot that receives the
failing test files, shell-quoted; `timeout_sec` (10–7200) bounds each command.
`gstack-base-red classify --failures <evidence.json> --base <base> --pause`
reads the block from `origin/<base>` (a PR cannot declare its own) and calls a
failure base-red only when all of these hold:
1. The evidence names every failure the lane reported (`total` equals the
number of `failures`), else `BASE_RED_UNNAMED`.
2. No failing test file, and no path of the PR's diff (`git diff --name-only
<merge-base>..HEAD`) that a failure's signature names, is changed by the PR.
3. In scratch worktrees under the state root, `setup` then `select` fails at
the base head and at HEAD, each output naming the same test id and the same
first error line (digits and whitespace normalized). A setup that fails is
`BASE_RED_REPRO_ERROR`.
Anything else is `IN_BRANCH` and /ship continues its usual triage, exactly as
without the block. A base-red run writes a pause record (`PAUSE: <file>` under
`$GSTACK_STATE_ROOT/ship/base-red/`, repository and branch hashed, 24-hour
expiry), finishes every other step, and writes its receipt with `--base-red
<file>`: the receipt's `base_red` block (`cases`, `files`, `base`, `lane`,
`expires_at`) makes it **conditional**. `gstack-ship-receipt result` prints
`SHIP_PAUSED: base-red …` and `status=incomplete`; `ready`, `train build`,
`verify --train`, `carry` and every other receipt consumer refuse a conditional
receipt with `QUEUE_RECEIPT_CONDITIONAL`.
To resume, a parent runs `gstack-base-red wait --pause <file>` as a background
operation. On each new base head (`git ls-remote`) it reruns `setup` and
`select` on the paused files there and exits 0 (`BASE_GREEN: <sha>`) on the
first head that passes; a head that still fails keeps it waiting, and it exits 3
on its timeout (`BASE_RED_WAIT_TIMEOUT`) or the pause's expiry
(`BASE_RED_PAUSE_EXPIRED`). Then `/ship --resume <record>` reruns Step 5 on HEAD
merged with the new base in a scratch worktree and, when every lane passes,
rewrites the receipt without `base_red`.
<a id="receipt-carry"></a>
## Carrying a receipt across a base merge or a test-only fix
docs/standalone-binaries.md
+1
View File
@@ -18,6 +18,7 @@ Beyond the slash-command skills, gstack ships standalone CLIs for workflows that
| `gstack-review-log` | **Review-pass receipts** — `--start <skill>` captures the working-tree fingerprint before a diff review; `'<JSON>' --finish <token>` consumes that single-use, repository/branch/skill-scoped receipt. Binding requires matching start/end content and reviewer-reported `completed:true` and `converged:true`; it is not independent proof that a model read the source. `--check-shared-libs <token>` reads a current finding as JSON on stdin and checks prior Skip decisions against the actual branch, capture and eligible raw source blobs. It returns `reusable:false` when proof is missing or unsafe, including older records without logger-versioned coverage. Final review logging computes shared-code fingerprints and coverage rather than trusting supplied proof. |
| `gstack-review-read` | **Review freshness** — emits review records with computed `review_freshness.status` and `reason`: CURRENT, STALE, or UNVERIFIED for diff reviews. `/ship` and `/land-and-deploy` use the same grade; a matching commit alone never certifies a diff review. [Dashboard rules](skills.md#review-readiness-dashboard). |
| `gstack-evidence` | **Verification-evidence ledger** — `run --label <lane> -- <cmd>` transparently wraps any test command (the child's exit code always passes through) and records what ran against which working-tree fingerprint (tracked and untracked files that are not gitignored, minus an untracked `.bun/` folder at the repo root, which Bun can create during a run); a run whose files changed while it ran is recorded without a fingerprint and with up to five of the changed paths, which `check` names in its STALE reason. `check` grades each label FRESH/STALE/MISSING with `--expect-cmd`, `--max-age`, and `--allow-paths` binding. /ship and /land-and-deploy cite fresh evidence instead of re-running suites. Per-run logs are 0600, capped at 2MB, pruned after 30 days; the ledger and logs stay machine-local by design. |
| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
| `gstack-pr-wait` | **PR event watcher** — `gstack-pr-wait <pr> [--repo owner/name] [--until green\|ready\|any-change] [--interval 20] [--timeout 60] [--exclude CHECK,...]` polls one pull request through `gh` and prints one stdout line per real event (a check failed with its link, all checks finished, a comment or review from someone other than the author, a review bot finished on the head, the base moved and its new commits touch this PR's files, BEHIND or DIRTY, merged or closed), with progress on stderr, so a monitor wakes the agent once per event instead of on a timer. `green` gates on every check on the head (an advisory bot's review is never waited for, its checks gate like any other), as `gstack-ci-gate` does; a check the base branch requires (rulesets and classic branch protection) is pending until it appears, never green by omission; two identical polls confirm a check set is complete; `--exclude` leaves a named non-required check out of the verdict as `gstack-ci-gate --exclude` does (a required name is refused and gates). Exit 0 when the condition is met, 1 when it needs action or `--timeout` passes (`PR_WAIT_BLOCKED`), 2 on bad input, a second watcher for the same PR or a local error (`PR_WAIT_REFUSED`). State under `<state root>/pr-wait/`, baseline discarded after 6 hours. Review bots are configuration: `gstack-config set pr_wait_gating_bots greptile,macroscope`, `pr_wait_advisory_bots coderabbit` (advisory bots are reported, never waited on). `/land-and-deploy` Step 3 is one call of it. Design ported from tstack's pr-wait (MIT, Theo Browne; NOTICE.md). [Exit codes](troubleshooting.md#gstack-pr-wait). |
| `gstack-issue-guard` | **Tracker-text trust envelope** — fetches GitHub issue/PR text (`issue <n>`, `pr-body`, `pr-comments`, or `--stdin`) and wraps it in a labeled envelope so agents treat it as data: injection-shaped lines get labeled even through fullwidth and invisible-character evasion, and forged envelope banners are defused. Every tracker-text ingress in gstack routes through it, enforced by a CI scanner. |
| `gstack-ios-qa-daemon` | **iOS QA daemon** — Mac-side broker between an agent and a connected iPhone or iPad over USB CoreDevice. Loopback by default; `--tailnet` opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on `~/.gstack/ios-qa-daemon.pid`. See [docs/howto-ios-testing-with-gstack.md](howto-ios-testing-with-gstack.md). |
docs/troubleshooting.md
+111 -7
View File
@@ -3293,8 +3293,9 @@ repository's lockfile), then rerun `gstack-regen check` or the pre-gate.
**Meaning.** `.gstack/pregate.json` has an unknown key, a wrong type, a glob
that escapes the repository, a runner without `{files}`, or `secrets` in `warn`.
The user-owned `<state root>/projects/<slug>/pregate.json` also refuses when it
sets `sealed` or `price_file` (both are read only from the base branch's copy)
or exists but cannot be read (a dangling symlink on its path counts).
sets `sealed`, `price_file`, a non-empty `run_profiles` or `remote_gate_runs`
(all read only from the base branch's copy) or exists but cannot be read (a
dangling symlink on its path counts).
**Fix.** `gstack-pregate validate`. `secrets` and required-lane obligations
cannot be downgraded; every other check id in `warn` is honoured only from the
@@ -3490,6 +3491,12 @@ reports failures. The obligation stays uncleared. A run URL alone never clears
this obligation; a `test_backend/<backend>` obligation takes a run URL, recorded
as attested rather than validated.
`gstack-pregate clear --run <id|url>` refusals use this code too: the line
reads `gstack-pregate clear: receipt refused — REUSE: refused reason=<code> —
…`, and the GitHub Actions runs offered did not verify against the base
branch's `run_profiles`. Each reason code and
what it checks is listed in [run evidence](pregate.md#run-evidence).
Another tree is accepted in exactly one case: it differs from `pregate.json`'s
tree only in the repo's stamp rule (`.gstack/ship-policy.json` on the base
branch: `stamp_files`, `stamp_patterns`, the built-in version, manifest,
@@ -3585,7 +3592,9 @@ run, named by `workflow.yml/job`, or the small-machine `pregate.json/remote_gate
tier) has no receipt; or a `remote_gate` receipt accepted from a stamp-only
predecessor tree records `accepted_for` as a tree other than `pregate.json`'s,
or a `gated_tree` other than its receipt string's `tree=` (the file was edited
after `clear`), so it clears nothing here.
after `clear`), so it clears nothing here; or a `clear --run` receipt names
another tree than `pregate.json`'s with no substitution recorded (`run
receipt(s) verified for another tree`).
**Fix.** Dispatch that job, then
`gstack-pregate clear --remote <workflow.yml/job> --run-url <url>` (or
@@ -3593,7 +3602,11 @@ after `clear`), so it clears nothing here.
`clear --remote pregate.json/remote_gate --receipt <gate output>`: the gate's
own run on this tree, or a run on a predecessor tree that differs only by the
repo's stamp rule ([gate ahead of the stamp](pregate.md#remote-gate-stamp-only));
`verify` then prints the accepted substitution on its ok line. The ship receipt
`verify` then prints the accepted substitution on its ok line. When the base
branch declares run profiles, `clear --remote <lane> --run <id|url>` clears a
lane from verified GitHub Actions runs on this tree (for the remote gate, one
run per `remote_gate_runs` profile, with `--allow-repo-commands`); a profiled
lane refuses `--bundle-lane` ([run evidence](pregate.md#run-evidence)). The ship receipt
carries each obligation with its receipt.
<a id="issue-unreadable"></a>
@@ -3644,14 +3657,44 @@ parentheses; grep the code.
### `PREREG_INCOMPLETE`
**Meaning.** `gstack-eval-plan prereg --check <file>` found a required
preregistration section missing or empty: `## Bars`, `## Arms`, `## Decoys`,
`## Stop rules` or `## Held-out exposure` (every held-out set the plan
touches, its prior exposure and the reservation on it).
preregistration section missing or empty: `## Bars`, `## Arms`,
`## Comparison`, `## Decoys`, `## Stop rules` or `## Held-out exposure`
(every held-out set the plan touches, its prior exposure and the reservation
on it). It also fires when `## Comparison` has no `stack:` line, a value other
than `loaded` or `none`, or `stack: none` without a `why:` line, and when a
`## Diagnostics` declaration is malformed.
**Fix.** Fill the named sections before any priced call. A plan that cannot
name its stop rule or its held-out exposure is not preregistered; the pilot
refuses to start without a passing check.
<a id="prereg-no-gstack-arm"></a>
### `PREREG_NO_GSTACK_ARM`
**Meaning.** The preregistration says `stack: loaded` (some arm loads gstack or
another skill stack) and its `## No-gstack arm` section is missing, lacks one
of `arms:`, `inputs:`, `prompt:`, `panel:` or `report:`, lists an arm that
is not a `<model>#no-gstack` Arms line, or leaves a model in the comparison
without its own no-gstack arm. Arms match on whole identities, so
`claude-opus-5#no-gstack` does not cover `claude-opus-5-5`.
**Fix.** Add one `<model>#no-gstack` line per model under `## Arms`, list them
on the `arms:` line, and fill the other four keys: the same inputs, a plain
prompt with no skill text, the same panel and judge, and where the result row
goes beside the variants. When no arm loads a skill stack, write
`stack: none` with a `why:` line instead. The rule is in
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
<a id="eval-arm-duplicate"></a>
### `EVAL_ARM_DUPLICATE`
**Meaning.** `gstack-eval-plan pilot start --arms` named the same arm identity
twice, so two arms would share one set of pilot attempts.
**Fix.** Give each arm its own identity: `<model>` or `<model>#<loadout>`, for
example `claude-opus-5-5` and `claude-opus-5-5#no-gstack`. Nothing was
reserved.
<a id="eval-price-missing"></a>
### `EVAL_PRICE_MISSING`
@@ -4729,6 +4772,67 @@ written for the pinned head is the binding one and `verify --train` accepts it.
`gstack-ship-queue ready --pr <n> --screen|--no-screen`. A lane never stamps
to fix this: the stamp comes only when the slot is called.
<a id="queue-receipt-conditional"></a>
### `QUEUE_RECEIPT_CONDITIONAL`
**Meaning.** The PR body's ship receipt carries a `base_red` block: /ship's
Step 5 failed on cases `gstack-base-red` proved are the base branch's own, so
/ship paused before READY ([base_red](ship-policy.md#base-red)). Nothing
READYs, trains, carries or merges on it.
**Fix.** Wait for the base to pass those cases (`gstack-base-red wait --pause
<file>`), then run `/ship --resume <record>`, which reruns Step 5 and rewrites
the receipt without `base_red`.
<a id="base-red-unconfigured"></a>
### `BASE_RED_UNCONFIGURED`
**Meaning.** `origin/<base>`'s ship policy has no `base_red` block, so
`gstack-base-red` cannot reproduce a failure there and every Step 5 failure
stays in-branch. An invalid ship policy on the base counts the same (the
invalid key is printed first). `wait` prints it when the block was removed
after the pause.
**Fix.** Use /ship's usual triage. To let /ship pause on a red base, add
`base_red` to `.gstack/ship-policy.json` on the base branch
([base_red](ship-policy.md#base-red)).
<a id="base-red-unnamed"></a>
### `BASE_RED_UNNAMED`
**Meaning.** The failure evidence names fewer failures than the lane reported,
so an unnamed failure could be the PR's.
**Fix.** List every failing test (`id`, `file`, first error line as
`signature`) with `total` equal to the lane's failure count, and classify again.
<a id="base-red-repro-error"></a>
### `BASE_RED_REPRO_ERROR`
**Meaning.** `gstack-base-red` could not run the reproduction: the base or HEAD
did not resolve, the PR's diff was unreadable, the scratch checkout failed, or
`setup` exited non-zero. Every failure stays in-branch.
**Fix.** Read the printed detail. Fetch the base, or fix the `base_red.setup`
command on the base branch, then classify again.
<a id="base-red-pause-expired"></a>
### `BASE_RED_PAUSE_EXPIRED`
**Meaning.** The pause is older than 24 hours, so its classification is stale.
**Fix.** Report the paused PR to its Captain or owner, then start /ship again on
the current base; it classifies afresh.
<a id="base-red-wait-timeout"></a>
### `BASE_RED_WAIT_TIMEOUT`
**Meaning.** No new base head passed the paused cases before `gstack-base-red
wait`'s `--timeout`.
**Fix.** Run `wait` again (the pause stays valid until it expires), or find who
owns the base's red.
<a id="queue-receipt-depth-raised"></a>
### `QUEUE_RECEIPT_DEPTH_RAISED`
docs/unattended.md
+6 -2
View File
@@ -148,8 +148,12 @@ The other endings, each greppable: when gh cannot read the PR,
`GSTACK_RESULT: skill=ship status=incomplete run=-` (exit 1); a receipt that is
missing or names another head, `GSTACK_RESULT: skill=ship status=refused run=-`
with `QUEUE_RECEIPT_MISSING` or `QUEUE_RECEIPT_STALE` on stderr (exit 3), the
line a run that stops before publication prints itself; and `result --pr <n>`
refuses `QUEUE_RECEIPT_STALE` when HEAD is not the PR's pushed head.
line a run that stops before publication prints itself; `result --pr <n>`
refuses `QUEUE_RECEIPT_STALE` when HEAD is not the PR's pushed head; and a
receipt with a `base_red` block (Step 5 paused on a red base) prints
`SHIP_PAUSED: base-red …` on stderr, writes the run record and ends
`GSTACK_RESULT: skill=ship status=incomplete run=<dir>` (exit 1): the lane is
waiting on the base, not failed ([base_red](ship-policy.md#base-red)).
The one publication the unattended rules allow: when an open PR for the branch
exists and HEAD is its pushed head, Step 18 takes the pr-body section's
eval-plan/SKILL.md
+16 -2
View File
@@ -135,10 +135,12 @@ Write (or open) the preregistration file. Each section is required and must be f
| Section | What it holds |
|---|---|
| Bars | one line per metric: name, pass bar, direction, what a miss means for the decision |
| Arms | one line per arm: model id (in the price table), settings, per-item cost from Step 1b |
| Arms | one line per arm: the arm identity first (`<model>` or `<model>#<loadout>`), settings, per-item cost from Step 1b |
| Comparison | `stack: loaded` when any arm loads gstack or another skill stack; otherwise `stack: none` and a `why:` line |
| Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves |
| Stop rules | the spend cap, a bar missed on the pilot, a scorer below its bar, `budget_exhausted` |
| Held-out exposure | every held-out set the plan touches: prior exposure (which runs, which models), the reservation on it |
| No-gstack arm | required with `stack: loaded`: `arms:`, `inputs:`, `prompt:`, `panel:`, `report:` (below) |
```bash
~/.claude/skills/gstack/bin/gstack-eval-plan prereg --init <path>
@@ -149,6 +151,16 @@ Write (or open) the preregistration file. Each section is required and must be f
(`PREREG_INCOMPLETE`, exit 1); fill them and rerun. Do not continue on an incomplete
preregistration.
**No-gstack arm.** A head-to-head where any arm loads gstack or another skill stack
(review variants, preamble loadouts, skill diets, a model panel under a stack) also
runs each model with no skill stack at all. Add `<model>#no-gstack` to Arms for every
model, and fill `## No-gstack arm`: `arms:` lists those arms; `inputs:` the same rows
and fixture; `prompt:` the task text only, with no SKILL.md, preamble, hooks or installed
skills, and the same tools; `panel:` the same E2E kind, trials, judge and blinding labels;
`report:` its row beside the variants in every results table and verdict.
`PREREG_NO_GSTACK_ARM` (exit 1) names what is missing. The PREREG line shows
`stack=loaded no_gstack_arm=ok` when it is complete, or `stack=none` for an opt-out.
When the summarizer counts diagnostics, declare them before the freeze commit in a
`## Diagnostics` section: `summarizer: <command naming {fixture}>`, `fixture: <synthetic
records file>`, and one `| diagnostic | expected |` row per diagnostic, with the count the
@@ -169,7 +181,9 @@ falling back to the repo's own policy document (`--policy-doc <path>`), or from
`--arms <picked-models>` when the user named them. Each `ARM:` line carries its source
and the per-item cost at the stated token estimate; `price=MISSING` (`EVAL_PRICE_MISSING`)
means the model has no row in `lib/pricing.ts`: add the row from the provider's pricing
page or drop the arm. Copy the priced arms into the Arms section.
page or drop the arm. Copy the priced arms into the Arms section. An arm identity
`<model>#<loadout>` (for example `claude-opus-5-5#no-gstack`) is priced by its model and
gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`.
When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families
newest first; `docs/eval-models.md` in gstack), `arms`, `dry-run`, `pilot start` and
eval-plan/SKILL.md.tmpl
+16 -2
View File
@@ -57,10 +57,12 @@ Write (or open) the preregistration file. Each section is required and must be f
| Section | What it holds |
|---|---|
| Bars | one line per metric: name, pass bar, direction, what a miss means for the decision |
| Arms | one line per arm: model id (in the price table), settings, per-item cost from Step 1b |
| Arms | one line per arm: the arm identity first (`<model>` or `<model>#<loadout>`), settings, per-item cost from Step 1b |
| Comparison | `stack: loaded` when any arm loads gstack or another skill stack; otherwise `stack: none` and a `why:` line |
| Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves |
| Stop rules | the spend cap, a bar missed on the pilot, a scorer below its bar, `budget_exhausted` |
| Held-out exposure | every held-out set the plan touches: prior exposure (which runs, which models), the reservation on it |
| No-gstack arm | required with `stack: loaded`: `arms:`, `inputs:`, `prompt:`, `panel:`, `report:` (below) |
```bash
~/.claude/skills/gstack/bin/gstack-eval-plan prereg --init <path>
@@ -71,6 +73,16 @@ Write (or open) the preregistration file. Each section is required and must be f
(`PREREG_INCOMPLETE`, exit 1); fill them and rerun. Do not continue on an incomplete
preregistration.
**No-gstack arm.** A head-to-head where any arm loads gstack or another skill stack
(review variants, preamble loadouts, skill diets, a model panel under a stack) also
runs each model with no skill stack at all. Add `<model>#no-gstack` to Arms for every
model, and fill `## No-gstack arm`: `arms:` lists those arms; `inputs:` the same rows
and fixture; `prompt:` the task text only, with no SKILL.md, preamble, hooks or installed
skills, and the same tools; `panel:` the same E2E kind, trials, judge and blinding labels;
`report:` its row beside the variants in every results table and verdict.
`PREREG_NO_GSTACK_ARM` (exit 1) names what is missing. The PREREG line shows
`stack=loaded no_gstack_arm=ok` when it is complete, or `stack=none` for an opt-out.
When the summarizer counts diagnostics, declare them before the freeze commit in a
`## Diagnostics` section: `summarizer: <command naming {fixture}>`, `fixture: <synthetic
records file>`, and one `| diagnostic | expected |` row per diagnostic, with the count the
@@ -91,7 +103,9 @@ falling back to the repo's own policy document (`--policy-doc <path>`), or from
`--arms <picked-models>` when the user named them. Each `ARM:` line carries its source
and the per-item cost at the stated token estimate; `price=MISSING` (`EVAL_PRICE_MISSING`)
means the model has no row in `lib/pricing.ts`: add the row from the provider's pricing
page or drop the arm. Copy the priced arms into the Arms section.
page or drop the arm. Copy the priced arms into the Arms section. An arm identity
`<model>#<loadout>` (for example `claude-opus-5-5#no-gstack`) is priced by its model and
gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`.
When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families
newest first; `docs/eval-models.md` in gstack), `arms`, `dry-run`, `pilot start` and
lib/base-red.ts
+197
View File
@@ -0,0 +1,197 @@
/**
* gstack-base-red — /ship Step 5's proof that a failure belongs to the base
* branch, not to the PR (docs/ship-policy.md#base-red). A failure is base-red
* only when the base branch's own ship policy declares how to reproduce one
* (`base_red: { setup, select, timeout_sec }`), every failure Step 5 counted is
* named, no failing test file or path its signature names is in the PR's diff,
* and the restricted command fails with the same test id and signature both at
* the base head and at the PR head, each in a scratch worktree under the state
* root. Any other outcome is in-branch, so /ship stops exactly as before.
*
* A base-red /ship finishes its other steps, writes a pause record here and
* ends with a conditional receipt; `wait` re-runs the cases on each new base
* head and returns once they pass there, and `/ship --resume` re-runs Step 5.
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import type { ShipReceipt } from './headless-artifacts';
import { type BaseRedPolicy } from './ship-policy';
import { git, sha256, writeJsonAtomic } from './ship-queue-gate';
import { runBashSync } from './script-invocation';
import { resolveStateRoot } from './state-root';
export const PAUSE_HOURS = 24;
export interface FailureCase { id: string; file: string; signature: string }
export interface FailureEvidence { lane: string; total: number; failures: FailureCase[] }
export interface CaseVerdict { id: string; file: string; verdict: 'BASE_RED' | 'IN_BRANCH'; reason: string }
export type ClassifyCode = 'BASE_RED' | 'IN_BRANCH' | 'BASE_RED_UNCONFIGURED' | 'BASE_RED_UNNAMED' | 'BASE_RED_REPRO_ERROR';
export interface ClassifyResult { code: ClassifyCode; base: string; base_sha: string; head: string; lane: string; cases: CaseVerdict[]; detail: string }
export interface RunOutput { status: number | null; out: string; error?: string }
export type Runner = (cwd: string, command: string, timeoutSec: number) => RunOutput;
export interface PauseRecord {
schema_version: 1; kind: 'base-red-pause'; branch: string; head: string; base: string; base_sha: string; lane: string;
cases: FailureCase[]; created_at: string; expires_at: string;
}
export const shellRunner: Runner = (cwd, command, timeoutSec) => {
const r = runBashSync(command, { cwd, encoding: 'utf8', timeout: timeoutSec * 1000, env: { ...process.env, CI: process.env.CI ?? '1' }, maxBuffer: 64 * 1024 * 1024 });
return { status: r.status, out: `${r.stdout ?? ''}\n${r.stderr ?? ''}`, error: r.error ? r.error.message : r.signal ? `killed by ${r.signal}` : undefined };
};
/** Step 5's failure evidence file: `{ lane, total, failures: [{ id, file, signature }] }`. */
export function parseEvidence(raw: unknown): FailureEvidence | string {
const e = raw as Record<string, unknown> | null;
if (!e || typeof e !== 'object' || Array.isArray(e)) return 'must be { lane, total, failures: [{ id, file, signature }] }';
if (typeof e.lane !== 'string' || !e.lane) return 'lane must be the Step 5 lane label';
if (!Number.isInteger(e.total) || (e.total as number) < 1) return 'total must be the failure count the lane reported (an integer >= 1)';
if (!Array.isArray(e.failures)) return 'failures must be an array';
for (const [i, f] of e.failures.entries()) {
const c = f as Record<string, unknown> | null;
if (!c || typeof c !== 'object') return `failures[${i}] must be { id, file, signature }`;
for (const k of ['id', 'file', 'signature'] as const) if (typeof c[k] !== 'string' || !(c[k] as string).trim()) return `failures[${i}].${k} must be a non-empty string`;
if (path.isAbsolute(c.file as string) || (c.file as string).split(/[\\/]/).includes('..')) return `failures[${i}].file must be repo-relative`;
}
return e as unknown as FailureEvidence;
}
const normalize = (s: string) => s.replace(/\d+/g, '0').replace(/\s+/g, ' ').trim();
const quote = (s: string) => `'${s.replace(/'/g, `'\\''`)}'`;
/** Whether a restricted run reproduces a case: non-zero exit, the test id and the normalized signature in its output. */
function reproduces(run: RunOutput, c: FailureCase): string | null {
if (run.status === 0) return 'passes';
const out = normalize(run.out);
if (!out.includes(normalize(c.id))) return 'fails without naming this test';
if (!out.includes(normalize(c.signature).slice(0, 160))) return 'fails with a different signature';
return null;
}
function scratchRoot(): string {
return path.join(resolveStateRoot(), 'ship', 'base-red', 'worktrees');
}
/** Check out `sha` in a scratch worktree, run setup then select on `files`, remove the worktree. */
export function runAt(repoRoot: string, sha: string, policy: BaseRedPolicy, files: string[], runner: Runner): RunOutput & { setupFailed?: boolean } {
fs.mkdirSync(scratchRoot(), { recursive: true });
const dir = fs.mkdtempSync(path.join(scratchRoot(), `${sha.slice(0, 12)}-`));
try {
const add = git(repoRoot, ['worktree', 'add', '--detach', '-q', dir, sha]);
if (!add.ok) return { status: null, out: '', error: `cannot check out ${sha.slice(0, 12)}: ${add.err.slice(0, 160)}`, setupFailed: true };
if (policy.setup) {
const s = runner(dir, policy.setup, policy.timeout_sec);
if (s.status !== 0) return { status: s.status, out: s.out, error: `setup exited ${s.status ?? s.error ?? 'unknown'}`, setupFailed: true };
}
return runner(dir, policy.select.split('{files}').join(files.map(quote).join(' ')), policy.timeout_sec);
} finally {
git(repoRoot, ['worktree', 'remove', '--force', dir]);
fs.rmSync(dir, { recursive: true, force: true });
}
}
export function classifyBaseRed(o: { repoRoot: string; base: string; head: string; evidence: FailureEvidence; policy: BaseRedPolicy | null; runner?: Runner }): ClassifyResult {
const runner = o.runner ?? shellRunner;
const baseSha = git(o.repoRoot, ['rev-parse', '--verify', '-q', `origin/${o.base}^{commit}`]).out;
const headSha = git(o.repoRoot, ['rev-parse', '--verify', '-q', `${o.head}^{commit}`]).out;
const all = (verdict: CaseVerdict['verdict'], reason: string) => o.evidence.failures.map(c => ({ id: c.id, file: c.file, verdict, reason }));
const out = (code: ClassifyCode, cases: CaseVerdict[], detail: string): ClassifyResult => ({ code, base: o.base, base_sha: baseSha, head: headSha, lane: o.evidence.lane, cases, detail });
if (!o.policy) return out('BASE_RED_UNCONFIGURED', all('IN_BRANCH', 'the base branch declares no base_red'), `origin/${o.base}'s ship policy has no base_red block`);
if (!baseSha || !headSha) return out('BASE_RED_REPRO_ERROR', all('IN_BRANCH', 'unresolved commit'), `cannot resolve origin/${o.base} or ${o.head}`);
if (o.evidence.failures.length !== o.evidence.total) return out('BASE_RED_UNNAMED', all('IN_BRANCH', 'not every failure is named'), `the lane reported ${o.evidence.total} failures and the evidence names ${o.evidence.failures.length}`);
const mergeBase = git(o.repoRoot, ['merge-base', baseSha, headSha]).out;
const diff = git(o.repoRoot, ['diff', '--name-only', `${mergeBase || baseSha}..${headSha}`]);
if (!mergeBase || !diff.ok) return out('BASE_RED_REPRO_ERROR', all('IN_BRANCH', 'diff unreadable'), `cannot read the PR's diff against origin/${o.base}`);
const changed = diff.out.split('\n').filter(Boolean);
const touched = o.evidence.failures.map(c => ({ c, path: changed.find(p => p === c.file || c.signature.includes(p)) }));
if (touched.some(t => t.path)) {
return out('IN_BRANCH', touched.map(({ c, path: p }) => ({ id: c.id, file: c.file, verdict: 'IN_BRANCH', reason: p ? `the PR changes ${p}` : 'another failure is in-branch' })), 'the PR touches a failing test or a path its failure names');
}
const files = [...new Set(o.evidence.failures.map(c => c.file))];
const atBase = runAt(o.repoRoot, baseSha, o.policy, files, runner);
if (atBase.setupFailed) return out('BASE_RED_REPRO_ERROR', all('IN_BRANCH', 'reproduction failed at base'), atBase.error ?? 'setup failed at base');
const atHead = runAt(o.repoRoot, headSha, o.policy, files, runner);
if (atHead.setupFailed) return out('BASE_RED_REPRO_ERROR', all('IN_BRANCH', 'reproduction failed at head'), atHead.error ?? 'setup failed at head');
const cases = o.evidence.failures.map((c): CaseVerdict => {
const base = reproduces(atBase, c);
if (base) return { id: c.id, file: c.file, verdict: 'IN_BRANCH', reason: `${base} at base ${baseSha.slice(0, 12)}` };
const head = reproduces(atHead, c);
if (head) return { id: c.id, file: c.file, verdict: 'IN_BRANCH', reason: `the restricted command ${head} at head` };
return { id: c.id, file: c.file, verdict: 'BASE_RED', reason: `fails the same way at base ${baseSha.slice(0, 12)}` };
});
const red = cases.every(c => c.verdict === 'BASE_RED');
return out(red ? 'BASE_RED' : 'IN_BRANCH', cases, red ? `every failure reproduces at origin/${o.base} ${baseSha.slice(0, 12)}` : 'at least one failure is not reproduced on the base');
}
export function classifyLines(r: ClassifyResult): string[] {
return [
...r.cases.map(c => c.verdict === 'BASE_RED' ? `BASE_RED: ${c.id} (${c.file}) base=${r.base_sha.slice(0, 12)}` : `IN_BRANCH: ${c.id} (${c.file}): ${c.reason}`),
`BASE_RED_VERDICT: ${r.code === 'BASE_RED' ? 'base-red' : 'in-branch'} lane=${r.lane} base=origin/${r.base}@${r.base_sha.slice(0, 12) || '-'} (${r.code}: ${r.detail})`,
];
}
/** `<state root>/ship/base-red/<repo>/<branch>.json`, both hashed so a branch name never becomes a path. */
export function pausePath(repoRoot: string, branch: string): string {
return path.join(resolveStateRoot(), 'ship', 'base-red', sha256(fs.realpathSync(repoRoot)).slice(0, 16), `${sha256(branch).slice(0, 16)}.json`);
}
export function writePause(repoRoot: string, branch: string, r: ClassifyResult, evidence: FailureEvidence, now = new Date()): { file: string; record: PauseRecord } {
const record: PauseRecord = {
schema_version: 1, kind: 'base-red-pause', branch, head: r.head, base: r.base, base_sha: r.base_sha, lane: r.lane, cases: evidence.failures,
created_at: now.toISOString(), expires_at: new Date(now.getTime() + PAUSE_HOURS * 3600_000).toISOString(),
};
const file = pausePath(repoRoot, branch);
writeJsonAtomic(file, record);
return { file, record };
}
export function readPause(file: string): PauseRecord | string {
let raw: unknown;
try { raw = JSON.parse(fs.readFileSync(file, 'utf8')); } catch (e: any) { return `cannot read ${file}: ${e.message}`; }
const p = raw as PauseRecord;
if (p?.kind !== 'base-red-pause' || p.schema_version !== 1 || !Array.isArray(p.cases) || !p.cases.length || typeof p.base_sha !== 'string' || typeof p.expires_at !== 'string') return `${file} is not a base-red pause record`;
return p;
}
/** The receipt's `base_red` block: what makes a receipt conditional (lib/ship-receipt.ts bindReceipt refuses it by default). */
export function receiptBaseRed(p: PauseRecord): NonNullable<ShipReceipt['base_red']> {
return { cases: p.cases.map(c => c.id), files: [...new Set(p.cases.map(c => c.file))], base: p.base_sha, lane: p.lane, expires_at: p.expires_at };
}
export type WaitResult = { code: 'BASE_GREEN'; sha: string; checked: number } | { code: 'BASE_RED_PAUSE_EXPIRED' | 'BASE_RED_WAIT_TIMEOUT'; sha: string; checked: number; detail: string };
/**
* Wait for the base to stop failing the paused cases: on each new base head
* (read with `git ls-remote`), fetch it and re-run setup and select there;
* return when that run passes. A base that moved but still fails keeps waiting.
*/
export function waitBaseGreen(o: {
repoRoot: string; pause: PauseRecord; policy: BaseRedPolicy; timeoutSec: number; pollSec: number; remote?: string;
runner?: Runner; now?: () => number; sleep?: (ms: number) => void; lsRemote?: () => string | null;
}): WaitResult {
const remote = o.remote ?? 'origin';
const now = o.now ?? Date.now;
const sleep = o.sleep ?? ((ms: number) => Bun.sleepSync(ms));
const lsRemote = o.lsRemote ?? (() => {
const r = git(o.repoRoot, ['ls-remote', remote, `refs/heads/${o.pause.base}`]);
return r.ok ? r.out.split(/\s/)[0] || null : null;
});
const files = [...new Set(o.pause.cases.map(c => c.file))];
const deadline = now() + o.timeoutSec * 1000;
const expires = Date.parse(o.pause.expires_at);
let last = o.pause.base_sha;
let checked = 0;
for (;;) {
if (now() >= expires) return { code: 'BASE_RED_PAUSE_EXPIRED', sha: last, checked, detail: `the pause expired at ${o.pause.expires_at}` };
const sha = lsRemote();
if (sha && sha !== last) {
git(o.repoRoot, ['fetch', '--no-tags', '-q', remote, `+refs/heads/${o.pause.base}:refs/remotes/${remote}/${o.pause.base}`]);
checked++;
const run = runAt(o.repoRoot, sha, o.policy, files, o.runner ?? shellRunner);
if (!run.setupFailed && run.status === 0) return { code: 'BASE_GREEN', sha, checked };
last = sha;
}
if (now() >= deadline) return { code: 'BASE_RED_WAIT_TIMEOUT', sha: last, checked, detail: `no base head passed the paused cases within ${o.timeoutSec}s` };
sleep(Math.min(o.pollSec * 1000, Math.max(0, deadline - now())));
}
}
lib/eval-plan.ts
+90 -15
View File
@@ -23,8 +23,14 @@ import { readLedger, release, reserve, settle, SpendCapExceeded, summarize, type
// ---------------------------------------------------------------------------
// preregistration
// ---------------------------------------------------------------------------
export const PREREG_SECTIONS = ['Bars', 'Arms', 'Decoys', 'Stop rules', 'Held-out exposure'] as const;
export interface PreregCheck { ok: boolean; present: string[]; missing: string[]; empty: string[]; diagnostics: PreregDiagnostics | null }
export const PREREG_SECTIONS = ['Bars', 'Arms', 'Comparison', 'Decoys', 'Stop rules', 'Held-out exposure'] as const;
/** `## Comparison`: `stack: loaded` when any arm loads gstack or another skill stack, `stack: none` with a `why:` otherwise. */
export interface ComparisonCheck { stack: 'loaded' | 'none' | null; why: string | null; errors: string[] }
/** `## No-gstack arm`, required when `stack: loaded`: one `<model>#no-gstack` Arms line per model, and the inputs, prompt, panel and report it shares with the variants. */
export interface NoGstackArmCheck { required: boolean; ok: boolean; arms: string[]; problems: string[] }
export interface PreregCheck { ok: boolean; present: string[]; missing: string[]; empty: string[]; comparison: ComparisonCheck; no_gstack: NoGstackArmCheck; diagnostics: PreregDiagnostics | null }
export const NO_GSTACK_LOADOUT = 'no-gstack';
export const NO_GSTACK_KEYS = ['arms', 'inputs', 'prompt', 'panel', 'report'] as const;
export function preregTemplate(title = 'Eval preregistration'): string {
return `# ${title}
@@ -33,7 +39,13 @@ export function preregTemplate(title = 'Eval preregistration'): string {
<!-- One line per metric: name, the pass bar, the direction, and what a miss means for the decision. -->
## Arms
<!-- One line per arm: model id (must be in lib/pricing.ts), settings, and the per-item cost from \`gstack-eval-plan arms\`. -->
<!-- One line per arm: the arm identity first (\`<model>\` or \`<model>#<loadout>\`, the model in lib/pricing.ts, e.g. \`claude-opus-5-5#no-gstack\`), settings, and the per-item cost from \`gstack-eval-plan arms\`. -->
## Comparison
<!-- \`stack: loaded\` when any arm loads gstack or another skill stack (review variants, preamble loadouts, skill diets, a model panel under a stack); otherwise \`stack: none\` and a \`why:\` line. -->
## No-gstack arm
<!-- Required with \`stack: loaded\`: \`arms:\` one \`<model>#no-gstack\` Arms line per model; \`inputs:\` the same rows and fixture; \`prompt:\` the task only (no SKILL.md, preamble, hooks or installed skills); \`panel:\` the same E2E kind, trials and judge; \`report:\` its row beside the variants. -->
## Decoys
<!-- Inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves. -->
@@ -47,14 +59,71 @@ export function preregTemplate(title = 'Eval preregistration'): string {
`;
}
/** A section's content lines (blank lines and one-line `<!-- -->` guidance dropped); null when the heading is absent. */
/** A section's content lines (blank lines and `<!-- -->` guidance, one-line or multi-line, dropped); null when the heading is absent. */
function sectionBody(text: string, name: string): string[] | null {
const lines = text.split('\n');
const headings = lines.map((l, i) => ({ i, m: /^#{1,6}\s+(.+?)\s*$/.exec(l) })).filter(x => x.m) as Array<{ i: number; m: RegExpExecArray }>;
const idx = headings.findIndex(h => h.m[1]!.toLowerCase().replace(/[^a-z]/g, '') === name.toLowerCase().replace(/[^a-z]/g, ''));
if (idx < 0) return null;
const end = idx + 1 < headings.length ? headings[idx + 1]!.i : lines.length;
return lines.slice(headings[idx]!.i + 1, end).filter(l => l.trim() && !/^\s*<!--.*-->\s*$/.test(l));
return lines.slice(headings[idx]!.i + 1, end).join('\n').replace(/<!--[\s\S]*?-->/g, '').split('\n').filter(l => l.trim());
}
/** The model an arm identity `<model>[#<loadout>]` is priced by. */
export const armModel = (id: string) => id.split('#')[0]!;
const armLoadout = (id: string) => (id.includes('#') ? id.slice(id.indexOf('#') + 1) : null);
/** The first token of each Arms line (bullets, table pipes and backticks stripped; a table's header row skipped). */
export function armIdentities(text: string): string[] {
const body = sectionBody(text, 'Arms') ?? [];
const row = (l: string) => /^\s*\|/.test(l);
const separator = (l: string | undefined) => !!l && /^[\s:|-]+$/.test(l) && l.includes('-');
const ids: string[] = [];
body.forEach((line, i) => {
if (separator(line) || (row(line) && separator(body[i + 1]))) return;
const token = line.replace(/^\s*(?:[-*+]|\d+[.)])?\s*\|?\s*/, '').split(/[\s|]+/)[0]!.replace(/[`,;:]+$/g, '').replace(/^`/, '');
if (token) ids.push(token);
});
return ids;
}
const keyValues = (body: string[]) => {
const kv: Record<string, string> = {};
for (const line of body) {
const m = /^\s*(?:[-*]\s+)?([a-z]+):(.*)$/i.exec(line);
if (m) kv[m[1]!.toLowerCase()] = m[2]!.trim();
}
return kv;
};
function checkComparison(text: string): ComparisonCheck {
const body = sectionBody(text, 'Comparison');
if (!body?.length) return { stack: null, why: null, errors: [] };
const kv = keyValues(body);
const why = kv.why || null;
if (kv.stack === undefined) return { stack: null, why, errors: ['no stack: line (stack: loaded or stack: none)'] };
if (kv.stack !== 'loaded' && kv.stack !== 'none') return { stack: null, why, errors: [`stack: ${kv.stack} is neither loaded nor none`] };
return { stack: kv.stack, why, errors: kv.stack === 'none' && !why ? ['stack: none needs a why: line saying why no arm loads a skill stack'] : [] };
}
function checkNoGstackArm(text: string, stack: ComparisonCheck['stack']): NoGstackArmCheck {
if (stack !== 'loaded') return { required: false, ok: true, arms: [], problems: [] };
const body = sectionBody(text, 'No-gstack arm');
if (!body?.length) return { required: true, ok: false, arms: [], problems: ['missing section'] };
const kv = keyValues(body);
const problems = NO_GSTACK_KEYS.filter(k => !kv[k]).map(k => `missing ${k}`);
const listed = (kv.arms ?? '').split(/[\s,]+/).map(a => a.replace(/^`|`$/g, '')).filter(Boolean);
const ids = armIdentities(text);
for (const a of listed) {
if (armLoadout(a) !== NO_GSTACK_LOADOUT) problems.push(`${a} is not a <model>#${NO_GSTACK_LOADOUT} identity`);
else if (!ids.includes(a)) problems.push(`${a} is not an Arms line`);
}
const loaded = [...new Set(ids.filter(id => armLoadout(id) !== NO_GSTACK_LOADOUT).map(armModel))];
for (const m of loaded) {
const arm = `${m}#${NO_GSTACK_LOADOUT}`;
if (!ids.includes(arm) || !listed.includes(arm)) problems.push(`no no-gstack arm for ${m}`);
}
return { required: true, ok: problems.length === 0, arms: listed, problems };
}
export function checkPrereg(text: string): PreregCheck {
@@ -63,8 +132,10 @@ export function checkPrereg(text: string): PreregCheck {
const body = sectionBody(text, name);
if (body === null) missing.push(name); else if (body.length === 0) empty.push(name); else present.push(name);
}
const comparison = checkComparison(text);
const no_gstack = checkNoGstackArm(text, comparison.stack);
const diagnostics = parsePreregDiagnostics(text);
return { ok: missing.length === 0 && empty.length === 0 && !diagnostics?.errors.length, present, missing, empty, diagnostics };
return { ok: missing.length === 0 && empty.length === 0 && !comparison.errors.length && no_gstack.ok && !diagnostics?.errors.length, present, missing, empty, comparison, no_gstack, diagnostics };
}
// ---------------------------------------------------------------------------
@@ -152,8 +223,8 @@ export const DEFAULT_TOKENS: TokenEstimate = { input: 4_000, output: 2_000 };
const POLICY_DOCS = ['docs/model-policy.md', 'MODEL_POLICY.md', 'docs/MODELS.md'];
function priceArm(model: string, source: string, provider: string | null, tokens: TokenEstimate): Arm {
const row = pricingRow(model);
return { model, provider, source, priced: !!row, input_per_mtok: row?.input_per_mtok ?? null, output_per_mtok: row?.output_per_mtok ?? null, per_item_usd: row ? estimateCostUsd(tokens, model) : null };
const row = pricingRow(armModel(model));
return { model, provider, source, priced: !!row, input_per_mtok: row?.input_per_mtok ?? null, output_per_mtok: row?.output_per_mtok ?? null, per_item_usd: row ? estimateCostUsd(tokens, armModel(model)) : null };
}
/** Models named in the repo's policy document that the price table knows. */
@@ -249,10 +320,10 @@ export function dryRun(o: DryRunOptions): DryRunResult {
if (missing.length) bad.push({ line: i + 1, ok: false, missing, input_tokens: t });
});
const retryText = o.retryPrompt && fs.existsSync(o.retryPrompt) ? fs.readFileSync(o.retryPrompt, 'utf8').trim() : null;
const missingPrice = o.arms.filter(a => !hasPricing(a));
const missingPrice = o.arms.filter(a => !hasPricing(armModel(a)));
const perItem: Record<string, number | null> = {};
const avgIn = rows ? Math.ceil(inputTokens / rows) : 0;
for (const a of o.arms) perItem[a] = hasPricing(a) ? estimateCostUsd({ input: avgIn, output: budget }, a) : null;
for (const a of o.arms) perItem[a] = hasPricing(armModel(a)) ? estimateCostUsd({ input: avgIn, output: budget }, armModel(a)) : null;
const checks = {
price_table: { ok: missingPrice.length === 0 && o.arms.length > 0, missing: missingPrice },
output_budget: { ok: budget >= REASONING_OUTPUT_FLOOR, budget, floor: REASONING_OUTPUT_FLOOR },
@@ -260,7 +331,7 @@ export function dryRun(o: DryRunOptions): DryRunResult {
} as DryRunResult['checks'];
if (o.cap !== undefined) {
const perArm: Record<string, number | null> = {};
for (const a of o.arms) perArm[a] = hasPricing(a) ? estimateCostUsd({ input: maxInput, output: budget }, a) : null;
for (const a of o.arms) perArm[a] = hasPricing(armModel(a)) ? estimateCostUsd({ input: maxInput, output: budget }, armModel(a)) : null;
checks.reservation_fits_cap = { ok: Object.values(perArm).every(v => v === null || v <= o.cap!), cap: o.cap, max_input_tokens: maxInput, per_arm: perArm };
}
if (o.modelRules) checks.model_rules = { ok: o.modelRules.ok, source: o.modelRules.source, refused: o.modelRules.errors.map(e => e.message) };
@@ -303,15 +374,19 @@ export function startPilot(o: PilotStartOptions): { record: PilotRecord; error?:
const dry = JSON.parse(fs.readFileSync(o.dryRun, 'utf8')) as DryRunResult;
if (!dry.pass) return { record: null as unknown as PilotRecord, error: { code: 'EVAL_DRY_RUN_FAILED', message: `${o.dryRun} did not pass; the pilot never starts on a failed dry run` } };
const tokens = o.tokens ?? { input: dry.rows ? Math.ceil(dry.est_input_tokens / dry.rows) : DEFAULT_TOKENS.input, output: dry.checks.output_budget.budget };
const record: PilotRecord = { schema_version: 1, n: o.n, arms: o.arms, cap_usd: o.cap, dry_run: o.dryRun, attempts: [], started_at: (o.now ?? new Date()).toISOString(), status: 'reserved' };
const keyOf = (a: string) => `${armModel(a).replace(/[^A-Za-z0-9.-]/g, '_')}${armLoadout(a) !== null ? `.${armLoadout(a)!.replace(/[^A-Za-z0-9.-]/g, '_')}` : ''}`;
const keys = o.arms.map(keyOf);
const duplicates = [...new Set(o.arms.filter((_, i) => keys.indexOf(keys[i]!) !== i).map(a => o.arms.filter(b => keyOf(b) === keyOf(a)).join(' = ')))];
if (duplicates.length) return { record, error: { code: 'EVAL_ARM_DUPLICATE', message: `arm ${duplicates.join(', ')} is listed more than once or shares one ledger attempt name; give each arm its own <model>#<loadout> identity` } };
fs.mkdirSync(o.out, { recursive: true });
const ledgerFile = path.join(o.out, PILOT_FILES.spend);
const record: PilotRecord = { schema_version: 1, n: o.n, arms: o.arms, cap_usd: o.cap, dry_run: o.dryRun, attempts: [], started_at: (o.now ?? new Date()).toISOString(), status: 'reserved' };
let error: { code: ResultCodeName; message: string } | undefined;
outer: for (const model of o.arms) {
if (!hasPricing(model)) { error = { code: 'EVAL_PRICE_MISSING', message: `${model} has no row in lib/pricing.ts` }; break; }
if (!hasPricing(armModel(model))) { error = { code: 'EVAL_PRICE_MISSING', message: `${armModel(model)} has no row in lib/pricing.ts` }; break; }
for (let i = 1; i <= o.n; i++) {
const estimate = estimateCostUsd(tokens, model);
const attempt = `pilot-${model.replace(/[^A-Za-z0-9.-]/g, '_')}-${i}`;
const estimate = estimateCostUsd(tokens, armModel(model));
const attempt = `pilot-${keyOf(model)}-${i}`;
try { reserve(ledgerFile, attempt, estimate, { cap: o.cap, label: `pilot ${model} #${i}`, now: o.now }); }
catch (e) {
if (e instanceof SpendCapExceeded) { record.status = 'budget_exhausted'; error = { code: 'EVAL_BUDGET_EXHAUSTED', message: e.message }; break outer; }
lib/flaky-run.ts
+368
View File
@@ -0,0 +1,368 @@
/**
* gstack-flaky run (GSTA-158): run a test command, read its JUnit, rerun once
* when tests failed, and record flaky passes on a ledger. This is CREW rule 16
* ("run the failed test again and note that it is flaky") for any runner that
* writes JUnit, the per-test form of the free suite's GSTACK_FREE_RETRY_FLAKY.
*
* Rules (docs/flaky.md):
* - A failed test gets one rerun when the runner made no retries of its own and
* at most maxReruns tests failed; more failures are a real red and nothing
* is rerun. When the runner retried (Playwright `retries` with retry-aware
* JUnit), its attempts stand and nothing is rerun again.
* - The rerun repeats the whole command by default, and counts only on the
* runner's own full green verdict: exit 0, no failed test, every first-run
* test present, nothing counted outside the report. Only that verdict covers
* every reason a run can fail (a test it never reached, a coverage
* threshold, a fail-on-flaky policy).
* - rerunScope 'test' (opt-in) reruns the failed test alone, and only with
* proof that the first run was complete: the runner's own summary, its
* test-failure exit status 1, no early stop, no coverage and no runner-flaky
* test; otherwise, and always for generic runners, the whole command reruns.
* - A rerun makes a flaky pass only when its command exited 0 and the test ran
* exactly once and passed on its first attempt. The command is an argv run
* without a shell; a targeted rerun appends the adapter's selection, each one
* argv element.
* - The JUnit file is removed before every attempt; a report that attempt did
* not write is missing. Missing, unreadable or empty JUnit is never green; a
* non-zero exit with no failed test and a runner killed by a signal are never
* green. When the runner's own summary counts failures or errors the JUnit
* omits (bun files that fail to load), the run is red and nothing is rerun.
* The first report is restored at the --junit path when the run ends; each
* rerun's report is kept beside it as <name>.rerun-<n>.xml.
*/
import * as fs from 'fs';
import * as path from 'path';
import { spawn as spawnChild, spawnSync } from 'child_process';
import { finalAttempt, JunitError, readJunit, type JunitReport, type JunitRunner, type JunitTest } from './junit';
import { redactFindingSpans } from './redact-engine';
import { resolveStateRoot } from './state-root';
export type LedgerKind = 'flaky-pass' | 'rerun-fail';
export interface LedgerLine {
ts: string;
runner: 'junit';
kind: LedgerKind;
id: string;
file: string;
/** 'runner' when the runner's own retries decided, 'gstack-flaky' for our rerun. */
rerun_by: 'runner' | 'gstack-flaky';
/** For gstack-flaky reruns: the failed test alone, or the whole command when the first run could not be proven complete. */
rerun_scope?: 'test' | 'command';
attempts: string[];
tree?: string;
git_sha?: string;
branch?: string;
run_url?: string;
excerpt: string[];
}
export interface RunOptions {
command: string[];
junit: string;
runner?: JunitRunner;
maxReruns: number;
/** 'command' (default): rerun the whole command. 'test': rerun the failed test alone when the first run is proven complete. */
rerunScope?: 'command' | 'test';
cwd?: string;
env?: NodeJS.ProcessEnv;
/** Runs one attempt; resolves with its exit status and the tail of its combined output. */
spawn?: (argv: string[], env: NodeJS.ProcessEnv, cwd: string) => Promise<Attempted>;
now?: () => Date;
}
export interface TestOutcome {
test: JunitTest;
status: 'flaky-pass' | 'fail' | 'rerun-fail';
attempts: string[];
reason?: string;
}
export interface Attempted { exit: number; output: string; signal?: string }
export interface RunResult {
exitCode: 0 | 1 | 3;
code?: string;
message?: string;
tests: number;
commandExit: number;
outcomes: TestOutcome[];
reruns: number;
/** Why the rerun was the whole command (the default, or why a requested targeted rerun was refused); absent for targeted reruns. */
wholeRerun?: string;
ledger: LedgerLine[];
}
export const DEFAULT_MAX_RERUNS = 3;
const escapeRegex = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
/** Selection arguments a targeted rerun appends for one test, each one argv element. */
export function rerunSelection(runner: 'playwright' | 'bun', t: JunitTest): string[] {
if (runner === 'bun') return [t.file, '--test-name-pattern', `^${escapeRegex(t.path.join(' '))}$`];
const args = [escapeRegex(t.file), '--grep', `(?:^|\\s)${escapeRegex(t.path.join(' '))}(?:\\s|$)`];
return t.project ? [...args, '--project', t.project] : args;
}
const OUTPUT_TAIL = 256 * 1024;
/** Set once gstack-flaky itself is asked to stop; every attempt after that counts as killed, whatever exit the runner chose. */
let cancelled: string | undefined;
/** Stream the child's output live (a quiet runner trips CI idle watchdogs) and keep its tail for reconciliation. */
function defaultSpawn(argv: string[], env: NodeJS.ProcessEnv, cwd: string): Promise<Attempted> {
return new Promise((resolve, reject) => {
let tail = '';
const keep = (chunk: Buffer) => { tail = (tail + chunk.toString('utf-8')).slice(-OUTPUT_TAIL); };
const child = spawnChild(argv[0]!, argv.slice(1), { stdio: ['inherit', 'pipe', 'pipe'], env, cwd });
const forward = (s: NodeJS.Signals) => { cancelled ??= s; child.kill(s); };
const release = () => { process.off('SIGINT', forward); process.off('SIGTERM', forward); };
process.on('SIGINT', forward);
process.on('SIGTERM', forward);
child.stdout!.on('data', (c: Buffer) => { process.stdout.write(c); keep(c); });
child.stderr!.on('data', (c: Buffer) => { process.stderr.write(c); keep(c); });
child.on('error', e => { release(); reject(new JunitError('FLAKY_COMMAND_NOT_STARTED', `could not start ${argv[0]}: ${e.message}`)); });
child.on('close', (code, signal) => {
release();
const killed = signal ?? cancelled;
resolve({ exit: code ?? 128, output: tail, ...(killed ? { signal: killed } : {}) });
});
});
}
const plain = (output: string) => output.replace(/\x1b\[[0-9;]*m/g, '');
/**
* Failures a runner counted that its JUnit omits: bun leaves files that fail to
* load out of the report ("N errors"), Playwright reports errors outside any
* test. A rerun cannot clear those, so they keep the run red.
*/
export function unreportedFailures(runner: JunitRunner, output: string, reportedFailed: number): string | null {
const text = plain(output);
const last = (re: RegExp) => { const m = [...text.matchAll(re)].pop(); return m ? Number(m[1]) : 0; };
if (runner === 'bun') {
const errors = last(/^\s*(\d+) errors?\s*$/gm);
const fails = last(/^\s*(\d+) fail\s*$/gm);
if (errors > 0) return `bun reported ${errors} error(s) outside the JUnit report (a file failed to load or threw between tests)`;
if (fails > reportedFailed) return `bun counted ${fails} failing tests, the JUnit report ${reportedFailed}`;
}
if (runner === 'playwright') {
const outside = last(/(\d+) errors? (?:was|were) not a part of any test/g);
const fails = last(/^\s*(\d+) failed\s*$/gm);
if (outside > 0) return `Playwright reported ${outside} error(s) outside any test`;
if (fails > reportedFailed) return `Playwright counted ${fails} failed tests, the JUnit report ${reportedFailed}`;
}
return null;
}
/**
* Why the first run cannot be trusted to have run every test, so a rerun must
* repeat the whole command; null when the runner's own output proves it
* complete and its exit status explained by the failed tests alone. A run that
* stopped early reports only the tests it reached, and a targeted rerun would
* never run the rest; a coverage threshold or a fail-on-flaky policy fails the
* run for a reason a narrowed rerun cannot reproduce.
*/
export function incompleteRun(runner: JunitRunner, output: string, exit: number, command: string[] = [], runnerFlaky = 0): string | null {
if (runner === 'generic') return 'a generic runner cannot prove every test ran';
if (exit !== 1) return `the command exited ${exit}, not the runner's test-failure status 1`;
const text = plain(output);
if (command.some(a => a === '--coverage' || a.startsWith('--coverage=')) || /%\s*Funcs\s*\|\s*%\s*Lines|coverage threshold/i.test(text)) return 'coverage is measured, and a narrowed rerun cannot reproduce a coverage failure';
if (runnerFlaky) return `the runner retried ${runnerFlaky} flaky test(s), which a fail-on-flaky policy can fail the run on`;
if (runner === 'bun') {
if (/Bailed out after/.test(text)) return 'bun stopped early (--bail)';
if (!/^\s*Ran \d+ tests? across \d+ files?/m.test(text)) return 'bun printed no run summary';
return null;
}
const notRun = text.match(/\b(\d+) (did not run|interrupted)\b/);
if (notRun) return `Playwright reports ${notRun[1]} ${notRun[2]}`;
if (/Testing stopped early|Timed out waiting [\d.]+s for the (?:entire )?test run/.test(text)) return 'Playwright stopped early';
if (!/^\s*\d+ failed\s*$/m.test(text)) return 'Playwright printed no summary (add a console reporter: --reporter=line,junit)';
return null;
}
function git(args: string[], cwd: string): string | undefined {
const r = spawnSync('git', args, { cwd, encoding: 'utf-8', timeout: 10_000 });
return r.status === 0 ? r.stdout.trim() || undefined : undefined;
}
export function provenance(cwd: string, env: NodeJS.ProcessEnv): Pick<LedgerLine, 'tree' | 'git_sha' | 'branch' | 'run_url'> {
const runUrl = env.GITHUB_SERVER_URL && env.GITHUB_REPOSITORY && env.GITHUB_RUN_ID
? `${env.GITHUB_SERVER_URL}/${env.GITHUB_REPOSITORY}/actions/runs/${env.GITHUB_RUN_ID}${env.GITHUB_RUN_ATTEMPT ? `/attempts/${env.GITHUB_RUN_ATTEMPT}` : ''}`
: undefined;
return {
tree: git(['rev-parse', 'HEAD^{tree}'], cwd),
git_sha: git(['rev-parse', 'HEAD'], cwd),
branch: env.GITHUB_HEAD_REF || env.GITHUB_REF_NAME || git(['rev-parse', '--abbrev-ref', 'HEAD'], cwd),
run_url: runUrl,
};
}
/** Test ids and paths leave this process only through the redaction engine: a parameterized title can carry a credential. */
export function safeText(s: string): string {
return redactFindingSpans(s) ?? '[withheld: the redaction scan could not run]';
}
/** Escape a value for a GitHub workflow command, so a test title cannot end the command or start another. */
export const workflowEscape = (s: string) => s.replace(/%/g, '%25').replace(/\r/g, '%0D').replace(/\n/g, '%0A');
/** Failure text leaves this process only through the redaction engine; when it cannot scan, nothing leaves. */
export function safeExcerpt(lines: string[]): string[] {
if (!lines.length) return [];
const redacted = redactFindingSpans(lines.join('\n'));
return redacted === null ? ['[excerpt withheld: the redaction scan could not run]'] : redacted.split('\n');
}
/** One attempt: remove the report, run, and read a report this attempt wrote. */
async function attempt(opts: RunOptions, argv: string[], cwd: string, env: NodeJS.ProcessEnv, runner = opts.runner): Promise<Attempted & { report?: JunitReport; error?: JunitError }> {
const junit = path.resolve(cwd, opts.junit);
fs.rmSync(junit, { force: true });
const ran = await (opts.spawn ?? defaultSpawn)(argv, env, cwd);
if (!fs.existsSync(junit)) return { ...ran, error: new JunitError('FLAKY_NO_RESULTS', `the command wrote no JUnit report at ${opts.junit} (exit ${ran.exit})`) };
try { return { ...ran, report: readJunit(junit, runner) }; } catch (e) {
if (e instanceof JunitError) return { ...ran, error: e };
throw e;
}
}
/**
* What keeps a rerun from clearing anything even when it exited 0: failures
* its runner counted outside the report and, for a whole-command rerun, any
* failed test or any first-run test missing from its report.
*/
function rerunProblem(again: Attempted & { report?: JunitReport }, first: JunitReport, whole: boolean): string | undefined {
if (!again.report) return undefined;
const failedNow = again.report.tests.filter(x => finalAttempt(x) === 'fail');
const unreported = unreportedFailures(first.runner, again.output, failedNow.length);
if (unreported) return unreported;
if (!whole) return undefined;
if (failedNow.length) return `${failedNow.length} test(s) failed in the whole-command rerun`;
const seen = new Set(again.report.tests.map(x => x.id));
const missing = first.tests.filter(x => !seen.has(x.id)).length;
return missing ? `${missing} test(s) of the first run are missing from the whole-command rerun` : undefined;
}
/** A rerun clears a test only when its command exited 0 and the test ran once and passed on its first attempt. */
function rerunVerdict(again: Attempted & { report?: JunitReport; error?: JunitError }, t: JunitTest, problem?: string): { status: 'flaky-pass' | 'rerun-fail'; attempts: string[]; reason?: string } {
const match = again.report?.tests.find(x => x.id === t.id);
if (!match) return { status: 'rerun-fail', attempts: [...t.attempts, 'fail'], reason: again.error ? `rerun: ${again.error.message}` : 'rerun: the report does not contain this test' };
const attempts = [...t.attempts, ...match.attempts];
if (again.signal) return { status: 'rerun-fail', attempts, reason: `rerun: killed by ${again.signal}` };
if (again.exit !== 0) return { status: 'rerun-fail', attempts, reason: `rerun: the command exited ${again.exit}` };
if (problem) return { status: 'rerun-fail', attempts, reason: `rerun: ${problem}` };
if (match.executions !== 1 || match.attempts.join() !== 'pass') return { status: 'rerun-fail', attempts, reason: `rerun: the test did not pass once on its first attempt (${match.attempts.join(',')})` };
return { status: 'flaky-pass', attempts };
}
export async function runFlaky(opts: RunOptions): Promise<RunResult> {
const cwd = opts.cwd ?? process.cwd();
const env = { ...(opts.env ?? process.env), PLAYWRIGHT_JUNIT_INCLUDE_RETRIES: '1' };
const now = opts.now ?? (() => new Date());
const empty = { tests: 0, outcomes: [], reruns: 0, ledger: [] };
const first = await attempt(opts, opts.command, cwd, env);
if (first.signal) return { exitCode: 3, code: 'FLAKY_COMMAND_KILLED', message: `the command was killed by ${first.signal}`, commandExit: first.exit, ...empty };
if (!first.report) return { exitCode: 3, code: first.error!.code, message: first.error!.message, commandExit: first.exit, ...empty };
const report = first.report;
if (report.retriesHidden) {
return { exitCode: 3, code: 'FLAKY_RETRIES_HIDDEN', commandExit: first.exit, ...empty, tests: report.tests.length,
message: 'Playwright retried but its JUnit omits the retries; set PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1 (gstack-flaky run sets it for the command it starts)' };
}
const failed = report.tests.filter(t => finalAttempt(t) === 'fail');
const runnerFlaky = report.tests.filter(t => finalAttempt(t) !== 'fail' && t.attempts.includes('fail'));
if (first.exit !== 0 && failed.length === 0) {
return { exitCode: 3, code: 'FLAKY_COMMAND_FAILED', commandExit: first.exit, ...empty, tests: report.tests.length,
message: `the command exited ${first.exit} but its JUnit reports no failed test; a crash or a non-test step failed` };
}
const unreported = unreportedFailures(report.runner, first.output, failed.length);
if (unreported) {
return { exitCode: 1, code: 'FLAKY_UNREPORTED_FAILURES', commandExit: first.exit, ...empty, tests: report.tests.length,
outcomes: failed.map(t => ({ test: t, status: 'fail' as const, attempts: t.attempts, reason: 'not rerun: the runner reported failures outside the JUnit report' })),
message: `${unreported}; nothing was rerun` };
}
const prov = provenance(cwd, env);
const line = (t: JunitTest, kind: LedgerKind, by: LedgerLine['rerun_by'], attempts: string[], scope?: LedgerLine['rerun_scope']): LedgerLine =>
({ ts: now().toISOString(), runner: 'junit', kind, id: safeText(t.id), file: safeText(t.file), rerun_by: by, ...(scope ? { rerun_scope: scope } : {}), attempts, ...prov, excerpt: safeExcerpt(t.excerpt) });
const ledger: LedgerLine[] = runnerFlaky.map(t => line(t, 'flaky-pass', 'runner', t.attempts));
const outcomes: TestOutcome[] = runnerFlaky.map(t => ({ test: t, status: 'flaky-pass' as const, attempts: t.attempts }));
const eligible: JunitTest[] = [];
for (const t of failed) {
if (t.executions > 1) outcomes.push({ test: t, status: 'fail', attempts: t.attempts, reason: `it ran ${t.executions} times in this report and one failed; not rerun` });
else if (t.attempts.length > 1) outcomes.push({ test: t, status: 'fail', attempts: t.attempts, reason: `the runner already retried (${t.attempts.length} attempts)` });
else if (failed.length > opts.maxReruns) outcomes.push({ test: t, status: 'fail', attempts: t.attempts, reason: `${failed.length} tests failed, more than --max-reruns ${opts.maxReruns}; not rerun` });
else eligible.push(t);
}
const wholeRerun = !eligible.length ? undefined
: opts.rerunScope !== 'test' ? 'the default; --rerun-scope test reruns the failed test alone'
: incompleteRun(report.runner, first.output, first.exit, opts.command, runnerFlaky.length) ?? undefined;
const junit = path.resolve(cwd, opts.junit);
const firstXml = fs.readFileSync(junit);
let reruns = 0;
const rerun = async (argv: string[]) => {
reruns++;
const again = await attempt(opts, argv, cwd, env, report.runner);
if (fs.existsSync(junit)) fs.renameSync(junit, `${junit.replace(/\.xml$/i, '')}.rerun-${reruns}.xml`);
return again;
};
const record = (t: JunitTest, v: ReturnType<typeof rerunVerdict>, scope: 'test' | 'command') => {
outcomes.push({ test: t, status: v.status, attempts: v.attempts, ...(v.reason ? { reason: v.reason } : {}) });
ledger.push(line(t, v.status, 'gstack-flaky', v.attempts, scope));
};
try {
if (wholeRerun && outcomes.some(o => o.status === 'fail')) {
for (const t of eligible) outcomes.push({ test: t, status: 'fail', attempts: t.attempts, reason: `not rerun: the run is already red, and a rerun would repeat the whole command (${wholeRerun})` });
} else if (wholeRerun) {
const again = await rerun(opts.command);
const problem = rerunProblem(again, report, true);
for (const t of eligible) record(t, rerunVerdict(again, t, problem), 'command');
} else if (eligible.length) {
let interrupted = false;
for (const t of eligible) {
if (interrupted) { outcomes.push({ test: t, status: 'fail', attempts: t.attempts, reason: 'not rerun: an earlier rerun was killed' }); continue; }
const again = await rerun([...opts.command, ...rerunSelection(report.runner as 'playwright' | 'bun', t)]);
interrupted = !!again.signal;
record(t, rerunVerdict(again, t, rerunProblem(again, report, false)), 'test');
}
}
} finally {
if (reruns) fs.writeFileSync(junit, firstXml);
}
const red = outcomes.some(o => o.status !== 'flaky-pass');
return { exitCode: red ? 1 : 0, tests: report.tests.length, commandExit: first.exit, outcomes, reruns, ...(wholeRerun && reruns ? { wholeRerun } : {}), ledger };
}
export function ledgerPath(flag: string | undefined, env: NodeJS.ProcessEnv = process.env): string {
if (flag) return flag;
if (env.GSTACK_FLAKE_LEDGER_JUNIT) return env.GSTACK_FLAKE_LEDGER_JUNIT;
return path.join(resolveStateRoot(env), 'flake-ledger-junit.jsonl');
}
/** Append ledger lines; an I/O error is a warning and never changes the verdict. */
export function appendLedger(lines: LedgerLine[], file: string, warn: (s: string) => void): void {
if (!lines.length) return;
try {
fs.mkdirSync(path.dirname(file), { recursive: true });
fs.appendFileSync(file, lines.map(l => JSON.stringify(l)).join('\n') + '\n');
} catch (e) {
warn(`FLAKY_LEDGER_WRITE_FAILED: could not append ${file} (${(e as Error).message}); the verdict is unaffected`);
}
}
/** Lines the bin prints: one per non-passing test, then the result line. */
export function renderResult(r: RunResult, ledger: string): string[] {
const out: string[] = [];
for (const o of r.outcomes) out.push(`FLAKY: ${o.status} ${safeText(o.test.id)} attempts=${o.attempts.join(',')}${o.reason ? ` (${o.reason})` : ''}`);
const flaky = r.outcomes.filter(o => o.status === 'flaky-pass').length;
const failed = r.outcomes.length - flaky;
if (r.wholeRerun) out.push(`FLAKY_RERUN: scope=command (${r.wholeRerun})`);
out.push(`FLAKY_RESULT: tests=${r.tests} failed=${failed} flaky=${flaky} reruns=${r.reruns} command_exit=${r.commandExit} verdict=${r.exitCode === 0 ? 'pass' : 'fail'} ledger=${r.ledger.length ? ledger : 'none'}`);
return out;
}
/** A GitHub step summary for the flaky passes, so they are seen before any report reads the ledger. */
export function stepSummary(r: RunResult): string {
const rows = r.outcomes.filter(o => o.status === 'flaky-pass');
if (!rows.length) return '';
const cell = (s: string) => s.replace(/\|/g, '\\|').replace(/`/g, "'").replace(/[\r\n]+/g, ' ');
return ['### Flaky passes (gstack-flaky)', '', '| Test | Attempts | First error |', '|---|---|---|',
...rows.map(o => `| \`${cell(safeText(o.test.id))}\` | ${o.attempts.join(', ')} | ${cell(safeExcerpt(o.test.excerpt)[0] ?? '')} |`), ''].join('\n');
}
lib/headless-artifacts.ts
+3
View File
@@ -113,6 +113,8 @@ export interface ShipReceipt {
review_loop_exception?: { accepted_by: string; date: string; via: string; reason: string; fix_check: { head: string; findings: number; files?: string[] } };
/** The targeted pass on money, gate, auth and secret code (lib/review-targeted.ts), bound to the receipt's product_patch_id; `done` only with its output collected. */
review_targeted?: { classes: string[]; rules: string; brief_sha: string; product_patch_id: string; status: 'done' | 'failed' | 'not-run'; findings: number; raised_by: string };
/** Step 5 failures gstack-base-red proved are the base's own (lib/base-red.ts): a conditional receipt the queue refuses until /ship --resume rewrites it without this block. */
base_red?: { cases: string[]; files: string[]; base: string; lane: string; expires_at: string };
}
export const RESULT_STATUSES: readonly ResultStatus[] = ['complete', 'gate_pending', 'amend_pending', 'incomplete', 'interrupted', 'refused'];
@@ -235,6 +237,7 @@ export const ARTIFACT_SCHEMAS = {
review_loop: { type: 'object', required: ['passes', 'blocking_found', 'deferred'], properties: { passes: { type: 'integer', minimum: 0 }, blocking_found: { type: 'integer', minimum: 0 }, deferred: { type: 'integer', minimum: 0 }, noted: { type: 'integer', minimum: 0 }, fixed: { type: 'integer', minimum: 0 }, scope: str, fix_checks: { type: 'integer', minimum: 0 }, dispatches: { type: 'object', description: 'reviewer dispatches by role' }, est_input_tokens: { type: 'integer', minimum: 0 } } },
carry: { type: 'object', required: ['from', 'reason', 'product_patch_id'], properties: { from: str, reason: str, product_patch_id: str, changed_paths: strList, merge_commits: { type: 'integer', minimum: 0 }, range_diff: str, covers: strList, reruns: strList, trees: strList, ancestry: { enum: ['linear', 'rewritten'] } } },
head_tree: str, product_patch_id: str, gstack_version: str,
base_red: { type: 'object', required: ['cases', 'files', 'base', 'lane', 'expires_at'], properties: { cases: strList, files: strList, base: str, lane: str, expires_at: str } },
review_depth: { type: 'object', required: ['depth', 'class', 'runtime_lines', 'rules', 'base', 'gate_scan'], properties: { depth: { enum: ['light', 'standard', 'full'] }, class: { enum: ['docs', 'tests', 'code', 'gate'] }, runtime_lines: { type: 'integer', minimum: 0 }, rules: str, base: str, registry: nullableStr, gate_scan: { type: 'object', required: ['tagged', 'head'], properties: { tagged: { type: ['integer', 'null'], minimum: 0 }, head: str } }, reasons: strList } },
review_targeted: { type: 'object', required: ['classes', 'rules', 'brief_sha', 'product_patch_id', 'status', 'findings', 'raised_by'], properties: { classes: { type: 'array', items: { enum: ['money', 'gate', 'auth', 'secret'] } }, rules: str, brief_sha: str, product_patch_id: str, status: { enum: ['done', 'failed', 'not-run'] }, findings: { type: 'integer', minimum: 0 }, raised_by: str } },
review_loop_exception: { type: 'object', required: ['accepted_by', 'date', 'via', 'reason', 'fix_check'], properties: { accepted_by: str, date: str, via: str, reason: str, fix_check: { type: 'object', required: ['head', 'findings'], properties: { head: str, findings: { type: 'integer', minimum: 0 }, files: strList } } } },
lib/junit.ts
+204
View File
@@ -0,0 +1,204 @@
/**
* JUnit XML reader for gstack-flaky (GSTA-158). One reader for every runner
* that writes JUnit: Playwright (`--reporter=junit`, retry-aware with
* PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1), `bun test --reporter=junit`, jest,
* pytest and vitest. No XML dependency: the subset JUnit uses (elements,
* attributes, text, CDATA, comments, processing instructions) is tokenized
* here.
*
* A test's id is `<file>::<project>::<title>`. The title is the describe path
* plus the test name joined by " > ", so Playwright's " › " and bun's nested
* testsuites give the same shape. The project is Playwright's project name
* (its testsuite `hostname`) and empty for every other runner, so a test
* that runs in two browser projects is two ids.
*
* Attempts: Playwright's retry-aware report lists each failed retry as
* `rerunFailure` (final result failed) or `flakyFailure` (final result
* passed). A Playwright failure whose text shows "Retry #" without those
* elements means retries ran but were not reported (`retriesHidden`). A test
* that appears more than once (repeat-each, a duplicate name) fails when any
* of its executions failed.
*/
import * as fs from 'fs';
export type JunitRunner = 'playwright' | 'bun' | 'generic';
export type Attempt = 'pass' | 'fail' | 'skip';
export interface JunitTest {
id: string;
file: string;
project: string;
/** Describe path plus test name. */
path: string[];
attempts: Attempt[];
/** Testcase elements with this id in the report (more than one: repeat-each or a duplicate name). */
executions: number;
/** First lines of the final failure, or of the first failed attempt. */
excerpt: string[];
}
export interface JunitReport {
runner: JunitRunner;
tests: JunitTest[];
retriesHidden: boolean;
}
export class JunitError extends Error {
constructor(readonly code: string, message: string) { super(message); }
}
interface XmlNode { name: string; attrs: Record<string, string>; children: XmlNode[]; text: string; parent: XmlNode | null }
const ENTITIES: Record<string, string> = { lt: '<', gt: '>', amp: '&', quot: '"', apos: "'" };
function decode(s: string): string {
return s.replace(/&(#x[0-9a-fA-F]+|#\d+|[a-z]+);/g, (m, e: string) => {
if (e[0] === '#') {
const code = e[1] === 'x' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
return Number.isFinite(code) && code <= 0x10ffff ? String.fromCodePoint(code) : m;
}
return ENTITIES[e] ?? m;
});
}
const TAG = /^<(\/?)([A-Za-z_][\w:.-]*)((?:\s+[^\s=/>]+\s*=\s*(?:"[^"]*"|'[^']*'))*)\s*(\/?)>/;
const ATTR = /([^\s=/>]+)\s*=\s*(?:"([^"]*)"|'([^']*)')/g;
export function parseXml(xml: string): XmlNode {
const root: XmlNode = { name: '#root', attrs: {}, children: [], text: '', parent: null };
let cur = root;
let i = 0;
while (i < xml.length) {
const lt = xml.indexOf('<', i);
if (lt < 0) { cur.text += decode(xml.slice(i)); break; }
if (lt > i) cur.text += decode(xml.slice(i, lt));
const rest = xml.slice(lt, lt + 9);
if (rest.startsWith('<![CDATA[')) {
const end = xml.indexOf(']]>', lt);
if (end < 0) throw new JunitError('FLAKY_JUNIT_INVALID', 'unterminated CDATA section');
cur.text += xml.slice(lt + 9, end);
i = end + 3;
continue;
}
if (rest.startsWith('<!--')) {
const end = xml.indexOf('-->', lt);
if (end < 0) throw new JunitError('FLAKY_JUNIT_INVALID', 'unterminated comment');
i = end + 3;
continue;
}
if (rest.startsWith('<?') || rest.startsWith('<!')) {
const end = xml.indexOf('>', lt);
if (end < 0) throw new JunitError('FLAKY_JUNIT_INVALID', 'unterminated declaration');
i = end + 1;
continue;
}
const m = TAG.exec(xml.slice(lt));
if (!m) throw new JunitError('FLAKY_JUNIT_INVALID', `malformed tag at offset ${lt}`);
const [whole, close, name, attrText, selfClose] = m;
if (close) {
if (cur.name !== name) throw new JunitError('FLAKY_JUNIT_INVALID', `</${name}> closes <${cur.name}> at offset ${lt}`);
cur = cur.parent!;
} else {
const attrs: Record<string, string> = {};
for (const a of attrText!.matchAll(ATTR)) attrs[a[1]!] = decode(a[2] ?? a[3] ?? '');
const node: XmlNode = { name: name!, attrs, children: [], text: '', parent: cur };
cur.children.push(node);
if (!selfClose) cur = node;
}
i = lt + whole!.length;
}
if (cur !== root) throw new JunitError('FLAKY_JUNIT_INVALID', `<${cur.name}> is never closed`);
return root;
}
function allNodes(node: XmlNode, name: string, out: XmlNode[] = []): XmlNode[] {
for (const c of node.children) {
if (c.name === name) out.push(c);
allNodes(c, name, out);
}
return out;
}
export function detectRunner(root: XmlNode): JunitRunner {
const top = root.children.find(c => c.name === 'testsuites');
if (top?.attrs.name === 'bun test') return 'bun';
const cases = allNodes(root, 'testcase');
if (cases.some(c => (c.attrs.name ?? '').includes(' › '))) return 'playwright';
const suites = allNodes(root, 'testsuite');
if (suites.length && suites.every(s => 'timestamp' in s.attrs && 'hostname' in s.attrs && !('file' in s.attrs))
&& cases.every(c => c.parent?.name === 'testsuite' && c.attrs.classname === c.parent.attrs.name)) return 'playwright';
return 'generic';
}
function excerptOf(node: XmlNode | undefined): string[] {
if (!node) return [];
const body = (node.attrs.message ? node.attrs.message + '\n' : '') + node.text;
return body.replace(/\x1b\[[0-9;]*m/g, '').split('\n').map(l => l.trimEnd()).filter(l => l.trim()).slice(0, 8);
}
export const testId = (file: string, project: string, path: string[]) => `${file}::${project}::${path.join(' > ')}`;
function readCase(tc: XmlNode, runner: JunitRunner): Omit<JunitTest, 'id' | 'executions'> & { hidden: boolean } {
const suite = tc.parent!;
let file: string, project = '', path: string[];
if (runner === 'playwright') {
file = tc.attrs.classname || suite.attrs.name || '';
project = suite.attrs.hostname ?? '';
path = (tc.attrs.name ?? '').split(' › ');
} else if (runner === 'bun') {
file = tc.attrs.file ?? '';
path = [];
for (let s: XmlNode | null = suite; s && s.name === 'testsuite'; s = s.parent) {
if ('file' in s.attrs && s.attrs.name === s.attrs.file && s.parent?.name !== 'testsuite') break;
path.unshift(s.attrs.name ?? '');
}
path.push(tc.attrs.name ?? '');
} else {
file = tc.attrs.file || tc.attrs.classname || suite.attrs.file || suite.attrs.name || '';
path = [tc.attrs.name ?? ''];
}
const kids = (n: string) => tc.children.filter(c => c.name === n);
const final = kids('failure')[0] ?? kids('error')[0];
const skipped = kids('skipped').length > 0;
const flaky = [...kids('flakyFailure'), ...kids('flakyError')];
const rerun = [...kids('rerunFailure'), ...kids('rerunError')];
let attempts: Attempt[];
if (final) attempts = [...rerun.map(() => 'fail' as const), 'fail'];
else if (skipped) attempts = ['skip'];
else attempts = [...flaky.map(() => 'fail' as const), 'pass'];
const hidden = runner === 'playwright' && !!final && rerun.length === 0 && /\bRetry #\d+/.test(final.text);
return { file, project, path, attempts, excerpt: excerptOf(final ?? flaky[0]), hidden };
}
export function parseJunit(xml: string, runner?: JunitRunner): JunitReport {
const root = parseXml(xml);
const kind = runner ?? detectRunner(root);
const byId = new Map<string, JunitTest>();
let retriesHidden = false;
for (const tc of allNodes(root, 'testcase')) {
const { hidden, ...t } = readCase(tc, kind);
retriesHidden ||= hidden;
const id = testId(t.file, t.project, t.path);
const seen = byId.get(id);
if (seen) {
const failed = finalAttempt(seen) === 'fail' || t.attempts.at(-1) === 'fail';
seen.attempts = [...seen.attempts, ...t.attempts].sort((a, b) => failed ? Number(a === 'fail') - Number(b === 'fail') : 0);
seen.executions++;
if (!seen.excerpt.length) seen.excerpt = t.excerpt;
} else byId.set(id, { id, ...t, executions: 1 });
}
if (byId.size === 0) throw new JunitError('FLAKY_NO_RESULTS', 'the JUnit report holds no testcase');
return { runner: kind, tests: [...byId.values()], retriesHidden };
}
/** Read and parse one JUnit file; missing, unreadable and empty files are errors, never green. */
export function readJunit(file: string, runner?: JunitRunner): JunitReport {
let xml: string;
try { xml = fs.readFileSync(file, 'utf-8'); } catch (e) {
throw new JunitError('FLAKY_NO_RESULTS', `cannot read ${file}: ${(e as Error).message}`);
}
if (!xml.trim()) throw new JunitError('FLAKY_NO_RESULTS', `${file} is empty`);
return parseJunit(xml, runner);
}
/** The final outcome of a test: its last attempt. */
export const finalAttempt = (t: JunitTest): Attempt => t.attempts[t.attempts.length - 1]!;
lib/pregate/config.ts
+17 -5
View File
@@ -36,6 +36,12 @@
* remote_gate the repo's remote test gate command: under 8 cores the test
* tier is a requires-remote obligation naming it, and
* `--remote` runs it (it prints `PREGATE_REMOTE: tree=… tests=… failed=…`)
* run_profiles name → `{ workflow, events, jobs, artifact, max_age_hours?, os? }`:
* the GitHub Actions runs `clear --run` accepts as evidence
* (lib/run-evidence.ts); a profile named after a lane clears it
* remote_gate_runs `{ profiles, inventory }`: the run profiles that together
* clear the remote gate obligation on the identical tree, and
* the command listing the gate's test files they must cover
* allowlist `{ finding, paths?, pattern?, reason }` entries the `secrets`
* check accepts as benign: a gstack-redact finding id or a
* non-secret category, scoped by path globs and/or a regex
@@ -53,6 +59,7 @@ import * as path from 'node:path';
import { spawnSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import type { ResultCodeName } from '../result-codes';
import { runProfileErrors, type RemoteGateRuns, type RunProfile } from '../run-evidence';
import { sealedErrors } from '../sealed-guard';
import { containedRelative, globToRegExp } from '../ship-policy';
@@ -84,15 +91,17 @@ export interface PregateConfig {
timeout_s: number | null;
remote_gate: string | null;
allowlist: AllowlistEntry[];
run_profiles: Record<string, RunProfile>;
remote_gate_runs: RemoteGateRuns | null;
}
export interface Guard { id: string; when: string[]; command: string; fix?: string; timeout_s?: number }
export interface AllowlistEntry { finding: string; paths?: string[]; pattern?: string; reason: string }
export const DEFAULT_CONFIG: PregateConfig = { runner: null, dependencies: {}, empty_lanes: [], selection: {}, warn: [], strays: { allow: [] }, local: [], guards: [], serial_patterns: DEFAULT_SERIAL_PATTERNS, price_file: null, stall_timeout_s: null, timeout_s: null, remote_gate: null, allowlist: [] };
export const DEFAULT_CONFIG: PregateConfig = { runner: null, dependencies: {}, empty_lanes: [], selection: {}, warn: [], strays: { allow: [] }, local: [], guards: [], serial_patterns: DEFAULT_SERIAL_PATTERNS, price_file: null, stall_timeout_s: null, timeout_s: null, remote_gate: null, allowlist: [], run_profiles: {}, remote_gate_runs: null };
export interface ConfigError { code: ResultCodeName; path: string; message: string }
export const MAX_LIST = 256;
const MAX_COMMAND = 512;
const KEYS = ['runner', 'dependencies', 'empty_lanes', 'selection', 'warn', 'strays', 'local', 'guards', 'serial_patterns', 'price_file', 'sealed', 'stall_timeout_s', 'timeout_s', 'remote_gate', 'allowlist'];
const KEYS = ['runner', 'dependencies', 'empty_lanes', 'selection', 'warn', 'strays', 'local', 'guards', 'serial_patterns', 'price_file', 'sealed', 'stall_timeout_s', 'timeout_s', 'remote_gate', 'allowlist', 'run_profiles', 'remote_gate_runs'];
const LANE_ID = /^[A-Za-z0-9_.-]+\.ya?ml\/[A-Za-z0-9_-]+$/;
const MAX_GUARDS = 32;
const GUARD_KEYS = ['id', 'when', 'command', 'fix', 'timeout_s'];
@@ -183,6 +192,7 @@ export function validateConfig(raw: unknown, knownChecks: readonly string[] = []
const why = commandError(obj.remote_gate);
if (why) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: 'remote_gate', message: why });
}
for (const e of runProfileErrors(obj.run_profiles, obj.remote_gate_runs)) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: e.path, message: e.message });
if ('guards' in obj) guardErrors(obj.guards, errors);
if ('allowlist' in obj) allowlistErrors(obj.allowlist, errors);
if ('sealed' in obj) for (const e of sealedErrors(obj.sealed)) errors.push({ code: 'PREGATE_CONFIG_INVALID', path: e.path, message: e.message });
@@ -251,7 +261,7 @@ export function parseConfig(text: string, knownChecks: readonly string[] = []):
const errors = validateConfig(raw, knownChecks);
if (errors.length) return { config: DEFAULT_CONFIG, errors };
const o = raw as Partial<PregateConfig>;
return { config: { $schema: o.$schema, runner: o.runner ?? null, dependencies: o.dependencies ?? {}, empty_lanes: o.empty_lanes ?? [], selection: o.selection ?? {}, warn: o.warn ?? [], strays: { allow: o.strays?.allow ?? [] }, local: o.local ?? [], guards: o.guards ?? [], serial_patterns: o.serial_patterns ?? DEFAULT_SERIAL_PATTERNS, price_file: o.price_file ?? null, stall_timeout_s: o.stall_timeout_s ?? null, timeout_s: o.timeout_s ?? null, remote_gate: o.remote_gate ?? null, allowlist: o.allowlist ?? [] }, errors: [] };
return { config: { $schema: o.$schema, runner: o.runner ?? null, dependencies: o.dependencies ?? {}, empty_lanes: o.empty_lanes ?? [], selection: o.selection ?? {}, warn: o.warn ?? [], strays: { allow: o.strays?.allow ?? [] }, local: o.local ?? [], guards: o.guards ?? [], serial_patterns: o.serial_patterns ?? DEFAULT_SERIAL_PATTERNS, price_file: o.price_file ?? null, stall_timeout_s: o.stall_timeout_s ?? null, timeout_s: o.timeout_s ?? null, remote_gate: o.remote_gate ?? null, allowlist: o.allowlist ?? [], run_profiles: o.run_profiles ?? {}, remote_gate_runs: o.remote_gate_runs ?? null }, errors: [] };
}
export type ConfigLoad = { config: PregateConfig; errors: ConfigError[]; present: boolean; label: string; kind: 'base' | 'head' | 'user' | 'none'; /** sha256 of the user file's bytes (kind 'user' only). */ digest?: string } | { unavailable: true; base: string; detail: string };
@@ -304,10 +314,10 @@ export function loadConfigFromUser(file: string, knownChecks: readonly string[]
return { ...parseConfig('{}', knownChecks), errors: [{ code: 'PREGATE_CONFIG_INVALID', path: file, message: `cannot read the user-owned policy: ${(e as Error).message}` }], present: true, label: `user ${file}`, kind: 'user' };
}
const load = parseConfig(text, knownChecks);
// `sealed` (the pre-push guard) and `price_file` (gstack-eval-plan) read only the git copies, so a non-null value in the user file would look active and do nothing.
// `sealed` (the pre-push guard), `price_file` (gstack-eval-plan) and the run profiles (`clear --run`) read only the git copies, so a non-null value in the user file would look active and do nothing.
let raw: unknown = null;
try { raw = JSON.parse(text); } catch { /* parseConfig reported it */ }
const ignored = raw && typeof raw === 'object' && !Array.isArray(raw) ? ['sealed', 'price_file'].filter(k => (raw as Record<string, unknown>)[k] != null) : [];
const ignored = raw && typeof raw === 'object' && !Array.isArray(raw) ? ['sealed', 'price_file', 'run_profiles', 'remote_gate_runs'].filter(k => { const v = (raw as Record<string, unknown>)[k]; return v != null && !(k === 'run_profiles' && typeof v === 'object' && Object.keys(v).length === 0); }) : [];
// parseConfig names the repo copy for a JSON error; this file is not it.
const errors = [...load.errors.map(e => e.path === PREGATE_CONFIG_PATH ? { ...e, path: file } : e), ...ignored.map(k => ({ code: 'PREGATE_CONFIG_INVALID' as const, path: k, message: `the user-owned policy covers gstack-pregate run only; ${k} is read from ${PREGATE_CONFIG_PATH} on the base branch` }))];
return { ...load, errors, present: true, label: `user ${file}`, kind: 'user', digest: createHash('sha256').update(text).digest('hex') };
@@ -333,6 +343,8 @@ export function explainConfig(): string {
'stall_timeout_s: seconds without test output before the lanes check kills the runner\'s process group and records it incomplete (default 300; 0 disables; --stall-timeout overrides)',
'timeout_s: seconds each runner process may take (the unit-file group, each serial file, each guard without its own timeout_s, and the remote gate\'s deadline) when --timeout is not passed; 1 to 3600, default 600',
'remote_gate: the repo\'s remote test gate command; under 8 cores the test tier becomes a requires-remote obligation naming it, and gstack-pregate --remote runs it (it must print PREGATE_REMOTE: tree=<tree> tests=<n> failed=<n>) (docs/pregate.md#remote-gate)',
'run_profiles: name → { workflow, events, jobs, artifact, max_age_hours?, os? }; the GitHub Actions runs gstack-pregate clear --run verifies as evidence on the identical tree (docs/pregate.md#run-evidence)',
'remote_gate_runs: { profiles, inventory }; the run profiles that together clear the remote gate obligation, and the command listing the gate\'s test files they must cover (docs/pregate.md#run-evidence)',
`allowlist: [{ finding, paths?, pattern?, reason }]: benign findings the secrets check accepts and names as allowlisted. finding is a gstack-redact id (env.kv, google.api_key, pii.phone.e164, …) or a category (${ALLOWLISTABLE_CATEGORIES.join(', ')}); paths are globs; pattern is a regex matched at the finding's position in the added line; one of the two is required, reason always. Never ${NON_ALLOWLISTABLE.join(' or ')} or engine.* (docs/pregate.md#allowlist)`,
'no .gstack/pregate.json on the base: <state root>/projects/<slug>/pregate.json is read instead (state root: GSTACK_STATE_ROOT, else GSTACK_HOME, else GSTACK_STATE_DIR, else a gstack plugin data dir, else ~/.gstack, per docs/state-root.md; run --explain prints the resolved path; outside the repo, so a PR cannot reach it; a path inside the checkout is ignored)',
].join('\n') + '\n';
lib/pregate/index.ts
+100 -12
View File
@@ -23,10 +23,11 @@ import { regenCheck } from './checks/regen';
import { secretsCheck } from './checks/secrets';
import { straysCheck } from './checks/strays';
import { timingCheck } from './checks/timing';
import { NON_DOWNGRADABLE, userPolicyAbsent } from './config';
import { NON_DOWNGRADABLE, PREGATE_CONFIG_PATH, userPolicyAbsent } from './config';
import { sha256, withFingerprint } from './context';
import { baseBranchOf, baseMismatchFix, describeSubstitution, substituteStampOnlyTree } from './remote-receipt';
import { resolveBaseBranch } from '../ship-policy';
import { parseRunReceipt, parseRunRef, runChanged, runReceiptFields, uncoveredFiles, verifyRun, type RemoteGateRuns, type RunDeps, type RunEvidence, type RunProfile } from '../run-evidence';
import type { CheckDef, CheckResult, PregateContext, Stage } from './types';
export const CHECKS: readonly CheckDef[] = [regenCheck, secretsCheck, straysCheck, literalsCheck, timingCheck, guardsCheck, lanesCheck];
@@ -173,6 +174,9 @@ export function verifyPregate(repoRoot: string, file: PregateFile, treeNow: Wtre
// A receipt accepted from a stamp-only predecessor tree was accepted for one head tree; recorded for another, it clears nothing here.
const misapplied = (file.requires_remote ?? []).filter(r => r.substitution && (r.substitution.accepted_for !== file.tree || !(r.receipt ?? '').includes(`tree=${r.substitution.gated_tree.slice(0, 12)} `)));
if (misapplied.length) return { ok: false, code: 'PREGATE_REMOTE_UNCLEARED', message: `${misapplied.length} remote receipt(s) accepted for another tree or recorded against another gated tree: ${misapplied.map(r => `${r.lane} (accepted for ${r.substitution!.accepted_for.slice(0, 12)} from gated ${r.substitution!.gated_tree.slice(0, 12)}, pregate.json is ${file.tree.slice(0, 12)})`).join(', ')}; clear again`, current, uncleared: misapplied };
// A run receipt was verified online at clear for its tree; offline it must still name this tree (or carry its substitution).
const foreignRuns = (file.requires_remote ?? []).filter(r => { const run = parseRunReceipt(r.receipt ?? ''); return run && !r.substitution && run.tree !== file.tree.slice(0, 12); });
if (foreignRuns.length) return { ok: false, code: 'PREGATE_REMOTE_UNCLEARED', message: `${foreignRuns.length} run receipt(s) verified for another tree: ${foreignRuns.map(r => `${r.lane} (${r.receipt})`).join(', ')}; clear again with runs on tree ${file.tree.slice(0, 12)}`, current, uncleared: foreignRuns };
const substituted = (file.requires_remote ?? []).filter(r => r.substitution).map(r => `${r.lane}: ${describeSubstitution(r.substitution!)}`);
return { ok: true, message: `pregate ${file.verdict} is current for tree ${file.tree.slice(0, 12)}${substituted.length ? `; ${substituted.join('; ')}` : ''}`, current, uncleared };
}
@@ -198,7 +202,9 @@ function userPolicyChange(file: PregateFile): string | null {
return `${absent} now holds a user-owned policy, and pregate.json was recorded under the defaults; rerun gstack-pregate`;
}
export interface ClearInput { runUrl?: string; bundleLane?: string; receiptPath?: string }
export interface ClearInput { runUrl?: string; bundleLane?: string; receiptPath?: string; runs?: string[]; evidence?: RunCheck }
/** What a run clear needs: the base branch's run profiles, the verifier's dependencies, the clock, and the remote gate's inventory. */
export interface RunCheck { repo: string; deps: RunDeps; profiles: Record<string, RunProfile>; gateRuns: RemoteGateRuns | null; now: Date; inventory: (command: string, tree: string) => string[] | { error: string } }
export type ClearDecision = { receipt: string; attested: boolean; substitution?: PregateReceiptSubstitution } | { refused: 'usage' | 'PREGATE_RECEIPT_INVALID'; message: string; fix?: string };
/**
@@ -212,11 +218,19 @@ export type ClearDecision = { receipt: string; attested: boolean; substitution?:
* the tree mismatch refuses as before. A `test_backend/<backend>` obligation
* has no machine-checkable receipt, so its run URL is recorded as attested.
* Platform lanes take a run URL or an evidence-bundle lane label.
*
* `--run` (and `--run-url` on a lane the base branch profiles) goes to
* clearByRuns, which accepts only verified Actions runs; a profiled lane
* refuses `--bundle-lane`, and the remote gate clears by runs only when the
* base declares remote_gate_runs.
*/
export function clearDecision(file: PregateFile, lane: string, input: ClearInput, repoRoot: string): ClearDecision {
const list = file.requires_remote ?? [];
const target = list.find(r => r.lane === lane);
if (!target) return { refused: 'usage', message: `no requires-remote obligation named ${lane}; obligations: ${list.map(r => r.lane).join(', ') || '(none)'}` };
const profiled = input.evidence && (lane === REMOTE_GATE_LANE ? input.evidence.gateRuns : input.evidence.profiles[lane]);
if (input.runs?.length || (input.runUrl && profiled)) return clearByRuns(file, lane, [...(input.runs ?? []), ...(input.runUrl ? [input.runUrl] : [])], input.evidence, repoRoot);
if (input.bundleLane && profiled && lane !== REMOTE_GATE_LANE) return { refused: 'usage', message: `${lane} has a run profile on the base branch; clear it with --run <id|url>, which verifies the run` };
if (target.lane === REMOTE_GATE_LANE) {
if (!input.receiptPath) return { refused: 'usage', message: `${lane} clears only with --receipt <file> holding the remote gate's PREGATE_REMOTE: tree=… tests=… failed=… line (a run URL is not checkable); or rerun gstack-pregate --remote` };
let text: string;
@@ -226,16 +240,9 @@ export function clearDecision(file: PregateFile, lane: string, input: ClearInput
if (!report) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${input.receiptPath} has no PREGATE_REMOTE: line; ${lane} stays uncleared` };
const own = remoteReportProblem(report, report.tree);
if (own) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${own}; ${lane} stays uncleared` };
let substitution: PregateReceiptSubstitution | undefined;
if (report.tree !== file.tree) {
// The stamp rule is the ship's own base branch's policy (origin/HEAD, as regate reads it), never a branch the artifact names.
const base = resolveBaseBranch(repoRoot);
const recorded = baseBranchOf(file.base);
if (recorded && recorded !== base) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${treeMismatch(report.tree, file.tree)}; pregate.json was recorded against origin/${recorded}, not this repository's base origin/${base}, so the stamp rule that would class the difference is not the base's; ${lane} stays uncleared`, fix: baseMismatchFix(repoRoot, base, recorded) };
const s = substituteStampOnlyTree(repoRoot, base, report.tree, file.tree);
if ('problem' in s) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${s.problem}; ${lane} stays uncleared`, fix: s.fix };
substitution = s.substitution;
}
const sub = treeSubstitution(file, report.tree, lane, repoRoot);
if ('refused' in sub) return sub;
const substitution = sub.substitution;
const problem = remoteReportProblem(report, substitution ? report.tree : file.tree);
if (problem) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${problem}; ${lane} stays uncleared` };
const accepted = substitution ? ` accepted-for=${file.tree.slice(0, 12)} stamp-only=${substitution.files.length}` : '';
@@ -251,6 +258,87 @@ export function clearDecision(file: PregateFile, lane: string, input: ClearInput
return { receipt, attested: false };
}
/** The stamp-only substitution that lets a receipt for `gatedTree` clear pregate.json's tree, none when the trees are equal, or the refusal. */
function treeSubstitution(file: PregateFile, gatedTree: string, lane: string, repoRoot: string): { substitution?: PregateReceiptSubstitution } | Extract<ClearDecision, { refused: string }> {
if (gatedTree === file.tree) return {};
// The stamp rule is the ship's own base branch's policy (origin/HEAD, as regate reads it), never a branch the artifact names.
const base = resolveBaseBranch(repoRoot);
const recorded = baseBranchOf(file.base);
if (recorded && recorded !== base) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${treeMismatch(gatedTree, file.tree)}; pregate.json was recorded against origin/${recorded}, not this repository's base origin/${base}, so the stamp rule that would class the difference is not the base's; ${lane} stays uncleared`, fix: baseMismatchFix(repoRoot, base, recorded) };
const s = substituteStampOnlyTree(repoRoot, base, gatedTree, file.tree);
if ('problem' in s) return { refused: 'PREGATE_RECEIPT_INVALID', message: `${s.problem}; ${lane} stays uncleared`, fix: s.fix };
return { substitution: s.substitution };
}
/**
* `clear --run`: GitHub Actions runs as the obligation's evidence (lib/run-evidence.ts). The remote gate needs one
* verified run per `remote_gate_runs` profile, all on one tree, whose tree.json file lists together cover every file
* the gate's inventory command prints; any other lane needs one verified run of the profile named after it. The run
* tree must be pregate.json's tree or a stamp-only predecessor of it, exactly as a gate report's tree must.
*/
function clearByRuns(file: PregateFile, lane: string, refs: string[], check: RunCheck | undefined, repoRoot: string): ClearDecision {
if (!check) return { refused: 'usage', message: `--run needs the base branch's run_profiles; none could be read` };
const isGate = lane === REMOTE_GATE_LANE;
const names = isGate ? check.gateRuns?.profiles ?? [] : check.profiles[lane] ? [lane] : [];
if (!names.length) return { refused: 'usage', message: isGate ? `--run clears ${lane} only when the base branch's ${PREGATE_CONFIG_PATH} declares remote_gate_runs` : `--run clears ${lane} only when the base branch's ${PREGATE_CONFIG_PATH} declares a run_profiles entry named ${lane}` };
if (refs.length !== names.length) return { refused: 'usage', message: `${lane} clears with exactly ${names.length} run(s), one per profile (${names.join(', ')}); got ${refs.length}` };
const ids = refs.map(ref => parseRunRef(ref, check.repo) ?? ref);
if (new Set(ids).size !== ids.length) return { refused: 'usage', message: `${lane} needs a different run for each profile; got ${refs.join(', ')}` };
// The profiles are the base branch's; a pregate.json recorded against another base names another authority.
const base = resolveBaseBranch(repoRoot);
const recorded = baseBranchOf(file.base);
if (recorded && recorded !== base) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=base_mismatch — pregate.json was recorded against origin/${recorded}, but run profiles come from this repository's base origin/${base}; ${lane} stays uncleared`, fix: baseMismatchFix(repoRoot, base, recorded) };
const verdicts = new Map<string, ReturnType<typeof verifyRun>>();
const verdict = (name: string, ref: string) => {
const key = `${name}\0${ref}`;
if (!verdicts.has(key)) verdicts.set(key, verifyRun(check.deps, { repo: check.repo, runRef: ref, profileName: name, profile: check.profiles[name]!, vetoHeads: file.head ? [file.head] : [], now: check.now }));
return verdicts.get(key)!;
};
const missingProfile = names.find(name => !check.profiles[name]);
if (missingProfile) return { refused: 'usage', message: `run profile ${missingProfile} is not declared` };
// Every profile gets its own verified run; search the whole assignment so a run valid for two profiles cannot starve one.
const assign = (i: number, used: Set<string>): RunEvidence[] | null => {
if (i === names.length) return [];
for (const ref of refs) {
if (used.has(ref)) continue;
const v = verdict(names[i]!, ref);
if (!v.ok) continue;
const rest = assign(i + 1, new Set([...used, ref]));
if (rest) return [v.evidence, ...rest];
}
return null;
};
const evidence = assign(0, new Set());
if (!evidence) {
const failed = names.map(name => `${name}: ${refs.map(ref => { const v = verdict(name, ref); return v.ok ? `${ref} ok` : `${ref} reason=${v.reason} ${v.detail}`; }).join('; ')}`);
return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=no_verified_run — no run per profile verifies (${failed.join(' | ')}); ${lane} stays uncleared` };
}
const trees = [...new Set(evidence.map(e => e.tree))];
if (trees.length > 1) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=tree_mismatch — the runs tested different trees (${trees.map(t => t.slice(0, 12)).join(', ')}); ${lane} stays uncleared` };
const sub = treeSubstitution(file, trees[0]!, lane, repoRoot);
if ('refused' in sub) return { ...sub, message: `REUSE: refused reason=tree_not_stamp_only — ${sub.message}` };
const idle = evidence.filter(e => (e.tests ?? e.files.length) === 0);
if (idle.length) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=no_tests — ${idle.map(e => `run ${e.run_id} (${e.profile})`).join(', ')} records no tests; ${lane} stays uncleared` };
const required = new Set(file.requires_remote?.find(r => r.lane === lane)?.tests ?? []);
if (isGate) {
const listed = check.inventory(check.gateRuns!.inventory, file.tree);
if ('error' in listed) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=inventory_failed — the remote gate's inventory command failed: ${listed.error}; ${lane} stays uncleared` };
if (!listed.length) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=inventory_empty — the remote gate's inventory command listed no test files; ${lane} stays uncleared` };
for (const f of listed) required.add(f);
}
const missing = uncoveredFiles([...required], evidence);
if (missing.length) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=${isGate ? 'inventory_uncovered' : 'tests_uncovered'} — ${missing.length} of the ${required.size} test file(s) ${lane} must cover appear in no run's tree.json (${missing.slice(0, 5).join(', ')}${missing.length > 5 ? ', …' : ''}); ${lane} stays uncleared` };
for (const e of evidence) {
let changed: string | null;
try { changed = runChanged(check.deps, e.run_id, e.run_attempt); }
catch (err: any) { changed = `reading run ${e.run_id} again failed: ${String(err?.message ?? err).slice(0, 200)}`; }
if (changed) return { refused: 'PREGATE_RECEIPT_INVALID', message: `REUSE: refused reason=run_changed — ${changed}; ${lane} stays uncleared` };
}
const tests = evidence.reduce((n, e) => n + (e.tests ?? e.files.length), 0);
const accepted = sub.substitution ? ` accepted-for=${file.tree.slice(0, 12)} stamp-only=${sub.substitution.files.length}` : '';
return { receipt: `${isGate ? 'remote-gate' : 'run'} tree=${trees[0]!.slice(0, 12)} tests=${tests}${accepted} ${runReceiptFields(evidence, check.now)}`, attested: false, substitution: sub.substitution };
}
/** Record the receipt that clears one obligation (a run URL, an evidence-bundle lane label, or a remote gate report, with the stamp-only substitution when one was accepted). */
export function clearObligation(file: PregateFile, lane: string, receipt: string, by: string, substitution?: PregateReceiptSubstitution): PregateFile {
const list = file.requires_remote ?? [];
lib/result-codes-eval.ts
+33
View File
@@ -0,0 +1,33 @@
/**
* Result codes for the /eval-plan preregistration (bin/gstack-eval-plan prereg
* --check, the priced pilot's arm list). Kept beside lib/result-codes.ts, which
* spreads this table into RESULT_CODES; the same rules apply: one stable
* docs/troubleshooting.md anchor, a summary and a fix clause each. The
* no-gstack arm rule is documented in docs/TESTING_INTERNALS.md#no-gstack-arm.
*/
import type { ResultCode } from './result-codes';
export const EVAL_RESULT_CODES = {
PREREG_INCOMPLETE: {
anchor: 'prereg-incomplete',
summary: 'the preregistration file lacks a required section, a section is empty, or the Comparison or Diagnostics declaration is malformed',
fix: 'fill every section `gstack-eval-plan prereg --check` names (bars, arms, comparison, decoys, stop rules, held-out exposure) before any priced call',
},
// Field list 2 (2026-10-16), W8: preregistered diagnostics run before the freeze commit.
PREREG_DIAGNOSTIC_UNTESTED: {
anchor: 'prereg-diagnostic-untested',
summary: 'the declared summarizer, run on the declared synthetic fixture, did not report every preregistered diagnostic at its expected count',
fix: 'fix the summarizer or the declared definition (expected count, fixture) until every PREREG_DIAGNOSTIC line says ok, then commit the freeze; never freeze a diagnostic the summarizer has not counted',
},
// Garry, 2026-10-31: "all of our head to head tests have to test no gstack from here".
PREREG_NO_GSTACK_ARM: {
anchor: 'prereg-no-gstack-arm',
summary: 'the Comparison says stack: loaded and the No-gstack arm section is missing, lacks a key, or does not give every model its own <model>#no-gstack Arms line',
fix: 'add `## No-gstack arm` with arms:, inputs:, prompt:, panel: and report:, and list each <model>#no-gstack arm under Arms; when no arm loads a skill stack, say stack: none with a why: line',
},
EVAL_ARM_DUPLICATE: {
anchor: 'eval-arm-duplicate',
summary: 'the pilot was given the same arm identity twice, so two arms would share one set of attempts',
fix: 'give each arm its own identity, <model> or <model>#<loadout> (for example claude-opus-5-5#no-gstack beside claude-opus-5-5); nothing was reserved',
},
} as const satisfies Record<string, ResultCode>;
lib/result-codes-release.ts
+30
View File
@@ -153,6 +153,36 @@ export const RELEASE_RESULT_CODES = {
summary: 'the PR body’s /ship receipt is for another head than the one being READY’d or merged (a push after /ship, or a receipt copied from another PR)',
fix: 'rerun /ship on the current head so the receipt names it, then gstack-ship-queue ready again; a slot’s stamp commit keeps the receipt written for its pinned head',
},
QUEUE_RECEIPT_CONDITIONAL: {
anchor: 'queue-receipt-conditional',
summary: 'the PR body’s /ship receipt is conditional: Step 5 failed on cases gstack-base-red proved are the base branch’s own, so /ship paused before READY',
fix: 'wait for the base to pass those cases (gstack-base-red wait --pause <file>), then /ship --resume <record>, which re-runs Step 5 and rewrites the receipt without base_red; nothing READYs, trains or merges on a conditional receipt',
},
BASE_RED_UNCONFIGURED: {
anchor: 'base-red-unconfigured',
summary: 'the base branch’s ship policy declares no base_red block, so gstack-base-red cannot reproduce a Step 5 failure there and every failure stays in-branch',
fix: 'handle the failure with /ship’s usual triage; to let /ship pause on a red base, add base_red { setup, select, timeout_sec } to .gstack/ship-policy.json on the base branch',
},
BASE_RED_UNNAMED: {
anchor: 'base-red-unnamed',
summary: 'the failure evidence names fewer failures than the Step 5 lane reported, so an unnamed failure could be in-branch',
fix: 'list every failing test in the evidence file (id, file, first error line), with total equal to the lane’s failure count, and classify again',
},
BASE_RED_REPRO_ERROR: {
anchor: 'base-red-repro-error',
summary: 'gstack-base-red could not reproduce at the base or the PR head: a commit did not resolve, the scratch checkout failed, or the setup command exited non-zero',
fix: 'read the printed detail; fix the base_red setup command on the base branch or fetch the base, then classify again. Until then every failure is in-branch',
},
BASE_RED_PAUSE_EXPIRED: {
anchor: 'base-red-pause-expired',
summary: 'the base-red pause is older than 24 hours, so its classification is stale',
fix: 'report the paused PR to its Captain (or owner), then start /ship again on the current base; it classifies afresh',
},
BASE_RED_WAIT_TIMEOUT: {
anchor: 'base-red-wait-timeout',
summary: 'no new base head passed the paused cases before gstack-base-red wait’s timeout',
fix: 'run gstack-base-red wait --pause <file> again (the pause stays valid until it expires), or check who owns the base red',
},
QUEUE_RECEIPT_DEPTH_RAISED: {
anchor: 'queue-receipt-depth-raised',
summary: 'the PR body’s /ship receipt records a review depth below the one its head needs (the queue recomputed it from git), or a current receipt records no review_depth at all',
lib/result-codes.ts
+2 -11
View File
@@ -13,6 +13,7 @@
import { RELEASE_RESULT_CODES } from './result-codes-release';
import { CAPY_RESULT_CODES } from './result-codes-capy';
import { BUDGET_RESULT_CODES } from './result-codes-budget';
import { EVAL_RESULT_CODES } from './result-codes-eval';
export interface ResultCode {
/** Stable `<a id>` in docs/troubleshooting.md. */
@@ -435,11 +436,6 @@ export const RESULT_CODES = {
},
// PR E (multi-agent wave): gstack-eval-plan, gstack-lane-check, gstack-evidence verify,
// gstack-owner-brief, gstack-contributor-mode, gstack-banned-terms.
PREREG_INCOMPLETE: {
anchor: 'prereg-incomplete',
summary: 'the preregistration file lacks a required section or a section is empty',
fix: 'fill every section `gstack-eval-plan prereg --check` names (bars, arms, decoys, stop rules, held-out exposure) before any priced call',
},
EVAL_PRICE_MISSING: {
anchor: 'eval-price-missing',
summary: 'an arm names a model with no row in the price table, so its admission cannot be estimated',
@@ -671,12 +667,6 @@ export const RESULT_CODES = {
summary: '`gstack-autoplan export --public <dir>` writes only into an empty directory or over an earlier public copy of the same run',
fix: 'pass an empty or new --public directory, or the directory that already holds this run\'s public copy',
},
// Field list 2 (2026-10-16), W8: preregistered diagnostics run before the freeze commit.
PREREG_DIAGNOSTIC_UNTESTED: {
anchor: 'prereg-diagnostic-untested',
summary: 'the declared summarizer, run on the declared synthetic fixture, did not report every preregistered diagnostic at its expected count',
fix: 'fix the summarizer or the declared definition (expected count, fixture) until every PREREG_DIAGNOSTIC line says ok, then commit the freeze; never freeze a diagnostic the summarizer has not counted',
},
// Field list 2 (2026-10-16), W9: .gstack/eval-models.json.
EVAL_MODELS_INVALID: {
anchor: 'eval-models-invalid',
@@ -772,6 +762,7 @@ export const RESULT_CODES = {
...RELEASE_RESULT_CODES,
...CAPY_RESULT_CODES,
...BUDGET_RESULT_CODES,
...EVAL_RESULT_CODES,
} as const satisfies Record<string, ResultCode>;
export type ResultCodeName = keyof typeof RESULT_CODES;
lib/run-evidence.ts
+261
View File
@@ -0,0 +1,261 @@
/**
* run-evidence — a GitHub Actions run as evidence for one tree (GSTA-156).
*
* A consumer that would otherwise run a suite again (pregate's remote gate
* obligation, a platform lane) may accept a run that already passed on the
* identical tree, but only on the runner's own record, never on prose: the
* GitHub API's record of the run, the tree git computes from the run's head
* commit, and the profile's artifact (its tree.json) the run uploaded itself.
* `verifyRun` refuses unless every check holds:
*
* the run belongs to this repository and ran for a same-repository head;
* its workflow file and event are the profile's; it is the newest attempt,
* completed with success; every job the profile names appears exactly once
* in that attempt and succeeded; its artifact names this run, attempt,
* event and head (the API always answers with the newest attempt, so an
* older green attempt under a newer red one is refused), is not a reuse
* record, records the tree git computes for the head, and holds well-formed
* files (a list of paths) and tests (a non-negative integer); no run of the
* same workflow on that head, or on any other head the consumer names, that
* started at or after this run was created failed, was cancelled or is still
* running (a red re-run of an older run counts, and re-running this run does
* not hide a failure in between); it finished within the profile's
* max_age_hours (default 24); its runner OS matches when the profile names
* one; and a second read of the run still shows the same successful attempt
* (run_changed otherwise).
*
* Profiles are the base branch's `.gstack/pregate.json` `run_profiles`, so a
* pull request can tighten them but never relax them. The workflow a run
* executed and the tree.json it uploaded are the run head's own, so a diff to
* that workflow is reviewed like any other gate change. Every refusal names a
* reason code; an API or git error refuses (fail closed). Verification runs
* once, online, at `gstack-pregate clear`; the receipt it writes carries the
* evidence, and `gstack-pregate verify` checks that receipt offline.
*/
import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { sha256Hex, writeReceipt } from './egress-receipt';
import { runBashSync } from './script-invocation';
export interface RunProfile { workflow: string; events: string[]; jobs: string[]; artifact: string; max_age_hours?: number; os?: string }
export interface RemoteGateRuns { profiles: string[]; inventory: string }
export interface ApiRun {
id: number; run_attempt: number; status: string; conclusion: string | null; event: string; head_sha: string;
path?: string; created_at?: string; run_started_at?: string; updated_at?: string;
repository?: { full_name: string } | null; head_repository?: { full_name: string } | null;
}
export interface ApiJob { name: string; conclusion: string | null; completed_at?: string | null }
export interface TreeRecord { run_id?: unknown; run_attempt?: unknown; event?: unknown; sha?: unknown; head_sha?: unknown; tree?: unknown; reused?: unknown; reuse?: unknown; os?: unknown; files?: unknown; tests?: unknown }
/** Everything verifyRun reads, injectable so every refusal has a fixture. */
export interface RunDeps {
run(id: number): ApiRun;
runsForHead(workflow: string, sha: string): ApiRun[];
jobs(id: number, attempt: number): ApiJob[];
artifact(id: number, name: string): TreeRecord;
commitTree(sha: string): string | null;
}
export type RunRefusal = 'run_ref_invalid' | 'other_repository' | 'fork_run' | 'workflow_mismatch' | 'event_mismatch' | 'run_in_progress'
| 'run_not_success' | 'job_missing' | 'job_ambiguous' | 'job_not_success' | 'artifact_missing' | 'artifact_mismatch'
| 'artifact_reused' | 'tree_mismatch' | 'newer_run_vetoes' | 'run_changed' | 'run_stale' | 'os_mismatch' | 'api_error';
export interface RunEvidence { run_id: number; run_attempt: number; head_sha: string; tree: string; profile: string; event: string; finished_at: string; files: string[]; tests: number | null }
export type RunVerdict = { ok: true; evidence: RunEvidence } | { ok: false; reason: RunRefusal; detail: string };
const SHA = /^[0-9a-f]{40}$/;
const PROFILE_KEYS = ['workflow', 'events', 'jobs', 'artifact', 'max_age_hours', 'os'];
const WORKFLOW_FILE = /^[A-Za-z0-9_.-]+\.ya?ml$/;
const NAME = /^[A-Za-z0-9_.\/-]{1,128}$/;
export const DEFAULT_MAX_AGE_HOURS = 24;
const MAX_ARTIFACT_BYTES = 8 * 1024 * 1024;
/** A numeric run id, or a run URL on this repository (`…/actions/runs/<id>[/attempts/<n>][/job/<j>]`). */
export function parseRunRef(ref: string, repo: string): number | null {
if (/^\d{1,15}$/.test(ref)) return Number(ref);
const m = /^https:\/\/github\.com\/([\w.-]+\/[\w.-]+)\/actions\/runs\/(\d{1,15})(?:\/(?:attempts\/\d+|job\/\d+))?\/?$/.exec(ref);
return m && m[1]!.toLowerCase() === repo.toLowerCase() ? Number(m[2]) : null;
}
/** `<key>: <message>` errors for `run_profiles` and `remote_gate_runs`; empty when valid. Shared with lib/pregate/config.ts. */
export function runProfileErrors(profiles: unknown, gateRuns: unknown): Array<{ path: string; message: string }> {
const errors: Array<{ path: string; message: string }> = [];
const names = new Set<string>();
if (profiles !== undefined) {
if (!profiles || typeof profiles !== 'object' || Array.isArray(profiles)) errors.push({ path: 'run_profiles', message: 'must be an object of name → { workflow, events, jobs, artifact, max_age_hours?, os? }' });
else for (const [name, raw] of Object.entries(profiles as Record<string, unknown>)) {
const at = `run_profiles[${JSON.stringify(name)}]`;
names.add(name);
if (!NAME.test(name)) errors.push({ path: at, message: 'name must be 1-128 letters, digits, . _ / -' });
const p = raw as Record<string, unknown>;
if (!p || typeof p !== 'object' || Array.isArray(p)) { errors.push({ path: at, message: 'must be an object' }); continue; }
for (const k of Object.keys(p)) if (!PROFILE_KEYS.includes(k)) errors.push({ path: `${at}.${k}`, message: `unknown key; valid keys: ${PROFILE_KEYS.join(', ')}` });
if (typeof p.workflow !== 'string' || !WORKFLOW_FILE.test(p.workflow)) errors.push({ path: `${at}.workflow`, message: 'must be a workflow file name like test.yml' });
for (const k of ['events', 'jobs'] as const) {
const v = p[k];
if (!Array.isArray(v) || v.length === 0 || v.length > 32 || v.some(s => typeof s !== 'string' || !s.trim())) errors.push({ path: `${at}.${k}`, message: 'must be a non-empty array of up to 32 names' });
}
if (Array.isArray(p.events) && p.events.some(e => e !== 'push' && e !== 'workflow_dispatch')) errors.push({ path: `${at}.events`, message: 'events are push or workflow_dispatch (a pull_request run tests the merge commit, not the head tree)' });
if (typeof p.artifact !== 'string' || !NAME.test(p.artifact)) errors.push({ path: `${at}.artifact`, message: 'must be the artifact name holding tree.json' });
if ('max_age_hours' in p && (!Number.isInteger(p.max_age_hours) || (p.max_age_hours as number) < 1 || (p.max_age_hours as number) > 168)) errors.push({ path: `${at}.max_age_hours`, message: 'must be an integer from 1 to 168' });
if ('os' in p && (typeof p.os !== 'string' || !/^(Linux|macOS|Windows)$/.test(p.os))) errors.push({ path: `${at}.os`, message: 'must be Linux, macOS or Windows' });
}
}
if (gateRuns !== undefined && gateRuns !== null) {
const g = gateRuns as Record<string, unknown>;
if (!g || typeof g !== 'object' || Array.isArray(g)) errors.push({ path: 'remote_gate_runs', message: 'must be { profiles: [names], inventory: command }' });
else {
for (const k of Object.keys(g)) if (k !== 'profiles' && k !== 'inventory') errors.push({ path: `remote_gate_runs.${k}`, message: 'unknown key; valid keys: profiles, inventory' });
if (!Array.isArray(g.profiles) || g.profiles.length === 0 || g.profiles.some(p => typeof p !== 'string')) errors.push({ path: 'remote_gate_runs.profiles', message: 'must be a non-empty array of run_profiles names' });
else {
for (const p of g.profiles as string[]) if (!names.has(p)) errors.push({ path: 'remote_gate_runs.profiles', message: `${JSON.stringify(p)} is not a run_profiles name` });
if (new Set(g.profiles).size !== g.profiles.length) errors.push({ path: 'remote_gate_runs.profiles', message: 'names each profile once' });
}
if (typeof g.inventory !== 'string' || !g.inventory.trim() || g.inventory.length > 512 || /[\0\r\n]/.test(g.inventory)) errors.push({ path: 'remote_gate_runs.inventory', message: 'must be one command line under 512 characters that prints the remote gate\'s test files, one per line' });
}
}
return errors;
}
const fail = (reason: RunRefusal, detail: string): RunVerdict => ({ ok: false, reason, detail });
/** When a run's newest attempt started. A run vetoes the candidate when it started at or after the second the candidate was created: a red re-run of an older run counts, and re-running the candidate does not hide a failure in between. */
const started = (r: ApiRun): string => r.run_started_at ?? r.created_at ?? '';
const normalPath = (f: string): string => f.replace(/^(\.\/)+/, '');
/**
* The verdict for run `runRef` under `profile`. `vetoHeads` are further heads
* whose newer runs of the same workflow can veto (the consumer's own head).
*/
export function verifyRun(deps: RunDeps, o: { repo: string; runRef: string; profileName: string; profile: RunProfile; vetoHeads?: string[]; now: Date }): RunVerdict {
const id = parseRunRef(o.runRef, o.repo);
if (id === null) return fail('run_ref_invalid', `${JSON.stringify(o.runRef)} is not a run id or a run URL on ${o.repo}`);
const p = o.profile;
try {
const run = deps.run(id);
const at = `run ${id} attempt ${run.run_attempt}`;
if ((run.repository?.full_name ?? '').toLowerCase() !== o.repo.toLowerCase()) return fail('other_repository', `${at} belongs to ${run.repository?.full_name ?? 'an unknown repository'}, not ${o.repo}`);
if ((run.head_repository?.full_name ?? '').toLowerCase() !== o.repo.toLowerCase()) return fail('fork_run', `${at} ran for ${run.head_repository?.full_name ?? 'an unknown repository'}, not ${o.repo}`);
if (run.path !== `.github/workflows/${p.workflow}`) return fail('workflow_mismatch', `${at} ran ${run.path ?? 'an unknown workflow'}, not .github/workflows/${p.workflow}`);
if (!p.events.includes(run.event)) return fail('event_mismatch', `${at} is a ${run.event} run; ${o.profileName} accepts ${p.events.join(', ')}`);
if (run.status !== 'completed') return fail('run_in_progress', `${at} is ${run.status}`);
if (run.conclusion !== 'success') return fail('run_not_success', `${at} concluded ${run.conclusion}`);
if (!SHA.test(run.head_sha)) return fail('api_error', `${at} has no head commit`);
const jobs = deps.jobs(id, run.run_attempt);
const completed = jobs.map(j => j.completed_at).filter((t): t is string => !!t).sort().pop();
const finishedAt = completed ?? run.updated_at;
const finished = Date.parse(finishedAt ?? '');
const maxAge = p.max_age_hours ?? DEFAULT_MAX_AGE_HOURS;
if (!Number.isFinite(finished)) return fail('api_error', `${at} has no completion time`);
if (o.now.getTime() - finished > maxAge * 3600_000) return fail('run_stale', `${at} finished ${finishedAt}, more than ${maxAge} h before now`);
for (const name of p.jobs) {
const hits = jobs.filter(j => j.name === name);
if (hits.length === 0) return fail('job_missing', `${at} has no job named ${name}`);
if (hits.length > 1) return fail('job_ambiguous', `${at} has ${hits.length} jobs named ${name}`);
if (hits[0]!.conclusion !== 'success') return fail('job_not_success', `${at}'s ${name} job concluded ${hits[0]!.conclusion}`);
}
const heads = [...new Set([run.head_sha, ...(o.vetoHeads ?? []).filter(h => SHA.test(h))])];
for (const head of heads) {
const newer = deps.runsForHead(p.workflow, head).filter(r => !(r.id === id && r.run_attempt === run.run_attempt) && started(r) >= (run.created_at ?? '') && !(r.status === 'completed' && r.conclusion === 'success'));
if (newer.length) return fail('newer_run_vetoes', `newer run ${newer[0]!.id} of ${p.workflow} on ${head.slice(0, 12)} is ${newer[0]!.status === 'completed' ? newer[0]!.conclusion : newer[0]!.status}`);
}
let rec: TreeRecord;
try { rec = deps.artifact(id, p.artifact); }
catch (e: any) { return fail('artifact_missing', `${at} has no readable ${p.artifact} tree.json: ${String(e?.message ?? e).slice(0, 200)}`); }
if (!rec || typeof rec !== 'object') return fail('artifact_mismatch', `${at}'s ${p.artifact} tree.json is not a JSON object`);
if (rec.reused === true || rec.reuse === true) return fail('artifact_reused', `${at} reused another run; a reuse is never evidence`);
const recHeads = rec.head_sha === undefined && rec.sha === undefined ? [['head', undefined, run.head_sha] as const] : ([['sha', rec.sha, run.head_sha], ['head_sha', rec.head_sha, run.head_sha]] as const).filter(([, got]) => got !== undefined);
const wrong = ([['run_id', rec.run_id, id], ['run_attempt', rec.run_attempt, run.run_attempt], ['event', rec.event, run.event], ...recHeads] as const).filter(([, got, want]) => got !== want);
if (wrong.length) return fail('artifact_mismatch', `${at}'s ${p.artifact} records ${wrong.map(([k, got, want]) => `${k} ${String(got)} (expected ${want})`).join(', ')}`);
const headTree = deps.commitTree(run.head_sha);
if (!headTree) return fail('api_error', `git cannot read the tree of ${run.head_sha}`);
if (rec.tree !== headTree) return fail('tree_mismatch', `${at} recorded tree ${String(rec.tree)}, but its head ${run.head_sha.slice(0, 12)} has tree ${headTree}`);
if (p.os && rec.os !== p.os) return fail('os_mismatch', `${at} ran on ${String(rec.os ?? 'an unrecorded OS')}; ${o.profileName} requires ${p.os}`);
if (rec.files !== undefined && (!Array.isArray(rec.files) || rec.files.some(f => typeof f !== 'string'))) return fail('artifact_mismatch', `${at}'s ${p.artifact} files is not a list of paths`);
if (rec.tests !== undefined && (!Number.isInteger(rec.tests) || (rec.tests as number) < 0)) return fail('artifact_mismatch', `${at}'s ${p.artifact} tests is not a non-negative integer`);
const files = ((rec.files ?? []) as string[]).map(normalPath);
const tests = rec.tests === undefined ? null : (rec.tests as number);
const changed = runChanged(deps, id, run.run_attempt);
if (changed) return fail('run_changed', changed);
return { ok: true, evidence: { run_id: id, run_attempt: run.run_attempt, head_sha: run.head_sha, tree: headTree, profile: o.profileName, event: run.event, finished_at: finishedAt!, files, tests } };
} catch (e: any) {
return fail('api_error', `reading run ${id} failed: ${String(e?.message ?? e).slice(0, 300)}`);
}
}
/** Why run `id` no longer stands as attempt `attempt`, a success; null while it does. Read again after the evidence, and for every run once a whole clear is assembled. */
export function runChanged(deps: RunDeps, id: number, attempt: number): string | null {
const again = deps.run(id);
return again.run_attempt === attempt && again.status === 'completed' && again.conclusion === 'success' ? null : `run ${id} moved to attempt ${again.run_attempt} (${again.status === 'completed' ? again.conclusion : again.status}) while it was being verified`;
}
/** The remote gate's inventory: `command`'s non-empty output lines under pipefail, so a pipeline whose producer fails refuses instead of listing part of the gate. */
export function runInventory(command: string, cwd: string): string[] | { error: string } {
const r = runBashSync(`set -o pipefail\n${command}`, { cwd, encoding: 'utf8', timeout: 300_000, maxBuffer: 64 * 1024 * 1024 });
if (r.status !== 0 || r.error) return { error: `${command} exited ${r.status ?? r.error}: ${(r.stderr ?? '').trim().slice(0, 200)}` };
return [...new Set(r.stdout.split('\n').map(l => l.trim()).filter(Boolean))];
}
/** The files in `inventory` that no verified run's tree.json lists. */
export function uncoveredFiles(inventory: string[], evidence: RunEvidence[]): string[] {
const ran = new Set(evidence.flatMap(e => e.files.map(normalPath)));
return inventory.filter(f => !ran.has(normalPath(f)));
}
/** The receipt suffix a run clear records: runs, profiles and the verification time, read back by `parseRunReceipt`. */
export function runReceiptFields(evidence: RunEvidence[], now: Date): string {
return `runs=${evidence.map(e => `${e.run_id}/${e.run_attempt}`).join(',')} profiles=${evidence.map(e => e.profile).join(',')} heads=${[...new Set(evidence.map(e => e.head_sha.slice(0, 12)))].join(',')} verified=${now.toISOString()}`;
}
export function parseRunReceipt(receipt: string): { runs: string[]; profiles: string[]; tree: string | null } | null {
const runs = / runs=(\S+)/.exec(receipt)?.[1];
const profiles = / profiles=(\S+)/.exec(receipt)?.[1];
if (!runs || !profiles || !/ verified=\S+/.test(receipt)) return null;
return { runs: runs.split(','), profiles: profiles.split(','), tree: /\btree=([0-9a-f]{12})\b/.exec(receipt)?.[1] ?? null };
}
/** Egress receipts for the reads below (fail-open, like contributor-mode): only this repository's name, run ids and a commit sha leave the machine. */
const receiptFor = (host: string, payloadClass: string, request: string) => ({ sink: 'run-evidence', host, payloadClass, bytes: 0, sha256: sha256Hex(request), consent: 'user ran gstack-pregate clear --run (or --run-url on a profiled lane); reads its own repository\'s Actions runs and fetches a run head by sha' });
const receiptFailed = (err: unknown) => process.stderr.write(`[run-evidence] egress receipt could not be written (${(err as Error).message}); proceeding (fail-open)\n`);
function gh(args: string[], timeoutMs = 60_000): string {
try { writeReceipt(receiptFor('api.github.com', 'actions-run-read', args.join(' '))); } catch (err) { receiptFailed(err); }
const r = spawnSync('gh', args, { encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
if (r.status !== 0 || r.error) throw new Error(`gh ${args.slice(0, 2).join(' ')} failed: ${(r.stderr || String(r.error ?? '')).trim().slice(0, 300)}`);
return r.stdout;
}
/** The gh- and git-backed dependencies for `repo`, reading trees in `repoRoot` (fetching a missing head from origin once). */
export function ghRunDeps(repo: string, repoRoot: string): RunDeps {
const json = <T>(p: string): T => JSON.parse(gh(['api', p]));
// Every page, one JSON value per line: a veto or a required job past the first 100 entries must still be seen.
const all = <T>(p: string, key: string): T[] => gh(['api', '--paginate', p, '--jq', `.${key}[]`]).split('\n').filter(Boolean).map(l => JSON.parse(l) as T);
return {
run: id => json<ApiRun>(`repos/${repo}/actions/runs/${id}`),
runsForHead: (workflow, sha) => all<ApiRun>(`repos/${repo}/actions/workflows/${workflow}/runs?head_sha=${sha}&per_page=100`, 'workflow_runs'),
jobs: (id, attempt) => all<ApiJob>(`repos/${repo}/actions/runs/${id}/attempts/${attempt}/jobs?per_page=100`, 'jobs'),
artifact: (id, name) => {
const listed = all<{ name: string; size_in_bytes: number; expired: boolean }>(`repos/${repo}/actions/runs/${id}/artifacts?name=${encodeURIComponent(name)}&per_page=100`, 'artifacts').filter(a => a.name === name && !a.expired);
if (listed.length !== 1) throw new Error(`run ${id} has ${listed.length} unexpired artifacts named ${name}`);
if (!(listed[0]!.size_in_bytes <= MAX_ARTIFACT_BYTES)) throw new Error(`artifact ${name} is ${listed[0]!.size_in_bytes} bytes, over ${MAX_ARTIFACT_BYTES}`);
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-run-evidence-'));
try {
gh(['run', 'download', String(id), '--repo', repo, '--name', name, '--dir', dir], 120_000);
const file = path.join(dir, 'tree.json');
if (fs.statSync(file).size > MAX_ARTIFACT_BYTES) throw new Error(`tree.json is over ${MAX_ARTIFACT_BYTES} bytes`);
return JSON.parse(fs.readFileSync(file, 'utf8')) as TreeRecord;
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
},
commitTree: sha => {
const read = () => spawnSync('git', ['rev-parse', '--verify', '--quiet', `${sha}^{tree}`], { cwd: repoRoot, encoding: 'utf8', timeout: 30_000 });
let r = read();
if (r.status !== 0) {
try { writeReceipt(receiptFor('origin', 'git-fetch-by-sha', sha)); } catch (err) { receiptFailed(err); }
spawnSync('git', ['fetch', '--quiet', '--no-tags', 'origin', sha], { cwd: repoRoot, timeout: 120_000 });
r = read();
}
const tree = (r.stdout ?? '').trim();
return r.status === 0 && /^[0-9a-f]{40,64}$/.test(tree) ? tree : null;
},
};
}
lib/ship-policy.ts
+25
View File
@@ -75,8 +75,13 @@ export interface ShipPolicy {
train: TrainPolicy | null;
/** Documentation caps /ship Step 14.5 checks before the verification gate (`gstack-ship-receipt docs-caps`); null declares none. */
docs: { caps: Array<{ path: string; max_bytes: number }> } | null;
/** How /ship reproduces a Step 5 failure on the base branch before calling it base-red (docs/ship-policy.md#base-red); null never calls one base-red. */
base_red: BaseRedPolicy | null;
}
/** `base_red`: `setup` installs dependencies in a scratch checkout (null when none are needed); `select` runs the failing test files, named by its `{files}` slot. */
export interface BaseRedPolicy { setup: string | null; select: string; timeout_sec: number }
/** The `train` block: the gate workflow trains dispatch and how its verdict is read. */
export interface TrainPolicy {
/** Workflow file dispatched for the full gate and the bisect (it accepts the inputs in TRAIN_DISPATCH_INPUTS). */
@@ -121,6 +126,7 @@ export const DEFAULT_POLICY: ShipPolicy = {
min_gstack_version: null,
train: null,
docs: null,
base_red: null,
};
export type PolicyKey = Exclude<keyof ShipPolicy, '$schema'>;
@@ -150,6 +156,7 @@ export const POLICY_EXPLANATIONS: Record<PolicyKey, string> = {
full_e2e: 'true: every merge needs a passing full-gate result (gstack-ship-queue gate-ahead --record, or the lane\'s --gate-exit 0 --gate-log on the gated head) bound to a tree equal to the merging one modulo stamps; ready, stamp and verify refuse without one (QUEUE_GATE_MISSING).',
train: '{ gate_workflow, free_workflow, full_gate_check, record_artifact, max_size, fill_hours, screen_selection, trial_reservation_usd } or null: turns on merge trains in gstack-ship-queue (train build|status|verdict|bisect|rebuild|prune, ready --screen|--no-screen, verify --train). null keeps today\'s per-PR queue unchanged.',
docs: '{ caps: [{ path, max_bytes }] } or null: documentation files with a byte cap. /ship Step 14.5 runs `gstack-ship-receipt docs-caps` before the verification gate and prints one DOCS_CAP: line per cap; a file over its cap (or absent) blocks the ship until the doc is trimmed or the cap raised on the base branch. null declares none.',
base_red: '{ setup, select, timeout_sec } or null: how /ship Step 5 proves a failure is the base branch\'s own (docs/ship-policy.md#base-red). gstack-base-red checks out the base head and the PR head in scratch worktrees, runs setup (dependency install, or null), then select with the failing test files in its {files} slot; only the same failures in both, in files the PR does not touch, are base-red, and /ship then pauses before READY instead of failing. Read from the base branch only. null keeps every failure in-branch.',
min_gstack_version: 'The oldest gstack that may read this file. Any key newer than 1.91.98.0 set to a non-default value requires it at that key’s introducing version (validate prints each); a gstack older than it refuses the file (POLICY_GSTACK_TOO_OLD).',
};
@@ -183,6 +190,7 @@ export const DOCS_CAPS_KEY_VERSION = '1.91.270.0';
export const CHANGELOG_FRAGMENTS_KEY_VERSION = '1.91.336.0';
/** `bump: never` is a value of an older key, so the per-key floor check misses it; compatErrors checks it by value. */
export const BUMP_NEVER_KEY_VERSION = '1.91.353.0';
export const BASE_RED_KEY_VERSION = '1.91.374.0';
export const POLICY_KEY_INTRODUCED: Record<PolicyKey, string> = {
bump: '1.91.73.0', queue_mode: '1.91.73.0', release_tool: '1.91.73.0', release_outputs: '1.91.73.0', stamp_paths: '1.91.73.0',
changelog: '1.91.73.0', mirror: '1.91.73.0', history: '1.91.73.0', preregistration_shas: '1.91.73.0', stamp_sensitive_gates: '1.91.73.0',
@@ -190,6 +198,7 @@ export const POLICY_KEY_INTRODUCED: Record<PolicyKey, string> = {
migrations: '1.91.98.0', gate_command: SHIP_QUEUE_KEYS_VERSION, full_e2e: SHIP_QUEUE_KEYS_VERSION, min_gstack_version: SHIP_QUEUE_KEYS_VERSION,
train: TRAIN_KEYS_VERSION, review_scope: REVIEW_SCOPE_KEY_VERSION, docs: DOCS_CAPS_KEY_VERSION,
changelog_fragments: CHANGELOG_FRAGMENTS_KEY_VERSION,
base_red: BASE_RED_KEY_VERSION,
};
const newer = (a: string, b: string): boolean => cmpVersion(parseVersion(a)!, parseVersion(b)!) > 0;
@@ -373,6 +382,7 @@ export function validatePolicy(raw: unknown, opts: { gstackVersion?: string } =
if ('train' in obj && obj.train !== null) trainErrors(obj.train, errors);
if ('docs' in obj && obj.docs !== null) docsErrors(obj.docs, errors);
if ('changelog_fragments' in obj && obj.changelog_fragments !== null) fragmentsErrors(obj.changelog_fragments, errors);
if ('base_red' in obj && obj.base_red !== null) baseRedErrors(obj.base_red, errors);
if ('stamp_paths' in obj) listErrors('stamp_paths', obj.stamp_paths, errors, globError);
for (const key of ['changelog', 'mirror'] as const) {
if (!(key in obj) || obj[key] === null) continue;
@@ -457,6 +467,21 @@ function docsErrors(raw: unknown, errors: PolicyError[]): void {
});
}
function baseRedErrors(raw: unknown, errors: PolicyError[]): void {
const b = raw as Record<string, unknown> | null;
if (!b || typeof b !== 'object' || Array.isArray(b)) { errors.push({ code: 'POLICY_INVALID', path: 'base_red', message: 'must be { setup, select, timeout_sec } or null' }); return; }
for (const k of Object.keys(b)) if (!['setup', 'select', 'timeout_sec'].includes(k)) errors.push({ code: 'POLICY_INVALID', path: `base_red.${k}`, message: 'unknown key' });
const command = (key: string, value: unknown, nullable: boolean) => {
if (nullable && value === null) return;
if (typeof value !== 'string' || !value.trim()) errors.push({ code: 'POLICY_INVALID', path: `base_red.${key}`, message: `must be a non-empty string${nullable ? ' or null' : ''}` });
else if (value.length > MAX_COMMAND || /[\0\r\n]/.test(value)) errors.push({ code: 'POLICY_CONTAINMENT', path: `base_red.${key}`, message: `must be one line under ${MAX_COMMAND} characters` });
};
command('setup', b.setup, true);
command('select', b.select, false);
if (typeof b.select === 'string' && !b.select.includes('{files}')) errors.push({ code: 'POLICY_INVALID', path: 'base_red.select', message: 'needs a {files} slot for the failing test files' });
if (!Number.isInteger(b.timeout_sec) || (b.timeout_sec as number) < 10 || (b.timeout_sec as number) > 7200) errors.push({ code: 'POLICY_INVALID', path: 'base_red.timeout_sec', message: 'must be an integer from 10 to 7200' });
}
function fragmentsErrors(raw: unknown, errors: PolicyError[]): void {
const d = raw as Record<string, unknown> | null;
if (!d || typeof d !== 'object' || Array.isArray(d)) { errors.push({ code: 'POLICY_INVALID', path: 'changelog_fragments', message: 'must be { dir, required_checks? } or null' }); return; }
lib/ship-receipt.ts
+9 -3
View File
@@ -19,7 +19,7 @@ const KEY_ORDER: Array<keyof ShipReceipt | 'docs'> = [
'schema_version', 'pr', 'head', 'version', 'base', 'gated_tree', 'predecessor', 'tree', 'gate_reuse', 'gate', 'ci', 'spend_usd',
'session_kind', 'artifacts_consumed', 'run', 'thread', 'policy', 'queue_mode', 'preregistration', 'history',
'head_tree', 'product_patch_id', 'gstack_version', 'review_depth', 'pregate', 'requires_remote', 'issues', 'approval', 'docs', 'review_loop',
'review_loop_exception', 'review_targeted', 'carry',
'review_loop_exception', 'review_targeted', 'carry', 'base_red',
];
/** Keys in the fixed order so identifiers lead; undefined values are dropped. */
@@ -54,7 +54,7 @@ export function extractReceipt(body: string): { found: boolean; receipt?: ShipRe
}
}
export type ReceiptBindingCode = 'QUEUE_RECEIPT_MISSING' | 'QUEUE_RECEIPT_STALE';
export type ReceiptBindingCode = 'QUEUE_RECEIPT_MISSING' | 'QUEUE_RECEIPT_STALE' | 'QUEUE_RECEIPT_CONDITIONAL';
export type ReceiptBinding =
| { ok: true; receipt: ShipReceipt; head: string; line: string }
| { ok: false; code: ReceiptBindingCode; message: string; line: string };
@@ -67,8 +67,12 @@ export type ReceiptBinding =
* head is the pin's stamp commit (the stamp changes only stamp files, so the
* receipt written for the pin stays the binding one). `line` is the
* informational `SHIP_RECEIPT:` line the READY block prints either way.
* A receipt with a `base_red` block (lib/base-red.ts) is conditional: its
* Step 5 did not pass, so it binds only for a caller that asks for it
* (`/ship`'s terminal line, which reports the pause) and is refused
* everywhere else.
*/
export function bindReceipt(body: string, pr: string | undefined, accepted: readonly string[]): ReceiptBinding {
export function bindReceipt(body: string, pr: string | undefined, accepted: readonly string[], opts: { conditional?: boolean } = {}): ReceiptBinding {
const subject = pr === undefined ? "the receipt's" : `PR #${pr}'s`;
const ex = extractReceipt(body);
if (!ex.found) return { ok: false, code: 'QUEUE_RECEIPT_MISSING', message: `${subject} body carries no ship receipt`, line: 'SHIP_RECEIPT: none in the PR body' };
@@ -84,6 +88,7 @@ export function bindReceipt(body: string, pr: string | undefined, accepted: read
return { ok: false, code: 'QUEUE_RECEIPT_STALE', message: `${subject} ship receipt is for ${oneLine(r.head.slice(0, 12))}, not the head ${want} being merged`, line };
}
if (pr !== undefined && r.pr !== undefined && String(r.pr) !== pr) return { ok: false, code: 'QUEUE_RECEIPT_STALE', message: `${subject} ship receipt names pr=${r.pr}, not this PR`, line };
if (r.base_red && !opts.conditional) return { ok: false, code: 'QUEUE_RECEIPT_CONDITIONAL', message: `${subject} ship receipt is paused on a red base (${oneLine(r.base_red.cases.join(', ')).slice(0, 160)} fail at ${oneLine(r.base_red.base.slice(0, 12))}); /ship --resume rewrites it once the base passes them`, line };
return { ok: true, receipt: r, head, line };
}
@@ -126,6 +131,7 @@ export function receiptSummaryLine(r: ShipReceipt): string {
if (loop) parts.push(`passes=${loop.passes} blocking_found=${loop.blocking_found} deferred=${loop.deferred}`
+ (loop.fix_checks ? ` fix_checks=${loop.fix_checks}` : '') + (loop.dispatches ? ` dispatches=${Object.values(loop.dispatches).reduce((a, c) => a + c, 0)}` : ''));
if (r.carry && typeof r.carry.from === 'string') parts.push(`carried_from=${r.carry.from.slice(0, 12)}`);
if (r.base_red && typeof r.base_red.base === 'string') parts.push(`base_red=${r.base_red.base.slice(0, 12)}`);
return `SHIP_RECEIPT: ${oneLine(parts.join(' '))}`;
}
package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gstack",
"version": "1.91.371",
"version": "1.91.378",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
scripts/pr-dependencies.ts
+1 -1
View File
@@ -89,7 +89,7 @@ export const SELECTION_DATA_MODULES: readonly string[] = [
// The build entry list: every compiled binary, which skills reach through their `--outfile` path instead.
'scripts/build.sh',
// The reason-code table: its `fix:` prose names the command a person reruns, not a file the module consumes.
'lib/result-codes.ts', 'lib/result-codes-release.ts', 'lib/result-codes-capy.ts', 'lib/result-codes-budget.ts',
'lib/result-codes.ts', 'lib/result-codes-release.ts', 'lib/result-codes-capy.ts', 'lib/result-codes-budget.ts', 'lib/result-codes-eval.ts',
// The exercised-path map: observed commands per case, read only by exercisedConsumers() (shadow data; a change selects no paid case).
'test/helpers/exercised-commands.json',
];
scripts/ubicloud/pregate-remote.sh
+36 -5
View File
@@ -27,10 +27,18 @@
# teardown) times the size's price. Without a key it runs the same files
# locally in parallel shards and reports spent=0.
#
# Reattach: the tree and the planned set's digest go into ubi-runner's run key
# (--key-input), so a rerun after the local runner died (a machine stop, a
# SIGKILL) reattaches to the VM still running this gate instead of paying for a
# second one. Its minutes since created_at count against the cap: the wall
# budget shrinks by them (PREGATE_REMOTE_CAP_SPENT when none is left), and
# spent= bills from the original create.
#
# Exit codes: 0 pass (or a paid-only selection, tests=0); 1 test failures (or
# a rejected file list); 2 missing input; 3 tree or plan mismatch (or the tree
# moved during the run); 4 infrastructure failure (create, setup, SSH, stop
# deadline); 5 the spend cap or pregate's deadline refuses the run. Every refusal line starts with a
# deadline, another gate on this machine owning the VM to reattach to); 5 the
# spend cap or pregate's deadline refuses the run. Every refusal line starts with a
# PREGATE_REMOTE_* reason code (docs/pregate.md#remote-gate). Nothing is
# uploaded or spent before the local tree check.
set -euo pipefail
@@ -147,8 +155,6 @@ if [ -n "$outer_s" ]; then
fi
if [ "$outer_stop" -lt "$stop_after" ]; then stop_after=$outer_stop stop_why="$outer_from ${outer_s}s minus 120 s teardown reserve and 15 s margin"; fi
fi
say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)"
digest=$(tr -d '\r' <"$files" | grep . | LC_ALL=C sort | sha256sum | cut -d' ' -f1)
prelude="$logs/vm-setup.sh"
{
@@ -183,6 +189,28 @@ args=(run -s "$size" --src "$root" --setup "$prelude"
--pull "/tmp/gstack-free-test-*:$logs"
--pull "work/$(basename "$root")/.context/free-test-logs:$logs")
[ -z "${GSTACK_FREE_JOBS:-}" ] || args+=(--env "GSTACK_FREE_JOBS=$GSTACK_FREE_JOBS")
args+=(--key-input "pregate-remote tree=$tree plan=$digest")
# A rerun of this gate for the same tree and planned set has the same
# ubi-runner run key, so it reattaches to the VM a killed run left behind
# (`ubi-runner: reattached to <vm> created_at=<epoch>`). The minutes that VM
# has billed since created_at come off the cap's wall budget first.
prior_vm=""
attach=$(bash "$here/ubi-runner.sh" "${args[@]}" --probe -- "bash ~/.pregate-run.sh" 2>/dev/null \
| sed -n 's/^UBI_ATTACH_INFO: vm=\([^ ]*\) created_at=\([0-9]*\) key=[0-9a-f]* busy=\([01]\)$/\1 \2 \3/p' | tail -1) || attach=""
if [ -n "$attach" ]; then
read -r prior_vm prior_created prior_busy <<<"$attach"
[ "$prior_busy" = 0 ] || stop 4 PREGATE_REMOTE_ATTACH_BUSY "another gate on this machine still runs VM $prior_vm for this tree and planned set" "wait for it; this run created nothing and spent nothing"
prior_min=$(( ($(date +%s) - prior_created + 59) / 60 ))
left_s=$(( (wall_min - 3 - prior_min) * 60 ))
if [ "$left_s" -le 0 ]; then
stop 5 PREGATE_REMOTE_CAP_SPENT "VM $prior_vm, created ${prior_min} min ago for this tree and planned set, has already used the cap's $wall_min min wall (\$$(usd "$prior_min") of \$$cap)" \
"nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down $prior_vm" "or rerun with a higher --spend-cap to let it finish: $rerun"
fi
if [ "$left_s" -lt "$stop_after" ]; then stop_after=$left_s stop_why="the cap's $wall_min min wall minus 3 min for teardown and $prior_min min already billed since $prior_vm was created"; fi
say "reattaching to VM $prior_vm, created ${prior_min} min ago by an earlier run of this gate for the same tree and planned set"
fi
say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)"
# The VM run gets its own session (python3, which ubi-runner already needs, for
# setsid on Linux and macOS): pregate's --timeout kill signals this
@@ -220,7 +248,7 @@ while :; do
rm -f "$log.pid" "$log.timedout" "$log.left"
t0=$(date +%s)
vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$(( (stop_after + 900 + 3599) / 3600 ))"
say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
[ -n "$prior_vm" ] || say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
UBI_LEFT_FILE="$log.left" python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
bash -c "$vm_session" pregate-remote-vm "$stop_after" "$log" "$stream" "$here/ubi-runner.sh" "${args[@]}" -n "$vm_name" &
vmrun=$!
@@ -236,6 +264,9 @@ while :; do
tailer=""
t_end=$(date +%s)
t_create=$(at 'ubi-runner: creating ')
# A reattached VM is billed from its original create.
reattached=$(sed -n 's/^[0-9]* ubi-runner: reattached to [^ ]* created_at=\([0-9][0-9]*\) .*/\1/p' "$log" | tail -1)
[ -z "$reattached" ] || t_create=$reattached
if [ -n "$t_create" ] && ! grep -q 'ubi-runner: create failed' "$log"; then minutes=$(( (t_end - t_create + 59) / 60 )); billed=$(( billed + (minutes > 0 ? minutes : 1) )); fi
if [ "$rc" -ne 0 ] && [ "$attempt" -eq 1 ] && grep -q 'vCPU quota exhausted' "$log"; then
say "vCPU quota refused the VM; one retry in ${PREGATE_REMOTE_QUOTA_WAIT_S:-60} s"
@@ -279,7 +310,7 @@ if [ -n "$result" ] && [ "$rc" -ne 124 ]; then
echo "PREGATE_REMOTE: tree=$vm_tree tests=$(field tests "$result") failed=$(field failed "$result") spent=$spent"
exit $(( rc == 0 ? 0 : 1 ))
fi
vm=$(sed -n 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' "$log" | tail -1)
vm=$(sed -n -e 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' -e 's/^[0-9]* ubi-runner: reattached to \([^ ]*\) .*/\1/p' "$log" | tail -1)
down=$(grep -E 'ubi-runner: (destroyed|gone:|WARNING|FAILED to destroy)' "$log" | tail -1 | cut -d' ' -f2- || true)
cause=$(grep -E 'ubi-runner: ' "$log" | grep -vE 'ubi-runner: (destroyed|gone:|destroy requested|waiting for an in-flight)' | tail -1 | cut -d' ' -f2- || true)
[ "$rc" -ne 124 ] || cause="the VM was told to stop after ${stop_after}s ($stop_why) and torn down"
scripts/ubicloud/ubi-runner.sh
+628 -40
View File
@@ -33,8 +33,9 @@
# Teardown: `up` records each VM's state (location, create status) before it
# sends the create request, and `down` destroys and polls until the VM is
# confirmed gone or provably never existed, waiting out a create that may still
# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, HUP, QUIT) runs
# that `down`; further signals are ignored until it finishes. A create refused
# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, QUIT, and HUP
# until a remote phase is live) runs that `down`; further signals are ignored
# until it finishes. A create refused
# for vCPU quota prints the project's usage per owner.
#
# The owner-age sweep is off by default (the TTL reap below always runs). With UBI_GC_HOURS set to a positive
@@ -69,6 +70,33 @@
# UBI_RUN_TIMEOUT seconds (0, unlimited), exiting 124 with the last 40 lines,
# so a hung verify fails fast instead of holding the VM until its TTL.
#
# Detached phases and reattach: `run` and `verify` start their setup, command
# and tier phases on the VM detached from the ssh connection (setsid; stdout to
# ~/.ubirun-<phase>.out, stderr to .err, the session leader's pid in .pid, and
# the exit status in .exit, written by rename once the streams are closed) and
# stream both files back. Before creating a VM they compute a run key locally
# (the checkout's content tree, the command, the setup script's content, every
# --env entry, each --pass name with a sha256 of its value, size, location,
# image, storage, origin/HEAD, tiers and this script's own sha256) and look for
# a run record (<state dir>/<vm>/run) with the same key whose VM still exists:
# its runner alive (lease pid and start time) refuses with UBI_ATTACH_BUSY,
# otherwise the run reattaches, printing `reattached to <vm> created_at=<epoch>
# key=<k12>`, and continues from the first phase without an exit file. After
# unpack the VM recomputes the tree and refuses a mismatch (UBI_VM_TREE_MISMATCH).
# A leader gone with no exit file exits 125 (UBI_ATTACH_LOST). Signals: INT
# and TERM tear the VM down as before; HUP once a remote phase is live detaches
# (UBI_DETACHED, exit 129) and leaves the VM under its TTL; SIGKILL or a machine
# stop leaves the record and the VM, and the next run of the same command
# reattaches. `attach NAME` resumes a recorded run explicitly. Reattach never
# extends a TTL, and the reaper still applies.
#
# Tier checkpoints: a .gstack/ubicloud.json with `tiers` runs each tier as its
# own detached phase after setup, and records start/pass/fail per tier under
# UBI_CHECKPOINT_DIR/<run key>/ (write-once JSON plus the tier's log, published
# under a lock with a monotonic generation). A rerun reuses a pass younger than
# UBI_CHECKPOINT_MAX_AGE whose log re-hashes, so a resume on a fresh VM reruns
# only setup and the tiers without a pass.
#
# `verify` is the one entry point for a repository's full remote verify: it
# reads .gstack/ubicloud.json (setup, command, size, location, env, pass, pull)
# and runs it through `run`, so lanes share one committed definition instead of
@@ -100,6 +128,11 @@ PREFIX="ubirun"
if [ -n "${UBI_VM_RECORD:-}" ]; then VM_RECORD="$UBI_VM_RECORD"
elif [ -d "$HOME/.capy/work" ]; then VM_RECORD="$HOME/.capy/work/ubi-runner/vms.tsv"
else VM_RECORD="$STATE_ROOT/vms.tsv"; fi
if [ -n "${UBI_CHECKPOINT_DIR:-}" ]; then CHECKPOINT_DIR="$UBI_CHECKPOINT_DIR"
elif [ -d "$HOME/.capy/work" ]; then CHECKPOINT_DIR="$HOME/.capy/work/ubi-runner/checkpoints"
else CHECKPOINT_DIR="$STATE_ROOT/checkpoints"; fi
CHECKPOINT_MAX_AGE="${UBI_CHECKPOINT_MAX_AGE:-86400}"
REATTACH_MARGIN="${UBI_REATTACH_MARGIN:-900}"
die() { echo "ubi-runner: $*" >&2; exit 1; }
log() { echo "ubi-runner: $*" >&2; }
@@ -109,7 +142,7 @@ for bin in curl python3 ssh ssh-keygen tar; do
done
UBICLOUD_API_KEY="${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}"
[[ "$GC_HOURS" =~ ^[0-9]+$ ]] || die "UBI_GC_HOURS must be a whole number of hours (unset or 0 disables the sweep)"
for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT; do
for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT CHECKPOINT_MAX_AGE REATTACH_MARGIN; do
[[ "${!v}" =~ ^[0-9]+$ ]] || die "UBI_$v must be a whole number of seconds (0 disables it)"
done
[ -z "$QUOTA_VCPUS" ] || [[ "$QUOTA_VCPUS" =~ ^[0-9]+$ ]] || die "UBI_QUOTA_VCPUS must be a whole number of vCPUs"
@@ -296,6 +329,410 @@ watched() {
return "$rc"
}
proc_start() { ps -o lstart= -p "$1" 2>/dev/null | tr -s ' ' || true; }
fsize() { if [ -f "$1" ]; then wc -c <"$1" | tr -d ' '; else echo 0; fi; }
sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum | cut -d' ' -f1; else shasum -a 256 | cut -d' ' -f1; fi; }
# TREE_SH: print the content tree of the current directory, the same way here
# and on the VM: at a checkout's root, `git add -A` into a copy of its index
# (tracked plus untracked files that are not ignored, the files pack uploads);
# elsewhere, the same into a scratch repository over the directory.
TREE_SH='idx=$(mktemp -d "${TMPDIR:-/tmp}/ubirun-tree.XXXXXX") || exit 1
if [ "$(git rev-parse --show-toplevel 2>/dev/null)" = "$(pwd -P)" ]; then
gi=$(git rev-parse --git-path index)
if [ "$(git config --bool core.splitIndex)" != true ] && [ -f "$gi" ]; then cp "$gi" "$idx/index"; fi
else
git init -q --bare "$idx/git" && export GIT_DIR="$idx/git" GIT_WORK_TREE=.
fi
GIT_INDEX_FILE="$idx/index" git add -A >/dev/null 2>&1 && GIT_INDEX_FILE="$idx/index" git write-tree
rc=$?; rm -rf "$idx"; exit $rc'
# vm_tree NAME QDEST: the content tree of QDEST on the VM.
vm_tree() {
remote "$1" "command -v git >/dev/null || { sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git; } >/dev/null 2>&1; cd $2 && bash -c $(sq "$TREE_SH")" </dev/null
}
base_sha() { git -C "$1" rev-parse -q --verify 'refs/remotes/origin/HEAD^{commit}' 2>/dev/null || echo none; }
# phase_status NAME PHASE: "exit <rc>" once the phase wrote its exit file,
# "alive" while its session leader runs, "lost" when the leader is gone with no
# exit file, "none" when it never started.
phase_status() {
local p="~/.ubirun-$2"
remote "$1" "if [ -f $p.exit ]; then echo exit \$(cat $p.exit); elif [ -s $p.pid ] && kill -0 \$(cat $p.pid) 2>/dev/null; then echo alive; elif [ -s $p.pid ]; then echo lost; else echo none; fi" </dev/null
}
# phase_start NAME PHASE INNER: start INNER on the VM in its own session,
# detached from this ssh connection. The leader records its pid, runs INNER with
# stdout and stderr in files, and writes the exit status by rename only after
# INNER's streams are closed.
phase_start() {
local p="~/.ubirun-$2" inner
inner=$(sq "echo \$\$ >$p.pid.tmp && mv $p.pid.tmp $p.pid; ( $3 ) >$p.out 2>$p.err </dev/null; echo \$? >$p.exit.tmp && mv $p.exit.tmp $p.exit")
remote "$1" "rm -f $p.exit $p.pid; : >$p.out; : >$p.err; if command -v setsid >/dev/null 2>&1; then setsid -f bash -c $inner </dev/null >/dev/null 2>&1; else nohup bash -c $inner </dev/null >/dev/null 2>&1 & fi" </dev/null >/dev/null
}
# stream_cmd PHASE OUT_OFFSET ERR_OFFSET: follow both output files from the
# given byte offsets until the session leader exits.
stream_cmd() {
local p="~/.ubirun-$1"
printf '%s' "i=0; while [ ! -s $p.pid ] && [ ! -f $p.exit ] && [ \$i -lt 600 ]; do sleep 0.1; i=\$((i + 1)); done; pid=\$(cat $p.pid 2>/dev/null); f=; [ -z \"\$pid\" ] || f=\"-f --pid=\$pid\"; tail -c +$(( $3 + 1 )) \$f $p.err >&2 & tail -c +$(( $2 + 1 )) \$f $p.out; wait"
}
PHASE_LIVE="" STREAM_PID="" RUN_KEY="" TREE=""
# on_hup: once a remote phase is live, HUP detaches: the VM keeps running under
# its TTL and the next run of the same command reattaches. Before that, HUP
# tears down like INT and TERM.
on_hup() {
[ -n "$PHASE_LIVE" ] || exit 130
[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null || true
trap - EXIT
rm -f "$(state_dir "$RUN_VM")/lease"
leftover "left=1 owner=$OWNER vm=$RUN_VM detached=1"
echo "UBI_DETACHED: vm=$RUN_VM key=${RUN_KEY:0:12} reattach: rerun the same command" >&2
exit 129
}
# detached NAME PHASE INNER: run INNER as a detached phase (or reattach to it),
# streaming its output to this process and to the VM's state dir
# (<phase>.out, <phase>.err and the combined <phase>.log) under the same
# heartbeat and watchdog as watched. RUN_TIMEOUT counts from the phase's
# recorded start, not from a reattach. Returns the phase's exit status, 124
# after a watchdog kill, 125 when the session is gone without an exit status.
detached() {
local name=$1 phase=$2 inner=$3 dir st rc=0 tries=0 start now idle elapsed last_beat why old_traps spid sst oo oe
load "$name"
dir=$(state_dir "$name")
local lout="$dir/$phase.out" lerr="$dir/$phase.err" out="$dir/$phase.log" p="~/.ubirun-$phase"
st=none
if [ -f "$dir/phase.$phase.start" ]; then
st=$(phase_status "$name" "$phase") || st=""
fi
case $st in
none)
: >"$lout"; : >"$lerr"; : >"$out"
date +%s >"$dir/phase.$phase.start"
phase_start "$name" "$phase" "$inner" || { log "could not start $phase on $name"; return 255; } ;;
alive) log "$phase is still running on $name; following it" ;;
"exit "*) ;;
lost) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; return 125 ;;
*) log "could not read the state of $phase on $name"; return 255 ;;
esac
PHASE_LIVE=1
start=$(cat "$dir/phase.$phase.start")
# A runner killed on this machine leaves its stream client behind; it would
# append to the same local copies.
if [ -f "$dir/stream" ] && read -r spid sst <"$dir/stream" && [ -n "$spid" ] && [ "$(proc_start "$spid")" = "$sst" ]; then
kill -TERM -- "-$spid" 2>/dev/null || true
fi
oo=$(fsize "$lout") oe=$(fsize "$lerr")
if [ "$st" != none ]; then head -c "$oo" "$lout"; head -c "$oe" "$lerr" >&2; fi
old_traps=$(trap -p INT TERM)
trap '[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null; exit 130' INT TERM
last_beat=$(date +%s)
while :; do
touch "$out"
# shellcheck disable=SC2046
( exec python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
ssh $(ssh_opts) "ubi@$IP" "umask 022; $(stream_cmd "$phase" "$oo" "$oe")" ) \
</dev/null > >(tee -a "$lout" | tee -a "$out") 2> >(tee -a "$lerr" | tee -a "$out" >&2) &
STREAM_PID=$!
echo "$STREAM_PID $(proc_start "$STREAM_PID")" >"$dir/stream"
why=""
while :; do
sleep 1
kill -0 "$STREAM_PID" 2>/dev/null || break
now=$(date +%s)
elapsed=$(( now - start ))
idle=$(( now - $(mtime "$out") ))
if [ "$IDLE_TIMEOUT" -gt 0 ] && [ "$idle" -ge "$IDLE_TIMEOUT" ]; then
why="printed nothing for ${idle}s (UBI_IDLE_TIMEOUT=$IDLE_TIMEOUT)"
elif [ "$RUN_TIMEOUT" -gt 0 ] && [ "$elapsed" -ge "$RUN_TIMEOUT" ]; then
why="ran ${elapsed}s (UBI_RUN_TIMEOUT=$RUN_TIMEOUT)"
fi
[ -z "$why" ] || break
if [ "$HEARTBEAT" -gt 0 ] && [ $(( now - last_beat )) -ge "$HEARTBEAT" ]; then
log "heartbeat: $phase ${elapsed}s elapsed, last output ${idle}s ago"
last_beat=$now
fi
done
if [ -n "$why" ]; then
log "watchdog: $phase $why; killing it. Last output:"
tail -n 40 "$out" >&2
kill -TERM -- "-$STREAM_PID" 2>/dev/null || kill "$STREAM_PID" 2>/dev/null || true
wait "$STREAM_PID" 2>/dev/null || true
STREAM_PID=""
# A phase that finished in the instant before the kill keeps its exit 0.
if [ "$(phase_status "$name" "$phase" 2>/dev/null)" = "exit 0" ]; then
log "watchdog: $phase finished before it could be killed; its exit 0 stands"
rc=0
else
remote_cancel "$name" "$p.pid"
rc=124
fi
break
fi
wait "$STREAM_PID" 2>/dev/null || true
STREAM_PID=""
st=$(phase_status "$name" "$phase") || st=""
case $st in
"exit "*) rc=${st#exit }; break ;;
lost|none) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; rc=125; break ;;
esac
tries=$(( tries + 1 ))
if [ "$tries" -ge 5 ]; then log "lost the connection to $phase on $name 5 times; giving up (the VM keeps it until teardown)"; rc=255; break; fi
log "the stream of $phase on $name dropped; following it again"
sleep "$POLL"
oo=$(fsize "$lout") oe=$(fsize "$lerr")
done
rm -f "$dir/stream"
trap - INT TERM; eval "$old_traps"
return "$rc"
}
# write_record DIR: the run record a later run of the same command reattaches
# by (key, tree, and this invocation for `attach NAME`). Remote files of each
# phase: ~/.ubirun-<phase>.{out,err,exit,pid} (setup, command, tier-<name>).
qwords() { local x; for x in "$@"; do printf '%q ' "$x"; done; }
write_record() {
{
printf 'RUN_KEY=%q\nTREE=%q\nPULL_ROOT=%q\nRESUME=%q\nRERUN_FAILED=%q\n' "$RUN_KEY" "$TREE" "$PULL_ROOT" "$RESUME" "$RERUN_FAILED"
printf 'RUN_ARGS=(%s)\n' "$(qwords "${RUN_ARGS[@]}")"
printf 'TIER_NAMES=(%s)\n' "$(qwords ${TIER_NAMES[@]+"${TIER_NAMES[@]}"})"
printf 'TIER_CMDS=(%s)\n' "$(qwords ${TIER_CMDS[@]+"${TIER_CMDS[@]}"})"
echo "# remote files: ~/.ubirun-<phase>.{out,err,exit,pid}; phase starts: phase.<phase>.start; lease: lease (pid start time)"
} >"$1/run.$$" && mv "$1/run.$$" "$1/run"
}
lease_take() { echo "$$ $(proc_start $$)" >"$1/lease.$$" && mv "$1/lease.$$" "$1/lease"; }
lease_busy() {
local pid st
[ -f "$1/lease" ] || return 1
read -r pid st <"$1/lease" || true
[ -n "$pid" ] && [ "$pid" != "$$" ] && kill -0 "$pid" 2>/dev/null && [ "$(proc_start "$pid")" = "$st" ]
}
# vm_alive NAME: the recorded VM still answers show. One confirmed gone is
# cleaned up (state dir and scratch record), so its record is never tried again.
vm_alive() {
local n=$1 out code
( load "$n"; [ "$CREATED" = yes ] && [ -n "$IP" ] ) 2>/dev/null || return 1
out=$(mktemp)
code=$(load "$n"; api "$out" vm "$LOCATION/$n" show)
rm -f "$out"
case $code in
200) return 0 ;;
404) log "run record for $n: the VM is gone; starting a new one"; cmd_down "$n" >/dev/null 2>&1 || true; return 1 ;;
*) return 1 ;;
esac
}
# ttl_short NAME SETUP: true when NAME's TTL (its created_at plus the name's
# -t<hours>, minus now) does not cover the run still to do plus
# REATTACH_MARGIN, with the numbers in TTL_WHY. The run still to do is the
# UBI_RUN_TIMEOUT bound of each phase that has not finished (setup when SETUP is
# set, then the command or every tier the plan does not reuse), counted from a
# started phase's recorded start; it is 0 when UBI_RUN_TIMEOUT is 0.
TTL_WHY=""
ttl_short() {
local name=$1 dir created left now need=0 p i phases=()
[[ "$name" =~ -t([0-9]{1,3})$ ]] || return 1
dir=$(state_dir "$name")
created=$(load "$name"; echo "$CREATE_AT")
now=$(date +%s)
left=$(( created + 10#${BASH_REMATCH[1]} * 3600 - now ))
[ -z "$2" ] || phases+=(setup)
if [ ${#TIER_NAMES[@]} -gt 0 ]; then
for i in "${!TIER_NAMES[@]}"; do [ "${TIER_ACT[i]}" = reuse ] || phases+=("tier-${TIER_NAMES[i]}"); done
else
phases+=(command)
fi
if [ "$RUN_TIMEOUT" -gt 0 ]; then
for p in "${phases[@]}"; do
if [ -f "$dir/phase.$p.start" ]; then i=$(( RUN_TIMEOUT - (now - $(cat "$dir/phase.$p.start")) )); [ "$i" -gt 0 ] || i=0
else i=$RUN_TIMEOUT; fi
need=$(( need + i ))
done
fi
TTL_WHY="ttl_left=${left}s needs=$(( need + REATTACH_MARGIN ))s (run still to do ${need}s + UBI_REATTACH_MARGIN=${REATTACH_MARGIN}s)"
[ "$left" -lt $(( need + REATTACH_MARGIN )) ]
}
# find_record: the VM of a run record with this RUN_KEY whose VM still exists.
find_record() {
local f n
for f in "$STATE_ROOT"/*/run; do
[ -f "$f" ] || continue
grep -qx "RUN_KEY=$RUN_KEY" "$f" || continue
n=$(basename "$(dirname "$f")")
if vm_alive "$n"; then echo "$n"; return 0; fi
done
}
# Tier checkpoints: write-once records <tier>.<epoch>.<pid>.json beside a copy
# of the tier's log, under CHECKPOINT_DIR/<run key>/, published under a lock
# with a monotonic generation; the newest generation per tier decides.
CKPT_PY='
import fcntl, hashlib, json, os, shutil, sys, time
op, d = sys.argv[1], sys.argv[2]
def records():
out = []
for f in (os.listdir(d) if os.path.isdir(d) else []):
if not f.endswith(".json") or f.startswith("."):
continue
try:
r = json.load(open(os.path.join(d, f)))
except (OSError, ValueError):
continue
if isinstance(r, dict) and isinstance(r.get("generation"), int):
r["_file"] = f
out.append(r)
return out
def newest(rs, tier):
rs = [r for r in rs if r.get("tier") == tier]
return max(rs, key=lambda r: r["generation"]) if rs else None
def sha(p):
h = hashlib.sha256()
with open(p, "rb") as fh:
for b in iter(lambda: fh.read(65536), b""):
h.update(b)
return h.hexdigest()
if op == "decide":
now, maxage, resume, rerun = int(sys.argv[3]), int(sys.argv[4]), sys.argv[5] == "1", sys.argv[6] == "1"
rs = records()
for tier in sys.argv[7:]:
r = newest(rs, tier)
f = r["_file"] if r else "-"
age = now - int(r.get("finished_at") or r.get("started_at") or now) if r else 0
dur = int(r.get("duration_s") or 0) if r else 0
code = str(r.get("exit", "-")) if r else "-"
log = os.path.join(d, str(r.get("log"))) if r and r.get("log") else ""
if not resume:
act, why = "run", "--no-resume"
elif r is None:
act, why = "run", "no record"
elif r.get("status") == "start":
act, why = "run", "interrupted (only a start record, %s)" % f
elif r.get("status") == "pass":
if age >= maxage:
act, why = "run", "pass %s is %ds old (UBI_CHECKPOINT_MAX_AGE=%d)" % (f, age, maxage)
elif not log or not os.path.isfile(log) or sha(log) != r.get("log_sha256"):
act, why = "run", "pass %s: its log does not re-hash" % f
else:
act, why = "reuse", "-"
elif r.get("status") == "fail":
act, why = ("rerun", "prior fail %s (exit %s; --rerun-failed)" % (f, code)) if rerun else ("blocked", "-")
else:
act, why = "run", "unreadable record %s" % f
print("\t".join([tier, act, f, str(age), str(dur), code, why]))
elif op == "publish":
tier, status, vm, started, finished, code, log, startgen, key, tree, pid = sys.argv[3:14]
os.makedirs(d, exist_ok=True)
with open(os.path.join(d, ".lock"), "a") as lk:
fcntl.flock(lk, fcntl.LOCK_EX)
rs = records()
gen = max([r["generation"] for r in rs] + [0]) + 1
if status != "start":
r = newest(rs, tier)
if r is None or str(r["generation"]) != startgen:
print("stale writer: its start record is generation %s, the newest for %s is %s" % (startgen, tier, r["generation"] if r else "none"), file=sys.stderr)
sys.exit(2)
rec = {"tier": tier, "status": status, "generation": gen, "vm": vm, "key": key, "tree": tree, "started_at": int(started)}
if status != "start":
rec.update(finished_at=int(finished), duration_s=int(finished) - int(started), exit=int(code))
epoch = int(time.time())
while os.path.exists(os.path.join(d, "%s.%d.%s.json" % (tier, epoch, pid))) or os.path.exists(os.path.join(d, "%s.%d.%s.log" % (tier, epoch, pid))):
epoch += 1
stem = "%s.%d.%s" % (tier, epoch, pid)
if status != "start":
shutil.copyfile(log, os.path.join(d, stem + ".log"))
rec["log"], rec["log_sha256"] = stem + ".log", sha(os.path.join(d, stem + ".log"))
tmp = os.path.join(d, ".%s.tmp" % stem)
with open(tmp, "w") as fh:
json.dump(rec, fh, sort_keys=True)
fh.write("\n")
os.link(tmp, os.path.join(d, stem + ".json"))
os.unlink(tmp)
print(gen)
'
# ckpt_publish TIER STATUS VM STARTED FINISHED EXIT LOG STARTGEN: print the new
# record's generation; fails (nothing published) for a stale writer.
ckpt_publish() { python3 -c "$CKPT_PY" publish "$CHECKPOINT_DIR/$RUN_KEY" "$@" "$RUN_KEY" "$TREE" "$$"; }
TIER_ACT=() TIER_TO_RUN=0 TIER_PASSED=0 TIER_REUSED="" TIER_FAILED="" TIER_PRIOR="" TIER_SAVED=0 TIER_BLOCKED="" TIER_BLOCKED_RC=1
# tier_plan: decide each tier from its newest checkpoint and print one
# UBI_VERIFY_RESUME line per tier before anything runs.
tier_plan() {
local plan t act f age dur code why
plan=$(python3 -c "$CKPT_PY" decide "$CHECKPOINT_DIR/$RUN_KEY" "$(date +%s)" "$CHECKPOINT_MAX_AGE" "$RESUME" "$RERUN_FAILED" "${TIER_NAMES[@]}") \
|| die "verify: could not read the tier checkpoints in $CHECKPOINT_DIR/$RUN_KEY"
while IFS=$'\t' read -r t act f age dur code why; do
case $act in
reuse)
echo "UBI_VERIFY_RESUME: tier=$t reuse $f (${age}s)" >&2
TIER_REUSED=${TIER_REUSED:+$TIER_REUSED,}$t TIER_SAVED=$(( TIER_SAVED + dur )) TIER_PASSED=$(( TIER_PASSED + 1 )) ;;
blocked)
echo "UBI_VERIFY_RESUME: tier=$t fail $f (${age}s, exit $code): not rerun without --rerun-failed" >&2
if [ -z "$TIER_BLOCKED" ]; then TIER_BLOCKED=$t; if [[ "$code" =~ ^[0-9]+$ ]]; then TIER_BLOCKED_RC=$code; fi; fi ;;
rerun)
echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
TIER_PRIOR=${TIER_PRIOR:+$TIER_PRIOR,}$t:$code TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
*)
echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
esac
TIER_ACT+=("$act")
done <<<"$plan"
}
verify_line() {
echo "UBI_VERIFY: tree=${TREE:0:12} key=${RUN_KEY:0:12} tiers=${#TIER_NAMES[@]} passed=$TIER_PASSED reused=${TIER_REUSED:-none} failed=${TIER_FAILED:-none} prior_failures=${TIER_PRIOR:-none} saved_s=$TIER_SAVED"
}
# run_tiers NAME QDEST: run every tier the plan did not reuse, in order, each
# as its own detached phase; stop at the first failure and return its status.
# A pass is recorded only when the VM's tree after the tier equals the tree
# before it.
run_tiers() {
local name=$1 qdest=$2 dir i t phase before after trc started finished gen
dir=$(state_dir "$name")
for i in "${!TIER_NAMES[@]}"; do
[ "${TIER_ACT[i]}" != reuse ] || continue
t=${TIER_NAMES[i]} phase="tier-${TIER_NAMES[i]}"
if [ ! -f "$dir/phase.$phase.start" ]; then
before=$(vm_tree "$name" "$qdest") || before=unavailable
echo "$before" >"$dir/phase.$phase.tree"
if gen=$(ckpt_publish "$t" start "$name" "$(date +%s)" 0 0 "" 0); then echo "$gen" >"$dir/phase.$phase.gen"
else echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t start record (the tier still runs)" >&2; fi
fi
log "running tier $t on $name: ${TIER_CMDS[i]}"
trc=0
detached "$name" "$phase" "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "${TIER_CMDS[i]}")" || trc=$?
started=$(cat "$dir/phase.$phase.start" 2>/dev/null || date +%s) finished=$(date +%s)
log "tier $t exited $trc after $(( finished - started ))s"
case $trc in 125|255) TIER_FAILED=$t; return "$trc" ;; esac
gen=$(cat "$dir/phase.$phase.gen" 2>/dev/null || echo 0)
if [ "$trc" -ne 0 ]; then
ckpt_publish "$t" fail "$name" "$started" "$finished" "$trc" "$dir/$phase.log" "$gen" >/dev/null \
|| echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t fail record (the result stands)" >&2
TIER_FAILED=$t
return "$trc"
fi
TIER_PASSED=$(( TIER_PASSED + 1 ))
after=$(vm_tree "$name" "$qdest") || after=unavailable
before=$(cat "$dir/phase.$phase.tree" 2>/dev/null || echo unavailable)
if [ "$after" != "$before" ] || [ "$after" = unavailable ]; then
echo "UBI_VERIFY_TREE_MOVED: tier=$t before=${before:0:12} after=${after:0:12}: a tier must leave the source tree as it found it; no pass record" >&2
continue
fi
ckpt_publish "$t" pass "$name" "$started" "$finished" 0 "$dir/$phase.log" "$gen" >/dev/null \
|| echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t pass record (the result stands)" >&2
done
}
# vm_rows: "location name owner expires created" for every VM in the project;
# owner is "-" for untagged ubirun-<epoch>-* names and "." for VMs this runner
# did not name; expires is the epoch the name's -t<hours> TTL ends at, or "-";
@@ -770,9 +1207,11 @@ run_teardown() {
[ "$rc" -ne 0 ] || exit 1
}
TIER_NAMES=() TIER_CMDS=() RESUME=1 RERUN_FAILED=0 FORCE_ATTACH="" RUN_ARGS=()
cmd_run() {
require_owner
local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=()
local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=() passes=() key_inputs=() probe=0
RUN_ARGS=("$@")
while [ $# -gt 0 ]; do
case $1 in
-S|--storage) check_storage "$2"; up_args+=("$1" "$2"); shift 2 ;;
@@ -781,60 +1220,161 @@ cmd_run() {
--dest) dest=$2; shift 2 ;;
--setup) setup=$2; shift 2 ;;
--env) envs+=("$2"); shift 2 ;;
--pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; envs+=("$2=${!2}"); shift 2 ;;
--pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; passes+=("$2"); shift 2 ;;
--pull) pulls+=("$2"); shift 2 ;;
--keep) keep=1; shift ;;
--key-input) key_inputs+=("$2"); shift 2 ;;
--probe) probe=1; shift ;;
--resume) RESUME=1; shift ;;
--no-resume) RESUME=0; shift ;;
--rerun-failed) RERUN_FAILED=1; shift ;;
--) shift; break ;;
*) die "run: unknown option $1" ;;
esac
done
[ $# -gt 0 ] || die "run: missing command after --"
[ $# -gt 0 ] || [ ${#TIER_NAMES[@]} -gt 0 ] || die "run: missing command after --"
[ -z "$setup" ] || [ -f "$setup" ] || die "setup script not found: $setup"
command -v git >/dev/null || die "git is required: the run key fingerprints the checkout with it"
src=$(cd "$src" && pwd)
[ -n "$dest" ] || dest="work/$(basename "$src")"
local i name=""
local i name="" size=$DEFAULT_SIZE location=$DEFAULT_LOCATION image=ubuntu-noble storage="" e n
for (( i = 0; i < ${#up_args[@]}; i += 2 )); do
case ${up_args[i]} in -n|--name) name=${up_args[i+1]} ;; esac
case ${up_args[i]} in
-n|--name) name=${up_args[i+1]} ;;
-s|--size) size=${up_args[i+1]} ;;
-l|--location) location=${up_args[i+1]} ;;
-b|--image) image=${up_args[i+1]} ;;
-S|--storage) storage=${up_args[i+1]} ;;
esac
done
# The run key: everything that decides what the VM runs, computed before any
# VM exists. Pass values enter only as their sha256.
if [ -z "$FORCE_ATTACH" ]; then
TREE=$(cd "$src" && bash -c "$TREE_SH") || die "could not fingerprint $src (git add -A into a temporary index failed)"
RUN_KEY=$({
echo "ubi-runner run key v1"
echo "tree $TREE"
printf 'command %q\n' "$*"
if [ -n "$setup" ]; then echo "setup $(sha256 <"$setup")"; else echo "setup none"; fi
for e in ${envs[@]+"${envs[@]}"}; do printf 'env %q\n' "$e"; done
for n in ${passes[@]+"${passes[@]}"}; do printf 'pass %s %s\n' "$n" "$(printf %s "${!n}" | sha256)"; done
printf 'vm %q %q %q %q\n' "$size" "$location" "$image" "$storage"
echo "base $(base_sha "$src")"
echo "runner $(sha256 <"${BASH_SOURCE[0]}")"
for e in ${key_inputs[@]+"${key_inputs[@]}"}; do printf 'input %q\n' "$e"; done
for i in "${!TIER_NAMES[@]}"; do printf 'tier %q %q\n' "${TIER_NAMES[i]}" "${TIER_CMDS[i]}"; done
} | sha256)
fi
[ ${#TIER_NAMES[@]} -eq 0 ] || tier_plan
local attach=""
if [ -n "$FORCE_ATTACH" ]; then
vm_alive "$FORCE_ATTACH" || die "attach: $FORCE_ATTACH no longer exists; rerun the original command to start fresh"
attach=$FORCE_ATTACH
else
attach=$(find_record)
fi
if [ "$probe" = 1 ]; then
if [ -z "$attach" ]; then echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12}"
else
local busy=0
! lease_busy "$(state_dir "$attach")" || busy=1
if [ "$busy" = 0 ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12} ttl_short vm=$attach $TTL_WHY"
else
echo "UBI_ATTACH_INFO: vm=$attach created_at=$(load "$attach"; echo "$CREATE_AT") key=${RUN_KEY:0:12} busy=$busy"
fi
fi
return 0
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && { [ -n "$TIER_BLOCKED" ] || [ "$TIER_TO_RUN" -eq 0 ]; } && [ -n "$attach" ] && ! lease_busy "$(state_dir "$attach")"; then
log "verify: no tier needs $attach, which a dead runner left behind; destroying it"
cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && [ -n "$TIER_BLOCKED" ]; then
TIER_FAILED=$TIER_BLOCKED
verify_line
return "$TIER_BLOCKED_RC"
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && [ "$TIER_TO_RUN" -eq 0 ]; then
log "verify: every tier passed under this key; no VM needed"
verify_line
return 0
fi
if [ -n "$attach" ] && lease_busy "$(state_dir "$attach")"; then
die "UBI_ATTACH_BUSY: vm=$attach key=${RUN_KEY:0:12}: runner pid $(cut -d' ' -f1 <"$(state_dir "$attach")/lease") on this machine still owns it; wait for it or stop it, then rerun"
fi
if [ -n "$attach" ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
log "UBI_REATTACH_TTL_SHORT: vm=$attach key=${RUN_KEY:0:12} $TTL_WHY: starting a fresh VM (tier checkpoints carry over) and destroying $attach"
cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
attach=""
fi
[ -z "$attach" ] || name=$attach
[ -n "$name" ] || { name=$(new_name); up_args+=(-n "$name"); }
RUN_VM=$name
# Armed before the create request: an interrupted `up` still destroys its VM.
# A second signal must not cut teardown short, so the EXIT handler ignores them.
trap 'run_teardown "$?"' EXIT
trap 'exit 130' INT TERM HUP QUIT
cmd_up "${up_args[@]}" >/dev/null
if [[ "$name" =~ -t([0-9]+)$ ]]; then
log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
trap 'exit 130' INT TERM QUIT
trap on_hup HUP
if [ -n "$attach" ]; then
load "$name"
log "reattached to $name created_at=$CREATE_AT key=${RUN_KEY:0:12}"
else
log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
cmd_up "${up_args[@]}" >/dev/null
trap 'exit 130' INT TERM QUIT
trap on_hup HUP
if [[ "$name" =~ -t([0-9]+)$ ]]; then
log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
else
log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
fi
fi
local dir
dir=$(state_dir "$name")
write_record "$dir"
lease_take "$dir"
if [ "$keep" = 1 ]; then
trap - EXIT
# A kept VM's finished run is not reattached by the next run; a detach (HUP) keeps the record.
trap 'rm -f "$(state_dir "$RUN_VM")/run"' EXIT
leftover "left=1 owner=$OWNER vm=$name kept=1"
log "--keep: leaving $name running; destroy with: $0 down $name"
fi
local e
for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done \
| remote "$name" 'umask 077 && cat > ~/.ubirun-env'
cmd_sync "$name" "$src" "$dest"
local qdest rc=0
local qdest rc=0 vmt
qdest=$(printf %q "$dest")
if [ ! -f "$dir/synced" ]; then
{ for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done
for n in ${passes[@]+"${passes[@]}"}; do printf 'export %s=%q\n' "$n" "${!n}"; done; } \
| remote "$name" 'umask 077 && cat > ~/.ubirun-env'
cmd_sync "$name" "$src" "$dest"
# A subdirectory of a checkout uploads the whole .git with only its own
# files, so its tree is compared VM to VM (tiers) but not to this side.
if [ -z "$(git -C "$src" rev-parse --show-prefix 2>/dev/null)" ]; then
vmt=$(vm_tree "$name" "$qdest") || vmt=unavailable
if [ "$vmt" != "$TREE" ]; then
log "UBI_VM_TREE_MISMATCH: vm=$name tree=${vmt:0:12} expected=${TREE:0:12}: the unpacked checkout is not the one this run was keyed on"
exit 3
fi
fi
[ -z "$setup" ] || remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup"
: >"$dir/synced"
fi
if [ -n "$setup" ]; then
remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup"
log "running setup $(basename "$setup")"
watched "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; }
detached "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; }
fi
local start
start=$(date +%s)
log "running on $name: $*"
watched "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$?
log "command exited $rc after $(( $(date +%s) - start ))s"
if [ ${#TIER_NAMES[@]} -gt 0 ]; then
run_tiers "$name" "$qdest" || rc=$?
else
log "running on $name: $*"
detached "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$?
log "command exited $rc after $(( $(date +%s) - $(cat "$dir/phase.command.start" 2>/dev/null || date +%s) ))s"
fi
local p
for p in ${pulls[@]+"${pulls[@]}"}; do
@@ -845,9 +1385,23 @@ cmd_run() {
fi
cmd_pull "$name" "${p%%:*}" "${p#*:}" || log "pull failed: $p"
done
[ ${#TIER_NAMES[@]} -eq 0 ] || verify_line
return "$rc"
}
# attach NAME: resume the run recorded for NAME on this machine (the same
# invocation, key and tiers), as a rerun of the same command would.
cmd_attach() {
local name=${1:-}
[ -n "$name" ] || die "attach: missing NAME"
valid_name "$name" || die "invalid VM name '$name'"
[ -f "$(state_dir "$name")/run" ] || die "attach: no run record for $name on this machine ($(state_dir "$name")/run); a machine replacement loses it"
# shellcheck disable=SC1091
. "$(state_dir "$name")/run"
FORCE_ATTACH=$name
cmd_run "${RUN_ARGS[@]}"
}
# verify [run opts] [--trust-config] [-- COMMAND]: run the repository's
# committed remote verify. .gstack/ubicloud.json under --src (default .)
# supplies setup, command, size, location, storage, image, env, pass and pull;
@@ -860,7 +1414,7 @@ cmd_run() {
# local variable it lists under pass is forwarded only when this command line
# also names it (--pass NAME) or grants every listed name (--trust-config).
cmd_verify() {
local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=()
local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=() cfg_tier_names=() cfg_tier_cmds=()
for (( i = 1; i <= $#; i++ )); do
[ "${!i}" = -- ] && break
if [ "${!i}" = --src ]; then j=$(( i + 1 )); src=${!j}; fi
@@ -869,7 +1423,7 @@ cmd_verify() {
[ -f "$cfg" ] || die "verify: no $cfg; commit one ({\"setup\": \"scripts/ubicloud/bootstrap.sh\", \"command\": \"bash scripts/ubicloud/verify.sh\"}) so every lane runs the same verify instead of its own wrapper"
err=$(mktemp)
parsed=$(python3 -c '
import json, os, shlex, sys
import json, os, re, shlex, sys
cfg, src = sys.argv[1], sys.argv[2]
try:
c = json.load(open(cfg))
@@ -895,7 +1449,25 @@ for item in c.get("pull", []):
if not sep or not remote:
print("pull entry %r must be REMOTE_GLOB:LOCAL_DIR" % (item,), file=sys.stderr); sys.exit(2)
args += ["--pull", q(remote + ":" + inside(local, "pull destination"))]
tiers = c.get("tiers")
names = []
if tiers is not None:
if "command" in c:
print("UBI_CONFIG_TIERS_AND_COMMAND: give either command or tiers, not both", file=sys.stderr); sys.exit(2)
if not isinstance(tiers, list) or not tiers:
print("UBI_CONFIG_TIER_NAME: tiers must be a non-empty list of {\"name\", \"command\"}", file=sys.stderr); sys.exit(2)
for t in tiers:
name = t.get("name") if isinstance(t, dict) else None
if not isinstance(name, str) or not re.fullmatch(r"[a-z0-9][a-z0-9_-]{0,31}", name):
print("UBI_CONFIG_TIER_NAME: tier name %r must match ^[a-z0-9][a-z0-9_-]{0,31}$" % (name,), file=sys.stderr); sys.exit(2)
if name in names:
print("UBI_CONFIG_TIER_NAME: tier name %r appears twice" % name, file=sys.stderr); sys.exit(2)
if not isinstance(t.get("command"), str) or not t["command"].strip():
print("UBI_CONFIG_TIER_NAME: tier %r needs a command" % name, file=sys.stderr); sys.exit(2)
names.append(name)
setup = c.get("setup", "")
print("cfg_tier_names=(" + " ".join(q(n) for n in names) + ")")
print("cfg_tier_cmds=(" + " ".join(q(t["command"]) for t in (tiers or [])) + ")")
print("cfg_args=(" + " ".join(args) + ")")
print("cfg_pass=(" + " ".join(q(x) for x in c.get("pass", [])) + ")")
print("cfg_setup=" + q(inside(setup, "setup") if setup else ""))
@@ -912,7 +1484,7 @@ print("cfg_cmd=" + q(c.get("command", "")))
case $1 in
--) shift; user_cmd=("$@"); break ;;
--trust-config) trust=1; shift ;;
--keep) user_pre+=("$1"); shift ;;
--keep|--resume|--no-resume|--rerun-failed|--probe) user_pre+=("$1"); shift ;;
--pass|-S|--storage|-s|--size|-l|--location|-n|--name|-b|--image|--src|--dest|--setup|--env|--pull)
[ $# -ge 2 ] || die "verify: $1 needs a value"
[ "$1" != --pass ] || granted+=("$2")
@@ -927,11 +1499,16 @@ print("cfg_cmd=" + q(c.get("command", "")))
missing+=("$n")
done
[ ${#missing[@]} -eq 0 ] || die "verify: $cfg asks to forward local ${missing[*]}; grant each on this command line with --pass NAME, or every name it lists with --trust-config"
[ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --"
[ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd")
# A trailing -- COMMAND replaces the configured tiers as it replaces command.
if [ ${#user_cmd[@]} -eq 0 ] && [ ${#cfg_tier_names[@]} -gt 0 ]; then
TIER_NAMES=("${cfg_tier_names[@]}") TIER_CMDS=("${cfg_tier_cmds[@]}")
else
[ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --"
[ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd")
fi
log "verify: $cfg"
PULL_ROOT=$(cd "$src" && pwd)
cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- "${user_cmd[@]}"
cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- ${user_cmd[@]+"${user_cmd[@]}"}
}
usage() {
@@ -939,13 +1516,17 @@ usage() {
usage: ubi-runner.sh <command> [args]
run [up opts] [--src DIR] [--dest PATH] [--setup FILE] [--env K=V]...
[--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] -- COMMAND
up + sync + setup + command + pull + destroy; exits with COMMAND's status
verify [run opts] [--trust-config] [-- COMMAND]
[--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] [--key-input S]... [--probe] -- COMMAND
up + sync + setup + command + pull + destroy; exits with COMMAND's status.
A rerun of the same command (same run key) reattaches to its live VM;
--probe prints UBI_ATTACH_INFO (and the tier plan) and exits
verify [run opts] [--trust-config] [--resume|--no-resume] [--rerun-failed] [-- COMMAND]
run the repository's committed verify (.gstack/ubicloud.json under --src);
its setup script and every pull destination stay inside --src (checked
again at pull time), and its pass names are forwarded only when granted
here (--pass NAME, or all: --trust-config)
here (--pass NAME, or all: --trust-config). With tiers, a pass checkpoint
under the same run key is reused (resume is the default)
attach NAME resume the run recorded for NAME on this machine, as a rerun would
up [-s SIZE] [-l LOCATION] [-n NAME] [-S STORAGE_GIB] [-b IMAGE]
create a VM and wait for SSH; prints its name
(STORAGE_GIB: one of $ALLOWED_STORAGE)
@@ -965,6 +1546,11 @@ usage: ubi-runner.sh <command> [args]
owner's TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS
cli ARGS... raw Ubicloud CLI passthrough (e.g. cli vm list)
signals: INT and TERM tear the VM down. HUP, once a remote phase is live, detaches
(UBI_DETACHED, exit 129; the VM runs on under its TTL). SIGKILL or a machine stop leaves
the run record and the VM; rerun the same command to reattach (UBI_ATTACH_BUSY while
another runner owns it; exit 125 UBI_ATTACH_LOST when the remote session left no status).
VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; up, run and verify need
UBI_OWNER (e.g. your thread code); run and verify end with a UBICLOUD: left=<n> line.
defaults: size=$DEFAULT_SIZE location=$DEFAULT_LOCATION (env UBI_SIZE, UBI_LOCATION)
@@ -976,7 +1562,8 @@ env: UBI_TTL_HOURS (${TTL_HOURS}h in the name, 1-999), UBI_REAP (0: up skips the
UBI_QUOTA_WAIT (${QUOTA_WAIT}s to wait for quota room instead of refusing),
UBI_GC_HOURS (unset/0: no sweep on up), UBI_CREATE_GRACE (${CREATE_GRACE}s),
UBI_DOWN_TIMEOUT (${DOWN_TIMEOUT}s), UBI_API_TIMEOUT (${API_TIMEOUT}s), UBI_VM_RECORD (scratch VM-id file),
UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640")
UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640"),
UBI_CHECKPOINT_DIR (tier checkpoints; $CHECKPOINT_DIR), UBI_CHECKPOINT_MAX_AGE (${CHECKPOINT_MAX_AGE}s a pass is reused)
EOF
}
@@ -989,6 +1576,7 @@ esac
case $cmd in
run) cmd_run "$@" ;;
verify) cmd_verify "$@" ;;
attach) cmd_attach "$@" ;;
up) trap 'trap "" INT TERM HUP QUIT; [ -z "$UP_NAME" ] || [ -n "$UP_READY" ] || cmd_down "$UP_NAME" || log "WARNING: failed to destroy $UP_NAME; run: $0 down $UP_NAME"' EXIT
trap 'exit 130' INT TERM HUP QUIT
cmd_up "$@" ;;
ship/sections/pr-body.md
+2 -1
View File
@@ -224,7 +224,8 @@ rm -f .gstack/tmp/review-loop.json
~/.claude/skills/gstack/bin/gstack-ship-receipt write --pr <pr-number> --base <base> --gate @.gstack/tmp/ship-gate.json --pregate .gstack/tmp/pregate.json --issues @.gstack/tmp/issues.json --review-loop @.gstack/tmp/review-loop.json --docs @.gstack/tmp/docs-audit.json --body "$PR_BODY_FILE"
```
`--docs` reads the record Step 14.5 saved (`{status, scope[, exception]}`): the
A Step 5 red-base pause adds `--base-red <pause file>` (a conditional receipt the queue
refuses, `QUEUE_RECEIPT_CONDITIONAL`). `--docs` reads the record Step 14.5 saved (`{status, scope[, exception]}`): the
receipt carries it as `docs`, and a `blocked` status without the owner's exception,
or a body missing the exception line, is `DOCS_AUDIT_BLOCKED` with nothing written.
When the owner accepted a stop after a fix-check edit, add
ship/sections/pr-body.md.tmpl
+2 -1
View File
@@ -207,7 +207,8 @@ rm -f .gstack/tmp/review-loop.json
~/.claude/skills/gstack/bin/gstack-ship-receipt write --pr <pr-number> --base <base> --gate @.gstack/tmp/ship-gate.json --pregate .gstack/tmp/pregate.json --issues @.gstack/tmp/issues.json --review-loop @.gstack/tmp/review-loop.json --docs @.gstack/tmp/docs-audit.json --body "$PR_BODY_FILE"
```
`--docs` reads the record Step 14.5 saved (`{status, scope[, exception]}`): the
A Step 5 red-base pause adds `--base-red <pause file>` (a conditional receipt the queue
refuses, `QUEUE_RECEIPT_CONDITIONAL`). `--docs` reads the record Step 14.5 saved (`{status, scope[, exception]}`): the
receipt carries it as `docs`, and a `blocked` status without the owner's exception,
or a body missing the exception line, is `DOCS_AUDIT_BLOCKED` with nothing written.
When the owner accepted a stop after a fix-check edit, add
ship/sections/tests.md
+27 -1
View File
@@ -368,7 +368,33 @@ esac && rm -f "$TITLE_FILE" "$BODY_FILE"
- Continue with the workflow.
- Note in output: "Pre-existing test failure skipped: <test-name>"
**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, or skipped), continue to Step 6.
**Red base (before T3).** When T1 calls every failure likely pre-existing,
prove it before asking: write `.gstack/tmp/base-red.json` with your file-write tool
(`{"lane":"<label>","total":<failures the lane reported>,"failures":[{"id":"<test
name>","file":"<test file>","signature":"<first error line>"}]}`), then run
```bash
~/.claude/skills/gstack/bin/gstack-base-red classify --failures .gstack/tmp/base-red.json --base <base> --pause
```
It reads `base_red` from the base branch's ship policy and reruns those files at
the base head and at HEAD in scratch worktrees. Exit 0 with `BASE_RED_VERDICT:
base-red` and `PAUSE: <file>`: skip T3, record the pause file and its cases in
the invocation record, and continue to Step 6; Step 16 treats exactly these cases
as this run's waived failures (any other failure is new), and the receipt write
adds `--base-red <pause file>`, so Step 22 ends `SHIP_PAUSED` with
`status=incomplete` instead of READY. Exit 1 (`IN_BRANCH:` lines,
`BASE_RED_UNCONFIGURED`, `BASE_RED_UNNAMED`, `BASE_RED_REPRO_ERROR`): continue T3
as written.
**Resume a red-base pause.** A parent waits with `gstack-base-red wait --pause
<file>` (a background operation; `BASE_GREEN:` once a new base head passes the
cases), then runs `/ship --resume <record>`: if HEAD is still the paused head,
rerun this Step 5 on HEAD merged with the new base in a scratch worktree (never a
pushed base merge); all green rewrites the receipt without `--base-red` and runs
Step 22. Any failure is triaged here as new; a moved HEAD starts /ship fresh.
**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, skipped, or base-red and paused), continue to Step 6.
**If all pass:** Report each lane's pass count (or `count unavailable`) in one line and continue to Step 6.
ship/sections/tests.md.tmpl
+27 -1
View File
@@ -56,7 +56,33 @@ Failure Ownership Triage:
{{TEST_FAILURE_TRIAGE}}
**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, or skipped), continue to Step 6.
**Red base (before T3).** When T1 calls every failure likely pre-existing,
prove it before asking: write `.gstack/tmp/base-red.json` with your file-write tool
(`{"lane":"<label>","total":<failures the lane reported>,"failures":[{"id":"<test
name>","file":"<test file>","signature":"<first error line>"}]}`), then run
```bash
~/.claude/skills/gstack/bin/gstack-base-red classify --failures .gstack/tmp/base-red.json --base <base> --pause
```
It reads `base_red` from the base branch's ship policy and reruns those files at
the base head and at HEAD in scratch worktrees. Exit 0 with `BASE_RED_VERDICT:
base-red` and `PAUSE: <file>`: skip T3, record the pause file and its cases in
the invocation record, and continue to Step 6; Step 16 treats exactly these cases
as this run's waived failures (any other failure is new), and the receipt write
adds `--base-red <pause file>`, so Step 22 ends `SHIP_PAUSED` with
`status=incomplete` instead of READY. Exit 1 (`IN_BRANCH:` lines,
`BASE_RED_UNCONFIGURED`, `BASE_RED_UNNAMED`, `BASE_RED_REPRO_ERROR`): continue T3
as written.
**Resume a red-base pause.** A parent waits with `gstack-base-red wait --pause
<file>` (a background operation; `BASE_GREEN:` once a new base head passes the
cases), then runs `/ship --resume <record>`: if HEAD is still the paused head,
rerun this Step 5 on HEAD merged with the new base in a scratch worktree (never a
pushed base merge); all green rewrites the receipt without `--base-red` and runs
Step 22. Any failure is triaged here as new; a moved HEAD starts /ship fresh.
**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, skipped, or base-red and paused), continue to Step 6.
**If all pass:** Report each lane's pass count (or `count unavailable`) in one line and continue to Step 6.
test/base-red.test.ts
+284
View File
@@ -0,0 +1,284 @@
/**
* gstack-base-red (docs/ship-policy.md#base-red): a Step 5 failure is base-red
* only when the base branch's policy declares how to reproduce it, every
* failure is named, the PR touches none of the failing files or the paths
* their signatures name, and the same test fails the same way at the base
* head and at the PR head in scratch worktrees. A conditional receipt is
* refused by every receipt consumer except /ship's own terminal line, which
* reports SHIP_PAUSED; wait returns only once a new base head passes.
*/
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { classifyBaseRed, parseEvidence, pausePath, readPause, receiptBaseRed, shellRunner, waitBaseGreen, writePause, type FailureEvidence, type Runner } from '../lib/base-red';
import { EXIT } from '../lib/headless-artifacts';
import { bindReceipt, extractReceipt, placeReceiptFirst, receiptSummaryLine } from '../lib/ship-receipt';
import { BASE_RED_KEY_VERSION, validatePolicy, type BaseRedPolicy } from '../lib/ship-policy';
const ROOT = path.resolve(import.meta.dir, '..');
const ID = { GIT_AUTHOR_NAME: 'fixture', GIT_AUTHOR_EMAIL: 'fixture@example.invalid', GIT_COMMITTER_NAME: 'fixture', GIT_COMMITTER_EMAIL: 'fixture@example.invalid' };
const SELECT = 'for f in {files}; do bash "$f" || status=1; done; exit ${status:-0}';
const POLICY: BaseRedPolicy = { setup: null, select: SELECT, timeout_sec: 60 };
const FAIL_TEST = 'echo "(fail) totals add up"; echo "error: expected 3 got 4 in lib/sum.sh"; exit 1\n';
const PASS_TEST = 'echo "(pass) totals add up"\n';
let root = '';
let work = '';
let state = '';
const prevState = process.env.GSTACK_STATE_ROOT;
const git = (...args: string[]) => {
const r = spawnSync('git', args, { cwd: work, encoding: 'utf8', timeout: 30_000, env: { ...process.env, ...ID } });
if (r.status !== 0) throw new Error(`git ${args.join(' ')}: ${r.stderr}`);
return r.stdout.trim();
};
const write = (rel: string, text: string) => { fs.mkdirSync(path.dirname(path.join(work, rel)), { recursive: true }); fs.writeFileSync(path.join(work, rel), text); };
const policyJson = (base_red: BaseRedPolicy | null) => JSON.stringify({ base_red, min_gstack_version: base_red ? BASE_RED_KEY_VERSION : null }) + '\n';
const evidence = (over: Partial<FailureEvidence> = {}): FailureEvidence => ({ lane: 'tests', total: 1, failures: [{ id: 'totals add up', file: 't/sum.test.sh', signature: 'error: expected 3 got 4' }], ...over });
const counting = () => { const calls: string[] = []; const runner: Runner = (cwd, cmd, s) => { calls.push(cmd); return shellRunner(cwd, cmd, s); }; return { calls, runner }; };
/** origin/main carries a red test; the PR branch changes only README.md. */
function fixture(base: { test?: string; policy?: BaseRedPolicy | null } = {}) {
write('t/sum.test.sh', base.test ?? FAIL_TEST);
write('lib/sum.sh', 'echo 4\n');
write('README.md', 'base\n');
write('.gstack/ship-policy.json', policyJson(base.policy === undefined ? POLICY : base.policy));
git('add', '-A'); git('commit', '-qm', 'base'); git('push', '-q', 'origin', 'HEAD:main'); git('fetch', '-q', 'origin');
git('checkout', '-q', '-b', 'feat');
write('README.md', 'pr\n'); git('commit', '-qam', 'pr: readme');
}
beforeEach(() => {
root = fs.mkdtempSync(path.join(os.tmpdir(), 'base-red-'));
work = path.join(root, 'work');
state = path.join(root, 'state');
process.env.GSTACK_STATE_ROOT = state;
spawnSync('git', ['init', '-q', '--bare', '-b', 'main', path.join(root, 'origin.git')], { timeout: 30_000 });
fs.mkdirSync(work);
git('init', '-q', '-b', 'main'); git('remote', 'add', 'origin', path.join(root, 'origin.git'));
});
afterEach(() => {
if (prevState === undefined) delete process.env.GSTACK_STATE_ROOT; else process.env.GSTACK_STATE_ROOT = prevState;
fs.rmSync(root, { recursive: true, force: true });
});
describe('classify', () => {
test('a failure that reproduces at base and head in a file the PR does not touch is base-red, and leaves no worktree behind', () => {
fixture();
const r = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence(), policy: POLICY });
expect(r.code).toBe('BASE_RED');
expect(r.cases).toEqual([{ id: 'totals add up', file: 't/sum.test.sh', verdict: 'BASE_RED', reason: `fails the same way at base ${r.base_sha.slice(0, 12)}` }]);
expect(git('worktree', 'list').split('\n')).toHaveLength(1);
expect(fs.readdirSync(path.join(state, 'ship', 'base-red', 'worktrees'))).toEqual([]);
});
test('the PR touching the failing test, or a path its signature names, is in-branch without running anything', () => {
fixture();
write('t/sum.test.sh', FAIL_TEST + '# touched\n'); git('commit', '-qam', 'touch test');
const a = counting();
const touched = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence(), policy: POLICY, runner: a.runner });
expect([touched.code, touched.cases[0]!.reason, a.calls]).toEqual(['IN_BRANCH', 'the PR changes t/sum.test.sh', []]);
git('reset', '-q', '--hard', 'HEAD~1');
write('lib/sum.sh', 'echo 5\n'); git('commit', '-qam', 'touch lib');
const named = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence({ failures: [{ id: 'totals add up', file: 't/sum.test.sh', signature: 'error: expected 3 got 4 in lib/sum.sh' }] }), policy: POLICY, runner: a.runner });
expect([named.code, named.cases[0]!.reason, a.calls]).toEqual(['IN_BRANCH', 'the PR changes lib/sum.sh', []]);
});
test('a test that passes on the base, or fails there with another signature, is in-branch', () => {
fixture({ test: PASS_TEST });
const passes = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence(), policy: POLICY });
expect(passes.code).toBe('IN_BRANCH');
expect(passes.cases[0]!.reason).toStartWith('passes at base ');
const other = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence({ failures: [{ id: 'totals add up', file: 't/sum.test.sh', signature: 'error: timeout after 5000ms' }] }), policy: { ...POLICY, select: 'bash -c "echo totals add up; echo error: expected; exit 1" # {files}' } });
expect([other.code, other.cases[0]!.reason.split(' at base')[0]]).toEqual(['IN_BRANCH', 'fails with a different signature']);
});
test('no base_red policy, an unnamed failure, or a failing setup keeps every failure in-branch', () => {
fixture({ policy: null });
expect(classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence(), policy: null }).code).toBe('BASE_RED_UNCONFIGURED');
expect(classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence({ total: 2 }), policy: POLICY }).code).toBe('BASE_RED_UNNAMED');
const setup = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence(), policy: { ...POLICY, setup: 'exit 7' } });
expect([setup.code, setup.detail, setup.cases.every(c => c.verdict === 'IN_BRANCH')]).toEqual(['BASE_RED_REPRO_ERROR', 'setup exited 7', true]);
});
test('evidence must name the lane, the total and repo-relative files', () => {
expect(parseEvidence({ lane: 'tests', total: 0, failures: [] })).toContain('total');
expect(parseEvidence({ lane: 'tests', total: 1, failures: [{ id: 'x', file: '../escape.sh', signature: 's' }] })).toContain('repo-relative');
expect(typeof parseEvidence(evidence())).toBe('object');
});
});
describe('pause, receipt and wait', () => {
test('the pause path hashes the branch; the receipt block from it is refused by default and bound only on request', () => {
fixture();
const ev = evidence();
const r = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: ev, policy: POLICY });
const { file } = writePause(work, 'feat/../../etc', r, ev, new Date('2026-10-29T00:00:00Z'));
expect(file).toBe(pausePath(work, 'feat/../../etc'));
expect(path.basename(file)).toMatch(/^[0-9a-f]{16}\.json$/);
const pause = readPause(file);
if (typeof pause === 'string') throw new Error(pause);
expect(pause.expires_at).toBe('2026-10-30T00:00:00.000Z');
const head = git('rev-parse', 'HEAD');
const body = placeReceiptFirst('## Summary\n', { schema_version: 1, pr: 7, head, base: 'main', base_red: receiptBaseRed(pause) } as any);
expect(extractReceipt(body).errors).toEqual([]);
const refused = bindReceipt(body, '7', [head]);
expect(refused.ok ? 'bound' : refused.code).toBe('QUEUE_RECEIPT_CONDITIONAL');
expect(bindReceipt(body, '7', [head], { conditional: true }).ok).toBe(true);
});
test('the terminal line reports SHIP_PAUSED with status=incomplete, and carry refuses a conditional receipt', () => {
fixture();
const ev = evidence();
fs.writeFileSync(path.join(root, 'ev.json'), JSON.stringify(ev));
const bin = (name: string, args: string[]) => spawnSync('bun', [path.join(ROOT, 'bin', name), ...args], { cwd: work, encoding: 'utf8', timeout: 120_000, env: { ...process.env, ...ID, GSTACK_STATE_ROOT: state } });
const c = bin('gstack-base-red', ['classify', '--failures', path.join(root, 'ev.json'), '--base', 'main', '--pause']);
expect(c.status).toBe(EXIT.ok);
const pauseLine = c.stdout.split('\n').find(l => l.startsWith('PAUSE: '))!;
expect(c.stdout).toContain('BASE_RED_VERDICT: base-red lane=tests');
const pause = readPause(pauseLine.slice(7));
if (typeof pause === 'string') throw new Error(pause);
const head = git('rev-parse', 'HEAD');
const bodyFile = path.join(root, 'body.md');
fs.writeFileSync(bodyFile, placeReceiptFirst('## Summary\n', { schema_version: 1, pr: 7, head, base: 'main', base_red: receiptBaseRed(pause) } as any));
const res = bin('gstack-ship-receipt', ['result', '--body', bodyFile, '--head', head, '--out', path.join(root, 'run')]);
expect(res.status).toBe(EXIT.fail);
expect(res.stdout.trim()).toBe(`GSTACK_RESULT: skill=ship status=incomplete run=${path.join(root, 'run')}`);
expect(res.stderr).toContain(`SHIP_PAUSED: base-red cases=1 base=${pause.base_sha.slice(0, 12)} lane=tests`);
expect(JSON.parse(fs.readFileSync(path.join(root, 'run', 'ship-run.json'), 'utf8')).status).toBe('incomplete');
const carry = bin('gstack-ship-receipt', ['carry', '--body', bodyFile]);
expect([carry.status, carry.stdout.split('\n')[0]]).toEqual([EXIT.refused, 'CARRY: refused (QUEUE_RECEIPT_CONDITIONAL)']);
});
test('wait skips a base move that still fails and returns on the first head that passes', () => {
fixture();
const ev = evidence();
const r = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: ev, policy: POLICY });
const pause = writePause(work, 'feat', r, ev).record;
git('checkout', '-q', 'main');
write('README.md', 'still red\n'); git('commit', '-qam', 'unrelated'); const stillRed = git('rev-parse', 'HEAD');
write('t/sum.test.sh', PASS_TEST); git('commit', '-qam', 'fix'); const green = git('rev-parse', 'HEAD');
git('push', '-q', 'origin', 'main');
const heads = [pause.base_sha, stillRed, stillRed, green];
let t = 0;
const res = waitBaseGreen({ repoRoot: work, pause, policy: POLICY, timeoutSec: 600, pollSec: 5, lsRemote: () => heads.shift() ?? green, now: () => t, sleep: ms => { t += ms; } });
expect(res).toEqual({ code: 'BASE_GREEN', sha: green, checked: 2 });
const expired = waitBaseGreen({ repoRoot: work, pause: { ...pause, expires_at: '2000-01-01T00:00:00Z' }, policy: POLICY, timeoutSec: 60, pollSec: 5, lsRemote: () => green });
expect(expired.code).toBe('BASE_RED_PAUSE_EXPIRED');
let u = 0;
const timeout = waitBaseGreen({ repoRoot: work, pause, policy: POLICY, timeoutSec: 30, pollSec: 5, lsRemote: () => pause.base_sha, now: () => u, sleep: ms => { u += ms; } });
expect([timeout.code, timeout.checked]).toEqual(['BASE_RED_WAIT_TIMEOUT', 0]);
});
});
describe('policy', () => {
test('base_red needs {files} in select, bounded timeout, no unknown keys and the introducing min_gstack_version', () => {
const codes = (base_red: unknown, min: string | null = BASE_RED_KEY_VERSION) => validatePolicy({ base_red, min_gstack_version: min }).map(e => `${e.code}@${e.path}`);
expect(codes(POLICY)).toEqual([]);
expect(codes({ ...POLICY, setup: 'bun install --frozen-lockfile' })).toEqual([]);
expect(codes({ ...POLICY, select: 'bun test' })).toEqual(['POLICY_INVALID@base_red.select']);
expect(codes({ ...POLICY, timeout_sec: 5 })).toEqual(['POLICY_INVALID@base_red.timeout_sec']);
expect(codes({ ...POLICY, extra: 1 })).toEqual(['POLICY_INVALID@base_red.extra']);
expect(codes({ ...POLICY, select: 'a {files}\nrm -rf /' })).toEqual(['POLICY_CONTAINMENT@base_red.select']);
expect(codes(POLICY, null)).toEqual(['POLICY_INVALID@base_red']);
});
});
describe('coverage audit additions', () => {
// Generated by /ship Step 7 coverage audit (PR #3611).
// Value: protects=a failure reproduced at base but not by the restricted run at the PR head stays in-branch, and one unreproduced case keeps the whole verdict in-branch;
// fails_when=the head reproduction or the every-case aggregation is dropped, so a PR that changes behaviour is paused as base-red; why_new=existing tests vary only the base run, with one case;
// seam=none; no_claim=real test runners (shell scripts stand in for the lane's tests)
test('a case the head run does not reproduce, or a case the base output never names, keeps the verdict in-branch', () => {
fixture();
write('pr-only.txt', 'x\n'); git('add', '-A'); git('commit', '-qm', 'pr: marker');
const headPasses = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence(), policy: { ...POLICY, select: `[ -f pr-only.txt ] && { echo "(pass) totals add up"; exit 0; }; ${SELECT}` } });
expect([headPasses.code, headPasses.cases[0]!.reason]).toEqual(['IN_BRANCH', 'the restricted command passes at head']);
const ghost = { id: 'ghost case', file: 't/sum.test.sh', signature: 'error: expected 3 got 4' };
const mixed = classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: evidence({ total: 2, failures: [evidence().failures[0]!, ghost] }), policy: POLICY });
expect(mixed.code).toBe('IN_BRANCH');
expect(mixed.cases.map(c => c.verdict)).toEqual(['BASE_RED', 'IN_BRANCH']);
expect(mixed.cases[1]!.reason).toStartWith('fails without naming this test at base ');
});
// Generated by /ship Step 7 coverage audit (PR #3611).
// Value: protects=an unresolvable PR head is BASE_RED_REPRO_ERROR with every case in-branch and nothing run; fails_when=the commit-resolution guard is removed or inverted;
// why_new=no existing test resolves a missing commit; seam=none; no_claim=none beyond protects
test('an unresolvable head is a reproduction error that runs nothing', () => {
fixture();
const a = counting();
const r = classifyBaseRed({ repoRoot: work, base: 'main', head: 'no-such-ref', evidence: evidence(), policy: POLICY, runner: a.runner });
expect([r.code, r.cases.every(c => c.verdict === 'IN_BRANCH'), a.calls]).toEqual(['BASE_RED_REPRO_ERROR', true, []]);
expect([r.detail, r.cases[0]!.reason]).toEqual([`cannot resolve origin/main or no-such-ref`, 'unresolved commit']);
});
// Generated by /ship Step 7 coverage audit (PR #3611).
// Value: protects=readPause rejects missing or non-pause files, and wait refuses BASE_RED_UNCONFIGURED once the base no longer declares base_red;
// fails_when=readPause accepts any JSON or wait polls without a usable base policy; why_new=existing tests read only valid pauses and never run the wait CLI;
// seam=none; no_claim=remote fetch over a network (local bare origin)
test('readPause rejects non-pause files and the wait CLI refuses when the base declares no base_red', () => {
fixture({ policy: null });
const bogus = path.join(root, 'bogus.json');
fs.writeFileSync(bogus, JSON.stringify({ kind: 'other', schema_version: 1, cases: [] }));
expect(readPause(bogus)).toContain('is not a base-red pause record');
expect(readPause(path.join(root, 'missing.json'))).toStartWith('cannot read ');
const pauseFile = path.join(root, 'pause.json');
fs.writeFileSync(pauseFile, JSON.stringify({ schema_version: 1, kind: 'base-red-pause', branch: 'feat', head: git('rev-parse', 'HEAD'), base: 'main', base_sha: git('rev-parse', 'origin/main'), lane: 'tests', cases: evidence().failures, created_at: '2026-10-29T00:00:00Z', expires_at: '2999-01-01T00:00:00Z' }));
const w = spawnSync('bun', [path.join(ROOT, 'bin', 'gstack-base-red'), 'wait', '--pause', pauseFile, '--timeout', '30', '--poll', '5'], { cwd: work, encoding: 'utf8', timeout: 60_000, env: { ...process.env, ...ID, GSTACK_STATE_ROOT: state } });
expect(w.status).toBe(EXIT.refused);
expect(w.stderr).toContain('BASE_RED_UNCONFIGURED');
});
// Generated by /ship Step 7 coverage audit (PR #3611), pass 2.
// Value: protects=write --base-red adds the base_red block from a valid pause, and an unreadable or non-pause file exits usage with nothing written;
// fails_when=the flag is ignored, so an unconditional receipt is written for a paused run; why_new=no test runs write with --base-red;
// seam=none; no_claim=PR body upload via gh (local body file only)
test('write --base-red makes the receipt conditional, and a bad pause file writes nothing', () => {
fixture();
const ev = evidence();
const { file, record } = writePause(work, 'feat', classifyBaseRed({ repoRoot: work, base: 'main', head: 'HEAD', evidence: ev, policy: POLICY }), ev);
const body = path.join(root, 'body.md');
const run = (pause: string) => spawnSync('bun', [path.join(ROOT, 'bin', 'gstack-ship-receipt'), 'write', '--pr', '7', '--base', 'main', '--session-kind', 'x', '--body', body, '--base-red', pause], { cwd: work, encoding: 'utf8', timeout: 120_000, env: { ...process.env, ...ID, GSTACK_STATE_ROOT: state } });
for (const bad of [path.join(root, 'missing.json'), path.join(root, 'ev.json')]) {
fs.writeFileSync(path.join(root, 'ev.json'), JSON.stringify(ev));
fs.writeFileSync(body, 'prose\n');
const r = run(bad);
expect([r.status, r.stderr.includes('--base-red: '), r.stderr.includes('nothing written'), fs.readFileSync(body, 'utf8')]).toEqual([EXIT.usage, true, true, 'prose\n']);
}
const ok = run(file);
expect(ok.status).toBe(EXIT.ok);
const head = git('rev-parse', 'HEAD');
const written = fs.readFileSync(body, 'utf8');
expect(extractReceipt(written).receipt?.base_red).toEqual(receiptBaseRed(record));
const bound = bindReceipt(written, '7', [head]);
expect(bound.ok ? 'bound' : bound.code).toBe('QUEUE_RECEIPT_CONDITIONAL');
});
// Generated by /ship Step 7 coverage audit (PR #3611), pass 2.
// Value: protects=the SHIP_RECEIPT summary line names base_red=<base12> for a conditional receipt and omits it otherwise; fails_when=the base_red part is dropped or printed unconditionally;
// why_new=summary-line tests cover carried_from only; seam=none; no_claim=none beyond protects
test('the summary line names the red base only for a conditional receipt', () => {
const base = 'c'.repeat(40);
const plain = { schema_version: 1, head: 'a'.repeat(40), base: 'main' } as any;
expect(receiptSummaryLine({ ...plain, base_red: { cases: ['x'], files: ['t/x.sh'], base, lane: 'tests', expires_at: '2026-10-30T00:00:00Z' } })).toContain(` base_red=${base.slice(0, 12)}`);
expect(receiptSummaryLine(plain)).not.toContain('base_red=');
});
// Generated by /ship Step 7 coverage audit (PR #3611), pass 2.
// Value: protects=classify exits usage on a missing --failures, an unreadable file or invalid evidence before any git fetch; fails_when=evidence checks move after the policy fetch or stop refusing;
// why_new=parseEvidence is unit-tested only, never through the CLI; seam=none; no_claim=none beyond protects (GIT_TRACE proves no fetch ran)
test('classify refuses bad evidence with a usage exit before fetching anything', () => {
git('commit', '-q', '--allow-empty', '-m', 'init');
const trace = path.join(root, 'git-trace.log');
const bad = path.join(root, 'bad.json');
fs.writeFileSync(bad, JSON.stringify({ lane: 'tests', total: 1, failures: [{ id: 'x', file: '/abs.sh', signature: 's' }] }));
const cases: Array<[string[], string]> = [[[], '--failures <evidence.json> is required'], [['--failures', path.join(root, 'missing.json')], 'cannot read '], [['--failures', bad], 'must be repo-relative']];
for (const [args, why] of cases) {
const r = spawnSync('bun', [path.join(ROOT, 'bin', 'gstack-base-red'), 'classify', ...args], { cwd: work, encoding: 'utf8', timeout: 60_000, env: { ...process.env, ...ID, GSTACK_STATE_ROOT: state, GIT_TRACE: trace } });
expect([r.status, r.stderr.includes(why)]).toEqual([EXIT.usage, true]);
}
const traced = fs.readFileSync(trace, 'utf8');
expect([traced.includes('rev-parse'), traced.includes('fetch')]).toEqual([true, false]);
});
});
test/egress-receipt-wiring.test.ts
+3
View File
@@ -98,6 +98,9 @@ const MODULE_SINKS = [
// repository permission (`gh api .../collaborators/<user>/permission`); the
// author login and repo slug leave the machine, sink 'contributor-mode'.
'lib/contributor-mode.ts',
// GSTA-156: gstack-pregate clear --run reads the repository's own Actions runs
// through gh and fetches a missing run head from origin by sha; sink 'run-evidence'.
'lib/run-evidence.ts',
// The Memorable bridge hook: gstack-owned code that hands each prompt to a
// vendor CLI. hosts/ has no curl/fetch for the scanner to see, so the
// receipt wiring is pinned here explicitly.
test/eval-models.test.ts
+13
View File
@@ -142,6 +142,19 @@ describe('gstack-eval-plan lints every arm (bin, fixture repo with an origin)',
expect(head.err).toContain('(EVAL_MODEL_GENERATION_UNRESOLVED)');
});
test('a <model>#<loadout> arm is linted as its model: a deny still refuses it and an exact allow accepts it', () => {
seed(GBRAIN_RULES);
const r = run(['arms', '--arms', 'claude-opus-5-5#no-gstack,gpt-5.4-mini#no-gstack']);
expect(r.code).toBe(1);
expect(r.out).toContain('EVAL_MODEL: claude-opus-5-5 scope=full ok (allow claude-*; newest in families.claude-opus) where=flag:--arms (claude-opus-5-5#no-gstack)');
expect(r.out).toContain('EVAL_MODEL: gpt-5.4-mini scope=full denied (deny gpt-5.4-mini) where=flag:--arms (gpt-5.4-mini#no-gstack)');
expect(r.err).toContain('(EVAL_MODEL_DENIED)');
fs.writeFileSync(path.join(repo, '.gstack', 'eval-models.json'), JSON.stringify({ schema_version: 1, allow: ['claude-opus-5-5'], deny: [] }));
const exact = run(['arms', '--arms', 'claude-opus-5-5,claude-opus-5-5#no-gstack', '--models-from', 'head']);
expect(exact.code).toBe(0);
expect(exact.out).toContain('where=flag:--arms (claude-opus-5-5#no-gstack)');
});
test('dry-run and pilot start: Fable fails the full scope (no reservation), passes --scope smoke; the dry-run record carries the check', () => {
seed(GBRAIN_RULES);
const out = path.join(host.base, 'out');
test/eval-plan-no-gstack-arm.test.ts
+199
View File
@@ -0,0 +1,199 @@
/**
* The no-gstack arm (Garry, 2026-10-31: "all of our head to head tests have to
* test no gstack from here"). A preregistration says in `## Comparison` whether
* any arm loads gstack or another skill stack (`stack: loaded`) or none does
* (`stack: none` with a `why:`). A loaded comparison needs a `## No-gstack arm`
* section naming one `<model>#no-gstack` arm per model, each also an Arms line,
* plus the inputs, prompt, panel and report it shares with the variants. Arms are
* identities `<model>[#<loadout>]`, priced by the model, so the no-gstack arm on
* the same model gets its own pilot reservations.
*/
import { afterEach, describe, expect, test } from 'bun:test';
import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { armModel, checkPrereg, dryRun, preregTemplate, resolveArms, startPilot } from '../lib/eval-plan';
const BIN = path.resolve(import.meta.dir, '..', 'bin', 'gstack-eval-plan');
const dirs: string[] = [];
const tmp = () => { const d = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-no-gstack-arm-')); dirs.push(d); return d; };
afterEach(() => { while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, force: true }); });
const BASE = ['Bars', 'Decoys', 'Stop rules', 'Held-out exposure'].map(s => `## ${s}\n- filled in\n`).join('\n');
const ARMS = (...ids: string[]) => `\n## Arms\n${ids.map(id => `- ${id} default settings, $0.01 per item`).join('\n')}\n`;
const LOADED = '\n## Comparison\nstack: loaded\n';
const NO_GSTACK = (o: Partial<Record<'arms' | 'inputs' | 'prompt' | 'panel' | 'report', string>> = {}) => {
const v = { arms: 'claude-opus-5-5#no-gstack', inputs: 'the same rows and fixture as the variants', prompt: 'the task text only; no SKILL.md, preamble, hooks or installed skills', panel: 'behavior, 3 trials, the same judge and labels', report: 'one row beside the variants in the results table', ...o };
return `\n## No-gstack arm\n${Object.entries(v).filter(([, x]) => x !== undefined).map(([k, x]) => `${k}: ${x}`).join('\n')}\n`;
};
const VARIANTS = ARMS('claude-opus-5-5#review-g', 'claude-opus-5-5#review-h', 'claude-opus-5-5#no-gstack');
describe('the Comparison declaration', () => {
test('Comparison is a required section; the blank template fails and multi-line guidance never counts as content', () => {
const t = preregTemplate('x');
expect(t).toContain('## Comparison');
expect(t).toContain('## No-gstack arm');
const blank = checkPrereg(t);
expect(blank.empty).toContain('Comparison');
expect(checkPrereg(BASE + VARIANTS + '\n## Comparison\n<!-- stack: loaded\n more guidance -->\n').empty).toContain('Comparison');
});
test('stack: none needs a why: line and passes without the arm; an unknown value fails', () => {
const none = checkPrereg(BASE + ARMS('claude-opus-5-5') + '\n## Comparison\nstack: none\nwhy: scorer calibration; no arm loads a skill stack\n');
expect(none.ok).toBe(true);
expect(none.comparison.stack).toBe('none');
expect(none.no_gstack.required).toBe(false);
const noWhy = checkPrereg(BASE + ARMS('claude-opus-5-5') + '\n## Comparison\nstack: none\n');
expect(noWhy.ok).toBe(false);
expect(noWhy.comparison.errors.join(';')).toContain('why:');
const odd = checkPrereg(BASE + ARMS('claude-opus-5-5') + '\n## Comparison\nstack: maybe\n');
expect(odd.ok).toBe(false);
expect(odd.comparison.errors.join(';')).toContain('stack: maybe');
});
});
describe('the No-gstack arm section', () => {
test('a loaded comparison passes with a complete section and fails without it', () => {
expect(checkPrereg(BASE + VARIANTS + LOADED + NO_GSTACK()).ok).toBe(true);
const missing = checkPrereg(BASE + VARIANTS + LOADED);
expect(missing.ok).toBe(false);
expect(missing.no_gstack).toMatchObject({ required: true, ok: false });
expect(missing.no_gstack.problems).toEqual(['missing section']);
});
for (const key of ['arms', 'inputs', 'prompt', 'panel', 'report'] as const) {
test(`a missing or blank ${key}: line fails naming it`, () => {
for (const v of [undefined, ' ']) {
const r = checkPrereg(BASE + VARIANTS + LOADED + NO_GSTACK({ [key]: v } as Record<string, string>));
expect(r.ok).toBe(false);
expect(r.no_gstack.problems).toContain(`missing ${key}`);
}
});
}
test('every model that loads a stack needs its own <model>#no-gstack arm: a two-model panel needs two', () => {
const panel = ARMS('claude-opus-5-5', 'gpt-6.1-sol', 'claude-opus-5-5#no-gstack');
const one = checkPrereg(BASE + panel + LOADED + NO_GSTACK());
expect(one.ok).toBe(false);
expect(one.no_gstack.problems).toEqual(['no no-gstack arm for gpt-6.1-sol']);
const both = checkPrereg(BASE + ARMS('claude-opus-5-5', 'gpt-6.1-sol', 'claude-opus-5-5#no-gstack', 'gpt-6.1-sol#no-gstack') + LOADED + NO_GSTACK({ arms: 'claude-opus-5-5#no-gstack, gpt-6.1-sol#no-gstack' }));
expect(both.ok).toBe(true);
});
test('a model panel under another skill stack (not gstack) is loaded too and needs the same arms', () => {
const other = ARMS('claude-opus-5-5#other-stack', 'gpt-6.1-sol#other-stack');
const r = checkPrereg(BASE + other + LOADED + NO_GSTACK({ arms: 'claude-opus-5-5#no-gstack' }));
expect(r.ok).toBe(false);
expect(r.no_gstack.problems).toEqual(['claude-opus-5-5#no-gstack is not an Arms line', 'no no-gstack arm for claude-opus-5-5', 'no no-gstack arm for gpt-6.1-sol']);
});
test('a listed arm must be an Arms line and a <model>#no-gstack identity, matched on whole identities', () => {
const notListed = checkPrereg(BASE + ARMS('claude-opus-5-5#review-g') + LOADED + NO_GSTACK());
expect(notListed.no_gstack.problems).toContain('claude-opus-5-5#no-gstack is not an Arms line');
const prefix = checkPrereg(BASE + ARMS('claude-opus-5-5#review-g', 'claude-opus-5#no-gstack') + LOADED + NO_GSTACK({ arms: 'claude-opus-5#no-gstack' }));
expect(prefix.no_gstack.problems).toContain('no no-gstack arm for claude-opus-5-5');
const wrongLoadout = checkPrereg(BASE + ARMS('claude-opus-5-5#review-g', 'claude-opus-5-5#plain') + LOADED + NO_GSTACK({ arms: 'claude-opus-5-5#plain' }));
expect(wrongLoadout.no_gstack.problems).toContain('claude-opus-5-5#plain is not a <model>#no-gstack identity');
const prose = checkPrereg(BASE + '\n## Arms\n- claude-opus-5-5#review-g, compared with claude-opus-5-5#no-gstack in prose\n' + LOADED + NO_GSTACK());
expect(prose.no_gstack.problems).toContain('claude-opus-5-5#no-gstack is not an Arms line');
});
test('a table in Arms: the header row is not an arm', () => {
const table = '\n## Arms\n| model | settings |\n|---|---|\n| claude-opus-5-5#review-g | default |\n| claude-opus-5-5#no-gstack | default |\n';
expect(checkPrereg(BASE + table + LOADED + NO_GSTACK()).ok).toBe(true);
});
});
describe('arm identities in pricing and the pilot', () => {
test('armModel strips the loadout; arms and the dry run price <model>#no-gstack by its model', () => {
expect(armModel('anthropic/claude-opus-5-5#no-gstack')).toBe('anthropic/claude-opus-5-5');
expect(armModel('gpt-6.1-sol')).toBe('gpt-6.1-sol');
const d = tmp();
const arms = resolveArms({ cwd: d, explicit: ['claude-opus-5-5#no-gstack'] });
expect(arms.errors).toEqual([]);
expect(arms.arms[0]).toMatchObject({ model: 'claude-opus-5-5#no-gstack', priced: true });
const rows = path.join(d, 'rows.jsonl');
fs.writeFileSync(rows, JSON.stringify({ prompt: 'hello there' }) + '\n');
const dry = dryRun({ input: rows, arms: ['claude-opus-5-5', 'claude-opus-5-5#no-gstack'] });
expect(dry.checks.price_table.ok).toBe(true);
expect(dry.per_item_usd['claude-opus-5-5#no-gstack']).toBe(dry.per_item_usd['claude-opus-5-5']);
});
const passingDryRun = (d: string) => {
const rows = path.join(d, 'rows.jsonl');
fs.writeFileSync(rows, JSON.stringify({ prompt: 'hello there' }) + '\n');
const file = path.join(d, 'dry-run.json');
const r = dryRun({ input: rows, arms: ['claude-opus-5-5'], retryPrompt: (fs.writeFileSync(path.join(d, 'retry.md'), 'try again\n'), path.join(d, 'retry.md')) });
fs.writeFileSync(file, JSON.stringify(r));
expect(r.pass, JSON.stringify(r.checks)).toBe(true);
return file;
};
test('the pilot reserves a variant and its no-gstack arm on the same model under distinct attempts; the total is the sum', () => {
const d = tmp();
const r = startPilot({ out: path.join(d, 'pilot'), cap: 100, n: 2, arms: ['claude-opus-5-5', 'claude-opus-5-5#no-gstack'], dryRun: passingDryRun(d) });
expect(r.error).toBeUndefined();
const ids = r.record.attempts.map(a => a.attempt);
expect(new Set(ids).size).toBe(4);
const ledger = JSON.parse(fs.readFileSync(path.join(d, 'pilot', fs.readdirSync(path.join(d, 'pilot')).find(f => f.includes('spend'))!), 'utf8'));
const reserved = JSON.stringify(ledger);
for (const id of ids) expect(reserved).toContain(id);
const each = r.record.attempts[0]!.estimate_usd;
expect(r.record.attempts.every(a => a.estimate_usd === each)).toBe(true);
});
test('a duplicate arm identity is refused before anything is reserved (EVAL_ARM_DUPLICATE)', () => {
const d = tmp();
const r = startPilot({ out: path.join(d, 'pilot'), cap: 100, n: 1, arms: ['claude-opus-5-5', 'claude-opus-5-5'], dryRun: passingDryRun(d) });
expect(r.error?.code).toBe('EVAL_ARM_DUPLICATE');
expect(r.record.attempts).toEqual([]);
});
test('distinct identities that share one ledger attempt name are refused before anything is reserved', () => {
for (const arms of [['claude-opus-5-5#review/a', 'claude-opus-5-5#review_a'], ['gpt-6#1-sol', 'gpt-6.1-sol']]) {
const d = tmp();
const r = startPilot({ out: path.join(d, 'pilot'), cap: 100, n: 1, arms, dryRun: passingDryRun(d) });
expect(r.error?.code).toBe('EVAL_ARM_DUPLICATE');
expect(r.error?.message).toContain(arms.join(' = '));
expect(r.record.attempts).toEqual([]);
expect(fs.existsSync(path.join(d, 'pilot'))).toBe(false);
}
});
});
describe('gstack-eval-plan prereg --check', () => {
const run = (text: string) => {
const d = tmp();
const file = path.join(d, 'prereg.md');
fs.writeFileSync(file, text);
const r = spawnSync(BIN, ['prereg', '--check', file], { cwd: d, encoding: 'utf8', timeout: 60_000 });
return { code: r.status, out: r.stdout ?? '', err: r.stderr ?? '' };
};
test('a loaded comparison without the arm: one PREREG line out of 6, PREREG_NO_GSTACK_ARM on stderr, exit 1', () => {
const r = run(BASE + VARIANTS + LOADED);
expect(r.code).toBe(1);
expect(r.out).toMatch(/^PREREG: incomplete present=6\/6 stack=loaded no_gstack_arm=missing file=/m);
expect(r.err).toContain('missing section');
expect(r.err).toContain('(PREREG_NO_GSTACK_ARM)');
expect(r.err).not.toContain('(PREREG_INCOMPLETE)');
});
test('missing sections and arm problems both print, PREREG_INCOMPLETE first', () => {
const r = run(VARIANTS + LOADED);
expect(r.code).toBe(1);
const a = r.err.indexOf('(PREREG_INCOMPLETE)'), b = r.err.indexOf('(PREREG_NO_GSTACK_ARM)');
expect(a).toBeGreaterThanOrEqual(0);
expect(b).toBeGreaterThan(a);
});
test('complete: PREREG: ok with stack=loaded no_gstack_arm=ok; an opt-out prints stack=none', () => {
const ok = run(BASE + VARIANTS + LOADED + NO_GSTACK());
expect(ok.code, ok.out + ok.err).toBe(0);
expect(ok.out).toMatch(/^PREREG: ok present=6\/6 stack=loaded no_gstack_arm=ok file=/m);
const none = run(BASE + ARMS('claude-opus-5-5') + '\n## Comparison\nstack: none\nwhy: scorer calibration\n');
expect(none.code, none.out + none.err).toBe(0);
expect(none.out).toMatch(/^PREREG: ok present=6\/6 stack=none file=/m);
});
});
test/eval-plan-prereg-diagnostics.test.ts
+4 -4
View File
@@ -14,7 +14,7 @@ import { checkPrereg, parsePreregDiagnostics, preregTemplate } from '../lib/eval
import { capyHost, type CapyHost } from './helpers/capy-host';
const BIN = path.resolve(import.meta.dir, '..', 'bin', 'gstack-eval-plan');
const FILLED = ['Bars', 'Arms', 'Decoys', 'Stop rules', 'Held-out exposure'].map(s => `## ${s}\n- filled in\n`).join('\n');
const FILLED = ['Bars', 'Arms', 'Decoys', 'Stop rules', 'Held-out exposure'].map(s => `## ${s}\n- filled in\n`).join('\n') + '\n## Comparison\nstack: none\nwhy: one arm, no skill stack\n';
const SUMMARIZER = [
'touch ran.marker',
'r=$(grep -c \'"kind":"refusal"\' "$1")',
@@ -54,7 +54,7 @@ describe('prereg --check: the Diagnostics section', () => {
const r = check(FILLED + diagnostics([['refusals', '2'], ['truncated', '1']]));
expect(r.code).toBe(3);
expect(r.ran).toBe(false);
expect(r.out).toContain('PREREG: ok present=5/5 diagnostics=2 file=');
expect(r.out).toContain('PREREG: ok present=6/6 stack=none diagnostics=2 file=');
expect(r.out).toContain(`PREREG_COMMAND: summarizer="sh summarize.sh {fixture}" fixture=fixtures/synthetic records.jsonl cwd=${fs.realpathSync(repo)}`);
expect(r.err).toContain('(REPO_COMMANDS_NOT_ALLOWED)');
expect(r.out).not.toContain('PREREG_DIAGNOSTICS:');
@@ -104,7 +104,7 @@ describe('prereg --check: the Diagnostics section', () => {
const r = check(FILLED + diagnostics([['refusals', 'two']], { summarizer: 'sh summarize.sh fixtures/x.jsonl' }), '--allow-repo-commands');
expect(r.code).toBe(1);
expect(r.ran).toBe(false);
expect(r.out).toContain('PREREG: incomplete present=5/5 diagnostics=invalid');
expect(r.out).toContain('PREREG: incomplete present=6/6 stack=none diagnostics=invalid');
expect(r.err).toContain('diagnostics: row "| refusals | two |"');
expect(r.err).toContain('diagnostics: the summarizer command never names the fixture ({fixture})');
expect(r.err).toContain('diagnostics: no | diagnostic | expected | rows');
@@ -114,7 +114,7 @@ describe('prereg --check: the Diagnostics section', () => {
test('without a Diagnostics section the check is unchanged; the template mentions the section without adding a heading', () => {
const r = check(FILLED);
expect(r.code).toBe(0);
expect(r.out).toBe(`PREREG: ok present=5/5 file=${path.join(repo, 'evals', 'prereg.md')}\n`);
expect(r.out).toBe(`PREREG: ok present=6/6 stack=none file=${path.join(repo, 'evals', 'prereg.md')}\n`);
expect(parsePreregDiagnostics(FILLED)).toBeNull();
expect(parsePreregDiagnostics(FILLED + '\n## Diagnostics\n<!-- none -->\n')).toBeNull();
expect(preregTemplate()).toContain('## Diagnostics');
test/fixtures/context-budget.json
Vendored
+1 -1
View File
@@ -9,7 +9,7 @@
"browser-skills/hackernews-frontpage": 371,
"canary": 7738,
"captain": 6551,
"capy/skill/gstack": 5136,
"capy/skill/gstack": 5312,
"careful": 1239,
"codex": 9890,
"context-restore": 6287,
test/fixtures/junit/bun.xml
Vendored
+17
View File
@@ -0,0 +1,17 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="bun test" tests="4" assertions="2" failures="1" skipped="1" time="0.02640424">
<testsuite name="a.test.ts" file="a.test.ts" tests="4" assertions="2" failures="1" skipped="1" time="0.014448064" hostname="ci-host">
<testsuite name="outer" file="a.test.ts" line="2" tests="3" assertions="2" failures="1" skipped="1" time="0" hostname="ci-host">
<testsuite name="inner" file="a.test.ts" line="2" tests="2" assertions="2" failures="1" skipped="0" time="0.006" hostname="ci-host">
<testcase name="passes" classname="inner &gt; outer" time="0.000069" file="a.test.ts" line="2" assertions="1" />
<testcase name="fails" classname="inner &gt; outer" time="0.006322" file="a.test.ts" line="2" assertions="1">
<failure type="AssertionError" message="expect(received).toBe(expected)&#10;&#10;Expected: 2&#10;Received: 1&#10;">AssertionError: expect(received).toBe(expected)&#10;&#10;Expected: 2&#10;Received: 1&#10;&#10; at a.test.ts:2:131&#10;</failure>
</testcase>
</testsuite>
<testcase name="skipped" classname="outer" time="0" file="a.test.ts" line="2" assertions="0">
<skipped />
</testcase>
</testsuite>
<testcase name="top level" classname="" time="0.000039" file="a.test.ts" line="3" assertions="0" />
</testsuite>
</testsuites>
test/fixtures/junit/playwright-r0.xml
Vendored
+138
View File
@@ -0,0 +1,138 @@
<testsuites id="" name="" tests="10" failures="4" skipped="2" errors="0" time="3.23467">
<testsuite name="a.spec.ts" timestamp="2026-10-29T16:02:06.214Z" hostname="alpha" tests="5" failures="2" skipped="1" time="0.03" errors="0">
<testcase name="group one › passes" classname="a.spec.ts" time="0.011">
</testcase>
<testcase name="group one › fails always" classname="a.spec.ts" time="0.004">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-alpha/error-context.md]]
]]>
</system-out>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [alpha] › a.spec.ts:5:7 › group one › fails always ───────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-alpha/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › flaky once" classname="a.spec.ts" time="0.009">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-flaky-once-alpha/error-context.md]]
]]>
</system-out>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [alpha] › a.spec.ts:6:7 › group one › flaky once ─────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: "second"
Received: "first"
4 | test('passes', () => { expect(1).toBe(1); });
5 | test('fails always', () => { expect(1).toBe(2); });
> 6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
| ^
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
9 | });
at /repo/tests/a.spec.ts:6:152
Error Context: ../test-results/a-group-one-flaky-once-alpha/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › skipped" classname="a.spec.ts">
<properties>
<property name="skip" value="">
</property>
</properties>
<skipped>
</skipped>
</testcase>
<testcase name="group one › title with $(rm -rf) [x] (y)? *" classname="a.spec.ts" time="0.006">
</testcase>
</testsuite>
<testsuite name="a.spec.ts" timestamp="2026-10-29T16:02:06.214Z" hostname="beta" tests="5" failures="2" skipped="1" time="0.027" errors="0">
<testcase name="group one › passes" classname="a.spec.ts" time="0.005">
</testcase>
<testcase name="group one › fails always" classname="a.spec.ts" time="0.002">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-beta/error-context.md]]
]]>
</system-out>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [beta] › a.spec.ts:5:7 › group one › fails always ────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-beta/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › flaky once" classname="a.spec.ts" time="0.013">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-flaky-once-beta/error-context.md]]
]]>
</system-out>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [beta] › a.spec.ts:6:7 › group one › flaky once ──────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: "second"
Received: "first"
4 | test('passes', () => { expect(1).toBe(1); });
5 | test('fails always', () => { expect(1).toBe(2); });
> 6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
| ^
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
9 | });
at /repo/tests/a.spec.ts:6:152
Error Context: ../test-results/a-group-one-flaky-once-beta/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › skipped" classname="a.spec.ts">
<properties>
<property name="skip" value="">
</property>
</properties>
<skipped>
</skipped>
</testcase>
<testcase name="group one › title with $(rm -rf) [x] (y)? *" classname="a.spec.ts" time="0.007">
</testcase>
</testsuite>
</testsuites>
test/fixtures/junit/playwright-r2-inc.xml
Vendored
+244
View File
@@ -0,0 +1,244 @@
<testsuites id="" name="" tests="10" failures="2" skipped="2" errors="0" time="4.805343">
<testsuite name="a.spec.ts" timestamp="2026-10-29T16:02:16.450Z" hostname="alpha" tests="5" failures="1" skipped="1" time="0.043" errors="0">
<testcase name="group one › passes" classname="a.spec.ts" time="0.007">
</testcase>
<testcase name="group one › fails always" classname="a.spec.ts" time="0.005">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-alpha/error-context.md]]
]]>
</system-out>
<rerunFailure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe" time="0.008">
<stackTrace>
<![CDATA[Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
at /repo/tests/a.spec.ts:5:42]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-alpha-retry1/error-context.md]]
]]>
</system-out>
</rerunFailure>
<rerunFailure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe" time="0.009">
<stackTrace>
<![CDATA[Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
at /repo/tests/a.spec.ts:5:42]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-alpha-retry2/error-context.md]]
]]>
</system-out>
</rerunFailure>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [alpha] › a.spec.ts:5:7 › group one › fails always ───────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-alpha/error-context.md
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-alpha-retry1/error-context.md
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-alpha-retry2/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › flaky once" classname="a.spec.ts" time="0.002">
<flakyFailure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe" time="0.009">
<stackTrace>
<![CDATA[Error: expect(received).toBe(expected) // Object.is equality
Expected: "second"
Received: "first"
at /repo/tests/a.spec.ts:6:152]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-flaky-once-alpha/error-context.md]]
]]>
</system-out>
</flakyFailure>
</testcase>
<testcase name="group one › skipped" classname="a.spec.ts">
<properties>
<property name="skip" value="">
</property>
</properties>
<skipped>
</skipped>
</testcase>
<testcase name="group one › title with $(rm -rf) [x] (y)? *" classname="a.spec.ts" time="0.003">
</testcase>
</testsuite>
<testsuite name="a.spec.ts" timestamp="2026-10-29T16:02:16.450Z" hostname="beta" tests="5" failures="1" skipped="1" time="0.033" errors="0">
<testcase name="group one › passes" classname="a.spec.ts" time="0.004">
</testcase>
<testcase name="group one › fails always" classname="a.spec.ts" time="0.002">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-beta/error-context.md]]
]]>
</system-out>
<rerunFailure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe" time="0.006">
<stackTrace>
<![CDATA[Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
at /repo/tests/a.spec.ts:5:42]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-beta-retry1/error-context.md]]
]]>
</system-out>
</rerunFailure>
<rerunFailure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe" time="0.006">
<stackTrace>
<![CDATA[Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
at /repo/tests/a.spec.ts:5:42]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-beta-retry2/error-context.md]]
]]>
</system-out>
</rerunFailure>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [beta] › a.spec.ts:5:7 › group one › fails always ────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-beta/error-context.md
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-beta-retry1/error-context.md
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-beta-retry2/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › flaky once" classname="a.spec.ts" time="0.002">
<flakyFailure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe" time="0.01">
<stackTrace>
<![CDATA[Error: expect(received).toBe(expected) // Object.is equality
Expected: "second"
Received: "first"
at /repo/tests/a.spec.ts:6:152]]>
</stackTrace>
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-flaky-once-beta/error-context.md]]
]]>
</system-out>
</flakyFailure>
</testcase>
<testcase name="group one › skipped" classname="a.spec.ts">
<properties>
<property name="skip" value="">
</property>
</properties>
<skipped>
</skipped>
</testcase>
<testcase name="group one › title with $(rm -rf) [x] (y)? *" classname="a.spec.ts" time="0.003">
</testcase>
</testsuite>
</testsuites>
test/fixtures/junit/playwright-r2.xml
Vendored
+178
View File
@@ -0,0 +1,178 @@
<testsuites id="" name="" tests="10" failures="2" skipped="2" errors="0" time="4.985760999999999">
<testsuite name="a.spec.ts" timestamp="2026-10-29T16:02:10.182Z" hostname="alpha" tests="5" failures="1" skipped="1" time="0.044" errors="0">
<testcase name="group one › passes" classname="a.spec.ts" time="0.005">
</testcase>
<testcase name="group one › fails always" classname="a.spec.ts" time="0.021">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-alpha/error-context.md]]
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-alpha-retry1/error-context.md]]
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-alpha-retry2/error-context.md]]
]]>
</system-out>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [alpha] › a.spec.ts:5:7 › group one › fails always ───────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-alpha/error-context.md
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-alpha-retry1/error-context.md
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-alpha-retry2/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › flaky once" classname="a.spec.ts" time="0.014">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-flaky-once-alpha/error-context.md]]
]]>
</system-out>
</testcase>
<testcase name="group one › skipped" classname="a.spec.ts">
<properties>
<property name="skip" value="">
</property>
</properties>
<skipped>
</skipped>
</testcase>
<testcase name="group one › title with $(rm -rf) [x] (y)? *" classname="a.spec.ts" time="0.004">
</testcase>
</testsuite>
<testsuite name="a.spec.ts" timestamp="2026-10-29T16:02:10.182Z" hostname="beta" tests="5" failures="1" skipped="1" time="0.04" errors="0">
<testcase name="group one › passes" classname="a.spec.ts" time="0.005">
</testcase>
<testcase name="group one › fails always" classname="a.spec.ts" time="0.019">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-beta/error-context.md]]
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-beta-retry1/error-context.md]]
[[ATTACHMENT|pw/test-results/a-group-one-fails-always-beta-retry2/error-context.md]]
]]>
</system-out>
<failure message="expect(received).toBe(expected) // Object.is equality" type="expect.toBe">
<![CDATA[ [beta] › a.spec.ts:5:7 › group one › fails always ────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-beta/error-context.md
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-beta-retry1/error-context.md
Retry #2 ───────────────────────────────────────────────────────────────────────────────────────
Error: expect(received).toBe(expected) // Object.is equality
Expected: 2
Received: 1
3 | test.describe('group one', () => {
4 | test('passes', () => { expect(1).toBe(1); });
> 5 | test('fails always', () => { expect(1).toBe(2); });
| ^
6 | test('flaky once', ({}, info) => { const f = '/tmp/pwflake-' + info.project.name; if (!fs.existsSync(f)) { fs.writeFileSync(f, '1'); expect('first').toBe('second'); } });
7 | test.skip('skipped', () => {});
8 | test('title with $(rm -rf) [x] (y)? *', () => { expect(1).toBe(1); });
at /repo/tests/a.spec.ts:5:42
Error Context: ../test-results/a-group-one-fails-always-beta-retry2/error-context.md
]]>
</failure>
</testcase>
<testcase name="group one › flaky once" classname="a.spec.ts" time="0.012">
<system-out>
<![CDATA[
[[ATTACHMENT|pw/test-results/a-group-one-flaky-once-beta/error-context.md]]
]]>
</system-out>
</testcase>
<testcase name="group one › skipped" classname="a.spec.ts">
<properties>
<property name="skip" value="">
</property>
</properties>
<skipped>
</skipped>
</testcase>
<testcase name="group one › title with $(rm -rf) [x] (y)? *" classname="a.spec.ts" time="0.004">
</testcase>
</testsuite>
</testsuites>
test/fixtures/module-size-ratchet.json
Vendored
+1
View File
@@ -101,6 +101,7 @@
{ "file": "lib/result-codes-release.ts", "movedFrom": "lib/result-codes.ts (the release-path rows: ship queue, pregate timing, approval classes, state matrix; field list 2 wave B)" },
{ "file": "lib/result-codes-capy.ts", "movedFrom": "lib/result-codes.ts (the Capy rows: approval record, Captain brief, PR screenshots; GSTA-61)" },
{ "file": "lib/result-codes-budget.ts", "movedFrom": "lib/result-codes.ts (the /ship token budget and handoff rows; GSTA-152)" },
{ "file": "lib/result-codes-eval.ts", "movedFrom": "lib/result-codes.ts (the /eval-plan preregistration rows and the no-gstack arm; GSTA-168)" },
{ "file": "lib/ship-budget.ts", "movedFrom": "(new: /ship token budget ledger, meters and admission, GSTA-152)" },
{ "file": "lib/ship-budget-cli.ts", "movedFrom": "(new: bin/gstack-ship-budget start/check/settle/close/resume, GSTA-152)" },
{ "file": "lib/ship-handoff.ts", "movedFrom": "(new: /ship fresh-context handoff, verify, return and reconcile, GSTA-152)" },
test/flaky-run.test.ts
+459
View File
@@ -0,0 +1,459 @@
/**
* gstack-flaky run (GSTA-158): the JUnit reader against reports captured from
* Playwright 1.64 and bun 1.4 (test/fixtures/junit, paths rewritten to /repo),
* the rerun-once rules, and the bin end to end with a fake runner.
*/
import { afterAll, describe, expect, test } from 'bun:test';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { spawn, spawnSync } from 'child_process';
import { JunitError, parseJunit, readJunit, testId } from '../lib/junit';
import { appendLedger, incompleteRun, ledgerPath, renderResult, rerunSelection, runFlaky, safeExcerpt, stepSummary, unreportedFailures, workflowEscape } from '../lib/flaky-run';
const ROOT = path.resolve(import.meta.dir, '..');
const FIX = path.join(ROOT, 'test', 'fixtures', 'junit');
const fixture = (name: string) => fs.readFileSync(path.join(FIX, name), 'utf-8');
const TMP = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-flaky-'));
const tmp = () => fs.mkdtempSync(path.join(TMP, 't-'));
afterAll(() => fs.rmSync(TMP, { recursive: true, force: true }));
describe('JUnit reader', () => {
test('bun: nested describe path, skip, failure excerpt', () => {
const r = parseJunit(fixture('bun.xml'));
expect(r.runner).toBe('bun');
expect(r.tests.map(t => [t.id, t.attempts.join(',')])).toEqual([
['a.test.ts::::outer > inner > passes', 'pass'],
['a.test.ts::::outer > inner > fails', 'fail'],
['a.test.ts::::outer > skipped', 'skip'],
['a.test.ts::::top level', 'pass'],
]);
expect(r.tests[1]!.excerpt[0]).toContain('expect(received).toBe(expected)');
});
test('playwright: one id per project, retry-aware attempts', () => {
const r = parseJunit(fixture('playwright-r2-inc.xml'));
expect(r.runner).toBe('playwright');
expect(r.retriesHidden).toBe(false);
const byId = Object.fromEntries(r.tests.map(t => [t.id, t.attempts.join(',')]));
expect(byId['a.spec.ts::alpha::group one > fails always']).toBe('fail,fail,fail');
expect(byId['a.spec.ts::beta::group one > fails always']).toBe('fail,fail,fail');
expect(byId['a.spec.ts::alpha::group one > flaky once']).toBe('fail,pass');
expect(byId['a.spec.ts::alpha::group one > skipped']).toBe('skip');
expect(byId['a.spec.ts::alpha::group one > title with $(rm -rf) [x] (y)? *']).toBe('pass');
});
test('playwright retries without retry data are detected, not read as one attempt', () => {
expect(parseJunit(fixture('playwright-r2.xml')).retriesHidden).toBe(true);
expect(parseJunit(fixture('playwright-r0.xml')).retriesHidden).toBe(false);
});
test('missing, empty, zero-testcase and malformed reports are errors, never green', () => {
const dir = tmp();
const code = (fn: () => unknown) => { try { fn(); return 'ok'; } catch (e) { return (e as JunitError).code; } };
expect(code(() => readJunit(path.join(dir, 'absent.xml')))).toBe('FLAKY_NO_RESULTS');
fs.writeFileSync(path.join(dir, 'empty.xml'), ' \n');
expect(code(() => readJunit(path.join(dir, 'empty.xml')))).toBe('FLAKY_NO_RESULTS');
expect(code(() => parseJunit('<testsuites name="x"><testsuite name="s" tests="0"/></testsuites>'))).toBe('FLAKY_NO_RESULTS');
expect(code(() => parseJunit('<testsuites><testsuite name="s"><testcase name="a"></testsuite></testsuites>'))).toBe('FLAKY_JUNIT_INVALID');
});
test('generic runners (jest, pytest): file from the testcase, entities decoded', () => {
const r = parseJunit('<?xml version="1.0"?><testsuites><testsuite name="pytest"><testcase classname="tests.test_a" name="test_x[a&amp;b]" file="tests/test_a.py"><error message="boom"/></testcase></testsuite></testsuites>');
expect(r.runner).toBe('generic');
expect(r.tests[0]!.id).toBe('tests/test_a.py::::test_x[a&b]');
expect(r.tests[0]!.attempts).toEqual(['fail']);
});
});
describe('rerun selection', () => {
const pw = parseJunit(fixture('playwright-r0.xml')).tests.find(t => t.path.at(-1)!.startsWith('title with') && t.project === 'beta')!;
test('playwright: file and title regex-escaped, project kept, each one argv element', () => {
expect(rerunSelection('playwright', pw)).toEqual([
'a\\.spec\\.ts', '--grep', '(?:^|\\s)group one title with \\$\\(rm -rf\\) \\[x\\] \\(y\\)\\? \\*(?:\\s|$)', '--project', 'beta',
]);
});
test('bun: anchored space-joined path', () => {
const t = parseJunit(fixture('bun.xml')).tests[1]!;
expect(rerunSelection('bun', t)).toEqual(['a.test.ts', '--test-name-pattern', '^outer inner fails$']);
});
});
describe('complete-run proof for a targeted rerun', () => {
test('a complete run needs exit 1 and the runner summary without an early stop', () => {
expect(incompleteRun('bun', ' 4 pass\n 1 fail\nRan 5 tests across 2 files. [9ms]\n', 1)).toBeNull();
expect(incompleteRun('bun', 'Bailed out after 1 failure\nRan 1 test across 1 file.\n', 1)).toContain('--bail');
expect(incompleteRun('bun', ' 1 fail\n', 1)).toContain('no run summary');
expect(incompleteRun('bun', 'Ran 5 tests across 2 files.\n', 2)).toContain('exited 2');
expect(incompleteRun('playwright', ' 1 failed\n [alpha] x\n 4 passed (1s)\n', 1)).toBeNull();
expect(incompleteRun('playwright', 'Testing stopped early after 1 maximum allowed failures.\n 1 failed\n 2 did not run\n', 1)).toContain('2 did not run');
expect(incompleteRun('playwright', ' 1 failed\n 3 interrupted\n', 1)).toContain('3 interrupted');
expect(incompleteRun('playwright', 'Timed out waiting 600s for the test suite to run\nTimed out waiting 600s for the entire test run\n 1 failed\n', 1)).toContain('stopped early');
expect(incompleteRun('playwright', '', 1)).toContain('no summary');
expect(incompleteRun('generic', 'Ran 5 tests', 1)).toContain('generic');
expect(incompleteRun('bun', 'All files | % Funcs | % Lines |\nRan 5 tests across 2 files.\n', 1)).toContain('coverage');
expect(incompleteRun('bun', 'Ran 5 tests across 2 files.\n', 1, ['bun', 'test', '--coverage'])).toContain('coverage');
expect(incompleteRun('playwright', ' 1 failed\n 1 flaky\n 4 passed\n', 1, [], 1)).toContain('fail-on-flaky');
});
});
/** A fake runner: argv selection decides which canned report the attempt writes. */
const BUN_SUMMARY = 'Ran 2 tests across 1 file. [3ms]\n';
function fakeRun(reports: (argv: string[], n: number) => string | null, exits: (argv: string[], n: number) => number, outputs: (argv: string[], n: number) => string = () => BUN_SUMMARY, signals: (n: number) => string | undefined = () => undefined) {
const calls: string[][] = [];
return {
calls,
spawn(junit: string) {
return async (argv: string[], _env: NodeJS.ProcessEnv, cwd: string) => {
calls.push(argv);
const xml = reports(argv, calls.length);
if (xml !== null) fs.writeFileSync(path.resolve(cwd, junit), xml);
const signal = signals(calls.length);
return { exit: exits(argv, calls.length), output: outputs(argv, calls.length), ...(signal ? { signal } : {}) };
};
},
};
}
const suite = (cases: string) => `<testsuites name="bun test"><testsuite name="a.test.ts" file="a.test.ts">${cases}</testsuite></testsuites>`;
const pass = (name: string) => `<testcase name="${name}" classname="" file="a.test.ts"/>`;
const fail = (name: string, msg = 'boom') => `<testcase name="${name}" classname="" file="a.test.ts"><failure message="${msg}"/></testcase>`;
describe('runFlaky', () => {
const base = (spawn: ReturnType<typeof fakeRun>, dir: string, extra: Partial<Parameters<typeof runFlaky>[0]> = {}) =>
runFlaky({ command: ['bun', 'test'], junit: 'r.xml', maxReruns: 3, cwd: dir, spawn: spawn.spawn('r.xml'), env: {}, now: () => new Date('2026-10-29T00:00:00Z'), ...extra });
test('a green run with no failures is green and reruns nothing', async () => {
const dir = tmp();
const f = fakeRun(() => suite(pass('a') + pass('b')), () => 0);
expect(await base(f, dir)).toMatchObject({ exitCode: 0, reruns: 0, ledger: [] });
expect(f.calls).toHaveLength(1);
});
test('a coverage failure beside a flaky test is not cleared by a narrowed rerun', async () => {
const dir = tmp();
const cov = () => '----|---------|---------|\nAll files | % Funcs | % Lines |\nRan 2 tests across 1 file.\n';
const f = fakeRun((_a, n) => n === 1 ? suite(pass('a') + fail('b')) : suite(pass('a') + pass('b')), () => 1, cov);
const r = await base(f, dir, { rerunScope: 'test' });
expect(f.calls[1]).toEqual(['bun', 'test']);
expect(r).toMatchObject({ exitCode: 1, wholeRerun: 'coverage is measured, and a narrowed rerun cannot reproduce a coverage failure' });
});
test('by default the rerun is the whole command and counts only on its full green verdict', async () => {
const dir = tmp();
const green = fakeRun((_a, n) => n === 1 ? suite(pass('a') + fail('b')) : suite(pass('a') + pass('b')), (_a, n) => n === 1 ? 1 : 0);
const r = await base(green, dir);
expect(green.calls).toEqual([['bun', 'test'], ['bun', 'test']]);
expect(r).toMatchObject({ exitCode: 0, wholeRerun: 'the default; --rerun-scope test reruns the failed test alone' });
expect(r.ledger[0]).toMatchObject({ kind: 'flaky-pass', rerun_scope: 'command' });
const otherFails = fakeRun((_a, n) => n === 1 ? suite(pass('a') + fail('b')) : suite(fail('a') + pass('b')), (_a, n) => n === 1 ? 1 : 0);
expect((await base(otherFails, dir)).outcomes[0]).toMatchObject({ status: 'rerun-fail', reason: 'rerun: 1 test(s) failed in the whole-command rerun' });
});
test('a failed test that passes on rerun is a flaky pass and goes on the ledger', async () => {
const dir = tmp();
const f = fakeRun((argv, n) => n === 1 ? suite(pass('a') + fail('b')) : suite(pass('b')), (_a, n) => n === 1 ? 1 : 0);
const r = await base(f, dir, { rerunScope: 'test' });
expect(r.exitCode).toBe(0);
expect(f.calls[1]).toEqual(['bun', 'test', 'a.test.ts', '--test-name-pattern', '^b$']);
expect(r.outcomes.map(o => [o.status, o.attempts.join(',')])).toEqual([['flaky-pass', 'fail,pass']]);
expect(r.ledger).toHaveLength(1);
expect(r.ledger[0]).toMatchObject({ kind: 'flaky-pass', id: 'a.test.ts::::b', rerun_by: 'gstack-flaky', attempts: ['fail', 'pass'], excerpt: ['boom'] });
expect(renderResult(r, '/l.jsonl').at(-1)).toBe('FLAKY_RESULT: tests=2 failed=0 flaky=1 reruns=1 command_exit=1 verdict=pass ledger=/l.jsonl');
expect(stepSummary(r)).toContain('a.test.ts::::b');
});
test('a test that fails its rerun is red and is ledgered as rerun-fail', async () => {
const dir = tmp();
const f = fakeRun(() => suite(fail('b')), () => 1);
const r = await base(f, dir);
expect(r.exitCode).toBe(1);
expect(r.outcomes[0]).toMatchObject({ status: 'rerun-fail', attempts: ['fail', 'fail'] });
expect(r.ledger[0]!.kind).toBe('rerun-fail');
});
test('more failures than --max-reruns: a real red, nothing rerun', async () => {
const dir = tmp();
const f = fakeRun(() => suite(fail('a') + fail('b') + fail('c') + fail('d')), () => 1);
const r = await base(f, dir);
expect(r.exitCode).toBe(1);
expect(f.calls).toHaveLength(1);
expect(r.reruns).toBe(0);
expect(r.outcomes.every(o => o.status === 'fail' && /more than --max-reruns 3/.test(o.reason!))).toBe(true);
expect(r.ledger).toEqual([]);
});
test('runner retries stand: no second rerun, and a runner flaky pass is ledgered', async () => {
const dir = tmp();
const f = fakeRun(() => fixture('playwright-r2-inc.xml'), () => 1);
const r = await base(f, dir);
expect(f.calls).toHaveLength(1);
expect(r.exitCode).toBe(1);
expect(r.ledger.filter(l => l.kind === 'flaky-pass').map(l => [l.id, l.rerun_by])).toEqual([
['a.spec.ts::alpha::group one > flaky once', 'runner'], ['a.spec.ts::beta::group one > flaky once', 'runner'],
]);
expect(r.outcomes.filter(o => o.status === 'fail').every(o => /already retried \(3 attempts\)/.test(o.reason!))).toBe(true);
});
test('hidden Playwright retries refuse with a code instead of a guess', async () => {
const dir = tmp();
const r = await base(fakeRun(() => fixture('playwright-r2.xml'), () => 1), dir);
expect([r.exitCode, r.code]).toEqual([3, 'FLAKY_RETRIES_HIDDEN']);
});
test('no report, a stale report, and a crash with no failed test are never green', async () => {
const dir = tmp();
expect(await base(fakeRun(() => null, () => 0), dir)).toMatchObject({ exitCode: 3, code: 'FLAKY_NO_RESULTS' });
fs.writeFileSync(path.join(dir, 'r.xml'), suite(pass('stale')));
expect(await base(fakeRun(() => null, () => 0), dir)).toMatchObject({ exitCode: 3, code: 'FLAKY_NO_RESULTS' });
expect(await base(fakeRun(() => suite(pass('a')), () => 2), dir)).toMatchObject({ exitCode: 3, code: 'FLAKY_COMMAND_FAILED', commandExit: 2 });
});
test('a rerun that does not report the targeted test fails it; other tests in the rerun are ignored', async () => {
const dir = tmp();
const f = fakeRun((_a, n) => n === 1 ? suite(fail('b')) : suite(pass('other')), (_a, n) => n === 1 ? 1 : 0);
const r = await base(f, dir);
expect(r.exitCode).toBe(1);
expect(r.outcomes[0]).toMatchObject({ status: 'rerun-fail', reason: 'rerun: the report does not contain this test' });
});
test('a rerun whose command exits non-zero is not a flaky pass, even with a passing report', async () => {
const dir = tmp();
const f = fakeRun((_a, n) => n === 1 ? suite(fail('b')) : suite(pass('b')), () => 1);
expect((await base(f, dir)).outcomes[0]).toMatchObject({ status: 'rerun-fail', reason: 'rerun: the command exited 1' });
});
test('a test reported twice with one failure is red and not rerun', async () => {
const dir = tmp();
const r = parseJunit(suite(pass('b') + fail('b')));
expect(r.tests[0]).toMatchObject({ executions: 2, attempts: ['pass', 'fail'] });
const f = fakeRun(() => suite(fail('b') + pass('b') + fail('c')), () => 1);
const out = await base(f, dir, { rerunScope: 'test' });
expect(out.outcomes.find(o => o.test.id.endsWith('::b'))).toMatchObject({ status: 'fail', reason: 'it ran 2 times in this report and one failed; not rerun' });
expect(f.calls).toHaveLength(2);
});
test('a runner-caught flake is ledgered even when a duplicate execution of the test was skipped', async () => {
const dir = tmp();
const caught = '<testcase name="b" classname="" file="a.test.ts"><flakyFailure message="x"/></testcase>';
const r = await base(fakeRun(() => suite(caught + '<testcase name="b" classname="" file="a.test.ts"><skipped/></testcase>'), () => 0), dir);
expect(r.exitCode).toBe(0);
expect(r.ledger.map(l => [l.kind, l.rerun_by])).toEqual([['flaky-pass', 'runner']]);
});
test('the Playwright environment flag reaches every attempt', async () => {
const dir = tmp();
const seen: string[] = [];
await runFlaky({ command: ['x'], junit: 'r.xml', maxReruns: 3, cwd: dir, env: {}, spawn: async (_a, env, cwd) => {
seen.push(env.PLAYWRIGHT_JUNIT_INCLUDE_RETRIES ?? '');
fs.writeFileSync(path.join(cwd, 'r.xml'), suite(seen.length === 1 ? fail('b') : pass('b')));
return { exit: seen.length === 1 ? 1 : 0, output: '' };
} });
expect(seen).toEqual(['1', '1']);
});
test('bun load errors the JUnit omits keep the run red and nothing is rerun', async () => {
const dir = tmp();
const f = fakeRun(() => suite(fail('b')), () => 1, () => '\n 0 pass\n 3 fail\n 2 errors\nRan 3 tests across 3 files.\n');
const r = await base(f, dir);
expect(r).toMatchObject({ exitCode: 1, code: 'FLAKY_UNREPORTED_FAILURES', reruns: 0 });
expect(f.calls).toHaveLength(1);
});
test('runner summaries: Playwright errors outside tests and count mismatches are unreported failures', () => {
expect(unreportedFailures('playwright', ' 1 error was not a part of any test, see above for details\n 1 failed\n', 1)).toContain('outside any test');
expect(unreportedFailures('playwright', ' 2 failed\n 10 passed\n', 1)).toContain('counted 2 failed');
expect(unreportedFailures('playwright', ' 1 failed\n [chromium] x\n 10 passed\n', 1)).toBeNull();
expect(unreportedFailures('bun', ' 4 pass\n 1 fail\n', 1)).toBeNull();
expect(unreportedFailures('generic', '5 failed', 0)).toBeNull();
});
test('a run that stopped early reruns the whole command, so tests it never reached must pass too', async () => {
const dir = tmp();
const bail = () => 'Bailed out after 1 failure\nRan 1 test across 1 file.\n';
const green = fakeRun((_a, n) => n === 1 ? suite(fail('b')) : suite(pass('b') + pass('never-reached')), (_a, n) => n === 1 ? 1 : 0, bail);
const r = await base(green, dir, { rerunScope: 'test' });
expect(green.calls).toEqual([['bun', 'test'], ['bun', 'test']]);
expect(r).toMatchObject({ exitCode: 0, wholeRerun: 'bun stopped early (--bail)' });
expect(r.ledger[0]).toMatchObject({ kind: 'flaky-pass', rerun_scope: 'command' });
expect(renderResult(r, '/l')).toContain('FLAKY_RERUN: scope=command (bun stopped early (--bail))');
const red = fakeRun((_a, n) => n === 1 ? suite(fail('b')) : suite(pass('b') + fail('never-reached')), () => 1, bail);
expect((await base(red, dir, { rerunScope: 'test' })).outcomes[0]).toMatchObject({ status: 'rerun-fail', reason: 'rerun: the command exited 1' });
});
test('generic runners rerun the whole command even when a targeted rerun is asked for', async () => {
const dir = tmp();
const f = fakeRun((_a, n) => n === 1 ? suite(fail('b')) : suite(pass('b')), (_a, n) => n === 1 ? 1 : 0);
const r = await base(f, dir, { runner: 'generic', rerunScope: 'test' });
expect(f.calls[1]).toEqual(['bun', 'test']);
expect(r).toMatchObject({ exitCode: 0, wholeRerun: 'a generic runner cannot prove every test ran' });
});
test('a whole-command rerun clears nothing when any test fails in it, a first-run test is missing, it is killed or it retried', async () => {
const dir = tmp();
const generic = (second: string, exit = 0, signal?: string) => base(fakeRun((_a, n) => n === 1 ? suite(fail('a') + pass('b')) : second, () => exit && 1, undefined, n => n === 2 ? signal : undefined), dir, { runner: 'generic' });
expect((await generic(suite(pass('a') + fail('b')))).outcomes[0]).toMatchObject({ status: 'rerun-fail', reason: 'rerun: 1 test(s) failed in the whole-command rerun' });
expect((await generic(suite(pass('a')))).outcomes[0]).toMatchObject({ status: 'rerun-fail', reason: 'rerun: 1 test(s) of the first run are missing from the whole-command rerun' });
expect((await generic(suite(pass('a') + pass('b')), 0, 'SIGTERM')).outcomes[0]).toMatchObject({ status: 'rerun-fail', reason: 'rerun: killed by SIGTERM' });
const retried = await generic(suite('<testcase name="a" classname="" file="a.test.ts"><flakyFailure message="x"/></testcase>' + pass('b')));
expect(retried.outcomes.find(o => o.status === 'rerun-fail')).toMatchObject({ attempts: ['fail', 'fail', 'pass'] });
expect(retried.ledger.find(l => l.kind === 'rerun-fail')).toMatchObject({ rerun_scope: 'command' });
expect(retried.exitCode).toBe(1);
});
test('an incomplete run that is already red is not rerun as a whole', async () => {
const dir = tmp();
const f = fakeRun(() => suite(fail('b') + pass('b') + fail('c')), () => 1, () => '');
const r = await base(f, dir);
expect(f.calls).toHaveLength(1);
expect(r.outcomes.find(o => o.test.id.endsWith('::c'))!.reason).toContain('already red');
});
test('a rerun the runner retried again does not clear the test', async () => {
const dir = tmp();
const f = fakeRun((_a, n) => n === 1 ? suite(fail('b')) : suite('<testcase name="b" classname="" file="a.test.ts"><flakyFailure message="x"/></testcase>'), (_a, n) => n === 1 ? 1 : 0);
expect((await base(f, dir)).outcomes[0]).toMatchObject({ status: 'rerun-fail', attempts: ['fail', 'fail', 'pass'] });
});
test('a killed runner is never green and stops further reruns', async () => {
const dir = tmp();
expect(await base(fakeRun(() => suite(fail('b')), () => 128, undefined, () => 'SIGTERM'), dir)).toMatchObject({ exitCode: 3, code: 'FLAKY_COMMAND_KILLED' });
const f = fakeRun((_a, n) => n === 1 ? suite(fail('b') + fail('c')) : suite(pass('b') + pass('c')), (_a, n) => n === 1 ? 1 : 128, undefined, n => n === 2 ? 'SIGTERM' : undefined);
const r = await base(f, dir, { rerunScope: 'test' });
expect(f.calls).toHaveLength(2);
expect(r.outcomes.map(o => o.reason)).toEqual(['rerun: killed by SIGTERM', 'not rerun: an earlier rerun was killed']);
expect(r.exitCode).toBe(1);
});
test('the first report is restored at --junit and each rerun report is kept beside it', async () => {
const dir = tmp();
const firstXml = suite(pass('a') + fail('b'));
await base(fakeRun((_a, n) => n === 1 ? firstXml : suite(pass('b')), (_a, n) => n === 1 ? 1 : 0), dir);
expect(fs.readFileSync(path.join(dir, 'r.xml'), 'utf-8')).toBe(firstXml);
expect(fs.readFileSync(path.join(dir, 'r.rerun-1.xml'), 'utf-8')).toBe(suite(pass('b')));
});
test('credentials in a test title never reach the ledger, the result lines or the step summary', async () => {
const dir = tmp();
const key = ['sk', 'ant', 'api03', 'B'.repeat(93) + 'AA'].join('-');
const r = await base(fakeRun((_a, n) => n === 1 ? suite(fail(`login with ${key}`)) : suite(pass(`login with ${key}`)), (_a, n) => n === 1 ? 1 : 0), dir);
expect(r.exitCode).toBe(0);
expect(JSON.stringify(r.ledger)).not.toContain(key);
expect(renderResult(r, '/l').join('\n')).not.toContain(key);
expect(stepSummary(r)).not.toContain(key);
});
test('workflow commands escape a title that would end or start a command', () => {
expect(workflowEscape('a%b\r\n::error::x')).toBe('a%25b%0D%0A::error::x');
});
test('ledger: a write error is a warning; the path comes from --ledger, then GSTACK_FLAKE_LEDGER_JUNIT, then the state root', () => {
const dir = tmp();
fs.writeFileSync(path.join(dir, 'file'), '');
const warns: string[] = [];
appendLedger([{ ts: 't', runner: 'junit', kind: 'flaky-pass', id: 'x', file: 'x', rerun_by: 'gstack-flaky', attempts: [], excerpt: [] }], path.join(dir, 'file', 'l.jsonl'), w => warns.push(w));
expect(warns[0]).toContain('FLAKY_LEDGER_WRITE_FAILED');
expect(ledgerPath('/flag', { GSTACK_FLAKE_LEDGER_JUNIT: '/env' })).toBe('/flag');
expect(ledgerPath(undefined, { GSTACK_FLAKE_LEDGER_JUNIT: '/env', GSTACK_FLAKE_LEDGER: '/free' })).toBe('/env');
expect(ledgerPath(undefined, { GSTACK_STATE_ROOT: dir, GSTACK_FLAKE_LEDGER: '/free' })).toBe(path.join(dir, 'flake-ledger-junit.jsonl'));
});
test('credentials in failure text never reach the ledger', () => {
const key = ['sk', 'ant', 'api03', 'A'.repeat(93) + 'AA'].join('-');
const out = safeExcerpt([`Error: request failed with key ${key}`]);
expect(out.join('\n')).not.toContain(key);
});
});
describe('bin/gstack-flaky', () => {
test('run: a fake bun runner flakes once, the bin reruns it and exits 0 with a ledger line', () => {
const dir = tmp();
const runner = path.join(dir, 'runner.sh');
fs.writeFileSync(runner, [
'#!/usr/bin/env bash',
'n=$(cat count 2>/dev/null || echo 0); echo $((n+1)) > count',
`if [ "$n" = 0 ]; then printf '%s' '${suite(pass('a') + fail('b'))}' > r.xml; echo 'Ran 2 tests across 1 file.' >&2; exit 1; fi`,
'[ "$1" = a.test.ts ] && [ "$2" = --test-name-pattern ] && [ "$3" = "^b$" ] || exit 9',
`printf '%s' '${suite(pass('b'))}' > r.xml`,
].join('\n'), { mode: 0o755 });
const ledger = path.join(dir, 'ledger.jsonl');
const r = spawnSync(path.join(ROOT, 'bin', 'gstack-flaky'), ['run', '--junit', 'r.xml', '--runner', 'bun', '--rerun-scope', 'test', '--ledger', ledger, '--', runner], { cwd: dir, encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GITHUB_ACTIONS: '', GITHUB_STEP_SUMMARY: '' } });
expect(r.stdout).toContain('FLAKY: flaky-pass a.test.ts::::b attempts=fail,pass');
expect(r.stdout).toContain(`verdict=pass ledger=${ledger}`);
expect(r.status).toBe(0);
expect(JSON.parse(fs.readFileSync(ledger, 'utf-8').trim()).kind).toBe('flaky-pass');
});
test('usage, a command that cannot start, and missing reports exit with their codes', () => {
const dir = tmp();
const bin = path.join(ROOT, 'bin', 'gstack-flaky');
expect(spawnSync(bin, ['run', '--', 'true'], { cwd: dir, encoding: 'utf-8', timeout: 30_000 }).status).toBe(2);
fs.writeFileSync(path.join(dir, 'keep.ts'), 'x');
expect(spawnSync(bin, ['run', '--junit', 'keep.ts', '--', 'true'], { cwd: dir, encoding: 'utf-8', timeout: 30_000 }).status).toBe(2);
expect(fs.existsSync(path.join(dir, 'keep.ts'))).toBe(true);
fs.writeFileSync(path.join(dir, 'pom.xml'), '<project><modelVersion>4.0.0</modelVersion></project>');
expect(spawnSync(bin, ['run', '--junit', 'pom.xml', '--', 'true'], { cwd: dir, encoding: 'utf-8', timeout: 30_000 }).status).toBe(2);
expect(fs.existsSync(path.join(dir, 'pom.xml'))).toBe(true);
expect(spawnSync(bin, ['run', '--junit', 'r.xml', '--rerun-scope', 'file', '--', 'true'], { cwd: dir, encoding: 'utf-8', timeout: 30_000 }).status).toBe(2);
fs.writeFileSync(path.join(dir, 'locked.xml'), '<testsuites/>', { mode: 0o000 });
if (process.getuid?.() !== 0) {
const locked = spawnSync(bin, ['run', '--junit', 'locked.xml', '--', 'true'], { cwd: dir, encoding: 'utf-8', timeout: 30_000 });
expect([locked.status, locked.stderr.includes('cannot be read')]).toEqual([2, true]);
}
const missing = spawnSync(bin, ['run', '--junit', 'r.xml', '--', path.join(dir, 'no-such-runner')], { cwd: dir, encoding: 'utf-8', timeout: 30_000 });
expect(missing.status).toBe(3);
expect(missing.stderr).toContain('(FLAKY_COMMAND_NOT_STARTED)');
const r = spawnSync(bin, ['run', '--junit', 'r.xml', '--', 'true'], { cwd: dir, encoding: 'utf-8', timeout: 30_000 });
expect(r.status).toBe(3);
expect(r.stderr).toContain('(FLAKY_NO_RESULTS)');
expect(r.stderr).toContain('fix: ');
});
test('a runner that dies by a signal exits 3 FLAKY_COMMAND_KILLED and is not rerun', () => {
const dir = tmp();
const runner = path.join(dir, 'runner.sh');
fs.writeFileSync(runner, ['#!/usr/bin/env bash', 'echo x >> count', `printf '%s' '${suite(fail('b'))}' > r.xml`, 'kill -TERM $$'].join('\n'), { mode: 0o755 });
const r = spawnSync(path.join(ROOT, 'bin', 'gstack-flaky'), ['run', '--junit', 'r.xml', '--runner', 'bun', '--ledger', path.join(dir, 'l.jsonl'), '--', runner], { cwd: dir, encoding: 'utf-8', timeout: 30_000 });
expect(r.status).toBe(3);
expect(r.stderr).toContain('(FLAKY_COMMAND_KILLED)');
expect(fs.readFileSync(path.join(dir, 'count'), 'utf-8').trim().split('\n')).toHaveLength(1);
});
test('cancelling gstack-flaky is latched even when the runner traps the signal and exits by code', async () => {
const dir = tmp();
const runner = path.join(dir, 'runner.sh');
fs.writeFileSync(runner, [
'#!/usr/bin/env bash',
'echo x >> count',
'trap "cp report.xml r.xml; exit 143" TERM',
'touch ready',
'for i in $(seq 1 300); do sleep 0.1; done',
].join('\n'), { mode: 0o755 });
fs.writeFileSync(path.join(dir, 'report.xml'), suite(fail('b')));
const child = spawn(path.join(ROOT, 'bin', 'gstack-flaky'), ['run', '--junit', 'r.xml', '--runner', 'generic', '--ledger', path.join(dir, 'l.jsonl'), '--', runner], { cwd: dir, stdio: ['ignore', 'ignore', 'pipe'] });
let stderr = '';
child.stderr.on('data', (c: Buffer) => { stderr += c.toString(); });
const done = new Promise<number | null>(resolve => child.on('close', code => resolve(code)));
const deadline = Date.now() + 20_000;
while (!fs.existsSync(path.join(dir, 'ready')) && Date.now() < deadline) await Bun.sleep(50);
child.kill('SIGTERM');
expect(await done).toBe(3);
expect(stderr).toContain('(FLAKY_COMMAND_KILLED)');
expect(fs.readFileSync(path.join(dir, 'count'), 'utf-8').trim().split('\n')).toHaveLength(1);
expect(fs.existsSync(path.join(dir, 'l.jsonl'))).toBe(false);
}, 30_000);
test('every FLAKY_ code the bin and the library emit is in the docs table', () => {
const doc = fs.readFileSync(path.join(ROOT, 'docs', 'flaky.md'), 'utf-8');
const src = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-flaky'), 'utf-8') + fs.readFileSync(path.join(ROOT, 'lib', 'flaky-run.ts'), 'utf-8') + fs.readFileSync(path.join(ROOT, 'lib', 'junit.ts'), 'utf-8');
const codes = [...new Set(src.match(/FLAKY_[A-Z_]+(?=['":])/g))].filter(c => c !== 'FLAKY_RESULT' && c !== 'FLAKY_RERUN');
expect(codes.length).toBeGreaterThan(5);
expect(codes.filter(c => !doc.includes(`| \`${c}\` |`))).toEqual([]);
});
test('ids prints one id per test', () => {
const r = spawnSync(path.join(ROOT, 'bin', 'gstack-flaky'), ['ids', '--junit', path.join(FIX, 'bun.xml')], { encoding: 'utf-8', timeout: 30_000 });
expect(r.stdout.trim().split('\n')).toEqual([testId('a.test.ts', '', ['outer', 'inner', 'passes']), 'a.test.ts::::outer > inner > fails', 'a.test.ts::::outer > skipped', 'a.test.ts::::top level']);
});
});
test/gstack-doctor.test.ts
+1 -1
View File
@@ -27,7 +27,7 @@ const COPIED = [
'autoplan/SKILL.md', // its frontmatter registers the autoplan hook the hooks-row test breaks
'bin/gstack-launch-probe.sh',
'bin/gstack-doctor-check.sh', 'bin/gstack-doctor-components.sh', 'bin/gstack-runtime-pins.ts', 'lib/runtime-pins.ts',
'bin/gstack-windows-bash.ts', 'lib/script-invocation.ts', 'lib/bun-isolation.ts', 'lib/result-codes.ts', 'lib/result-codes-release.ts', 'lib/result-codes-capy.ts', 'lib/result-codes-budget.ts', // the windows bash row
'bin/gstack-windows-bash.ts', 'lib/script-invocation.ts', 'lib/bun-isolation.ts', 'lib/result-codes.ts', 'lib/result-codes-release.ts', 'lib/result-codes-capy.ts', 'lib/result-codes-budget.ts', 'lib/result-codes-eval.ts', // the windows bash row
];
const bases: string[] = [];
afterEach(() => { for (const b of bases.splice(0)) fs.rmSync(b, { recursive: true, force: true }); });
test/helpers/free-fixtures-data.ts
+1
View File
@@ -151,6 +151,7 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
'test/fixtures/investigate-owned-question-39195396438.json': ['test/workflow-boundaries-fixture.test.ts'],
'test/fixtures/judge-calibration/**': ['scripts/judge-calibration.ts', 'test/judge-calibration.test.ts'],
'test/fixtures/judge-panel-replay-2026-10-05.json': ['test/judge-panel-median.test.ts'],
'test/fixtures/junit/**': ['test/flaky-run.test.ts'],
'test/fixtures/launch-env-echo/**': ['test/launch-env-echo-ratchet.test.ts'],
'test/fixtures/merge-train-3345/**': ['test/changelog-merge-damage.test.ts', 'test/compare-pr-version.test.ts'],
'test/fixtures/merge-train-3453/**': ['test/changelog-merge-damage.test.ts'],
test/plan-tune-cathedral.test.ts
+1 -1
View File
@@ -70,7 +70,7 @@ function scaffoldFixture(workDir: string): { workDir: string; stateRoot: string;
// These imports are resolved relative to the copied install, not the source checkout.
const libDir = path.join(workDir, 'lib');
fs.mkdirSync(libDir, { recursive: true });
for (const file of ['jsonl-store.ts', 'is-conductor.ts', 'state-root.ts', 'bin-context.ts', 'remote-identity.ts', 'script-invocation.ts', 'bun-isolation.ts', 'result-codes.ts', 'result-codes-release.ts', 'result-codes-capy.ts', 'result-codes-budget.ts']) {
for (const file of ['jsonl-store.ts', 'is-conductor.ts', 'state-root.ts', 'bin-context.ts', 'remote-identity.ts', 'script-invocation.ts', 'bun-isolation.ts', 'result-codes.ts', 'result-codes-release.ts', 'result-codes-capy.ts', 'result-codes-budget.ts', 'result-codes-eval.ts']) {
fs.copyFileSync(path.join(ROOT, 'lib', file), path.join(libDir, file));
}
test/pregate-clear-runs.test.ts
+246
View File
@@ -0,0 +1,246 @@
/**
* gstack-pregate clear --run (GSTA-156): GitHub Actions runs clear the remote gate obligation, or a lane with a run
* profile, on the identical tree or a stamp-only predecessor, covering the gate's inventory; verify then checks the
* recorded receipt offline. The run API is a fake; trees come from the fixture repository's git.
*/
import { afterEach, describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { EXIT, type PregateFile } from '../lib/headless-artifacts';
import { buildContext } from '../lib/pregate/context';
import { clearDecision, clearObligation, runPregate, verifyPregate, type RunCheck } from '../lib/pregate/index';
import type { ApiRun, RunDeps, RunProfile } from '../lib/run-evidence';
import { BASE_PREGATE, cleanup, commit, makeFixtureRepo, readFile, runBin, type FixtureRepo } from './helpers/pregate-fixture';
const repos: FixtureRepo[] = [];
afterEach(() => { while (repos.length) cleanup(repos.pop()!); });
const LANE = 'pregate.json/remote_gate';
const NOW = new Date('2026-10-29T12:00:00Z');
const POLICY = { bump: 'patch', queue_mode: 'stamp-at-merge', changelog: 'CHANGELOG.md', stamp_files: ['VERSION', 'package.json', 'CHANGELOG.md'] };
const PROFILES: Record<string, RunProfile> = {
test: { workflow: 'test.yml', events: ['workflow_dispatch'], jobs: ['full-suite'], artifact: 'full-suite-tree' },
e2e: { workflow: 'e2e.yml', events: ['workflow_dispatch'], jobs: ['e2e-full-suite'], artifact: 'e2e-full-suite-tree' },
};
const GATE_RUNS = { profiles: ['test', 'e2e'], inventory: 'printf "test/a.test.ts\\ntest/b.test.ts\\n"' };
function fixture(extra: Record<string, unknown> = {}): FixtureRepo {
const repo = makeFixtureRepo({
pregate: { ...BASE_PREGATE, ...extra, remote_gate: 'node scripts/remote-gate.js', dependencies: { ...BASE_PREGATE.dependencies, 'CHANGELOG.md': [], VERSION: [] } },
baseFiles: { '.gstack/ship-policy.json': JSON.stringify(POLICY, null, 2) + '\n', VERSION: '1.0.0\n', 'CHANGELOG.md': '# Changelog\n\n## [Unreleased]\n\n- add\n' },
});
repos.push(repo);
repo.git('checkout', '-q', 'main');
commit(repo, { 'scripts/remote-gate.js': "console.log('PREGATE_REMOTE: tree=' + process.env.PREGATE_TREE + ' tests=3 failed=0');\n" }, 'remote gate');
repo.git('push', '-q', 'origin', 'main');
repo.git('checkout', '-q', 'feature');
repo.git('merge', '-q', '--ff-only', 'main');
commit(repo, { 'lib/util.js': 'exports.add = (a, b) => a + b + 0;\n' }, 'the real change');
return repo;
}
function stamp(repo: FixtureRepo): string {
const pkg = JSON.parse(readFile(repo, 'package.json'));
pkg.version = '1.0.1';
return commit(repo, { VERSION: '1.0.1\n', 'package.json': JSON.stringify(pkg, null, 2) + '\n', 'CHANGELOG.md': '# Changelog\n\n## [1.0.1] - 2026-10-29\n\n- add\n' }, 'v1.0.1 stamp');
}
async function pregateAt(repo: FixtureRepo): Promise<PregateFile> {
const loaded = buildContext({ repoRoot: repo.work, base: 'main', policyFrom: 'base', allowRepoCommands: true, env: { ...process.env, ...repo.env }, progress: () => {}, cores: 4, testBackend: 'local' });
if (!('ctx' in loaded)) throw new Error('fixture context refused');
const file = await runPregate(loaded.ctx, { stage: 'tests' });
expect(file.requires_remote).toEqual([expect.objectContaining({ lane: LANE, receipt: null })]);
return file;
}
/** A run of `profile` on `head` with a tree.json listing `files`, as GitHub would report it. */
interface FakeRun { id: number; profile: string; head: string; files: string[]; conclusion?: string; updated?: string }
function check(repo: FixtureRepo, runs: FakeRun[], over: Partial<RunCheck> = {}): RunCheck {
const byId = new Map(runs.map(r => [r.id, r]));
const api = (r: FakeRun): ApiRun => ({
id: r.id, run_attempt: 1, status: 'completed', conclusion: r.conclusion ?? 'success', event: 'workflow_dispatch', head_sha: r.head,
path: `.github/workflows/${PROFILES[r.profile]!.workflow}`, created_at: '2026-10-29T10:00:00Z', updated_at: r.updated ?? '2026-10-29T10:30:00Z',
repository: { full_name: 'fx/fx' }, head_repository: { full_name: 'fx/fx' },
});
const deps: RunDeps = {
run: id => { const r = byId.get(id); if (!r) throw new Error('HTTP 404'); return api(r); },
runsForHead: (wf, sha) => runs.filter(r => r.head === sha && PROFILES[r.profile]!.workflow === wf).map(api),
jobs: id => [{ name: PROFILES[byId.get(id)!.profile]!.jobs[0]!, conclusion: 'success' }],
artifact: id => { const r = byId.get(id)!; return { run_id: id, run_attempt: 1, event: 'workflow_dispatch', sha: r.head, tree: repo.git('rev-parse', `${r.head}^{tree}`), files: r.files, tests: r.files.length * 5 }; },
commitTree: sha => repo.git('rev-parse', `${sha}^{tree}`),
};
return { repo: 'fx/fx', deps, profiles: PROFILES, gateRuns: GATE_RUNS, now: NOW, inventory: () => ['test/a.test.ts', 'test/b.test.ts'], ...over };
}
describe.skipIf(process.platform === 'win32')('clear --run: Actions runs as remote gate evidence', () => {
test('one verified run per profile on the identical tree clears the gate, and verify accepts the receipt offline', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const d = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 201, profile: 'e2e', head, files: ['test/b.test.ts'] }, { id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/util.test.js'] }]) }, repo.work);
if ('refused' in d) throw new Error(d.message);
expect(d.receipt).toBe(`remote-gate tree=${file.tree.slice(0, 12)} tests=15 runs=200/1,201/1 profiles=test,e2e heads=${head.slice(0, 12)} verified=2026-10-29T12:00:00.000Z`);
expect(d.substitution).toBeUndefined();
const cleared = clearObligation(file, LANE, d.receipt, 'test');
expect(verifyPregate(repo.work, cleared, { tree: file.tree } as any)).toMatchObject({ ok: true });
const out = path.join(repo.work, '.gstack/tmp/pregate.json');
fs.mkdirSync(path.dirname(out), { recursive: true });
fs.writeFileSync(out, JSON.stringify(cleared));
const bound = runBin(repo, 'gstack-ship-receipt', ['write', '--pr', '7', '--base', 'main', '--version', '1.0.0', '--session-kind', 'interactive', '--pregate', out, '--json']);
expect([bound.status, bound.err]).toEqual([EXIT.ok, expect.any(String)]);
expect(bound.out).toContain(d.receipt);
}, 120_000);
test('runs on a stamp-only predecessor are accepted for the stamped head and recorded as a substitution', async () => {
const repo = fixture();
const gated = repo.git('rev-parse', 'HEAD');
stamp(repo);
const file = await pregateAt(repo);
const d = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 200, profile: 'test', head: gated, files: ['test/a.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'e2e', head: gated, files: ['test/b.test.ts'] }]) }, repo.work);
if ('refused' in d) throw new Error(d.message);
expect(d.receipt).toContain(`accepted-for=${file.tree.slice(0, 12)} stamp-only=3 runs=200/1,201/1`);
expect(d.substitution?.accepted_for).toBe(file.tree);
expect(verifyPregate(repo.work, clearObligation(file, LANE, d.receipt, 'test', d.substitution), { tree: file.tree } as any)).toMatchObject({ ok: true });
}, 120_000);
test('refusals: a non-stamp difference, an uncovered inventory file, a failed run, a missing profile, no remote_gate_runs', async () => {
const repo = fixture();
const older = repo.git('rev-parse', 'HEAD~1');
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const both = (h: string, files: string[][] = [['test/a.test.ts', 'test/util.test.js'], ['test/b.test.ts']], conclusion?: string): FakeRun[] => [{ id: 200, profile: 'test', head: h, files: files[0]!, conclusion }, { id: 201, profile: 'e2e', head: h, files: files[1]! }];
const msg = (d: ReturnType<typeof clearDecision>) => ('refused' in d ? `${d.refused}: ${d.message}` : `cleared ${d.receipt}`);
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(older)) }, repo.work))).toMatch(/^PREGATE_RECEIPT_INVALID: .*lib\/util\.js/);
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, [['test/a.test.ts', 'test/util.test.js'], ['test/c.test.ts']])) }, repo.work))).toContain('1 of the 3 test file(s) pregate.json/remote_gate must cover appear in no run\'s tree.json (test/b.test.ts)');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, [['test/a.test.ts', 'test/b.test.ts'], ['test/b.test.ts']])) }, repo.work))).toContain('(test/util.test.js)');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, [['test/a.test.ts', 'test/util.test.js'], []])) }, repo.work))).toContain('reason=no_tests — run 201 (e2e) records no tests');
expect(msg(clearDecision(file, LANE, { runs: ['200', 'https://github.com/fx/fx/actions/runs/200'], evidence: check(repo, both(head)) }, repo.work))).toBe('usage: pregate.json/remote_gate needs a different run for each profile; got 200, https://github.com/fx/fx/actions/runs/200');
expect(msg(clearDecision({ ...file, base: 'origin/release@0123456789ab' }, LANE, { runs: ['200', '201'], evidence: check(repo, both(head)) }, repo.work))).toMatch(/^PREGATE_RECEIPT_INVALID: REUSE: refused reason=base_mismatch/);
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head, undefined, 'failure')) }, repo.work))).toContain('reason=no_verified_run');
expect(msg(clearDecision(file, LANE, { runs: ['200'], evidence: check(repo, both(head)) }, repo.work))).toBe('usage: pregate.json/remote_gate clears with exactly 2 run(s), one per profile (test, e2e); got 1');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head), { gateRuns: null }) }, repo.work))).toContain('only when the base branch\'s .gstack/pregate.json declares remote_gate_runs');
expect(msg(clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, both(head), { inventory: () => ({ error: 'boom' }) }) }, repo.work))).toContain('inventory command failed: boom');
}, 120_000);
test('runs that tested different trees, or that record no tests, never clear the gate', async () => {
const repo = fixture();
const older = repo.git('rev-parse', 'HEAD~1');
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const split = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/b.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'e2e', head: older, files: ['test/b.test.ts'] }]) }, repo.work);
expect('refused' in split && split.message).toContain('the runs tested different trees');
const empty = clearDecision(file, LANE, { runs: ['200', '201'], evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/util.test.js'] }, { id: 201, profile: 'e2e', head, files: ['test/b.test.ts'] }], { inventory: () => [] }) }, repo.work);
expect('refused' in empty && empty.message).toContain('inventory command listed no test files');
}, 120_000);
test('a run valid for two profiles does not starve the other: the profile-to-run assignment is searched, not taken greedily', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const profiles = { ...PROFILES, wide: PROFILES.test!, narrow: { ...PROFILES.test!, max_age_hours: 1 } };
const runs: FakeRun[] = [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'test', head, files: ['test/b.test.ts'], updated: '2026-10-29T11:30:00Z' }];
const d = clearDecision(file, LANE, { runs: ['201', '200'], evidence: check(repo, runs, { profiles, gateRuns: { profiles: ['wide', 'narrow'], inventory: 'true' } }) }, repo.work);
if ('refused' in d) throw new Error(d.message);
expect(d.receipt).toContain('runs=200/1,201/1 profiles=wide,narrow ');
}, 120_000);
test('a lane with its own profile clears from --run-url through the verifier; a lane without one keeps the plain run-URL record, and so does a repository whose run profiles cannot be read (a non-GitHub origin), with a warning', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const base = await pregateAt(repo);
const PLATFORM = 'test.yml/full-suite';
const file: PregateFile = { ...base, requires_remote: [...base.requires_remote!, { lane: PLATFORM, platform: 'macos', workflow: 'test.yml', job: 'full-suite', tests: ['test/a.test.ts'], receipt: null }] };
const url = 'https://github.com/fx/fx/actions/runs/200';
const profiled = clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts'] }], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
if ('refused' in profiled) throw new Error(profiled.message);
expect(profiled).toMatchObject({ attested: false });
expect(profiled.receipt).toStartWith(`run tree=${file.tree.slice(0, 12)} tests=5 runs=200/1 profiles=${PLATFORM} `);
const failed = clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts'], conclusion: 'failure' }], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
expect('refused' in failed && failed.refused).toBe('PREGATE_RECEIPT_INVALID');
expect(clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, []) }, repo.work)).toEqual({ receipt: url, attested: false });
const bundled = clearDecision(file, PLATFORM, { bundleLane: 'macos', evidence: check(repo, [], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
expect('refused' in bundled && `${bundled.refused}: ${bundled.message}`).toBe(`usage: ${PLATFORM} has a run profile on the base branch; clear it with --run <id|url>, which verifies the run`);
expect(clearDecision(file, PLATFORM, { bundleLane: 'macos', evidence: check(repo, []) }, repo.work)).toEqual({ receipt: 'bundle-lane:macos', attested: false });
const unprofiled = clearDecision(file, PLATFORM, { runs: ['200'], evidence: check(repo, []) }, repo.work);
expect('refused' in unprofiled && `${unprofiled.refused}: ${unprofiled.message}`).toContain(`usage: --run clears ${PLATFORM} only when the base branch's .gstack/pregate.json declares a run_profiles entry named`);
const gateFromUrl = clearDecision(file, LANE, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/b.test.ts', 'test/util.test.js'] }]) }, repo.work);
expect('refused' in gateFromUrl && `${gateFromUrl.refused}: ${gateFromUrl.message}`).toBe('usage: pregate.json/remote_gate clears with exactly 2 run(s), one per profile (test, e2e); got 1');
const oneProfile = clearDecision(file, LANE, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/b.test.ts', 'test/util.test.js'] }], { gateRuns: { profiles: ['test'], inventory: 'true' } }) }, repo.work);
if ('refused' in oneProfile) throw new Error(oneProfile.message);
expect(oneProfile.receipt).toStartWith(`remote-gate tree=${file.tree.slice(0, 12)} tests=15 runs=200/1 profiles=test `);
const uncovered = clearDecision(file, PLATFORM, { runUrl: url, evidence: check(repo, [{ id: 200, profile: 'test', head, files: ['test/smoke.test.ts'] }], { profiles: { ...PROFILES, [PLATFORM]: PROFILES.test! } }) }, repo.work);
expect('refused' in uncovered && uncovered.message).toContain(`reason=tests_uncovered — 1 of the 1 test file(s) ${PLATFORM} must cover appear in no run's tree.json (test/a.test.ts)`);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', PLATFORM, '--run-url', url]);
expect([r.status, r.err]).toEqual([EXIT.ok, expect.stringContaining('origin is not a github.com repository')]);
expect(r.err).toContain(`${PLATFORM} was recorded without run verification`);
const viaRun = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run', '200', '--run', '201', '--allow-repo-commands']);
expect([viaRun.status, viaRun.err]).toEqual([EXIT.fail, expect.stringContaining('origin is not a github.com repository')]);
}, 120_000);
test('a lane the base profiles never falls back to an unverified record when the GitHub origin cannot be read', async () => {
const repo = fixture({ run_profiles: PROFILES, remote_gate_runs: GATE_RUNS });
const file = await pregateAt(repo);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run-url', 'https://github.com/fx/fx/actions/runs/200']);
expect([r.status, r.err]).toEqual([EXIT.fail, expect.stringContaining(`${LANE} may have a run profile on the base branch, so it clears only with a verified --run`)]);
expect(JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')).requires_remote[0].receipt).toBeNull();
}, 120_000);
test('an unreadable base config refuses --run-url even on a lane with no profile, since the base may profile it', async () => {
const repo = fixture();
const base = await pregateAt(repo);
const PLATFORM = 'test.yml/full-suite';
const file: PregateFile = { ...base, requires_remote: [...base.requires_remote!, { lane: PLATFORM, platform: 'macos', workflow: 'test.yml', job: 'full-suite', tests: ['test/a.test.ts'], receipt: null }] };
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
repo.git('update-ref', '-d', 'refs/remotes/origin/main');
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', PLATFORM, '--run-url', 'https://github.com/fx/fx/actions/runs/200']);
expect([r.status, r.err]).toEqual([EXIT.fail, expect.stringContaining(`${PLATFORM} may have a run profile on the base branch, so it clears only with a verified --run`)]);
expect(JSON.parse(fs.readFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), 'utf8')).requires_remote.find((o: { lane: string }) => o.lane === PLATFORM).receipt).toBeNull();
}, 120_000);
test('offline verify refuses a run receipt recorded for another tree', async () => {
const repo = fixture();
const file = await pregateAt(repo);
const forged = clearObligation(file, LANE, `remote-gate tree=${'0'.repeat(12)} tests=10 runs=200/1 profiles=test heads=${'1'.repeat(12)} verified=2026-10-29T12:00:00.000Z`, 'test');
expect(verifyPregate(repo.work, forged, { tree: file.tree } as any)).toMatchObject({ ok: false, code: 'PREGATE_REMOTE_UNCLEARED' });
}, 120_000);
test('every run is read again once the clear is assembled: a run that turns red while another profile or the inventory is read refuses', async () => {
const repo = fixture();
const head = repo.git('rev-parse', 'HEAD');
const file = await pregateAt(repo);
const base = check(repo, [{ id: 200, profile: 'test', head, files: ['test/a.test.ts', 'test/util.test.js'] }, { id: 201, profile: 'e2e', head, files: ['test/b.test.ts'] }]);
let flipped = false;
const evidence: RunCheck = { ...base, deps: { ...base.deps, run: id => flipped && id === 200 ? { ...base.deps.run(id), run_attempt: 2, conclusion: 'failure' } : base.deps.run(id) }, inventory: () => { flipped = true; return ['test/a.test.ts', 'test/b.test.ts']; } };
const d = clearDecision(file, LANE, { runs: ['200', '201'], evidence }, repo.work);
expect('refused' in d && d.message).toContain('REUSE: refused reason=run_changed — run 200 moved to attempt 2 (failure)');
}, 120_000);
test('origin is read before url.insteadOf, so a Capy git.capy.ai proxy still names the GitHub repository', async () => {
const repo = fixture();
const file = await pregateAt(repo);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
repo.git('remote', 'set-url', 'origin', 'https://github.com/fx/fx.git');
repo.git('config', 'url.https://git.capy.ai/.insteadOf', 'https://github.com/');
expect(repo.git('remote', 'get-url', 'origin')).toBe('https://git.capy.ai/fx/fx.git');
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run', '200', '--run', '201', '--allow-repo-commands']);
expect(r.err).not.toContain('origin is not a github.com repository');
expect([r.status, r.err]).toEqual([EXIT.usage, expect.stringContaining('declares remote_gate_runs')]);
}, 120_000);
test('the bin refuses --run when the base declares no run profiles, and keeps the --receipt route unchanged', async () => {
const repo = fixture();
const file = await pregateAt(repo);
fs.mkdirSync(path.join(repo.work, '.gstack/tmp'), { recursive: true });
fs.writeFileSync(path.join(repo.work, '.gstack/tmp/pregate.json'), JSON.stringify(file));
repo.git('remote', 'set-url', 'origin', 'https://github.com/fx/fx.git');
const r = runBin(repo, 'gstack-pregate', ['clear', '--remote', LANE, '--run', '200', '--run', '201', '--allow-repo-commands']);
expect(r.status).toBe(EXIT.usage);
expect(r.err).toContain('--run clears pregate.json/remote_gate only when the base branch\'s .gstack/pregate.json declares remote_gate_runs');
}, 120_000);
});
test/pregate-user-policy.test.ts
+5 -1
View File
@@ -157,12 +157,16 @@ describe('gstack-pregate with a user-owned policy', () => {
expect(invalid.err).toContain(`fix: edit ${userPolicy} directly`);
expect(invalid.err).not.toContain('.gstack/pregate.json has');
for (const [key, value] of [['sealed', { paths: ['evals/**'] }], ['price_file', 'evals/calls.json']] as const) {
const profile = { workflow: 'test.yml', events: ['push'], jobs: ['full-suite'], artifact: 'full-suite-tree' };
for (const [key, value] of [['sealed', { paths: ['evals/**'] }], ['price_file', 'evals/calls.json'], ['run_profiles', { test: profile }], ['remote_gate_runs', { profiles: ['test'], inventory: 'true' }]] as const) {
writePolicy(userPolicy, { [key]: value });
const r = run();
expect(r.status).toBe(EXIT.refused);
expect(r.err).toContain(`covers gstack-pregate run only; ${key}`);
}
// An empty run_profiles is the documented default, not a policy, so it is accepted.
writePolicy(userPolicy, { run_profiles: {}, local: LOCAL_ALL });
expect(run().err).not.toContain('PREGATE_CONFIG_INVALID');
// A null price_file is the documented default, not a policy, so it is accepted.
writePolicy(userPolicy, { price_file: null, local: LOCAL_ALL });
expect(run().err).not.toContain('PREGATE_CONFIG_INVALID');
test/pricing-eval-plan.test.ts
+2 -1
View File
@@ -50,7 +50,8 @@ describe('preregistration', () => {
expect(check.missing).toEqual([]);
expect(check.empty.length).toBe(PREREG_SECTIONS.length);
expect(check.ok).toBe(false);
const filled = t.replace(/^(##\s+.+)$/gm, '$1\n- filled in');
expect(PREREG_SECTIONS).toContain('Comparison');
const filled = t.replace(/^(##\s+.+)$/gm, '$1\n- filled in').replace('## Comparison\n- filled in', '## Comparison\nstack: none\nwhy: one arm, no skill stack');
expect(checkPrereg(filled)).toMatchObject({ ok: true, missing: [], empty: [] });
expect(checkPrereg('# nothing here\n').missing).toEqual([...PREREG_SECTIONS]);
});
test/run-evidence.test.ts
+183
View File
@@ -0,0 +1,183 @@
/**
* lib/run-evidence.ts: a GitHub Actions run as evidence for one tree (GSTA-156). Every refusal reason has a fixture;
* the dependencies are fakes, so nothing here reaches GitHub.
*/
import { describe, expect, test } from 'bun:test';
import { parseRunReceipt, parseRunRef, runInventory, runProfileErrors, runReceiptFields, uncoveredFiles, verifyRun, type ApiJob, type ApiRun, type RunDeps, type RunProfile, type TreeRecord } from '../lib/run-evidence';
const REPO = 'garrytan/gbrain';
const HEAD = 'a'.repeat(40);
const TREE = 'b'.repeat(40);
const NOW = new Date('2026-10-29T12:00:00Z');
const PROFILE: RunProfile = { workflow: 'test.yml', events: ['workflow_dispatch', 'push'], jobs: ['full-suite'], artifact: 'full-suite-tree' };
interface World { run: ApiRun; others: ApiRun[]; jobs: ApiJob[]; artifact: TreeRecord | Error; trees: Record<string, string | null>; throwOn?: 'run'; later?: ApiRun }
function world(over: Partial<World> = {}): World {
const run: ApiRun = {
id: 100, run_attempt: 1, status: 'completed', conclusion: 'success', event: 'workflow_dispatch', head_sha: HEAD,
path: '.github/workflows/test.yml', created_at: '2026-10-29T10:00:00Z', updated_at: '2026-10-29T10:20:00Z',
repository: { full_name: REPO }, head_repository: { full_name: REPO },
};
return {
run, others: [], jobs: [{ name: 'full-suite', conclusion: 'success' }, { name: 'lint', conclusion: 'success' }],
artifact: { run_id: 100, run_attempt: 1, event: 'workflow_dispatch', sha: HEAD, tree: TREE, files: ['test/a.test.ts', 'test/b.test.ts'], tests: 12 },
trees: { [HEAD]: TREE }, ...over,
};
}
function deps(w: World): RunDeps {
let reads = 0;
return {
run: id => { if (w.throwOn === 'run') throw new Error('HTTP 502'); if (id !== w.run.id) throw new Error('HTTP 404'); return reads++ && w.later ? w.later : w.run; },
runsForHead: (_wf, sha) => [w.run, ...w.others].filter(r => r.head_sha === sha),
jobs: () => w.jobs,
artifact: () => { if (w.artifact instanceof Error) throw w.artifact; return w.artifact; },
commitTree: sha => w.trees[sha] ?? null,
};
}
const check = (w: World, extra: { runRef?: string; profile?: RunProfile; vetoHeads?: string[] } = {}) =>
verifyRun(deps(w), { repo: REPO, runRef: extra.runRef ?? '100', profileName: 'test', profile: extra.profile ?? PROFILE, vetoHeads: extra.vetoHeads, now: NOW });
const reason = (w: World, extra: Parameters<typeof check>[1] = {}) => { const v = check(w, extra); return v.ok ? 'ok' : v.reason; };
describe('verifyRun', () => {
test('a completed same-repository run with its job, artifact and head tree agreeing is evidence for that tree', () => {
const v = check(world());
expect(v).toEqual({ ok: true, evidence: { run_id: 100, run_attempt: 1, head_sha: HEAD, tree: TREE, profile: 'test', event: 'workflow_dispatch', finished_at: '2026-10-29T10:20:00Z', files: ['test/a.test.ts', 'test/b.test.ts'], tests: 12 } });
});
test('a run URL on this repository is accepted; another repository, a malformed ref and an unknown host are not', () => {
expect(reason(world(), { runRef: `https://github.com/${REPO}/actions/runs/100/attempts/1` })).toBe('ok');
expect(reason(world(), { runRef: 'https://github.com/someone/else/actions/runs/100' })).toBe('run_ref_invalid');
expect(reason(world(), { runRef: 'https://example.com/garrytan/gbrain/actions/runs/100' })).toBe('run_ref_invalid');
expect(reason(world(), { runRef: '100; rm -rf /' })).toBe('run_ref_invalid');
});
test('each refusal names its reason', () => {
const base = world();
const cases: Array<[string, World]> = [
['other_repository', world({ run: { ...base.run, repository: { full_name: 'x/y' } } })],
['fork_run', world({ run: { ...base.run, head_repository: { full_name: 'fork/gbrain' } } })],
['workflow_mismatch', world({ run: { ...base.run, path: '.github/workflows/e2e.yml' } })],
['event_mismatch', world({ run: { ...base.run, event: 'pull_request' } })],
['run_in_progress', world({ run: { ...base.run, status: 'in_progress', conclusion: null } })],
['run_not_success', world({ run: { ...base.run, conclusion: 'failure' } })],
['run_not_success', world({ run: { ...base.run, conclusion: 'cancelled' } })],
['run_stale', world({ run: { ...base.run, updated_at: '2026-10-28T10:00:00Z' } })],
['job_missing', world({ jobs: [{ name: 'lint', conclusion: 'success' }] })],
['job_ambiguous', world({ jobs: [{ name: 'full-suite', conclusion: 'success' }, { name: 'full-suite', conclusion: 'success' }] })],
['job_not_success', world({ jobs: [{ name: 'full-suite', conclusion: 'skipped' }] })],
['artifact_missing', world({ artifact: new Error('no artifact named full-suite-tree') })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), run_id: 99 } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), run_attempt: 2 } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), sha: 'c'.repeat(40) } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), head_sha: 'c'.repeat(40) } })],
['artifact_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), sha: undefined } })],
['artifact_reused', world({ artifact: { ...(base.artifact as TreeRecord), reused: true } })],
['tree_mismatch', world({ artifact: { ...(base.artifact as TreeRecord), tree: 'd'.repeat(40) } })],
['api_error', world({ trees: {} })],
['api_error', world({ throwOn: 'run' })],
];
for (const [want, w] of cases) expect([want, reason(w)]).toEqual([want, want]);
});
test('the newest attempt decides: a run whose latest attempt failed is refused even if an earlier attempt passed', () => {
const w = world();
expect(reason(world({ run: { ...w.run, run_attempt: 2, conclusion: 'failure' }, artifact: { ...(w.artifact as TreeRecord), run_attempt: 2 } }))).toBe('run_not_success');
});
test('a newer failed, cancelled or running run of the workflow on the head vetoes; an older one or a newer green does not', () => {
const w = world();
const newer = (status: string, conclusion: string | null): ApiRun => ({ ...w.run, id: 101, created_at: '2026-10-29T11:00:00Z', status, conclusion });
expect(reason(world({ others: [newer('completed', 'failure')] }))).toBe('newer_run_vetoes');
expect(reason(world({ others: [newer('completed', 'cancelled')] }))).toBe('newer_run_vetoes');
expect(reason(world({ others: [newer('in_progress', null)] }))).toBe('newer_run_vetoes');
expect(reason(world({ others: [newer('completed', 'success')] }))).toBe('ok');
expect(reason(world({ others: [{ ...newer('completed', 'failure'), created_at: '2026-10-29T09:00:00Z' }] }))).toBe('ok');
// A re-run keeps its run's created_at; its newest attempt started after the candidate, so its red result still vetoes.
expect(reason(world({ others: [{ ...newer('completed', 'failure'), created_at: '2026-10-29T09:00:00Z', run_started_at: '2026-10-29T11:00:00Z' }] }))).toBe('newer_run_vetoes');
// Re-running the candidate itself does not hide a failure that started after it was first created.
expect(reason(world({ run: { ...w.run, run_started_at: '2026-10-29T12:00:00Z' }, others: [newer('completed', 'failure')] }))).toBe('newer_run_vetoes');
});
test('a newer failure on another head the consumer names (same tree, the lane\'s own head) vetoes too', () => {
const w = world();
const other = 'e'.repeat(40);
const red: ApiRun = { ...w.run, id: 102, head_sha: other, created_at: '2026-10-29T11:30:00Z', conclusion: 'failure' };
expect(reason(world({ others: [red] }))).toBe('ok');
expect(reason(world({ others: [red] }), { vetoHeads: [other] })).toBe('newer_run_vetoes');
});
test('the run is read again after the evidence: a newer attempt of the same run, seen or started meanwhile, refuses', () => {
const w = world();
expect(reason(world({ later: { ...w.run, run_attempt: 2, status: 'in_progress', conclusion: null } }))).toBe('run_changed');
expect(reason(world({ others: [{ ...w.run, run_attempt: 2, run_started_at: '2026-10-29T11:00:00Z', status: 'completed', conclusion: 'failure' }] }))).toBe('newer_run_vetoes');
});
test('tree.json counts and file lists must be well formed', () => {
const art = world().artifact as TreeRecord;
expect(reason(world({ artifact: { ...art, tests: -1 } }))).toBe('artifact_mismatch');
expect(reason(world({ artifact: { ...art, tests: 1.5 } }))).toBe('artifact_mismatch');
expect(reason(world({ artifact: { ...art, files: 'test/a.test.ts' } }))).toBe('artifact_mismatch');
const v = check(world({ artifact: { ...art, files: ['./test/a.test.ts'], tests: undefined } }));
expect(v.ok && [v.evidence.files, v.evidence.tests]).toEqual([['test/a.test.ts'], null]);
});
test('a profile that names an OS needs the artifact to record it', () => {
const w = world();
expect(reason(w, { profile: { ...PROFILE, os: 'Linux' } })).toBe('os_mismatch');
expect(reason(world({ artifact: { ...(w.artifact as TreeRecord), os: 'Linux' } }), { profile: { ...PROFILE, os: 'Linux' } })).toBe('ok');
});
test('a failed run created in the same second as the candidate vetoes it: ties count as newer', () => {
const w = world();
expect(reason(world({ others: [{ ...w.run, id: 101, status: 'completed', conclusion: 'failure' }] }))).toBe('newer_run_vetoes');
});
test('freshness reads the jobs\' latest completed_at when the API gives it, not a later updated_at', () => {
const w = world();
const jobs: ApiJob[] = [{ name: 'full-suite', conclusion: 'success', completed_at: '2026-10-28T10:00:00Z' }, { name: 'lint', conclusion: 'success', completed_at: '2026-10-28T09:00:00Z' }];
expect(reason(world({ jobs }))).toBe('run_stale');
const v = check(world({ jobs: jobs.map(j => ({ ...j, completed_at: '2026-10-29T10:10:00Z' })) }));
expect(v.ok && v.evidence.finished_at).toBe('2026-10-29T10:10:00Z');
expect(w.run.updated_at).toBe('2026-10-29T10:20:00Z');
});
test('max_age_hours widens the freshness window', () => {
const stale = world({ run: { ...world().run, updated_at: '2026-10-28T10:00:00Z' } });
expect(reason(stale)).toBe('run_stale');
expect(reason(stale, { profile: { ...PROFILE, max_age_hours: 48 } })).toBe('ok');
});
});
describe('runInventory', () => {
test('lists unique non-empty lines, and a pipeline whose producer fails refuses instead of listing part of the gate', () => {
const cwd = process.cwd();
expect(runInventory("printf 'test/b.test.ts\\n\\ntest/a.test.ts\\ntest/b.test.ts\\n'", cwd)).toEqual(['test/b.test.ts', 'test/a.test.ts']);
expect(runInventory("{ printf 'test/a.test.ts\\n'; exit 17; } | sort", cwd)).toMatchObject({ error: expect.stringContaining('exited 17') });
});
});
describe('profiles, coverage and receipts', () => {
test('runProfileErrors accepts a well-formed pair and names every malformed field', () => {
expect(runProfileErrors({ test: PROFILE, e2e: { ...PROFILE, workflow: 'e2e.yml', jobs: ['e2e-full-suite'], artifact: 'e2e-full-suite-tree', os: 'Linux', max_age_hours: 12 } }, { profiles: ['test', 'e2e'], inventory: 'bash scripts/pregate-remote-gate.sh --list' })).toEqual([]);
const errors = runProfileErrors({ bad: { workflow: '../x.yml', events: ['pull_request'], jobs: [], artifact: '', max_age_hours: 0, os: 'BeOS', extra: 1 } }, { profiles: ['missing'], inventory: 'a\nb' }).map(e => e.path);
expect(errors).toEqual(expect.arrayContaining(['run_profiles["bad"].extra', 'run_profiles["bad"].workflow', 'run_profiles["bad"].events', 'run_profiles["bad"].jobs', 'run_profiles["bad"].artifact', 'run_profiles["bad"].max_age_hours', 'run_profiles["bad"].os', 'remote_gate_runs.profiles', 'remote_gate_runs.inventory']));
expect(runProfileErrors({ test: PROFILE }, { profiles: ['test', 'test'], inventory: 'true' })).toEqual([{ path: 'remote_gate_runs.profiles', message: 'names each profile once' }]);
});
test('uncoveredFiles lists the inventory files no run recorded', () => {
const ev = (files: string[]) => ({ run_id: 1, run_attempt: 1, head_sha: HEAD, tree: TREE, profile: 'p', event: 'push', finished_at: '', files, tests: null });
expect(uncoveredFiles(['a', 'b', 'c'], [ev(['a']), ev(['b'])])).toEqual(['c']);
expect(uncoveredFiles(['a'], [ev(['a', 'z'])])).toEqual([]);
expect(uncoveredFiles(['./test/x.test.ts'], [ev(['test/x.test.ts'])])).toEqual([]);
});
test('the receipt fields round-trip and keep the tree prefix verify reads', () => {
const v = check(world());
if (!v.ok) throw new Error(v.detail);
const receipt = `remote-gate tree=${TREE.slice(0, 12)} tests=12 ${runReceiptFields([v.evidence], NOW)}`;
expect(receipt).toBe(`remote-gate tree=${TREE.slice(0, 12)} tests=12 runs=100/1 profiles=test heads=${HEAD.slice(0, 12)} verified=2026-10-29T12:00:00.000Z`);
expect(parseRunReceipt(receipt)).toEqual({ runs: ['100/1'], profiles: ['test'], tree: TREE.slice(0, 12) });
expect(parseRunReceipt(`remote-gate tree=${TREE.slice(0, 12)} tests=3 receipt=/x sha256=abc`)).toBeNull();
expect(parseRunRef('123', REPO)).toBe(123);
});
});
test/size-ledger/ship/gsta157-base-red-pause.json
+4
View File
@@ -0,0 +1,4 @@
{
"bytes": 1699,
"why": "GSTA-157: Step 5 red-base pause (gstack-base-red classify before T3, resume after BASE_GREEN) and the conditional receipt flag in the PR-body step"
}
test/timeline-stop-hook-slug.test.ts
+1
View File
@@ -33,6 +33,7 @@ const HOOK_FILES = [
'lib/result-codes-release.ts',
'lib/result-codes-capy.ts',
'lib/result-codes-budget.ts',
'lib/result-codes-eval.ts',
];
const SLUG = 'stop-hook-slug-test';
const BUDGET_MS = 2000;
test/ubicloud-pregate-remote.test.ts
+48 -3
View File
@@ -37,12 +37,22 @@ const UBI_RUNNER_STUB = `#!/usr/bin/env bash
# VM directory, runs setup and the command there, and prints ubi-runner's log
# lines. STUB_FAIL picks a failure: create, quota, quota-once, ssh. STUB_HANG
# sleeps before the command; TERM tears down as ubi-runner's trap does.
# Every teardown is appended to $STUB_CALLS.down.
# Every teardown is appended to $STUB_CALLS.down. STUB_REATTACH=<vm>:<created_at>
# plays a live run record for this key: --probe reports it (busy per
# STUB_REATTACH_BUSY) and run prints ubi-runner's reattached line instead of
# creating a VM.
set -uo pipefail
[ "$1" != owner ] || { echo stub; exit 0; }
left() { [ "\${STUB_LEFT:-0}" = none ] || echo "UBICLOUD: left=\${STUB_LEFT:-0} owner=stub vm=$name others=0" | tee -a "\${UBI_LEFT_FILE:-/dev/null}" >&2; }
[ "$1" = run ] || exit 2
shift
case " $* " in
*" --probe "*)
printf '%s\\n' "$*" >>"$STUB_CALLS.probe"
if [ -n "\${STUB_REATTACH:-}" ]; then echo "UBI_ATTACH_INFO: vm=\${STUB_REATTACH%%:*} created_at=\${STUB_REATTACH#*:} key=0123456789ab busy=\${STUB_REATTACH_BUSY:-0}"
else echo "UBI_ATTACH_INFO: none key=0123456789ab"; fi
exit 0 ;;
esac
printf '%s\\n' "$*" >>"$STUB_CALLS"
src=. setup="" size="" name=unnamed envs=()
while [ $# -gt 0 ]; do
@@ -52,16 +62,20 @@ while [ $# -gt 0 ]; do
--src) src=$2; shift 2 ;;
--setup) setup=$2; shift 2 ;;
--env) envs+=("$2"); shift 2 ;;
--pull) shift 2 ;;
--pull|--key-input) shift 2 ;;
--) shift; break ;;
*) echo "stub: unknown option $1" >&2; exit 2 ;;
esac
done
cmd=$*
log() { echo "ubi-runner: $*" >&2; }
log "creating eu-central-h1/$name ($size)"
calls=$(wc -l <"$STUB_CALLS")
fail=\${STUB_FAIL:-}
if [ -n "\${STUB_REATTACH:-}" ]; then
name=\${STUB_REATTACH%%:*}
log "reattached to $name created_at=\${STUB_REATTACH#*:} key=0123456789ab"
else
log "creating eu-central-h1/$name ($size)"
if [ "$fail" = quota ] || { [ "$fail" = quota-once ] && [ "$calls" -eq 1 ]; }; then
log "quota refused eu-central-h1/$name ($size): it needs 16 vCPUs and the project already uses 64 of 64"
log "create failed: vCPU quota exhausted; wait for running VMs to finish, or use fewer or smaller VMs (ci:ubicloud --vms N)"
@@ -77,6 +91,7 @@ if [ "$fail" = create ]; then
exit 1
fi
log "ready: $name (192.0.2.1)"
fi
trap 'log "destroyed eu-central-h1/$name"; left; echo "term $name" >>"$STUB_CALLS.down"; exit 130' TERM
vm=$STUB_VM_HOME
dest="$vm/work/$(basename "$src")"
@@ -282,6 +297,36 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
expect(twice.out).toContain('create failed: vCPU quota exhausted');
}, 90_000);
test('a rerun for the same tree and planned set reattaches: the receipt still parses, spent= bills from the original created_at, and a used-up wall budget refuses', () => {
const created = Math.floor(Date.now() / 1000) - 5 * 60 - 10;
const f = fixture();
const r = gate(f, ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}` });
expect(r.exit, r.out).toBe(0);
expect(r.out).toContain('reattaching to VM ubirun-stub-old, created 6 min ago by an earlier run of this gate for the same tree and planned set');
expect(r.out).toContain("the VM is told to stop after 2160s (the cap's 45 min wall minus 3 min for teardown and 6 min already billed since ubirun-stub-old was created)");
expect(r.out).not.toContain('if this run is killed before its teardown line');
expect(r.out.trim().split('\n').at(-1)).toMatch(/^PREGATE_REMOTE: /);
expect(r.report).toMatchObject({ tree: f.tree, tests: 1, failed: 0 });
// Billed from created_at (5 min 10 s before the gate started), not from the reattach: 6 min at $0.00831.
expect(r.report!.spent).toBe('0.05');
expect(remoteReportProblem(r.report!, f.tree)).toBeNull();
const plan = new RegExp(`--key-input pregate-remote tree=${f.tree} plan=[0-9a-f]{64}`);
expect(r.calls[0]).toMatch(plan);
expect(fs.readFileSync(f.calls + '.probe', 'utf8')).toMatch(plan);
const spent = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created - 40 * 60}` });
expect(spent.exit, spent.out).toBe(5);
expect(spent.out).toContain("PREGATE_REMOTE_CAP_SPENT: VM ubirun-stub-old, created 46 min ago for this tree and planned set, has already used the cap's 45 min wall ($0.39 of $2)");
expect(spent.out).toContain('nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down ubirun-stub-old');
expect(spent.report).toBeNull();
expect(spent.calls).toEqual([]);
const busy = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}`, STUB_REATTACH_BUSY: '1' });
expect(busy.exit, busy.out).toBe(4);
expect(busy.out).toContain('PREGATE_REMOTE_ATTACH_BUSY: another gate on this machine still runs VM ubirun-stub-old');
expect(busy.calls).toEqual([]);
}, 90_000);
test('the VM refuses a tree that differs after unpack, a planned set that differs from the list, and a tree that moved during the run', () => {
const edited = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_VM_EDIT: 'test/alpha.test.ts' });
expect(edited.exit, edited.out).toBe(3);
test/ubicloud-runner.test.ts
+391 -31
View File
@@ -389,18 +389,23 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
/**
* Fake Ubicloud API that can create: create adds the name to the listed file
* (or answers the quota refusal in refuse with HTTP 400), show is running at
* 192.0.2.1 while listed, destroy unlists. The fake ssh records every remote
* command in ssh.log and plays the role the command asks for: it stores
* ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, drains tar
* uploads, runs the setup phase per setup-mode (ok or hang) and echoes the
* command phase. No network, no VM.
* 192.0.2.1 while listed, destroy unlists. The fake VM is the directory vm/:
* the fake ssh records every remote command in ssh.log, keeps copies of
* ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, serves pulls
* from remote-fs/, and runs every other command there for real (HOME=vm/,
* vmbin/ first on PATH), so uploads unpack, detached phases write their pid,
* output and exit files, tail --pid follows them and pkill ends them. The stub
* make in vmbin/ echoes `ran: make ARGS`, or runs command-mode or
* make-<target>.sh from the fixture root. No network, no VM.
*/
function fakeVm(opts: { listed?: string; refuse?: string } = {}) {
const root = mkdtempSync(join(tmpdir(), 'ubi-run-'));
const bin = join(root, 'bin'), state = join(root, 'state'), log = join(root, 'calls.log'), sshLog = join(root, 'ssh.log'), record = join(root, 'vms.tsv');
const vm = join(root, 'vm'), vmbin = join(root, 'vmbin'), srcDir = join(root, 'src'), checkpoints = join(root, 'checkpoints');
mkdirSync(vm); mkdirSync(vmbin); mkdirSync(srcDir);
writeFileSync(join(srcDir, 'README'), 'fixture checkout\n');
mkdirSync(bin); mkdirSync(state, { recursive: true });
writeFileSync(join(root, 'listed'), opts.listed ?? '');
writeFileSync(join(root, 'setup-mode'), 'ok');
writeFileSync(join(bin, 'curl'), '#!/usr/bin/env bash\n' +
'out=""; while [ $# -gt 0 ]; do [ "$1" = -o ] && out=$2; shift; done\n' +
'body=$(cat); echo "$body" >> ' + JSON.stringify(log) + '\n' +
@@ -415,26 +420,33 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
' *) : > "$out"; printf 200 ;;\n' +
'esac\n');
writeFileSync(join(bin, 'ssh'), '#!/usr/bin/env bash\n' +
'root=' + JSON.stringify(root) + '\n' +
'root=' + JSON.stringify(root) + '; vm="$root/vm"\n' +
'cmd=${*: -1}\n' +
'printf \'%s\\n\' "$cmd" >> "$root/ssh.log"\n' +
'export HOME="$vm" PATH="$root/vmbin:$PATH"\n' +
'cd "$vm" || exit 255\n' +
'case "$cmd" in\n' +
' *pkill*) echo "$cmd" >> "$root/cancel.log" ;;\n' +
' *pkill*) echo "$cmd" >> "$root/cancel.log"; exec bash -c "$cmd" ;;\n' +
' *captured*) cat > "$root/captured" ;;\n' +
' *"cat > ~/.ubirun-env"*) cat > "$root/ubirun-env" ;;\n' +
' *"cat > ~/.ubirun-setup.sh"*) cat > "$root/ubirun-setup.sh" ;;\n' +
' *"tar -xzf -"*) cat > /dev/null ;;\n' +
' *"cat > ~/.ubirun-env"*) tee "$root/ubirun-env" > "$vm/.ubirun-env" ;;\n' +
' *"cat > ~/.ubirun-setup.sh"*) tee "$root/ubirun-setup.sh" > "$vm/.ubirun-setup.sh" ;;\n' +
' *"tar -czf -"*) base=${cmd##* }; tar -czf - -C "$root/remote-fs" "$base" ;;\n' +
' *"bash -l ~/.ubirun-setup.sh"*) echo "setup start"; if [ "$(cat "$root/setup-mode")" = hang ]; then sleep 20; fi; echo "setup done" ;;\n' +
' *"bash -lc "*) if [ -f "$root/command-mode" ]; then bash -c "$(cat "$root/command-mode")"; else echo "ran: $cmd"; fi ;;\n' +
' *) : ;;\n' +
' *"ls -d -- "*) : ;;\n' +
' *) exec bash -c "$cmd" ;;\n' +
'esac\n');
chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755);
const env: Record<string, string> = { ...(process.env as Record<string, string>), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0' };
writeFileSync(join(vmbin, 'make'), '#!/usr/bin/env bash\n' +
'root=' + JSON.stringify(root) + '\n' +
'if [ -f "$root/command-mode" ]; then exec bash -c "$(cat "$root/command-mode")"; fi\n' +
'echo "ran: make $*"\n' +
'[ ! -f "$root/make-$1.sh" ] || exec bash "$root/make-$1.sh"\n');
chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755); chmodSync(join(vmbin, 'make'), 0o755);
const env: Record<string, string> = { ...(process.env as Record<string, string>), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0', UBI_CHECKPOINT_DIR: checkpoints };
const run = (args: string[], extra: Record<string, string> = {}, stdin?: string) =>
Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: stdin === undefined ? undefined : Buffer.from(stdin), timeout: 90_000 });
const spawn = (args: string[], extra: Record<string, string> = {}) =>
Bun.spawn(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: 'ignore', stdout: 'pipe', stderr: 'pipe', timeout: 120_000 });
return {
root, state, record, run,
root, state, record, run, spawn, vm, src: srcDir, checkpoints,
calls: () => (existsSync(log) ? readFileSync(log, 'utf8') : ''),
ssh: () => (existsSync(sshLog) ? readFileSync(sshLog, 'utf8') : ''),
listed: () => readFileSync(join(root, 'listed'), 'utf8'),
@@ -455,17 +467,18 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
test('run: a hung setup is killed by the watchdog, exits 124, never reaches the command, and the VM is destroyed', () => {
const f = fakeVm();
try {
writeFileSync(join(f.root, 'setup-mode'), 'hang');
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo hi\n');
const r = run(f, ['run', '--src', f.root, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' });
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo "setup start"; sleep 20\n');
const r = run(f, ['run', '--src', f.src, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' });
const err = r.stderr.toString();
expect(r.exitCode, err).toBe(124);
expect(err).toMatch(/watchdog: setup printed nothing for \d+s \(UBI_IDLE_TIMEOUT=2\)/);
expect(err).toContain('setup failed (exit 124)');
expect(f.ssh()).not.toContain('bash -lc');
expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/cat ~\/\.ubirun-setup-\d+\.sid/);
expect(f.ssh()).toMatch(/setsid -w bash -c 'printf %s \$\$ >~\/\.ubirun-setup-\d+\.sid; cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh'; else bash -c '[^']*'; fi$/m);
expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/rm -f ~\/\.ubirun-setup-\d+\.sid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/);
expect(err).toContain('setup start');
// The setup runs detached in its own session: its leader's pid, both streams and the exit status are files on the VM.
expect(f.ssh()).toMatch(/setsid -f bash -c 'echo \$\$ >~\/\.ubirun-setup\.pid\.tmp && mv ~\/\.ubirun-setup\.pid\.tmp ~\/\.ubirun-setup\.pid; \( cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh \) >~\/\.ubirun-setup\.out 2>~\/\.ubirun-setup\.err <\/dev\/null; echo \$\? >~\/\.ubirun-setup\.exit\.tmp && mv ~\/\.ubirun-setup\.exit\.tmp ~\/\.ubirun-setup\.exit' <\/dev\/null/m);
expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/sid=\$\(cat ~\/\.ubirun-setup\.pid 2>\/dev\/null\); rm -f ~\/\.ubirun-setup\.pid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/);
expect(existsSync(join(f.vm, '.ubirun-setup.exit'))).toBe(false);
expect(f.calls()).toContain('destroy');
expect(err).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3/);
expect(f.listed()).toBe('');
@@ -477,7 +490,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const f = fakeVm();
try {
writeFileSync(join(f.root, 'command-mode'), 'for i in 1 2 3 4 5 6 7 8 9 10; do echo "tick $i"; sleep 1; done');
const r = run(f, ['run', '--src', f.root, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' });
const r = run(f, ['run', '--src', f.src, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' });
expect(r.exitCode).toBe(124);
expect(r.stderr.toString()).toMatch(/watchdog: command ran \d+s \(UBI_RUN_TIMEOUT=3\); killing it/);
expect(r.stderr.toString()).toContain('command exited 124 after');
@@ -489,7 +502,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const f = fakeVm({ listed: 'eu-central-h1 ubirun-thrd1-9999999999-sib1-t3\n' });
try {
const leftFile = join(f.root, 'left');
const r = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile });
const r = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile });
const err = r.stderr.toString();
expect(r.exitCode, err).toBe(0);
expect(err).toMatch(/UBICLOUD: left=0 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
@@ -499,7 +512,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const parser = read('scripts/ubicloud/pregate-remote.sh').match(/left=\$\(sed -n '([^']+)' "\$log\.left"/)![1];
expect(Bun.spawnSync(['sed', '-n', parser, leftFile], { timeout: 30_000 }).stdout.toString()).toBe('0\n');
// Value: protects=run --keep reports its live VM as UBICLOUD: left=1 ... kept=1 and never a clean left=0; fails_when=the kept line is dropped or --keep still prints the teardown line; why_new=no test runs run --keep; seam=none; no_claim=real VM lifetime (fake API and ssh)
const kept = f.run(['run', '--src', f.root, '--keep', '--', 'true'], { UBI_REAP: '0' });
const kept = f.run(['run', '--src', f.src, '--keep', '--', 'true'], { UBI_REAP: '0' });
const keptErr = kept.stderr.toString();
expect(kept.exitCode, keptErr).toBe(0);
expect(keptErr).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 kept=1\n/);
@@ -512,14 +525,14 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const f = fakeVm();
try {
writeFileSync(join(f.root, 'listed.destroy-fails'), '');
const leaked = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
const leaked = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
const err = leaked.stderr.toString();
expect(leaked.exitCode, err).toBe(1);
expect(err).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=(ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3) others=0\n[^]*VM cleanup not confirmed for \1 \(left=1\); destroy it with: \S+ down \1/);
expect(f.listed()).toContain('ubirun-thrd1-');
// Value: protects=a failing command whose VM also leaks keeps its own exit status, not 1; fails_when=run_teardown exits 1 whenever left is not 0; why_new=the leak case above only runs a passing command; seam=none; no_claim=real destroy failure modes (fake API returns 500)
writeFileSync(join(f.root, 'command-mode'), 'exit 7');
const failedLeak = f.run(['run', '--src', f.root, '--', 'false'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
const failedLeak = f.run(['run', '--src', f.src, '--', 'make test'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
expect(failedLeak.exitCode, failedLeak.stderr.toString()).toBe(7);
expect(failedLeak.stderr.toString()).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
const blind = fakeVm();
@@ -529,7 +542,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
writeFileSync(join(blind.root, 'bin/curl'), '#!/usr/bin/env bash\nif [ -f ' + JSON.stringify(join(blind.root, 'armed')) + ' ]; then touch ' + JSON.stringify(join(blind.root, 'listed.list-fails')) + '; fi\nexec ' + JSON.stringify(join(blind.root, 'bin/curl.real')) + ' "$@"\n');
chmodSync(join(blind.root, 'bin/curl'), 0o755);
writeFileSync(join(blind.root, 'command-mode'), 'touch ' + JSON.stringify(join(blind.root, 'armed')));
const r = blind.run(['run', '--src', blind.root, '--', 'true'], { UBI_REAP: '0' });
const r = blind.run(['run', '--src', blind.src, '--', 'make test'], { UBI_REAP: '0' });
expect(r.exitCode, r.stderr.toString()).toBe(1);
expect(r.stderr.toString()).toMatch(/UBICLOUD: left=unknown owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=unknown\n/);
} finally { rmSync(blind.root, { recursive: true, force: true }); }
@@ -551,7 +564,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
test('run: --pass forwards a named local variable only; UBICLOUD_* and non-names are refused before any API call', () => {
const f = fakeVm();
try {
const ok = run(f, ['run', '--src', f.root, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' });
const ok = run(f, ['run', '--src', f.src, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' });
expect(ok.exitCode).toBe(0);
expect(ok.stderr.toString()).toContain('command exited 0 after');
const envFile = readFileSync(join(f.root, 'ubirun-env'), 'utf8');
@@ -561,7 +574,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
expect(ok.stdout.toString() + ok.stderr.toString()).not.toContain('sekret');
for (const bad of ['UBICLOUD_API_KEY', 'UBICLOUD_API_TOKEN', 'X;Y', 'HOME/..']) {
const before = f.calls();
const r = run(f, ['run', '--src', f.root, '--pass', bad, '--', 'true']);
const r = run(f, ['run', '--src', f.src, '--pass', bad, '--', 'true']);
expect(r.exitCode).toBe(1);
expect(r.stderr.toString()).toMatch(bad.startsWith('UBICLOUD_') ? /never forwards its own Ubicloud credentials/ : /not a variable name/);
expect(f.calls()).toBe(before);
@@ -753,4 +766,351 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
}
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 30_000);
// GSTA-158 P3: detached phases, reattach and tier checkpoints.
const until = async (cond: () => boolean, ms: number) => {
const deadline = Date.now() + ms;
while (!cond()) { if (Date.now() > deadline) throw new Error('condition not met in time'); await Bun.sleep(100); }
};
const vmDirs = (f: ReturnType<typeof fakeVm>) => (existsSync(f.state) ? readdirSync(f.state).filter(d => d.startsWith('ubirun-')) : []);
const phaseStarted = (f: ReturnType<typeof fakeVm>, phase: string) => vmDirs(f).some(d => existsSync(join(f.state, d, 'phase.' + phase + '.start')));
const creates = (f: ReturnType<typeof fakeVm>) => f.calls().split('\n').filter(l => l.includes('"create"')).length;
const killSession = async (f: ReturnType<typeof fakeVm>, phase: string) => {
const pidFile = join(f.vm, '.ubirun-' + phase + '.pid');
await until(() => existsSync(pidFile) && readFileSync(pidFile, 'utf8').trim() !== '', 10_000);
const pid = Number(readFileSync(pidFile, 'utf8').trim());
try { process.kill(-pid, 'SIGKILL'); } catch { /* already gone */ }
await until(() => { try { process.kill(pid, 0); return false; } catch { return true; } }, 10_000);
};
const gitSrc = (f: ReturnType<typeof fakeVm>, cfg?: unknown) => {
const dir = cfg === undefined ? join(f.root, 'repo') : src(f, cfg);
mkdirSync(dir, { recursive: true });
writeFileSync(join(dir, 'README'), 'fixture repo\n');
for (const args of [['init', '-q', '-b', 'main'], ['add', '-A'], ['commit', '-qm', 'fixture']]) {
const r = Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', ...args], { cwd: dir, timeout: 30_000 });
if (r.exitCode !== 0) throw new Error('git ' + args.join(' ') + ': ' + r.stderr.toString());
}
return dir;
};
const TIERS = { setup: 'scripts/boot.sh', tiers: [{ name: 't1', command: 'make t1' }, { name: 't2', command: 'make t2' }, { name: 't3', command: 'make t3' }] };
const bootCounts = (f: ReturnType<typeof fakeVm>, dir: string) => writeFileSync(join(dir, 'scripts/boot.sh'), 'echo boot >> ~/setup-count\n');
const verifyLine = (out: string) => out.split('\n').find(l => l.startsWith('UBI_VERIFY: ')) ?? '';
test('run: a SIGKILLed runner leaves its VM; the same command reattaches, skips the finished setup and reports the command\'s exit', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo setup >> ~/setup-count\n');
writeFileSync(join(f.root, 'make-test.sh'), 'echo begin; sleep 4; echo end; exit 3\n');
const args = ['run', '--src', repo, '--setup', setup, '--', 'make test'];
const a = f.spawn(args);
await until(() => vmDirs(f).some(d => existsSync(join(f.state, d, 'command.out')) && readFileSync(join(f.state, d, 'command.out'), 'utf8').includes('begin')), 30_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
const name = vmDirs(f)[0]!;
expect(f.listed()).toContain(name);
const b = f.run(args);
const err = b.stderr.toString();
expect(b.exitCode, err).toBe(3);
expect(err).toMatch(new RegExp('ubi-runner: reattached to ' + name + ' created_at=\\d{10} key=[0-9a-f]{12}\n'));
expect(b.stdout.toString().split('\n').filter(l => l === 'begin' || l === 'end')).toEqual(['begin', 'end']);
expect(err).toMatch(/command exited 3 after \d+s/);
expect(creates(f)).toBe(1);
expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('setup\n');
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: a second live runner of the same command refuses with UBI_ATTACH_BUSY and creates nothing', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 4; echo done\n');
const args = ['run', '--src', repo, '--', 'make test'];
const a = f.spawn(args);
await until(() => phaseStarted(f, 'command'), 30_000);
const b = f.run(args);
expect(b.exitCode).toBe(1);
expect(b.stderr.toString()).toMatch(new RegExp('UBI_ATTACH_BUSY: vm=' + vmDirs(f)[0] + ' key=[0-9a-f]{12}: runner pid ' + a.pid + ' '));
expect(await a.exited).toBe(0);
expect(await new Response(a.stdout).text()).toContain('done');
expect(creates(f)).toBe(1);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: HUP once the command is live detaches (UBI_DETACHED, exit 129, VM kept, no pass value on disk); a rerun reattaches and finishes', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3; echo finished\n');
const args = ['run', '--src', repo, '--pass', 'UBI_TEST_TOKEN', '--', 'make test'];
const a = f.spawn(args, { UBI_TEST_TOKEN: 'hup-secret-value' });
await until(() => phaseStarted(f, 'command'), 30_000);
await Bun.sleep(500);
process.kill(a.pid, 'SIGHUP');
expect(await a.exited).toBe(129);
const name = vmDirs(f)[0]!;
const err = await new Response(a.stderr).text();
expect(err).toMatch(new RegExp('UBI_DETACHED: vm=' + name + ' key=[0-9a-f]{12} reattach: rerun the same command'));
expect(f.listed()).toContain(name);
expect(f.calls()).not.toContain('destroy');
for (const file of readdirSync(join(f.state, name))) {
const p = join(f.state, name, file);
if (lstatSync(p).isFile()) expect(readFileSync(p, 'utf8')).not.toContain('hup-secret-value');
}
const b = f.run(args, { UBI_TEST_TOKEN: 'hup-secret-value' });
expect(b.exitCode, b.stderr.toString()).toBe(0);
expect(b.stderr.toString()).toContain('reattached to ' + name);
expect(b.stdout.toString()).toContain('finished');
expect(creates(f)).toBe(1);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: INT and TERM during a live command still tear the VM down', async () => {
for (const sig of ['SIGTERM', 'SIGINT'] as const) {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3\n');
const a = f.spawn(['run', '--src', repo, '--', 'make test']);
await until(() => phaseStarted(f, 'command'), 30_000);
await Bun.sleep(500);
process.kill(a.pid, sig);
expect(await a.exited).toBe(130);
expect(await new Response(a.stderr).text()).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-/);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}
}, 90_000);
test('run: a remote session gone without an exit status exits 125 with UBI_ATTACH_LOST and the VM name', async () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
writeFileSync(join(f.root, 'make-test.sh'), 'sleep 30\n');
const args = ['run', '--src', repo, '--', 'make test'];
const a = f.spawn(args);
await until(() => existsSync(join(f.vm, '.ubirun-command.pid')) && readFileSync(join(f.vm, '.ubirun-command.pid'), 'utf8').trim() !== '', 30_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
await killSession(f, 'command');
const name = vmDirs(f)[0]!;
const b = f.run(args);
expect(b.exitCode).toBe(125);
expect(b.stderr.toString()).toContain('UBI_ATTACH_LOST: vm=' + name + ' phase=command');
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
test('run: the run key covers tree, command, setup content, env, pass value, image, origin/HEAD and the runner itself; a pass value never appears in it', () => {
const f = fakeVm();
try {
const repo = gitSrc(f);
const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo one\n');
const runner = join(f.root, 'runner-copy.sh'); writeFileSync(runner, readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8'));
const key = (opts: { args?: string[]; env?: Record<string, string>; cmd?: string; script?: string } = {}) => {
const r = Bun.spawnSync(['bash', opts.script ?? join(DIR, 'ubi-runner.sh'), 'run', '--probe', '--src', repo, '--setup', setup, '--env', 'MODE=a', '--pass', 'UBI_TEST_TOKEN', ...(opts.args ?? []), '--', opts.cmd ?? 'make test'],
{ env: { ...process.env, PATH: join(f.root, 'bin') + ':' + process.env.PATH, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: f.state, UBI_OWNER: 'thrd1', UBI_TEST_TOKEN: 'v1', ...(opts.env ?? {}) }, timeout: 30_000 });
const out = r.stdout.toString() + r.stderr.toString();
expect(out).not.toContain('v1');
const m = /^UBI_ATTACH_INFO: none key=([0-9a-f]{12})$/m.exec(out);
if (!m) throw new Error(out);
return m[1]!;
};
const base = key();
expect(key()).toBe(base);
const variants: Record<string, string> = {};
variants.command = key({ cmd: 'make check' });
variants.env = key({ args: ['--env', 'MODE=b'] });
variants.pass = key({ env: { UBI_TEST_TOKEN: 'v2' } });
variants.image = key({ args: ['-b', 'ubuntu-jammy'] });
variants.size = key({ args: ['-s', 'standard-8'] });
writeFileSync(runner, readFileSync(runner, 'utf8') + '# edited\n');
variants.runner = key({ script: runner });
writeFileSync(setup, 'echo two\n');
variants.setup = key();
writeFileSync(setup, 'echo one\n');
expect(key()).toBe(base);
Bun.spawnSync(['git', 'update-ref', 'refs/remotes/origin/HEAD', 'HEAD'], { cwd: repo, timeout: 10_000 });
variants.base = key();
writeFileSync(join(repo, 'new-file.txt'), 'untracked source\n');
variants.tree = key();
for (const [what, k] of Object.entries(variants)) expect({ what, same: k === base }).toEqual({ what, same: false });
expect(new Set(Object.values(variants)).size).toBe(Object.keys(variants).length);
expect(f.calls()).not.toContain('"create"');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 60_000);
test('verify tiers: config refuses command with tiers, bad and duplicate tier names, before any API call', () => {
const f = fakeVm();
try {
const cases: Array<[unknown, RegExp]> = [
[{ command: 'make', tiers: [{ name: 'a', command: 'x' }] }, /UBI_CONFIG_TIERS_AND_COMMAND/],
[{ tiers: [{ name: 'Unit', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name 'Unit' must match/],
[{ tiers: [{ name: '-x', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name '-x'/],
[{ tiers: [{ name: 'a'.repeat(33), command: 'x' }] }, /UBI_CONFIG_TIER_NAME/],
[{ tiers: [{ name: 'a.b', command: 'x' }] }, /UBI_CONFIG_TIER_NAME/],
[{ tiers: [{ name: 'a', command: 'x' }, { name: 'a', command: 'y' }] }, /UBI_CONFIG_TIER_NAME: tier name 'a' appears twice/],
[{ tiers: [{ name: 'a' }] }, /UBI_CONFIG_TIER_NAME: tier 'a' needs a command/],
[{ tiers: [] }, /UBI_CONFIG_TIER_NAME: tiers must be a non-empty list/],
];
for (const [cfg, want] of cases) {
const dir = src(f, cfg);
const r = run(f, ['verify', '--src', dir]);
expect(r.exitCode).toBe(1);
expect(r.stderr.toString()).toMatch(want);
}
expect(f.calls()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 30_000);
test('verify tiers: a run killed during tier 3 with its VM gone resumes on a new VM: setup reruns, t1 and t2 are reused, t3 reruns', async () => {
const f = fakeVm();
try {
const dir = gitSrc(f, TIERS);
bootCounts(f, dir);
Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 });
writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n');
const a = f.spawn(['verify', '--src', dir]);
await until(() => existsSync(join(f.root, 't3-started')), 60_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
await killSession(f, 'tier-t3');
const first = vmDirs(f)[0]!;
writeFileSync(join(f.root, 'listed'), '');
writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n');
const b = f.run(['verify', '--src', dir]);
const err = b.stderr.toString(), out = b.stdout.toString();
expect(b.exitCode, err).toBe(0);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse t1\.\d{10}\.\d+\.json \(\d+s\)\n/);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse t2\.\d{10}\.\d+\.json \(\d+s\)\n/);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t3 run: interrupted \(only a start record, t3\.\d{10}\.\d+\.json\)\n/);
expect(err).toContain('run record for ' + first + ': the VM is gone; starting a new one');
expect(err).not.toContain('reattached to');
expect(creates(f)).toBe(2);
expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('boot\nboot\n');
expect(out).not.toContain('ran: make t1');
expect(out).toContain('t3 ok');
expect(verifyLine(out)).toMatch(/^UBI_VERIFY: tree=[0-9a-f]{12} key=[0-9a-f]{12} tiers=3 passed=3 reused=t1,t2 failed=none prior_failures=none saved_s=\d+$/);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 120_000);
test('verify tiers: a fail record blocks a rerun until --rerun-failed (then prior_failures); passes are reused unless stale, tampered, --no-resume or the tree changed', () => {
const f = fakeVm();
try {
const dir = gitSrc(f, TIERS);
writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 broke; exit 7\n');
const first = run(f, ['verify', '--src', dir]);
expect(first.exitCode, first.stderr.toString()).toBe(7);
expect(first.stdout.toString()).not.toContain('ran: make t3');
expect(verifyLine(first.stdout.toString())).toMatch(/ tiers=3 passed=1 reused=none failed=t2 prior_failures=none saved_s=0$/);
expect(creates(f)).toBe(1);
const blocked = run(f, ['verify', '--src', dir]);
expect(blocked.exitCode).toBe(7);
expect(blocked.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 fail t2\.\d{10}\.\d+\.json \(\d+s, exit 7\): not rerun without --rerun-failed/);
expect(verifyLine(blocked.stdout.toString())).toMatch(/ failed=t2 prior_failures=none /);
expect(creates(f)).toBe(1);
writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 fixed\n');
const rerun = run(f, ['verify', '--src', dir, '--rerun-failed']);
expect(rerun.exitCode, rerun.stderr.toString()).toBe(0);
expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /);
expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 run: prior fail t2\.\d{10}\.\d+\.json \(exit 7; --rerun-failed\)/);
expect(verifyLine(rerun.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1 failed=none prior_failures=t2:7 saved_s=\d+$/);
expect(creates(f)).toBe(2);
const all = run(f, ['verify', '--src', dir]);
expect(all.exitCode).toBe(0);
expect(verifyLine(all.stdout.toString())).toMatch(/ passed=3 reused=t1,t2,t3 failed=none prior_failures=none saved_s=\d+$/);
expect(all.stderr.toString()).toContain('every tier passed under this key; no VM needed');
expect(creates(f)).toBe(2);
const probe = (args: string[] = [], extra: Record<string, string> = {}) => run(f, ['verify', '--src', dir, '--probe', ...args], extra).stderr.toString();
expect(probe(['--no-resume'])).toMatch(/tier=t1 run: --no-resume\n[^]*tier=t2 run: --no-resume\n[^]*tier=t3 run: --no-resume\n/);
expect(probe([], { UBI_CHECKPOINT_MAX_AGE: '0' })).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json is \d+s old \(UBI_CHECKPOINT_MAX_AGE=0\)/);
const keyDir = join(f.checkpoints, readdirSync(f.checkpoints)[0]!);
const t1Log = readdirSync(keyDir).filter(n => n.startsWith('t1.') && n.endsWith('.log')).sort().at(-1)!;
writeFileSync(join(keyDir, t1Log), 'tampered\n');
const tampered = probe();
expect(tampered).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json: its log does not re-hash/);
expect(tampered).toMatch(/tier=t2 reuse /);
writeFileSync(join(dir, 'new-source.ts'), 'export {}\n');
const moved = probe();
for (const t of ['t1', 't2', 't3']) expect(moved).toContain('UBI_VERIFY_RESUME: tier=' + t + ' run: no record');
expect(creates(f)).toBe(2);
// Records are write-once JSON; every one carries its generation.
const gens = readdirSync(keyDir).filter(n => n.endsWith('.json')).map(n => JSON.parse(readFileSync(join(keyDir, n), 'utf8')).generation as number).sort((x, y) => x - y);
expect(gens).toEqual(gens.map((_, i) => i + 1));
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 120_000);
test('verify tiers: a reattach whose VM TTL no longer covers the run left plus UBI_REATTACH_MARGIN starts a fresh VM, destroys the old one and reuses the passed tiers', async () => {
const f = fakeVm();
try {
const dir = gitSrc(f, TIERS);
bootCounts(f, dir);
Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 });
writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n');
const env = { UBI_REAP: '0', UBI_TTL_HOURS: '1' };
const a = f.spawn(['verify', '--src', dir], env);
await until(() => existsSync(join(f.root, 't3-started')), 60_000);
process.kill(a.pid, 'SIGKILL');
await a.exited;
const first = vmDirs(f)[0]!;
expect(first).toMatch(/-t1$/);
const probe = (extra: Record<string, string>) => run(f, ['verify', '--src', dir, '--probe'], { ...env, ...extra }).stdout.toString();
expect(probe({})).toMatch(new RegExp('UBI_ATTACH_INFO: vm=' + first + ' created_at=\\d{10} '));
// A 1 h TTL leaves about 3600 s: t3's 1200 s run timeout plus a 2700 s margin does not fit.
const short = { UBI_RUN_TIMEOUT: '1200', UBI_REATTACH_MARGIN: '2700' };
expect(probe(short)).toMatch(new RegExp('UBI_ATTACH_INFO: none key=[0-9a-f]{12} ttl_short vm=' + first + ' ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\)'));
expect(creates(f)).toBe(1);
writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n');
const b = run(f, ['verify', '--src', dir], { ...env, ...short });
const err = b.stderr.toString();
expect(b.exitCode, err).toBe(0);
expect(err).toMatch(new RegExp('UBI_REATTACH_TTL_SHORT: vm=' + first + ' key=[0-9a-f]{12} ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\): starting a fresh VM \\(tier checkpoints carry over\\) and destroying ' + first));
expect(err).not.toContain('reattached to');
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /);
expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse /);
expect(verifyLine(b.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1,t2 failed=none /);
expect(creates(f)).toBe(2);
expect(f.listed()).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 120_000);
test('verify tiers: a writer whose start record is no longer the newest publishes nothing (stale writer)', () => {
const py = /\nCKPT_PY='\n([^']*)'\n/.exec(readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8'))![1]!;
const root = mkdtempSync(join(tmpdir(), 'ubi-ckpt-'));
try {
const d = join(root, 'key');
const log = join(root, 'tier.log'); writeFileSync(log, 'ok\n');
const publish = (status: string, startgen: string, pid: string) => Bun.spawnSync(['python3', '-c', py, 'publish', d, 't1', status, 'vm', '100', '200', '0', status === 'start' ? '' : log, startgen, 'k', 'tree', pid], { timeout: 30_000 });
expect(publish('start', '0', '1').stdout.toString().trim()).toBe('1');
expect(publish('start', '0', '2').stdout.toString().trim()).toBe('2');
const stale = publish('pass', '1', '1');
expect(stale.exitCode).toBe(2);
expect(stale.stderr.toString()).toContain('stale writer: its start record is generation 1, the newest for t1 is 2');
expect(readdirSync(d).filter(n => n.endsWith('.json')).length).toBe(2);
expect(publish('pass', '2', '2').stdout.toString().trim()).toBe('3');
} finally { rmSync(root, { recursive: true, force: true }); }
});
test('verify tiers: a tier that edits the source gets no pass record (UBI_VERIFY_TREE_MOVED) and reruns next time', () => {
const f = fakeVm();
try {
const dir = gitSrc(f, { tiers: [{ name: 'mover', command: 'make mover' }, { name: 'clean', command: 'make clean' }] });
writeFileSync(join(f.root, 'make-mover.sh'), 'echo edited > moved.txt\n');
const r = run(f, ['verify', '--src', dir]);
expect(r.exitCode, r.stderr.toString()).toBe(0);
expect(r.stderr.toString()).toMatch(/UBI_VERIFY_TREE_MOVED: tier=mover before=[0-9a-f]{12} after=[0-9a-f]{12}: a tier must leave the source tree as it found it; no pass record/);
expect(verifyLine(r.stdout.toString())).toMatch(/ tiers=2 passed=2 reused=none failed=none /);
const next = run(f, ['verify', '--src', dir, '--probe']).stderr.toString();
expect(next).toMatch(/tier=mover run: interrupted \(only a start record/);
expect(next).toMatch(/tier=clean reuse /);
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 90_000);
});
ubicloud/SKILL.md
+15 -5
View File
@@ -237,6 +237,10 @@ and for browser tests `xvfb` plus
`bun run test:ubicloud` (the free suite with the CI lane's environment) and
`gstack-pregate --stage tests --remote` under 8 cores.
A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]`
for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a
fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`.
---
## Step 4: Read the result
@@ -266,9 +270,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown.
## Resume checklist after a wake or a machine swap
1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is
still running without a live runner.
2. A background log without its exit sentinel after the VM is gone is an
interrupted run, not a pass; rerun it.
3. Measure before/after pairs on the same size and location, never one side
1. Rerun the same command first: a run whose runner died or detached
(`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a
finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125
`UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over).
2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no
rerun will reattach to.
3. After a machine swap nothing reattaches; tier checkpoints on a Drive
(`UBI_CHECKPOINT_DIR`) still count.
4. A background log with no exit sentinel and no VM is an interrupted run, not
a pass; rerun it.
5. Measure before/after pairs on the same size and location, never one side
local and the other remote.
ubicloud/SKILL.md.tmpl
+15 -5
View File
@@ -163,6 +163,10 @@ and for browser tests `xvfb` plus
`bun run test:ubicloud` (the free suite with the CI lane's environment) and
`gstack-pregate --stage tests --remote` under 8 cores.
A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]`
for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a
fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`.
---
## Step 4: Read the result
@@ -192,9 +196,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown.
## Resume checklist after a wake or a machine swap
1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is
still running without a live runner.
2. A background log without its exit sentinel after the VM is gone is an
interrupted run, not a pass; rerun it.
3. Measure before/after pairs on the same size and location, never one side
1. Rerun the same command first: a run whose runner died or detached
(`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a
finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125
`UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over).
2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no
rerun will reattach to.
3. After a machine swap nothing reattaches; tier checkpoints on a Drive
(`UBI_CHECKPOINT_DIR`) still count.
4. A background log with no exit sentinel and no VM is an interrupted run, not
a pass; rerun it.
5. Measure before/after pairs on the same size and location, never one side
local and the other remote.