v1.92.60.0 test-free-shards: --ci-run --jobs k sub-shards inside one CI job (k=1 default) (GSTA-203 PR 2) (#3730)

* test-free-shards: --ci-run --jobs k sub-shards inside one CI job (k=1 default), per-file JUnit counts (GSTA-203 PR 2)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* shard-jobs: serial sub-shard gets what is left of the shard wall; register shard-jobs.ts in ship-measure touchfiles

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* shard-jobs: validate every sub-shard summary, bounded JUnit read, stricter --ci-verify records, unique count paths, no serial launch without wall

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* shard-jobs tests: non-hex plan id fixture; harness-version: shard-jobs FREE_ONLY entry is non-behavioral

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(test:free): exact root-bound JUnit path matching; serial timeout text names the wall the parallel phase left

Adversarial pass 2 (GSTA-203 PR 2): a plain relative path now matches a planned
file exactly, and an absolute or ../-prefixed path matches only when its prefix is
the child's root, so an unplanned browse/test/x.test.ts is never credited to
test/x.test.ts. The serial sub-shard's FREE_SUBSHARD_TIMEOUT text says how much of
the shard wall the parallel phase left it.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(testing): real-home tripwire covers --jobs k>1 parallel sub-shards (docs audit docs-203-1)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-02 05:42:27 +00:00
1 parent d7154dba13
commit 68195c372e
14 files changed
+1005 -23

No files matched your search

+4
View File
@@ -1,5 +1,9 @@
# Changelog
## [1.92.60.0] - 2026-11-02
Free test shards can now run their files as several sub-shards side by side inside one CI job. `bun run scripts/test-free-shards.ts --ci-run <plan> --shard <i> --result <file> --jobs <k>` splits the shard into k parts that each run in their own sandboxed `bun test` process. The default stays at 1, so CI runs exactly as before until a workflow opts in. Bad `--jobs` values stop with `FREE_JOBS_INVALID`, a sub-shard that runs out of time prints `FREE_SUBSHARD_TIMEOUT` with a serial reproduce command, and every CI shard result now records per-file test and `expect()` counts. See docs/TESTING_INTERNALS.md#sub-shards-inside-one-ci-job.
## [1.92.59.0] - 2026-11-02
**`test:health` and the factory scorecard leave reused main pushes out of their free-suite flake counts (#3725).** Since #3709, a main push whose merged pull request already passed on the same tree reuses that run. It finishes green, runs no shards and uploads nothing, so both readers counted it as a run with zero flakes. As reuse grows, that pulls real flake rates under the 5% bar, and `bun run test:health --enforce` stops flagging files without any error.
+1 -1
View File
@@ -1 +1 @@
1.92.59.0
1.92.60.0
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.92.59.0 — regenerate/re-copy after upgrading gstack
# gstack digest v1.92.60.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
+54 -4
View File
@@ -1428,10 +1428,13 @@ No free test may write the developer's real home. `scripts/lib/free-home-guard.t
snapshots `~/.gstack`, `~/.claude`, `~/.codex`, `~/.agents` and `~/.config/gstack`
(live session logs excluded) and fails the run when an entry changes.
- A shard that runs alone (`--shard`, CI, the exclusive host-state shard, the
flaky retry) owns its window, so the failure names its files.
- Full-suite shards run concurrently on one HOME and cannot tell whose write a
change was. The runner guards that whole phase once and names no shard.
- A shard that runs alone (`--shard`, a CI shard at `--jobs 1` or its trailing
serial sub-shard, the exclusive host-state shard, the flaky retry) owns its
window, so the failure names its files.
- Full-suite shards, and the parallel sub-shards of a CI shard at `--jobs k > 1`
([sub-shards](#sub-shards-inside-one-ci-job)), run concurrently on one HOME and
cannot tell whose write a change was. The runner guards that whole phase once
and names no shard.
- `bun run scripts/test-free-shards.ts --attribute-home` names the writer: it
runs every file alone with a private HOME (browser cache and git identity
still come from the real home) and reports each file that wrote a watched
@@ -1565,6 +1568,53 @@ refresh to a maintainer.
besides the bot's one refresh commit, later runs leave it alone. Only runs on
`main` push the branch; a dispatch from another ref reports only.
### Sub-shards inside one CI job
`--ci-run <plan> --shard i --result <file> --jobs k` runs one planned CI shard as
k sub-shards side by side on the same machine (`scripts/lib/shard-jobs.ts`). Each
sub-shard is an ordinary free shard child: one `bun test` process, serial
inside, with its own sandbox, process-group kill and log. Bun's worker
`--parallel` mode is still not used (decision V3/D6 in
`scripts/test-free-shards.ts`). The default is `k = 1`, which runs today's one
child, so no workflow changes behavior until it passes `--jobs`.
- `--jobs` applies only to `--ci-run`. It takes a positive whole number up to the
shard's file count (an empty shard takes 1); anything else stops with
`FREE_JOBS_INVALID`. `GSTACK_FREE_JOBS` sets the local full-suite process count
and never changes `--ci-run`. The exclusive host-state shard has one file, so
a workflow passes it `--jobs 1` or no `--jobs`.
- Files are split into disjoint sub-shards packed by the duration seed. Each
sub-shard has an id (`<plan id prefix>:<shard>.<n>`) bound to its file list.
Files in `TREE_MUTATING` or `SLOW_UNDER_LOAD` run after the parallel
sub-shards settle, serially, in one trailing sub-shard.
- The parallel phase uses the concurrent-phase real-home guard, the same one
full-suite mode uses; the serial sub-shard guards itself and names its files.
- The CI shard keeps its k = 1 wall deadline: the parallel sub-shards get it,
and the serial sub-shard gets whatever is left of it, so k never extends a
deadline. The parent waits for every child to settle.
- The parent merges every sub-shard result into the shard's one `outcome`
before the single flaky-retry pass, so the five-file retry cap stays per CI
shard. A missing, duplicate or incomplete sub-shard result fails the shard
with `FREE_SUBSHARD_REJECTED` and vetoes the retry. `--ci-verify` re-checks
the `jobs` and `subShards` records a `k > 1` result carries.
- Sub-shard runs never write `scripts/free-test-durations.json`; only
`--record-durations` does, and only from one-file children.
`FREE_SUBSHARD_TIMEOUT` means a sub-shard hit its wall at `k > 1`. The message
names the file in flight, k, the runner and the CI run id, and prints a serial
reproduce command. If the file passes serially, add it to `SLOW_UNDER_LOAD` in
`scripts/lib/shard-jobs.ts` with that run id and a reason. Never raise the
timeout. `SLOW_UNDER_LOAD` is empty today.
Every `--ci-run` result, at any k, records per-file test and `expect()` counts
in `outcome.counts` (`tests`, `pass`, `fail`, `skip`, `todo`, `expects` per
file, plus shard totals). They come from bun's JUnit reporter, which the runner
writes into the shard sandbox; on the pinned Bun 1.4.2 it attributes both counts
per file inside a multi-file child and leaves the console stream unchanged.
`complete: false` (with a `problem`) means the report was missing or did not add
up to bun's own test count. Counts are evidence for comparing k values; they
never decide a verdict.
### Windows free lane
Windows has minimal support ([support tiers](support-tiers.md)). The required
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gstack",
"version": "1.92.59",
"version": "1.92.60",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
+1 -1
View File
@@ -17,7 +17,7 @@
"scripts/lib/published-text.ts": {"blob":"3f2e4d616b5dd0204035710c66b64d49055c5714","since":6},
"scripts/lib/shard-engine.ts": {"blob":"dc714d7d897aed185c0ec3d2c92ad22d954039e5","since":6,"nonBehavioral":"createShardSandbox drops the Capy launch markers (GSTACK_SESSION_KIND, GSTACK_CAPY, GSTACK_EPHEMERAL, GSTACK_INSTALL_VERDICT) from shard environments; CI never sets them, so no case's input or verdict changes there, and on a Capy machine fixtures now see the session kind they set themselves instead of the installer's"},
"scripts/test-paid-shards.ts": {"blob":"5439d0a8f6e1569fa7c924af20d67c30c81d5d30","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"},
"scripts/test-pr-profile.ts": {"blob":"25e2d88dd2e1caae5213fb396b977891bc67549a","since":6,"nonBehavioral":"scripts/test-pr-profile.ts only lists .github/workflows/windows-smoke.yml among files no paid case reads; no case's selection, prompt or verdict can change"},
"scripts/test-pr-profile.ts": {"blob":"44ae88bebf8f716f70a3bd836c8bedb03e6653e8","since":6,"nonBehavioral":"FREE_ONLY_PR_FILES gains scripts/lib/shard-jobs.ts (free-runner module) and .github/workflows/windows-smoke.yml (a workflow no paid case reads); both change only which PR paths select paid cases, never how a case runs or is judged"},
"scripts/test-strict-output.ts": {"blob":"01185d20dc9761b04b9b2f751ec98664fe0eaf93","since":6},
"test/helpers/agent-sdk-runner.ts": {"blob":"210bba92d241a90df578b2c7fc943ac5755d0025","since":6},
"test/helpers/auq-substance-panel.ts": {"blob":"4881260ef793668e4963a062db6d06a5e1f7a40f","since":6},
+422
View File
@@ -0,0 +1,422 @@
/**
* Sub-shards inside one CI job: `--ci-run <plan> --shard i --result r --jobs k`
* (GSTA-203 PR 2, principle "buy cores, not shards").
*
* A CI shard's files split into k disjoint, identity-bound sub-shards that run
* side by side through the free runner's own process-shard path (one bun child
* each, serial inside, own sandbox, group kill, log), never Bun's worker
* `--parallel` (decision V3/D6 in scripts/test-free-shards.ts). The rules:
*
* - k = 1 (the default) is today's single child, unchanged.
* - Files in TREE_MUTATING or SLOW_UNDER_LOAD run after the parallel
* sub-shards settle, serially, in one trailing sub-shard.
* - The parallel phase uses the concurrent-phase home guard; every child
* uses the shared guard inside it, exactly as full-suite mode does.
* - The whole CI shard keeps its k = 1 wall: the parallel sub-shards get it,
* and the serial sub-shard gets what is left of it, so k never extends a
* deadline; every launched child settles before the parent goes on.
* - The parent merges every sub-shard result before the single retry pass,
* so the five-file retry cap stays per CI shard; a missing, duplicate or
* incomplete sub-shard result fails the shard.
* - Sub-shard runs never write the duration seed (only `--record-durations`
* does, from one-file children).
*
* Per-file test and expect() counts come from bun's JUnit reporter, which
* attributes both per file inside a multi-file child on the pinned Bun 1.4.2
* (the console stream is byte-identical with or without it). Counts are
* evidence for k > 1 comparisons; they never decide a verdict.
*/
import * as fs from 'node:fs';
import * as os from 'node:os';
import type { FreeCiPlan, FreeCiResult, FreeShardOutcome } from '../test-free-shards';
import { guardFreeHome, sharedFreeHome, type FreeHomeGuard, type FreeHomeGuardFactory } from './free-home-guard';
import { normalizeRelativePath } from './shard-engine';
/**
* Files that pass alone but time out as a sub-shard under load. Each entry
* names the CI run id where it timed out (FREE_SUBSHARD_TIMEOUT prints it).
* They run in the trailing serial sub-shard under `--jobs`; at k = 1 nothing
* changes. Keys are pinned against the live file census by
* test/shard-jobs.test.ts. Never raise a timeout instead of adding an entry.
*/
export const SLOW_UNDER_LOAD: Record<string, { reason: string; runId: string }> = {};
export const FREE_JOBS_INVALID = 'FREE_JOBS_INVALID';
export const FREE_SUBSHARD_TIMEOUT = 'FREE_SUBSHARD_TIMEOUT';
export const FREE_SUBSHARD_REJECTED = 'FREE_SUBSHARD_REJECTED';
const DOCS = 'docs/TESTING_INTERNALS.md#sub-shards-inside-one-ci-job';
const JOBS_FIX = `--jobs applies only to --ci-run and defaults to 1; GSTACK_FREE_JOBS sets the local full-suite process count and never changes --ci-run. See ${DOCS}`;
/** `--jobs <k>` at parse time: digits only, at least 1. */
export function parseJobsFlag(raw: string | undefined): number {
if (raw === undefined || !/^\d+$/.test(raw) || Number(raw) < 1) {
throw new Error(`${FREE_JOBS_INVALID}: --jobs needs a positive whole number; got ${JSON.stringify(raw ?? '')}. ${JOBS_FIX}`);
}
return Number(raw);
}
export function rejectJobsWithoutCiRun(): never {
throw new Error(`${FREE_JOBS_INVALID}: --jobs needs --ci-run. ${JOBS_FIX}`);
}
/** `--jobs <k>` against the planned shard: k may not exceed its file count (an empty shard allows 1). */
export function validateJobsForShard(jobs: number, fileCount: number, shard: number): void {
const max = Math.max(1, fileCount);
if (jobs > max) {
throw new Error(`${FREE_JOBS_INVALID}: --jobs ${jobs} is more than CI shard ${shard}'s ${fileCount} file(s); use 1 to ${max}. ${JOBS_FIX}`);
}
}
export interface FileTestCounts { tests: number; pass: number; fail: number; skip: number; todo: number; expects: number }
export interface ShardTestCounts {
source: 'bun-junit';
/** Every reported file was planned and the per-file tests add up to bun's own `Ran N tests` count. */
complete: boolean;
problem?: string;
files: Record<string, FileTestCounts>;
totals: FileTestCounts;
}
const zeroCounts = (): FileTestCounts => ({ tests: 0, pass: 0, fail: 0, skip: 0, todo: 0, expects: 0 });
const addCounts = (into: FileTestCounts, from: FileTestCounts) => {
for (const key of Object.keys(into) as Array<keyof FileTestCounts>) into[key] += from[key];
};
const TAG = /<(\/?)(testsuites|testsuite|testcase|failure|error|skipped)\b((?:[^>"']|"[^"]*"|'[^']*')*?)(\/?)>/g;
const ATTR = /([\w:-]+)="([^"]*)"/g;
const ENTITIES: Record<string, string> = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'" };
const decode = (text: string) => text.replace(/&(#x[0-9a-f]+|#\d+|\w+);/gi, (whole, entity: string) => entity[0] === '#'
? String.fromCodePoint(entity[1].toLowerCase() === 'x' ? parseInt(entity.slice(2), 16) : Number(entity.slice(1)))
: ENTITIES[entity] ?? whole);
const attributes = (text: string): Record<string, string> =>
Object.fromEntries([...text.matchAll(ATTR)].map(([, key, value]) => [key, decode(value)]));
/**
* Per-file counts from bun's JUnit XML. A testcase is a fail when it holds a
* failure or error, a todo when skipped with message TODO, a skip when
* skipped otherwise, else a pass; expects sum each testcase's assertions.
* Printed paths are matched back to planned files. Bun prints them relative
* to the child's real cwd, so a plain relative path must match exactly; an
* absolute or ../-prefixed path matches only when what precedes the planned
* file is the child's root. Files that never ran a test are absent.
*/
export function parseJunitCounts(xml: string | null, planned: readonly string[], testsRan: number | null, root = process.cwd()): ShardTestCounts {
const files: Record<string, FileTestCounts> = {};
const totals = zeroCounts();
const result = (complete: boolean, problem?: string): ShardTestCounts =>
({ source: 'bun-junit', complete, ...(problem ? { problem } : {}), files, totals });
if (xml === null) return result(false, 'bun wrote no JUnit report');
const plannedSet = new Set(planned);
const unknown = new Set<string>();
const roots = new Set([root, (() => { try { return fs.realpathSync(root); } catch { return root; } })()].map(normalizeRelativePath));
const isRoot = (prefix: string) => [...roots].some(dir => dir === prefix || dir.endsWith(`/${prefix}`));
const canonical = (printed: string) => {
const normalized = normalizeRelativePath(printed);
const stripped = normalized.replace(/^(?:\.{1,2}\/)+/, '');
if (plannedSet.has(stripped)) return stripped;
if (stripped === normalized && !/^(?:\/|[A-Za-z]:\/)/.test(stripped)) return stripped;
const matches = planned.filter(file => stripped.endsWith(`/${file}`) && isRoot(stripped.slice(0, -file.length - 1)));
return matches.length === 1 ? matches[0] : stripped;
};
let current: { file: string; state: 'pass' | 'fail' | 'skip' | 'todo'; expects: number } | null = null;
const finish = () => {
if (!current) return;
const entry = files[current.file] ??= zeroCounts();
entry.tests += 1;
entry[current.state] += 1;
entry.expects += current.expects;
if (!plannedSet.has(current.file)) unknown.add(current.file);
current = null;
};
for (const [, closing, name, body, selfClosing] of xml.matchAll(TAG)) {
if (name === 'testcase') {
if (closing) { finish(); continue; }
const attrs = attributes(body);
current = { file: canonical(attrs.file ?? ''), state: 'pass', expects: Number(attrs.assertions ?? 0) || 0 };
if (selfClosing) finish();
} else if (current && !closing && (name === 'failure' || name === 'error')) {
current.state = 'fail';
} else if (current && !closing && name === 'skipped' && current.state === 'pass') {
current.state = attributes(body).message === 'TODO' ? 'todo' : 'skip';
}
}
finish();
for (const entry of Object.values(files)) addCounts(totals, entry);
if (unknown.size) return result(false, `JUnit named unplanned file(s): ${[...unknown].sort().join(', ')}`);
if (testsRan === null) return result(false, "bun's summary has no test count");
if (totals.tests !== testsRan) return result(false, `JUnit counted ${totals.tests} tests, bun's summary ${testsRan}`);
return result(true);
}
const JUNIT_MAX_BYTES = 64 * 1024 * 1024;
/** The child-written report, only when it is a regular file of bounded size (never a link, FIFO or device). */
export function readJunitReport(file: string): string | null {
let fd: number;
try { fd = fs.openSync(file, fs.constants.O_RDONLY | (fs.constants.O_NOFOLLOW ?? 0) | (fs.constants.O_NONBLOCK ?? 0)); } catch { return null; }
try {
const stat = fs.fstatSync(fd);
return stat.isFile() && stat.size <= JUNIT_MAX_BYTES ? fs.readFileSync(fd, 'utf8') : null;
} catch { return null; } finally { fs.closeSync(fd); }
}
/** One shard's counts from its sub-shards' counts; complete only when every part is and no file repeats. */
export function mergeTestCounts(parts: Array<ShardTestCounts | undefined>): ShardTestCounts | undefined {
if (parts.every(part => !part)) return undefined;
const files: Record<string, FileTestCounts> = {};
const totals = zeroCounts();
const problems: string[] = [];
parts.forEach((part, index) => {
if (!part) { problems.push(`sub-shard ${index + 1} has no counts`); return; }
if (!part.complete) problems.push(`sub-shard ${index + 1}: ${part.problem ?? 'incomplete'}`);
for (const [file, counts] of Object.entries(part.files)) {
if (files[file]) problems.push(`${file} counted twice`);
files[file] = { ...counts };
}
addCounts(totals, part.totals);
});
return { source: 'bun-junit', complete: problems.length === 0, ...(problems.length ? { problem: problems.join('; ') } : {}), files, totals };
}
export interface SubShardAssignment {
/** `<plan id prefix>:<shard>.<index>`: binds a result to one plan, shard and file set. */
id: string;
index: number;
phase: 'parallel' | 'serial';
files: string[];
}
/**
* Split one CI shard into disjoint sub-shards: up to k parallel ones packed by
* `pack`, then one serial sub-shard for the files `isSerial` names. Throws
* unless the assignments cover the shard's files exactly once.
*/
export function assignSubShards(
planId: string, shard: { shard: number; files: string[] }, jobs: number,
isSerial: (file: string) => boolean, pack: (files: string[], count: number) => string[][],
): SubShardAssignment[] {
const serial = shard.files.filter(isSerial);
const parallel = shard.files.filter(file => !isSerial(file));
const groups = parallel.length ? pack(parallel, Math.min(jobs, parallel.length)).filter(files => files.length) : [];
const assignments: SubShardAssignment[] = [
...groups.map(files => ({ phase: 'parallel' as const, files })),
...(serial.length ? [{ phase: 'serial' as const, files: serial }] : []),
].map((entry, index) => ({ id: `${planId.slice(0, 12)}:${shard.shard}.${index + 1}`, index: index + 1, ...entry }));
const assigned = assignments.flatMap(entry => entry.files);
if (new Set(assigned).size !== assigned.length || assigned.length !== shard.files.length
|| !shard.files.every(file => assigned.includes(file))) {
throw new Error(`${FREE_SUBSHARD_REJECTED}: sub-shard assignment for CI shard ${shard.shard} does not cover its files exactly once`);
}
return assignments;
}
export type SubShardSettlement = { assignment: SubShardAssignment } & ({ outcome: FreeShardOutcome } | { error: string });
/** Why one sub-shard's result cannot stand for its assignment; empty when it can. */
export function subShardProblems(settled: SubShardSettlement): string[] {
const { assignment } = settled;
if (!('outcome' in settled)) return [`${assignment.id} has no result (${settled.error})`];
const { outcome } = settled;
const problems: string[] = [];
if (JSON.stringify(outcome.files) !== JSON.stringify(assignment.files)) problems.push(`${assignment.id} reported another file set`);
const summary = outcome.summary;
if (!summary || (summary.sawTerminalSummary && summary.filesRan !== assignment.files.length)) {
problems.push(`${assignment.id} reported ${summary?.filesRan ?? 'no'} executed file(s) for ${assignment.files.length} assigned`);
} else if (outcome.status === 'passed' && (summary.filesRan !== assignment.files.length || summary.sawTerminalSummary !== true
|| outcome.exitCode !== 0 || outcome.failingFiles.length || outcome.unattributedFailures)) {
problems.push(`${assignment.id} passed without a complete execution summary`);
}
return problems;
}
export interface SubShardRecord {
id: string;
phase: SubShardAssignment['phase'];
files: string[];
status: FreeShardOutcome['status'] | 'missing';
exitCode: number | null;
elapsedMs: number;
summary?: FreeShardOutcome['summary'];
failingFiles: string[];
unattributedFailures: number;
}
/**
* One CI shard outcome from every sub-shard: the planned file order, summed
* summaries, the union of failing files, and the worst status. Every rejected
* sub-shard adds unattributed evidence, which also vetoes the retry pass.
*/
export function mergeSubShardOutcomes(
shard: { shard: number; files: string[] }, settled: SubShardSettlement[], elapsedMs: number, extraProblems: string[] = [],
): { outcome: FreeShardOutcome; records: SubShardRecord[]; problems: string[] } {
const problems = [...extraProblems];
const assigned = settled.flatMap(entry => entry.assignment.files);
if (new Set(assigned).size !== assigned.length) problems.push('a file is assigned to more than one sub-shard');
if (assigned.length !== shard.files.length || !shard.files.every(file => assigned.includes(file))) problems.push('sub-shards do not cover the CI shard');
for (const entry of settled) problems.push(...subShardProblems(entry));
const outcomes = settled.flatMap(entry => ('outcome' in entry ? [entry.outcome] : []));
const missing = settled.length - outcomes.length;
const status: FreeShardOutcome['status'] = outcomes.some(outcome => outcome.status === 'timed-out') ? 'timed-out'
: problems.length || outcomes.some(outcome => outcome.status !== 'passed') ? 'failed' : 'passed';
const testsRan = outcomes.map(outcome => outcome.summary?.testsRan ?? null);
const failures = outcomes.flatMap(outcome => outcome.failures ?? []);
const inFlight = outcomes.flatMap(outcome => outcome.inFlight ?? []);
const counts = mergeTestCounts(settled.map(entry => ('outcome' in entry ? entry.outcome.counts : undefined)));
const outcome: FreeShardOutcome = {
shard: shard.shard,
files: shard.files,
status,
exitCode: status === 'passed' ? 0 : outcomes.find(entry => entry.exitCode)?.exitCode ?? 1,
elapsedMs,
groupPid: null,
failingFiles: status === 'passed' ? [] : [...new Set(outcomes.flatMap(entry => entry.failingFiles))],
unattributedFailures: status === 'passed' ? 0 : outcomes.reduce((sum, entry) => sum + entry.unattributedFailures, 0) + problems.length,
...(failures.length ? { failures } : {}),
summary: {
testsRan: missing || testsRan.some(value => value === null) ? null : testsRan.reduce<number>((sum, value) => sum + value!, 0),
filesRan: outcomes.reduce((sum, entry) => sum + (entry.summary?.filesRan ?? 0), 0),
sawTerminalSummary: !missing && outcomes.every(entry => entry.files.length === 0 || entry.summary?.sawTerminalSummary === true),
},
...(counts ? { counts } : {}),
...(inFlight.length ? { inFlight } : {}),
};
const records = settled.map((entry): SubShardRecord => {
const { id, phase, files } = entry.assignment;
if (!('outcome' in entry)) return { id, phase, files, status: 'missing', exitCode: null, elapsedMs: 0, failingFiles: [], unattributedFailures: 1 };
const { status: subStatus, exitCode, elapsedMs: subElapsed, summary, failingFiles, unattributedFailures } = entry.outcome;
return { id, phase, files, status: subStatus, exitCode, elapsedMs: subElapsed, summary, failingFiles, unattributedFailures };
});
return { outcome, records, problems };
}
/** `--ci-verify` re-checks a k > 1 result's sub-shard records; empty means they stand. */
export function subShardRecordProblems(result: FreeCiResult): string[] {
if (result.jobs === undefined && result.subShards === undefined) return [];
const records = result.subShards ?? [];
if (!Number.isInteger(result.jobs) || result.jobs! < 2 || !records.length) return ['sub-shard result without a valid --jobs and records'];
const files = records.flatMap(record => record.files);
const problems: string[] = [];
if (new Set(files).size !== files.length || files.length !== result.outcome.files.length
|| !result.outcome.files.every(file => files.includes(file))) problems.push('sub-shard records do not cover the shard exactly once');
const prefix = `${result.planId.slice(0, 12)}:${result.outcome.shard}.`;
if (records.some((record, index) => record.id !== `${prefix}${index + 1}`)
|| records.filter(record => record.phase === 'parallel').length > result.jobs!
|| records.some((record, index) => record.phase === 'serial' ? index !== records.length - 1 : record.phase !== 'parallel')) {
problems.push('sub-shard ids or phases do not match the plan and --jobs');
}
if (records.some(record => record.status === 'missing' || record.summary?.filesRan !== record.files.length
|| (record.files.length > 0 && record.summary?.sawTerminalSummary !== true))) problems.push('a sub-shard did not run every assigned file');
if (result.outcome.status === 'passed' && records.some(record => record.status !== 'passed')) problems.push('a passing shard holds a failed sub-shard');
return problems;
}
export interface CiRunContext { planPath: string; runId: string; runner: string }
/** Where this run is: the CI run id (with attempt) and runner, else `local` and the host. */
export function ciRunContext(planPath: string, env: NodeJS.ProcessEnv = process.env): CiRunContext {
const runId = env.GITHUB_RUN_ID ? `${env.GITHUB_RUN_ID}${env.GITHUB_RUN_ATTEMPT ? `/attempt ${env.GITHUB_RUN_ATTEMPT}` : ''}` : 'local';
const cpus = os.availableParallelism?.() ?? os.cpus().length;
return { planPath, runId, runner: `${env.RUNNER_NAME || os.hostname()} (${process.platform}, ${cpus} CPUs)` };
}
const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`;
/** The FREE_SUBSHARD_TIMEOUT text: the file, k, runner, a serial reproduce command and the fix. */
export function subShardTimeoutText(
shard: number, jobs: number, record: SubShardRecord, inFlight: string[], wallTimeoutMs: number, context: CiRunContext, shardWallMs = wallTimeoutMs,
): string[] {
const suspects = inFlight.filter(file => record.files.includes(file));
const files = suspects.length ? suspects : record.files;
const seconds = (ms: number) => Math.round(ms / 1000);
const fix = record.phase !== 'serial'
? `if it passes serially, add it to SLOW_UNDER_LOAD in scripts/lib/shard-jobs.ts with run id ${context.runId}; never raise the timeout`
: wallTimeoutMs < shardWallMs
? `the parallel sub-shards left this serial sub-shard ${seconds(wallTimeoutMs)}s of the ${seconds(shardWallMs)}s shard wall; if the k=1 command passes, the parallel phase used up the budget and this shard needs a lower --jobs; never raise the timeout`
: 'this file already runs alone after the parallel sub-shards; fix or split it, never raise the timeout';
return [
`[test:free] ${FREE_SUBSHARD_TIMEOUT}: CI shard ${shard} sub-shard ${record.id} (${record.files.length} files) hit its ${seconds(wallTimeoutMs)}s wall at --jobs ${jobs} on ${context.runner}, run ${context.runId}.`,
`[test:free] in flight: ${suspects.length ? suspects.join(', ') : 'unknown (no open file chunk; see the sub-shard log)'}`,
`[test:free] reproduce serially: printf '%s\\n' ${files.map(quote).join(' ')} > files.txt && GSTACK_FREE_JOBS=1 bun run scripts/test-free-shards.ts --files-from files.txt`,
`[test:free] or the whole CI shard at k=1: bun run scripts/test-free-shards.ts --ci-run ${quote(context.planPath)} --shard ${shard} --result shard-${shard}.json --jobs 1`,
`[test:free] ${fix}. See ${DOCS}`,
];
}
export interface RunShardOptions { wallTimeoutMs: number; verbose: boolean; testCounts: true; homeGuard?: FreeHomeGuardFactory }
export interface CiShardInput {
plan: FreeCiPlan;
shard: FreeCiPlan['shards'][number];
revision: string;
/** Sub-shard count; 1 is today's single child. */
jobs: number;
wallTimeoutMs: number;
verbose: boolean;
context: CiRunContext;
}
export interface CiShardDeps {
runShard(files: string[], shardNumber: number, totalShards: number, options: RunShardOptions): Promise<FreeShardOutcome>;
/** The free runner's single flaky-retry pass over every collected outcome. */
retry(outcomes: FreeShardOutcome[]): Promise<{ exitCode: number; retry: FreeShardOutcome | null }>;
pack(files: string[], count: number): string[][];
isSerial(file: string): boolean;
terminationRequested(): boolean;
phaseGuard?(files: string[], shards: number): FreeHomeGuard;
log?(line: string): void;
}
/** Run one planned CI shard at k sub-shards and build its result JSON. */
export async function runCiShard(input: CiShardInput, deps: CiShardDeps): Promise<{ result: FreeCiResult; exitCode: number }> {
const { plan, shard, revision, jobs, wallTimeoutMs, verbose } = input;
const log = deps.log ?? ((line: string) => console.log(line));
if (jobs === 1) {
const outcome = await deps.runShard(shard.files, shard.shard, plan.shards.length, { wallTimeoutMs, verbose, testCounts: true });
const retried = await deps.retry([outcome]);
return { result: { planId: plan.id, revision, outcome, retry: retried.retry }, exitCode: retried.exitCode };
}
const startedAt = Date.now();
const assignments = assignSubShards(plan.id, shard, jobs, deps.isSerial, deps.pack);
const total = assignments.length;
log(`[test:free] CI shard ${shard.shard}/${plan.shards.length} at --jobs ${jobs}: ${assignments
.map(entry => `sub-shard ${entry.id} = runner shard ${entry.index}/${total} (${entry.phase}, ${entry.files.length} files)`).join('; ')}`);
const walls = new Map<string, number>();
const run = (entry: SubShardAssignment, homeGuard?: FreeHomeGuardFactory, wall = wallTimeoutMs): Promise<SubShardSettlement> => {
walls.set(entry.id, wall);
return deps.runShard(entry.files, entry.index, total, { wallTimeoutMs: wall, verbose, testCounts: true, ...(homeGuard ? { homeGuard } : {}) })
.then(outcome => ({ assignment: entry, outcome }), (error: unknown) => ({ assignment: entry, error: error instanceof Error ? error.message : String(error) }));
};
const parallel = assignments.filter(entry => entry.phase === 'parallel');
const serial = assignments.filter(entry => entry.phase === 'serial');
const phaseProblems: string[] = [];
const settled: SubShardSettlement[] = [];
if (parallel.length) {
const guard = deps.phaseGuard ?? ((files: string[], shards: number) => guardFreeHome(files, process.env, { kind: 'concurrent', shards }));
const phaseHome = guard(parallel.flatMap(entry => entry.files), parallel.length);
settled.push(...await Promise.all(parallel.map(entry => run(entry, sharedFreeHome))));
const homeChange = phaseHome.verify();
if (homeChange) phaseProblems.push(homeChange);
}
for (const entry of serial) {
const left = startedAt + wallTimeoutMs - Date.now();
settled.push(deps.terminationRequested() ? { assignment: entry, error: 'cancelled before the serial phase' }
: left <= 0 ? { assignment: entry, error: 'no wall left after the parallel phase' } : await run(entry, undefined, left));
}
const merged = mergeSubShardOutcomes(shard, settled, Date.now() - startedAt, phaseProblems);
for (const problem of merged.problems) console.error(`[test:free] ${FREE_SUBSHARD_REJECTED}: CI shard ${shard.shard}: ${problem}`);
for (const record of merged.records.filter(entry => entry.status === 'timed-out')) {
for (const line of subShardTimeoutText(shard.shard, jobs, record, merged.outcome.inFlight ?? [], walls.get(record.id) ?? wallTimeoutMs, input.context, wallTimeoutMs)) console.error(line);
}
log(`[test:free] CI shard ${shard.shard}: ${merged.outcome.status} across ${total} sub-shards in ${Math.round(merged.outcome.elapsedMs / 1000)}s`);
const retried = await deps.retry([merged.outcome]);
return {
result: { planId: plan.id, revision, outcome: merged.outcome, retry: retried.retry, jobs, subShards: merged.records },
exitCode: retried.exitCode,
};
}
/** The duration seed takes only one-file children's walls, never a sub-shard's or a shard's. */
export function assertSoloSeedSample(file: string, outcome: Pick<FreeShardOutcome, 'files' | 'summary'>): void {
if (outcome.files.length !== 1 || outcome.files[0] !== file || (outcome.summary?.filesRan ?? 0) > 1) {
throw new Error(`Duration seed sample for ${file} came from a ${outcome.files.length}-file child; seeds are recorded only from one-file serial children.`);
}
}
+51 -12
View File
@@ -77,6 +77,11 @@
* bun run scripts/test-free-shards.ts --attribute-home # name real-home writers: each file alone, private HOME
* bun run scripts/test-free-shards.ts --ci-plan plan.json --shards 20
* bun run scripts/test-free-shards.ts --ci-run plan.json --shard 1 --result result.json
* bun run scripts/test-free-shards.ts --ci-run plan.json --shard 1 --result result.json --jobs 4
* # k sub-shards side by side inside one CI job (default 1 = one child, today's behavior;
* # scripts/lib/shard-jobs.ts). --jobs applies only to --ci-run: a positive whole number up
* # to the shard's file count, else FREE_JOBS_INVALID. GSTACK_FREE_JOBS sets the local
* # full-suite process count and never changes --ci-run; --jobs is its only knob.
* bun run scripts/test-free-shards.ts --ci-verify plan.json --results results/
*/
@@ -91,6 +96,10 @@ import { resolveStateRoot } from '../lib/state-root';
import { attributeFreeHomeWriters, guardFreeHome, sharedFreeHome, type FreeHomeGuardFactory } from './lib/free-home-guard';
import { appendStepSummary, ciHealthSummary, SEED_REFRESH_COMMAND, unseededWarning, windowsCurationLine } from './lib/free-ci-health';
import { curateWindowsSafe, type CurationResult } from './lib/windows-curation';
import {
assertSoloSeedSample, ciRunContext, parseJobsFlag, parseJunitCounts, readJunitReport, rejectJobsWithoutCiRun, runCiShard, SLOW_UNDER_LOAD, subShardRecordProblems,
validateJobsForShard, type ShardTestCounts, type SubShardRecord,
} from './lib/shard-jobs';
export { curateWindowsSafe, detectWindowsFragility, KNOWN_WINDOWS_INCOMPATIBLE, type CurationResult } from './lib/windows-curation';
import {
BunTestOutputClassifier,
@@ -447,6 +456,9 @@ export interface FreeCiResult {
revision: string;
outcome: FreeShardOutcome;
retry: FreeShardOutcome | null;
/** Present only at --jobs k > 1: k and one record per sub-shard. */
jobs?: number;
subShards?: SubShardRecord[];
}
/** Preserve the full-suite retry cap across independently running CI jobs. */
@@ -472,6 +484,8 @@ export function verifyFreeCiResults(plan: FreeCiPlan, results: FreeCiResult[]):
|| JSON.stringify(outcome.files) !== JSON.stringify(shard.files)) throw new Error('CI result identity, shard or file coverage mismatch');
seen.add(outcome.shard);
if (!hasCompleteCiSummary(outcome)) throw new Error('Missing or incomplete CI execution summary');
const subShardProblems = subShardRecordProblems(result);
if (subShardProblems.length) throw new Error(`CI shard ${outcome.shard} sub-shards rejected: ${subShardProblems.join('; ')}`);
if (outcome.status === 'passed') {
if (outcome.exitCode !== 0 || outcome.failingFiles.length || outcome.unattributedFailures || result.retry) throw new Error('Inconsistent passing CI result');
} else {
@@ -542,6 +556,8 @@ type CliOptions = {
result: string | null;
results: string | null;
filesFrom: string | null;
/** Set only by --jobs (with --ci-run); absent means k = 1. */
jobs?: number;
};
export function parseCliOptions(argv: string[]): CliOptions {
@@ -556,6 +572,7 @@ export function parseCliOptions(argv: string[]): CliOptions {
let wallTimeoutExplicit = false;
let quick = false;
let attributeHome = false;
let jobs: number | undefined;
const paths: Record<'ciPlan' | 'ciRun' | 'ciVerify' | 'result' | 'results' | 'filesFrom', string | null> = {
ciPlan: null, ciRun: null, ciVerify: null, result: null, results: null, filesFrom: null,
};
@@ -579,6 +596,7 @@ export function parseCliOptions(argv: string[]): CliOptions {
'--result': pathFlag('--result', 'result'),
'--results': pathFlag('--results', 'results'),
'--files-from': pathFlag('--files-from', 'filesFrom'),
'--jobs': (next) => { jobs = parseJobsFlag(next()); },
'--shards': (next) => {
const value = next();
if (!value) throw new Error('Missing value for --shards');
@@ -602,8 +620,9 @@ export function parseCliOptions(argv: string[]): CliOptions {
if (ciModes > 1 || (ciModes && (quick || listOnly || dryRun || recordDurations || attributeHome))) throw new Error('CI modes cannot be combined with other selection modes');
if (paths.ciRun && (shardIndex === null || !paths.result)) throw new Error('--ci-run requires --shard and --result');
if (paths.ciVerify && !paths.results) throw new Error('--ci-verify requires --results');
if (jobs !== undefined && !paths.ciRun) rejectJobsWithoutCiRun();
if (quick && (recordDurations || windowsOnly || shardIndex !== null)) throw new Error('--quick cannot change recording, Windows or shard selection');
return { dryRun, listOnly, recordDurations, windowsOnly, verbose, shardCount, shardIndex, wallTimeoutMs, wallTimeoutExplicit, quick, attributeHome, ...paths };
return { dryRun, listOnly, recordDurations, windowsOnly, verbose, shardCount, shardIndex, wallTimeoutMs, wallTimeoutExplicit, quick, attributeHome, ...paths, ...(jobs === undefined ? {} : { jobs }) };
}
const describeLines = (lines: string[]) => `${lines.slice(0, 5).join(', ')}${lines.length > 5 ? ` (+${lines.length - 5} more)` : ''}`;
@@ -1075,6 +1094,10 @@ export interface FreeShardOutcome {
unattributedFailures: number;
/** Named failures with their excerpts, so a flaky pass records what failed the first time. */
failures?: FreeRunFailure[];
/** Per-file test and expect() counts from bun's JUnit report (--ci-run only). */
counts?: ShardTestCounts;
/** Wedge suspects at a wall-timeout kill (FreeRunReport.inFlight). */
inFlight?: string[];
}
export interface ShardCommand {
@@ -1087,8 +1110,10 @@ export interface RunFreeShardOptions {
wallTimeoutMs?: number;
rootDir?: string;
env?: NodeJS.ProcessEnv;
/** Override the spawned command. Tests inject fake pass/fail/slow commands. */
commandFor?: (files: string[]) => ShardCommand;
/** Override the spawned command. Tests inject fake pass/fail/slow commands; `junitPath` is set with testCounts. */
commandFor?: (files: string[], junitPath?: string) => ShardCommand;
/** Ask bun for a JUnit report in the shard sandbox and record per-file counts on the outcome (--ci-run). */
testCounts?: boolean;
/** Suppress ALL child output from the console (tests). The classifier and the log file still see every byte. */
quiet?: boolean;
/** Forward the full child stream to the console (legacy firehose). Default: the quiet filtered console. */
@@ -1633,12 +1658,14 @@ export async function runFreeShard(
const shardLog = openShardLog(logPath, label, 0o600);
log(`[test:free] full log: ${logPath}`);
const { command, args } = options.commandFor
? options.commandFor(files)
: { command: process.execPath, args: buildShardArgs(files, { rootDir }) };
// realpath: the browser tracker refuses a state dir whose path resolves elsewhere.
const { stateDir, env } = createShardSandbox('gstack-free-shard-', options.env ?? process.env, { realpath: true });
// The JUnit reporter leaves the console stream byte-identical (Bun 1.4.2); it only adds per-file counts.
const junitPath = options.testCounts ? path.join(stateDir, 'junit.xml') : undefined;
const { command, args } = options.commandFor
? options.commandFor(files, junitPath)
: { command: process.execPath, args: [...buildShardArgs(files, { rootDir }), ...(junitPath ? ['--reporter=junit', `--reporter-outfile=${junitPath}`] : [])] };
let junitXml = null as string | null;
// CLI renders otherwise share the repo's .gstack/browse.json, where concurrent
// shards and prior daemons replace each other's state; override inherited state.
env.BROWSE_STATE_FILE = path.join(stateDir, '.gstack', 'browse.json');
@@ -1703,6 +1730,7 @@ export async function runFreeShard(
shardLog.write(diagnostic + '\n');
}
await new Promise<void>((resolve) => shardLog.stream.end(() => resolve()));
if (junitPath) junitXml = readJunitReport(junitPath);
try {
if (!cleanupError) fs.rmSync(stateDir, { recursive: true, force: true });
} catch {
@@ -1745,6 +1773,8 @@ export async function runFreeShard(
shard: shardNumber, files, status, exitCode, elapsedMs: Date.now() - startedAt, groupPid, failingFiles, unattributedFailures,
...(status === 'passed' ? {} : { failures: report.failures }),
summary: { testsRan: report.testsRan, filesRan: report.filesRan, sawTerminalSummary: report.sawTerminalSummary },
...(junitPath ? { counts: parseJunitCounts(junitXml, files, report.testsRan, rootDir) } : {}),
...(status === 'timed-out' && report.inFlight.length ? { inFlight: report.inFlight } : {}),
};
log(shardEpilogue(outcome, totalShards));
for (const line of buildRunEpilogue(status, report, outcome.elapsedMs, logPath)) log(line);
@@ -1799,6 +1829,7 @@ async function recordFreeTestDurations(files: string[], jobs: number, wallTimeou
const outcome = await runFreeShard([file], index + 1, files.length, {
wallTimeoutMs, quiet: true, homeGuard: sharedFreeHome,
});
assertSoloSeedSample(file, outcome);
durations[normalizeRelativePath(file)] = outcome.elapsedMs;
if (outcome.status !== 'passed') failed.push(file);
});
@@ -1944,14 +1975,22 @@ async function main(): Promise<number> {
}
const shard = plan.shards[options.shardIndex! - 1];
if (!shard || shard.shard !== options.shardIndex) throw new Error('CI shard index is outside the plan');
const outcome = await runFreeShard(shard.files, shard.shard, plan.shards.length, {
const jobs = options.jobs ?? 1;
validateJobsForShard(jobs, shard.files.length, shard.shard);
const durations = loadFreeTestDurations() ?? {};
const { result, exitCode } = await runCiShard({
plan, shard, revision, jobs, verbose: options.verbose, context: ciRunContext(options.ciRun!),
wallTimeoutMs: options.wallTimeoutExplicit ? options.wallTimeoutMs
: wallTimeoutForPackedShard(shard.predictedMs, options.wallTimeoutMs, shard.files.length),
verbose: options.verbose,
}, {
runShard: (files, shardNumber, totalShards, runOptions) => runFreeShard(files, shardNumber, totalShards, runOptions),
retry: outcomes => retryFailedFreeFiles(outcomes, plan.shards.length, options),
pack: (files, count) => packShardsByDuration(files, count, durations).shards,
isSerial: file => file in TREE_MUTATING || file in SLOW_UNDER_LOAD,
terminationRequested: isTerminationRequested,
});
const retried = await retryFailedFreeFiles([outcome], plan.shards.length, options);
writeJson(options.result!, { planId: plan.id, revision, outcome, retry: retried.retry } satisfies FreeCiResult);
return retried.exitCode;
writeJson(options.result!, result);
return exitCode;
}
let files = allFiles;
+1
View File
@@ -237,6 +237,7 @@ export const PAID_WORKFLOW_FILES = [
export const FREE_ONLY_PR_FILES = [
'scripts/test-free-shards.ts',
'scripts/lib/free-home-guard.ts', // Imported only by the free shard runner.
'scripts/lib/shard-jobs.ts', // Free runner's --ci-run --jobs sub-shards; ship-measure loads it through test-free-shards.ts, so its touchfiles list it.
'test/helpers/auq-parallel-worker.ts',
'test/helpers/stored-zip.ts', // Read only by the free test-health report tests.
'scripts/free-test-durations.json',
+1
View File
@@ -59,6 +59,7 @@
{ "file": "scripts/eval-bg.ts", "movedFrom": "(new: eval:bg dispatch/local backend, W5d)" },
{ "file": "test/helpers/module-size.ts", "movedFrom": "(new: ratchet (c) counter)" },
{ "file": "scripts/lib/free-home-guard.ts", "movedFrom": "(new: free-lane home-write tripwire, kept out of scripts/test-free-shards.ts)" },
{ "file": "scripts/lib/shard-jobs.ts", "movedFrom": "(new: --ci-run --jobs sub-shards and per-file JUnit counts, kept out of scripts/test-free-shards.ts and scripts/lib/shard-engine.ts, GSTA-203)" },
{ "file": "browse/src/routes/activity.ts", "movedFrom": "browse/src/server.ts" },
{ "file": "browse/src/routes/commands.ts", "movedFrom": "browse/src/server.ts" },
{ "file": "browse/src/routes/core.ts", "movedFrom": "browse/src/server.ts" },
+1 -1
View File
@@ -417,7 +417,7 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
'tpa-apple-ban': [ 'scripts/resolvers/third-party-actions.ts', 'ship/SKILL.md.tmpl', 'ship/sections/apple-release.md.tmpl', 'scripts/gen-skill-docs.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-third-party-actions.test.ts', 'test/helpers/third-party-actions.ts',
'lib/eval-model.ts'
],
'ship-measure-seeded-flake': ['ship/**', 'scripts/ship-measure.ts', 'scripts/lib/measure-bar.ts', 'scripts/test-free-shards.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/paid-select.ts', 'scripts/lib/free-home-guard.ts', 'bin/gstack-config', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/testing.ts', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-ship-measure-loop.test.ts', 'test/helpers/ship-measure-seeded-fixture.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/skill-fixture.ts'],
'ship-measure-seeded-flake': ['ship/**', 'scripts/ship-measure.ts', 'scripts/lib/measure-bar.ts', 'scripts/test-free-shards.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/paid-select.ts', 'scripts/lib/free-home-guard.ts', 'scripts/lib/shard-jobs.ts', 'bin/gstack-config', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/testing.ts', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-ship-measure-loop.test.ts', 'test/helpers/ship-measure-seeded-fixture.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/skill-fixture.ts'],
'ship-section-loading': [ 'ship/**', 'scripts/resolvers/sections.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-ship-section-loading.test.ts',
'scripts/resolvers/testing.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
'plan-ceo-section-loading': [ 'ETHOS.md',
+465
View File
@@ -0,0 +1,465 @@
/**
* Sub-shards inside one CI job (`--ci-run … --jobs k`, scripts/lib/shard-jobs.ts):
* the k = 1 golden path, identity-bound assignment, rejection of missing,
* duplicate and incomplete sub-shard results, one retry pass per CI shard,
* serial TREE_MUTATING and SLOW_UNDER_LOAD files, the timeout text, and per-file
* test and expect() counts from real bun children at k = 1 and k > 1.
*/
import { describe, expect, spyOn, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import {
collectFreeTestFiles, createFreeCiPlan, eligibleFreeRetryFiles, parseCliOptions, runFreeShard, TREE_MUTATING, verifyFreeCiResults,
type FreeCiResult, type FreeShardOutcome,
} from '../scripts/test-free-shards';
import {
assertSoloSeedSample, assignSubShards, ciRunContext, FREE_JOBS_INVALID, mergeSubShardOutcomes, parseJobsFlag, parseJunitCounts,
readJunitReport, runCiShard, SLOW_UNDER_LOAD, subShardRecordProblems, subShardTimeoutText, validateJobsForShard, type CiShardDeps, type RunShardOptions,
} from '../scripts/lib/shard-jobs';
const ROOT = path.resolve(import.meta.dir, '..');
const EXCLUSIVE = Object.keys(TREE_MUTATING)[0];
const files = Array.from({ length: 7 }, (_, i) => `test/sample-${i}.test.ts`);
const durations = Object.fromEntries(files.map((file, i) => [file, (i + 1) * 1_000]));
const context = { planPath: 'plan.json', runId: '123/attempt 2', runner: 'runner-7 (linux, 8 CPUs)' };
const passed = (shardFiles: string[], shard = 1): FreeShardOutcome => ({
shard, files: shardFiles, status: 'passed', exitCode: 0, elapsedMs: 5, groupPid: null, failingFiles: [], unattributedFailures: 0,
summary: { testsRan: shardFiles.length * 2, filesRan: shardFiles.length, sawTerminalSummary: true },
});
const failed = (shardFiles: string[], failing: string[], shard = 1): FreeShardOutcome => ({
...passed(shardFiles, shard), status: 'failed', exitCode: 1, failingFiles: failing,
});
function harness(shardFiles: string[], respond: (files: string[]) => FreeShardOutcome | Promise<FreeShardOutcome> = passed) {
const calls: Array<{ files: string[]; shardNumber: number; totalShards: number; options: RunShardOptions; startedAt: number; endedAt?: number }> = [];
const retries: FreeShardOutcome[][] = [];
const guards: Array<{ files: string[]; shards: number }> = [];
let homeChange: string | null = null;
let terminate = false;
const deps: CiShardDeps = {
async runShard(runFiles, shardNumber, totalShards, options) {
const call: (typeof calls)[number] = { files: runFiles, shardNumber, totalShards, options, startedAt: performance.now() };
calls.push(call);
await new Promise(resolve => setTimeout(resolve, 20));
try { return await respond(runFiles); } finally { call.endedAt = performance.now(); }
},
async retry(outcomes) {
retries.push(outcomes);
return { exitCode: outcomes.every(outcome => outcome.status === 'passed') ? 0 : 1, retry: null };
},
pack: (packFiles, count) => Array.from({ length: count }, (_, i) => packFiles.filter((_, j) => j % count === i)),
isSerial: file => file in TREE_MUTATING || file in SLOW_UNDER_LOAD,
terminationRequested: () => terminate,
phaseGuard: (guarded, shards) => { guards.push({ files: guarded, shards }); return { verify: () => homeChange }; },
log: () => {},
};
const plan = { version: 1 as const, revision: 'revision-a', id: 'plan-id-for-tests', shards: [{ shard: 1, files: shardFiles, predictedMs: 0 }] };
const shard = plan.shards[0];
const run = (jobs: number) => runCiShard({ plan, shard, revision: 'revision-a', jobs, wallTimeoutMs: 360_000, verbose: false, context }, deps);
return { calls, retries, guards, deps, plan, shard, run, setHomeChange: (value: string) => { homeChange = value; }, cancel: () => { terminate = true; } };
}
describe('--jobs validation (FREE_JOBS_INVALID)', () => {
test('accepts positive whole numbers and rejects everything else with the code and the precedence note', () => {
expect(parseJobsFlag('1')).toBe(1);
expect(parseJobsFlag('4')).toBe(4);
for (const raw of ['0', '-1', '2.5', 'abc', '', ' 2', undefined]) {
expect(() => parseJobsFlag(raw)).toThrow(FREE_JOBS_INVALID);
expect(() => parseJobsFlag(raw)).toThrow(/GSTACK_FREE_JOBS sets the local full-suite process count and never changes --ci-run/);
}
});
test('--jobs belongs to --ci-run only, and an absent --jobs leaves the parsed options unchanged', () => {
const base = ['--ci-run', 'plan.json', '--shard', '1', '--result', 'r.json'];
expect(parseCliOptions([...base, '--jobs', '4']).jobs).toBe(4);
expect('jobs' in parseCliOptions(base)).toBe(false);
expect(() => parseCliOptions([...base, '--jobs', '0'])).toThrow(FREE_JOBS_INVALID);
expect(() => parseCliOptions([...base, '--jobs'])).toThrow(FREE_JOBS_INVALID);
expect(() => parseCliOptions(['--jobs', '2'])).toThrow(`${FREE_JOBS_INVALID}: --jobs needs --ci-run`);
expect(() => parseCliOptions(['--ci-plan', 'p.json', '--jobs', '2'])).toThrow(FREE_JOBS_INVALID);
});
test('k may not exceed the shard file count; an empty shard still takes 1', () => {
expect(() => validateJobsForShard(4, 4, 3)).not.toThrow();
expect(() => validateJobsForShard(1, 0, 21)).not.toThrow();
expect(() => validateJobsForShard(5, 4, 3)).toThrow(`${FREE_JOBS_INVALID}: --jobs 5 is more than CI shard 3's 4 file(s); use 1 to 4.`);
expect(() => validateJobsForShard(2, 0, 21)).toThrow(FREE_JOBS_INVALID);
});
});
describe('k = 1 golden: PR 2 changes nothing in CI', () => {
test('one child over the whole planned shard with the same wall, one retry pass, and the same result keys', async () => {
const h = harness(files);
const { result, exitCode } = await h.run(1);
expect(h.calls.map(({ files: f, shardNumber, totalShards, options }) => ({ f, shardNumber, totalShards, options }))).toEqual([
{ f: h.shard.files, shardNumber: 1, totalShards: 1, options: { wallTimeoutMs: 360_000, verbose: false, testCounts: true } },
]);
expect(h.guards).toEqual([]);
expect(h.retries).toEqual([[passed(h.shard.files)]]);
expect(h.calls[0].options.wallTimeoutMs).toBe(360_000);
expect(Object.keys(result)).toEqual(['planId', 'revision', 'outcome', 'retry']);
expect(result).toEqual({ planId: h.plan.id, revision: 'revision-a', outcome: passed(h.shard.files), retry: null });
expect(exitCode).toBe(0);
});
test('a k = 1 failure reaches the retry pass as the child reported it', async () => {
const h = harness(files, runFiles => failed(runFiles, [runFiles[0]]));
const { result, exitCode } = await h.run(1);
expect(h.retries).toEqual([[failed(h.shard.files, [h.shard.files[0]])]]);
expect(result.outcome).toBe(h.retries[0][0]);
expect(exitCode).toBe(1);
});
});
describe('sub-shard assignment', () => {
const pack = (packFiles: string[], count: number) => Array.from({ length: count }, (_, i) => packFiles.filter((_, j) => j % count === i));
test('disjoint, identity-bound, covering the shard once, with exclusive files in one trailing serial sub-shard', () => {
const shardFiles = [...files, EXCLUSIVE];
const assignments = assignSubShards('plan-id-for-tests', { shard: 4, files: shardFiles }, 3, file => file in TREE_MUTATING, pack);
expect(assignments.map(entry => [entry.id, entry.phase, entry.files.length])).toEqual([
['plan-id-for-:4.1', 'parallel', 3], ['plan-id-for-:4.2', 'parallel', 2], ['plan-id-for-:4.3', 'parallel', 2], ['plan-id-for-:4.4', 'serial', 1],
]);
expect(assignments.at(-1)!.files).toEqual([EXCLUSIVE]);
expect(assignments.flatMap(entry => entry.files).sort()).toEqual([...shardFiles].sort());
});
test('k is clamped to the parallel file count and a shard of only exclusive files stays one serial child', () => {
expect(assignSubShards('p', { shard: 1, files: files.slice(0, 2) }, 4, () => false, pack).length).toBe(2);
expect(assignSubShards('p', { shard: 1, files: [EXCLUSIVE] }, 1, file => file in TREE_MUTATING, pack)
.map(entry => entry.phase)).toEqual(['serial']);
});
test('a packer that drops or repeats a file is rejected before any child starts', () => {
expect(() => assignSubShards('p', { shard: 2, files }, 2, () => false, packFiles => [packFiles, [packFiles[0]]])).toThrow(/FREE_SUBSHARD_REJECTED/);
expect(() => assignSubShards('p', { shard: 2, files }, 2, () => false, packFiles => [packFiles.slice(1)])).toThrow(/exactly once/);
});
});
describe('k > 1: merge, rejection and one retry pass per CI shard', () => {
test('parallel sub-shards share the concurrent home guard, then exclusive files run alone after they settle', async () => {
const h = harness([...files, EXCLUSIVE]);
const { result, exitCode } = await h.run(3);
expect(exitCode).toBe(0);
const parallel = h.calls.filter(call => !call.files.includes(EXCLUSIVE));
const serial = h.calls.find(call => call.files.includes(EXCLUSIVE))!;
expect(parallel).toHaveLength(3);
expect(parallel.every(call => typeof call.options.homeGuard === 'function')).toBe(true);
expect(serial.options.homeGuard).toBeUndefined();
expect(serial.files).toEqual([EXCLUSIVE]);
expect(serial.startedAt).toBeGreaterThanOrEqual(Math.max(...parallel.map(call => call.endedAt!)));
expect(h.guards).toEqual([{ files: parallel.flatMap(call => call.files), shards: 3 }]);
expect(h.calls.every(call => call.totalShards === 4)).toBe(true);
expect(parallel.every(call => call.options.wallTimeoutMs === 360_000)).toBe(true);
expect(serial.options.wallTimeoutMs).toBeLessThan(360_000);
expect(serial.options.wallTimeoutMs).toBeGreaterThan(350_000);
expect(result.jobs).toBe(3);
expect(result.outcome).toMatchObject({ shard: 1, files: h.shard.files, status: 'passed', exitCode: 0, failingFiles: [], unattributedFailures: 0,
summary: { testsRan: 16, filesRan: 8, sawTerminalSummary: true } });
expect(result.subShards!.map(record => record.status)).toEqual(['passed', 'passed', 'passed', 'passed']);
expect(h.retries).toHaveLength(1);
expect(() => verifyFreeCiResults(h.plan, [result])).not.toThrow();
});
test('failures from every sub-shard are collected first, so k never multiplies the five-file retry cap', async () => {
const h = harness(files, runFiles => failed(runFiles, runFiles.slice(0, 2)));
const { result, exitCode } = await h.run(3);
expect(h.retries).toHaveLength(1);
expect(h.retries[0]).toHaveLength(1);
expect(result.outcome.failingFiles).toHaveLength(6);
expect(eligibleFreeRetryFiles(h.retries[0])).toBeNull();
expect(exitCode).toBe(1);
const spread = harness(files, runFiles => failed(runFiles, runFiles.slice(0, 1)));
await spread.run(3);
expect(eligibleFreeRetryFiles(spread.retries[0])!.sort()).toEqual(spread.calls.map(call => call.files[0]).sort());
});
test('a missing, duplicate or incomplete sub-shard result fails the CI shard and vetoes the retry', async () => {
const errors = spyOn(console, 'error').mockImplementation(() => {});
try {
const cases: Array<(runFiles: string[]) => FreeShardOutcome | Promise<FreeShardOutcome>> = [
runFiles => runFiles.includes(files[0]) ? Promise.reject(new Error('spawn failed')) : passed(runFiles),
runFiles => passed(runFiles.includes(files[0]) ? [...runFiles, files[1]] : runFiles),
runFiles => runFiles.includes(files[0]) ? { ...passed(runFiles), summary: { testsRan: 1, filesRan: runFiles.length - 1, sawTerminalSummary: true } } : passed(runFiles),
runFiles => runFiles.includes(files[0]) ? { ...passed(runFiles), summary: undefined } : passed(runFiles),
];
for (const respond of cases) {
errors.mockClear();
const h = harness(files, respond);
const { result, exitCode } = await h.run(3);
expect(h.calls).toHaveLength(3);
expect(result.outcome.status).toBe('failed');
expect(result.outcome.unattributedFailures).toBeGreaterThan(0);
expect(eligibleFreeRetryFiles([result.outcome])).toBeNull();
expect(exitCode).toBe(1);
expect(errors.mock.calls.map(call => String(call[0])).some(line => line.startsWith('[test:free] FREE_SUBSHARD_REJECTED: CI shard 1: plan-id-for-:1.'))).toBe(true);
}
} finally { errors.mockRestore(); }
});
test('a failed sub-shard that ran too few or too many files is rejected, so counts cannot cancel and no retry repairs it', async () => {
const errors = spyOn(console, 'error').mockImplementation(() => {});
try {
const h = harness(files.slice(0, 4), runFiles => ({ ...failed(runFiles, [runFiles[0]]),
summary: { testsRan: 2, filesRan: runFiles.includes(files[0]) ? 1 : 3, sawTerminalSummary: true } }));
const { result } = await h.run(2);
expect(result.outcome.summary!.filesRan).toBe(4);
expect(result.outcome.unattributedFailures).toBeGreaterThanOrEqual(2);
expect(eligibleFreeRetryFiles([result.outcome])).toBeNull();
const forged: FreeCiResult = { ...result, outcome: { ...result.outcome, unattributedFailures: 0 },
retry: { ...passed([files[0], files[1]]), shard: 2 } };
expect(() => verifyFreeCiResults({ ...h.plan, revision: 'revision-a' } as never, [forged])).toThrow(/did not run every assigned file|identity/);
expect(subShardRecordProblems(forged)).toContain('a sub-shard did not run every assigned file');
} finally { errors.mockRestore(); }
});
test('the serial sub-shard gets only what is left of the wall, and none left means it is not launched', async () => {
const errors = spyOn(console, 'error').mockImplementation(() => {});
try {
const h = harness([...files, EXCLUSIVE], async runFiles => {
if (!runFiles.includes(EXCLUSIVE)) await new Promise(resolve => setTimeout(resolve, 60));
return passed(runFiles);
});
const { result } = await runCiShard({ plan: h.plan, shard: h.shard, revision: 'revision-a', jobs: 2, wallTimeoutMs: 50, verbose: false, context }, h.deps);
expect(h.calls.some(call => call.files.includes(EXCLUSIVE))).toBe(false);
expect(result.subShards!.at(-1)).toMatchObject({ status: 'missing', phase: 'serial' });
expect(result.outcome.status).toBe('failed');
} finally { errors.mockRestore(); }
});
test('a real-home change in the concurrent phase and a cancelled serial phase both fail the shard', async () => {
const errors = spyOn(console, 'error').mockImplementation(() => {});
try {
const home = harness(files);
home.setHomeChange('real home changed while 3 shards ran concurrently');
expect((await home.run(3)).result.outcome.status).toBe('failed');
const cancelled = harness([...files, EXCLUSIVE]);
cancelled.cancel();
const { result } = await cancelled.run(2);
expect(cancelled.calls.some(call => call.files.includes(EXCLUSIVE))).toBe(false);
expect(result.subShards!.at(-1)).toMatchObject({ status: 'missing', files: [EXCLUSIVE] });
expect(result.outcome.status).toBe('failed');
} finally { errors.mockRestore(); }
});
test('a sub-shard timeout prints FREE_SUBSHARD_TIMEOUT with the file, k, runner, run id and a serial reproduce', async () => {
const errors = spyOn(console, 'error').mockImplementation(() => {});
try {
const h = harness(files, runFiles => runFiles.includes(files[3])
? { ...failed(runFiles, []), status: 'timed-out', exitCode: null, inFlight: [files[3]] } : passed(runFiles));
const { result, exitCode } = await h.run(3);
expect(result.outcome.status).toBe('timed-out');
expect(exitCode).toBe(1);
const text = errors.mock.calls.map(call => String(call[0])).join('\n');
expect(text).toContain('FREE_SUBSHARD_TIMEOUT: CI shard 1 sub-shard plan-id-for-:1.1 (3 files)');
expect(text).toContain('hit its 360s wall at --jobs 3 on runner-7 (linux, 8 CPUs), run 123/attempt 2.');
expect(text).toContain(`in flight: ${files[3]}`);
expect(text).toContain(`printf '%s\\n' '${files[3]}' > files.txt && GSTACK_FREE_JOBS=1 bun run scripts/test-free-shards.ts --files-from files.txt`);
expect(text).toContain("--ci-run 'plan.json' --shard 1 --result shard-1.json --jobs 1");
expect(text).toContain('add it to SLOW_UNDER_LOAD in scripts/lib/shard-jobs.ts with run id 123/attempt 2; never raise the timeout.');
} finally { errors.mockRestore(); }
});
test('--ci-verify rejects tampered sub-shard records', () => {
const plan = createFreeCiPlan(files, 1, durations, 'revision-a');
const shard = plan.shards[0];
const assignments = assignSubShards(plan.id, shard, 2, () => false, packFiles => [packFiles.slice(0, 3), packFiles.slice(3)]);
const merged = mergeSubShardOutcomes(shard, assignments.map(assignment => ({ assignment, outcome: passed(assignment.files) })), 10);
const result: FreeCiResult = { planId: plan.id, revision: 'revision-a', outcome: merged.outcome, retry: null, jobs: 2, subShards: merged.records };
expect(() => verifyFreeCiResults(plan, [result])).not.toThrow();
const renamed = structuredClone(result);
renamed.subShards![0].id = 'other:1.1';
expect(() => verifyFreeCiResults(plan, [renamed])).toThrow(/ids or phases/);
const tooMany = structuredClone(result);
tooMany.jobs = 1;
expect(subShardRecordProblems({ ...tooMany, jobs: 2, subShards: [...tooMany.subShards!, { ...tooMany.subShards![0], id: `${plan.id.slice(0, 12)}:1.3`, files: [] }] }))
.toContain('sub-shard ids or phases do not match the plan and --jobs');
const overlap = structuredClone(result);
overlap.subShards![1].files = [...overlap.subShards![1].files, overlap.subShards![0].files[0]];
expect(() => verifyFreeCiResults(plan, [overlap])).toThrow(/sub-shards rejected/);
const hidden = structuredClone(result);
hidden.subShards![0].status = 'failed';
expect(() => verifyFreeCiResults(plan, [hidden])).toThrow(/a passing shard holds a failed sub-shard/);
expect(() => verifyFreeCiResults(plan, [{ ...result, jobs: 1 }])).toThrow(/valid --jobs/);
});
});
describe('per-file test and expect() counts from bun JUnit', () => {
const xml = (file: string) => `<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="bun test" tests="6" assertions="9" failures="1" skipped="2">
<testsuite name="${file}" file="${file}" tests="3" assertions="9" failures="1" skipped="2">
<testsuite name="grp" file="${file}" tests="1" assertions="5">
<testcase name="x &gt; y &quot;q&quot;" classname="grp" file="${file}" line="2" assertions="5" />
</testsuite>
<testcase name="todo" classname="" file="${file}" line="3" assertions="0"><skipped message="TODO" /></testcase>
<testcase name="skip" classname="" file="${file}" line="4" assertions="0"><skipped /></testcase>
<testcase name="fails" classname="" file="${file}" line="5" assertions="4"><failure type="AssertionError" message="a &lt;b&gt;">x</failure></testcase>
</testsuite>
</testsuites>`;
test('classifies every testcase and sums assertions per file; matches cwd-relative paths back to planned files', () => {
const counts = parseJunitCounts(xml('../../test/a&amp;b.test.ts'), ['test/a&b.test.ts'], 4);
expect(counts).toEqual({ source: 'bun-junit', complete: true, files: { 'test/a&b.test.ts': { tests: 4, pass: 1, fail: 1, skip: 1, todo: 1, expects: 9 } },
totals: { tests: 4, pass: 1, fail: 1, skip: 1, todo: 1, expects: 9 } });
});
test('an ambiguous short path is never credited to one of two planned files that share it', () => {
const counts = parseJunitCounts(xml('a.test.ts'), ['test/a.test.ts', 'browse/test/a.test.ts'], 4);
expect(counts.complete).toBe(false);
expect(counts.problem).toContain('unplanned');
});
test('an absolute path is credited to the planned file under the root; an unplanned file never borrows a shorter planned path', () => {
const planned = ['test/telemetry.test.ts', 'browse/test/telemetry.test.ts'];
expect(Object.keys(parseJunitCounts(xml('/repo/browse/test/telemetry.test.ts'), planned, 4, '/repo').files)).toEqual(['browse/test/telemetry.test.ts']);
expect(Object.keys(parseJunitCounts(xml('../../repo/test/telemetry.test.ts'), planned, 4, '/repo').files)).toEqual(['test/telemetry.test.ts']);
const borrowed = parseJunitCounts(xml('browse/test/x.test.ts'), ['test/x.test.ts'], 4, '/repo');
expect(borrowed).toMatchObject({ complete: false, problem: expect.stringContaining('browse/test/x.test.ts') });
expect(parseJunitCounts(xml('/elsewhere/test/x.test.ts'), ['test/x.test.ts'], 4, '/repo').complete).toBe(false);
});
test('the report is read only as a bounded regular file, never through a link or a FIFO', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-junit-'));
try {
fs.writeFileSync(path.join(dir, 'real.xml'), '<testsuites/>');
expect(readJunitReport(path.join(dir, 'real.xml'))).toBe('<testsuites/>');
expect(readJunitReport(path.join(dir, 'absent.xml'))).toBeNull();
if (process.platform !== 'win32') {
fs.symlinkSync(path.join(dir, 'real.xml'), path.join(dir, 'link.xml'));
expect(readJunitReport(path.join(dir, 'link.xml'))).toBeNull();
const fifo = Bun.spawnSync(['mkfifo', path.join(dir, 'fifo.xml')], { timeout: 5_000 });
if (fifo.exitCode === 0) expect(readJunitReport(path.join(dir, 'fifo.xml'))).toBeNull();
}
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
});
test('a missing report, an unplanned file or a count that disagrees with bun summary is incomplete, never silently complete', () => {
expect(parseJunitCounts(null, ['test/a.test.ts'], 4)).toMatchObject({ complete: false, problem: 'bun wrote no JUnit report' });
expect(parseJunitCounts(xml('test/z.test.ts'), ['test/a.test.ts'], 4)).toMatchObject({ complete: false, problem: expect.stringContaining('unplanned') });
expect(parseJunitCounts(xml('test/a.test.ts'), ['test/a.test.ts'], 5)).toMatchObject({ complete: false, problem: "JUnit counted 4 tests, bun's summary 5" });
expect(parseJunitCounts(xml('test/a.test.ts'), ['test/a.test.ts'], null).complete).toBe(false);
});
test('real bun children: counts are identical per file at k = 1 and k = 2, add up to bun summary, and leave the stream unchanged', async () => {
const root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-shard-jobs-')));
try {
const fixtures: Record<string, string> = {
't/a.test.ts': "import { test, expect } from 'bun:test';\ntest('a1', () => { expect(1).toBe(1); expect(2).toBe(2); });\ntest('a2', () => { expect(3).toBe(3); });\ntest.skip('a3', () => {});\n",
't/b.test.ts': "import { test, expect, describe } from 'bun:test';\ndescribe('g', () => { test('b1', () => { for (let i = 0; i < 5; i++) expect(i).toBe(i); }); });\ntest.todo('b2');\n",
't/c.test.ts': "import { test, expect } from 'bun:test';\ntest('c1', () => { expect([1]).toEqual([1]); });\n",
't/d.test.ts': "import { test, expect } from 'bun:test';\ntest.each([1, 2, 3])('d%p', n => { expect(n).toBeGreaterThan(0); expect(n).toBeLessThan(4); });\n",
};
for (const [file, body] of Object.entries(fixtures)) {
fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true });
fs.writeFileSync(path.join(root, file), body);
}
const planned = Object.keys(fixtures);
const plan = createFreeCiPlan(planned, 1, {}, 'rev');
const pids: number[] = [];
const deps = (logs: string[]): CiShardDeps => ({
runShard: async (runFiles, n, total, options) => {
const logFilePath = path.join(root, `stream-${logs.length}-${runFiles.length}-${n}-${total}.log`);
logs.push(logFilePath);
const outcome = await runFreeShard(runFiles, n, total, { ...options, rootDir: root, quiet: true, log: () => {}, logFilePath });
if (outcome.groupPid) pids.push(outcome.groupPid);
return outcome;
},
retry: async outcomes => ({ exitCode: outcomes[0].status === 'passed' ? 0 : 1, retry: null }),
pack: (packFiles, count) => Array.from({ length: count }, (_, i) => packFiles.filter((_, j) => j % count === i)),
isSerial: () => false,
terminationRequested: () => false,
log: () => {},
});
const input = { plan, shard: plan.shards[0], revision: 'rev', wallTimeoutMs: 60_000, verbose: false, context };
const serialLogs: string[] = [];
const one = await runCiShard({ ...input, jobs: 1 }, deps(serialLogs));
const two = await runCiShard({ ...input, jobs: 2 }, deps([]));
expect([one.exitCode, two.exitCode]).toEqual([0, 0]);
const expected = {
't/a.test.ts': { tests: 3, pass: 2, fail: 0, skip: 1, todo: 0, expects: 3 },
't/b.test.ts': { tests: 2, pass: 1, fail: 0, skip: 0, todo: 1, expects: 5 },
't/c.test.ts': { tests: 1, pass: 1, fail: 0, skip: 0, todo: 0, expects: 1 },
't/d.test.ts': { tests: 3, pass: 3, fail: 0, skip: 0, todo: 0, expects: 6 },
};
expect(one.result.outcome.counts).toMatchObject({ complete: true, files: expected, totals: { tests: 9, expects: 15 } });
expect(two.result.outcome.counts).toEqual(one.result.outcome.counts);
expect(two.result.subShards).toHaveLength(2);
expect(one.result.outcome.summary).toEqual(two.result.outcome.summary);
const bare = await runFreeShard(planned, 1, 1, { rootDir: root, quiet: true, log: () => {}, logFilePath: path.join(root, 'bare.log') });
expect(bare.counts).toBeUndefined();
const stream = (file: string) => fs.readFileSync(file, 'utf8').replace(/ \[[\d.]+m?s\]/g, '').split('\n').filter(line => !line.startsWith('[test:free]'));
expect(stream(serialLogs[0])).toEqual(stream(path.join(root, 'bare.log')));
for (const pid of pids) expect(() => process.kill(pid, 0)).toThrow();
} finally { fs.rmSync(root, { recursive: true, force: true }); }
}, 60_000);
test('a hung sub-shard is killed at its wall while its sibling finishes, and no child outlives the parent', async () => {
const root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-shard-jobs-')));
const errors = spyOn(console, 'error').mockImplementation(() => {});
try {
const planned = ['t/ok.test.ts', 't/hang.test.ts'];
for (const file of planned) {
fs.mkdirSync(path.join(root, 't'), { recursive: true });
fs.writeFileSync(path.join(root, file), "import { test, expect } from 'bun:test';\ntest('ok', () => { expect(1).toBe(1); });\n");
}
const plan = createFreeCiPlan(planned, 1, {}, 'rev');
const outcomes: FreeShardOutcome[] = [];
const startedAt = performance.now();
const { result } = await runCiShard({ plan, shard: plan.shards[0], revision: 'rev', jobs: 2, wallTimeoutMs: 2_000, verbose: false, context }, {
runShard: async (runFiles, n, total, options) => {
const outcome = await runFreeShard(runFiles, n, total, { ...options, rootDir: root, quiet: true, log: () => {},
...(runFiles.includes('t/hang.test.ts') ? { commandFor: () => ({ command: process.execPath, args: ['-e', 'setInterval(() => {}, 1000)'] }) } : {}) });
outcomes.push(outcome);
return outcome;
},
retry: async () => ({ exitCode: 1, retry: null }),
pack: packFiles => packFiles.map(file => [file]),
isSerial: () => false,
terminationRequested: () => false,
log: () => {},
});
expect(performance.now() - startedAt).toBeLessThan(20_000);
expect(result.outcome.status).toBe('timed-out');
expect(result.subShards!.map(record => [record.files[0], record.status]).sort()).toEqual([['t/hang.test.ts', 'timed-out'], ['t/ok.test.ts', 'passed']]);
for (const outcome of outcomes) expect(() => process.kill(outcome.groupPid!, 0)).toThrow();
expect(errors.mock.calls.some(call => String(call[0]).includes('FREE_SUBSHARD_TIMEOUT'))).toBe(true);
} finally { errors.mockRestore(); fs.rmSync(root, { recursive: true, force: true }); }
}, 60_000);
});
describe('serial lists and the duration seed', () => {
test('SLOW_UNDER_LOAD entries name live free files, a reason and a CI run id, and never overlap TREE_MUTATING', () => {
const census = new Set(collectFreeTestFiles());
for (const [file, entry] of Object.entries(SLOW_UNDER_LOAD)) {
expect(census.has(file), `SLOW_UNDER_LOAD names ${file}, which is not a free test file`).toBe(true);
expect(file in TREE_MUTATING).toBe(false);
expect(entry.reason.trim()).not.toBe('');
expect(entry.runId).toMatch(/^\d+(\/attempt \d+)?$/);
}
});
test('seeds take only one-file children; sub-shard runs have no path to the seed writer', () => {
expect(() => assertSoloSeedSample('test/a.test.ts', { files: ['test/a.test.ts'] })).not.toThrow();
expect(() => assertSoloSeedSample('test/a.test.ts', { files: ['test/a.test.ts', 'test/b.test.ts'] })).toThrow(/one-file serial children/);
expect(() => assertSoloSeedSample('test/a.test.ts', { files: ['test/a.test.ts'], summary: { testsRan: 9, filesRan: 2, sawTerminalSummary: true } })).toThrow(/one-file serial children/);
expect(fs.readFileSync(path.join(ROOT, 'scripts/lib/shard-jobs.ts'), 'utf8')).not.toContain('writeDurationSeed');
const runner = fs.readFileSync(path.join(ROOT, 'scripts/test-free-shards.ts'), 'utf8');
const recorder = runner.slice(runner.indexOf('async function recordFreeTestDurations('), runner.indexOf('async function retryFailedFreeFiles('));
expect(runner.split('writeDurationSeed(').length - 1).toBe(recorder.split('writeDurationSeed(').length - 1);
expect(recorder).toContain('assertSoloSeedSample(file, outcome)');
});
test('the run context names the CI run and attempt, else local', () => {
expect(ciRunContext('p.json', { GITHUB_RUN_ID: '42', GITHUB_RUN_ATTEMPT: '3', RUNNER_NAME: 'ubi-1' })).toMatchObject({ runId: '42/attempt 3', runner: expect.stringMatching(/^ubi-1 \(/) });
expect(ciRunContext('p.json', {}).runId).toBe('local');
expect(subShardTimeoutText(2, 4, { id: 'x:2.1', phase: 'parallel', files: ['test/a.test.ts'], status: 'timed-out', exitCode: null, elapsedMs: 1, failingFiles: [], unattributedFailures: 0 },
[], 10_000, context)[1]).toContain('in flight: unknown');
const serial = { id: 'x:2.5', phase: 'serial' as const, files: ['test/s.test.ts'], status: 'timed-out' as const, exitCode: null, elapsedMs: 1, failingFiles: [], unattributedFailures: 0 };
const squeezed = subShardTimeoutText(2, 4, serial, [], 4_000, context, 360_000);
expect(squeezed[0]).toContain('hit its 4s wall');
expect(squeezed[4]).toContain('left this serial sub-shard 4s of the 360s shard wall');
expect(subShardTimeoutText(2, 4, serial, [], 360_000, context, 360_000)[4]).toContain('already runs alone');
});
});
+1 -1
View File
@@ -196,7 +196,7 @@ describe('test-free-shards: exclusive host-state phase', () => {
let child: ReturnType<typeof Bun.spawn> | undefined;
let watchdog: ReturnType<typeof setTimeout> | undefined;
try {
for (const file of ['scripts/test-free-shards.ts', 'scripts/test-strict-output.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/free-home-guard.ts', 'scripts/lib/free-ci-health.ts', 'scripts/lib/windows-curation.ts', 'lib/state-root.ts',
for (const file of ['scripts/test-free-shards.ts', 'scripts/test-strict-output.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/free-home-guard.ts', 'scripts/lib/free-ci-health.ts', 'scripts/lib/windows-curation.ts', 'scripts/lib/shard-jobs.ts', 'lib/state-root.ts',
'test/helpers/paid-test-set.ts', 'test/helpers/touchfiles.ts', 'test/helpers/touchfiles-data.ts', 'test/helpers/test-selection.ts']) {
const target = path.join(directory, file);
fs.mkdirSync(path.dirname(target), { recursive: true });
+1 -1
View File
@@ -20,7 +20,7 @@ afterAll(() => fs.rmSync(SCRATCH, { recursive: true, force: true }));
const COPIED = ['scripts/ubicloud/pregate-remote.sh', 'bin/gstack-wtree', 'bin/gstack-path-spelling.sh',
'scripts/test-free-shards.ts', 'scripts/test-strict-output.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/free-home-guard.ts',
'scripts/lib/free-ci-health.ts', 'scripts/lib/windows-curation.ts', 'lib/state-root.ts', 'test/helpers/paid-test-set.ts',
'scripts/lib/free-ci-health.ts', 'scripts/lib/windows-curation.ts', 'scripts/lib/shard-jobs.ts', 'lib/state-root.ts', 'test/helpers/paid-test-set.ts',
'test/helpers/touchfiles.ts', 'test/helpers/touchfiles-data.ts', 'test/helpers/test-selection.ts'];
const TESTS: Record<string, string> = {
'test/alpha.test.ts': "import { expect, test } from 'bun:test';\ntest('alpha', () => { expect(1).toBe(1); });\n",