mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-09 22:48:57 +02:00
* fix(freeze): hook reads the same state root /freeze writes — fails closed under GSTACK_HOME (#1459, #1509) check-freeze.sh resolved its state dir as ${CLAUDE_PLUGIN_DATA:-$HOME/.gstack} while every writer (/freeze, /guard, /unfreeze, /investigate) resolves through bin/gstack-paths, GSTACK_HOME first. With GSTACK_HOME set, /freeze wrote freeze-dir.txt under GSTACK_HOME, the hook read $HOME/.gstack, found no file, and allowed everything — a deny-tier boundary failing open. One resolver now: gstack_hook_state_root() in careful/bin/hook-extract.sh (already sourced by both check-freeze.sh and check-careful.sh) implements the exact gstack-paths chain, including the CLAUDE_PLUGIN_ROOT guard that keeps a CLAUDE_PLUGIN_DATA leaked from another plugin from redirecting our state. check-freeze.sh and gstack_hook_log_fire both call it; nothing spawns gstack-paths from a hook. Tests: the GSTACK_HOME deny regression, GSTACK_HOME-over-CLAUDE_PLUGIN_DATA precedence, plugin-root guard both ways, and a byte-parity check against bin/gstack-paths across six env combinations. Existing freeze tests now pass CLAUDE_PLUGIN_ROOT like a real plugin install would. Idea from PR #1509 (@NikhileshNanduri); implemented natively against the shared resolver rather than a second fallback chain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(relink): never delete or link over a skill gstack does not own (#2119) gstack-relink runs on every ./setup. Its cleanup did `rm -rf` on any same-name entry whose SKILL.md was a symlink, with no readlink check, and its link step did `mkdir -p` then `ln -snf` onto any existing SKILL.md — on Linux that replaces a user's real file with a symlink into gstack (macOS refused by accident). setup's Windows mode-flip cleanup deleted any real dir whose name matched a gstack skill. A personal `qa` skill, or a fork installed under another path, was destroyed by the installer of a tool it never asked for. Ownership is now proven, never assumed. An entry is ours when it is a symlink resolving into INSTALL_DIR or RENDER_DIR, a real dir whose SKILL.md is such a symlink, or a real dir carrying the .gstack-owned marker setup now writes for Windows copy installs (legacy copies count when byte-identical to the source or carrying gen-skill-docs' AUTO-GENERATED header). Anything else — including an entry whose readlink fails — is foreign: left untouched, reported on stderr, and listed in relink's summary line. The same rule replaces setup's Windows name-match deletion; setup:1040 and gstack-uninstall:204 already gated on readlink, so this closes the last unguarded deleter of the class. Tests: foreign real dir in flat mode, foreign flat entry on a prefix flip, foreign directory symlink, RENDER_DIR-targeted entry (ours), marker-carrying copy (ours), marker-less copy (foreign); the Windows cleanup test now proves provenance three ways and keeps the user's own same-name skill. Idea and two regression cases from PR #2119 (@smblight); implemented on the destination entry, not only the symlink target. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(setup): Chromium bootstrap is best-effort and bounded — skills always register (#1900, #1901, #1902, #913, #2233) setup runs under `set -e`, and the Chromium bootstrap in section 2 sat ahead of skill registration in section 4 with a bare `bunx playwright install chromium`, an unbounded download, and an explicit `exit 1` after the post-install launch probe. On an offline, proxied, or AppArmor-restricted box the user ended with ZERO skills registered and a re-run that died at the same line; a wedged download hung setup indefinitely. Every browser failure now records a reason code in _PW_FAIL_REASON and setup continues: skipped (GSTACK_SKIP_PLAYWRIGHT=1, #913), chromium-install, chromium-install-timeout (the download is bounded by the existing _wait_with_deadline helper, default 600s, env GSTACK_PLAYWRIGHT_INSTALL_TIMEOUT, process tree killed via _kill_tree), chromium-install-locked (another setup holds the lock: this one registers skills and re-probes next time instead of exiting), windows-no-node, windows-node-modules, post-install-launch (with the GSTACK_CHROMIUM_NO_SANDBOX=1 hint for Ubuntu 24.04's userns policy, #2157). The daemon font refresh is skipped when Chromium is unavailable. The final summary names the skills that need the browser (/qa, /qa-only, /design-review, /browse, make-pdf, /pair-agent) and the fix for the recorded reason, and logs the reason code (never a path) through gstack-telemetry-log when telemetry is on. Tests: static invariants over the anchor-sliced block (no exit, every reason code, deadline helper, trap chaining, guarded refresh, summary contents) plus an integration harness that executes the real block with a stubbed probe and installer: install failure, hang killed at the deadline with the tree kill recorded, non-numeric knob fallback, live lock (continues, installer not run, lock preserved), stale lock reclaimed, post-install probe failure, and the skip flag. Credit @DavidMiserak (PR #1900) for the best-effort shape; re-implemented on the current block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(designs): preserve the time-attack fork-port residual evaluation The read-only evaluation of what remains portable from time-attack/gstack (583 raw candidates, 415 canonical, 287 with a residual, 48 adversarially refuted, 14 standing) lived only on a throwaway VM. This records the report, the lite residual index, the absorbed/superseded ledger, the refuter verdicts, and SHAS.md with the fork tip, upstream HEAD, merge-base, and a sha256 per file, so every scheduled fix in this wave series traces to its evidence. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: file the fork-port residual deferrals and document the Chromium bootstrap knobs TODOS.md gains the seven items the CEO and eng reviews of the fork-port residual plan deliberately deferred (shared ownership helper, config-key reader tripwire, "pre-existing" vocabulary, opt-in reply_language, .auth.json writer removal, the fork-derived-change rule for CONTRIBUTING, hook slug parity audit), each with rationale, and updates the two residual bullets for PR #2232 and PR #2233 with their dispositions. README's Troubleshooting section explains the best-effort Chromium bootstrap and its three knobs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(relink): canonicalize link targets before the ownership check Pre-landing review finding: the ownership gate compared readlink output textually against INSTALL_DIR and RENDER_DIR, so two shapes of gstack's OWN entries read as foreign and were left behind on a mode flip — a legacy relative link (`gstack/qa/SKILL.md`, resolved against $PWD instead of the link's directory) and an entry linked against the real path of a symlinked install dir (~/.claude/skills/gstack -> checkout). Both now resolve: relative targets anchor at the link's directory, the directory part is canonicalized with pwd -P (the basename stays verbatim so a dangling managed target is not misread), and both spellings of each root are accepted. Two regression tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(telemetry): one-shot setup events never sweep other sessions' pending markers gstack-telemetry-log finalizes every .pending-<session> marker that is not the caller's own as outcome:unknown and deletes it. setup's onboarding events (_setup_welcome, _setup_playwright) have no session of their own, so a Chromium bootstrap failure during a live skill session recorded a false unknown for that session and removed its marker. New --no-sweep flag skips the stale-marker pass; both setup call sites use it (the synthetic --session-id did not prevent the sweep). Surfaced by the Codex adversarial pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(hooks): partial upgrades fail closed for freeze and fall back for careful A hook script and its sourced helper can be copied at different times. With an older careful/bin/hook-extract.sh that lacks gstack_hook_state_root: - check-freeze.sh now emits a deny ("fail closed, re-run ./setup or /unfreeze") instead of dying under set -e with no decision JSON. - check-careful.sh falls back to ${GSTACK_HOME:-$HOME/.gstack} so project rules under the plain chain still load and a decision is always emitted (a warn hook must never break on a stale helper). gstack_hook_state_root prints its root without a trailing newline and both callers capture it with a printf-x sentinel, so a GSTACK_HOME ending in a newline round-trips byte-for-byte with the writer's %q form. gstack_hook_log_fire stays on ${GSTACK_HOME:-$HOME/.gstack}/analytics, the same two-step chain every other analytics writer and reader uses, so the usage log remains one file under a plugin install. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(setup): never link over, copy over, or reap a skill gstack does not own (#2119) The relink gate alone left three destructive sites open: - link_claude_skill_dirs runs BEFORE relink on every ./setup and used `ln -snf` (Linux replaces a user's real SKILL.md with a symlink into gstack) or, on Windows, rm -rf + cp followed by a marker that made the user's directory "ours" on the next flip. It and _install_alias_skill_md now consult _claude_entry_is_ours first and skip loudly. - cleanup_prefixed_claude_symlinks kept a bare name-match deletion and a `*gstack*` substring match. Symlink arms use anchored `gstack/` segment patterns; the Windows real-file arm proves provenance (marker, byte-identity with our source, or the full two-line gen-skill-docs banner within the first 40 lines, never a one-line substring another generator could emit). cleanup_old_claude_symlinks uses the same banner rule. - gstack-relink's fast path judged absolute targets before canonicalizing, so `/x/gstack/../foreign/SKILL.md` counted as ours; dot-segment targets now canonicalize first. Its banner rule matches setup's. The `.gstack-owned` marker records the owning payload's realpath. Entries skipped by setup or relink are listed in the final setup summary. Chromium bootstrap refinements from the pre-landing review: an INT/TERM trap kills the installer's process tree; the Windows npm chain no longer masks an install failure; GSTACK_SKIP_PLAYWRIGHT=1 is reported as a choice rather than a failure and sends no telemetry; the timeout knob is normalized (0, 000, non-numeric, or more than nine digits fall back to the 600s default instead of killing on the first poll or never killing). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: README Chromium note outside the CLAUDE.md fence; report banner stripped; deferrals name the four gate sites - README: the Chromium troubleshooting paragraph sat inside the CLAUDE.md snippet code fence, so copy-paste put it into users' CLAUDE.md. Moved to the troubleshooting list. - docs/designs/fork-port-residual-2026-09/REPORT.md: the scratch-run preamble banner is gone; SHAS.md re-hashed. - TODOS: the ownership-gate deferral names the four sites and the marker-path idea for the fork-with-banner residual. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(todos): the bootstrap block coverage gap is pinned except the quarantine helper Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(setup,relink): ownership proof has two strengths; weak proof never deletes a directory or discards a differing file The first #2119 gate treated a byte-identical or banner-bearing real-file SKILL.md as full ownership, so a prefix flip could rm -rf a user's directory (their own qa skill started from a gstack SKILL.md, plus my-templates/) and the link pass could replace their customized file with a symlink. Two strengths now: - STRONG: the .gstack-owned marker (we created the directory), or a directory holding nothing but symlinks and the marker (deleting it loses no data). Only strong proof removes a directory whole. - WEAK: byte-identity with our source or the two-line gen-skill-docs banner on a real file. Weak proof covers that SKILL.md and our runtime-asset links only; a differing file is moved to ${GSTACK_HOME:-~/.gstack}/backups/skills/<ts>/<skill>/ before we link over it, and setup/relink print one summary line naming what moved. The marker is written on every platform now (path-independent proof for Windows copies and for checkouts whose path carries no gstack segment), but only for a directory gstack creates: a directory we merely link into (unclaimed, or a legacy install) never becomes deletable whole. A directory with no SKILL.md at all is unclaimed: the link pass may add our file, the cleanup pass has nothing to remove. Also from the review passes: the banner check reads 8192 bytes, not 40 lines (investigate, office-hours, plan-ceo-review and design-consultation carry the banner past line 40 and were left "foreign" on pre-marker Windows installs); a link into a checkout named without a gstack segment (git worktree add ../gstack-<branch>) is ours when that tree carries setup + VERSION + bin/; relink's fast path is gone so both files canonicalize before judging; relink's root alias (_gstack-command) is gated and stamped like every other entry; relink reports the bare entry name with setup's wording and setup dedupes when forwarding (_run_relink_quiet); the summary names the browser skills as examples. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(setup): Chromium-install lock reclaim is atomic and pid-validated; abandoned locks expire; the tree kill walks /proc without pgrep - A pid file holding "", "-1" or "0" counted as a live holder (kill -0 -1 signals every process and succeeds), locking Chromium out for good. A pid must be a positive integer; anything else is stale. - Two setups judging the same lock stale raced on rm -rf + mkdir and the loser deleted the winner's fresh lock. The stale dir is renamed first (atomic), so exactly one reclaims. - A lock dir with no pid file (killed between mkdir and echo) was never reclaimed; it now expires once older than the install bound. - _kill_tree needed pgrep; debian-slim and git-bash ship none, so the bound killed only the wrapper subshell and the installer kept running. Without pgrep the children are found by walking /proc/*/stat. - The timeout knob is normalized in one place with one comment; the trap's exit 130 is the only exit the block may contain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(freeze): an unexpected non-zero death denies via an EXIT backstop instead of exiting with no decision set -e plus a failing pipeline (a tool on PATH exiting non-zero, a deleted cwd) ended the deny-tier hook with no JSON, which Claude Code treats as non-blocking: the edit outside the boundary proceeded. The EXIT trap now prints a deny for any non-zero exit that happens before a decision was written; every deliberate output sets _FREEZE_DECIDED first so a late failure never prints a second object. Tests also pin careful's state-root precedence (GSTACK_HOME over CLAUDE_PLUGIN_DATA, plugin data when CLAUDE_PLUGIN_ROOT names gstack) and the specific "out of date" deny for a helper without gstack_hook_state_root. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(telemetry): guard the stale-marker sweep with an if, not a break inside the loop Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(todos): the ownership gate lives in six sites, and the cleanup arms inline their own chain Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test: the two remaining linker harnesses extract the ownership helpers; the marker is the one allowed dotfile setup-claude-skill-assets and user-render-out-dir-install slice link_claude_skill_dirs out of setup without the helpers it now calls, so the extracted function died with "command not found" (or, inside an if, degraded into "foreign, skipped"). Both harnesses now carry the full helper set and the globals. The hidden-files census allows .gstack-owned, which the linker writes for directories it creates rather than copying from the skill source. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(setup,relink): weak proof never costs the user a file — assets, flips, failed backups, foreign dir links, alias markers Third review cycle on the ownership model, every item reproduced against a fixture before the fix: - Runtime assets (sections/, templates/, checklist.md, ...) were refreshed with rm -rf regardless of who owned the directory, so an unclaimed or weakly-owned directory lost the user's same-named real files. Real assets are now replaced only in a directory gstack created or strongly owns (marker, or SKILL.md symlink into gstack), plus the legacy Windows real-copy shape; elsewhere they are kept and reported. Symlinks are never content and are always refreshed. - The prefix-flip cleanup deleted a customized banner-bearing SKILL.md that the link pass would have backed up. Both cleanups now compare the file against the source (raw, or with its name: line rewritten to the entry name, which is how alias and prefixed copies legitimately differ) and move a differing file to the backup root. - A failed backup (unwritable root) returned success and the caller linked over the file anyway. It now fails, and the entry is left untouched and reported. - A foreign DIRECTORY symlink whose target had no SKILL.md fell through to the "unclaimed directory" rule and was replaced by a real directory. A symlink that does not resolve into gstack is foreign, full stop. - The alias installers stamped .gstack-owned into pre-existing directories; they now follow the same created-or-already-marked rule. - A directory counts as "only links" only when every link resolves into gstack: a user's own symlink makes it mixed, so their link survives. - The gstack-tree heuristic requires bin/gstack-relink, not just a VERSION file, a setup script and a bin/ directory. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(setup): lock reclaim hands a fresh lock back; a live holder past the bound is stale; /proc walk strips through the last paren - Reclaim renamed the lock by path after judging it stale, so a second setup that had already reclaimed and re-created it lost its fresh lock and two installers ran. After the rename the moved directory's pid is re-read: a new live holder, or a fresh lock whose pid is not written yet, is moved straight back. - A pid file whose process is alive but whose lock is older than the install bound is stale too (the holder is past its own deadline, or the pid was recycled to an unrelated long-lived process); it was locked forever. - The /proc fallback stripped the comm field to the FIRST ") ", so a comm containing ") " hid a child from the kill. proc(5) says the last paren. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(freeze): mark the decision written after the helper prints, not before If gstack_hook_decision ever failed between the flag and its output the backstop would have stayed silent; setting the flag after the print keeps the deny backstop armed until a decision is actually on stdout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: bump version and changelog (v1.80.0.0) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: update project documentation for v1.80.0.0 README troubleshooting + manual uninstall cover the skill ownership gate (.gstack-owned marker, ~/.gstack/backups/skills/<ts>/, foreign same-name skills left untouched). CLAUDE.md and CONTRIBUTING carry the ownership and best-effort Chromium bootstrap invariants for people editing setup and gstack-relink. PROJECT_STRUCTURE gains careful/, freeze/, guard/, unfreeze/, gstack-upgrade/, gstack-relink, and the setup/relink/hook test files. TESTING_INTERNALS documents the anchor-sliced setup harness convention. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(setup): the final summary reports customized SKILL.md files moved to the backup root The linker moved a weakly-proven, customized SKILL.md aside before linking over it but never said so; only relink printed a "Moved N" line, and by the time relink runs the file is already a symlink. The summary now names each moved file and where it went, next to the foreign-entry report. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test: run assembled setup harness scripts from a temp file, not `bash -c` argv (Windows MSYS2 8 KB truncation) windows-free-tests (run 33907177851) failed in test/setup-alias-name-uniqueness.test.ts with bash: -c: line 178: unexpected EOF while looking for matching `' The harness slices functions out of `setup` and passed the joined script as one `bash -c` argv element. The ownership gate grew that script from 6.7 KB to 15.7 KB, and on Windows bash is an MSYS2 program: when its parent is a non-MSYS process (bun), msys-2.0.dll's build_argv() runs any argument containing `?*["'(){}` through globify()/glob(), which copies the pattern into a fixed `Char patbuf[8192]` and silently stops after 8192 - MB_CUR_MAX (8186 chars under C.UTF-8); GLOB_NOCHECK then returns the truncated text as the argument. Character 8186 lands inside the single-quoted sed token on line 178. Rebuilding the exact script with CI path shapes and cutting it at 8186-8190 characters reproduces the identical message locally; cmd.exe's 8191-UTF-16 cap and CreateProcess's 32767 do not fit the evidence. Fix: test/helpers/bash-script.ts writes the script to a temp file and runs `bash <path>` — a short glob-free argument that never enters globify. Every setup harness that assembled a script for `bash -c` (11 files, 22 sites) uses it; timeouts and env are preserved verbatim, spawn/timeout errors are appended to stderr, temp cleanup is best-effort. `spawnSync('bash', [<Windows absolute path>])` already passes on windows-latest in setup-help, uninstall-windows-copies and the migration tests. The Windows-curated list is byte-identical before and after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(test-free-shards): the rerun-refresh harness spawns bash <tempfile> via test/helpers/bash-script.ts, not bash -c Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1617 lines
75 KiB
TypeScript
Executable File
1617 lines
75 KiB
TypeScript
Executable File
#!/usr/bin/env bun
|
||
/**
|
||
* test-free-shards — enumerate, shard, curate, and run the free test suite.
|
||
*
|
||
* Four jobs:
|
||
* 1. Enumeration. Walk `browse/test/`, `test/`, `make-pdf/test/` and return
|
||
* every `*.test.{ts,tsx,js,jsx,mjs,cjs}` that isn't a paid-eval test.
|
||
* 2. Sharding. Stable-hash assign each test to one of N shards. Used by CI
|
||
* to parallelize the free suite when needed.
|
||
* 3. Curation (Windows-safe filter). Scan each test's content for POSIX-only
|
||
* patterns (`/bin/bash`, `sh -c`, raw `/tmp/`, `chmod`, `xargs`). Files
|
||
* that match are excluded from the Windows-safe subset — they would fail
|
||
* on `windows-latest` no matter how the runner shards them.
|
||
* 4. Execution. Spawn `bun test` children and refuse to trust their exit
|
||
* code alone: every byte of output is classified through
|
||
* scripts/test-strict-output.ts, so a child that exits 0 without bun's
|
||
* terminal summary (a mid-suite process.exit truncation), with `(fail)`
|
||
* result lines, or with fewer files run than planned is a FAILURE. An
|
||
* external wall-clock timeout SIGKILLs the child's process group and
|
||
* reports the shard as timed-out — distinct from failed.
|
||
*
|
||
* Execution strategy (decision ledger V3/D6 — evaluate the Bun built-in
|
||
* first; probed 2026-08 on Bun 1.3.13):
|
||
* - Full-suite runs (`bun test` via package.json, `bun run test:free`) use
|
||
* N CONCURRENT SHARD PROCESSES, serial within each (the paid runner's
|
||
* model). A single `--parallel` invocation was probed and initially
|
||
* adopted, then abandoned: three distinct Bun 1.3.13 worker pathologies
|
||
* (segfault + crash-retry wedge, skipped-file hooks stalling a worker,
|
||
* spawn-heavy files hanging under load) each stalled the whole
|
||
* invocation, while process shards isolate any wedge to its own shard.
|
||
* Original --parallel probe results, kept for the record: it
|
||
* showed --parallel (a) prints the standard `Ran N tests across M files`
|
||
* terminal summary, (b) exits non-zero when any file fails, (c) runs each
|
||
* file in its own worker process (distinct pids, no shared globals), and
|
||
* (d) converts a mid-suite process.exit(0) — which silently truncates a
|
||
* serial run at exit 0 — into a per-file `(crashed: exited)` failure with
|
||
* a complete summary and exit 1. Strictly SAFER than the serial path and
|
||
* ~2x faster on a 6-file probe (0.22s -> 0.11s wall, 280% CPU); the win
|
||
* grows with suite size since the serial suite measured 454s.
|
||
* - CI-matrix runs (`--shards M --shard i`) keep the hash-partitioned
|
||
* one-child-per-shard path. Cross-runner partitioning must be
|
||
* deterministic and per-file stable, so bun's own `--shard=M/N`
|
||
* (round-robin over sorted paths — every assignment shifts when a file
|
||
* lands) is not used, and there are no static per-file weight lists.
|
||
* Shard indices are STABLE: assignFilesToShards never renumbers on
|
||
* occupancy, and an empty shard is a fast no-op success.
|
||
*
|
||
* Adapted from the McGluut/gstack fork's test-free-shards.ts (190 LOC). The
|
||
* Windows-safe filter is upstream-original — codex flagged that sharding alone
|
||
* doesn't fix POSIX-bound tests, so we curate the subset that actually runs
|
||
* on the windows-latest CI job.
|
||
*
|
||
* Output contract (v1.66): the full child stream ALWAYS lands in a per-run
|
||
* log file under os.tmpdir() (path printed once at start and again in the
|
||
* epilogue). The console is quiet by default — only the runner's own
|
||
* [test:free] lines, `(fail)` result lines, bun error/crash markers
|
||
* (`error:`, `panic:`, `crashed`, `Unhandled error`), and the terminal
|
||
* `Ran N tests across M files` summary reach it; `--verbose` restores full
|
||
* forwarding. After every run a stable epilogue names the failing tests
|
||
* (attributed to files via bun's `path/to/file.test.ts:` chunk headers),
|
||
* crashed+retried workers, and — on a wall-timeout kill — the wedge-suspect
|
||
* files. The strict classifier consumes the FULL stream regardless of what
|
||
* the console shows.
|
||
*
|
||
* Exit codes: 0 pass, 1 fail, 124 wall-clock timeout.
|
||
*
|
||
* Usage:
|
||
* bun run scripts/test-free-shards.ts # full suite, N concurrent shard processes
|
||
* bun run scripts/test-free-shards.ts --list # show all
|
||
* bun run scripts/test-free-shards.ts --windows-only --list # show curated
|
||
* bun run scripts/test-free-shards.ts --windows-only # run curated
|
||
* bun run scripts/test-free-shards.ts --shards 4 --shard 1 # one shard (CI matrix)
|
||
* bun run scripts/test-free-shards.ts --wall-timeout 600 # override the kill deadline
|
||
* bun run scripts/test-free-shards.ts --verbose # forward the full child stream
|
||
*/
|
||
|
||
import * as fs from 'fs';
|
||
import * as os from 'os';
|
||
import * as path from 'path';
|
||
import { spawn, spawnSync } from 'child_process';
|
||
import { StringDecoder } from 'node:string_decoder';
|
||
import { isPaidTestFile } from '../test/helpers/paid-test-set';
|
||
import {
|
||
BunTestOutputClassifier,
|
||
exactTestFileSelectors,
|
||
installChildSignalForwarding,
|
||
isTerminationRequested,
|
||
killProcessGroup,
|
||
strictTestExitCode,
|
||
stripAnsiLine,
|
||
} from './test-strict-output';
|
||
|
||
const ROOT = path.resolve(import.meta.dir, '..');
|
||
// design/test was silently absent from BOTH the package.json test script and
|
||
// this list — design tests (including a teardown bomb) never ran in any CI
|
||
// or local free run. Keep the two lists in sync. This list is the single
|
||
// source of truth for free-suite roots: package.json's `test` script routes
|
||
// through this runner rather than passing its own directory globs.
|
||
export const TEST_ROOTS = [
|
||
'browse/test',
|
||
'test',
|
||
'make-pdf/test',
|
||
'design/test',
|
||
// v1.65 orphan wire-in (decision D3a): these ran under NO script or CI —
|
||
// written coverage that caught nothing. All were green on arrival.
|
||
'ios-qa/daemon/test',
|
||
'ios-qa/scripts',
|
||
'browser-skills',
|
||
] as const;
|
||
const TEST_FILE_REGEX = /\.test\.(?:[cm]?[jt]s|tsx|jsx)$/;
|
||
|
||
// POSIX-only patterns that indicate a test will fail on windows-latest no
|
||
// matter how the runner shards. Codex's v1.18.0.0 review flagged the first
|
||
// three as concrete examples in the existing free suite (test/ship-version-sync.test.ts:72,
|
||
// test/helpers/providers/claude.ts:22, package.json:12). We scan the test's
|
||
// own content here so the filter stays automatic as new tests land. The
|
||
// "Windows-incompatible APIs" patterns at the bottom were added after the
|
||
// first windows-free-tests CI run surfaced concrete failure modes.
|
||
const WINDOWS_FRAGILE_PATTERNS: Array<{ pattern: RegExp; reason: string }> = [
|
||
// Hardcoded POSIX shells / commands.
|
||
{ pattern: /['"`]\/bin\/(?:ba)?sh/, reason: 'hardcoded /bin/sh or /bin/bash' },
|
||
{ pattern: /spawnSync\(['"]sh['"],|spawn\(['"]sh['"],|exec\(['"]sh /, reason: 'spawn("sh", ...)' },
|
||
{ pattern: /['"]bash -c['"]|['"]sh -c['"]/, reason: 'bash -c / sh -c' },
|
||
{ pattern: /['"`]\/tmp\//, reason: 'raw /tmp/ path (use os.tmpdir())' },
|
||
{ pattern: /['"]chmod\b/, reason: 'chmod shell command' },
|
||
{ pattern: /['"]xargs\b/, reason: 'xargs pipeline' },
|
||
{ pattern: /\bwhich claude\b/, reason: 'which claude (use Bun.which)' },
|
||
// Windows-incompatible APIs.
|
||
{ pattern: /\.mode\s*&\s*0o[0-7]+/, reason: 'POSIX file mode bitmask (mode & 0o600 etc — Windows fakes mode bits)' },
|
||
{ pattern: /\.endsWith\(['"]\//, reason: 'hardcoded forward-slash path assertion (Windows uses \\\\)' },
|
||
{ pattern: /['"]\.\/[a-zA-Z][^"']*['"]\)\s*\.\s*toBe\(true\)/, reason: 'forward-slash path comparison' },
|
||
// Tests that spawn a bash shebang script in bin/ via spawnSync. Git Bash on
|
||
// Windows can run `bash /path/to/script` but spawnSync(scriptPath, ...)
|
||
// tries to execute the file directly via CreateProcess, which fails on the
|
||
// shebang. The pattern matches `, 'bin'` as a path-join argument (closing
|
||
// OR followed by another segment), which catches:
|
||
// - path.join(ROOT, 'bin', 'script-name') — typical
|
||
// - join(import.meta.dir, '..', 'bin', 'name') — destructured (diff-scope)
|
||
// - path.join(ROOT, 'bin') — bare BIN constant (brain-sync)
|
||
{ pattern: /,\s*['"]bin['"]\s*[,)]|['"]\.?\/?bin\/[a-z][\w-]+['"]/, reason: 'spawns bin/ shebang script (Windows CreateProcess does not parse shebangs)' },
|
||
// Tests that launch a real Playwright browser. The windows-free-tests CI job
|
||
// runs a curated subset that intentionally does NOT install Chromium —
|
||
// browser bring-up on Windows is a separate concern (see PR #1238). Tests
|
||
// matching `await foo.launch(` need Chromium and fail with "Executable
|
||
// doesn't exist" on the runner.
|
||
{ pattern: /await\s+\w+\.launch\(/, reason: 'launches Playwright browser (Chromium not installed in windows-free CI)' },
|
||
// Tests that spawn the browse server as a subprocess via `bun run server.ts`.
|
||
// The Bun → server.ts → Playwright path is the same one that doesn't work
|
||
// on Windows (PR #1238 windows-pty-bun-pty-fix). Tests typically set
|
||
// BROWSE_HEADLESS_SKIP=1 to skip the browser launch but still need a working
|
||
// server, which they don't get on Windows.
|
||
{ pattern: /BROWSE_HEADLESS_SKIP|spawn\(\[['"]bun['"],\s*['"]run['"]/, reason: 'spawns the browse server subprocess (Bun-driven path is Windows-broken)' },
|
||
];
|
||
|
||
// Explicit known-Windows-incompatible test files that don't fit a regex
|
||
// pattern. Listed here with the precise reason. Prefer adding a pattern above
|
||
// when possible; this list is for environment-/runtime-specific tests where
|
||
// the failure mode is structural rather than detectable via source-file scan.
|
||
export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }> = [
|
||
{
|
||
file: 'test/host-config.test.ts',
|
||
reason: 'asserts "claude" binary on PATH (only true when running inside Claude Code, not on bare CI runner)',
|
||
},
|
||
{
|
||
file: 'browse/test/findport.test.ts',
|
||
reason: 'asserts Bun.serve.stop() is fire-and-forget — Bun behavior differs on Windows for this polyfill',
|
||
},
|
||
// First full run of the expanded lane (v1.66, 13 → ~258 files) surfaced
|
||
// seven POSIX-bound files the content patterns cannot see (their
|
||
// POSIX-ness is what they TEST, or arrives via a variable). Receipts:
|
||
// PR #2593 windows-free-tests run 31918591602.
|
||
{
|
||
file: 'test/codex-under-codex-detection.test.ts',
|
||
reason: 'drives the rendered preflight bash under a hardcoded POSIX PATH (/usr/bin:/bin) — bash is unreachable through that PATH on Windows, so every case sees empty output (v1.67 windows lane run 95234224148)',
|
||
},
|
||
{
|
||
file: 'test/regression-pr1169-build-app-sed.test.ts',
|
||
reason: 'tests sed escape sequences in build-app.sh — sed/bash are the subject under test',
|
||
},
|
||
{
|
||
file: 'test/setup-conductor-worktree.test.ts',
|
||
reason: 'tests ln -snf symlink semantics in the setup script — POSIX ln is the subject under test',
|
||
},
|
||
{
|
||
file: 'test/artifacts-init-migration.test.ts',
|
||
reason: 'runs a bash migration script + jq against a scaffolded git state — POSIX toolchain paths break under cmd spawn',
|
||
},
|
||
{
|
||
file: 'test/gstack-decision-semantic.test.ts',
|
||
reason: 'installs a fake gbrain SHEBANG SHIM on PATH; Windows spawn cannot exec shebang scripts',
|
||
},
|
||
{
|
||
file: 'test/question-log-hook.test.ts',
|
||
reason: 'spawns the PostToolUse hook script (bash shebang) directly; Windows spawn cannot exec it',
|
||
},
|
||
{
|
||
file: 'browse/test/browser-skills-e2e.test.ts',
|
||
reason: 'asserts forward-slash tier paths (<repo>/browser-skills/) that resolve with backslashes on Windows',
|
||
},
|
||
{
|
||
file: 'design/test/variants-retry-after.test.ts',
|
||
reason: 'wall-clock retry-timing assertions — flaky on the slow windows-latest runner even with widened bounds',
|
||
},
|
||
// Round-2 census (PR #2593 run 31919227507) after the first seven:
|
||
{
|
||
file: 'test/skill-census.test.ts',
|
||
reason: 'census walk throws at module load on Windows (skill-census.ts:63) — the skills-tree symlink layout needs Developer Mode that CI runners lack',
|
||
},
|
||
{
|
||
file: 'browse/test/browser-manager-unit.test.ts',
|
||
reason: 'wedges the shard to its wall deadline on windows-latest (in-flight at kill); needs a Windows repro to diagnose — macOS + Linux lanes cover the file',
|
||
},
|
||
// Round-3 census (PR #2593 run 31919871680): the round-2 wedge had been
|
||
// TRUNCATING its shard, so these seven only surfaced once shard 2 completed.
|
||
// All the same POSIX-environment classes: PID/cmdline identity probing,
|
||
// bash scripts as the subject under test, env-scrubbed child spawns.
|
||
{
|
||
file: 'browse/test/server-embedder-terminal-port.test.ts',
|
||
reason: 'identity-based terminal-agent kill probes PID/cmdline with POSIX semantics; teardown asserts fail on windows-latest',
|
||
},
|
||
{
|
||
file: 'design/test/daemon-discovery.test.ts',
|
||
reason: 'verifyIdentity matches a spawned daemon via /proc-style cmdline probing — POSIX identity semantics',
|
||
},
|
||
{
|
||
file: 'test/context-save-hardening.test.ts',
|
||
reason: 'bash context-save/migration scripts (HOME-unset semantics, random-suffix path) are the subject under test',
|
||
},
|
||
{
|
||
file: 'test/eval-list-cli.test.ts',
|
||
reason: 'spawns the eval:list CLI via bun with a constructed env — bun resolution fails under Windows spawn',
|
||
},
|
||
{
|
||
file: 'test/memory-cache-injection.test.ts',
|
||
reason: 'exercises hook/deny-enforcement shell scripts — POSIX toolchain is the subject under test',
|
||
},
|
||
{
|
||
file: 'test/migrations-v1.65.0.0.test.ts',
|
||
reason: 'bash migration script (bunx re-fetch, .done markers) is the subject under test',
|
||
},
|
||
{
|
||
file: 'test/question-preference-hook.test.ts',
|
||
reason: 'spawns the PreToolUse preference hook (shebang script) directly; Windows spawn cannot exec it',
|
||
},
|
||
// Round-4 census (PR #2593 run 31920052810): unhandled errors with no
|
||
// (fail) lines — attributed statically (the lane had no log artifact yet).
|
||
{
|
||
file: 'browse/test/browser-skill-commands.test.ts',
|
||
reason: 'spawnSkill spawns bun with a constructed env — bun resolution fails under Windows spawn (unhandled, no (fail) line)',
|
||
},
|
||
{
|
||
file: 'browse/test/security-audit-r2.test.ts',
|
||
reason: 'symlink-attack fixtures (evil-link) need Developer Mode CI runners lack; expect(toThrow) fires unhandled on Windows',
|
||
},
|
||
];
|
||
|
||
// Force-include overrides: files a WINDOWS_FRAGILE_PATTERNS regex excludes for
|
||
// a reason that does not actually apply to them. Each entry documents WHY the
|
||
// pattern hit is a false positive — the point of these files is Windows
|
||
// coverage, so auto-excluding them defeats the regression tests they carry.
|
||
const KNOWN_WINDOWS_SAFE: Array<{ file: string; reason: string }> = [
|
||
{
|
||
file: 'test/setup-windows-rerun-refresh.test.ts',
|
||
// Trips the "spawns bin/ shebang script" pattern via path.join(..., 'bin',
|
||
// 'tool.sh') fixture paths, but every spawn goes through test/helpers/bash-script.ts
|
||
// (bash <tempfile>) — Git Bash executes it fine on windows-latest, with no argv-length ceiling. This file IS
|
||
// the #2444 Windows regression coverage (IS_WINDOWS=1 copy-refresh path);
|
||
// excluding it here would keep the bug class unexercised on the one
|
||
// platform it bites.
|
||
reason: 'bin/ hits are fixture path segments; spawns bash explicitly — the IS_WINDOWS=1 refresh path must run on windows-latest',
|
||
},
|
||
{
|
||
file: 'test/uninstall-windows-copies.test.ts',
|
||
// Trips the "spawns bin/ shebang script" pattern via the
|
||
// path.join(ROOT, 'bin', 'gstack-uninstall') constant, but the script is
|
||
// always spawned through spawnSync('bash', [UNINSTALL, ...]). This file
|
||
// carries the #2563 Windows real-dir-copy uninstall coverage — the bug
|
||
// ONLY reproduces on the copy install shape windows-latest exercises.
|
||
// The symlink-shape describe block self-skips on win32.
|
||
reason: 'bin/ hit is a bash-spawned script path; #2563 real-dir uninstall coverage must run on windows-latest',
|
||
},
|
||
{
|
||
file: 'browse/test/file-permissions.test.ts',
|
||
// Trips the POSIX-mode-bitmask pattern, but every `mode & 0o777` assertion
|
||
// is platform-guarded: win32-only tests return early, POSIX-only tests
|
||
// guard the bitmask behind `process.platform !== 'win32'`, and the
|
||
// symlink-skip regression test both wraps symlinkSync in try/catch
|
||
// (runners without Developer Mode can't create symlinks) and guards its
|
||
// bitmask — on win32 it asserts behavior (warns, skips, doesn't throw,
|
||
// target stays usable), never fake Windows mode bits (dirs stat 0o777
|
||
// there, so a 0o755 expectation fails on runner semantics, not our code).
|
||
// This file carries the win32-only icacls-by-SID regression tests, which
|
||
// can ONLY execute on windows-latest — excluding it here means the
|
||
// machine-account ACL lockout regression is never exercised on the one
|
||
// platform it bricks.
|
||
reason: 'every mode-bitmask assertion is guarded off win32 (behavior asserted instead); win32-only ACL regression tests must run on windows-latest',
|
||
},
|
||
{
|
||
file: 'browse/test/terminal-agent-owner-watchdog.test.ts',
|
||
// Trips the spawn(['bun','run',...]) pattern, whose reason is the
|
||
// Playwright-bound browse server. This test spawns terminal-agent.ts,
|
||
// which imports only fs/path/crypto + local helpers (no Playwright, no
|
||
// PTY at module scope) and boots under Bun on Windows — the owner-PID
|
||
// orphan leak it pins was reported on Windows (#2019).
|
||
reason: 'spawns terminal-agent (no Playwright), not the browse server; owner-orphan leak is a Windows defect',
|
||
},
|
||
];
|
||
|
||
export const DEFAULT_SHARD_COUNT = 20;
|
||
// Per-test timeout passed to `bun test --timeout`. 30s matches what
|
||
// package.json's `test` script used before it was repointed at this runner —
|
||
// the runner is now the single owner of that semantic.
|
||
export const FREE_TEST_TIMEOUT_MS = 30_000;
|
||
// External wall-clock deadline per spawned child (whole shard or the single
|
||
// full-suite --parallel invocation). A wedged child — a spinning main thread
|
||
// no in-process --timeout timer can interrupt — is SIGKILLed at the group
|
||
// level and reported 'timed-out', distinct from 'failed'.
|
||
// ~3.5x the observed full-suite wall (~100-160s). A wedged run should be
|
||
// killed-and-diagnosed (the epilogue prints the in-flight suspects) in
|
||
// minutes, not sat out — 15min of silence was pure diagnosis latency.
|
||
// Override per run with --wall-timeout <secs>.
|
||
export const DEFAULT_WALL_TIMEOUT_MS = 6 * 60_000;
|
||
/**
|
||
* Full-suite shards scale their wall deadline with shard size:
|
||
* max(DEFAULT_WALL_TIMEOUT_MS, files × PER_FILE_WALL_MS). The 6-min floor
|
||
* keeps wedge diagnosis fast on a typical ~70-file local shard, while a
|
||
* low-core machine (jobs=1 → the whole suite in one shard) or the Windows
|
||
* lane (~130 files/shard) gets proportional headroom instead of a false
|
||
* timed-out kill of a healthy run. Explicit --wall-timeout disables scaling.
|
||
*/
|
||
export const PER_FILE_WALL_MS = 5_000;
|
||
export function wallTimeoutForShard(fileCount: number, baseMs = DEFAULT_WALL_TIMEOUT_MS): number {
|
||
return Math.max(baseMs, fileCount * PER_FILE_WALL_MS);
|
||
}
|
||
|
||
/**
|
||
* Wall for a duration-packed shard. The count heuristic above assumes count
|
||
* approximates cost; LPT packing breaks that BY DESIGN (a shard may hold six
|
||
* slow Playwright files), so packed shards get max(base, predicted x 3) —
|
||
* generous against seed drift, still bounded.
|
||
*/
|
||
export function wallTimeoutForPackedShard(predictedMs: number, baseMs = DEFAULT_WALL_TIMEOUT_MS, fileCount = 0): number {
|
||
// Predictions transfer badly across machines: the committed duration seed
|
||
// is recorded on fast CI, and a syscall-supervised sandbox replays those
|
||
// files 2-4x slower (observed: a 253-file shard predicted ~242s wall-killed
|
||
// at its 725s predicted-x3 wall while genuinely still progressing). The
|
||
// packed wall may therefore be LOOSER than the count heuristic, never
|
||
// tighter — it keeps the per-file floor the runner has always guaranteed.
|
||
return Math.max(baseMs, Math.ceil(predictedMs * 3), fileCount * PER_FILE_WALL_MS);
|
||
}
|
||
/**
|
||
* Full-suite parallelism: leave RESERVED_CPUS cores for the parent runner +
|
||
* OS, cap at MAX_FULL_SUITE_JOBS — beyond ~6 concurrent bun processes the
|
||
* playwright-heavy shards contend on browser launches instead of finishing
|
||
* sooner (measured on an M-series dev box).
|
||
*
|
||
* GSTACK_FREE_JOBS overrides the computed count (the free runner's analogue
|
||
* of the paid runner's EVALS_JOBS). Exists for syscall-supervised sandboxes:
|
||
* on Vercel sandboxes, PID 1 (sandbox-init) installs a seccomp filter whose
|
||
* user-space supervisor saturates under ~6 concurrent bun+playwright shards
|
||
* and starts returning EACCES from plain file syscalls (measured: 200/200
|
||
* `git init` probes in fresh mktemp dirs fail with
|
||
* "Cannot access work tree: Permission denied" while the suite runs, 0/200
|
||
* when idle — access(dir, X_OK) = EACCES under strace). Fewer shards keep
|
||
* the supervisor inside its budget. Not clamped by MAX_FULL_SUITE_JOBS so a
|
||
* beefy box can also raise it deliberately.
|
||
*/
|
||
export const MAX_FULL_SUITE_JOBS = 6;
|
||
export const RESERVED_CPUS = 2;
|
||
|
||
export function fullSuiteJobs(): number {
|
||
const raw = process.env.GSTACK_FREE_JOBS;
|
||
if (raw !== undefined && raw !== '') {
|
||
// Strict digits-only: parseInt would silently truncate "2abc" -> 2 and
|
||
// "3.7" -> 3, defeating the loud-failure contract the error text claims.
|
||
if (!/^\d+$/.test(raw.trim()) || Number.parseInt(raw, 10) <= 0) {
|
||
throw new Error(`GSTACK_FREE_JOBS must be a positive integer, got: ${raw}`);
|
||
}
|
||
return Number.parseInt(raw, 10);
|
||
}
|
||
return Math.max(1, Math.min(MAX_FULL_SUITE_JOBS, os.cpus().length - RESERVED_CPUS));
|
||
}
|
||
|
||
/**
|
||
* Files that crash or wedge Bun's --parallel WORKERS but run fine in a plain
|
||
* serial process. Full-suite mode now uses shard PROCESSES (no workers), so
|
||
* this list is inert placement-wise — retained as the paper trail of why the
|
||
* one-invocation --parallel strategy was abandoned, and as the exclusion list
|
||
* should anyone re-attempt it on a newer Bun.
|
||
*/
|
||
export const WORKER_HOSTILE: Record<string, string> = {
|
||
'browse/test/security-live-playwright.test.ts':
|
||
'Bun 1.3.13 segfaults running this file in a --parallel worker ("panic: '
|
||
+ 'Segmentation fault ... a bug in Bun"), and the crashed-worker retry then '
|
||
+ 'wedges the whole invocation past the wall clock. Passes serially.',
|
||
};
|
||
|
||
/**
|
||
* TREE-SERIAL files: run in ONE serial shard AFTER the parallel shards.
|
||
* EMPTY since the 2026-08 dissolution — kept as a mechanism, not a museum:
|
||
* a test that must regenerate shared repo artifacts IN PLACE (and cannot
|
||
* render into an out-dir instead) earns an entry here with a reason, and
|
||
* the runner will serialize it again.
|
||
*
|
||
* How it emptied: gen-skill-docs gained a main() guard (imports stopped
|
||
* regenerating 71 files at load) and --out-dir grew to every host, so all
|
||
* eight mutators now render into mkdtemps — the live tree is never written
|
||
* by the suite (pinned by gen-skill-docs-import-purity + each migrated
|
||
* file's own porcelain/mtime assertions). With zero mutators, the four
|
||
* ratchet READERS (parity caps, size budgets, carve parity/ordering) get a
|
||
* quiet tree by construction in any shard, so they rejoined the parallel
|
||
* phase — the ~35-40s serial tail on every full-suite run is gone.
|
||
* Keys are pinned against the live file census by test-free-shards.test.ts —
|
||
* a renamed file fails the suite instead of silently dropping serialization.
|
||
*/
|
||
export const TREE_MUTATING: Record<string, string> = {};
|
||
|
||
export function normalizeRelativePath(filePath: string): string {
|
||
return filePath.replace(/\\/g, '/');
|
||
}
|
||
|
||
export function isFreeTestFile(relativePath: string): boolean {
|
||
const normalized = normalizeRelativePath(relativePath);
|
||
if (!TEST_FILE_REGEX.test(normalized)) return false;
|
||
return !isPaidTestFile(normalized);
|
||
}
|
||
|
||
/**
|
||
* Returns the first POSIX-only pattern hit in the file, or null if Windows-safe.
|
||
*/
|
||
export function detectWindowsFragility(absolutePath: string): { reason: string } | null {
|
||
let content: string;
|
||
try {
|
||
content = fs.readFileSync(absolutePath, 'utf-8');
|
||
} catch {
|
||
return null;
|
||
}
|
||
for (const { pattern, reason } of WINDOWS_FRAGILE_PATTERNS) {
|
||
if (pattern.test(content)) return { reason };
|
||
}
|
||
return null;
|
||
}
|
||
|
||
function walkTestFiles(dirPath: string): string[] {
|
||
const entries = fs.readdirSync(dirPath, { withFileTypes: true });
|
||
const files: string[] = [];
|
||
for (const entry of entries) {
|
||
const fullPath = path.join(dirPath, entry.name);
|
||
if (entry.isDirectory()) {
|
||
files.push(...walkTestFiles(fullPath));
|
||
continue;
|
||
}
|
||
if (TEST_FILE_REGEX.test(entry.name)) {
|
||
files.push(fullPath);
|
||
}
|
||
}
|
||
return files;
|
||
}
|
||
|
||
export function collectFreeTestFiles(rootDir = ROOT): string[] {
|
||
const discovered = new Set<string>();
|
||
for (const testRoot of TEST_ROOTS) {
|
||
const absoluteRoot = path.join(rootDir, testRoot);
|
||
if (!fs.existsSync(absoluteRoot)) continue;
|
||
for (const fullPath of walkTestFiles(absoluteRoot)) {
|
||
const relativePath = normalizeRelativePath(path.relative(rootDir, fullPath));
|
||
if (isFreeTestFile(relativePath)) {
|
||
discovered.add(relativePath);
|
||
}
|
||
}
|
||
}
|
||
return [...discovered].sort();
|
||
}
|
||
|
||
export interface CurationResult {
|
||
safe: string[];
|
||
excluded: Array<{ file: string; reason: string }>;
|
||
}
|
||
|
||
export function curateWindowsSafe(files: string[], rootDir = ROOT): CurationResult {
|
||
const safe: string[] = [];
|
||
const excluded: Array<{ file: string; reason: string }> = [];
|
||
const knownBad = new Map(KNOWN_WINDOWS_INCOMPATIBLE.map((e) => [e.file, e.reason]));
|
||
const knownSafe = new Set(KNOWN_WINDOWS_SAFE.map((e) => e.file));
|
||
for (const relativePath of files) {
|
||
const knownReason = knownBad.get(relativePath);
|
||
if (knownReason) {
|
||
excluded.push({ file: relativePath, reason: knownReason });
|
||
continue;
|
||
}
|
||
if (knownSafe.has(relativePath)) {
|
||
safe.push(relativePath);
|
||
continue;
|
||
}
|
||
const absolute = path.join(rootDir, relativePath);
|
||
const fragility = detectWindowsFragility(absolute);
|
||
if (fragility) {
|
||
excluded.push({ file: relativePath, reason: fragility.reason });
|
||
} else {
|
||
safe.push(relativePath);
|
||
}
|
||
}
|
||
return { safe, excluded };
|
||
}
|
||
|
||
export function stableHash(input: string): number {
|
||
let hash = 0x811c9dc5;
|
||
for (let index = 0; index < input.length; index += 1) {
|
||
hash ^= input.charCodeAt(index);
|
||
hash = Math.imul(hash, 0x01000193);
|
||
}
|
||
return hash >>> 0;
|
||
}
|
||
|
||
/**
|
||
* Hash-partition files across EXACTLY shardCount shards. Empty shards are
|
||
* preserved: a file's shard index is a pure function of its own path and the
|
||
* shard count, never of which other files happen to exist. A CI matrix keys
|
||
* runners off the index, so filtering empty shards (the old behavior) would
|
||
* renumber every later shard whenever occupancy shifted — runner 3 silently
|
||
* running shard 4's files. An empty shard is instead a fast no-op success at
|
||
* run time.
|
||
*/
|
||
export function assignFilesToShards(files: string[], shardCount: number): string[][] {
|
||
if (!Number.isInteger(shardCount) || shardCount <= 0) {
|
||
throw new Error(`Shard count must be a positive integer. Received: ${shardCount}`);
|
||
}
|
||
|
||
const shards = Array.from({ length: shardCount }, () => [] as string[]);
|
||
for (const file of files) {
|
||
const shardIndex = stableHash(file) % shardCount;
|
||
shards[shardIndex].push(file);
|
||
}
|
||
|
||
return shards.map(filesInShard => filesInShard.sort());
|
||
}
|
||
|
||
// ─── Duration-aware packing (full-suite path ONLY) ─────────────────────────
|
||
// Hash sharding balances file COUNTS (~1.15x spread) but not cost: the 15
|
||
// Playwright-launching files land 4/3/4/1/2/1 across 6 shards, giving a
|
||
// measured 28s–97s shard spread and ~40s of idle tail on every run. LPT
|
||
// packing over recorded per-file durations reclaims most of it. The `--shard`
|
||
// CI-matrix path is deliberately untouched — its contract is stable indices
|
||
// via assignFilesToShards/stableHash (empty shards no-op; see above).
|
||
//
|
||
// One store, no overlay: durations come from the committed seed
|
||
// (scripts/free-test-durations.json), refreshed occasionally via
|
||
// `--record-durations` (each file timed in its own child — exact, and immune
|
||
// to bun's stream buffering, where silent passers print no header to
|
||
// timestamp). GSTACK_FREE_TEST_DURATIONS overrides the path for experiments.
|
||
// The seed is a HINT, not a contract: missing file → hash-shard fallback;
|
||
// unknown file → 75th-percentile pessimism (placed early by LPT, bounding
|
||
// tail risk). Successor note: bun ≥1.3.14 ships native --timings/--shard LPT
|
||
// scheduling — when the repo unpins 1.3.13, this packer is the code to
|
||
// replace (keep it swappable).
|
||
|
||
export const FREE_TEST_DURATIONS_FILE = 'scripts/free-test-durations.json';
|
||
|
||
export function loadFreeTestDurations(rootDir = ROOT): Record<string, number> | null {
|
||
const file = process.env.GSTACK_FREE_TEST_DURATIONS
|
||
?? path.join(rootDir, FREE_TEST_DURATIONS_FILE);
|
||
let raw: string;
|
||
try {
|
||
raw = fs.readFileSync(file, 'utf-8');
|
||
} catch {
|
||
return null; // no seed — hash sharding, silently (fresh checkouts are normal)
|
||
}
|
||
try {
|
||
const parsed = JSON.parse(raw) as { durations?: Record<string, unknown> };
|
||
const entries = Object.entries(parsed.durations ?? {})
|
||
.filter((entry): entry is [string, number] =>
|
||
typeof entry[1] === 'number' && Number.isFinite(entry[1]) && entry[1] >= 0);
|
||
if (entries.length === 0) return null;
|
||
return Object.fromEntries(entries);
|
||
} catch (error) {
|
||
// A corrupt seed (bad merge) must cost a warning, never the suite.
|
||
console.error(`[test:free] WARNING: corrupt durations seed ${file} (${(error as Error).message}) — falling back to hash sharding`);
|
||
return null;
|
||
}
|
||
}
|
||
|
||
export interface PackedShards {
|
||
shards: string[][];
|
||
/** Predicted total per shard, aligned with `shards` — feeds walls + logs. */
|
||
predictedMs: number[];
|
||
}
|
||
|
||
/**
|
||
* Longest-processing-time-first bin packing: files sorted by predicted
|
||
* duration (desc, path-stable tiebreak) each go to the currently-lightest
|
||
* shard. Deterministic for a given (files, shardCount, durations).
|
||
*/
|
||
export function packShardsByDuration(
|
||
files: string[],
|
||
shardCount: number,
|
||
durations: Record<string, number>,
|
||
): PackedShards {
|
||
if (!Number.isInteger(shardCount) || shardCount <= 0) {
|
||
throw new Error(`Shard count must be a positive integer. Received: ${shardCount}`);
|
||
}
|
||
const known = files
|
||
.map((f) => durations[normalizeRelativePath(f)])
|
||
.filter((v): v is number => typeof v === 'number')
|
||
.sort((a, b) => a - b);
|
||
// Unknown files get the 75th percentile of known durations: pessimistic, so
|
||
// LPT places them early and a surprise long-runner can't recreate the tail.
|
||
const fallback = known.length > 0 ? known[Math.min(known.length - 1, Math.floor(known.length * 0.75))] : 1;
|
||
const predicted = (f: string): number => durations[normalizeRelativePath(f)] ?? fallback;
|
||
|
||
const ordered = [...files].sort((a, b) => predicted(b) - predicted(a) || (a < b ? -1 : 1));
|
||
const shards = Array.from({ length: shardCount }, () => [] as string[]);
|
||
const loads = new Array<number>(shardCount).fill(0);
|
||
for (const file of ordered) {
|
||
let lightest = 0;
|
||
for (let i = 1; i < shardCount; i += 1) {
|
||
if (loads[i] < loads[lightest]) lightest = i;
|
||
}
|
||
shards[lightest].push(file);
|
||
loads[lightest] += predicted(file);
|
||
}
|
||
return { shards: shards.map((s) => s.sort()), predictedMs: loads };
|
||
}
|
||
|
||
export interface BuildShardArgsOptions {
|
||
/**
|
||
* Pass bun's --parallel (worker-per-file, implies --isolate). No production
|
||
* caller today — full-suite mode uses N shard PROCESSES after the worker
|
||
* pathologies documented in main(); retained for a future re-attempt on a
|
||
* newer Bun (see WORKER_HOSTILE).
|
||
*/
|
||
parallel?: boolean;
|
||
rootDir?: string;
|
||
}
|
||
|
||
export function buildShardArgs(files: string[], options: BuildShardArgsOptions = {}): string[] {
|
||
// Exact absolute selectors: bun treats positional test paths as substring
|
||
// filters, so a relative `test/x.test.ts` would ALSO select
|
||
// `browse/test/x.test.ts` — shard bleed that double-runs files.
|
||
const selectors = exactTestFileSelectors(files, options.rootDir ?? ROOT);
|
||
const args = ['test', ...selectors, `--timeout=${FREE_TEST_TIMEOUT_MS}`];
|
||
if (options.parallel) args.push('--parallel');
|
||
else args.push('--max-concurrency=1');
|
||
return args;
|
||
}
|
||
|
||
type CliOptions = {
|
||
dryRun: boolean;
|
||
listOnly: boolean;
|
||
recordDurations: boolean;
|
||
windowsOnly: boolean;
|
||
verbose: boolean;
|
||
shardCount: number;
|
||
shardIndex: number | null;
|
||
wallTimeoutMs: number;
|
||
/** True when --wall-timeout was passed explicitly; full-suite mode only auto-scales the default. */
|
||
wallTimeoutExplicit: boolean;
|
||
};
|
||
|
||
function parseCliOptions(argv: string[]): CliOptions {
|
||
let dryRun = false;
|
||
let listOnly = false;
|
||
let recordDurations = false;
|
||
let windowsOnly = false;
|
||
let verbose = false;
|
||
let shardCount = DEFAULT_SHARD_COUNT;
|
||
let shardIndex: number | null = null;
|
||
let wallTimeoutMs = DEFAULT_WALL_TIMEOUT_MS;
|
||
let wallTimeoutExplicit = false;
|
||
|
||
for (let index = 0; index < argv.length; index += 1) {
|
||
const arg = argv[index];
|
||
if (arg === '--dry-run') { dryRun = true; continue; }
|
||
if (arg === '--list') { listOnly = true; continue; }
|
||
if (arg === '--record-durations') { recordDurations = true; continue; }
|
||
if (arg === '--windows-only') { windowsOnly = true; continue; }
|
||
if (arg === '--verbose') { verbose = true; continue; }
|
||
if (arg === '--shards') {
|
||
const value = argv[index + 1];
|
||
if (!value) throw new Error('Missing value for --shards');
|
||
shardCount = Number.parseInt(value, 10);
|
||
index += 1;
|
||
continue;
|
||
}
|
||
if (arg === '--shard') {
|
||
const value = argv[index + 1];
|
||
if (!value) throw new Error('Missing value for --shard');
|
||
shardIndex = Number.parseInt(value, 10);
|
||
index += 1;
|
||
continue;
|
||
}
|
||
if (arg === '--wall-timeout') {
|
||
const value = Number.parseInt(argv[index + 1] ?? '', 10);
|
||
if (!Number.isInteger(value) || value <= 0) throw new Error('--wall-timeout needs a positive integer (seconds)');
|
||
wallTimeoutMs = value * 1000;
|
||
wallTimeoutExplicit = true;
|
||
index += 1;
|
||
continue;
|
||
}
|
||
throw new Error(`Unknown argument: ${arg}`);
|
||
}
|
||
|
||
return { dryRun, listOnly, recordDurations, windowsOnly, verbose, shardCount, shardIndex, wallTimeoutMs, wallTimeoutExplicit };
|
||
}
|
||
|
||
function formatShardSummary(shards: string[][]): string[] {
|
||
return shards.map((files, index) => {
|
||
const preview = files.slice(0, 3).join(', ');
|
||
const suffix = files.length > 3 ? ', ...' : '';
|
||
return `Shard ${index + 1}/${shards.length}: ${files.length} files${preview ? ` -> ${preview}${suffix}` : ''}`;
|
||
});
|
||
}
|
||
|
||
/**
|
||
* True when a shard's output shows the run ended WITHOUT bun's final summary
|
||
* ("Ran N tests across ..."). A process.exit() fired mid-suite skips the
|
||
* summary AND hands back whatever code the caller passed — historically 0,
|
||
* which made a truncated shard indistinguishable from a green one. Exit code
|
||
* alone is therefore not evidence of completion; the summary line is.
|
||
*
|
||
* The runner itself now enforces this (and more) through
|
||
* scripts/test-strict-output.ts inside runFreeShard; this predicate remains
|
||
* the minimal documented primitive that test/exit-propagation.test.ts drives
|
||
* with genuine truncated and genuine complete bun runs.
|
||
*/
|
||
export function shardRunLooksTruncated(status: number | null, output: string): boolean {
|
||
if (status !== 0) return false; // already failing — not the silent case
|
||
return !/Ran \d+ tests? across \d+ files?/.test(output);
|
||
}
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// Output contract: console filtering + per-file failure attribution.
|
||
//
|
||
// Bun groups each file's output under a `path/to/file.test.ts:` header line
|
||
// (cwd-relative, sometimes ../-prefixed through a symlinked cwd). The
|
||
// reporter tracks the current header while consuming the stream, attributes
|
||
// `(fail)` lines and crash markers to files, and decides which lines reach
|
||
// the console in the default quiet mode. All matching happens on
|
||
// ANSI-stripped lines — colored `(fail)` lines defeated a prior grep.
|
||
// ---------------------------------------------------------------------------
|
||
|
||
const TEST_PATH_SOURCE = String.raw`\.test\.(?:[cm]?[jt]s|tsx|jsx)`;
|
||
/** A file chunk header: the path bun printed, terminated by a bare colon. */
|
||
const FILE_HEADER_RE = new RegExp(`^(\\S.*${TEST_PATH_SOURCE}):$`);
|
||
/** Same shape strict-output classifies as failed-test, with the name captured. */
|
||
const FAIL_RESULT_CAPTURE_RE = /^\(fail\) (.+) \[\d+(?:\.\d+)?(?:ns|us|µs|ms|s)\]$/;
|
||
/** bun --parallel retries a crashed worker once: `<icon> crashed running <path>, retrying`. */
|
||
const CRASH_RETRY_RE = new RegExp(`crashed running (\\S*${TEST_PATH_SOURCE}), retrying`);
|
||
/** The give-up marker after the retry also crashes: `✗ <path> (crashed: exited)`. */
|
||
const CRASH_FINAL_RE = new RegExp(`(\\S*${TEST_PATH_SOURCE}) \\(crashed: [^)]+\\)`);
|
||
const TERMINAL_SUMMARY_CAPTURE_RE = /^Ran (\d+) tests? across (\d+) files?\. \[/;
|
||
/** Substrings that must reach the console even in the default quiet mode. */
|
||
const CONSOLE_ALWAYS_MARKERS = ['error:', 'panic:', 'Unhandled error', 'crashed'] as const;
|
||
|
||
export type StreamOrigin = 'stdout' | 'stderr';
|
||
|
||
export interface FreeRunFailure {
|
||
/** Planned relative path when attributable, else the raw header path, else null. */
|
||
file: string | null;
|
||
testName: string;
|
||
}
|
||
|
||
export interface FreeRunReport {
|
||
testsRan: number | null;
|
||
filesRan: number | null;
|
||
sawTerminalSummary: boolean;
|
||
/** Deduped `(fail)` lines in arrival order, attributed to the current file header. */
|
||
failures: FreeRunFailure[];
|
||
/** Files that crashed a worker (bun retries once; a second crash is final). Deduped. */
|
||
crashedFiles: string[];
|
||
/**
|
||
* "# Unhandled error between tests" markers, attributed to the chunk they
|
||
* appeared in. These fail the shard via the strict classifier but produce
|
||
* NO (fail) lines — without surfacing them here, the epilogue reads
|
||
* "FAIL — 0 failing test(s)" and the culprit is undiscoverable from CI
|
||
* output (first Windows lane run: a module-load throw in skill-census).
|
||
*/
|
||
unhandledErrors: Array<{ file: string | null }>;
|
||
/**
|
||
* Wedge-suspect heuristic for a wall-timeout kill: files whose header was
|
||
* seen but whose chunk never ENDED (chunk end = the next file's header, or
|
||
* a final crash marker) before the terminal summary — i.e. "started but
|
||
* never produced a result chunk end". Result lines deliberately do NOT end
|
||
* a chunk: a file that printed a fail and then wedged stays listed. Known
|
||
* limits of the approximation:
|
||
* - Serial (--shard CI path): bun streams live but prints a file's header
|
||
* lazily, on its first output line — a wedged file that printed ANY
|
||
* line is listed; a fully silent wedge is not.
|
||
* - Parallel (full-suite path): bun buffers a file's whole chunk until it
|
||
* COMPLETES, so a wedged file usually never prints a header (see
|
||
* filesWithNoOutput), and the LAST flushed chunk before the kill has no
|
||
* closing header, so one completed noisy file can be over-listed.
|
||
*/
|
||
inFlight: string[];
|
||
/** Planned files never observed in the stream (silent passers + never-flushed wedges). */
|
||
filesWithNoOutput: number;
|
||
}
|
||
|
||
interface FileProgress {
|
||
headerSeen: boolean;
|
||
/** The file's chunk ended: a later file's header arrived, or it crashed out. */
|
||
ended: boolean;
|
||
}
|
||
|
||
/**
|
||
* Incrementally consumes the child's stdout/stderr (chunk boundaries need not
|
||
* align to lines), attributing results to files and forwarding only
|
||
* always-visible lines to `forward` (omit `forward` for verbose/quiet modes —
|
||
* attribution still runs so the epilogue works in every mode).
|
||
*/
|
||
export class FreeRunReporter {
|
||
private readonly decoders: Record<StreamOrigin, StringDecoder> = {
|
||
stdout: new StringDecoder('utf8'),
|
||
stderr: new StringDecoder('utf8'),
|
||
};
|
||
private readonly pending: Record<StreamOrigin, string> = { stdout: '', stderr: '' };
|
||
private readonly plannedSet: Set<string>;
|
||
private readonly canonicalCache = new Map<string, string>();
|
||
private readonly progress = new Map<string, FileProgress>();
|
||
private readonly failureKeys = new Set<string>();
|
||
private readonly failures: FreeRunFailure[] = [];
|
||
private readonly crashed = new Set<string>();
|
||
private currentFile: string | null = null;
|
||
private inRecap = false;
|
||
private readonly unhandled: Array<{ file: string | null }> = [];
|
||
private testsRan: number | null = null;
|
||
private filesRan: number | null = null;
|
||
private sawSummary = false;
|
||
|
||
constructor(
|
||
private readonly plannedFiles: string[],
|
||
private readonly forward?: (text: string, origin: StreamOrigin) => void,
|
||
) {
|
||
this.plannedSet = new Set(plannedFiles.map(normalizeRelativePath));
|
||
}
|
||
|
||
write(chunk: Uint8Array | string, origin: StreamOrigin): void {
|
||
this.pending[origin] += typeof chunk === 'string'
|
||
? chunk
|
||
: this.decoders[origin].write(Buffer.from(chunk));
|
||
let newline = this.pending[origin].indexOf('\n');
|
||
while (newline !== -1) {
|
||
this.handleLine(this.pending[origin].slice(0, newline), origin);
|
||
this.pending[origin] = this.pending[origin].slice(newline + 1);
|
||
newline = this.pending[origin].indexOf('\n');
|
||
}
|
||
}
|
||
|
||
/** Flush partial trailing lines (a stream killed mid-line still classifies). */
|
||
end(): void {
|
||
for (const origin of ['stdout', 'stderr'] as const) {
|
||
this.pending[origin] += this.decoders[origin].end();
|
||
if (this.pending[origin].length > 0) this.handleLine(this.pending[origin], origin);
|
||
this.pending[origin] = '';
|
||
}
|
||
}
|
||
|
||
report(): FreeRunReport {
|
||
const inFlight = this.sawSummary
|
||
? []
|
||
: [...this.progress.entries()]
|
||
.filter(([, p]) => p.headerSeen && !p.ended)
|
||
.map(([file]) => file)
|
||
.sort();
|
||
return {
|
||
testsRan: this.testsRan,
|
||
filesRan: this.filesRan,
|
||
sawTerminalSummary: this.sawSummary,
|
||
failures: [...this.failures],
|
||
crashedFiles: [...this.crashed].sort(),
|
||
unhandledErrors: [...this.unhandled],
|
||
inFlight,
|
||
filesWithNoOutput: this.plannedFiles.filter((f) => !this.progress.has(normalizeRelativePath(f))).length,
|
||
};
|
||
}
|
||
|
||
private handleLine(rawLine: string, origin: StreamOrigin): void {
|
||
// GitHub Actions: bun wraps each file's section in ::group::<header>.
|
||
// Without stripping, the real header fails FILE_HEADER_RE, failures get
|
||
// attributed to the PREVIOUS file, and the terminal recap's re-printed
|
||
// (fail) lines land under a second phantom file (observed on the first
|
||
// Linux run: 5 real failures reported as 10 across 2 files).
|
||
const line = stripAnsiLine(rawLine).replace(/^::group::/, '');
|
||
let visible = false;
|
||
|
||
// Bun's terminal recap ("N tests failed:") re-prints every (fail) line
|
||
// WITHOUT re-printing file headers. Attributing those to the stale
|
||
// currentFile invented a phantom failing file on the first Linux run
|
||
// (5 real failures reported as 10 across 2 files, one innocent).
|
||
if (/^\d+ tests? failed:$/.test(line)) {
|
||
this.inRecap = true;
|
||
if (this.currentFile) this.progressFor(this.currentFile).ended = true;
|
||
this.currentFile = null;
|
||
}
|
||
|
||
if (line === '# Unhandled error between tests') {
|
||
this.unhandled.push({ file: this.currentFile });
|
||
}
|
||
|
||
const header = FILE_HEADER_RE.exec(line);
|
||
if (header) {
|
||
const file = this.canonicalize(header[1]);
|
||
// A new header ends the previous file's chunk — that file is no longer
|
||
// a wedge suspect. (Bun 1.3.x prints NO (pass) lines, so chunk
|
||
// delimiters, not result lines, are the completion signal.)
|
||
if (this.currentFile && this.currentFile !== file) this.progressFor(this.currentFile).ended = true;
|
||
this.currentFile = file;
|
||
this.progressFor(file).headerSeen = true;
|
||
} else {
|
||
const fail = FAIL_RESULT_CAPTURE_RE.exec(line);
|
||
const retry = fail ? null : CRASH_RETRY_RE.exec(line);
|
||
const final = fail || retry ? null : CRASH_FINAL_RE.exec(line);
|
||
if (fail) {
|
||
visible = true;
|
||
// In the recap, a (fail) line only records a failure the main run
|
||
// somehow never attributed (belt and braces); known names dedupe.
|
||
const recapDuplicate = this.inRecap
|
||
&& this.failures.some((f) => f.testName === fail[1]);
|
||
const key = `${this.currentFile ?? ''}\u0000${fail[1]}`;
|
||
if (!recapDuplicate && !this.failureKeys.has(key)) {
|
||
this.failureKeys.add(key);
|
||
this.failures.push({ file: this.currentFile, testName: fail[1] });
|
||
}
|
||
} else if (retry) {
|
||
// The file will run again — a crash+retry does not end its chunk.
|
||
visible = true;
|
||
this.crashed.add(this.canonicalize(retry[1]));
|
||
} else if (final) {
|
||
visible = true;
|
||
const file = this.canonicalize(final[1]);
|
||
this.crashed.add(file);
|
||
this.progressFor(file).ended = true;
|
||
} else {
|
||
const summary = TERMINAL_SUMMARY_CAPTURE_RE.exec(line);
|
||
if (summary) {
|
||
visible = true;
|
||
this.sawSummary = true;
|
||
this.testsRan = Number.parseInt(summary[1], 10);
|
||
this.filesRan = Number.parseInt(summary[2], 10);
|
||
}
|
||
}
|
||
}
|
||
|
||
if (!visible) visible = CONSOLE_ALWAYS_MARKERS.some((marker) => line.includes(marker));
|
||
if (visible && this.forward) this.forward(`${rawLine.replace(/\r$/, '')}\n`, origin);
|
||
}
|
||
|
||
private progressFor(file: string): FileProgress {
|
||
let entry = this.progress.get(file);
|
||
if (!entry) {
|
||
entry = { headerSeen: false, ended: false };
|
||
this.progress.set(file, entry);
|
||
}
|
||
return entry;
|
||
}
|
||
|
||
/**
|
||
* Map a printed path back to its planned relative path. Bun prints paths
|
||
* relative to the child's (real)cwd, so a symlinked cwd (macOS /tmp) yields
|
||
* `../..`-prefixed forms — strip the prefix and suffix-match.
|
||
*/
|
||
private canonicalize(printedPath: string): string {
|
||
const cached = this.canonicalCache.get(printedPath);
|
||
if (cached) return cached;
|
||
const stripped = normalizeRelativePath(printedPath).replace(/^(?:\.{1,2}\/)+/, '');
|
||
let resolved = stripped;
|
||
if (!this.plannedSet.has(stripped)) {
|
||
const match = this.plannedFiles.find(
|
||
(planned) => stripped.endsWith(`/${planned}`) || planned.endsWith(`/${stripped}`),
|
||
);
|
||
if (match) resolved = match;
|
||
}
|
||
this.canonicalCache.set(printedPath, resolved);
|
||
return resolved;
|
||
}
|
||
}
|
||
|
||
/**
|
||
* The stable post-run epilogue. Success is one line; failure names every
|
||
* failing test (deduped, attributed) and crashed worker; a wall-timeout kill
|
||
* additionally prints the wedge-suspect list (see FreeRunReport.inFlight for
|
||
* the heuristic and its limits).
|
||
*/
|
||
export function buildRunEpilogue(
|
||
status: FreeShardStatus,
|
||
report: FreeRunReport,
|
||
elapsedMs: number,
|
||
logPath: string,
|
||
): string[] {
|
||
const seconds = Math.round(elapsedMs / 1000);
|
||
if (status === 'passed') {
|
||
return [
|
||
`[test:free] PASS — ${report.testsRan ?? '?'} tests, ${report.filesRan ?? '?'} files, ${seconds}s. Full log: ${logPath}`,
|
||
];
|
||
}
|
||
const failingFiles = new Set(report.failures.map((f) => f.file ?? '(unattributed)'));
|
||
const lines = [
|
||
`[test:free] FAIL — ${report.failures.length} failing test(s) in ${failingFiles.size} file(s), `
|
||
+ `${report.crashedFiles.length} crashed worker(s)${report.unhandledErrors.length > 0 ? `, ${report.unhandledErrors.length} unhandled error(s) between tests` : ''}. Full log: ${logPath}`,
|
||
];
|
||
for (const failure of report.failures) {
|
||
lines.push(` ✗ ${failure.file ?? '(unattributed)'} — ${failure.testName}`);
|
||
}
|
||
for (const file of report.crashedFiles) {
|
||
lines.push(` ⚠ crashed+retried: ${file}`);
|
||
}
|
||
for (const u of report.unhandledErrors) {
|
||
lines.push(` ⚠ unhandled error between tests (around ${u.file ?? 'unknown file'})`);
|
||
}
|
||
if (status === 'timed-out') {
|
||
if (report.inFlight.length > 0) {
|
||
lines.push(` ⏱ in flight at kill: ${report.inFlight.join(', ')}`);
|
||
} else {
|
||
lines.push(
|
||
' ⏱ in flight at kill: unknown — no open file chunk was observed '
|
||
+ '(bun --parallel buffers a file\'s output until it completes, so a silent wedge never prints); '
|
||
+ `${report.filesWithNoOutput} planned file(s) produced no output before the kill.`,
|
||
);
|
||
}
|
||
}
|
||
return lines;
|
||
}
|
||
|
||
export type FreeShardStatus = 'passed' | 'failed' | 'timed-out';
|
||
|
||
// ─── Flake ledger (WS1 telemetry) ───────────────────────────────────────────
|
||
// Single-writer JSONL: ONLY this parent runner appends (never shards, never
|
||
// tests — no concurrent-append hazard by construction). CI points
|
||
// GSTACK_FLAKE_LEDGER at $RUNNER_TEMP and uploads it as an artifact every
|
||
// run, so repeat offenders become an enumerable series instead of console
|
||
// scrollback. Fail-open with a loud stderr warning: a broken ledger must
|
||
// never red the only required lane.
|
||
|
||
export interface FlakeLedgerEntry {
|
||
ts: string;
|
||
runner: 'free';
|
||
kind: 'flaky-pass';
|
||
file: string;
|
||
/** Shard the original failure surfaced in, when attributable. */
|
||
shard?: number;
|
||
/** Code-state attribution (review finding): without branch/sha the series
|
||
* can't tie an entry to the state that produced it, and the WS16
|
||
* promotion evidence needs exactly that. */
|
||
branch?: string;
|
||
git_sha?: string;
|
||
}
|
||
|
||
export function flakeLedgerPath(env: NodeJS.ProcessEnv = process.env): string {
|
||
if (env.GSTACK_FLAKE_LEDGER) return env.GSTACK_FLAKE_LEDGER;
|
||
// Local default: per-PROJECT, not the machine-global tmpdir — sibling
|
||
// Conductor worktrees of DIFFERENT repos must not interleave into one
|
||
// series (review finding). CI always sets GSTACK_FLAKE_LEDGER explicitly.
|
||
try {
|
||
const slug = spawnSync('bash', ['-c', '~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null'], { stdio: 'pipe', timeout: 3000 })
|
||
.stdout?.toString().match(/^SLUG=(.+)$/m)?.[1];
|
||
if (slug) {
|
||
const dir = path.join(os.homedir(), '.gstack', 'projects', slug);
|
||
fs.mkdirSync(dir, { recursive: true });
|
||
return path.join(dir, 'flake-ledger.jsonl');
|
||
}
|
||
} catch { /* fall through */ }
|
||
return path.join(os.tmpdir(), 'gstack-flake-ledger.jsonl');
|
||
}
|
||
|
||
export function appendFlakeLedger(
|
||
entries: FlakeLedgerEntry[],
|
||
ledgerPath: string,
|
||
warn: (line: string) => void = (line) => console.error(line),
|
||
): boolean {
|
||
if (entries.length === 0) return true;
|
||
try {
|
||
fs.mkdirSync(path.dirname(ledgerPath), { recursive: true });
|
||
fs.appendFileSync(ledgerPath, entries.map((e) => JSON.stringify(e)).join('\n') + '\n');
|
||
return true;
|
||
} catch (error) {
|
||
warn(`[test:free] WARNING: could not append flake ledger at ${ledgerPath} `
|
||
+ `(${error instanceof Error ? error.message : String(error)}) — flaky-pass telemetry lost for this run, verdict unaffected`);
|
||
return false;
|
||
}
|
||
}
|
||
|
||
export interface FreeShardOutcome {
|
||
shard: number;
|
||
files: string[];
|
||
status: FreeShardStatus;
|
||
exitCode: number | null;
|
||
elapsedMs: number;
|
||
groupPid: number | null;
|
||
/**
|
||
* Repo-relative files with attributed test failures or crashes, deduped.
|
||
* Feeds the opt-in flaky retry pass (GSTACK_FREE_RETRY_FLAKY) — empty on
|
||
* pass, and empty when every failure was unattributed (retry would be
|
||
* meaningless without knowing what to re-run).
|
||
*/
|
||
failingFiles: string[];
|
||
/**
|
||
* Count of failure evidence the retry pass CANNOT re-run by file: fail
|
||
* lines seen before any file-chunk header, unhandled errors between tests,
|
||
* and a truncated run (no terminal summary). Nonzero vetoes the flaky
|
||
* retry for the whole run — retrying only failingFiles would re-run a
|
||
* subset and mask the rest as a FLAKY-PASS, re-opening the silent-truncation
|
||
* hole the strict classifier exists to close.
|
||
*/
|
||
unattributedFailures: number;
|
||
}
|
||
|
||
export interface ShardCommand {
|
||
command: string;
|
||
args: string[];
|
||
}
|
||
|
||
export interface RunFreeShardOptions {
|
||
/** External wall-clock deadline; on expiry the child's process GROUP is SIGKILLed. */
|
||
wallTimeoutMs?: number;
|
||
rootDir?: string;
|
||
env?: NodeJS.ProcessEnv;
|
||
/** Pass bun's --parallel. No production caller today (see BuildShardArgsOptions.parallel). */
|
||
parallel?: boolean;
|
||
/** Override the spawned command. Tests inject fake pass/fail/slow commands. */
|
||
commandFor?: (files: string[]) => ShardCommand;
|
||
/** Suppress ALL child output from the console (tests). The classifier and the log file still see every byte. */
|
||
quiet?: boolean;
|
||
/** Forward the full child stream to the console (legacy firehose). Default: the quiet filtered console. */
|
||
verbose?: boolean;
|
||
/**
|
||
* Console sink for child-stream output (tests inject to assert quiet vs
|
||
* verbose behavior). Default: process.stdout / process.stderr by origin.
|
||
* Runner-owned [test:free] lines go through `log`, not this sink.
|
||
*/
|
||
consoleWrite?: (text: string) => void;
|
||
/** Per-run full-stream log path (tests inject). Default: a timestamped file under os.tmpdir(). */
|
||
logFilePath?: string;
|
||
log?: (line: string) => void;
|
||
}
|
||
|
||
const EPILOGUE_WORD: Record<FreeShardStatus, string> = {
|
||
passed: 'pass',
|
||
failed: 'fail',
|
||
'timed-out': 'timed-out',
|
||
};
|
||
|
||
/** One line per shard, printed after the run: `[test:free] shard i/N: M files, XXs, pass|fail|timed-out`. */
|
||
function shardEpilogue(outcome: FreeShardOutcome, totalShards: number): string {
|
||
return `[test:free] shard ${outcome.shard}/${totalShards}: ${outcome.files.length} files, `
|
||
+ `${Math.round(outcome.elapsedMs / 1000)}s, ${EPILOGUE_WORD[outcome.status]}`;
|
||
}
|
||
|
||
/**
|
||
* Run one shard (or the whole suite, in --parallel full-suite mode) in its own
|
||
* bun process and classify the result strictly.
|
||
*
|
||
* Verdict integrity: the child's exit code is never trusted alone. Output is
|
||
* fed through BunTestOutputClassifier, and strictTestExitCode requires bun's
|
||
* terminal summary to report EXACTLY the planned file count — a shard that
|
||
* exits 0 without the summary (mid-suite process.exit truncation), with
|
||
* `(fail)` result lines, or having run fewer files than planned is a FAILURE.
|
||
* This is enforced for injected fake commands too (unlike the paid runner),
|
||
* so tests can pin the summary-missing => failure backstop; fake passing
|
||
* commands must print a synthetic `Ran N tests across M files. [Xms]` line.
|
||
*
|
||
* Per-shard temp isolation: each spawned child gets its own throwaway TMPDIR
|
||
* (TEMP/TMP on Windows) so shards can't trip over each other's temp files.
|
||
* Deliberately NOT GSTACK_HOME: injecting one shared scratch home for a whole
|
||
* invocation made 6,900 tests share a MUTABLE state dir — config tests wrote
|
||
* keys into it and relink/update-check tests then read them (measured: 12
|
||
* cross-contamination failures on the first full run). Tests that need
|
||
* GSTACK_HOME isolation mkdtemp their own per test — the repo convention —
|
||
* and the hermetic-env machinery covers E2E children.
|
||
*/
|
||
export async function runFreeShard(
|
||
files: string[],
|
||
shardNumber: number,
|
||
totalShards: number,
|
||
options: RunFreeShardOptions = {},
|
||
): Promise<FreeShardOutcome> {
|
||
const log = options.log ?? ((line: string) => console.log(line));
|
||
const label = `[test:free] shard ${shardNumber}/${totalShards}`;
|
||
|
||
// Empty shard = fast no-op SUCCESS. Indices are stable for the CI matrix,
|
||
// so an unoccupied index must not fail or shift work to a different runner.
|
||
if (files.length === 0) {
|
||
const outcome: FreeShardOutcome = {
|
||
shard: shardNumber, files: [], status: 'passed', exitCode: 0, elapsedMs: 0, groupPid: null, failingFiles: [], unattributedFailures: 0,
|
||
};
|
||
log(shardEpilogue(outcome, totalShards));
|
||
return outcome;
|
||
}
|
||
|
||
const rootDir = options.rootDir ?? ROOT;
|
||
const wallTimeoutMs = options.wallTimeoutMs ?? DEFAULT_WALL_TIMEOUT_MS;
|
||
log(`${label} (${files.length} files${options.parallel ? ', bun --parallel' : ''})`);
|
||
|
||
// Full-stream capture: EVERY child byte lands here, whatever the console
|
||
// shows. Printed once at start so a wedged or noisy run is inspectable
|
||
// without a re-run.
|
||
const logPath = options.logFilePath ?? nextDefaultLogPath();
|
||
const logStream = fs.createWriteStream(logPath);
|
||
let logWriteFailed = false;
|
||
logStream.on('error', (err) => {
|
||
if (logWriteFailed) return;
|
||
logWriteFailed = true;
|
||
console.error(`${label} could not write the full log at ${logPath}: ${err.message}`);
|
||
});
|
||
log(`[test:free] full log: ${logPath}`);
|
||
|
||
const { command, args } = options.commandFor
|
||
? options.commandFor(files)
|
||
: { command: process.execPath, args: buildShardArgs(files, { parallel: options.parallel, rootDir }) };
|
||
|
||
const env = { ...(options.env ?? process.env) };
|
||
const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-free-shard-'));
|
||
const childTmp = path.join(stateDir, 'tmp');
|
||
fs.mkdirSync(childTmp);
|
||
env.TMPDIR = childTmp;
|
||
env.TEMP = childTmp;
|
||
env.TMP = childTmp;
|
||
// Per-shard Chromium profile (same isolation idea as TMPDIR): nine test
|
||
// files launch in-process persistent contexts or daemons that default to
|
||
// the SHARED ~/.gstack/chromium-profile, and two concurrent shards on one
|
||
// profile dir kill each other's browser — observed live on CI once
|
||
// duration packing recomposed shards (handoff's launchPersistentContext
|
||
// died "Target page, context or browser has been closed" while a sibling
|
||
// shard's daemon logged "Chromium process crashed"). Hash sharding had
|
||
// masked the collision by chance placement. Within a shard, files run
|
||
// serially, so sharing the per-shard profile is safe; config tests that
|
||
// assert resolution order save/restore this env around their assertions.
|
||
env.CHROMIUM_PROFILE = path.join(stateDir, 'chromium-profile');
|
||
|
||
const startedAt = Date.now();
|
||
const child = spawn(command, args, {
|
||
cwd: rootDir,
|
||
env,
|
||
stdio: ['ignore', 'pipe', 'pipe'],
|
||
detached: process.platform !== 'win32',
|
||
windowsHide: true,
|
||
});
|
||
const groupPid = child.pid ?? null;
|
||
// Group-kill on parent SIGINT/SIGTERM too, not just on timeout.
|
||
const forwarding = installChildSignalForwarding({
|
||
kill: (signal?: NodeJS.Signals | number) => {
|
||
killProcessGroup(child, (signal as NodeJS.Signals) ?? 'SIGTERM');
|
||
return true;
|
||
},
|
||
});
|
||
|
||
const classifier = new BunTestOutputClassifier();
|
||
|
||
// Console policy: quiet => nothing; verbose => the raw firehose; default =>
|
||
// only always-visible lines (fail results, crash markers, error/panic
|
||
// markers, the terminal summary), selected by the reporter. The reporter
|
||
// consumes the stream in EVERY mode so the epilogue can attribute failures.
|
||
const emitToConsole = (text: string, origin: StreamOrigin): void => {
|
||
if (options.quiet) return;
|
||
if (options.consoleWrite) {
|
||
options.consoleWrite(text);
|
||
return;
|
||
}
|
||
(origin === 'stdout' ? process.stdout : process.stderr).write(text);
|
||
};
|
||
const reporter = new FreeRunReporter(files, options.verbose ? undefined : emitToConsole);
|
||
|
||
const consumeStream = (stream: NodeJS.ReadableStream, origin: StreamOrigin): Promise<void> =>
|
||
new Promise((resolve, reject) => {
|
||
stream.on('data', (chunk: Buffer | string) => {
|
||
classifier.write(chunk, origin); // strict verdict ALWAYS sees the full stream
|
||
if (!logWriteFailed) logStream.write(chunk);
|
||
reporter.write(chunk, origin);
|
||
if (options.verbose) emitToConsole(typeof chunk === 'string' ? chunk : chunk.toString('utf8'), origin);
|
||
});
|
||
stream.on('end', resolve);
|
||
stream.on('error', reject);
|
||
});
|
||
|
||
let timedOut = false;
|
||
const killTimer = setTimeout(() => {
|
||
timedOut = true;
|
||
killProcessGroup(child, 'SIGKILL');
|
||
}, wallTimeoutMs);
|
||
|
||
let exitCode: number | null = null;
|
||
try {
|
||
const streams: Array<Promise<void>> = [];
|
||
if (child.stdout) streams.push(consumeStream(child.stdout, 'stdout'));
|
||
if (child.stderr) streams.push(consumeStream(child.stderr, 'stderr'));
|
||
exitCode = await new Promise<number | null>((resolve, reject) => {
|
||
child.once('error', reject);
|
||
child.once('close', (code) => resolve(code));
|
||
});
|
||
await Promise.all(streams);
|
||
} finally {
|
||
clearTimeout(killTimer);
|
||
forwarding.dispose();
|
||
// Reap survivors of this shard even on the clean path.
|
||
killProcessGroup(child, 'SIGKILL');
|
||
reporter.end();
|
||
await new Promise<void>((resolve) => logStream.end(() => resolve()));
|
||
try {
|
||
fs.rmSync(stateDir, { recursive: true, force: true });
|
||
} catch {
|
||
// Best-effort cleanup of a throwaway temp dir — a locked file on
|
||
// Windows must not turn a real verdict into an exception.
|
||
}
|
||
}
|
||
|
||
const summary = classifier.end();
|
||
const status: FreeShardStatus = timedOut
|
||
? 'timed-out'
|
||
: strictTestExitCode(exitCode ?? 1, summary, files.length) === 0 ? 'passed' : 'failed';
|
||
|
||
if (status === 'timed-out') {
|
||
console.error(
|
||
`${label} exceeded the ${Math.round(wallTimeoutMs / 1000)}s wall-clock deadline — `
|
||
+ 'killed the process group. Reporting as TIMED-OUT (distinct from failed).',
|
||
);
|
||
} else if (status === 'failed' && (exitCode ?? 1) === 0) {
|
||
const reason = summary.failedTests > 0 || summary.unhandledBetweenTests > 0
|
||
? `printed ${summary.failedTests} failing result(s) and ${summary.unhandledBetweenTests} unhandled error(s) between tests`
|
||
: summary.terminalFileCounts.length === 0
|
||
? "never printed bun's terminal summary — the run was truncated (a process.exit fired mid-suite)"
|
||
: `bun's summary reported ${summary.terminalFileCounts.join(', ')} file(s), expected ${files.length}`;
|
||
console.error(`${label} exited 0 but ${reason}. Treating as FAILED.`);
|
||
} else if (status === 'failed') {
|
||
console.error(`${label} failed with exit code ${exitCode ?? 'signal'}`);
|
||
}
|
||
|
||
const report = reporter.report();
|
||
const failingFiles = status === 'passed' ? [] : [...new Set([
|
||
...report.failures.map((f) => f.file).filter((f): f is string => !!f),
|
||
...report.crashedFiles,
|
||
])];
|
||
const unattributedFailures = status === 'passed' ? 0
|
||
: report.failures.filter((f) => !f.file).length
|
||
+ report.unhandledErrors.length
|
||
+ (report.sawTerminalSummary ? 0 : 1);
|
||
const outcome: FreeShardOutcome = {
|
||
shard: shardNumber, files, status, exitCode, elapsedMs: Date.now() - startedAt, groupPid, failingFiles, unattributedFailures,
|
||
};
|
||
log(shardEpilogue(outcome, totalShards));
|
||
for (const line of buildRunEpilogue(status, report, outcome.elapsedMs, logPath)) log(line);
|
||
return outcome;
|
||
}
|
||
|
||
let logPathSequence = 0;
|
||
|
||
/** Timestamped per-run log file under os.tmpdir(); pid+sequence defeat same-ms collisions. */
|
||
function nextDefaultLogPath(): string {
|
||
const stamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||
logPathSequence += 1;
|
||
return path.join(os.tmpdir(), `gstack-free-test-${stamp}-${process.pid}-${logPathSequence}.log`);
|
||
}
|
||
|
||
function exitCodeFor(status: FreeShardStatus): number {
|
||
if (status === 'passed') return 0;
|
||
return status === 'timed-out' ? 124 : 1;
|
||
}
|
||
|
||
/**
|
||
* `--record-durations`: time every file in its own child (exact per-file wall,
|
||
* immune to bun's stream buffering) and write the committed seed atomically.
|
||
* Occasional + manual by design — CI never records (a hint refreshed by a
|
||
* human beats per-run churn), and the runtime (~serial suite / jobs) is fine
|
||
* for an operation run a few times a quarter.
|
||
*/
|
||
async function recordFreeTestDurations(files: string[], jobs: number): Promise<number> {
|
||
const durations: Record<string, number> = {};
|
||
const failed: string[] = [];
|
||
let cursor = 0;
|
||
console.log(`[test:free] recording per-file durations: ${files.length} files across ${jobs} workers`);
|
||
const worker = async (): Promise<void> => {
|
||
for (;;) {
|
||
const index = cursor;
|
||
cursor += 1;
|
||
if (index >= files.length) return;
|
||
const file = files[index];
|
||
const started = Date.now();
|
||
const child = spawn('bun', ['test', file, `--timeout=${FREE_TEST_TIMEOUT_MS}`], {
|
||
cwd: ROOT,
|
||
stdio: ['ignore', 'ignore', 'ignore'],
|
||
env: { ...process.env, GSTACK_HEADLESS: '1' },
|
||
});
|
||
const code = await new Promise<number>((resolve) => {
|
||
const timer = setTimeout(() => { child.kill('SIGKILL'); }, wallTimeoutForShard(1));
|
||
child.on('close', (c) => { clearTimeout(timer); resolve(c ?? 1); });
|
||
child.on('error', () => { clearTimeout(timer); resolve(1); });
|
||
});
|
||
durations[normalizeRelativePath(file)] = Date.now() - started;
|
||
if (code !== 0) failed.push(file);
|
||
}
|
||
};
|
||
await Promise.all(Array.from({ length: Math.max(1, jobs) }, () => worker()));
|
||
|
||
const target = process.env.GSTACK_FREE_TEST_DURATIONS ?? path.join(ROOT, FREE_TEST_DURATIONS_FILE);
|
||
const payload = {
|
||
version: 1,
|
||
recordedAt: new Date().toISOString(),
|
||
durations: Object.fromEntries(Object.entries(durations).sort(([a], [b]) => (a < b ? -1 : 1))),
|
||
};
|
||
// Atomic temp+rename (capture-context-budget's pattern): a killed recorder
|
||
// must never leave a truncated seed for loadFreeTestDurations to warn on.
|
||
const tmp = `${target}.tmp-${process.pid}`;
|
||
fs.writeFileSync(tmp, `${JSON.stringify(payload, null, 2)}\n`);
|
||
fs.renameSync(tmp, target);
|
||
console.log(`[test:free] wrote ${Object.keys(durations).length} durations to ${path.relative(ROOT, target)}`);
|
||
if (failed.length > 0) {
|
||
// Failures still recorded (a red file's duration is still a real cost),
|
||
// but surfaced loudly — recording from a broken tree deserves a look.
|
||
console.error(`[test:free] WARNING: ${failed.length} file(s) failed while recording:`);
|
||
for (const f of failed) console.error(` ✗ ${f}`);
|
||
return 1;
|
||
}
|
||
return 0;
|
||
}
|
||
|
||
async function main(): Promise<number> {
|
||
const options = parseCliOptions(process.argv.slice(2));
|
||
const allFiles = collectFreeTestFiles();
|
||
if (allFiles.length === 0) {
|
||
throw new Error('No free test files were discovered.');
|
||
}
|
||
|
||
let files = allFiles;
|
||
let curationReport: CurationResult | null = null;
|
||
if (options.windowsOnly) {
|
||
curationReport = curateWindowsSafe(allFiles);
|
||
files = curationReport.safe;
|
||
console.log(`[test:free] curated ${files.length} Windows-safe tests (${curationReport.excluded.length} excluded)`);
|
||
if (options.listOnly && curationReport.excluded.length > 0) {
|
||
console.log('\nExcluded (POSIX-fragile):');
|
||
for (const { file, reason } of curationReport.excluded) {
|
||
console.log(` - ${file} [${reason}]`);
|
||
}
|
||
}
|
||
}
|
||
|
||
if (options.listOnly) {
|
||
console.log(`\nDiscovered ${files.length} test files.`);
|
||
for (const file of files) console.log(` ${file}`);
|
||
return 0;
|
||
}
|
||
|
||
if (options.recordDurations) {
|
||
const jobs = Math.max(1, Math.min(MAX_FULL_SUITE_JOBS, os.cpus().length - RESERVED_CPUS));
|
||
return recordFreeTestDurations(files, jobs);
|
||
}
|
||
|
||
if (options.dryRun) {
|
||
const shards = assignFilesToShards(files, options.shardCount);
|
||
const occupied = shards.filter((s) => s.length > 0).length;
|
||
console.log(
|
||
`\nWould run ${files.length} files across ${shards.length} shards (${occupied} occupied). `
|
||
+ 'Without --shard, the full suite runs as N concurrent shard processes '
|
||
+ '(plus a serial tree-mutating shard) instead.',
|
||
);
|
||
for (const line of formatShardSummary(shards)) console.log(line);
|
||
return 0;
|
||
}
|
||
|
||
if (options.shardIndex !== null) {
|
||
// Bounds-check against the REQUESTED shard count, not post-assignment
|
||
// occupancy — indices must be stable for a CI matrix, and an empty shard
|
||
// is a valid fast no-op.
|
||
if (!Number.isInteger(options.shardIndex) || options.shardIndex < 1 || options.shardIndex > options.shardCount) {
|
||
throw new Error(`--shard must be between 1 and ${options.shardCount}. Received: ${options.shardIndex}`);
|
||
}
|
||
const shards = assignFilesToShards(files, options.shardCount);
|
||
const outcome = await runFreeShard(shards[options.shardIndex - 1], options.shardIndex, options.shardCount, {
|
||
wallTimeoutMs: options.wallTimeoutMs,
|
||
verbose: options.verbose,
|
||
});
|
||
return exitCodeFor(outcome.status);
|
||
}
|
||
|
||
// Full-suite mode: N concurrent shard PROCESSES, serial within each — the
|
||
// paid runner's proven model. One `bun test --parallel` invocation was
|
||
// tried first (decision V3) and abandoned after three distinct
|
||
// worker-runtime pathologies in a single day on Bun 1.3.13: a segfault
|
||
// whose crashed-worker retry wedged the run (security-live-playwright), a
|
||
// gated file's still-running file-level hooks stalling a worker
|
||
// (compare-board), and spawn-heavy files hanging workers under load
|
||
// (session-runner-timeout). Plain child processes have none of these:
|
||
// proven spawn semantics, per-shard group-kill, per-shard logs, and a
|
||
// wedge only ever costs its own shard. WORKER_HOSTILE files are moot in
|
||
// process shards (no workers) and fold back into normal assignment.
|
||
const jobs = fullSuiteJobs();
|
||
// Phase split: tree-mutating tests run AFTER the parallel shards, in one
|
||
// serial shard, so no concurrent shard ever reads a half-regenerated tree.
|
||
const mutators = files.filter((f) => f in TREE_MUTATING);
|
||
const readers = files.filter((f) => !(f in TREE_MUTATING));
|
||
const durations = loadFreeTestDurations();
|
||
const packed = durations ? packShardsByDuration(readers, jobs, durations) : null;
|
||
const shards = packed ? packed.shards : assignFilesToShards(readers, jobs);
|
||
const totalShards = jobs + (mutators.length > 0 ? 1 : 0);
|
||
console.log(`[test:free] full suite: ${readers.length} files across ${jobs} shard processes`
|
||
+ (packed ? ' (duration-packed)' : '')
|
||
+ (mutators.length > 0 ? `, then ${mutators.length} tree-mutating file(s) serially` : ''));
|
||
if (packed) {
|
||
// One line per shard so a packing regression is diagnosable from any log.
|
||
packed.predictedMs.forEach((ms, i) => {
|
||
console.log(`[test:free] shard ${i + 1}: ${shards[i].length} files, predicted ~${Math.round(ms / 1000)}s`);
|
||
});
|
||
}
|
||
const shardTimeout = (fileCount: number): number =>
|
||
options.wallTimeoutExplicit ? options.wallTimeoutMs : wallTimeoutForShard(fileCount, options.wallTimeoutMs);
|
||
const outcomes = await Promise.all(
|
||
shards.map((shardFiles, index) => runFreeShard(shardFiles, index + 1, totalShards, {
|
||
// Packed shards get duration-aware walls: LPT decouples file count from
|
||
// cost BY DESIGN, so the 5s/file heuristic would undersize a shard
|
||
// holding few expensive files.
|
||
wallTimeoutMs: packed && !options.wallTimeoutExplicit
|
||
? wallTimeoutForPackedShard(packed.predictedMs[index], options.wallTimeoutMs, shardFiles.length)
|
||
: shardTimeout(shardFiles.length),
|
||
verbose: options.verbose,
|
||
})),
|
||
);
|
||
let worst = Math.max(...outcomes.map((o) => exitCodeFor(o.status)));
|
||
// Cancellation stops the run: don't launch the serial tree-mutating shard
|
||
// after a SIGINT/SIGTERM already killed the parallel phase.
|
||
if (mutators.length > 0 && !isTerminationRequested()) {
|
||
const mutatorOutcome = await runFreeShard(mutators, totalShards, totalShards, {
|
||
wallTimeoutMs: shardTimeout(mutators.length),
|
||
verbose: options.verbose,
|
||
});
|
||
worst = Math.max(worst, exitCodeFor(mutatorOutcome.status));
|
||
if (mutatorOutcome.status !== 'passed') {
|
||
// Mutator safety rests on each test restoring default state itself; a
|
||
// SIGKILL at the wall deadline (or a mid-regeneration crash) defeats
|
||
// that by construction. Say so, loudly, before someone commits
|
||
// regenerated SKILL.md / .agents artifacts by accident.
|
||
const dirty = spawnSyncGitStatusGenerated();
|
||
if (dirty.length > 0) {
|
||
console.error('[test:free] ⚠ tree-mutating shard did not finish cleanly — generated artifacts may be mid-regeneration:');
|
||
for (const line of dirty.slice(0, 20)) console.error(`[test:free] ${line}`);
|
||
console.error('[test:free] restore with: bun run gen:skill-docs (or git checkout -- <paths>)');
|
||
}
|
||
}
|
||
outcomes.push(mutatorOutcome);
|
||
}
|
||
|
||
// Opt-in flaky retry (GSTACK_FREE_RETRY_FLAKY=1): when every failure is an
|
||
// attributed test failure (no timeouts, no unattributed carnage), re-run
|
||
// just the failing files ONCE in a fresh serial shard. A clean retry
|
||
// downgrades the run to a loud flaky-pass; a repeat failure stays a
|
||
// failure. Default OFF: dev boxes should see flakes, not absorb them.
|
||
// Exists for syscall-supervised sandboxes (see fullSuiteJobs) where a run
|
||
// lands 0-1 spurious browser-timing failures under an otherwise-green
|
||
// suite. Capped so a genuinely broken tree never masquerades as flaky.
|
||
const RETRY_CAP = 5;
|
||
if (
|
||
worst !== 0
|
||
&& process.env.GSTACK_FREE_RETRY_FLAKY === '1'
|
||
&& !isTerminationRequested()
|
||
&& outcomes.every((o) => o.status !== 'timed-out')
|
||
) {
|
||
const flakyFiles = [...new Set(outcomes.flatMap((o) => o.failingFiles))]
|
||
.filter((f): f is string => typeof f === 'string' && f.length > 0);
|
||
// "Fully attributed" is per-failure, not per-shard: a shard with one
|
||
// attributed failure PLUS a headerless failure / unhandled error /
|
||
// truncated run must veto the retry — re-running only failingFiles would
|
||
// mask the unattributable evidence as a FLAKY-PASS.
|
||
const allAttributed = outcomes.every((o) => o.status === 'passed'
|
||
|| (o.failingFiles.length > 0 && o.unattributedFailures === 0));
|
||
if (allAttributed && flakyFiles.length > 0 && flakyFiles.length <= RETRY_CAP) {
|
||
console.log(`[test:free] flaky-retry: re-running ${flakyFiles.length} failing file(s) once, serially: ${flakyFiles.join(', ')}`);
|
||
const retryOutcome = await runFreeShard(flakyFiles, totalShards + 1, totalShards + 1, {
|
||
wallTimeoutMs: shardTimeout(flakyFiles.length),
|
||
verbose: options.verbose,
|
||
});
|
||
if (retryOutcome.status === 'passed') {
|
||
console.log(`[test:free] FLAKY-PASS — ${flakyFiles.length} file(s) failed once and passed on serial retry: ${flakyFiles.join(', ')}`);
|
||
console.log('[test:free] treat repeat offenders as real flakes worth fixing, not noise.');
|
||
// Durable record (WS1): console lines vanish with the scrollback; the
|
||
// ledger makes repeat offenders rankable across runs (eval:flake-rank).
|
||
const ts = new Date().toISOString();
|
||
// Two separate calls: `rev-parse --abbrev-ref HEAD HEAD` abbreviates
|
||
// BOTH revs, printing the branch twice — git_sha recorded the branch
|
||
// name (codex adversarial finding).
|
||
const ledgerBranch = (spawnSync('git', ['rev-parse', '--abbrev-ref', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim();
|
||
const ledgerSha = (spawnSync('git', ['rev-parse', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim();
|
||
appendFlakeLedger(
|
||
flakyFiles.map((file) => ({
|
||
ts,
|
||
runner: 'free' as const,
|
||
kind: 'flaky-pass' as const,
|
||
file,
|
||
shard: outcomes.find((o) => o.failingFiles.includes(file))?.shard,
|
||
...(ledgerBranch ? { branch: ledgerBranch } : {}),
|
||
...(ledgerSha ? { git_sha: ledgerSha.slice(0, 12) } : {}),
|
||
})),
|
||
flakeLedgerPath(),
|
||
);
|
||
worst = 0;
|
||
} else {
|
||
console.error('[test:free] flaky-retry FAILED — the failures reproduce serially; not flaky.');
|
||
}
|
||
} else {
|
||
console.log(`[test:free] flaky-retry skipped: ${allAttributed ? `${flakyFiles.length} failing file(s) exceeds cap ${RETRY_CAP}` : 'failures not fully attributed'}.`);
|
||
}
|
||
}
|
||
return worst;
|
||
}
|
||
|
||
/** Dirty generated artifacts (SKILL.md / host outputs) after a failed mutator shard. */
|
||
function spawnSyncGitStatusGenerated(): string[] {
|
||
const result = spawnSync('git', ['status', '--porcelain'], { cwd: ROOT, encoding: 'utf8' });
|
||
if (result.status !== 0 || !result.stdout) return [];
|
||
return result.stdout.split('\n').filter((line) =>
|
||
/SKILL\.md$/.test(line) || line.includes('.agents/') || line.includes('.factory/'));
|
||
}
|
||
|
||
if (import.meta.main) {
|
||
try {
|
||
process.exitCode = await main();
|
||
} catch (error) {
|
||
console.error(`[test:free] ${error instanceof Error ? error.message : String(error)}`);
|
||
process.exitCode = 1;
|
||
}
|
||
}
|