mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
+44









Garry Tan
Lewis
Yugz
Florian
cdecook23
mb1810
Koda
Merit Blake
Nguyen Thanh Dat
pneumorea-ai
maxpetrusenkoagent
Matt Van Horn
Infiniteyieldai
Gabriel Rondon
Davon Johnson
Kevin Grasso
Xuan-Ce Wang
loulanyue
Thomas Reuser
Bit0ps
oli548
Slava
ryanlaiwy
李翊楠
Mammad M.
Lockyer
Spooks444
punksterlabs
Jonathan Zhang
Marek Cermak
y$un_
neallee
hardy
afshaker
cube
jacob-wang
miao
Jonah Berg
Jayesh Betala
Ryan Alberts
AlinValentin7
Evgenii Lopatin
Jino
snig-17
Sholto McNeilage
Phil Hie
Sinabina
Jimmy Zeisweiss
liutiming
Brandon Pugsley
benjamin beres
Som Samantray
Vaughan
Bharadwaj Mittapelli
2db0b3adc8
* docs: approved severe fix wave plan (Oct 2026) * D0: keep a project's .env and bunfig.toml out of compiled binaries and daemon children Compiled binaries are built with --no-compile-autoload-dotenv and --no-compile-autoload-bunfig; the browse server and terminal agent are spawned with --no-env-file and an empty config, so a repo's .env cannot set GSTACK_CHROMIUM_NO_SANDBOX and a bunfig preload cannot run code in them. Verified on Bun 1.4.0. * A9: --no-write is a dry run and leaves ingest state untouched --no-write stamped every prepared page ingested_at and saved state, so a later real run skipped those pages forever. It now prepares and reports pages without importing or changing the state file. Tests that used GSTACK_MEMORY_INGEST_NO_WRITE / --no-write to seed state move to an explicit seam (a fake gbrain that accepts every staged page), because the --no-write contract deliberately changes. * H #2885 #2827: vector geometry is not phone or card PII pii.phone.e164 accepts "." as a group separator, so a single float (viewBox="0 0 581.66796875 ...") and a run of path coordinates ("37.6188 101.694") read as phone numbers. pii.cc's \b stops at a decimal point, so the long fraction of an .excalidraw float is a Luhn candidate. Excalidraw's seed/versionNonce/updated integers read as bare phone numbers. A rendered /diagram triplet raised 18 (svg) and 97 (excalidraw) MEDIUM findings on main; it now raises none. - looksLikeDecimalCoordinates (ported from #2885, extended to a single decimal and refusing a leading "+", the E.164 marker) - insideDecimalNumber: a cc span glued to "<digit>." or ".<digit>" - isNumericMetadataValue: an unquoted JSON value under a seed, nonce or timestamp key; the same digits under "phone" still report Controls: E.164 in prose, dotted and spaced phones, "+"-prefixed runs, "phone"/"contact" JSON numbers, Luhn-valid cards in prose and at sentence end. Co-authored-by: Lewis <lewispeel@hotmail.com> * D1: pin the terminal-agent /ws Origin to the gstack extension (#1324) GSTACK_EXTENSION_ID moves to browse/src/extension-id.ts, shared by /extension-token and the terminal agent's /ws gate. /ws now returns 403 for every origin except the pinned extension. Forks override the ID with gstack-config set browse_extension_id <id>, honored by both routes; the BROWSE_EXTENSION_ID env var is no longer read. Test sites that used fake extension IDs now use the pinned origin. * H #2899: env.kv skips a function call assigned to a credential name `session = _FlakySession(...)`, `token = make_token(user, ...)` and `password = getpass.getpass()` raised env.kv MEDIUM, which blocks the fail-closed decision log. The exemption needs the line's evidence, not the span's shape: the value is unquoted, is an identifier or dotted path followed by "(", and the call closes on the line or opens a multi-line argument list. Quoted values, unclosed "(" inside an unquoted secret (API_SECRET=Abc123xyz(9Qm) and calls carrying a high-entropy literal argument still report. The literal check is shared with the #2912 env-read exemption. Co-authored-by: Yugz <yannduzelier@gmail.com> * H #2962: per-machine config filenames are not internal hostnames `CLAUDE.local.md` and `settings.local.json` reported internal.hostname MEDIUM (the pattern's \b stops at the dot before the extension), which fails closed in the decision log. A span immediately followed by a known config extension (.md .json .jsonc .yaml .yml .toml .ini .conf .txt) and a word boundary is a filename. Controls: `ssh build.local`, `printer.local.` at a sentence end, `printer.local/md`, `printer.local.mdx`, a host with a port, `api.corp`, `build-7.internal`. The `.env.local` vectors (#2740) pass on main already via isDotenvFilename and are pinned here as regressions. Co-authored-by: Florian <florian.cleyetmerle@gmail.com> * H #2784: four-part versions need a version declaration, not a shape `"version": "1.128.1.0"`, `<Version>`, `AssemblyVersion("1.0.0.0")` and Keep a Changelog headings raised pii.ip_public MEDIUM on every release, because a four-part version is byte-for-byte a dotted quad. Exempt only when the same line declares it a version immediately before the value (a version key or word, an XML tag, an attribute call, or a `## [x]` heading). The same digits after host:/server =/"upstream", bare on a line, or after prose that merely mentions a version still report. Unlike #2784, this does not exempt a value because the same string is declared a version elsewhere in the text: that let a real address sharing a diff with a matching version slip. The pre-push hook's VERSION-file rule (#2856) is unchanged. Co-authored-by: cdecook23 <decook23@gmail.com> * H #2913: postgres:postgres is exempt only on a loopback host The postgres image's default pair in compose files and CI DATABASE_URLs blocked pushes as db.url_with_password HIGH. It is now exempt only for postgres/postgresql URLs with exactly postgres:postgres on localhost or 127.0.0.1 (optional port). A bare single-label host (`db`, `postgres`), any other host, a host that merely starts with localhost (`localhost.evil...`, `localhost@prod-db`), any other user or password, and other schemes still block. #2913 exempted the pair on any host; this keeps the plan's narrower rule. Co-authored-by: mb1810 <michael.berger1810@gmail.com> * G1: ship without a version when no version source is configured Resolve the version source to four outcomes (valid, absent, ambiguous, broken) in lib/version-source.ts and read that one signal in gstack-version-bump (new NO_VERSION state; write refuses to invent a VERSION), gstack-next-version (no_version, no slot), /ship's bump, CHANGELOG and title steps, /land-and-deploy's drift check and /document-release. A configured version file that is missing, empty, unreadable or malformed stops with its path; 0.0.0.0 is never substituted. docs/skills.md documents .gstack/version-path. Migrates two tests that pinned the old fabricated-0.0.0.0 behavior. Co-authored-by: Koda <koda@edgeviewfinance.com.au> Co-authored-by: Merit Blake <merit@newsprint.com> * H #2949: eval credential screen checks every name segment The hermetic eval env admits GITHUB_* and EVALS_* metadata but rejects credential-shaped names, judged only by the last underscore segment. A trailing qualifier carried a credential through: GITHUB_APP_PRIVATE_KEY_BASE64, GITHUB_TOKEN_1, EVALS_API_KEY_FALLBACK reached eval children. The screen now rejects a name when any segment is a credential word; segments, not substrings, so GITHUB_PATH / GITHUB_TOKENIZER / GITHUB_KEYRING stay metadata, and explicit runner admissions still win. Co-authored-by: Nguyen Thanh Dat <ntdat812@gmail.com> * H #2915: cover chunked scanning of large multi-file push diffs Test-only coverage for behavior already on main: a clean two-file pushed diff over the engine's 1 MiB cap passes without engine.input_too_large, and a credential at the end of the second file (a later scan chunk) still blocks the push. Co-authored-by: pneumorea-ai <pneumorea@gmail.com> * A3: keep a project's DATABASE_URL away from a PGLite brain With a PGLite config, buildGbrainEnv returned the caller env unchanged, so a project's .env DATABASE_URL (autoloaded by Bun) reached gbrain: a false broken-db at best, gbrain talking to the app's own Postgres at worst. Inherited DATABASE_URL and GBRAIN_DATABASE_URL are now blanked for PGLite configs (GSTACK_RESPECT_ENV_DATABASE_URL=1 still keeps them), and every gbrain child runs from gbrain's config dir instead of the user's project, so it cannot reload the project's .env / .env.local. The old test that pinned pass-through for an engine:pglite config is migrated: pass-through now applies only to configs that name neither a database_url nor PGLite. Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> * C9: guard destructive gstack-upgrade and spec fences against unset or wrong paths Fresh-shell hosts (and Claude Code, which keeps cwd but not shell variables) ran cd "$INSTALL_DIR" with an empty variable, so git checkout/stash/reset targeted the user's project. Step 2 now prints an absolute path; every destructive fence starts with cd -- "${VAR:?...}" || exit 1 and checks it is gstack's own checkout before any git command. /spec's worktree and spawn fences get the same guard. * F4: keep every paid design variant across regenerate rounds $D variants now reserves a collision-safe round-<N>/ directory per run (atomic mkdir) and writes a manifest.json of what each round saved. The paid image request and the local write retry separately: a write failure retries the write (temp + rename) and, if the path stays unwritable, keeps the bytes in the temp dir instead of buying the image again. Responsive variants land in their round directory too. Readers updated: design-shotgun reserves a round dir and a private temp dir for its parallel agents and records approved_path/round in approved.json; design-html resolves approved_path (legacy letters still work) and finds round-dir variants; design-consultation checks the printed paths; the gallery shows round-dir variants. Variants that were already overwritten before this change cannot be recovered. Migrates the consultation contract test that pinned the flat path. Co-authored-by: Matt Van Horn <mvanhorn@gmail.com> * A2: stage dot-leading slugs gbrain can collect; network errors are db-unreachable A page slug segment starting with a dot (a .claude project) staged a file gbrain's import walker prunes, so the batch was refused on every run. Leading dots become "dot-" per segment, and slugs recorded in state under the old name map through the same function so no page is duplicated. A DNS or connection failure (ENOTFOUND, EAI_AGAIN, ECONNREFUSED, ...) no longer classifies as broken-db, which routed users to move a working config aside. The new db-unreachable status carries the reason "database host unreachable (<code> <host>); your gbrain config is unchanged. Fix: check network or VPN, then re-run /sync-gbrain.", skips local sync stages with that reason, and counts as usable for gstack-gbrain-detect --is-ok like timeout. Co-authored-by: Infiniteyieldai <infiniteyield.ai@gmail.com> * D2: one address classifier for navigation literals, DNS answers, redirects and page-driven navigations (#2811) classifyAddress() in url-validation.ts blocks 169.254.0.0/16 (metadata and container credential endpoints), 100.100.100.200, fe80::/10 and fc00::/7, including IPv4-mapped, IPv4-compatible and NAT64 spellings and decimal, octal and hex IPv4. validateNavigationUrl, the A/AAAA answer check and session-persist's cookie-domain guard all use it. BrowserManager now checks every navigation request (redirect hops, link clicks, scripted navigations): a blocked target resets the tab to about:blank and the write command fails with the reason. Connection-time DNS rebinding remains out of scope. Co-authored-by: Gabriel Rondon <grondon@gmail.com> * C4: external hosts point at carved sections relative to the installed skill Publishes the pointer form setup matches: on every non-Claude host a carved section is `sections/<file>` relative to the installed gstack-<skill> SKILL.md directory (QA's existing form), never $GSTACK_ROOT/<skill>/sections/, which does not exist in external runtime roots. No render changes yet: only QA is carved on external hosts today. * INV-1: gate outcome contract for outside reviews classifyOutsideReview() returns separate execution, findings and verdict fields from text, stderr and exit status; reason codes, anchors and fix actions live in lib/gate-outcomes.ts. validateOutsideReview() stays as the text-only wrapper and the two-argument CLI keeps exit 0/1/2 for importers and installed skills that predate a re-render. The --verdict form (implied by --stderr/--exit) prints VERDICT/FINDINGS/REASON and exits 0/3/4/1; the generated outside-voice caller moves to it and branches on each status (unverified prints OUTSIDE_STATUS: unverified, never completed). Adds docs/troubleshooting.md with one anchored section per reason code, the anchor test, and the gate-tail lint (query scripts may not swallow failures, gate calls may not end in || true, other best-effort tails say why). * G1: quote the base placeholder in document-release title sync The title-sync block is checked with bash -n; an unquoted <base> parses as a redirection. Migrates the test that pinned the old root VERSION read to the shared classifier call. * G3: give every logging-bin mktemp a ${TMPDIR:-/tmp} template Bare mktemp, mktemp -t and literal /tmp templates in the learnings, question and preference logs (plus jsonl-merge, distill, the two dashboards, /ship's PR body and /document-release's run dir) now use "${TMPDIR:-/tmp}/<name>.XXXXXX", the form the INV-3 lint accepts. When the temp file cannot be created, the learnings, question and preference logs now say the entry was not recorded and how to fix TMPDIR instead of exiting with only mktemp's own error. Three tests that pinned the bare PR_BODY_FILE=$(mktemp) and one that pinned the /tmp run dir migrate. GNU and BSD mktemp already honor TMPDIR without a template, so the macOS-sandbox drop in #2952 is unverified here; the template makes the temp location explicit on every mktemp. Co-authored-by: Davon Johnson <d.johnson@dayofshow.net> * B1b: structured outside-review gate fails on P0 as well as P1 validateOutsideReview() derived its structured pass/fail from a [P1] match, so a review reporting only [P0] findings passed. The wrapper now reads the classifier's findings verdict (P0 or P1), and the /review and /ship structured-gate prose requests [P0]-[P3] tags and fails on P0/P1; untagged output (OUTSIDE_STATUS: unverified) is missing coverage. The ship clarity pin moves from 'structured P1' to 'structured P0/P1' with the behavior. * E1: freeze normalizes boundary and target the same way on Windows; hooks spawn through cmd.exe (#2876, #2354) gstack_hook_normalize_path (careful/bin/hook-extract.sh) maps C:\..., c:/..., UNC \\server\share (leading // kept) and /cygdrive/c to one form; check-freeze runs both the saved boundary and file_path through it before the absolute test, after symlink resolution, and compares case-insensitively on Windows bash. freeze, guard and careful register hooks as bash -c "exec bash \"$HOME/...\"" so cmd.exe, which never expands $HOME, no longer turns the hook into a silent exit 127. Co-authored-by: Kevin Grasso <kevin@biblingo.com> Co-authored-by: Xuan-Ce Wang <261591073+xwang4-svg@users.noreply.github.com> * G4: keep CLI-to-daemon loopback calls off HTTP_PROXY (#494) The CLI appends 127.0.0.1, localhost and ::1 to the user's NO_PROXY (and no_proxy), keeping existing entries, before its first fetch. Verified on Bun 1.4.0: fetch honors NO_PROXY when it is set before the first request and ignores later changes, so it is set at the top of main(). * F1: bind the plan audit to this branch's plan, never the newest file Plan File Discovery now prefers an explicit binding (the plan-mode file or /autoplan ACTIVE_PLAN from the conversation, then a "Plan: <path>" line in the PR body), then lists candidates: repo-committed docs/designs/ files this branch changed or that name it, and personal plan files that name the branch. Candidates are offered, never picked silently. The repo-name match and the newest-file-of-the-day fallback, which audited PRs against unrelated plans, are gone. With nothing bound or chosen it prints the exact line 'Plan completion audit: not run (...). Fix: add "Plan: <path>" to the PR body, or run /autoplan.' /ship's parent now binds the plan before dispatching the audit subagent ({{PLAN_COMPLETION_GATE_SHIP:discovery}}), so the child never searches. Migrates three tests that pinned the old markers and notice. Co-authored-by: loulanyue <260355617@qq.com> * G5: browse type --selector <sel> <text> (#2936) --selector focuses the element (sel or @ref) and types into it. The bare form still types its text literally into the focused element; text that starts with -- goes after a -- separator. When the first word looks like a CSS selector, the result adds a hint to use --selector without echoing the typed text, and never refuses based on page contents. Help and command usage updated. Co-authored-by: Thomas Reuser <thomasreuser@gmail.com> * F2: design-doc discovery reads docs/designs, never a root DESIGN.md A root DESIGN.md is the design system /design-consultation writes, so preferring it unconditionally reviewed plans against a token file while printing "Design doc found" and never looked at docs/designs/ feature docs. Discovery now considers only docs/designs/*.md as the repo-local design doc (still freshness-compared with the private ~/.gstack copy). The CEO finding fixture seeds its supplied design as a docs/designs doc and its test follows. Co-authored-by: Bit0ps <alex.dreamscaper@gmail.com> * C4: rename the external carve test (typecheck:test union-order artifact) tsc prints a union's members in type-creation order; a test file sorting before ceo-mode-option.test.ts reordered one baseline diagnostic's text. * C1: every env-var host fence resolves its own gstack root Codex and the other env-var hosts run each block in a fresh shell, so GSTACK_ROOT/BIN, B and D set by the preamble were empty in later blocks (/bin/gstack-slug, B=/browse). scripts/resolvers/runtime-root.ts now owns root resolution: a valid exported GSTACK_ROOT (bin/ and lib/), else the per-install literal root (no git call), else repo-local, else the global root (CODEX_HOME on Codex); no root prints the setup fix line and exits. One post-render pass inserts it (<=400 bytes) into every fence that uses these variables without assigning them; B/D come from the same helper as BROWSE and DESIGN SETUP. Claude output is byte-identical. Migrated pins of the old preamble lines (gen-skill-docs.test.ts) and a fixture whose exported GSTACK_ROOT lacked lib/ (skill-positional-literals). * E8: headless browse honors GSTACK_CHROMIUM_PATH; extensions load in new headless mode (#2771, #1968, #2281) The headless launch passes GSTACK_CHROMIUM_PATH as executablePath, scopes the XProtect self-heal out for that bundle, and names the path in a launch failure with the fix. BROWSE_EXTENSIONS_DIR now launches a persistent context under the project state dir with --headless=new, so the extension runs and no window appears (the off-screen window was visible on macOS and the extension never ran in an isolated newContext). The GPU source guard test moves from the removed useHeadless flag to launch() vs launchHeaded(). Co-authored-by: oli548 <312288720+oli548@users.noreply.github.com> Co-authored-by: Slava <slava.istomin@tuta.io> Co-authored-by: ryanlaiwy <129729476+ryanlaiwy@users.noreply.github.com> Co-authored-by: 李翊楠 <yinanli1917@gmail.com> * C2: link design/dist and make-pdf/dist into every host runtime root and the agents sidecar Codex design and make-pdf skills fall back to $GSTACK_DESIGN and $GSTACK_MAKE_PDF under the runtime root, but setup linked only browse/dist, so both preflights reported NOT_AVAILABLE on a fresh install. One helper links dist/ only (make-pdf/ ships a Claude SKILL.md) into the Codex, Factory, OpenCode, Cursor, Copilot and Kiro roots, the .agents sidecar and the Cursor sidecar. Co-authored-by: Mammad M. <mammad@digit.az> * C3: install router and nested SKILL.md files as real copies; back up hand edits; --status checks Codex skips a symlinked SKILL.md file, so the gstack router linked into ~/.codex/skills/gstack was invisible. Every host runtime root (Codex, Factory, OpenCode, Cursor, Copilot, Kiro) and Kiro's per-skill installs now get real copies, rewritten on every run (the auto-update path runs setup --refresh-registered, so it refreshes them too). setup records each copy's hash in <state root>/skill-copies.tsv; a copy that no longer matches is saved to a uniquely named backup under <state root>/backups/skill-copies/ before _activate_runtime_root replaces the tree, and a failed backup keeps the old tree. setup prints what it refreshed or removed. ./setup --status now prints, per registered install, whether the router is a real file and whether every section pointer resolves, with the fix command. Test migration: setup-codex-model pinned the Kiro router/upgrade links as _link_or_copy; C3 deliberately changes them to copies. * INV-3: install-level host conformance checks (no symlinked SKILL.md, dist present, section pointers resolve) For codex, kiro, factory, opencode, cursor and copilot, the real-setup conformance kit now fails when an installed SKILL.md is a symlink, when design/dist or make-pdf/dist is missing from the runtime root, or when a 'Read `.../sections/<f>.md`' pointer in an installed SKILL.md does not resolve (bare pointers in carved skills, relative-to-installed-skill pointers, and $GSTACK_ROOT/~/absolute/../ pointers everywhere), and ./setup --status must report 'all resolve'. No new setup runs. Expected red until the generator lane lands: design-html's install-offer pointer and plan-eng-review's review-sections pointer name $GSTACK_ROOT/<skill>/sections/... on every external host. * B11: auto-update tracks pull, setup and migrations; announces only on success; retries with backoff gstack-session-update wrote the 'just upgraded' marker after SETUP_FAILED or SETUP_SKIPPED bun_missing, and re-ran setup only when HEAD moved, so a failed install was announced as an upgrade and never retried. Stages are now persisted in <state root>/session-update-pending; the marker is written only when every stage succeeded; a pending setup or migrations stage is retried even with HEAD unchanged after the update-check interval, then 6h, then 24h; and each session start prints one line naming the stage, the reason and the repair command while it is pending. Best-effort tails carry a same-line reason. * D1: gstack-config key browse_extension_id for forks and self-built extensions The browse daemon trusts one Chrome extension ID on /extension-token and the terminal-agent /ws. The override now lives only in gstack-config (never an env var a project .env could set): empty by default (gstack's published extension), a 32-letter a-p Chrome ID when set, "" clears it, and anything else is rejected with the stored value kept. The browse lane reads it. gstack-config-defaults: browse/src counts as a reader directory; the key is listed prose-only until the browse lane's reader lands (the stale-entry check then asks for the entry's removal). * G2: key the sticky slug cache on the git common dir A linked worktree used to resolve its slug fresh from the remote while the main checkout kept its sticky pre-remote identity, so learnings, checkpoints and timelines split into two buckets. gstack-slug and its TS twin in lib/bin-context.ts now key the cache on the git common dir (read from .git files, no git spawn, so cache hits stay git-free) and read entries in one deterministic order: the repository entry, then the main checkout's pre-upgrade entry, then this directory's own. Both the repository and per-path entries are written. When a worktree's own old entry named a different bucket that still holds data, that slug is recorded in projects/<slug>/.former-slugs and reported by --get FORMER_SLUGS. New --adopt-legacy --from <slug> merges it: JSONL logs append missing lines with de-duplication, missing files copy, differing files are listed as conflicts and never overwritten, the source bucket is kept, and it prints migrated/pending/conflicting counts; reruns are idempotent. --adopt-legacy --dismiss <slug> drops the reminder. Co-authored-by: loulanyue <260355617@qq.com> * C5: OpenCode /gstack-* commands load the skill by its frontmatter name OpenCode's skill tool knows gstack-review as `review` (its frontmatter name), and OpenCode's builtin /review shadows that name as a command. The managed /gstack-review command told the agent to load `gstack-review`, a name the skill tool does not list. Every managed command now names the frontmatter name. OpenCode's AskUserQuestion/ExitPlanMode mapping (#2759) already shipped in v1.91.16.0. Test migration: the #2629 commands test pinned the directory name. Co-authored-by: Lockyer <135391289+Lockyer228@users.noreply.github.com> * E4: Windows node-server bundle keeps sharp and socks external; CLI resolves its server script lazily (#2260, #2439) build-node-server.sh passes --external sharp and --external socks (Bun on Windows refuses to inline them; on Linux they were silently inlined). The CLI no longer resolves server.ts or server-node.mjs at module load: resolveServerLaunch() runs when a server starts, so a Windows minimal runtime with only server-node.mjs works and a missing script is reported only when one is needed. Co-authored-by: Spooks444 <193448029+Spooks444@users.noreply.github.com> Co-authored-by: punksterlabs <77217235+punksterlabs@users.noreply.github.com> * B1: Codex reviews whose sandbox could not start are unavailable, not a pass In containers without unprivileged user namespaces every Codex command fails, yet the review exits 0 and concludes 'no findings' (captured in test/fixtures/codex-sandbox/). The validator now takes execution evidence: 'codex exec --json' command events and the 'codex review' stderr transcript (a command that succeeded proves execution); sandbox failure lines on stderr or in failed command output return unavailable(sandbox_unavailable); exact 'could not run commands / read the diff' phrases are the last fallback. Review text that merely discusses bwrap or namespaces still passes. A free 'codex sandbox ... true' preflight (Linux, 10 s) runs before the paid model probe and every automatic outside review; CODEX_MODE gains sandbox_unavailable, and an inconclusive model probe reports unverified instead of ready (exit code kept for stale callers). Every Codex site (outside-voice resolver, /codex review, challenge, consult) takes its sandbox from _GSTACK_CODEX_SANDBOX and pipes its result through the validator; a static test parses rendered command positions for both. GSTACK_CODEX_NO_SANDBOX=1 (exactly 1, shell env only) switches to danger-full-access with a warning on every use. Sandbox pins in existing tests move from the literal read-only to the selected-sandbox variable; captured autoplan probes are judged against the probe block they executed. * C6: gstack-relink discovers skills structurally, so /browse survives a skill_prefix toggle relink skipped a hardcoded list of support-directory names that included browse, a real skill, so toggling skill_prefix removed /browse. A top-level directory is now a skill exactly when it carries a SKILL.md. Co-authored-by: Jonathan Zhang <topcoder1@Mac-Studio.local> * C4: carve ship and plan-ceo-review on every host; enforce the 160 KB byte gate The generated ship skill was 202-241 KB on all 10 non-Claude hosts (and Factory plan-ceo-review 160,039 bytes), over the 160,000-byte ceiling, because section carving ran only for Claude. Both skills now carve on every host, pointing at sections relative to the installed skill directory (literal $GSTACK_ROOT/<skill>/sections refs are rewritten the same way). The ceiling is measured in UTF-8 bytes and fails generation in repo mode; under --install-root it warns so setup never leaves a user without skills. qa-lazy-sections.test.ts pinned ship as inline-on-external-hosts; migrated to the carved policy and kept the inline check on review (Claude-only carve). * B5: learnings and timeline reads fail loudly when bun is missing gstack-learnings-search and gstack-timeline-read ended in '2>/dev/null || exit 0', so a missing bun or a crashed embedded script printed nothing and exited 0, identical to 'nothing recorded'. Both now exit 127 with '<script>: bun not found on PATH, so ... could not be read. Fix: install Bun (https://bun.sh), then re-run ./setup.' and let an embedded-script failure exit non-zero with its stderr. A successful run with no matches still exits 0 with empty output. The generated callers that print 'LEARNINGS: unavailable (...)' belong to the generator lane. Co-authored-by: loulanyue <260355617@qq.com> * B2: a retired-model 404 reports MODEL_UNUSABLE, not CODEX_MODE: ready The model probe matched only 400 / 'is not supported', so a retired model's 404 ('does not exist or you do not have access to it') and a custom provider's bare 404 fell through to the inconclusive branch and reported ready. Both now classify as MODEL_UNUSABLE with a hint naming the two causes (retired model, or a wrong base_url) and log codex_model_retired; a 'requires a newer version of Codex' 400 points at upgrading the CLI and logs codex_cli_outdated. Ported from #2843 onto the current probe. Co-authored-by: Marek Cermak <marek.cermak@strv.com> * B6: rank learnings by matched query tokens and disclose truncation A more specific query used to drop the exact match: the token-OR filter recalled every entry sharing one token, the sort used confidence alone, and the default --limit cut silently. Entries now rank by how many distinct query tokens they match, then confidence, then recency, and a query that matched more than fit says '-- N more matched, raise --limit to see them'. The hit count lives in a local map so a field stored on disk can never become a sort key, and the type grouping uses a null-prototype map so a row typed 'constructor' or '__proto__' cannot crash the run. Ranking code ported from #2796; regression vectors adapted from #2799 (its stemmer is not needed). Co-authored-by: loulanyue <260355617@qq.com> Co-authored-by: y$un_ <forrest.sun527@gmail.com> * C7: required team mode also blocks Copilot CLI's lowercase skill tool Copilot CLI applies Claude-format matchers by tool name; its skill tool is `skill`, which no Claude tool name maps to, so the 'Skill' matcher never fired and Copilot ran protected skills without gstack. The hook entry now matches 'Skill|skill', an entry an older team-init wrote is upgraded in place, and the deny is emitted in both decision formats (Copilot reads the top-level permissionDecision, Claude Code reads hookSpecificOutput) with exit 0 so both parse it. Test migration: the settings test pinned matcher 'Skill'. Co-authored-by: neallee <neallee@gmail.com> * B3: Codex auth probe accepts custom providers' env_key credentials The auth probe only knew CODEX_API_KEY, OPENAI_API_KEY and auth.json, so a working self-hosted or OpenAI-compatible provider configured with [model_providers.<id>] env_key = "X" reported AUTH_FAILED and /codex stopped. The probe now reads env_key from model_providers tables only (comments ignored, the name must be a shell identifier before it is dereferenced) and accepts a set variable; when none is set it names the missing key on stderr. Ported from #2192 with the table scoping added. Co-authored-by: hardy <932821449@qq.com> * E8: keep the extensions profile under the state dir at a path lock cleanup accepts cleanSingletonLocks only cleans directories named chromium-profile, so the BROWSE_EXTENSIONS_DIR profile moves to <state>/extensions/chromium-profile. * B10: plan-tune never reports CALIBRATED with zero signal events Stats printed CALIBRATED: true from logged-answer counts alone, while every answer could miss SIGNAL_MAP and leave each dimension at its 0.5 seed (#2623). --derive now records inferred.signal_events, the answers that actually moved a dimension; calibration requires at least one, and the summary prints 'not calibrated: no recorded signals' verbatim when there are none (or says the count is unknown for a profile derived before this change). The psychographic redesign stays out of scope. * A1: verify gbrain landing before stamping transcripts, plus --reconcile gbrain import exits 0 while skipping pages (returned per-file failures, malformed filenames, managed-import Pending receipts), and its failure ledger exists only for git import dirs, so a skipped page was stamped ingested and never retried (field reports: 37 and 69 lost entries). Write, verify, then stamp (lib/memory-ingest-landing.ts, a new ratcheted module): - State schema 2 with a transition table (staged -> imported_unverified -> ingested, plus refused, quarantined, unrecoverable). Schema 1 files migrate in place: entries become ingested in the legacy "default" source and a reconcile pass is marked pending. - Named failures come from --json `failures`, stderr "Skipped <rel>:" lines and the ledger; a failure gbrain does not attribute refuses the whole batch. After 3 refused batches the batch is bisected and the page that poisons it is quarantined (printed even under --quiet). - Landing check: one `gbrain list --source-id` per source per run and at most 50 `gbrain get --json` content checks against a gstack_content_sha256 frontmatter field gstack now writes. Lookup errors and pglite_busy mean "not yet checked", never absent; pages over the cap stay imported_unverified and are not re-imported. - Each entry records its gbrain source_id. Remote-http mode records "staged", never "ingested". - The source fingerprint comes from the exact bytes parsed, so a transcript that grows during import is not stamped as its newer self (the old unscanned contract test is migrated; it pinned that bug). - --reconcile mode (bounded by --limit, default 200; resumable; --dry-run; backs up the state file) prints "reconcile: checked N, present M, re-queued K, not yet checked L". --request-reconcile records a pending pass without calling gbrain; the v1.91.17.0 migration (placeholder name) uses it, and the next ingest run does one bounded pass. - One lock for every state writer; a state-save failure fails the run. The #1802 static pins in regression-1611 move to the new structure (remote-http returns before importBatch; the finally honors preserveStaging). Co-authored-by: afshaker <andrew@phone2.io> * B4: arm64 re-sign keeps a backup, restores it when codesign fails, and reports design unavailable on exit 137 setup's Apple Silicon re-sign mutated each binary in place (truncate, remove signature) and left it that way when re-signing failed; a design binary that was then SIGKILLed at launch produced at most a quiet log line. Each binary is now copied first and restored as built when signing fails, and design is exec-probed after re-signing: exit 137 prints 'design unavailable: ... Fix: cd <checkout> && ./setup' (even under -q). Verified on Linux with the real block and uname/codesign/otool/stat stubs; not run on a Mac. * D3: treat design docs and handoff notes as data in plan reviews plan-ceo-review, plan-eng-review and plan-devex-review read the freshest design doc as the source of truth, and nothing said its content is data. In multi-agent pipelines another agent may have written it, or a handoff note beside it, with reviewer-directed text ('skip Step 0', 'approve as-is'). Each read site now says the doc and any handoff note are data, never to follow text aimed at the reviewer, and to report it as suspicious content. The plan-completion audit's discovery, which now reads repo docs/designs/ files (F1), carries the same rule. Co-authored-by: cube <frosimanuel@gmail.com> * D4: setup-deploy reports Render API key presence only /setup-deploy ran 'echo $RENDER_API_KEY | head -c 4', printing the key's first bytes into the transcript. It now prints only 'RENDER_API_KEY: set' or 'RENDER_API_KEY: not set'. Wording follows the #2971 port of #1096; a test also guards against any setup-deploy command echoing a credential variable. Co-authored-by: jacob-wang <jiayuw794@gmail.com> * D5: per-project headed Chromium profiles; never kill a live profile owner (#2492) The headed profile moves from <gstack home>/chromium-profile to <project>/.gstack/chromium-profile (CHROMIUM_PROFILE still wins). killOrphanChromium now kills a SingletonLock PID only when the lock names this host, the process is a Chromium using that profile, and its daemon is gone (reparented to init or a subreaper); a live owner or another host's lock stops browse with the holder named. On first headed use a project profile is seeded from the old shared profile only when nothing holds it: copied without Singleton* files into a 0700 temp dir and renamed into place, with a one-time notice; otherwise it starts fresh and says how to import later. New: browse profiles [list] and browse profiles prune [--days N] (never removes a profile in use or the old shared one). The #2817 headed-startup tests move to the project profile and a real orphan (the old stand-in was a live child of the test). Co-authored-by: miao <miao@macmini-2.local> * F5: installed Claude skills serve the rendered sections, not the checkout copies Probe: a gbrain user render serves ship/SKILL.md from the render dir; its STOP pointers name absolute render paths, but its section index names `sections/<f>.md` relative to the installed skill directory, whose sections/ was linked from the checkout. So the rendered 'Save Results to Brain' blocks (plan reviews, office-hours, ship adversarial) were generated and never read. setup's _link_skill_runtime_assets and gstack-relink (the gbrain-refresh path) now serve a rendered counterpart per file, keep checkout-only assets (manifest.json), return an asset to the checkout once its render is gone, and never touch an entry that is not gstack's links. Co-authored-by: Jonah Berg <jonah.berg.g@gmail.com> * F1: ratchet /ship's parity union-size cap for this lane's content G1's NO_VERSION arms, F1's parent-side plan binding and D3's data line add 2,126 bytes to /ship's skeleton-plus-sections union, taking it from 1.4022x to 1.4135x of the v1.64.1.0 baseline (cap was 1.405). The cap moves to 1.414 with the measurement recorded, as earlier growth did; no other invariant changes. * E5: feed every Codex prompt on stdin instead of argv Prompts went to Codex as one argv string ('codex exec "$(cat file)"'), which fails over 32 KB on Windows (and over 128 KB on Linux, E2BIG) and mangles quotes. Every codex exec site (outside-voice resolver, /codex review custom instructions, challenge, consult new and resume) now runs 'codex exec -' with the prepared prompt file on stdin; challenge and consult write the prompt with a quoted heredoc so it is never shell-parsed. The bash-native timeout watchdog (stock macOS) started the command in the background, which gives it /dev/null as stdin; it now passes the caller's stdin through. The 'byte for byte' dispatch test drops its old argv trailing-LF exception, the consultation and autoplan fixtures follow the stdin form, and the static site test requires stdin at every rendered codex exec. * G6: prettyscreenshot --hide keeps a trailing image path as the output path (#1419) --hide collected every following non-flag argument as a selector, so the output path was hidden instead of written to. A trailing argument that looks like a path or ends in .png/.jpg/.jpeg/.webp is now the output path. Co-authored-by: Jayesh Betala <jayesh.betala7@gmail.com> * INV-3: render-level host conformance and generated-bash lint on every host One render per host checks every fence of every SKILL.md and section file: bash -n with angle-bracket placeholders normalized; the env-var host prelude and its B/D consumers run under env -i and set -u with stub binaries (no empty value, no /bin/, /browse, /design or /gstack- path); and a quote-aware lint (test/helpers/generated-bash-lint.ts) fails on a ~/ word inside double quotes, mktemp without a TMPDIR-respecting template, and cd/rm -r/mv/git -C on a variable the fence never assigned unless written ${VAR:?...} (cd must also handle failure). Negative controls plant each failure. Templated the bare mktemp calls in generator-owned resolvers and the upgrade fence; migrated the two tests that pinned the bare form. * G6: analytics averages quoted duration_s values (#1728) gstack-skill-end writes duration_s as a quoted string, and the per-skill average stripped everything from the leading quote, so every average read 0s. The awk now drops a leading quote before extracting digits. Co-authored-by: Ryan Alberts <ryan.a.alberts@gmail.com> * G6: gstack-ios-qa-daemon --help prints usage instead of starting the daemon (#1932) Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> * G1: tighten /ship's NO_VERSION wording to fit the union-size cap Same behavior in fewer bytes: the NO_VERSION arm, the broken-source exit, the unprefixed-title rule, the Step 20 null and the CHANGELOG skip. The cap ratchet commit measured this wording. * G6: rejected taste preferences gain confidence (#1777) bumpPref computed every entry's confidence from approved_count, so a rejected entry, which never gains approvals, stayed at 0: show ranked rejections by insertion order and the taste-drift warning could not fire from real use. Confidence now uses the entry's own bucket count. Also salvaged by Garry's #2971. Co-authored-by: Jayesh Betala <jayesh.betala7@gmail.com> * E2: Windows /cso build disables only C5105, and a failed MSVC compile is not reported as a missing toolchain The Windows SDK's winbase.h raises C5105, which /W4 /WX promotes to error C2220, so every MSVC build of the native /cso helper failed, and setup mapped every probe failure to 'install Visual Studio 2022 Build Tools'. Both compile lines add /wd5105 (only that diagnostic), and setup keeps the probe's output: a probe that found MSVC but failed to compile reports windows-msvc-compile with the compiler's first error. Not run on Windows: the mapping test stubs powershell.exe; the ps1 assertion runs everywhere. Co-authored-by: AlinValentin7 <AlinValentin7@users.noreply.github.com> * E6: gbrain-refresh hands the detection file to Python on stdin Under Git Bash, a native Windows Python first on PATH cannot open an MSYS /c/... path, so gbrain-refresh always summarized the detection as 'local-status: unknown' and brain-aware blocks were suppressed. bash now opens the file and Python reads stdin; the grep fallback without Python is unchanged. Verified on Linux with a stub Python that refuses path opens; not run on Windows. Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> * G6: context-save computes session duration on Linux (#2704) The duration step parsed 'ps -o lstart=' with BSD-only 'date -jf', so on Linux session_duration_s was always unknown. It now falls back to GNU 'date -d', and never parses an empty start (GNU reads that as midnight). Ported onto the current block instead of #2704's ~/.gstack/sessions mtime, which bypassed the state root. Co-authored-by: Evgenii Lopatin <e75533@gmail.com> * Q2: one-time notice naming the Codex provider and account for outside reviews Outside reviews are on by default and use whatever Codex login exists (#965). The first automatic Codex review on a machine now prints one non-blocking notice: the provider (config.toml model_provider, else openai), where the credential comes from (CODEX_API_KEY, OPENAI_API_KEY, the ChatGPT login or the key saved by codex login; never the value), that reviews stay on, and how to turn them off (gstack-config set codex_reviews disabled). A marker in the gstack state root keeps it to once per machine. Tests that run the generated command now use private state roots. Also shortens the no-sandbox provenance line. * Q1: validate browser output paths the way the kernel resolves them (#2971 salvage) validateOutputPath walks the destination one component at a time, resolving each existing component before the next, so link/../x, a dangling symlink and a dangling symlinked parent can no longer route a write outside the safe directories, while a new multi-level directory inside them is allowed. scrape validates its manifest and every downloaded file, not only --dir. Ported from PR #2971 onto current code. * G6: retro test census counts non-JS suites, skips generated output The census recognized only JS/TS-style names, so pytest and minitest (test_*.py|rb), XCTest (Tests/ dirs, FooTests.swift), Terraform (*.tftest.hcl) and Bats (*.bats) suites counted as zero, and a committed build directory counted as authored code (and its Test/ subdirs as test LOC). is_test now knows those conventions and never treats prose as a test, generated output is skipped in the numstat pass, TEST_FILES_CHANGED uses is_test, and TEST_FILES_TOTAL uses the same filename conventions minus generated paths. Merges #2812, #2809 and #2037's patterns (#2037's retro template edits target a block the metrics bin replaced). Co-authored-by: Jino <dgfrty00@gmail.com> Co-authored-by: snig-17 <santosh.m.tiwari@gmail.com> Co-authored-by: Sholto McNeilage <sholto@nodenarrative.com.au> * Q3: setup stops auto-installing the AskUserQuestion preference hook in Conductor and removes one it added The PreToolUse question-preference hook breaks Conductor's native AskUserQuestion round-trip even when it only defers (#2207), and setup auto-installed it on every Conductor workspace setup. Now a Conductor setup without an explicit opt-in never installs it and removes an existing one (the question-log and AUQ-error hooks stay), printing how to keep it (gstack-config set plan_tune_hooks yes). An explicit opt-in still installs it. gstack-settings-hook remove-source gains --event to scope the removal. Test migration: setup-plan-tune-hooks-noninteractive pinned the retired Conductor auto-opt-in. Co-authored-by: Phil Hie <philhie@outlook.com> * E5: watchdog stdin test feeds the wrapper through a redirect Callers run '_gstack_codex_timeout_wrapper ... codex exec - < file'. Bash keeps stdin for a backgrounded command inside a pipeline but not behind a redirect, so the pipe form passed on the old watchdog; the redirect form reproduces the lost prompt. * F4: type the variant-round fetch stub like the existing stubs The test-typecheck ratchet flagged the zero-argument stub cast to typeof fetch; it now takes the fetch arguments like variants-retry-after's stub. Test behavior unchanged. * A4: import transcripts into per-repository, non-federated gbrain sources `gbrain import` ran without --source-id, so transcripts from every project landed in the shared (often federated) default source. - Each transcript partition imports with --source-id into its originating repository's own source (gstack-transcripts-<repo>, built with constrainSourceId, now exported from lib/gbrain-sources.ts). The source is registered machine-local and --no-federated before its first import, and the registration is cached in state. Partitions import one after another (PGLite is a single writer); a failing source no longer blocks the others, except a timeout, whose checkpoint must survive. - If gbrain cannot register the source, that partition stays local and unstamped, and the run says why (even under --quiet). - --include-unattributed sends repo-less transcripts only to the machine-local gstack-transcripts-unattributed source, keeps them on the machine when the brain is remote (Postgres, Supabase or remote-HTTP), and never writes them to the publishable ~/.gstack/transcripts staging. - A page's recorded source is sticky: pages stamped before this release ("default") keep importing there, so no source gains a duplicate. - The staging dir name carries -src-<id> so a resumed import keeps its source. Test fakes that recorded only the last import now record every import of one run, and answer `gbrain sources`; remote-http scan tests use an attributed transcript, since unattributed ones no longer stage there. Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> * F3: load the taste profile, and report a failed slug lookup The taste-profile probe double-quoted ~/.claude/.../gstack-slug, so the tilde never expanded and /design-consultation and /design-shotgun always printed NO_TASTE_PROFILE. Use quoteSafePath(); stop hiding gstack-slug's stderr and print TASTE_PROFILE_UNAVAILABLE with the fix when the slug cannot be resolved. * B4: DESIGN_READY only when the design binary actually starts DESIGN SETUP (and office-hours' visual exploration) printed DESIGN_READY for any executable binary, so one SIGKILLed at launch by an invalid macOS code signature still passed. Launch it under the portable 10-second deadline (gtimeout, timeout, or perl alarm); otherwise print DESIGN_NOT_AVAILABLE with the exit code, 'timed out after 10s', or on 137 the code-signature hint and 'Fix: cd <checkout> && ./setup'. A success is cached by inode and mtime under the state root so Gatekeeper's first-launch scan is paid once. The probe launches the binary with no arguments (usage, exit 0): the design CLI has no --version command today. * C2: name the runtime root every way a generated preflight may read it in the dist test The generator lane's C1 resolves D from $GSTACK_ROOT instead of $GSTACK_DESIGN, so the test now exports GSTACK_ROOT as well; it still fails on main (no dist in the roots) and passes with and without the C1 change. * A5: brain-sync stage fails loud when the artifacts source indexed zero pages The stage reported "curated artifacts pushed" whenever gstack-brain-sync exited 0, but a git push is not gbrain indexing. After a successful push the stage now reads the maintained artifacts source's page count (GSTACK_BRAIN_SOURCE_ID, else the source whose path is the artifacts worktree or state root): zero pages is ERR with the fix command; a missing source or an unavailable count says so instead of claiming indexing. Co-authored-by: Jimmy Zeisweiss <4261258+jzeisweiss@users.noreply.github.com> * B1: state the effective read-only sandbox in /codex review prose The codex parity invariant requires the skill to state sandbox_mode="read-only"; the command now takes it from _GSTACK_CODEX_SANDBOX, so the prose names the effective default explicitly. * A6: report deleted-worktree gbrain sources; prune only with proof, on request A deleted Conductor worktree leaves its gstack-code-<hash> source behind and gbrain's autopilot fails on it forever. Local absence is not proof a source is dead (machines sharing a federated brain register their own paths, and its pages may be the only copy of uncommitted work), so gstack never removes one automatically. - Each sync counts consecutive misses for gstack code sources whose path is missing, prints "path unavailable" once per source with the review command, and records which repository each synced source belongs to. - New gstack-gbrain-sync --prune-gone-worktrees (with --dry-run, whose plan it always prints first) removes a source only when its path was missing on two consecutive syncs, its parent is readable, recomputing deriveCodeSourceId() for that path on this host yields its id, and the repository's git common dir still exists and no longer lists the worktree. It re-checks the path right before removing (through the #1734-guarded safeSourcesRemove) and logs each removal. * D0: pass the Bun child flags after 'run' so wrappers that rewrite 'bun run' argv keep working bun run --no-env-file --config=/dev/null <script> has the same effect (verified on Bun 1.4.0) and keeps argv[1] === 'run', which the publication-lock fixture's preload injection relies on. The sidebar-tabs source guard now expects the flags in the terminal-agent spawn. * E8: count the BROWSE_EXTENSIONS_DIR persistent launch in the signal-flags guard The new launch site carries handleSIGINT/SIGTERM/SIGHUP false like the other three, so the guard's expected count moves from 3 to 4. * D2: give the memory-leak reproducer's fake request isNavigationRequest() wirePageEvents now runs the navigation guard on every request event; the fake request adapter supplies the method real Playwright requests have. * A7: run only the resolve_symbol_edges dream phase, detected first /sync-gbrain --dream and the --full auto-build ran gbrain's full dream cycle (~35 minutes, LLM phases) when the call graph needs only the resolve_symbol_edges phase. The stage now runs gbrain dream [--source <id>] --phase resolve_symbol_edges when the installed gbrain accepts that phase (detected by behavior: gbrain validates --phase before printing --help). Otherwise it skips the forced dream and says the full cycle costs about 35 minutes and how to run it, instead of starting it. Co-authored-by: liutiming <timingliu@outlook.com> * D0: keep the dotenv isolation test runnable on the Windows lane The compiled probe gets its .exe name on Windows, and the terminal-agent case (owner identity comes from ps) is skipped there. * B9: read cycle_freshness from a doctor run that includes it /sync-gbrain Step 3.5 and the --full auto-dream decision read CYCLE from gbrain doctor --json --fast, which skips the DB checks that carry cycle_freshness, so the answer was always "unknown". Both now read gbrain doctor --json --scope=brain (DB checks, no skill-file walk), parse its JSON even when doctor exits 1 for a failing check, and match the issue that names this source: "never completed" is never, "last cycled Nh ago" is completed. When the installed gbrain does not report the check, they say "unknown: installed gbrain does not expose cycle_freshness" (lib readCycleStatus carries the reason to the --full SKIP row). The source-probe fixture that pinned the --fast call and a made-up message format now answers --scope=brain with gbrain's real message. * B4: compact the launch probe (cache keyed on inode, valid while newer than the binary) Keeps design-consultation inside its parity size budget. * C4: migrate the ship docs-dispatch test to the carved external ship On Codex and Factory the documentation dispatch now lives in sections/documentation.md behind a pointer relative to the installed skill, like Claude; the test pinned the old inlined layout. * B5: learnings callers report a failed search instead of reading as empty Every generated gstack-learnings-search caller ended in 2>/dev/null || true, so a missing bun or a failed search looked like 'nothing recorded'. Callers now keep stdout, capture stderr and the exit status, and print 'LEARNINGS: unavailable (<first stderr line>)' on failure (resolver callers in learnings.ts and review-army.ts, plus investigate, qa and ship's adversarial section). * F1: ship and review count unverified/unavailable outside reviews as missing coverage INV-1 consumers in this lane's templates: /ship's readiness note, its PR body Pre-Landing Review section and /review's completion rule now say an outside review reporting unverified or unavailable is missing coverage, never a pass. The PR body's Plan Completion section prints F1's not-run line instead of the old 'No plan file detected.' The /ship union-size cap moves to 1.415 for these lines (measured 1.4148). * A8: rename the artifacts repo with OWNER/REPO, rewrite the remote only after it The v1.27.0.0 migration passed a bare repo name to gh repo rename / gh repo edit, which always fails (gh wants [HOST/]OWNER/REPO), then step 2 rewrote ~/.gstack-artifacts-remote.txt to the never-renamed repository. - Qualify both names with the owner from the recorded remote URL (else the signed-in gh user) and surface gh's error. - Step 2 moves the remote file but rewrites the URL only once the rename is confirmed; a later successful retry rewrites it then. - A pending journal now counts as state to migrate, so the #1383 retry actually re-runs step 1 (it used to see no legacy file and declare the migration done). - The v1.91.17.0 migration (placeholder name) repairs installs already hit: when the remote names gstack-artifacts-<user> that does not exist but gstack-brain-<user> does, it restores the old URL in the remote file and ~/.gstack's origin and says so. It only calls gh for such a remote and never fails the upgrade. Co-authored-by: Brandon Pugsley <aibpaem@gmail.com> * B8: read gbrain JSON with bun instead of jq in install and wireup gstack-gbrain-install read package.json with jq but never checked for it, so on a machine without jq a successful install failed with "cannot read version". gstack-gbrain-source-wireup hard-failed without jq ("brew install jq" is not actionable on git-bash), so the wireup was skipped on every run. Both now read the JSON with bun, which they already require. Co-authored-by: benjamin beres <benjamin.beres@bienpreter.com> * B4: shorter probe messages (design-consultation parity budget) * B5: shorter capture variable names (parity budget) * E7: find the Aside CLI in ~/.local/bin when it is not on PATH (#2902) The installer puts aside in ~/.local/bin, which is not on the default macOS login PATH, so the probe said NEEDS_ASIDE ('download Aside') for a healthy install. The probe now falls back to ~/.local/bin/aside, drives the resolved binary, and prints READY: <path> when it is off PATH so later blocks run that path. Ported from PR #2965 onto the current _gs_d probe. Co-authored-by: Som Samantray <som.samantray@gmail.com> * A2: key the dot-slug catalog entry on the message users actually saw * E1: cmd.exe-safe investigate freeze hook On Windows Claude Code runs hook commands through cmd.exe, which cannot parse the single-quoted bash -c argument, so /investigate's scope-lock hook never ran. Use one double-quoted bash -c script that names the hook path directly (no S= variable: POSIX sh would expand $S to empty and skip the hook). * F4: design resolver reads per-round variant paths $D variants now writes $_DESIGN_DIR/round-<N>/variant-*.png and prints JSON with roundDir, manifest and paths. The mockup flow and comparison-board loops build the board from those printed paths instead of hardcoded variant-{A,B,C}.png, and approved.json records approved_path and round so a round-1 choice made after round 2 still resolves. * E4: generators tolerate Bun's spurious EEXIST on Windows (#2329) Bun on Windows throws EEXIST from a recursive mkdir of an existing directory, so gen:skill-docs and the llms.txt writer crashed once a second file landed in a directory. Both now use lib/fs-utils' mkdirpSync (ported from PR #2329 onto the existing helper instead of a new scripts/fs-utils.ts). Co-authored-by: Vaughan <vaughan@cairnsitguy.com.au> * C7: keep the required-mode hook's exit-2 deny Copilot CLI treats a preToolUse exit 2 as deny (its hooks reference), and #2413 established exit 2 plus hookSpecificOutput as the shape Claude Code enforces, so the deny output stays as on main; only the matcher and the in-place upgrade of older 'Skill' entries change. * C3: guard Kiro's router copies on a present source; give the QA pointer adapter the copy and dist helpers A copy fails where a symlink dangled, so the Kiro router and upgrade copies are written only when their render exists (as every other runtime root already does). test/qa-lazy-sections' extracted Kiro installer now supplies _link_runtime_dists (C2) and the SKILL.md copy helpers (C3). * Q3: office-hours asks open-ended questions in a Q<N> prose form (#2719) In Conductor, and whenever AskUserQuestion fails, prose is the only channel, and the prose fallback defined only the D<N> decision brief. Phase 2A/2B diagnostic questions have no option set, so the model dropped the phase. Office-hours' preamble now defines the Q<N> open-question form (question, why I'm asking, what a strong answer sounds like) and routes free-text replies to it; the diagnostic sections name it. Scoped to office-hours so every other skill's preamble is unchanged. Ported from PR #2729. Co-authored-by: Bharadwaj Mittapelli <bharadwajmittapelli@gmail.com> * A2: render db-unreachable like timeout gbrain's new db-unreachable status is a transient network failure; a setup run while offline must not strip brain-aware blocks from every skill. * A2: treat db-unreachable like timeout in gbrain-refresh The memory lane's new db-unreachable status (network or DNS failure, config intact) is transient, so gbrain-refresh keeps rendering brain-aware blocks for it exactly as it does for timeout instead of suppressing them. * C8: a render given disabled skills drops them from the router (#3018) The root router named every skill and ends 'When in doubt, invoke the skill', so it routed to skills the user disabled. gen-skill-docs takes --disabled-skills (runGeneration disabledSkills): the router drops their routing rules, keeps the other side of an 'A or B' rule, and names them as disabled. The router and gstack-upgrade cannot be disabled. With no disabled skills the render is byte-identical. * G2/G3/INV-1: skill-start former-bucket notice and visible learnings failure; TMPDIR mktemp in gbrain bins; best-effort reasons * C9: Step 2 falls back to the install registry's source checkout (DX-7) Codex users who installed from the README's ~/gstack clone got 'ERROR: gstack not found' from /gstack-upgrade: Step 2 only looked under host skill directories. It now falls back to the source checkout the install registry recorded for the current host. * B4: design --version prints the version, and the probe uses it The design CLI treated --version as an unknown command (exit 1), so the probe launched the binary with no arguments instead. Add --version (prints the gstack version, exit 0, before auth or network) and probe with it, so messages read '<path> --version exited <code>' as planned. * C4: no external render points at a section file that is not installed design-html's detector install offer and plan-eng-review's self-check pointed at $GSTACK_ROOT/<skill>/sections/<file> on every external host, where neither skill is carved and no section tree exists. design-html now inlines the offer off Claude, and a skill's pointer to its own {{SECTION}}-inlined section names the inlined text. * C1: skillify finds the browse SDK under its own host's install root The template hardcoded ~/.claude/skills/gstack for the canonical browse-client.ts, so external hosts read another host's install or nothing. A new INSTALLED_ROOT resolver (sibling of INSTALLED_BIN_DIR) names the host's global root, or the per-install root when setup renders for one. * C4: rewrite section refs on hosts whose root stays a literal ~/<globalRoot> Kiro, OpenCode, Cursor and Copilot render ~/.claude/skills/gstack as their own literal root, not $GSTACK_ROOT, so plan-eng-review's self-check pointer escaped the rewrite (found by the setup lane's installed-pointer check). * Integrator: outside-voice fences use the shared runtime prelude; TMPDIR mktemp and per-round variant paths in design templates * Integrator: C8 router omits disabled skills on every render (relink renders the served Claude router in place); Windows-safe E4/E6 entries; investigate hooks under the cmd.exe check; browse_extension_id reader; order-insensitive test typecheck identities tsc printed one unchanged ceo-mode-option diagnostic's object members in a different order (content?/isError?) once lib/fs-utils joined the generator's import graph; identities now sort object-type members, and old baselines normalize on read. * Q1 (#2971 salvage): bound iOS proxy responses One wall-clock deadline covers headers and body (stalled and dribbling bodies time out), the whole body is capped at 64 MiB (upstream_response_too_large), every terminal path clears its timer, removes listeners and destroys the request, and the daemon threads the deadline through both listeners. Ambiguous mutations are still never replayed after a timeout or abort. Ported from #2971's proxy-transport work plus the size bound. * I1: grade the AUQ matrix's (recommended) marker on option labels, the field the auto-decide hook reads The decision-brief format puts a Pros / cons block with 'A) ... (recommended)' in the question text and the same marker on the option label. The matrix counted both and failed all six skills in periodic run 37151477069 with 'expected exactly one (recommended) option, found 2'. The question-preference hook reads labels only, so the gate now checks the native question: a Recommendation line in the question text and exactly one label ending in (recommended). Fixtures move to the native question shape and add the real spec capture plus a marker-only-in-text negative control. * Q1: type the proxy-transport test doubles (typecheck:test ratchet) * INV-1: gate outcome rows and troubleshooting sections for every lane's new messages * Release v1.91.19.0 (next free patch in the queue: #3022 holds 1.91.17.0, #3023 holds 1.91.18.0); migration renamed; engines.bun >=1.4.0 to match the CI pin (#2815) * v1.91.19.0 docs and metadata: CHANGELOG, TODOS, README troubleshooting/upgrade/type --selector, BROWSER.md type and profiles, AGENTS.md conventions, CONTRIBUTING #2949, touchfiles for the wave's new lib/bin files * I1: give hermetic Codex runs a temp dir beside CODEX_HOME so Codex can create its PATH aliases Codex 0.160 refuses to create its helper aliases (codex-linux-sandbox among them) when CODEX_HOME sits under its temp dir, and the runner put the hermetic home under the shard TMPDIR. Periodic run 37151477069 shows the warning and 'Unable to spawn codex-linux-sandbox ... No viable candidates found in PATH' for every exec_command in codex-sol-scope-termination. * I1: Codex evals fail a review whose commands never ran and stop reading tool errors as skill-load problems validateCodexReview accepted codex-review-findings in periodic run 37151477069 although its only command failed in bwrap ('I couldn't run the review'); it now requires execution evidence from classifyOutsideReview (B1's INV-1 classifier). codex-sol-scope-termination failed on 'apply_patch verification failed: invalid hunk' because the skill-load check matched any 'invalid'; it now matches Codex 0.160's skill-loader diagnostics only. * I1: run paid eval containers with seccomp unconfined so Codex's sandbox can start Docker's default seccomp profile blocks the user namespaces bubblewrap needs, so every Codex command in a paid eval failed with 'bwrap: No permissions to create a new namespace' (codex-review-findings, run 37151477069). Reproduced in the CI image locally: default profile fails, seccomp=unconfined runs the command. The runner stays non-root. * I1: shared-libs plan actor reads one exclusion grammar for scope lists Periodic run 37151477069 refused 'Existing-caller migration and helper hardening stay out of scope' as a hardening proposal: the actor had two partial exclusion rules (unanchored is/are/remains outside, and an anchored list that only stays unchanged), the third refusal at this boundary after 36633323521 and 36776104571. One anchored predicate now covers a bare list of scope nouns (hyphenated compounds included) with is/are/stays/remains and unchanged/untouched/excluded/out of scope/outside/not part; any other verb in the clause is still read as a proposal. Adds the real capture and two adversarial controls. * Parity caps: measured v1.91.19.0 union/skeleton sizes for ship, review, design-consultation, office-hours and design-shotgun, with the reason beside each * Regenerate generated output for v1.91.19.0 (gen:skill-docs --host all, goldens, llms.txt, context-budget fixture) * I1: plan-eng-review writes the Test Plan Artifact even while test choices are pending Periodic run 37151477069 (plan-eng-review-artifact): a non-interactive review recorded every Test review choice as pending and never wrote the QA test plan, because Step 5 said to write the artifact 'when these test and eval choices are resolved' while the artifact section says to list unresolved choices as pending. Both now say to write it after the decision gate, with unanswered choices under Pending Decisions. The pinned sentence becomes a meaning-level check. * I3: plan-eng-review states when findings first enter the report and that the scope menu's recommendation needs no repo probe Judge rationales for 'plan-eng-review/SKILL.md sections' (actionability 3.67 in periodic run 37151477069, clarity 3 in 36385945043 and 35567915613) cite two concrete tensions: 'do not add findings before Scope Challenge C' versus the ledger saves C itself performs, and 'Recommendation: A when a branch diff exists' versus 'do not probe for session state'. The write policy now says findings first enter through Scope Challenge C's ledger saves; the selector says its Recommendation line advises the user and needs no repository inspection. The pinned sentence becomes the new meaning-level regex. * I3: plan-ceo-review defines the closing sequence's cleanup step Two of three judge samples for 'plan-ceo-review/SKILL.md modes' in periodic run 37151477069 (actionability at the 4.0 gate) flag 'perform permitted cleanup' as never defined. The only cleanup the skill has is archiving old CEO plans, which needs approval; step 5 now says so. * I3: /qa-only's value-card rule names only its own card location qa-only rendered the /qa line 'Put it in the 8e.5 record (/qa) or under each proposed test (/qa-only)'; judge samples for 'qa-only/SKILL.md workflow' (actionability at the 4.0 gate in run 37151477069, 3.67 in 36907895813) flag the /qa-only-irrelevant 8e.5 record. The qa-mode bar now renders the invoking skill's location; both skills get shorter. * test-of-test ratchet: a repo script or setup read by path counts as product code codex-invocation-sites (runs scripts/gen-skill-docs.ts), gstack-config-gbrain-refresh (copies bin/<name> through a variable) and setup-cso-windows-probe (reads setup and scripts/build-cso-windows.ps1) exercise product code the detector could not see. * I1: DX floor actor strips the editor hint from the footer line only (from #2971) The custom-field reply normalization removed 'ctrl+g to edit in <editor>' from every line, so question text that resembled the menu footer could be rewritten and still authenticate. Only the last non-empty line is the footer. Ported from Garry's #2971 with its negative control. * I4: attribute every census testcase; deselections never count as SKIPPED The paid report counted every skipped JUnit testcase as SKIPPED and every testcase whose name was not a registry id as unattributed. On the 2026-10-03 census (run 37151477069) that was SKIPPED 82 / unattributed 72 (periodic) and SKIPPED 282 / unattributed 24 (gate), and every case-shard sibling also wrote a skipped history record that reset its pass-rate series. - attributeJUnitCase: a name is its registry id or CASE_TEST_NAMES label, else the only id its file registers, else the keyless file itself. Skipped other-tier cases, case-shard siblings, isolated or unselected cases and Bun's '(unnamed)' hook placeholders are zero-credit deselections with a reason and write no history record. Anything else is listed by name as unattributed. - One result and one history record per case per shard; a genuinely skipped selected case counts as SKIPPED and is listed with its reason, also in report-summary.md and collector-outcomes.json. SKIPPED panels show their reason and skipped trials are no longer called assertions. - CASE_TEST_NAMES labels for the three /cso cases. - tierSkipReason: a computed registration whose lane-tier ids are all registered to other test files is excluded at plan time, so the four gate and two periodic files that ran with every test skipped no longer take a runner. Census rerun: periodic SKIPPED 0 / unattributed 0, gate SKIPPED 0 / unattributed 0; verdicts unchanged. * I5: split the deploy file per case; guard PR-lane shard walls at 600 s test/skill-e2e-deploy.test.ts ran 668 s as one gate shard in the 2026-10-03 census (six sequential cases, 51-201 s each). It is now case-sharded, so each case is its own shard with its own (smaller) supervision budget; no timeout or budget was raised. - CASE_SHARDED_FILES gains the deploy file (names already equal ids). - Gate duration seeds for the six case keys from the census JUnit walls, so packing stays at 12 gate-census slices (CI cap 230m -> 200m). - shardDurationViolations + PR_LANE_SHARD_LIMIT_MS: a free test fails when a planned gate shard's recorded wall (committed seed, and the last gate census in test/fixtures/paid-report-census/gate-census-durations.json) exceeds 600 s. - Tests that pinned deploy as one shard follow the split (cookie planner keys, reuse fixture key, release/PR floors). * TODOS: /review fingerprint under Codex's read-only .git sandbox (found by the live Codex check) * Free-suite repairs from the first integrated Ubicloud run Product: - Outside-voice fences carry runtimeRootPrelude(ctx) explicitly again, so the resolver output runs on its own (the post-render pass skips fences that already assign GSTACK_ROOT); outsideVoiceRuntime stays deleted. - D5: an explicit CHROMIUM_PROFILE (gbrowser's per-workspace gbd) is never seeded from the shared profile or registered for `browse profiles prune`; this was also the free suite's write to the real ~/.gstack/browse-profiles.json. - gstack-ios-qa-daemon: BASH_COMPAT=50 for its >512-byte --help heredoc. Tests migrated to the wave's deliberate contract changes: - B1 heading admits `unverified`; sandbox flag is ${_GSTACK_CODEX_SANDBOX:?}; verdict-form output (VERDICT/FINDINGS/REASON); fake codex follows the preflight + stdin + -o form; Q2 notice marked shown in a private state root. - C1: an exported root needs bin/ and lib/; a lone GSTACK_BIN is not honored. - C4: plan-ceo-review is carved on every host (carrier, discovery and methodology fixtures read sections/ when the render has it). - E7: READY prints the resolved Aside path. - INV-3: the cookie-approval reconstruction reverses the bun-install mktemp template. - F1's `Plan:` binding read gets a reasoned tracker-guard exemption. - Browse stubs stop their server instead of scheduling process.exit; the headless launch check only counts browsers its own launch started. * Regenerate generated output after the outside-voice prelude repair * I1: revert the seccomp-only eval container option; it does not let Codex's sandbox start Run 37158847998 on this branch and a stock Ubicloud VM both show that seccomp=unconfined only moves bubblewrap's failure from 'No permissions to create a new namespace' to 'Failed to make / slave: Permission denied'. Codex's sandbox started only with the host sysctl kernel.apparmor_restrict_unprivileged_userns=0 plus seccomp and AppArmor unconfined on the container, which a container job cannot set from its steps. The relaxation bought nothing, so it goes; the fix is filed as a TODO for the integrator. This reverts commitec97187. * I1: a Codex review eval needs a completed command, not just review-shaped text Run 37158847998 recorded codex-review-findings as passed again: the agent wrote 'I couldn't run the review because the execution sandbox failed (bwrap: Failed to make / slave)', the failure never reached stderr or a command event, and no phrase rule matched. validateCodexReview now also requires at least one command_execution event that completed with exit 0. Fixtures gain that event; the CI output is a new negative control. * Evals lane follow-ups: close the skillify gate TODO (I2 green), file its open reds, CHANGELOG lines; refresh measured parity annotations * Regenerate generated output after merging wave/evals * D2: bound the navigation guard's tab reset (commit, 5 s) Ubicloud run 3: a click to a link-local link reported the block, but the reset's page.goto('about:blank') waited the full 30 s default navigation timeout, so the command hung past the test deadline and the stale block leaked into the next command. The reset now waits for commit only, with a 5 s bound. * I1: run Codex paid slices on a host-run job where bubblewrap can start Docker's default seccomp/AppArmor profiles and the Ubuntu 24.04 host's kernel.apparmor_restrict_unprivileged_userns=1 keep Codex's bubblewrap sandbox from starting in a job container, so every Codex command failed. The planner now packs Codex shards into their own slices (plan.codexSlices, validated against the entries); evals-periodic runs those slices on a new host-run job that lifts the host restriction and runs the same CI image with seccomp and AppArmor unconfined. Artifact names match eval-slices, so the report reconciles one manifest. Gate plans have no Codex slices. * I1: the /codex review eval passes only when Codex executed Run 37158847998 recorded codex-review as passed while the sandbox preflight said unavailable and no review ran: the case only checked the session exit and an optional output file. It now requires the shared validator's executed verdict (findings, clean or unverified) in the codex command's own output and a written, non-fail-closed GATE. * TODOS: characterize the free-suite timing reds from Ubicloud run 4 * I1: QA evidence names the probes to rerun after an input change, before the one-shot materialize ship-exploratory-late-input returned inconclusive in gate census 37158847998: the fixture locale changed after captures 001 and 002, the agent reran only input 9, and materialize (which publishes once) recorded capture 001 as open. Its next line then said to rerun what is open, which the published verdict no longer allowed. Every capture now lists the commands that observed an older input snapshot and have no rerun on current inputs (revalidate, plus the next line), and materialize's next line says the verdict is final. * I1: the eng batching counter reads records under "Section N — <review> review" headings plan-eng-multi-finding-batching failed in 6 of 12 runs with reviewCount 0-1 (or a timeout with every ask counted as Step 0). In run 37158847998 the agent asked 11 separate questions, each with a saved ledger record, but filed records R2-R10 under '## Section 1 — Architecture review'. The counter strips 'Section 1' but not the dash separator, so it rejected every record outside the Decision ledger and the floor never filled. Quoted, historical and unrelated headings still do not count. * I1: plan-eng-review-artifact names its report file without making it the only output Runs 37151477069 and 37158847998 wrote the whole review, including the GSTACK REVIEW REPORT, into review-output.md and never ran the Test Plan Artifact block. #3017 made the artifact a hard assertion and dropped the prompt's explicit artifact instruction, leaving 'Write your review to review-output.md' as the one named destination. The prompt now names that file as the selected report file and says every other output keeps the destination the skill gives it, without naming the artifact. * browse: close returns when Chromium exits; Xvfb cleanup does not wait on its own zombie BrowserManager.close() detached every 'disconnected' listener before browser.close(). That also removed Playwright's internal listener that resolves close(), so every launched-mode close waited the full 5 s race timeout (measured 5,000 ms on origin/main and on this branch; raw Playwright closes in ~50 ms), and Playwright 1.62 has no Browser.process() for the SIGKILL fallback. The daemon's SIGINT shutdown therefore took over 5 s, which reddened the CLI lifecycle test under load (1 of 8 on main) and left browsers alive at shard cleanup. gstack now detaches only its own listeners: close takes ~50 ms. cleanupXvfb treated a terminated Xvfb child as alive until our own blocked event loop reaped it, so every close waited a full second then sent SIGKILL to a zombie. A zombie now counts as exited; the display test counts only running Xvfb processes. * I1: office-hours-brain-writeback reads the long skill with Read and skips the preamble 4 of 18 trials ended error_max_turns at the unchanged 12-turn budget with a gbrain search but no gbrain put. The failing trial in 37151477069 read the 1260-line rendered SKILL.md with cat, got truncated output, spent four more turns re-reading it with sed, then ran the preamble's skill-start block. The passing trials skipped the preamble. The prompt now says to read the skill with the Read tool and skip the preamble, onboarding, telemetry and contributor sections, as the plan-eng fixtures already do. Budget unchanged. * CI: no credential-shaped URLs in added lines (quality scan); release floor measured on the merged tree The quality job's credential scan of the PR diff flagged 7 db.url_with_password lines: three docs lines now describe the login instead of spelling a URL, and three tests build the URL from parts. paid-retry-supervision's periodic release floor is the merged tree's measured 35,711 (main's value; the evals lane's 33,821 predates main's new paid files). * qa-evidence: hold the stdout-pipe cases on a FIFO, not on a paused child stream Ubicloud run 4 finished the blocked-forever case in 71 ms (it should take the 5 s write deadline): the 2 MB write was not held, because saturation depended on how the runtime buffers a paused child pipe. On POSIX the child's stdout is now a FIFO the test reads only when it chooses to, so the kernel holds the write; Windows keeps the paused pipe. * TODOS/CHANGELOG: the three run-4 timing reds are fixed at their causes (daemon close latency, FIFO-held pipe tests) * I1: a QA helper call bun could not load is checked at the installed path, not failed outright review-exploratory-small-cli failed gate census 37162480720 with 'command outside declared caller observation interface': the actor dropped one character from the shard directory in a gstack-qa-evidence capture path, bun refused to load it ('Module not found', exit 1), and the next call ran the same capture correctly. A failed call whose only output is bun's refusal for a gstack-qa-evidence/deadline path is now held to the interface as the same command at the installed helper path, so a compound or undeclared command still fails. * I1: plan-eng-review's final self-check names its three outputs The fixture-prompt repair did not hold: in 37162480720 plan-eng-review-artifact again wrote one review-output.md, ran the skeleton's read-back gate on it and stopped without the QA Test Plan artifact (no gstack-paths/gstack-slug block). The transcript shows the agent's last check before finishing is the skeleton's Section self-check, which said only 'and outputs'. It now names the saved report, the QA Test Plan artifact and the Review Log, each at its own destination or presented as not persisted (the existing write policy). * I1: revert the plan-eng-review self-check wording; it did not change the artifact outcome A local run of plan-eng-review-artifact in the CI image (Claude Code 2.1.284) ate2f65ab22still wrote one review-output.md in 8 turns and ran no Bash, so the QA Test Plan artifact block never ran. Keep the skill text unchanged; the case is deferred with its evidence. * I1: seed the Codex files' periodic durations from their first executing run Every Codex file recorded ~1 s because its commands could not start in the container (codex-e2e*, outside-voice, workflow). Run 37162480720 executed them on the host-run job; its slice 24 walls replace the placeholders so the planner packs the Codex slice by real work (7 files, ~4.9 min estimated). * Refresh scripts/free-test-durations.json (bun run test:ubicloud --record-durations; covers the wave's new test files) * evals2 follow-ups: outside-voice-evidence helper in five case touchfiles; close the evals lane's Codex-CI, late-input, batching and brain-writeback TODOs (run 37162480720); CHANGELOG lines * fix(browse): take the next free X display when a concurrent allocator wins the race Pick-then-spawn let two allocators (parallel daemons or test shards) choose the same free display; the losing Xvfb exited and the handoff failed (run 6, handoff 'failure before launch'). spawnFreeXvfb now treats a display taken during startup as a collision and moves on. Regression test forces the race with a delayed Xvfb wrapper (pre-fix: fails; two-process repro 13/15 -> 0/15). * test(ci): read the host-run Codex eval job's docker env, mounts and uploads eval-codex-slices runs on the host and passes its executor env through docker -e flags and HOME through a $RUNNER_TEMP/eval-home mount. The evidence and coordination tests now map those back to the container view instead of assuming job.container, and the job's native-capture upload carries the qa-callers paths like the other executors. * I1: plan-eng-review-artifact runs the real interactive review in a PTY, replacing the composing fixture The claude -p case told the model to skip the preamble and questions and go straight to the review. In 37151477069, 37158847998, 37162480720 and a local CI-image run it wrote the whole review into one file without executing the workflow, so it measured composing, not /plan-eng-review. The interactive batching PTY run in 37158847998 did write the artifact during Test review, so there is no product defect. New test/skill-e2e-plan-eng-artifact.test.ts keeps the case id, tier (periodic), kind (rule) and budget (CAPTURE_LONG_MS): it drives the real /plan-eng-review over the same dashboard plan and sources, answers each question with its first option, stops once the QA test plan exists or the report completes, and asserts the same requirement (exactly one new eng-review test plan in the run's own state root, about the dashboard change). The composing case is retired from skill-e2e-plan.test.ts in this commit. runPlanSkillCounting reports engTestPlans (read before fixture cleanup) and passes them to isCollectionComplete. One paid run in the CI image (Claude Code 2.1.284): pass, collection_complete at 498 s with one test plan. The new file takes its own periodic runner (seeded at 498 s), so the periodic census plans 26 slices: the planner's --max-parallel and both executors' caps move from 25 to 26 so every slice still starts at once; the paid-file census pin moves from 106 to 107. * evals2 final: plan-eng-review-artifact touchfiles point at its PTY file and helpers; plan file wall drops the moved case Applies the lane's touchfiles-data patch (its outside-voice-evidence hunks were already in9a96fb3). skill-e2e-plan.test.ts lost one 600 s case to skill-e2e-plan-eng-artifact.test.ts, so its periodic wall is two 900 s + two 600 s (3,000,000 + reserve) and its timeout pin lists the remaining cases. * TODOS/CHANGELOG: close plan-eng-review-artifact (real PTY review, 498 s pass) * test: plan-eng-review-artifact is a PTY runner consumer (selected by claude-pty-runner and pty-screen changes) * test(windows): spawn cmd.exe hooks verbatim, use a .invalid UNC host, and match CRLF fixtures and native separators windows-free-tests reds on364f0f3, each verified on windows-latest (46 pass, 1 POSIX-only skip, 0 fail): - investigate-freeze-path cmd.exe cases: Bun MSVC-quoted the command line cmd.exe parses, so cmd saw \"bash and failed; Node's shell spawn (what Claude Code uses) passes it verbatim. - investigate-freeze-path UNC cases: a single-label host made each filesystem probe wait on network name resolution (cd 11 s, -L 2.8 s); a reserved .invalid host fails at once (55-80 ms). - paid-report-census: the junit fixture is CRLF in a Windows checkout. - codex-invocation-sites: rendered paths use backslashes on Windows. * eval(plan-design-with-ui): the Step 0D focus title may spell the seven dimensions as a word Run 37166586458 slice 5 (dfaf154) asked 'D2 - Review all seven design dimensions, or focus on specific areas?' and reached its review finding, but the focus predicate only accepted the numeral, so the case failed with no acknowledged Design focus. Both recognizers accept 7 or seven; the byte-exact capture is a free regression, and engineering/other-count/ no-'design' titles stay rejected. * eval(ship-docsync): git check-ignore is a declared read in the docs observation interface Run 37166586458 slice 2 (dfaf154): ship-docsync-completion's parent ran 'git check-ignore -v .qa-state' to verify the prompt's claim that the observer probe is Git-excluded. The read was outside the closed Git list, so the run's DOC_PATH replacement was unprovable and failed. check-ignore cannot write; it joins the list and the interface prose. Free regression replays the captured command against the fixture and keeps composition and redirects rejected. * fix(browse): read Xvfb's own startup error to classify a lost display race Run 10 (1ffd38e) failed handoff's concurrent promotion once: 'Xvfb on :100 exited during startup (code 1)' with the display free again by the exit check, so a holder that won the lock and closed within the 100 ms poll looked like a broken install. Startup stderr now goes to a private file; 'Server is already active for display' is a collision and moves on, and any other startup exit carries Xvfb's fatal line. Regression: a stub that reports the collision for one display and exits leaves spawnFreeXvfb on the next display (pre-fix: the misleading install error). * test: merged-main fixtures (E6 copies the Claude render library gstack-config now sources; the census fixture's plan-mode-no-op budget follows its v1.91.18.0 policy) * qa: materialize runs once per PROBE_DIR, and qa-only says §1 reuses the REPORT_DIR it resolved Run 37170610789 slice 3 (feaa28d): the qa-only workflow judge scored clarity 3.33 and named two ambiguities: §4 said materialize runs once per report root while mixed runs materialize each surface's PROBE_DIR, and SKILL.md deferred exploratory §1 without saying §1 reuses the REPORT_DIR resolved under Prepare Report Artifacts. A local run of the same judge on the repaired text: clarity 3.67, completeness 4.33, actionability 4 (pass); all three samples call the workflow internally consistent. * Regenerate generated output for the qa materialize/REPORT_DIR wording * eval: three detector defects from run 37170610789 (feaa28d) - safety-codex-consult-embed: the fake codex recorded argv only, but every codex exec takes its prompt on stdin since E5; it records stdin for the '-' form and the check reads both. - ship-docsync-missing-asset: 'ls ... 2>&1' on a read-only command is the same harmless stderr wrapper as '2>/dev/null'; stdout redirects, pipes and script calls stay rejected. - qa-functional-webhook-fix: a pre-repair capture whose command reran on the repaired inputs may be classified superseded (exploratory §4); the evidence check accepts that and still rejects superseded on a post-repair capture or one never rerun. Each free regression fails without its repair. * fix(test:free): a browser record removed mid-read by a shutting-down daemon is absent Run 13 (feaa28d) needed the flaky-retry for 'owned detached browser settlement > cancel': 2 unattributed failures instead of 1. A 16-vCPU stress run reproduced it once in 64 with the cause: 'browser ownership unavailable'. On SIGINT the cancelled daemon removes its state file; record() saw it exist, then lstat/read hit ENOENT, which failure() reports as an ownership error. ENOENT there now means absent. New mode cancel-vanishing-record forces the removal between the check and the read (pre-fix: 2 unattributed, deterministic; now 1). * test(windows): the 2>&1 wrapper regression uses relative reads (no native paths in literal commands) * test(plan-seed): wait for the fixture CLI's slash event instead of a fixed 50 ms Run 14 (b2f729b) needed the flaky-retry for 'startup-placeholder': the events log had paste/enter/end_turn but not yet the slash 50 ms after it was sent. The check now polls (bounded 5 s); the deadline assertion measures when submission settled, which is what it bounds. One test typecheck diagnostic goes away with the typed reader. * eval(plan-design-with-ui): recognize the Step 0D focus question by meaning, not one title spelling Theb2f729bgate run titled it "I've rated this plan 5/10 on design completeness. Review all 7 dimensions, or focus on specific areas?", the third native wording to miss a spelling-pinned regex. The native form is now a class: a title that offers all seven (numeral or word) dimensions or passes versus a focus and names design. Captured titles from 37166586458 and this run pass; other counts, engineering passes, titles without design or without a focus stay rejected. * eval(shared-libs-read-only): curl answers the provider rate-limit probe with the same 404 as gh Gate census 37174266054 slice 9: the model probed API availability with curl -o /dev/null https://api.github.com/rate_limit (GET, discarded). The gh transport already answers rate_limit as an unknown endpoint (404); the curl transport refused it, and the refusal counted as a read-only violation. curl now gets the same 404. POST, file output, other repos and raw hosts stay refused. Paid rerun (gate, 1 trial): PASS in 177 s. * eval(shared-libs-plan-callers): an exclusion may combine states (unchanged/out of scope) Periodic census 37174266054: every option ended 'Scheduler semantics, existing callers and helper hardening stay unchanged/out of scope', which the actor read as proposing hardening. Exclusion states may now be joined by '/', 'or' or 'and'; the captured question is answered, and clauses that pair a state with a real change ('stay unchanged/harden the helper') are still refused. Paid rerun (periodic, 1 trial): PASS in 296 s. * paid: --case plan-eng-review-artifact selects its PTY test by its registered name A local --case run matched 0 tests (no spend): the PTY test's Bun name is not its case id, and CASE_TEST_NAMES had no label for it. * eval(plan-eng-review-artifact): periodic checkpoint resumes at Test review; the fresh full run moves to marathon A fresh interactive /plan-eng-review cannot reach Test review inside the 600 s periodic budget: startup plus one-at-a-time Scope Challenge questions took 595 s with six questions and no Test review in census 37174266054 and in a local run (one earlier fresh run passed at 498 s). Per the eval-speed policy, the long run moves to the non-blocking marathon lane and the blocking case becomes a focused checkpoint. - plan-eng-review-artifact (periodic, 600 s unchanged): resumes the real interactive skill from a saved review record: the plan, the Scope Challenge result, Sections 1-2 complete, and a Decision ledger rebuilt from that census run's six actual native questions and answers (test/fixtures/plan-eng-artifact-resume-37174266054.json). The live skill runs Section 3, Test review. Assertion unchanged: exactly one new eng-review test plan in the run's own state root, about the dashboard change. Paid trial (local, Claude Code 2.1.284): PASS, 364 s. - plan-eng-review-artifact-full (marathon, PTY_LONG_MS): the fresh start-to-finish run with the same assertion; budget measured beside it. - test/plan-eng-resume.test.ts proves the rebuilt record is a valid resume point and rejects missing, unanswered, edited or duplicated records. - Registry: touchfiles, tiers, kinds, test-name labels, durations, paid file count 115, PTY consumer list; CHANGELOG and TODOS. * setup-deploy: say which deploy fields count as confirmed, how to resolve the default branch, and group the detection tests Periodic census 37176837432: the setup-deploy workflow judge's actionability mean was 3.67 (4, 4, 3). The low sample named the Fly/Heroku/Railway/GitHub Actions deploy trigger left undefined until Step 4's 'not already confirmed' catch-all and an unstated way to find <default branch>, and misread 'A || B && C' detection lines. Step 4 now defines a confirmed field and names the paths that leave the trigger unset, <default branch> resolves with gh repo view, and the two OR'd detection tests are grouped explicitly (same behavior). One judge run on the repaired text: 4/4/4, pass. * Regenerate generated output for the setup-deploy wording * TODOS: census data for the skipped CEO mode handoff (mode-routing, auto-decide-preserved); floor runner Edit-permission parse under collapsed TTY spaces * fix(browse): a blocked navigation that fails at once leaves the tab blank and the next command clean PR CI on8d2d269(free-suite 3): the redirect to [::ffff:169.254.170.2] failed immediately on the runner, so its chrome-error page committed around the guard's about:blank reset, and leaving the error page committed a second about:blank after the reset resolved, interrupting the next three tests' gotos. The guard now lets the blocked request settle (bounded at 1 s; a hanging target is still cut off by the reset), resets, and waits for the main frame to stay quiet before returning. Regression: a redirect to a blocked address on Chromium's unsafe port 1 fails at once (pre-fix: blank check and the next goto fail, deterministically). * qa: the checkpoint note is created for browser runs and composed for functional ones Periodic census 37178143007: the qa-only workflow judge scored actionability 3.67 again; one sample read step 2's 'Create exploration-NNN.json' as applying to functional runs before reaching 'Functional: do not write this file'. The sentence now names both surfaces up front; byte-neutral (the browser line drops its restated clause), qa union 1.1098 under its 1.110 cap. One judge run on the repaired text: 3.67/4/4, pass. * Regenerate generated output for the qa checkpoint wording * eval(section-loading): byte-range prints of a section count as a complete read Periodic census 37178143007: plan-ceo-section-loading printed review-sections.md in five byte ranges (head -c 16000, then tail -c +N | head -c 17000, one-byte overlaps) and the detector, which matched whole printed lines, saw the lines split at each range edge as missing. Outputs of the commands that name the section are also joined on their overlap. Regression: the captured ranges count; dropping a middle range or the last range does not (pre-fix: the complete ranges were refused). * test(plan-design-review): count design terms the review wrote into the plan * fix(aside): probe names the platform (GSTACK_PLATFORM override) so the macOS pitch never re-probes the OS * fix(plan-ceo-review): Step 0E mode handoff runs gstack-ceo-mode-handoff so the selected mode is printed, not only composed * chore: refresh context-budget fixture * test(auto-decide): accept Step 0E's documented AUTO_DECIDE line with its decisions suffix; replay the captured trial * docs: CHANGELOG and TODOS for the CEO mode handoff and platform line * test(plan-design-review): plan-term credit needs a rated summary; adapter regression for census 37179171083 * test(auto-decide): the printed handoff may precede Step 0E's provenance log; replay census 37182865432; TODOS for two rotating census reds * test(plan-ceo-section-loading): read the skill's ledger and currentDecision records for the stale-fill decision; replay three census reports * test: $B counts as a browser call only in command position; qa functional prompt defines its classification values * test(shared-libs-plan-callers): a verbless unchanged-scope list is an exclusion; replay census 37186854666; TODOS for two census reds * fix(settings-hook): retry a lock released between mkdir and the existence check instead of dropping the mutation * test(ship-docsync): the child's reply to a parent SendMessage counts as its spawned-marker report; replay PR run 37188668421 * test(browse): Xvfb stub-failure tests allocate from ranges no other allocator uses * perf(ship): documentation gate snapshots and compares its candidate with gstack-docs-candidate * fix(review): unverified Codex mode runs the passes in the mode summary too; stale-fill decision check accepts any ledger row ID * test(shared-libs-plan-callers): judge scope by forbidden verb+object actions, not an exclusion grammar; replay six refusals and four passing censuses * test(shared-libs-plan-callers): type the brief-line parameter * test(ship-docsync): an inline-code status quote is not a false current claim; replay PR run 37196811165 * fix(review,ship): Codex challenge and structured-review bodies run in unverified mode, matching their headers * docs(TODOS): overlay-harness control-arm error and format-approach provider refusals, with run links * docs(TODOS): qa-only workflow judge data point --------- Co-authored-by: Lewis <lewispeel@hotmail.com> Co-authored-by: Yugz <yannduzelier@gmail.com> Co-authored-by: Florian <florian.cleyetmerle@gmail.com> Co-authored-by: cdecook23 <decook23@gmail.com> Co-authored-by: mb1810 <michael.berger1810@gmail.com> Co-authored-by: Koda <koda@edgeviewfinance.com.au> Co-authored-by: Merit Blake <merit@newsprint.com> Co-authored-by: Nguyen Thanh Dat <ntdat812@gmail.com> Co-authored-by: pneumorea-ai <pneumorea@gmail.com> Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> Co-authored-by: Matt Van Horn <mvanhorn@gmail.com> Co-authored-by: Infiniteyieldai <infiniteyield.ai@gmail.com> Co-authored-by: Gabriel Rondon <grondon@gmail.com> Co-authored-by: Davon Johnson <d.johnson@dayofshow.net> Co-authored-by: Kevin Grasso <kevin@biblingo.com> Co-authored-by: Xuan-Ce Wang <261591073+xwang4-svg@users.noreply.github.com> Co-authored-by: loulanyue <260355617@qq.com> Co-authored-by: Thomas Reuser <thomasreuser@gmail.com> Co-authored-by: Bit0ps <alex.dreamscaper@gmail.com> Co-authored-by: oli548 <312288720+oli548@users.noreply.github.com> Co-authored-by: Slava <slava.istomin@tuta.io> Co-authored-by: ryanlaiwy <129729476+ryanlaiwy@users.noreply.github.com> Co-authored-by: 李翊楠 <yinanli1917@gmail.com> Co-authored-by: Mammad M. <mammad@digit.az> Co-authored-by: Lockyer <135391289+Lockyer228@users.noreply.github.com> Co-authored-by: Spooks444 <193448029+Spooks444@users.noreply.github.com> Co-authored-by: punksterlabs <77217235+punksterlabs@users.noreply.github.com> Co-authored-by: Jonathan Zhang <topcoder1@Mac-Studio.local> Co-authored-by: Marek Cermak <marek.cermak@strv.com> Co-authored-by: y$un_ <forrest.sun527@gmail.com> Co-authored-by: neallee <neallee@gmail.com> Co-authored-by: hardy <932821449@qq.com> Co-authored-by: afshaker <andrew@phone2.io> Co-authored-by: cube <frosimanuel@gmail.com> Co-authored-by: jacob-wang <jiayuw794@gmail.com> Co-authored-by: miao <miao@macmini-2.local> Co-authored-by: Jonah Berg <jonah.berg.g@gmail.com> Co-authored-by: Jayesh Betala <jayesh.betala7@gmail.com> Co-authored-by: Ryan Alberts <ryan.a.alberts@gmail.com> Co-authored-by: AlinValentin7 <AlinValentin7@users.noreply.github.com> Co-authored-by: Evgenii Lopatin <e75533@gmail.com> Co-authored-by: Jino <dgfrty00@gmail.com> Co-authored-by: snig-17 <santosh.m.tiwari@gmail.com> Co-authored-by: Sholto McNeilage <sholto@nodenarrative.com.au> Co-authored-by: Phil Hie <philhie@outlook.com> Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: Jimmy Zeisweiss <4261258+jzeisweiss@users.noreply.github.com> Co-authored-by: liutiming <timingliu@outlook.com> Co-authored-by: Brandon Pugsley <aibpaem@gmail.com> Co-authored-by: benjamin beres <benjamin.beres@bienpreter.com> Co-authored-by: Som Samantray <som.samantray@gmail.com> Co-authored-by: Vaughan <vaughan@cairnsitguy.com.au> Co-authored-by: Bharadwaj Mittapelli <bharadwajmittapelli@gmail.com>
2362 lines
115 KiB
TypeScript
Executable File
2362 lines
115 KiB
TypeScript
Executable File
#!/usr/bin/env bun
|
||
/**
|
||
* test-free-shards — enumerate, shard, curate, and run the free test suite.
|
||
*
|
||
* Four jobs:
|
||
* 1. Enumeration. Walk `browse/test/`, `test/`, `make-pdf/test/` and return
|
||
* every `*.test.{ts,tsx,js,jsx,mjs,cjs}` that isn't a paid-eval test.
|
||
* 2. Sharding. Duration-pack local and isolated CI runs. Legacy --shard
|
||
* selection retains stable hash assignment.
|
||
* 3. Curation (Windows-safe filter). Scan each test's content for POSIX-only
|
||
* patterns (`/bin/bash`, `sh -c`, raw `/tmp/`, `chmod`, `xargs`). Files
|
||
* that match are excluded from the Windows-safe subset — they would fail
|
||
* on `windows-latest` no matter how the runner shards them.
|
||
* 4. Execution. Run `bun test` children through the shared shard engine
|
||
* (scripts/lib/shard-engine.ts runShardChild) and refuse to trust their
|
||
* exit code alone: every byte of output is classified strictly, so a child that exits 0 without bun's
|
||
* terminal summary (a mid-suite process.exit truncation), with `(fail)`
|
||
* result lines, or with fewer files run than planned is a FAILURE. An
|
||
* external wall-clock timeout SIGKILLs the child's process group and
|
||
* reports the shard as timed-out — distinct from failed.
|
||
*
|
||
* Execution strategy (decision ledger V3/D6 — evaluate the Bun built-in
|
||
* first; probed 2026-08 on Bun 1.3.13):
|
||
* - Full-suite runs (`bun test` via package.json, `bun run test:free`) use
|
||
* N CONCURRENT SHARD PROCESSES, serial within each (the paid runner's
|
||
* model). A single `--parallel` invocation was probed and initially
|
||
* adopted, then abandoned: three distinct Bun 1.3.13 worker pathologies
|
||
* (segfault + crash-retry wedge, skipped-file hooks stalling a worker,
|
||
* spawn-heavy files hanging under load) each stalled the whole
|
||
* invocation, while process shards isolate any wedge to its own shard.
|
||
* Original --parallel probe results, kept for the record: it
|
||
* showed --parallel (a) prints the standard `Ran N tests across M files`
|
||
* terminal summary, (b) exits non-zero when any file fails, (c) runs each
|
||
* file in its own worker process (distinct pids, no shared globals), and
|
||
* (d) converts a mid-suite process.exit(0) — which silently truncates a
|
||
* serial run at exit 0 — into a per-file `(crashed: exited)` failure with
|
||
* a complete summary and exit 1. Strictly SAFER than the serial path and
|
||
* ~2x faster on a 6-file probe (0.22s -> 0.11s wall, 280% CPU); the win
|
||
* grows with suite size since the serial suite measured 454s.
|
||
* - Legacy runs (`--shards M --shard i`) keep the hash-partitioned
|
||
* one-child-per-shard path. Cross-runner partitioning must be
|
||
* deterministic and per-file stable, so bun's own `--shard=M/N`
|
||
* (round-robin over sorted paths — every assignment shifts when a file
|
||
* lands) is not used, and there are no static per-file weight lists.
|
||
* Shard indices are STABLE: assignFilesToShards never renumbers on
|
||
* occupancy, and an empty shard is a fast no-op success.
|
||
*
|
||
* Adapted from the McGluut/gstack fork's test-free-shards.ts (190 LOC). The
|
||
* Windows-safe filter is upstream-original — codex flagged that sharding alone
|
||
* doesn't fix POSIX-bound tests, so we curate the subset that actually runs
|
||
* on the windows-latest CI job.
|
||
*
|
||
* Output contract (v1.66): the full child stream ALWAYS lands in a per-run
|
||
* private log under .context/free-test-logs (path printed at start and in the
|
||
* epilogue). The console is quiet by default — only the runner's own
|
||
* [test:free] lines, `(fail)` result lines, bun error/crash markers
|
||
* (`error:`, `panic:`, `crashed`, `Unhandled error`), and the terminal
|
||
* `Ran N tests across M files` summary reach it; `--verbose` restores full
|
||
* forwarding. After every run a stable epilogue names the failing tests
|
||
* (attributed to files via bun's `path/to/file.test.ts:` chunk headers),
|
||
* crashed+retried workers, and — on a wall-timeout kill — the wedge-suspect
|
||
* files. The strict classifier consumes the FULL stream regardless of what
|
||
* the console shows.
|
||
*
|
||
* Exit codes: 0 pass, 1 fail, 124 wall-clock timeout.
|
||
*
|
||
* Usage:
|
||
* bun run scripts/test-free-shards.ts # full suite, N concurrent shard processes
|
||
* bun run scripts/test-free-shards.ts --list # show all
|
||
* bun run scripts/test-free-shards.ts --windows-only --list # show curated
|
||
* bun run scripts/test-free-shards.ts --windows-only # run curated
|
||
* bun run scripts/test-free-shards.ts --shards 4 --shard 1 # one shard (CI matrix)
|
||
* bun run scripts/test-free-shards.ts --wall-timeout 600 # override the kill deadline
|
||
* bun run scripts/test-free-shards.ts --verbose # forward the full child stream
|
||
* bun run scripts/test-free-shards.ts --quick # explicit fast subset, not acceptance
|
||
* bun run scripts/test-free-shards.ts --attribute-home # name real-home writers: each file alone, private HOME
|
||
* bun run scripts/test-free-shards.ts --ci-plan plan.json --shards 20
|
||
* bun run scripts/test-free-shards.ts --ci-run plan.json --shard 1 --result result.json
|
||
* bun run scripts/test-free-shards.ts --ci-verify plan.json --results results/
|
||
*/
|
||
|
||
import * as fs from 'fs';
|
||
import * as os from 'os';
|
||
import * as path from 'path';
|
||
import { spawn, spawnSync } from 'child_process';
|
||
import { StringDecoder } from 'node:string_decoder';
|
||
import { createHash, randomUUID } from 'node:crypto';
|
||
import { isPaidTestFile } from '../test/helpers/paid-test-set';
|
||
import { resolveStateRoot } from '../lib/state-root';
|
||
import { attributeFreeHomeWriters, guardFreeHome, sharedFreeHome, type FreeHomeGuardFactory } from './lib/free-home-guard';
|
||
import {
|
||
BunTestOutputClassifier,
|
||
createShardSandbox,
|
||
exactTestFileSelectors,
|
||
forEachFileAlone,
|
||
isTerminationRequested,
|
||
killProcessGroup,
|
||
BunFailureSummaryParser,
|
||
nextShardLogPath,
|
||
openShardLog,
|
||
parseBunFailureResult,
|
||
parseCliFlags,
|
||
normalizeRelativePath,
|
||
readDurationSeed,
|
||
runShardChild,
|
||
strictShardStatus,
|
||
stripAnsiLine,
|
||
writeDurationSeed,
|
||
zeroExecutionVerdict,
|
||
type LanePolicy,
|
||
type ShardChildResult,
|
||
} from './lib/shard-engine';
|
||
export { normalizeRelativePath } from './lib/shard-engine';
|
||
|
||
/**
|
||
* Free-lane classification policy. Seeds accept zero-duration files (fast
|
||
* files are real measurements). A shard with zero executed tests passes when
|
||
* bun's summary still counted every planned file; missing or short file
|
||
* counts already fail through the strict verdict.
|
||
*/
|
||
export const FREE_LANE_POLICY: LanePolicy = {
|
||
acceptsSeedDuration: (ms) => ms >= 0,
|
||
zeroExecution: () => 'passed',
|
||
};
|
||
|
||
const ROOT = path.resolve(import.meta.dir, '..');
|
||
// design/test was silently absent from BOTH the package.json test script and
|
||
// this list — design tests (including a teardown bomb) never ran in any CI
|
||
// or local free run. Keep the two lists in sync. This list is the single
|
||
// source of truth for free-suite roots: package.json's `test` script routes
|
||
// through this runner rather than passing its own directory globs.
|
||
export const TEST_ROOTS = [
|
||
'browse/test',
|
||
'test',
|
||
'make-pdf/test',
|
||
'design/test',
|
||
// v1.65 orphan wire-in (decision D3a): these ran under NO script or CI —
|
||
// written coverage that caught nothing. All were green on arrival.
|
||
'ios-qa/daemon/test',
|
||
'ios-qa/scripts',
|
||
'browser-skills',
|
||
] as const;
|
||
const TEST_FILE_REGEX = /\.test\.(?:[cm]?[jt]s|tsx|jsx)$/;
|
||
|
||
// POSIX-only patterns that indicate a test will fail on windows-latest no
|
||
// matter how the runner shards. Codex's v1.18.0.0 review flagged the first
|
||
// three as concrete examples in the existing free suite (test/ship-version-sync.test.ts:72,
|
||
// test/helpers/providers/claude.ts:22, package.json:12). We scan the test's
|
||
// own content here so the filter stays automatic as new tests land. The
|
||
// "Windows-incompatible APIs" patterns at the bottom were added after the
|
||
// first windows-free-tests CI run surfaced concrete failure modes.
|
||
const WINDOWS_FRAGILE_PATTERNS: Array<{ pattern: RegExp; reason: string }> = [
|
||
// Hardcoded POSIX shells / commands.
|
||
{ pattern: /['"`]\/bin\/(?:ba)?sh/, reason: 'hardcoded /bin/sh or /bin/bash' },
|
||
{ pattern: /spawnSync\(['"]sh['"],|spawn\(['"]sh['"],|exec\(['"]sh /, reason: 'spawn("sh", ...)' },
|
||
{ pattern: /['"]bash -c['"]|['"]sh -c['"]/, reason: 'bash -c / sh -c' },
|
||
{ pattern: /['"`]\/tmp\//, reason: 'raw /tmp/ path (use os.tmpdir())' },
|
||
{ pattern: /['"]chmod\b/, reason: 'chmod shell command' },
|
||
{ pattern: /['"]xargs\b/, reason: 'xargs pipeline' },
|
||
{ pattern: /\bwhich claude\b/, reason: 'which claude (use Bun.which)' },
|
||
// Windows-incompatible APIs.
|
||
{ pattern: /\.mode\s*&\s*0o[0-7]+/, reason: 'POSIX file mode bitmask (mode & 0o600 etc — Windows fakes mode bits)' },
|
||
{ pattern: /\.endsWith\(['"]\//, reason: 'hardcoded forward-slash path assertion (Windows uses \\\\)' },
|
||
{ pattern: /['"]\.\/[a-zA-Z][^"']*['"]\)\s*\.\s*toBe\(true\)/, reason: 'forward-slash path comparison' },
|
||
// Tests that spawn a bash shebang script in bin/ via spawnSync. Git Bash on
|
||
// Windows can run `bash /path/to/script` but spawnSync(scriptPath, ...)
|
||
// tries to execute the file directly via CreateProcess, which fails on the
|
||
// shebang. The pattern matches `, 'bin'` as a path-join argument (closing
|
||
// OR followed by another segment), which catches:
|
||
// - path.join(ROOT, 'bin', 'script-name') — typical
|
||
// - join(import.meta.dir, '..', 'bin', 'name') — destructured (diff-scope)
|
||
// - path.join(ROOT, 'bin') — bare BIN constant (brain-sync)
|
||
{ pattern: /,\s*['"]bin['"]\s*[,)]|['"]\.?\/?bin\/[a-z][\w-]+['"]/, reason: 'spawns bin/ shebang script (Windows CreateProcess does not parse shebangs)' },
|
||
// Tests that launch a real Playwright browser. The windows-free-tests CI job
|
||
// runs a curated subset that intentionally does NOT install Chromium —
|
||
// browser bring-up on Windows is a separate concern (see PR #1238). Tests
|
||
// matching `await foo.launch(` need Chromium and fail with "Executable
|
||
// doesn't exist" on the runner.
|
||
{ pattern: /await\s+\w+\.launch\(/, reason: 'launches Playwright browser (Chromium not installed in windows-free CI)' },
|
||
// Tests that spawn the browse server as a subprocess via `bun run server.ts`.
|
||
// The Bun → server.ts → Playwright path is the same one that doesn't work
|
||
// on Windows (PR #1238 windows-pty-bun-pty-fix). Tests typically set
|
||
// BROWSE_HEADLESS_SKIP=1 to skip the browser launch but still need a working
|
||
// server, which they don't get on Windows.
|
||
{ pattern: /BROWSE_HEADLESS_SKIP|spawn\(\[['"]bun['"],\s*['"]run['"]/, reason: 'spawns the browse server subprocess (Bun-driven path is Windows-broken)' },
|
||
];
|
||
|
||
// Explicit known-Windows-incompatible test files that don't fit a regex
|
||
// pattern. Listed here with the precise reason. Prefer adding a pattern above
|
||
// when possible; this list is for environment-/runtime-specific tests where
|
||
// the failure mode is structural rather than detectable via source-file scan.
|
||
export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }> = [
|
||
{
|
||
file: 'test/qa-evidence-producer.test.ts',
|
||
reason: 'executes the registered Linux native actor and its inotify observer; portable capture and Windows job behavior are covered by qa-evidence.test.ts',
|
||
},
|
||
{
|
||
file: 'test/qa-functional-fixture.test.ts',
|
||
reason: 'executes graceful POSIX signal cancellation; Bun on Windows uses TerminateProcess and cannot run the fixture SIGTERM cleanup handler',
|
||
},
|
||
{
|
||
file: 'test/qa-functional-observer-atomic.test.ts',
|
||
reason: 'exercises real Linux inotify inode and directory watches through libc.so.6; Windows has no equivalent kernel interface',
|
||
},
|
||
{
|
||
file: 'test/docsync-report-interface.test.ts',
|
||
reason: 'executes registered native documentation callbacks with their real Linux inotify write observer before the model boundary',
|
||
},
|
||
{
|
||
file: 'test/setup-gbrain-fixture.test.ts',
|
||
reason: 'the fixture invokes real POSIX detector/verifier helpers through executable shebang wrappers',
|
||
},
|
||
{
|
||
file: 'test/hermetic-skills-seeding.test.ts',
|
||
reason: 'seeds the POSIX PTY skill runtime, whose embedded shell paths require a POSIX temporary root',
|
||
},
|
||
{
|
||
file: 'test/hermetic-wiring.test.ts',
|
||
reason: 'its runtime contract check seeds the POSIX PTY skill runtime; the curated Windows lane does not run that harness',
|
||
},
|
||
{
|
||
file: 'test/pty-workspace-trust.test.ts',
|
||
reason: 'launches the POSIX PTY harness with a fake executable and bound skill runtime',
|
||
},
|
||
{
|
||
file: 'test/host-config.test.ts',
|
||
reason: 'asserts "claude" binary on PATH (only true when running inside Claude Code, not on bare CI runner)',
|
||
},
|
||
{
|
||
file: 'browse/test/findport.test.ts',
|
||
reason: 'asserts Bun.serve.stop() is fire-and-forget — Bun behavior differs on Windows for this polyfill',
|
||
},
|
||
// First full run of the expanded lane (v1.66, 13 → ~258 files) surfaced
|
||
// seven POSIX-bound files the content patterns cannot see (their
|
||
// POSIX-ness is what they TEST, or arrives via a variable). Receipts:
|
||
// PR #2593 windows-free-tests run 31918591602.
|
||
{
|
||
file: 'test/codex-under-codex-detection.test.ts',
|
||
reason: 'drives the rendered preflight bash under a hardcoded POSIX PATH (/usr/bin:/bin) — bash is unreachable through that PATH on Windows, so every case sees empty output (v1.67 windows lane run 95234224148)',
|
||
},
|
||
{
|
||
file: 'test/regression-pr1169-build-app-sed.test.ts',
|
||
reason: 'tests sed escape sequences in build-app.sh — sed/bash are the subject under test',
|
||
},
|
||
{
|
||
file: 'test/setup-conductor-worktree.test.ts',
|
||
reason: 'tests ln -snf symlink semantics in the setup script — POSIX ln is the subject under test',
|
||
},
|
||
{
|
||
file: 'test/artifacts-init-migration.test.ts',
|
||
reason: 'runs a bash migration script + jq against a scaffolded git state — POSIX toolchain paths break under cmd spawn',
|
||
},
|
||
{
|
||
file: 'test/gstack-decision-semantic.test.ts',
|
||
reason: 'installs a fake gbrain SHEBANG SHIM on PATH; Windows spawn cannot exec shebang scripts',
|
||
},
|
||
{
|
||
file: 'test/question-log-hook.test.ts',
|
||
reason: 'spawns the PostToolUse hook script (bash shebang) directly; Windows spawn cannot exec it',
|
||
},
|
||
{
|
||
file: 'browse/test/browser-skills-e2e.test.ts',
|
||
reason: 'asserts forward-slash tier paths (<repo>/browser-skills/) that resolve with backslashes on Windows',
|
||
},
|
||
{
|
||
file: 'design/test/variants-retry-after.test.ts',
|
||
reason: 'wall-clock retry-timing assertions — flaky on the slow windows-latest runner even with widened bounds',
|
||
},
|
||
// Round-2 census (PR #2593 run 31919227507) after the first seven:
|
||
{
|
||
file: 'test/skill-census.test.ts',
|
||
reason: 'census walk throws at module load on Windows (skill-census.ts:63) — the skills-tree symlink layout needs Developer Mode that CI runners lack',
|
||
},
|
||
{
|
||
file: 'browse/test/browser-manager-unit.test.ts',
|
||
reason: 'wedges the shard to its wall deadline on windows-latest (in-flight at kill); needs a Windows repro to diagnose — macOS + Linux lanes cover the file',
|
||
},
|
||
// Round-3 census (PR #2593 run 31919871680): the round-2 wedge had been
|
||
// TRUNCATING its shard, so these seven only surfaced once shard 2 completed.
|
||
// All the same POSIX-environment classes: PID/cmdline identity probing,
|
||
// bash scripts as the subject under test, env-scrubbed child spawns.
|
||
{
|
||
file: 'browse/test/server-embedder-terminal-port.test.ts',
|
||
reason: 'identity-based terminal-agent kill probes PID/cmdline with POSIX semantics; teardown asserts fail on windows-latest',
|
||
},
|
||
{
|
||
file: 'design/test/daemon-discovery.test.ts',
|
||
reason: 'verifyIdentity matches a spawned daemon via /proc-style cmdline probing — POSIX identity semantics',
|
||
},
|
||
{
|
||
file: 'test/context-save-hardening.test.ts',
|
||
reason: 'bash context-save/migration scripts (HOME-unset semantics, random-suffix path) are the subject under test',
|
||
},
|
||
{
|
||
file: 'test/eval-list-cli.test.ts',
|
||
reason: 'spawns the eval:list CLI via bun with a constructed env — bun resolution fails under Windows spawn',
|
||
},
|
||
{
|
||
file: 'test/memory-cache-injection.test.ts',
|
||
reason: 'exercises hook/deny-enforcement shell scripts — POSIX toolchain is the subject under test',
|
||
},
|
||
{
|
||
file: 'test/migrations-v1.65.0.0.test.ts',
|
||
reason: 'bash migration script (bunx re-fetch, .done markers) is the subject under test',
|
||
},
|
||
{
|
||
file: 'test/question-preference-hook.test.ts',
|
||
reason: 'spawns the PreToolUse preference hook (shebang script) directly; Windows spawn cannot exec it',
|
||
},
|
||
// Round-4 census (PR #2593 run 31920052810): unhandled errors with no
|
||
// (fail) lines — attributed statically (the lane had no log artifact yet).
|
||
{
|
||
file: 'browse/test/browser-skill-commands.test.ts',
|
||
reason: 'spawnSkill spawns bun with a constructed env — bun resolution fails under Windows spawn (unhandled, no (fail) line)',
|
||
},
|
||
{
|
||
file: 'browse/test/security-audit-r2.test.ts',
|
||
reason: 'symlink-attack fixtures (evil-link) need Developer Mode CI runners lack; expect(toThrow) fires unhandled on Windows',
|
||
},
|
||
// CSO comprehensive execution is qualified only for Linux containers behind
|
||
// the POSIX watchdog and Unix-domain registry broker. Keep the portable
|
||
// static/parser contracts in the Windows lane while leaving these exact
|
||
// containment suites to the Linux and macOS gates.
|
||
{
|
||
file: 'test/cso-preparation-adversarial.test.ts',
|
||
reason: 'exercises POSIX prepared-tree and archive-cache containment for qualified Linux Docker execution, which Windows does not admit',
|
||
},
|
||
{
|
||
file: 'test/cso-preparation-container.test.ts',
|
||
reason: 'asserts POSIX permission and symlink semantics for inert exports consumed by qualified Linux Docker execution',
|
||
},
|
||
{
|
||
file: 'test/cso-preparation-executor.test.ts',
|
||
reason: 'executes the Linux Docker acquisition path and its Unix-domain registry broker; comprehensive execution is unavailable on Windows',
|
||
},
|
||
{
|
||
file: 'test/cso-verification-cleanup.test.ts',
|
||
reason: 'spawns the POSIX detached watchdog used by contained repair verification, which Windows intentionally leaves unavailable',
|
||
},
|
||
{
|
||
file: 'test/cso-witness.test.ts',
|
||
reason: 'tests the contained repair witness with POSIX private-directory and compiled-helper assumptions; comprehensive execution is unavailable on Windows',
|
||
},
|
||
{
|
||
file: 'test/shard-engine-equivalence.test.ts',
|
||
reason: 'its classification golden was recorded from the POSIX runners (process-group wall kill); the win32 engine path is pinned by the mocked-platform case in shard-engine.test.ts',
|
||
},
|
||
{
|
||
file: 'test/cso-scanner-cli.test.ts',
|
||
reason: 'drives the prebuilt POSIX CSO launcher with /usr/bin/git and a POSIX-only PATH; native Windows launcher behavior is covered by the dedicated cso-windows-launcher gate',
|
||
},
|
||
];
|
||
|
||
// Force-include overrides: files a WINDOWS_FRAGILE_PATTERNS regex excludes for
|
||
// a reason that does not actually apply to them. Each entry documents WHY the
|
||
// pattern hit is a false positive — the point of these files is Windows
|
||
// coverage, so auto-excluding them defeats the regression tests they carry.
|
||
const KNOWN_WINDOWS_SAFE: Array<{ file: string; reason: string }> = [
|
||
{ file: 'test/ship-hook-windows-paths.test.ts', reason: 'runs bin/ helpers through explicit bash and Bun argv with forward-slash paths; never executes a shebang; the path-spelling simulation is skipIf win32' },
|
||
{ file: 'test/state-root-parity.test.ts', reason: 'runs the bash twin and lib/state-root.ts over an env table with PATH empty; no shebang execution, raw-string comparison is platform-neutral' },
|
||
{ file: 'test/generator-eexist.test.ts',
|
||
reason: 'E4: runs the generators through Bun argv with the Windows EEXIST emulation preload; no shebang execution' },
|
||
{ file: 'test/gstack-config-gbrain-refresh.test.ts', reason: 'E6: runs bin/gstack-config through explicit bash; its gbrain/python3 shims are found by bash PATH lookup, never launched by CreateProcess' },
|
||
{
|
||
file: 'test/qa-evidence.test.ts',
|
||
reason: 'invokes the production helper through Bun argv and exercises native Windows job cleanup, private file captures and backpressured receipt output',
|
||
},
|
||
{
|
||
file: 'test/qa-evidence-selection.test.ts',
|
||
reason: 'bin/ strings are literal dependency and Windows-selection assertions; no native actor or shebang command is launched',
|
||
},
|
||
{
|
||
file: 'test/qa-deadline.test.ts',
|
||
reason: 'launches the guard through Bun argv; mode assertions and POSIX signal cases are platform-gated, while Windows job cleanup must execute natively',
|
||
},
|
||
{
|
||
file: 'test/qa-deadline-selection.test.ts',
|
||
reason: 'bin/ strings are dependency-selection inputs; this suite never launches a shebang executable',
|
||
},
|
||
{
|
||
file: 'test/shared-libs-source-reads.test.ts',
|
||
reason: 'bin/ literal is a mocked launch assertion; actual worktree fingerprinting explicitly invokes Bash on Windows',
|
||
},
|
||
{
|
||
file: 'test/claude-code-windows-job.test.ts',
|
||
reason: 'invokes Bun directly; verifies Windows job containment at the standalone CLI boundary',
|
||
},
|
||
{
|
||
file: 'test/claude-code-runner.test.ts',
|
||
// The bin/ path is launched through process.execPath (Bun), never as a shebang; keep taskkill supervision in the Windows lane.
|
||
reason: 'invokes the runner via Bun argv; fake CLI and timeout descendant assertions cover native Windows taskkill',
|
||
},
|
||
{
|
||
file: 'test/setup-gbrain-remote-caller.test.ts',
|
||
// bin is an expected PATH component; the adapter injects the SDK boundary
|
||
// and never launches a shebang. Keep the native delimiter cases in CI.
|
||
reason: 'replays the registered SDK callback with fixture-only bin paths; covers native Windows PATH composition',
|
||
},
|
||
{
|
||
file: 'test/cso-windows-build-contract.test.ts',
|
||
// bin is a temporary staging directory. The adapter injects spawnSync;
|
||
// real PowerShell/native execution remains in cso-windows-launcher.
|
||
reason: 'replays the native build callbacks with an injected subprocess; bin paths are staging fixtures, not shebang launches',
|
||
},
|
||
{
|
||
file: 'test/setup-windows-rerun-refresh.test.ts',
|
||
// Trips the "spawns bin/ shebang script" pattern via path.join(..., 'bin',
|
||
// 'tool.sh') fixture paths, but every spawn goes through test/helpers/bash-script.ts
|
||
// (bash <tempfile>) — Git Bash executes it fine on windows-latest, with no argv-length ceiling. This file IS
|
||
// the #2444 Windows regression coverage (IS_WINDOWS=1 copy-refresh path);
|
||
// excluding it here would keep the bug class unexercised on the one
|
||
// platform it bites.
|
||
reason: 'bin/ hits are fixture path segments; spawns bash explicitly — the IS_WINDOWS=1 refresh path must run on windows-latest',
|
||
},
|
||
{
|
||
file: 'test/uninstall-windows-copies.test.ts',
|
||
// Trips the "spawns bin/ shebang script" pattern via the
|
||
// path.join(ROOT, 'bin', 'gstack-uninstall') constant, but the script is
|
||
// always spawned through spawnSync('bash', [UNINSTALL, ...]). This file
|
||
// carries the #2563 Windows real-dir-copy uninstall coverage — the bug
|
||
// ONLY reproduces on the copy install shape windows-latest exercises.
|
||
// The symlink-shape describe block self-skips on win32.
|
||
reason: 'bin/ hit is a bash-spawned script path; #2563 real-dir uninstall coverage must run on windows-latest',
|
||
},
|
||
{
|
||
file: 'browse/test/file-permissions.test.ts',
|
||
// Trips the POSIX-mode-bitmask pattern, but every `mode & 0o777` assertion
|
||
// is platform-guarded: win32-only tests return early, POSIX-only tests
|
||
// guard the bitmask behind `process.platform !== 'win32'`, and the
|
||
// symlink-skip regression test both wraps symlinkSync in try/catch
|
||
// (runners without Developer Mode can't create symlinks) and guards its
|
||
// bitmask — on win32 it asserts behavior (warns, skips, doesn't throw,
|
||
// target stays usable), never fake Windows mode bits (dirs stat 0o777
|
||
// there, so a 0o755 expectation fails on runner semantics, not our code).
|
||
// This file carries the win32-only icacls-by-SID regression tests, which
|
||
// can ONLY execute on windows-latest — excluding it here means the
|
||
// machine-account ACL lockout regression is never exercised on the one
|
||
// platform it bricks.
|
||
reason: 'every mode-bitmask assertion is guarded off win32 (behavior asserted instead); win32-only ACL regression tests must run on windows-latest',
|
||
},
|
||
{
|
||
file: 'browse/test/terminal-agent-owner-watchdog.test.ts',
|
||
// Trips the spawn(['bun','run',...]) pattern, whose reason is the
|
||
// Playwright-bound browse server. This test spawns terminal-agent.ts,
|
||
// which imports only fs/path/crypto + local helpers (no Playwright, no
|
||
// PTY at module scope) and boots under Bun on Windows — the owner-PID
|
||
// orphan leak it pins was reported on Windows (#2019).
|
||
reason: 'spawns terminal-agent (no Playwright), not the browse server; owner-orphan leak is a Windows defect',
|
||
},
|
||
];
|
||
|
||
export const DEFAULT_SHARD_COUNT = 20;
|
||
// Per-test timeout passed to `bun test --timeout`. 30s matches what
|
||
// package.json's `test` script used before it was repointed at this runner —
|
||
// the runner is now the single owner of that semantic.
|
||
export const FREE_TEST_TIMEOUT_MS = 30_000;
|
||
// External wall-clock deadline per spawned child (whole shard or the single
|
||
// full-suite --parallel invocation). A wedged child — a spinning main thread
|
||
// no in-process --timeout timer can interrupt — is SIGKILLed at the group
|
||
// level and reported 'timed-out', distinct from 'failed'.
|
||
// ~3.5x the observed full-suite wall (~100-160s). A wedged run should be
|
||
// killed-and-diagnosed (the epilogue prints the in-flight suspects) in
|
||
// minutes, not sat out — 15min of silence was pure diagnosis latency.
|
||
// Override per run with --wall-timeout <secs>.
|
||
export const DEFAULT_WALL_TIMEOUT_MS = 6 * 60_000;
|
||
/**
|
||
* Full-suite shards scale their wall deadline with shard size:
|
||
* max(DEFAULT_WALL_TIMEOUT_MS, files × PER_FILE_WALL_MS). The 6-min floor
|
||
* keeps wedge diagnosis fast on a typical ~70-file local shard, while a
|
||
* low-core machine (jobs=1 → the whole suite in one shard) or the Windows
|
||
* lane (~130 files/shard) gets proportional headroom instead of a false
|
||
* timed-out kill of a healthy run. Explicit --wall-timeout disables scaling.
|
||
*/
|
||
export const PER_FILE_WALL_MS = 5_000;
|
||
export function wallTimeoutForShard(fileCount: number, baseMs = DEFAULT_WALL_TIMEOUT_MS): number {
|
||
return Math.max(baseMs, fileCount * PER_FILE_WALL_MS);
|
||
}
|
||
|
||
/**
|
||
* Wall for a duration-packed shard. The count heuristic above assumes count
|
||
* approximates cost; LPT packing breaks that BY DESIGN (a shard may hold six
|
||
* slow Playwright files), so packed shards get max(base, predicted x 3) —
|
||
* generous against seed drift, still bounded.
|
||
*/
|
||
export function wallTimeoutForPackedShard(predictedMs: number, baseMs = DEFAULT_WALL_TIMEOUT_MS, fileCount = 0): number {
|
||
// Predictions transfer badly across machines: the committed duration seed
|
||
// is recorded on fast CI, and a syscall-supervised sandbox replays those
|
||
// files 2-4x slower (observed: a 253-file shard predicted ~242s wall-killed
|
||
// at its 725s predicted-x3 wall while genuinely still progressing). The
|
||
// packed wall may therefore be LOOSER than the count heuristic, never
|
||
// tighter — it keeps the per-file floor the runner has always guaranteed.
|
||
return Math.max(baseMs, Math.ceil(predictedMs * 3), fileCount * PER_FILE_WALL_MS);
|
||
}
|
||
/**
|
||
* Full-suite parallelism: use all available CPUs, with a floor of one and a
|
||
* per-platform cap (maxFullSuiteJobs). Shards stay serial internally; separate
|
||
* shard processes can overlap subprocess and I/O waits without a fixed CPU
|
||
* reserve. Prefer availableParallelism() to honor CPU affinity (Bun also
|
||
* honors a container's cgroup CPU quota there), falling back to cpus() on
|
||
* runtimes without it. macOS and Windows keep the cap of 6: beyond that,
|
||
* playwright-heavy shards contended on browser launches in the original
|
||
* M-series measurement. Linux caps at 16: on a 16-vCPU Ubicloud VM with the
|
||
* CI lane's environment (2026-09-28), 16 shards ran the complete suite in
|
||
* 137s versus 327s for 6. More shards are not a guaranteed speedup; compare
|
||
* complete-suite runs before raising either cap.
|
||
*
|
||
* GSTACK_FREE_JOBS overrides the computed count (the free runner's analogue
|
||
* of the paid runner's EVALS_JOBS). Exists for syscall-supervised sandboxes:
|
||
* on Vercel sandboxes, PID 1 (sandbox-init) installs a seccomp filter whose
|
||
* user-space supervisor saturates under ~6 concurrent bun+playwright shards
|
||
* and starts returning EACCES from plain file syscalls (measured: 200/200
|
||
* `git init` probes in fresh mktemp dirs fail with
|
||
* "Cannot access work tree: Permission denied" while the suite runs, 0/200
|
||
* when idle — access(dir, X_OK) = EACCES under strace). Fewer shards keep
|
||
* the supervisor inside its budget. Not clamped by maxFullSuiteJobs so a
|
||
* beefy box can also raise it deliberately.
|
||
*/
|
||
export const MAX_FULL_SUITE_JOBS = 6;
|
||
export const MAX_LINUX_FULL_SUITE_JOBS = 16;
|
||
|
||
export function maxFullSuiteJobs(platform: NodeJS.Platform = process.platform): number {
|
||
return platform === 'linux' ? MAX_LINUX_FULL_SUITE_JOBS : MAX_FULL_SUITE_JOBS;
|
||
}
|
||
|
||
export function fullSuiteJobs(platform: NodeJS.Platform = process.platform): number {
|
||
const raw = process.env.GSTACK_FREE_JOBS;
|
||
if (raw !== undefined && raw !== '') {
|
||
// Strict digits-only: parseInt would silently truncate "2abc" -> 2 and
|
||
// "3.7" -> 3, defeating the loud-failure contract the error text claims.
|
||
if (!/^\d+$/.test(raw.trim()) || Number.parseInt(raw, 10) <= 0) {
|
||
throw new Error(`GSTACK_FREE_JOBS must be a positive integer, got: ${raw}`);
|
||
}
|
||
return Number.parseInt(raw, 10);
|
||
}
|
||
const availableCpus = os.availableParallelism?.() ?? os.cpus().length;
|
||
return Math.max(1, Math.min(maxFullSuiteJobs(platform), availableCpus));
|
||
}
|
||
|
||
/**
|
||
* Files that crash or wedge Bun's --parallel WORKERS but run fine in a plain
|
||
* serial process. Full-suite mode now uses shard PROCESSES (no workers), so
|
||
* this list is inert placement-wise — retained as the paper trail of why the
|
||
* one-invocation --parallel strategy was abandoned, and as the exclusion list
|
||
* should anyone re-attempt it on a newer Bun.
|
||
*/
|
||
export const WORKER_HOSTILE: Record<string, string> = {
|
||
'browse/test/security-live-playwright.test.ts':
|
||
'Bun 1.3.13 segfaults running this file in a --parallel worker ("panic: '
|
||
+ 'Segmentation fault ... a bug in Bun"), and the crashed-worker retry then '
|
||
+ 'wedges the whole invocation past the wall clock. Passes serially.',
|
||
};
|
||
|
||
/**
|
||
* Exclusive host-state fixtures: run in ONE serial shard AFTER the parallel
|
||
* shards. The public name is retained for callers of the original tree-write
|
||
* classification. Entries need a concrete shared-state hazard that fixture
|
||
* directories cannot isolate, such as host-wide procfs visibility.
|
||
* Keys are pinned against the live file census by test-free-shards.test.ts —
|
||
* a renamed file fails the suite instead of silently dropping serialization.
|
||
*/
|
||
export const TREE_MUTATING: Record<string, string> = {
|
||
'test/bootstrap-retention.test.ts': 'Creates nondumpable same-UID processes visible to every host procfs census; must not overlap other native-retention fixtures.',
|
||
};
|
||
|
||
export function isFreeTestFile(relativePath: string): boolean {
|
||
const normalized = normalizeRelativePath(relativePath);
|
||
if (!TEST_FILE_REGEX.test(normalized)) return false;
|
||
return !isPaidTestFile(normalized);
|
||
}
|
||
|
||
/**
|
||
* Returns the first POSIX-only pattern hit in the file, or null if Windows-safe.
|
||
*/
|
||
export function detectWindowsFragility(absolutePath: string): { reason: string } | null {
|
||
let content: string;
|
||
try {
|
||
content = fs.readFileSync(absolutePath, 'utf-8');
|
||
} catch {
|
||
return null;
|
||
}
|
||
for (const { pattern, reason } of WINDOWS_FRAGILE_PATTERNS) {
|
||
if (pattern.test(content)) return { reason };
|
||
}
|
||
return null;
|
||
}
|
||
|
||
function walkTestFiles(dirPath: string): string[] {
|
||
const entries = fs.readdirSync(dirPath, { withFileTypes: true });
|
||
const files: string[] = [];
|
||
for (const entry of entries) {
|
||
const fullPath = path.join(dirPath, entry.name);
|
||
if (entry.isDirectory()) {
|
||
files.push(...walkTestFiles(fullPath));
|
||
continue;
|
||
}
|
||
if (TEST_FILE_REGEX.test(entry.name)) {
|
||
files.push(fullPath);
|
||
}
|
||
}
|
||
return files;
|
||
}
|
||
|
||
export function collectFreeTestFiles(rootDir = ROOT): string[] {
|
||
const discovered = new Set<string>();
|
||
for (const testRoot of TEST_ROOTS) {
|
||
const absoluteRoot = path.join(rootDir, testRoot);
|
||
if (!fs.existsSync(absoluteRoot)) continue;
|
||
for (const fullPath of walkTestFiles(absoluteRoot)) {
|
||
const relativePath = normalizeRelativePath(path.relative(rootDir, fullPath));
|
||
if (isFreeTestFile(relativePath)) {
|
||
discovered.add(relativePath);
|
||
}
|
||
}
|
||
}
|
||
return [...discovered].sort();
|
||
}
|
||
|
||
export interface CurationResult {
|
||
safe: string[];
|
||
excluded: Array<{ file: string; reason: string }>;
|
||
}
|
||
|
||
export function curateWindowsSafe(files: string[], rootDir = ROOT): CurationResult {
|
||
const safe: string[] = [];
|
||
const excluded: Array<{ file: string; reason: string }> = [];
|
||
const knownBad = new Map(KNOWN_WINDOWS_INCOMPATIBLE.map((e) => [e.file, e.reason]));
|
||
const knownSafe = new Set(KNOWN_WINDOWS_SAFE.map((e) => e.file));
|
||
for (const relativePath of files) {
|
||
const knownReason = knownBad.get(relativePath);
|
||
if (knownReason) {
|
||
excluded.push({ file: relativePath, reason: knownReason });
|
||
continue;
|
||
}
|
||
if (knownSafe.has(relativePath)) {
|
||
safe.push(relativePath);
|
||
continue;
|
||
}
|
||
const absolute = path.join(rootDir, relativePath);
|
||
const fragility = detectWindowsFragility(absolute);
|
||
if (fragility) {
|
||
excluded.push({ file: relativePath, reason: fragility.reason });
|
||
} else {
|
||
safe.push(relativePath);
|
||
}
|
||
}
|
||
return { safe, excluded };
|
||
}
|
||
|
||
export function stableHash(input: string): number {
|
||
let hash = 0x811c9dc5;
|
||
for (let index = 0; index < input.length; index += 1) {
|
||
hash ^= input.charCodeAt(index);
|
||
hash = Math.imul(hash, 0x01000193);
|
||
}
|
||
return hash >>> 0;
|
||
}
|
||
|
||
/**
|
||
* Hash-partition files across EXACTLY shardCount shards. Empty shards are
|
||
* preserved: a file's shard index is a pure function of its own path and the
|
||
* shard count, never of which other files happen to exist. A CI matrix keys
|
||
* runners off the index, so filtering empty shards (the old behavior) would
|
||
* renumber every later shard whenever occupancy shifted — runner 3 silently
|
||
* running shard 4's files. An empty shard is instead a fast no-op success at
|
||
* run time.
|
||
*/
|
||
export function assignFilesToShards(files: string[], shardCount: number): string[][] {
|
||
if (!Number.isInteger(shardCount) || shardCount <= 0) {
|
||
throw new Error(`Shard count must be a positive integer. Received: ${shardCount}`);
|
||
}
|
||
|
||
const shards = Array.from({ length: shardCount }, () => [] as string[]);
|
||
for (const file of files) {
|
||
const shardIndex = stableHash(file) % shardCount;
|
||
shards[shardIndex].push(file);
|
||
}
|
||
|
||
return shards.map(filesInShard => filesInShard.sort());
|
||
}
|
||
|
||
// ─── Duration-aware packing (local full suite and explicit CI plans) ───────
|
||
// Hash sharding balances file COUNTS (~1.15x spread) but not cost: the 15
|
||
// Playwright-launching files land 4/3/4/1/2/1 across 6 shards, giving a
|
||
// measured 28s–97s shard spread and ~40s of idle tail on every run. LPT
|
||
// packing over recorded per-file durations reclaims most of it. The `--shard`
|
||
// legacy path is deliberately untouched — its contract is stable indices
|
||
// via assignFilesToShards/stableHash (empty shards no-op; see above).
|
||
//
|
||
// One store, no overlay: durations come from the committed seed
|
||
// (scripts/free-test-durations.json), refreshed occasionally via
|
||
// `--record-durations` (each file timed in its own child — exact, and immune
|
||
// to bun's stream buffering, where silent passers print no header to
|
||
// timestamp). GSTACK_FREE_TEST_DURATIONS overrides the path for experiments.
|
||
// The seed is a HINT, not a contract: missing file → hash-shard fallback;
|
||
// unknown file → 75th-percentile pessimism (placed early by LPT, bounding
|
||
// tail risk). CI shares one plan rather than independently recomputing it.
|
||
|
||
export const FREE_TEST_DURATIONS_FILE = 'scripts/free-test-durations.json';
|
||
|
||
export function loadFreeTestDurations(rootDir = ROOT): Record<string, number> | null {
|
||
const file = process.env.GSTACK_FREE_TEST_DURATIONS
|
||
?? path.join(rootDir, FREE_TEST_DURATIONS_FILE);
|
||
const seed = readDurationSeed(file, FREE_LANE_POLICY.acceptsSeedDuration);
|
||
// No seed — hash sharding, silently (fresh checkouts are normal).
|
||
if (seed.status === 'missing') return null;
|
||
if (seed.status === 'corrupt') {
|
||
// A corrupt seed (bad merge) must cost a warning, never the suite.
|
||
console.error(`[test:free] WARNING: corrupt durations seed ${file} (${seed.error.message}) — falling back to hash sharding`);
|
||
return null;
|
||
}
|
||
return Object.keys(seed.durations).length === 0 ? null : seed.durations;
|
||
}
|
||
|
||
export interface PackedShards {
|
||
shards: string[][];
|
||
/** Predicted total per shard, aligned with `shards` — feeds walls + logs. */
|
||
predictedMs: number[];
|
||
}
|
||
|
||
/**
|
||
* Longest-processing-time-first bin packing: files sorted by predicted
|
||
* duration (desc, path-stable tiebreak) each go to the currently-lightest
|
||
* shard. Deterministic for a given (files, shardCount, durations).
|
||
*/
|
||
export function packShardsByDuration(
|
||
files: string[],
|
||
shardCount: number,
|
||
durations: Record<string, number>,
|
||
): PackedShards {
|
||
if (!Number.isInteger(shardCount) || shardCount <= 0) {
|
||
throw new Error(`Shard count must be a positive integer. Received: ${shardCount}`);
|
||
}
|
||
const known = files
|
||
.map((f) => durations[normalizeRelativePath(f)])
|
||
.filter((v): v is number => typeof v === 'number')
|
||
.sort((a, b) => a - b);
|
||
// Unknown files get the 75th percentile of known durations: pessimistic, so
|
||
// LPT places them early and a surprise long-runner can't recreate the tail.
|
||
const fallback = known.length > 0 ? known[Math.min(known.length - 1, Math.floor(known.length * 0.75))] : 1;
|
||
const predicted = (f: string): number => durations[normalizeRelativePath(f)] ?? fallback;
|
||
|
||
const ordered = [...files].sort((a, b) => predicted(b) - predicted(a) || (a < b ? -1 : 1));
|
||
const shards = Array.from({ length: shardCount }, () => [] as string[]);
|
||
const loads = new Array<number>(shardCount).fill(0);
|
||
for (const file of ordered) {
|
||
let lightest = 0;
|
||
for (let i = 1; i < shardCount; i += 1) {
|
||
if (loads[i] < loads[lightest]) lightest = i;
|
||
}
|
||
shards[lightest].push(file);
|
||
loads[lightest] += predicted(file);
|
||
}
|
||
return { shards: shards.map((s) => s.sort()), predictedMs: loads };
|
||
}
|
||
|
||
/**
|
||
* Files missing from the duration seed are packed at the 75th percentile, so
|
||
* one slow new file can become the whole run's long pole without anyone
|
||
* noticing. Name them (on stderr: --ci-plan's stdout is the CI matrix).
|
||
*/
|
||
export function unseededFreeFiles(files: string[], durations: Record<string, number>): string[] {
|
||
return files.filter((f) => durations[normalizeRelativePath(f)] === undefined);
|
||
}
|
||
|
||
function warnUnseededFreeFiles(files: string[], durations: Record<string, number>): void {
|
||
const unseeded = unseededFreeFiles(files, durations);
|
||
if (unseeded.length === 0) return;
|
||
const shown = unseeded.slice(0, 5).join(', ') + (unseeded.length > 5 ? `, +${unseeded.length - 5} more` : '');
|
||
console.error(`[test:free] ${unseeded.length} file(s) have no recorded duration and are packed at the 75th-percentile estimate: ${shown}.`
|
||
+ ' Refresh scripts/free-test-durations.json with `bun run test:ubicloud --record-durations`.');
|
||
}
|
||
|
||
export interface FreeCiPlan {
|
||
version: 1;
|
||
revision: string;
|
||
id: string;
|
||
shards: Array<{ shard: number; files: string[]; predictedMs: number }>;
|
||
}
|
||
|
||
const planDigest = (plan: Omit<FreeCiPlan, 'id'>): string =>
|
||
createHash('sha256').update(JSON.stringify(plan)).digest('hex');
|
||
|
||
/** One immutable plan is shared by isolated CI machines; never repack per job. */
|
||
export function createFreeCiPlan(files: string[], count: number, durations: Record<string, number>, revision: string): FreeCiPlan {
|
||
const readers = files.filter(file => !(file in TREE_MUTATING));
|
||
const exclusive = files.filter(file => file in TREE_MUTATING).sort();
|
||
const packed = packShardsByDuration(readers, count, durations);
|
||
const shards = packed.shards.map((files, index) => ({ shard: index + 1, files, predictedMs: packed.predictedMs[index] }));
|
||
if (exclusive.length) shards.push({ shard: shards.length + 1, files: exclusive, predictedMs: exclusive.reduce((ms, file) => ms + (durations[file] ?? 0), 0) });
|
||
const body = { version: 1 as const, revision, shards };
|
||
return { ...body, id: planDigest(body) };
|
||
}
|
||
|
||
export function validateFreeCiPlan(plan: FreeCiPlan, files: string[], revision: string): void {
|
||
const { id, version, shards } = plan;
|
||
if (version !== 1 || plan.revision !== revision || !Array.isArray(shards) || !shards.length
|
||
|| id !== planDigest({ version, revision: plan.revision, shards })) throw new Error('CI plan identity or revision mismatch');
|
||
if (shards.some((shard, index) => shard.shard !== index + 1 || !Array.isArray(shard.files)
|
||
|| !Number.isFinite(shard.predictedMs) || shard.predictedMs < 0)) throw new Error('Invalid CI shard plan');
|
||
const planned = shards.flatMap(shard => shard.files).sort();
|
||
if (new Set(planned).size !== planned.length || JSON.stringify(planned) !== JSON.stringify([...files].sort())) throw new Error('CI plan must cover every free file exactly once');
|
||
}
|
||
|
||
export interface FreeCiResult {
|
||
planId: string;
|
||
revision: string;
|
||
outcome: FreeShardOutcome;
|
||
retry: FreeShardOutcome | null;
|
||
}
|
||
|
||
/** Preserve the full-suite retry cap across independently running CI jobs. */
|
||
export function eligibleFreeRetryFiles(outcomes: FreeShardOutcome[]): string[] | null {
|
||
if (!outcomes.every(outcome => hasScopedFailureAttribution(outcome) && (outcome.status === 'passed'
|
||
|| (outcome.status === 'failed' && outcome.failingFiles.length > 0 && outcome.unattributedFailures === 0)))) return null;
|
||
const files = [...new Set(outcomes.flatMap(outcome => outcome.failingFiles))];
|
||
return files.length > 0 && files.length <= 5 ? files : null;
|
||
}
|
||
|
||
function hasScopedFailureAttribution(outcome: FreeShardOutcome): boolean {
|
||
return new Set(outcome.failingFiles).size === outcome.failingFiles.length
|
||
&& outcome.failingFiles.every(file => outcome.files.includes(file));
|
||
}
|
||
|
||
export function verifyFreeCiResults(plan: FreeCiPlan, results: FreeCiResult[]): void {
|
||
if (results.length !== plan.shards.length) throw new Error('Missing or duplicate CI shard results');
|
||
const seen = new Set<number>();
|
||
for (const result of results) {
|
||
const outcome = result.outcome;
|
||
const shard = plan.shards[outcome.shard - 1];
|
||
if (result.planId !== plan.id || result.revision !== plan.revision || !shard || seen.has(outcome.shard)
|
||
|| JSON.stringify(outcome.files) !== JSON.stringify(shard.files)) throw new Error('CI result identity, shard or file coverage mismatch');
|
||
seen.add(outcome.shard);
|
||
if (!hasCompleteCiSummary(outcome)) throw new Error('Missing or incomplete CI execution summary');
|
||
if (outcome.status === 'passed') {
|
||
if (outcome.exitCode !== 0 || outcome.failingFiles.length || outcome.unattributedFailures || result.retry) throw new Error('Inconsistent passing CI result');
|
||
} else {
|
||
const retryFiles = eligibleFreeRetryFiles([outcome]);
|
||
const retry = result.retry;
|
||
if (!retryFiles || !retry || retry.status !== 'passed' || retry.exitCode !== 0
|
||
|| retry.failingFiles.length || retry.unattributedFailures
|
||
|| !hasCompleteCiSummary(retry)
|
||
|| JSON.stringify([...retry.files].sort()) !== JSON.stringify(retryFiles.sort())) throw new Error('Failed or incomplete CI shard');
|
||
}
|
||
}
|
||
if (results.some(result => result.retry) && !eligibleFreeRetryFiles(results.map(result => result.outcome))) {
|
||
throw new Error('CI retries exceed the full-suite attribution or five-file limit');
|
||
}
|
||
}
|
||
|
||
function hasCompleteCiSummary(outcome: FreeShardOutcome): boolean {
|
||
const summary = outcome.summary;
|
||
if (!summary || !Number.isInteger(summary.testsRan) || summary.testsRan! < 0
|
||
|| summary.filesRan !== outcome.files.length) return false;
|
||
// Empty assigned shards deliberately do not launch Bun or invent a summary.
|
||
return outcome.files.length === 0
|
||
? summary.testsRan === 0 && summary.sawTerminalSummary === false
|
||
: summary.sawTerminalSummary === true;
|
||
}
|
||
|
||
export const QUICK_CORE = [
|
||
'test/strict-output.test.ts', 'test/gen-skill-docs.test.ts',
|
||
'test/skill-check-driver.test.ts',
|
||
'test/skill-ceo-section-ordering.test.ts',
|
||
'test/qa-functional-observer.test.ts', 'test/qa-checkpoint-evidence.test.ts',
|
||
'test/test-free-shards-capture.test.ts',
|
||
];
|
||
|
||
export function selectQuickFreeFiles(files: string[], durations: Record<string, number>): string[] {
|
||
return files.filter(file => isFreeTestFile(file)
|
||
&& (QUICK_CORE.includes(file) || (durations[file] !== undefined && durations[file] <= 2_000)));
|
||
}
|
||
|
||
export interface BuildShardArgsOptions {
|
||
/**
|
||
* Pass bun's --parallel (worker-per-file, implies --isolate). No production
|
||
* caller today — full-suite mode uses N shard PROCESSES after the worker
|
||
* pathologies documented in main(); retained for a future re-attempt on a
|
||
* newer Bun (see WORKER_HOSTILE).
|
||
*/
|
||
parallel?: boolean;
|
||
rootDir?: string;
|
||
}
|
||
|
||
export function buildShardArgs(files: string[], options: BuildShardArgsOptions = {}): string[] {
|
||
// Exact absolute selectors: bun treats positional test paths as substring
|
||
// filters, so a relative `test/x.test.ts` would ALSO select
|
||
// `browse/test/x.test.ts` — shard bleed that double-runs files.
|
||
const selectors = exactTestFileSelectors(files, options.rootDir ?? ROOT);
|
||
const args = ['test', ...selectors, `--timeout=${FREE_TEST_TIMEOUT_MS}`];
|
||
if (options.parallel) args.push('--parallel');
|
||
else args.push('--max-concurrency=1');
|
||
return args;
|
||
}
|
||
|
||
type CliOptions = {
|
||
dryRun: boolean;
|
||
listOnly: boolean;
|
||
recordDurations: boolean;
|
||
windowsOnly: boolean;
|
||
verbose: boolean;
|
||
shardCount: number;
|
||
shardIndex: number | null;
|
||
wallTimeoutMs: number;
|
||
/** True when --wall-timeout was passed explicitly; full-suite mode only auto-scales the default. */
|
||
wallTimeoutExplicit: boolean;
|
||
quick: boolean;
|
||
attributeHome: boolean;
|
||
ciPlan: string | null;
|
||
ciRun: string | null;
|
||
ciVerify: string | null;
|
||
result: string | null;
|
||
results: string | null;
|
||
};
|
||
|
||
export function parseCliOptions(argv: string[]): CliOptions {
|
||
let dryRun = false;
|
||
let listOnly = false;
|
||
let recordDurations = false;
|
||
let windowsOnly = false;
|
||
let verbose = false;
|
||
let shardCount = DEFAULT_SHARD_COUNT;
|
||
let shardIndex: number | null = null;
|
||
let wallTimeoutMs = DEFAULT_WALL_TIMEOUT_MS;
|
||
let wallTimeoutExplicit = false;
|
||
let quick = false;
|
||
let attributeHome = false;
|
||
const paths: Record<'ciPlan' | 'ciRun' | 'ciVerify' | 'result' | 'results', string | null> = {
|
||
ciPlan: null, ciRun: null, ciVerify: null, result: null, results: null,
|
||
};
|
||
const pathFlag = (flag: string, key: keyof typeof paths) => (next: () => string | undefined) => {
|
||
const value = next();
|
||
if (!value || value.startsWith('--')) throw new Error(`Missing path for ${flag}`);
|
||
paths[key] = value;
|
||
};
|
||
|
||
parseCliFlags(argv, {
|
||
'--dry-run': () => { dryRun = true; },
|
||
'--list': () => { listOnly = true; },
|
||
'--record-durations': () => { recordDurations = true; },
|
||
'--windows-only': () => { windowsOnly = true; },
|
||
'--verbose': () => { verbose = true; },
|
||
'--quick': () => { quick = true; },
|
||
'--attribute-home': () => { attributeHome = true; },
|
||
'--ci-plan': pathFlag('--ci-plan', 'ciPlan'),
|
||
'--ci-run': pathFlag('--ci-run', 'ciRun'),
|
||
'--ci-verify': pathFlag('--ci-verify', 'ciVerify'),
|
||
'--result': pathFlag('--result', 'result'),
|
||
'--results': pathFlag('--results', 'results'),
|
||
'--shards': (next) => {
|
||
const value = next();
|
||
if (!value) throw new Error('Missing value for --shards');
|
||
shardCount = Number.parseInt(value, 10);
|
||
},
|
||
'--shard': (next) => {
|
||
const value = next();
|
||
if (!value) throw new Error('Missing value for --shard');
|
||
shardIndex = Number.parseInt(value, 10);
|
||
},
|
||
'--wall-timeout': (next) => {
|
||
const value = Number.parseInt(next() ?? '', 10);
|
||
if (!Number.isInteger(value) || value <= 0) throw new Error('--wall-timeout needs a positive integer (seconds)');
|
||
wallTimeoutMs = value * 1000;
|
||
wallTimeoutExplicit = true;
|
||
},
|
||
});
|
||
|
||
const ciModes = [paths.ciPlan, paths.ciRun, paths.ciVerify].filter(Boolean).length;
|
||
if (ciModes > 1 || (ciModes && (quick || listOnly || dryRun || recordDurations || windowsOnly || attributeHome))) throw new Error('CI modes cannot be combined with other selection modes');
|
||
if (paths.ciRun && (shardIndex === null || !paths.result)) throw new Error('--ci-run requires --shard and --result');
|
||
if (paths.ciVerify && !paths.results) throw new Error('--ci-verify requires --results');
|
||
if (quick && (recordDurations || windowsOnly || shardIndex !== null)) throw new Error('--quick cannot change recording, Windows or shard selection');
|
||
return { dryRun, listOnly, recordDurations, windowsOnly, verbose, shardCount, shardIndex, wallTimeoutMs, wallTimeoutExplicit, quick, attributeHome, ...paths };
|
||
}
|
||
|
||
function formatShardSummary(shards: string[][]): string[] {
|
||
return shards.map((files, index) => {
|
||
const preview = files.slice(0, 3).join(', ');
|
||
const suffix = files.length > 3 ? ', ...' : '';
|
||
return `Shard ${index + 1}/${shards.length}: ${files.length} files${preview ? ` -> ${preview}${suffix}` : ''}`;
|
||
});
|
||
}
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// Output contract: console filtering + per-file failure attribution.
|
||
//
|
||
// Bun groups each file's output under a `path/to/file.test.ts:` header line
|
||
// (cwd-relative, sometimes ../-prefixed through a symlinked cwd). The
|
||
// reporter tracks the current header while consuming the stream, attributes
|
||
// `(fail)` lines and crash markers to files, and decides which lines reach
|
||
// the console in the default quiet mode. All matching happens on
|
||
// ANSI-stripped lines — colored `(fail)` lines defeated a prior grep.
|
||
// ---------------------------------------------------------------------------
|
||
|
||
const TEST_PATH_SOURCE = String.raw`\.test\.(?:[cm]?[jt]s|tsx|jsx)`;
|
||
/** A file chunk header: the path bun printed, terminated by a bare colon. */
|
||
const FILE_HEADER_RE = new RegExp(`^(\\S.*${TEST_PATH_SOURCE}):$`);
|
||
/** bun --parallel retries a crashed worker once: `<icon> crashed running <path>, retrying`. */
|
||
const CRASH_RETRY_RE = new RegExp(`crashed running (\\S*${TEST_PATH_SOURCE}), retrying`);
|
||
/** The give-up marker after the retry also crashes: `✗ <path> (crashed: exited)`. */
|
||
const CRASH_FINAL_RE = new RegExp(`(\\S*${TEST_PATH_SOURCE}) \\(crashed: [^)]+\\)`);
|
||
const TERMINAL_SUMMARY_CAPTURE_RE = /^Ran (\d+) tests? across (\d+) files?\. \[/;
|
||
/** Substrings that must reach the console even in the default quiet mode. */
|
||
const CONSOLE_ALWAYS_MARKERS = ['error:', 'panic:', 'Unhandled error', 'crashed'] as const;
|
||
|
||
export type StreamOrigin = 'stdout' | 'stderr';
|
||
|
||
export interface FreeRunFailure {
|
||
/** Planned relative path when attributable, else the raw header path, else null. */
|
||
file: string | null;
|
||
testName: string;
|
||
}
|
||
|
||
export interface FreeRunReport {
|
||
testsRan: number | null;
|
||
filesRan: number | null;
|
||
sawTerminalSummary: boolean;
|
||
/** Deduped `(fail)` lines in arrival order, attributed to the current file header. */
|
||
failures: FreeRunFailure[];
|
||
failedTests: number;
|
||
unreportedFailures: number;
|
||
/** Files that crashed a worker (bun retries once; a second crash is final). Deduped. */
|
||
crashedFiles: string[];
|
||
/**
|
||
* "# Unhandled error between tests" markers, attributed to the chunk they
|
||
* appeared in. These fail the shard via the strict classifier but produce
|
||
* NO (fail) lines — without surfacing them here, the epilogue reads
|
||
* "FAIL — 0 failing test(s)" and the culprit is undiscoverable from CI
|
||
* output (first Windows lane run: a module-load throw in skill-census).
|
||
*/
|
||
unhandledErrors: Array<{ file: string | null }>;
|
||
/**
|
||
* Wedge-suspect heuristic for a wall-timeout kill: files whose header was
|
||
* seen but whose chunk never ENDED (chunk end = the next file's header, or
|
||
* a final crash marker) before the terminal summary — i.e. "started but
|
||
* never produced a result chunk end". Result lines deliberately do NOT end
|
||
* a chunk: a file that printed a fail and then wedged stays listed. Known
|
||
* limits of the approximation:
|
||
* - Serial (--shard CI path): bun streams live but prints a file's header
|
||
* lazily, on its first output line — a wedged file that printed ANY
|
||
* line is listed; a fully silent wedge is not.
|
||
* - Parallel (full-suite path): bun buffers a file's whole chunk until it
|
||
* COMPLETES, so a wedged file usually never prints a header (see
|
||
* filesWithNoOutput), and the LAST flushed chunk before the kill has no
|
||
* closing header, so one completed noisy file can be over-listed.
|
||
*/
|
||
inFlight: string[];
|
||
/** Planned files never observed in the stream (silent passers + never-flushed wedges). */
|
||
filesWithNoOutput: number;
|
||
}
|
||
|
||
interface FileProgress {
|
||
headerSeen: boolean;
|
||
/** The file's chunk ended: a later file's header arrived, or it crashed out. */
|
||
ended: boolean;
|
||
}
|
||
|
||
/**
|
||
* Incrementally consumes the child's stdout/stderr (chunk boundaries need not
|
||
* align to lines), attributing results to files and forwarding only
|
||
* always-visible lines to `forward` (omit `forward` for verbose/quiet modes —
|
||
* attribution still runs so the epilogue works in every mode).
|
||
*/
|
||
export class FreeRunReporter {
|
||
private readonly decoders: Record<StreamOrigin, StringDecoder> = {
|
||
stdout: new StringDecoder('utf8'),
|
||
stderr: new StringDecoder('utf8'),
|
||
};
|
||
private readonly pending: Record<StreamOrigin, string> = { stdout: '', stderr: '' };
|
||
private readonly plannedSet: Set<string>;
|
||
private readonly canonicalCache = new Map<string, string>();
|
||
private readonly progress = new Map<string, FileProgress>();
|
||
private readonly failureKeys = new Set<string>();
|
||
private readonly failures: FreeRunFailure[] = [];
|
||
private namedFailureCount = 0;
|
||
private reportedFailedTests = 0;
|
||
private readonly failureSummary = new BunFailureSummaryParser();
|
||
private readonly crashed = new Set<string>();
|
||
private currentFile: string | null = null;
|
||
private inRecap = false;
|
||
private readonly unhandled: Array<{ file: string | null }> = [];
|
||
private testsRan: number | null = null;
|
||
private filesRan: number | null = null;
|
||
private sawSummary = false;
|
||
|
||
constructor(
|
||
private readonly plannedFiles: string[],
|
||
private readonly forward?: (text: string, origin: StreamOrigin) => void,
|
||
) {
|
||
this.plannedSet = new Set(plannedFiles.map(normalizeRelativePath));
|
||
}
|
||
|
||
write(chunk: Uint8Array | string, origin: StreamOrigin): void {
|
||
this.pending[origin] += typeof chunk === 'string'
|
||
? chunk
|
||
: this.decoders[origin].write(Buffer.from(chunk));
|
||
let newline = this.pending[origin].indexOf('\n');
|
||
while (newline !== -1) {
|
||
this.handleLine(this.pending[origin].slice(0, newline), origin);
|
||
this.pending[origin] = this.pending[origin].slice(newline + 1);
|
||
newline = this.pending[origin].indexOf('\n');
|
||
}
|
||
}
|
||
|
||
/** Flush partial trailing lines (a stream killed mid-line still classifies). */
|
||
end(): void {
|
||
for (const origin of ['stdout', 'stderr'] as const) {
|
||
this.pending[origin] += this.decoders[origin].end();
|
||
if (this.pending[origin].length > 0) this.handleLine(this.pending[origin], origin);
|
||
this.pending[origin] = '';
|
||
}
|
||
}
|
||
|
||
report(): FreeRunReport {
|
||
const inFlight = this.sawSummary
|
||
? []
|
||
: [...this.progress.entries()]
|
||
.filter(([, p]) => p.headerSeen && !p.ended)
|
||
.map(([file]) => file)
|
||
.sort();
|
||
return {
|
||
testsRan: this.testsRan,
|
||
filesRan: this.filesRan,
|
||
sawTerminalSummary: this.sawSummary,
|
||
failures: [...this.failures],
|
||
failedTests: Math.max(this.reportedFailedTests, this.failures.length),
|
||
unreportedFailures: Math.max(0, this.reportedFailedTests - this.namedFailureCount),
|
||
crashedFiles: [...this.crashed].sort(),
|
||
unhandledErrors: [...this.unhandled],
|
||
inFlight,
|
||
filesWithNoOutput: this.plannedFiles.filter((f) => !this.progress.has(normalizeRelativePath(f))).length,
|
||
};
|
||
}
|
||
|
||
private handleLine(rawLine: string, origin: StreamOrigin): void {
|
||
// GitHub Actions: bun wraps each file's section in ::group::<header>.
|
||
// Without stripping, the real header fails FILE_HEADER_RE, failures get
|
||
// attributed to the PREVIOUS file, and the terminal recap's re-printed
|
||
// (fail) lines land under a second phantom file (observed on the first
|
||
// Linux run: 5 real failures reported as 10 across 2 files).
|
||
const line = stripAnsiLine(rawLine).replace(/^::group::/, '');
|
||
let visible = false;
|
||
const failedCount = this.failureSummary.consume(line, origin);
|
||
if (failedCount !== null) {
|
||
this.reportedFailedTests = Math.max(this.reportedFailedTests, failedCount);
|
||
visible = failedCount > 0;
|
||
}
|
||
|
||
// Bun's terminal recap ("N tests failed:") re-prints every (fail) line
|
||
// WITHOUT re-printing file headers. Attributing those to the stale
|
||
// currentFile invented a phantom failing file on the first Linux run
|
||
// (5 real failures reported as 10 across 2 files, one innocent).
|
||
if (/^\d+ tests? failed:$/.test(line)) {
|
||
this.inRecap = true;
|
||
if (this.currentFile) this.progressFor(this.currentFile).ended = true;
|
||
this.currentFile = null;
|
||
}
|
||
|
||
if (line === '# Unhandled error between tests') {
|
||
this.unhandled.push({ file: this.currentFile });
|
||
}
|
||
|
||
const header = FILE_HEADER_RE.exec(line);
|
||
if (header) {
|
||
const file = this.canonicalize(header[1]);
|
||
// A new header ends the previous file's chunk — that file is no longer
|
||
// a wedge suspect. (Bun 1.3.x prints NO (pass) lines, so chunk
|
||
// delimiters, not result lines, are the completion signal.)
|
||
if (this.currentFile && this.currentFile !== file) this.progressFor(this.currentFile).ended = true;
|
||
this.currentFile = file;
|
||
this.progressFor(file).headerSeen = true;
|
||
} else {
|
||
const fail = parseBunFailureResult(line);
|
||
const retry = fail ? null : CRASH_RETRY_RE.exec(line);
|
||
const final = fail || retry ? null : CRASH_FINAL_RE.exec(line);
|
||
if (fail) {
|
||
visible = true;
|
||
// In the recap, a (fail) line only records a failure the main run
|
||
// somehow never attributed (belt and braces); known names dedupe.
|
||
const recapDuplicate = this.inRecap
|
||
&& this.failures.some((f) => f.testName === fail);
|
||
if (!recapDuplicate) this.namedFailureCount += 1;
|
||
const key = `${this.currentFile ?? ''}\u0000${fail}`;
|
||
if (!recapDuplicate && !this.failureKeys.has(key)) {
|
||
this.failureKeys.add(key);
|
||
this.failures.push({ file: this.currentFile, testName: fail });
|
||
}
|
||
} else if (retry) {
|
||
// The file will run again — a crash+retry does not end its chunk.
|
||
visible = true;
|
||
this.crashed.add(this.canonicalize(retry[1]));
|
||
} else if (final) {
|
||
visible = true;
|
||
const file = this.canonicalize(final[1]);
|
||
this.crashed.add(file);
|
||
this.progressFor(file).ended = true;
|
||
} else {
|
||
const summary = TERMINAL_SUMMARY_CAPTURE_RE.exec(line);
|
||
if (summary) {
|
||
visible = true;
|
||
this.sawSummary = true;
|
||
this.testsRan = Number.parseInt(summary[1], 10);
|
||
this.filesRan = Number.parseInt(summary[2], 10);
|
||
}
|
||
}
|
||
}
|
||
|
||
if (!visible) visible = CONSOLE_ALWAYS_MARKERS.some((marker) => line.includes(marker));
|
||
if (visible && this.forward) this.forward(`${rawLine.replace(/\r$/, '')}\n`, origin);
|
||
}
|
||
|
||
private progressFor(file: string): FileProgress {
|
||
let entry = this.progress.get(file);
|
||
if (!entry) {
|
||
entry = { headerSeen: false, ended: false };
|
||
this.progress.set(file, entry);
|
||
}
|
||
return entry;
|
||
}
|
||
|
||
/**
|
||
* Map a printed path back to its planned relative path. Bun prints paths
|
||
* relative to the child's (real)cwd, so a symlinked cwd (macOS /tmp) yields
|
||
* `../..`-prefixed forms — strip the prefix and suffix-match.
|
||
*/
|
||
private canonicalize(printedPath: string): string {
|
||
const cached = this.canonicalCache.get(printedPath);
|
||
if (cached) return cached;
|
||
const stripped = normalizeRelativePath(printedPath).replace(/^(?:\.{1,2}\/)+/, '');
|
||
let resolved = stripped;
|
||
if (!this.plannedSet.has(stripped)) {
|
||
const match = this.plannedFiles.find(
|
||
(planned) => stripped.endsWith(`/${planned}`) || planned.endsWith(`/${stripped}`),
|
||
);
|
||
if (match) resolved = match;
|
||
}
|
||
this.canonicalCache.set(printedPath, resolved);
|
||
return resolved;
|
||
}
|
||
}
|
||
|
||
/**
|
||
* The stable post-run epilogue. Success is one line; failure names every
|
||
* failing test (deduped, attributed) and crashed worker; a wall-timeout kill
|
||
* additionally prints the wedge-suspect list (see FreeRunReport.inFlight for
|
||
* the heuristic and its limits).
|
||
*/
|
||
export function buildRunEpilogue(
|
||
status: FreeShardStatus,
|
||
report: FreeRunReport,
|
||
elapsedMs: number,
|
||
logPath: string,
|
||
): string[] {
|
||
const seconds = Math.round(elapsedMs / 1000);
|
||
if (status === 'passed') {
|
||
return [
|
||
`[test:free] PASS — ${report.testsRan ?? '?'} tests, ${report.filesRan ?? '?'} files, ${seconds}s. Full log: ${logPath}`,
|
||
];
|
||
}
|
||
const failingFiles = new Set(report.failures.map((f) => f.file ?? '(unattributed)'));
|
||
const lines = [
|
||
`[test:free] FAIL — ${report.failedTests} failing test(s) in ${failingFiles.size} ${report.unreportedFailures > 0 ? 'identified ' : ''}file(s), `
|
||
+ `${report.crashedFiles.length} crashed worker(s)${report.unhandledErrors.length > 0 ? `, ${report.unhandledErrors.length} unhandled error(s) between tests` : ''}. Full log: ${logPath}`,
|
||
];
|
||
for (const failure of report.failures) {
|
||
lines.push(` ✗ ${failure.file ?? '(unattributed)'} — ${failure.testName}`);
|
||
}
|
||
if (report.unreportedFailures > 0) {
|
||
lines.push(` ⚠ ${report.unreportedFailures} failure(s) reported without named result lines`);
|
||
}
|
||
for (const file of report.crashedFiles) {
|
||
lines.push(` ⚠ crashed+retried: ${file}`);
|
||
}
|
||
for (const u of report.unhandledErrors) {
|
||
lines.push(` ⚠ unhandled error between tests (around ${u.file ?? 'unknown file'})`);
|
||
}
|
||
if (status === 'timed-out') {
|
||
if (report.inFlight.length > 0) {
|
||
lines.push(` ⏱ in flight at kill: ${report.inFlight.join(', ')}`);
|
||
} else {
|
||
lines.push(
|
||
' ⏱ in flight at kill: unknown — no open file chunk was observed '
|
||
+ '(bun --parallel buffers a file\'s output until it completes, so a silent wedge never prints); '
|
||
+ `${report.filesWithNoOutput} planned file(s) produced no output before the kill.`,
|
||
);
|
||
}
|
||
}
|
||
return lines;
|
||
}
|
||
|
||
export type FreeShardStatus = 'passed' | 'failed' | 'timed-out';
|
||
|
||
// ─── Flake ledger (WS1 telemetry) ───────────────────────────────────────────
|
||
// Single-writer JSONL: ONLY this parent runner appends (never shards, never
|
||
// tests — no concurrent-append hazard by construction). CI points
|
||
// GSTACK_FLAKE_LEDGER at $RUNNER_TEMP and uploads it as an artifact every
|
||
// run, so repeat offenders become an enumerable series instead of console
|
||
// scrollback. Fail-open with a loud stderr warning: a broken ledger must
|
||
// never red the only required lane.
|
||
|
||
export interface FlakeLedgerEntry {
|
||
ts: string;
|
||
runner: 'free';
|
||
kind: 'flaky-pass';
|
||
file: string;
|
||
/** Shard the original failure surfaced in, when attributable. */
|
||
shard?: number;
|
||
/** Code-state attribution (review finding): without branch/sha the series
|
||
* can't tie an entry to the state that produced it, and the WS16
|
||
* promotion evidence needs exactly that. */
|
||
branch?: string;
|
||
git_sha?: string;
|
||
}
|
||
|
||
export function flakeLedgerPath(env: NodeJS.ProcessEnv = process.env): string {
|
||
if (env.GSTACK_FLAKE_LEDGER) return env.GSTACK_FLAKE_LEDGER;
|
||
// Local default: per-PROJECT, not the machine-global tmpdir — sibling
|
||
// Conductor worktrees of DIFFERENT repos must not interleave into one
|
||
// series (review finding). CI always sets GSTACK_FLAKE_LEDGER explicitly.
|
||
try {
|
||
const slug = spawnSync('bash', ['-c', '~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null'], { stdio: 'pipe', timeout: 3000 })
|
||
.stdout?.toString().match(/^SLUG=(.+)$/m)?.[1];
|
||
if (slug) {
|
||
const dir = path.join(resolveStateRoot(env), 'projects', slug);
|
||
fs.mkdirSync(dir, { recursive: true });
|
||
return path.join(dir, 'flake-ledger.jsonl');
|
||
}
|
||
} catch { /* fall through */ }
|
||
return path.join(os.tmpdir(), 'gstack-flake-ledger.jsonl');
|
||
}
|
||
|
||
export function appendFlakeLedger(
|
||
entries: FlakeLedgerEntry[],
|
||
ledgerPath: string,
|
||
warn: (line: string) => void = (line) => console.error(line),
|
||
): boolean {
|
||
if (entries.length === 0) return true;
|
||
try {
|
||
fs.mkdirSync(path.dirname(ledgerPath), { recursive: true });
|
||
fs.appendFileSync(ledgerPath, entries.map((e) => JSON.stringify(e)).join('\n') + '\n');
|
||
return true;
|
||
} catch (error) {
|
||
warn(`[test:free] WARNING: could not append flake ledger at ${ledgerPath} `
|
||
+ `(${error instanceof Error ? error.message : String(error)}) — flaky-pass telemetry lost for this run, verdict unaffected`);
|
||
return false;
|
||
}
|
||
}
|
||
|
||
export interface FreeShardOutcome {
|
||
shard: number;
|
||
files: string[];
|
||
status: FreeShardStatus;
|
||
exitCode: number | null;
|
||
elapsedMs: number;
|
||
groupPid: number | null;
|
||
/** Required by CI receipts; optional for existing local caller fixtures. */
|
||
summary?: Pick<FreeRunReport, 'testsRan' | 'filesRan' | 'sawTerminalSummary'>;
|
||
/**
|
||
* Repo-relative files with attributed test failures or crashes, deduped.
|
||
* Feeds the opt-in flaky retry pass (GSTACK_FREE_RETRY_FLAKY) — empty on
|
||
* pass, and empty when every failure was unattributed (retry would be
|
||
* meaningless without knowing what to re-run).
|
||
*/
|
||
failingFiles: string[];
|
||
/**
|
||
* Count of failure evidence the retry pass CANNOT re-run by file: fail
|
||
* lines seen before any file-chunk header, unhandled errors between tests,
|
||
* and a truncated run (no terminal summary). Nonzero vetoes the flaky
|
||
* retry for the whole run — retrying only failingFiles would re-run a
|
||
* subset and mask the rest as a FLAKY-PASS, re-opening the silent-truncation
|
||
* hole the strict classifier exists to close.
|
||
*/
|
||
unattributedFailures: number;
|
||
}
|
||
|
||
export interface ShardCommand {
|
||
command: string;
|
||
args: string[];
|
||
}
|
||
|
||
export interface RunFreeShardOptions {
|
||
/** External wall-clock deadline; on expiry the child's process GROUP is SIGKILLed. */
|
||
wallTimeoutMs?: number;
|
||
rootDir?: string;
|
||
env?: NodeJS.ProcessEnv;
|
||
/** Pass bun's --parallel. No production caller today (see BuildShardArgsOptions.parallel). */
|
||
parallel?: boolean;
|
||
/** Override the spawned command. Tests inject fake pass/fail/slow commands. */
|
||
commandFor?: (files: string[]) => ShardCommand;
|
||
/** Suppress ALL child output from the console (tests). The classifier and the log file still see every byte. */
|
||
quiet?: boolean;
|
||
/** Forward the full child stream to the console (legacy firehose). Default: the quiet filtered console. */
|
||
verbose?: boolean;
|
||
/**
|
||
* Console sink for child-stream output (tests inject to assert quiet vs
|
||
* verbose behavior). Default: process.stdout / process.stderr by origin.
|
||
* Runner-owned [test:free] lines go through `log`, not this sink.
|
||
*/
|
||
consoleWrite?: (text: string) => void;
|
||
/** Per-run full-stream log path (tests inject). Default: a private retained file under .context/free-test-logs. */
|
||
logFilePath?: string;
|
||
log?: (line: string) => void;
|
||
/** Home tripwire for this shard. Default: guard the real home and name this shard's files. */
|
||
homeGuard?: FreeHomeGuardFactory;
|
||
}
|
||
|
||
const EPILOGUE_WORD: Record<FreeShardStatus, string> = {
|
||
passed: 'pass',
|
||
failed: 'fail',
|
||
'timed-out': 'timed-out',
|
||
};
|
||
|
||
function trackShardBrowser(stateDir: string, env: NodeJS.ProcessEnv) {
|
||
class BrowserCleanupError extends Error {}
|
||
class CaptureStopped extends Error {}
|
||
type Identity = { pid: number; parent: number; start: string; daemon: number; root: boolean };
|
||
type Capture = { abort: AbortController; deadline: number; probes: Set<Promise<unknown>>; records?: [string, string] };
|
||
const identities = new Map<number, Identity>();
|
||
const nativeStarts = new Map<string, string>();
|
||
const interrupted = new Set<string>();
|
||
const errors = new Set<string>();
|
||
const stateFile = env.BROWSE_STATE_FILE!;
|
||
let stopping = false;
|
||
let ready = true;
|
||
let closed = false;
|
||
let forced = false;
|
||
let cancellation = false;
|
||
let deadline = Infinity;
|
||
let forceAt = Infinity;
|
||
let active: Capture | null = null;
|
||
let pending: Promise<void> | null = null;
|
||
let observed = false;
|
||
let alive = true;
|
||
|
||
const check = (capture: Capture) => {
|
||
if (closed || capture.abort.signal.aborted || active !== capture || (!forced && performance.now() >= forceAt)) throw new CaptureStopped();
|
||
if (performance.now() >= Math.min(capture.deadline, deadline)) throw new BrowserCleanupError('browser ownership deadline exceeded');
|
||
};
|
||
const probe = async (capture: Capture, command: string, args: string[], timeout: number) => {
|
||
check(capture);
|
||
const remaining = Math.min(timeout, capture.deadline - performance.now() - 100, deadline - performance.now() - 100);
|
||
if (remaining <= 0) throw new BrowserCleanupError('browser ownership deadline exceeded');
|
||
const task = new Promise<{ status: number | null; stdout: string }>((resolve, reject) => {
|
||
const child = spawn(command, args, { detached: true, stdio: ['ignore', 'pipe', 'ignore'], windowsHide: true });
|
||
let output = '';
|
||
let failed = false;
|
||
let done = false;
|
||
let reaper: ReturnType<typeof setTimeout> | undefined;
|
||
const finish = (status: number | null) => {
|
||
if (done) return;
|
||
done = true;
|
||
clearTimeout(timer);
|
||
clearTimeout(reaper);
|
||
capture.abort.signal.removeEventListener('abort', stop);
|
||
child.stdout?.destroy();
|
||
child.unref();
|
||
if (failed) reject(new BrowserCleanupError('browser identity probe did not complete'));
|
||
else resolve({ status, stdout: output });
|
||
};
|
||
const stop = () => {
|
||
if (done || failed) return;
|
||
failed = true;
|
||
killProcessGroup(child, 'SIGKILL');
|
||
reaper = setTimeout(() => finish(null), 100);
|
||
};
|
||
const timer = setTimeout(() => { if (!closed) errors.add('browser identity probe did not complete'); stop(); }, Math.max(1, Math.min(remaining, forced ? Infinity : forceAt - performance.now())));
|
||
capture.abort.signal.addEventListener('abort', stop, { once: true });
|
||
child.once('error', () => { failed = true; finish(null); });
|
||
child.once('close', finish);
|
||
child.stdout?.on('data', chunk => {
|
||
output += chunk.toString();
|
||
if (output.length > 65536) stop();
|
||
});
|
||
});
|
||
capture.probes.add(task);
|
||
try {
|
||
const result = await task;
|
||
check(capture);
|
||
return result;
|
||
} finally { capture.probes.delete(task); }
|
||
};
|
||
|
||
const failure = (error: unknown) => {
|
||
if (error instanceof CaptureStopped) return;
|
||
const code = (error as NodeJS.ErrnoException)?.code;
|
||
const file = (error as NodeJS.ErrnoException & { path?: string })?.path;
|
||
const pid = typeof file === 'string' ? /^\/proc\/(\d+)\//.exec(file)?.[1] : undefined;
|
||
if (pid && identities.has(Number(pid)) && ['EACCES', 'EPERM', 'ENOENT', 'ESRCH'].includes(code ?? '')) return;
|
||
errors.add(error instanceof BrowserCleanupError ? error.message : 'browser ownership unavailable');
|
||
};
|
||
|
||
const inspectLinux = (pid: number): Omit<Identity, 'daemon' | 'root'> | null => {
|
||
try {
|
||
const raw = fs.readFileSync(`/proc/${pid}/stat`, 'utf8');
|
||
const fields = raw.slice(raw.lastIndexOf(') ') + 2).trim().split(/\s+/);
|
||
if (!/^\d+$/.test(fields[19] ?? '')) throw new BrowserCleanupError('process start identity unavailable');
|
||
return fields[0] === 'Z' || fields[0] === 'X' ? null
|
||
: { pid, parent: Number(fields[1]), start: fields[19] };
|
||
} catch (error) {
|
||
if ((error as NodeJS.ErrnoException).code === 'ENOENT' || (error as NodeJS.ErrnoException).code === 'ESRCH') return null;
|
||
throw error;
|
||
}
|
||
};
|
||
const inspect = async (capture: Capture, pid: number): Promise<Omit<Identity, 'daemon' | 'root'> | null> => {
|
||
check(capture);
|
||
if (!Number.isSafeInteger(pid) || pid <= 1) throw new BrowserCleanupError('invalid process identity');
|
||
if (process.platform === 'linux') return inspectLinux(pid);
|
||
const result = await probe(capture, 'ps', ['-p', String(pid), '-o', 'ppid=,stat=,lstart='], 500);
|
||
if (result.status === 1 && !result.stdout.trim()) return null;
|
||
if (result.status !== 0) throw new BrowserCleanupError('process identity unavailable');
|
||
const fields = result.stdout.trim().split(/\s+/);
|
||
return fields[1]?.startsWith('Z') ? null : { pid, parent: Number(fields[0]), start: fields.slice(2).join(' ') };
|
||
};
|
||
const required = [`BROWSE_STATE_FILE=${stateFile}`, `GSTACK_FREE_SHARD_ID=${env.GSTACK_FREE_SHARD_ID}`];
|
||
const boundLinux = (pid: number) => {
|
||
try {
|
||
const values = fs.readFileSync(`/proc/${pid}/environ`, 'utf8').split('\0');
|
||
return required.every(value => values.includes(value));
|
||
} catch (error) {
|
||
if ((error as NodeJS.ErrnoException).code === 'ENOENT' || (error as NodeJS.ErrnoException).code === 'ESRCH') return false;
|
||
throw error;
|
||
}
|
||
};
|
||
const bound = async (capture: Capture, pid: number): Promise<boolean> => {
|
||
check(capture);
|
||
if (process.platform === 'linux') return boundLinux(pid);
|
||
const [command, environment] = await Promise.all([
|
||
probe(capture, 'ps', ['-ww', '-p', String(pid), '-o', 'command='], 500),
|
||
probe(capture, 'ps', ['eww', '-p', String(pid), '-o', 'command='], 500),
|
||
]);
|
||
if (command.status !== 0 || environment.status !== 0) return false;
|
||
const prefix = command.stdout.trim();
|
||
if (!prefix || !environment.stdout.trim().startsWith(prefix + ' ')) return false;
|
||
const values = ' ' + environment.stdout.trim().slice(prefix.length).trim() + ' ';
|
||
return required.every(value => values.includes(' ' + value + ' '));
|
||
};
|
||
const live = async (capture: Capture, identity: Identity): Promise<boolean> => {
|
||
const current = await inspect(capture, identity.pid);
|
||
check(capture);
|
||
if (!current) return false;
|
||
if (!current.start || current.start !== identity.start) {
|
||
errors.add('captured process identity was replaced');
|
||
return false;
|
||
}
|
||
return true;
|
||
};
|
||
const record = (file: string): any => { try { // a record a shutting-down daemon removes mid-read is absent
|
||
if (!fs.existsSync(file)) return null;
|
||
const info = fs.lstatSync(file);
|
||
if (!info.isFile() || info.size > 65536) throw new BrowserCleanupError('unsafe browser state record');
|
||
return JSON.parse(fs.readFileSync(file, 'utf8'));
|
||
} catch (error) { if ((error as NodeJS.ErrnoException).code === 'ENOENT') return null; throw error; } };
|
||
const nativeStart = async (capture: Capture, identity: Identity): Promise<string> => {
|
||
check(capture);
|
||
const key = `${identity.pid}:${identity.start}`;
|
||
const recorded = nativeStarts.get(key);
|
||
if (recorded) return recorded;
|
||
if (!await live(capture, identity)) throw new BrowserCleanupError('process exited before its native identity was captured');
|
||
const result = await probe(capture, 'ps', ['-p', String(identity.pid), '-o', 'lstart='], 2000);
|
||
const value = result.status === 0 ? result.stdout.trim().replace(/\s+/g, ' ') : '';
|
||
if (!value || !await live(capture, identity)) throw new BrowserCleanupError('native process identity unavailable');
|
||
check(capture);
|
||
nativeStarts.set(key, value);
|
||
return value;
|
||
};
|
||
const remember = (capture: Capture, identity: Identity) => {
|
||
check(capture);
|
||
const previous = identities.get(identity.pid);
|
||
if (previous && previous.start !== identity.start) throw new BrowserCleanupError('captured process identity was replaced');
|
||
identities.set(identity.pid, identity);
|
||
};
|
||
const descendants = async (capture: Capture, parent: Identity, visited: Set<number>): Promise<void> => {
|
||
check(capture);
|
||
if (visited.has(parent.pid)) return;
|
||
visited.add(parent.pid);
|
||
if (visited.size > 256) throw new BrowserCleanupError('owned browser process limit exceeded');
|
||
if (!await live(capture, parent)) return;
|
||
let children: number[];
|
||
if (process.platform === 'linux') {
|
||
try {
|
||
children = fs.readFileSync(`/proc/${parent.pid}/task/${parent.pid}/children`, 'utf8').trim().split(/\s+/).filter(Boolean).map(Number);
|
||
} catch (error) {
|
||
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return;
|
||
throw error;
|
||
}
|
||
} else {
|
||
const result = await probe(capture, 'pgrep', ['-P', String(parent.pid)], 500);
|
||
if (result.status !== 0 && result.status !== 1) throw new BrowserCleanupError('owned child identities unavailable');
|
||
children = result.stdout.trim().split(/\s+/).filter(Boolean).map(Number);
|
||
}
|
||
for (const pid of children) {
|
||
const child = await inspect(capture, pid);
|
||
if (!child || child.parent !== parent.pid || !await live(capture, parent)) continue;
|
||
const identity = { ...child, daemon: parent.daemon, root: false };
|
||
remember(capture, identity);
|
||
await descendants(capture, identity, visited);
|
||
}
|
||
};
|
||
const capture = async (operation: Capture) => {
|
||
check(operation);
|
||
ready = true;
|
||
if (process.platform === 'win32') return;
|
||
try {
|
||
if (fs.realpathSync(stateDir) !== stateDir) throw new BrowserCleanupError('shard directory was replaced');
|
||
const directory = path.dirname(stateFile);
|
||
if (fs.existsSync(directory) && (!fs.lstatSync(directory).isDirectory()
|
||
|| fs.lstatSync(directory).isSymbolicLink())) throw new BrowserCleanupError('browser directory was replaced');
|
||
const state = record(stateFile);
|
||
if (state?.pid !== undefined) {
|
||
const current = await inspect(operation, state.pid);
|
||
if (current) {
|
||
if (!await bound(operation, current.pid)) {
|
||
if (!await inspect(operation, current.pid)) return;
|
||
throw new BrowserCleanupError('daemon is not bound to this shard');
|
||
}
|
||
remember(operation, { ...current, daemon: current.pid, root: true });
|
||
} else if (!identities.has(state.pid)) {
|
||
throw new BrowserCleanupError('daemon exited before ownership was captured');
|
||
}
|
||
}
|
||
for (const identity of identities.values()) if (identity.root) await descendants(operation, identity, new Set());
|
||
const validateChild = async (pid: unknown, start: unknown, daemon: unknown) => {
|
||
check(operation);
|
||
if (!Number.isSafeInteger(pid) || (pid as number) <= 1) throw new BrowserCleanupError('invalid browser child identity');
|
||
const identity = identities.get(pid as number);
|
||
if (!identity || identity.daemon !== daemon || identity.root) throw new BrowserCleanupError('browser child ownership is unconfirmed');
|
||
if (await live(operation, identity) && (typeof start !== 'string' || !start
|
||
|| await nativeStart(operation, identity) !== start.replace(/\s+/g, ' '))) throw new BrowserCleanupError('browser child identity was replaced');
|
||
};
|
||
const agent = record(path.join(directory, 'terminal-agent-pid'));
|
||
const validations: Promise<unknown>[] = [];
|
||
if (agent) {
|
||
const daemon = identities.get(agent.ownerPid);
|
||
if (!daemon?.root || (state?.pid !== undefined && agent.ownerPid !== state.pid)) throw new BrowserCleanupError('terminal owner is unconfirmed');
|
||
validations.push(nativeStart(operation, daemon).then(start => {
|
||
if (start !== agent.ownerStartTime?.replace(/\s+/g, ' ')) throw new BrowserCleanupError('terminal owner is unconfirmed');
|
||
}));
|
||
if (agent.pid === 0) ready = false;
|
||
else validations.push(validateChild(agent.pid, agent.startTime, agent.ownerPid));
|
||
}
|
||
if (state?.chromiumPid !== undefined) validations.push(validateChild(state.chromiumPid, state.chromiumStartTime, state.pid));
|
||
const results = await Promise.allSettled(validations);
|
||
check(operation);
|
||
for (const result of results) if (result.status === 'rejected') throw result.reason;
|
||
operation.records = [JSON.stringify(state), JSON.stringify(agent)];
|
||
} catch (error) {
|
||
check(operation);
|
||
ready = false;
|
||
failure(error);
|
||
}
|
||
};
|
||
const signalOwned = async (operation: Capture, force: boolean) => {
|
||
for (const identity of identities.values()) {
|
||
if (!force && (!identity.root || !ready || errors.size > 0)) continue;
|
||
const key = `${identity.pid}:${identity.start}`;
|
||
if (!force && interrupted.has(key)) continue;
|
||
try {
|
||
if (!await live(operation, identity)) continue;
|
||
if (identity.root && !await bound(operation, identity.pid)) {
|
||
if (!await live(operation, identity)) continue;
|
||
throw new BrowserCleanupError('daemon environment changed before termination');
|
||
}
|
||
if (!await live(operation, identity)) continue;
|
||
check(operation);
|
||
if (!force && (!operation.records || JSON.stringify(record(stateFile)) !== operation.records[0]
|
||
|| JSON.stringify(record(path.join(path.dirname(stateFile), 'terminal-agent-pid'))) !== operation.records[1])) {
|
||
ready = false;
|
||
continue;
|
||
}
|
||
process.kill(identity.pid, force ? 'SIGKILL' : 'SIGINT');
|
||
if (!force) interrupted.add(key);
|
||
} catch (error) {
|
||
check(operation);
|
||
if ((error as NodeJS.ErrnoException).code !== 'ESRCH') failure(error);
|
||
}
|
||
}
|
||
};
|
||
const forceLinux = () => {
|
||
if (process.platform !== 'linux') return;
|
||
for (const identity of identities.values()) {
|
||
try {
|
||
const current = inspectLinux(identity.pid);
|
||
if (!current) continue;
|
||
if (current.start !== identity.start) throw new BrowserCleanupError('captured process identity was replaced');
|
||
if (identity.root && !boundLinux(identity.pid)) {
|
||
if (!inspectLinux(identity.pid)) continue;
|
||
throw new BrowserCleanupError('daemon environment changed before termination');
|
||
}
|
||
if (inspectLinux(identity.pid)?.start === identity.start) process.kill(identity.pid, 'SIGKILL');
|
||
} catch (error) {
|
||
if ((error as NodeJS.ErrnoException).code !== 'ESRCH') failure(error);
|
||
}
|
||
}
|
||
};
|
||
const enqueue = () => {
|
||
if (closed || pending || process.platform === 'win32') return;
|
||
const operation: Capture = { abort: new AbortController(), deadline: Math.min(performance.now() + 10000, deadline), probes: new Set() };
|
||
active = operation;
|
||
pending = (async () => {
|
||
try {
|
||
if (!forced) await capture(operation);
|
||
if (stopping) {
|
||
await signalOwned(operation, forced);
|
||
let stillAlive = false;
|
||
for (const identity of identities.values()) if (await live(operation, identity)) stillAlive = true;
|
||
check(operation);
|
||
alive = stillAlive;
|
||
observed = true;
|
||
}
|
||
} catch (error) {
|
||
if (!closed && !operation.abort.signal.aborted) failure(error);
|
||
} finally {
|
||
operation.abort.abort();
|
||
await Promise.allSettled([...operation.probes]);
|
||
if (active === operation) active = null;
|
||
}
|
||
})().finally(() => { pending = null; });
|
||
};
|
||
const signal = (force: boolean) => {
|
||
if (closed) return;
|
||
if (!cancellation) {
|
||
cancellation = true;
|
||
stopping = true;
|
||
deadline = Math.min(deadline, performance.now() + 5500);
|
||
forceAt = Math.min(forceAt, performance.now() + 5000);
|
||
active?.abort.abort();
|
||
}
|
||
if (force) {
|
||
forced = true;
|
||
active?.abort.abort();
|
||
forceLinux();
|
||
}
|
||
if (pending) void pending.then(enqueue);
|
||
else enqueue();
|
||
};
|
||
const timer = setInterval(enqueue, process.platform === 'darwin' ? 1000 : 250);
|
||
timer.unref();
|
||
return {
|
||
signal,
|
||
async settle(): Promise<string | null> {
|
||
clearInterval(timer);
|
||
if (process.platform === 'win32') { closed = true; return null; }
|
||
stopping = true;
|
||
deadline = Math.min(deadline, performance.now() + 10000);
|
||
forceAt = Math.min(forceAt, performance.now() + 5000);
|
||
active?.abort.abort();
|
||
try {
|
||
while (true) {
|
||
if (performance.now() >= deadline) {
|
||
errors.add('owned browser settlement deadline exceeded');
|
||
break;
|
||
}
|
||
if (!forced && performance.now() >= forceAt) {
|
||
forced = true;
|
||
active?.abort.abort();
|
||
forceLinux();
|
||
}
|
||
if (pending) await pending;
|
||
enqueue();
|
||
if (pending) await pending;
|
||
if (observed && !alive) break;
|
||
await new Promise(resolve => setTimeout(resolve, Math.min(50, Math.max(0, deadline - performance.now()))));
|
||
}
|
||
} catch {
|
||
errors.add('owned browser settlement could not be verified');
|
||
} finally {
|
||
closed = true;
|
||
clearInterval(timer);
|
||
active?.abort.abort();
|
||
if (pending) await pending;
|
||
}
|
||
return errors.size ? [...errors].join('; ') : null;
|
||
},
|
||
};
|
||
}
|
||
|
||
/**
|
||
* Drain one child pipe into `onChunk`. Capture failures are recorded as data
|
||
* in `failures`, never thrown: the caller still waits for the child's real exit.
|
||
*/
|
||
function captureFreeStream(
|
||
stream: NodeJS.ReadableStream | null,
|
||
origin: StreamOrigin,
|
||
failures: Map<StreamOrigin, Error>,
|
||
onChunk: (chunk: Buffer | string) => void,
|
||
): Promise<void> {
|
||
return new Promise<void>((resolve) => {
|
||
if (!stream) {
|
||
failures.set(origin, new Error('configured pipe is missing'));
|
||
resolve();
|
||
return;
|
||
}
|
||
const readable = stream as NodeJS.ReadableStream & { readableEnded: boolean; destroyed: boolean; errored: Error | null };
|
||
let ended = readable.readableEnded;
|
||
const incomplete = (error?: Error | null): void => {
|
||
// A delayed error replaces the initial destroyed-stream diagnostic
|
||
// with its original cause.
|
||
if (error) failures.set(origin, error);
|
||
else if (!failures.has(origin)) failures.set(origin, new Error('stream closed before end'));
|
||
resolve();
|
||
};
|
||
// Even an already-destroyed pipe can emit error on the next tick.
|
||
stream.on('error', incomplete);
|
||
stream.once('end', () => { ended = true; resolve(); });
|
||
stream.once('close', () => {
|
||
if (!ended) incomplete(readable.errored);
|
||
else resolve();
|
||
});
|
||
stream.on('data', onChunk);
|
||
if (ended) resolve();
|
||
else if (readable.destroyed) incomplete(readable.errored);
|
||
});
|
||
}
|
||
|
||
/** Why a shard did not pass, on stderr (the epilogue repeats the names). */
|
||
function explainFreeVerdict(label: string, status: FreeShardStatus, facts: {
|
||
cleanupError: string | null; stateDir: string; evidenceComplete: boolean; exitCode: number | null;
|
||
summary: ReturnType<BunTestOutputClassifier['end']>; expectedFiles: number; wallTimeoutMs: number;
|
||
}): void {
|
||
const { summary, exitCode } = facts;
|
||
if (facts.cleanupError) console.error(`${label} shard cleanup or home containment failed: ${facts.cleanupError}; retained ${facts.stateDir}`);
|
||
if (status === 'timed-out') {
|
||
console.error(
|
||
`${label} exceeded the ${Math.round(facts.wallTimeoutMs / 1000)}s wall-clock deadline — `
|
||
+ 'killed the process group. Reporting as TIMED-OUT (distinct from failed).',
|
||
);
|
||
} else if (status === 'failed' && facts.evidenceComplete && (exitCode ?? 1) === 0) {
|
||
const reason = summary.failedTests > 0 || summary.unhandledBetweenTests > 0
|
||
? `reported ${summary.failedTests} failing test(s) and ${summary.unhandledBetweenTests} unhandled error(s) between tests`
|
||
: summary.terminalFileCounts.length === 0
|
||
? "never printed bun's terminal summary — the run was truncated (a process.exit fired mid-suite)"
|
||
: `bun's summary reported ${summary.terminalFileCounts.join(', ')} file(s), expected ${facts.expectedFiles}`;
|
||
console.error(`${label} exited 0 but ${reason}. Treating as FAILED.`);
|
||
} else if (status === 'failed' && (exitCode ?? 1) !== 0) {
|
||
console.error(`${label} failed with exit code ${exitCode ?? 'signal'}`);
|
||
}
|
||
}
|
||
|
||
/** The recovery step and, only when the failure scope is complete, a focused rerun. */
|
||
function logFreeRecovery(log: (line: string) => void, outcome: FreeShardOutcome, facts: {
|
||
cleanupError: string | null; logWriteFailed: boolean; captureIncomplete: boolean; rootDir: string;
|
||
}): void {
|
||
const problem = facts.cleanupError ? 'Owned-process cleanup or home containment is unconfirmed; inspect the reported paths and retained state before another run.'
|
||
: facts.logWriteFailed ? 'The evidence log could not be retained; repair the log destination before another run.'
|
||
: facts.captureIncomplete ? 'Evidence capture is incomplete; repair the stream or early exit before another run.'
|
||
: outcome.status === 'timed-out' ? 'Execution exceeded its deadline; inspect the last completed step before changing code or rerunning.'
|
||
: 'A test or module failed; the root cause is not established. Inspect the full log and repair the cause first.';
|
||
log(`[test:free] Recovery: ${problem} See docs/TESTING_INTERNALS.md.`);
|
||
const focused = outcome.failingFiles.filter(file => outcome.files.includes(file) && fs.existsSync(path.resolve(facts.rootDir, file)));
|
||
if (!outcome.unattributedFailures && focused.length) {
|
||
log(`[test:free] After repair, focused check: bun test ${focused.map(file => `'${file.replaceAll("'", "'\\''")}'`).join(' ')}`);
|
||
} else {
|
||
log('[test:free] No complete narrower failure scope is available; do not treat a subset rerun as complete coverage.');
|
||
}
|
||
}
|
||
|
||
/** One line per shard, printed after the run: `[test:free] shard i/N: M files, XXs, pass|fail|timed-out`. */
|
||
function shardEpilogue(outcome: FreeShardOutcome, totalShards: number): string {
|
||
return `[test:free] shard ${outcome.shard}/${totalShards}: ${outcome.files.length} files, `
|
||
+ `${Math.round(outcome.elapsedMs / 1000)}s, ${EPILOGUE_WORD[outcome.status]}`;
|
||
}
|
||
|
||
/**
|
||
* Run one shard (or the whole suite, in --parallel full-suite mode) in its own
|
||
* bun process and classify the result strictly.
|
||
*
|
||
* Verdict integrity: the child's exit code is never trusted alone. Output is
|
||
* fed through BunTestOutputClassifier, and strictTestExitCode requires bun's
|
||
* terminal summary to report EXACTLY the planned file count — a shard that
|
||
* exits 0 without the summary (mid-suite process.exit truncation), with
|
||
* `(fail)` result lines, or having run fewer files than planned is a FAILURE.
|
||
* This is enforced for injected fake commands too (unlike the paid runner),
|
||
* so tests can pin the summary-missing => failure backstop; fake passing
|
||
* commands must print a synthetic `Ran N tests across M files. [Xms]` line.
|
||
*
|
||
* Per-shard temp isolation: each spawned child gets its own throwaway TMPDIR
|
||
* (TEMP/TMP on Windows) so shards can't trip over each other's temp files.
|
||
* Deliberately NOT GSTACK_HOME: injecting one shared scratch home for a whole
|
||
* invocation made 6,900 tests share a MUTABLE state dir — config tests wrote
|
||
* keys into it and relink/update-check tests then read them (measured: 12
|
||
* cross-contamination failures on the first full run). Tests that need
|
||
* GSTACK_HOME isolation mkdtemp their own per test — the repo convention —
|
||
* and the hermetic-env machinery covers E2E children.
|
||
*/
|
||
export async function runFreeShard(
|
||
files: string[],
|
||
shardNumber: number,
|
||
totalShards: number,
|
||
options: RunFreeShardOptions = {},
|
||
): Promise<FreeShardOutcome> {
|
||
const log = options.log ?? ((line: string) => console.log(line));
|
||
const label = `[test:free] shard ${shardNumber}/${totalShards}`;
|
||
|
||
// Empty shard = fast no-op SUCCESS. Indices are stable for the CI matrix,
|
||
// so an unoccupied index must not fail or shift work to a different runner.
|
||
if (files.length === 0) {
|
||
const outcome: FreeShardOutcome = {
|
||
shard: shardNumber, files: [], status: 'passed', exitCode: 0, elapsedMs: 0, groupPid: null, failingFiles: [], unattributedFailures: 0,
|
||
summary: { testsRan: 0, filesRan: 0, sawTerminalSummary: false },
|
||
};
|
||
log(shardEpilogue(outcome, totalShards));
|
||
return outcome;
|
||
}
|
||
|
||
const rootDir = options.rootDir ?? ROOT;
|
||
const wallTimeoutMs = options.wallTimeoutMs ?? DEFAULT_WALL_TIMEOUT_MS;
|
||
log(`${label} (${files.length} files${options.parallel ? ', bun --parallel' : ''})`);
|
||
|
||
// Full-stream capture: EVERY child byte lands here, whatever the console
|
||
// shows. Printed once at start so a wedged or noisy run is inspectable
|
||
// without a re-run.
|
||
const logPath = options.logFilePath ?? nextDefaultLogPath(rootDir);
|
||
const shardLog = openShardLog(logPath, label, 0o600);
|
||
log(`[test:free] full log: ${logPath}`);
|
||
|
||
const { command, args } = options.commandFor
|
||
? options.commandFor(files)
|
||
: { command: process.execPath, args: buildShardArgs(files, { parallel: options.parallel, rootDir }) };
|
||
|
||
// realpath: the browser tracker refuses a state dir whose path resolves elsewhere.
|
||
const { stateDir, env } = createShardSandbox('gstack-free-shard-', options.env ?? process.env, { realpath: true });
|
||
// CLI renders otherwise share the repo's .gstack/browse.json, where concurrent
|
||
// shards and prior daemons replace each other's state; override inherited state.
|
||
env.BROWSE_STATE_FILE = path.join(stateDir, '.gstack', 'browse.json');
|
||
env.GSTACK_FREE_SHARD_ID = randomUUID();
|
||
const home = options.homeGuard ? options.homeGuard(files, env, stateDir) : guardFreeHome(files, env);
|
||
|
||
const startedAt = Date.now();
|
||
const classifier = new BunTestOutputClassifier();
|
||
|
||
// Console policy: quiet => nothing; verbose => the raw firehose; default =>
|
||
// only always-visible lines (fail results, crash markers, error/panic
|
||
// markers, the terminal summary), selected by the reporter. The reporter
|
||
// consumes the stream in EVERY mode so the epilogue can attribute failures.
|
||
const emitToConsole = (text: string, origin: StreamOrigin): void => {
|
||
if (options.quiet) return;
|
||
if (options.consoleWrite) {
|
||
options.consoleWrite(text);
|
||
return;
|
||
}
|
||
(origin === 'stdout' ? process.stdout : process.stderr).write(text);
|
||
};
|
||
const reporter = new FreeRunReporter(files, options.verbose ? undefined : emitToConsole);
|
||
|
||
const captureFailures = new Map<StreamOrigin, Error>();
|
||
const drained = new Set<StreamOrigin>();
|
||
const consumeStream = (stream: NodeJS.ReadableStream | null, origin: StreamOrigin): Promise<void> =>
|
||
captureFreeStream(stream, origin, captureFailures, (chunk) => {
|
||
classifier.write(chunk, origin); // strict verdict ALWAYS sees the full stream
|
||
shardLog.write(chunk);
|
||
reporter.write(chunk, origin);
|
||
if (options.verbose) emitToConsole(typeof chunk === 'string' ? chunk : chunk.toString('utf8'), origin);
|
||
}).then(() => { drained.add(origin); });
|
||
|
||
let child: ShardChildResult = { exitCode: null, timedOut: false, groupPid: null };
|
||
let cleanupError = null as string | null;
|
||
try {
|
||
// Shared spawn/detached/group-kill/wall-timer/reap lifecycle. The browser
|
||
// tracker rides along: forwarded signals reach it, and it settles after
|
||
// the final group kill.
|
||
child = await runShardChild({
|
||
command, args, cwd: rootDir, env, timeoutMs: wallTimeoutMs,
|
||
attach: () => {
|
||
const browser = trackShardBrowser(stateDir, env);
|
||
return { signal: (force) => browser.signal(force), settle: async () => { cleanupError = await browser.settle() ?? home.verify(); } };
|
||
},
|
||
hookStreams: (spawned) => [consumeStream(spawned.stdout, 'stdout'), consumeStream(spawned.stderr, 'stderr')],
|
||
});
|
||
} catch (error) {
|
||
child = (error as { shardResult?: ShardChildResult } | null)?.shardResult ?? child;
|
||
throw error;
|
||
} finally {
|
||
// A wall-expired child is not drained: its unread tail is lost evidence.
|
||
for (const origin of ['stdout', 'stderr'] as const) {
|
||
if (!drained.has(origin) && !captureFailures.has(origin)) captureFailures.set(origin, new Error('stream did not drain before the wall deadline'));
|
||
}
|
||
reporter.end();
|
||
for (const [origin, error] of captureFailures) {
|
||
const diagnostic = `${label} ${origin} capture incomplete: ${error.message} `
|
||
+ `(child exit ${child.exitCode ?? 'signal'}). Full log: ${logPath}`;
|
||
console.error(diagnostic);
|
||
shardLog.write(diagnostic + '\n');
|
||
}
|
||
await new Promise<void>((resolve) => shardLog.stream.end(() => resolve()));
|
||
try {
|
||
if (!cleanupError) fs.rmSync(stateDir, { recursive: true, force: true });
|
||
} catch {
|
||
// Best-effort cleanup of a throwaway temp dir — a locked file on
|
||
// Windows must not turn a real verdict into an exception.
|
||
if (process.platform !== 'win32') cleanupError = 'could not remove the owned shard directory';
|
||
}
|
||
}
|
||
|
||
const { exitCode, timedOut, groupPid } = child;
|
||
const logWriteFailed = shardLog.failed;
|
||
const summary = classifier.end();
|
||
let status: FreeShardStatus = strictShardStatus({
|
||
timedOut, exitCode, summary, expectedFiles: files.length,
|
||
evidenceComplete: !cleanupError && !logWriteFailed && captureFailures.size === 0,
|
||
});
|
||
if (status === 'passed' && zeroExecutionVerdict(reporter.report().testsRan, FREE_LANE_POLICY, { promisedAll: true }) === 'passed-empty') {
|
||
status = 'failed';
|
||
}
|
||
|
||
explainFreeVerdict(label, status, {
|
||
cleanupError, stateDir, exitCode, summary, expectedFiles: files.length, wallTimeoutMs,
|
||
evidenceComplete: !cleanupError && !logWriteFailed && captureFailures.size === 0,
|
||
});
|
||
|
||
const report = reporter.report();
|
||
const failingFiles = status === 'passed' ? [] : [...new Set([
|
||
...report.failures.map((f) => f.file).filter((f): f is string => !!f),
|
||
...report.crashedFiles,
|
||
])];
|
||
const unattributedFailures = status === 'passed' ? 0
|
||
: report.failures.filter((f) => !f.file).length
|
||
+ report.unreportedFailures
|
||
+ report.unhandledErrors.length
|
||
+ captureFailures.size
|
||
+ (logWriteFailed ? 1 : 0)
|
||
+ (cleanupError ? 1 : 0)
|
||
+ (report.sawTerminalSummary ? 0 : 1);
|
||
const outcome: FreeShardOutcome = {
|
||
shard: shardNumber, files, status, exitCode, elapsedMs: Date.now() - startedAt, groupPid, failingFiles, unattributedFailures,
|
||
summary: { testsRan: report.testsRan, filesRan: report.filesRan, sawTerminalSummary: report.sawTerminalSummary },
|
||
};
|
||
log(shardEpilogue(outcome, totalShards));
|
||
for (const line of buildRunEpilogue(status, report, outcome.elapsedMs, logPath)) log(line);
|
||
if (status !== 'passed') {
|
||
logFreeRecovery(log, outcome, {
|
||
cleanupError, logWriteFailed, rootDir, captureIncomplete: captureFailures.size > 0 || !report.sawTerminalSummary,
|
||
});
|
||
}
|
||
return outcome;
|
||
}
|
||
|
||
/** Retained private log under .context/free-test-logs; never through a link. */
|
||
function nextDefaultLogPath(rootDir: string): string {
|
||
let directory = fs.realpathSync(rootDir);
|
||
for (const part of ['.context', 'free-test-logs']) {
|
||
directory = path.join(directory, part);
|
||
const existing = fs.lstatSync(directory, { throwIfNoEntry: false });
|
||
if (existing && (!existing.isDirectory() || existing.isSymbolicLink())) throw new Error('Free-test log directory must not traverse links');
|
||
if (!existing) fs.mkdirSync(directory, { mode: 0o700 });
|
||
}
|
||
fs.chmodSync(directory, 0o700);
|
||
return nextShardLogPath(directory, 'gstack-free-test');
|
||
}
|
||
|
||
export function exitCodeFor(status: FreeShardStatus): number {
|
||
if (status === 'passed') return 0;
|
||
return status === 'timed-out' ? 124 : 1;
|
||
}
|
||
|
||
/**
|
||
* `--record-durations`: time every file in its own child (exact per-file wall,
|
||
* immune to bun's stream buffering) and write the committed seed atomically.
|
||
* Uses the same isolated, strictly classified children as a normal run.
|
||
* Run against an immutable checkout; never time while editing its inputs.
|
||
*/
|
||
async function recordFreeTestDurations(files: string[], jobs: number): Promise<number> {
|
||
const durations: Record<string, number> = {};
|
||
const failed: string[] = [];
|
||
const expectedCount = files.length;
|
||
console.log(`[test:free] recording per-file durations: ${files.length} files across ${jobs} workers`);
|
||
const phaseHome = guardFreeHome(files, process.env, { kind: 'concurrent', shards: jobs });
|
||
// As in full-suite mode, finish parallel work before exclusive host-state fixtures.
|
||
await forEachFileAlone(files.filter(file => !(file in TREE_MUTATING)), files.filter(file => file in TREE_MUTATING), jobs, async (file, index) => {
|
||
const outcome = await runFreeShard([file], index + 1, files.length, {
|
||
wallTimeoutMs: wallTimeoutForShard(1), quiet: true, homeGuard: sharedFreeHome,
|
||
});
|
||
durations[normalizeRelativePath(file)] = outcome.elapsedMs;
|
||
if (outcome.status !== 'passed') failed.push(file);
|
||
});
|
||
const homeChange = phaseHome.verify();
|
||
if (homeChange) console.error(`[test:free] ${homeChange}`);
|
||
if (Object.keys(durations).length !== expectedCount) {
|
||
throw new Error('Duration recording was interrupted; the seed was not replaced.');
|
||
}
|
||
|
||
const target = process.env.GSTACK_FREE_TEST_DURATIONS ?? path.join(ROOT, FREE_TEST_DURATIONS_FILE);
|
||
// Atomic: a killed recorder never leaves a truncated seed behind.
|
||
writeDurationSeed(target, durations);
|
||
console.log(`[test:free] wrote ${Object.keys(durations).length} durations to ${path.relative(ROOT, target)}`);
|
||
if (failed.length > 0 || homeChange) {
|
||
// Failures still recorded (a red file's duration is still a real cost),
|
||
// but surfaced loudly — recording from a broken tree deserves a look.
|
||
console.error(`[test:free] WARNING: ${failed.length} file(s) failed while recording:`);
|
||
for (const f of failed) console.error(` ✗ ${f}`);
|
||
return 1;
|
||
}
|
||
return 0;
|
||
}
|
||
|
||
async function retryFailedFreeFiles(
|
||
outcomes: FreeShardOutcome[], totalShards: number,
|
||
options: Pick<CliOptions, 'wallTimeoutExplicit' | 'wallTimeoutMs' | 'verbose'>,
|
||
): Promise<{ exitCode: number; retry: FreeShardOutcome | null }> {
|
||
let worst = Math.max(...outcomes.map(outcome => exitCodeFor(outcome.status)));
|
||
let retry: FreeShardOutcome | null = null;
|
||
const shardTimeout = (count: number) => options.wallTimeoutExplicit
|
||
? options.wallTimeoutMs : wallTimeoutForShard(count, options.wallTimeoutMs);
|
||
// Opt-in flaky retry (GSTACK_FREE_RETRY_FLAKY=1): when every failure is an
|
||
// attributed test failure (no timeouts, no unattributed carnage), re-run
|
||
// just the failing files ONCE in a fresh serial shard. A clean retry
|
||
// downgrades the run to a loud flaky-pass; a repeat failure stays a
|
||
// failure. Default OFF: dev boxes should see flakes, not absorb them.
|
||
// Exists for syscall-supervised sandboxes (see fullSuiteJobs) where a run
|
||
// lands 0-1 spurious browser-timing failures under an otherwise-green
|
||
// suite. Capped so a genuinely broken tree never masquerades as flaky.
|
||
const RETRY_CAP = 5;
|
||
if (
|
||
worst !== 0
|
||
&& process.env.GSTACK_FREE_RETRY_FLAKY === '1'
|
||
&& !isTerminationRequested()
|
||
&& outcomes.every((o) => o.status !== 'timed-out')
|
||
) {
|
||
const flakyFiles = [...new Set(outcomes.flatMap((o) => o.failingFiles))]
|
||
.filter((f): f is string => typeof f === 'string' && f.length > 0);
|
||
// "Fully attributed" is per-failure, not per-shard: a shard with one
|
||
// attributed failure PLUS a headerless failure / unhandled error /
|
||
// truncated run must veto the retry — re-running only failingFiles would
|
||
// mask the unattributable evidence as a FLAKY-PASS.
|
||
const allAttributed = outcomes.every((o) => hasScopedFailureAttribution(o) && (o.status === 'passed'
|
||
|| (o.failingFiles.length > 0 && o.unattributedFailures === 0)));
|
||
if (allAttributed && flakyFiles.length > 0 && flakyFiles.length <= RETRY_CAP) {
|
||
console.log(`[test:free] flaky-retry: re-running ${flakyFiles.length} failing file(s) once, serially: ${flakyFiles.join(', ')}`);
|
||
const retryOutcome = await runFreeShard(flakyFiles, totalShards + 1, totalShards + 1, {
|
||
wallTimeoutMs: shardTimeout(flakyFiles.length),
|
||
verbose: options.verbose,
|
||
});
|
||
retry = retryOutcome;
|
||
if (retryOutcome.status === 'passed') {
|
||
console.log(`[test:free] FLAKY-PASS — ${flakyFiles.length} file(s) failed once and passed on serial retry: ${flakyFiles.join(', ')}`);
|
||
console.log('[test:free] treat repeat offenders as real flakes worth fixing, not noise.');
|
||
// Durable record (WS1): console lines vanish with the scrollback; the
|
||
// ledger makes repeat offenders rankable across runs (eval:flake-rank).
|
||
const ts = new Date().toISOString();
|
||
// Two separate calls: `rev-parse --abbrev-ref HEAD HEAD` abbreviates
|
||
// BOTH revs, printing the branch twice — git_sha recorded the branch
|
||
// name (codex adversarial finding).
|
||
const ledgerBranch = (spawnSync('git', ['rev-parse', '--abbrev-ref', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim();
|
||
const ledgerSha = (spawnSync('git', ['rev-parse', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim();
|
||
appendFlakeLedger(
|
||
flakyFiles.map((file) => ({
|
||
ts,
|
||
runner: 'free' as const,
|
||
kind: 'flaky-pass' as const,
|
||
file,
|
||
shard: outcomes.find((o) => o.failingFiles.includes(file))?.shard,
|
||
...(ledgerBranch ? { branch: ledgerBranch } : {}),
|
||
...(ledgerSha ? { git_sha: ledgerSha.slice(0, 12) } : {}),
|
||
})),
|
||
flakeLedgerPath(),
|
||
);
|
||
worst = 0;
|
||
} else {
|
||
console.error('[test:free] flaky-retry FAILED — the failures reproduce serially; not flaky.');
|
||
}
|
||
} else {
|
||
console.log(`[test:free] flaky-retry skipped: ${allAttributed ? `${flakyFiles.length} failing file(s) exceeds cap ${RETRY_CAP}` : 'failures not fully attributed'}.`);
|
||
}
|
||
}
|
||
return { exitCode: worst, retry };
|
||
}
|
||
|
||
async function main(): Promise<number> {
|
||
const options = parseCliOptions(process.argv.slice(2));
|
||
const allFiles = collectFreeTestFiles();
|
||
if (allFiles.length === 0) {
|
||
throw new Error('No free test files were discovered.');
|
||
}
|
||
|
||
if (options.ciPlan || options.ciRun || options.ciVerify) {
|
||
const git = spawnSync('git', ['rev-parse', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5_000 });
|
||
if (git.status !== 0 || !git.stdout.trim()) throw new Error('Cannot bind CI plan to the checkout revision');
|
||
const revision = git.stdout.trim();
|
||
const writeJson = (file: string, value: unknown) => {
|
||
fs.mkdirSync(path.dirname(path.resolve(file)), { recursive: true });
|
||
const temporary = `${file}.tmp-${process.pid}`;
|
||
fs.writeFileSync(temporary, JSON.stringify(value, null, 2) + '\n');
|
||
fs.renameSync(temporary, file);
|
||
};
|
||
if (options.ciPlan) {
|
||
const durations = loadFreeTestDurations() ?? {};
|
||
warnUnseededFreeFiles(allFiles, durations);
|
||
const plan = createFreeCiPlan(allFiles, options.shardCount, durations, revision);
|
||
validateFreeCiPlan(plan, allFiles, revision);
|
||
writeJson(options.ciPlan, plan);
|
||
console.log(JSON.stringify({ shard: plan.shards.map(shard => shard.shard) }));
|
||
return 0;
|
||
}
|
||
const plan = JSON.parse(fs.readFileSync((options.ciRun ?? options.ciVerify)!, 'utf8')) as FreeCiPlan;
|
||
validateFreeCiPlan(plan, allFiles, revision);
|
||
if (options.ciVerify) {
|
||
const results = fs.readdirSync(options.results!).filter(file => file.endsWith('.json'))
|
||
.map(file => JSON.parse(fs.readFileSync(path.join(options.results!, file), 'utf8')) as FreeCiResult);
|
||
verifyFreeCiResults(plan, results);
|
||
console.log(`[test:free] CI PASS: ${allFiles.length} files across ${results.length} isolated shards; slowest ${Math.round(Math.max(...results.map(result => result.outcome.elapsedMs + (result.retry?.elapsedMs ?? 0))) / 1000)}s including retries`);
|
||
return 0;
|
||
}
|
||
const shard = plan.shards[options.shardIndex! - 1];
|
||
if (!shard || shard.shard !== options.shardIndex) throw new Error('CI shard index is outside the plan');
|
||
const outcome = await runFreeShard(shard.files, shard.shard, plan.shards.length, {
|
||
wallTimeoutMs: options.wallTimeoutExplicit ? options.wallTimeoutMs
|
||
: wallTimeoutForPackedShard(shard.predictedMs, options.wallTimeoutMs, shard.files.length),
|
||
verbose: options.verbose,
|
||
});
|
||
const retried = await retryFailedFreeFiles([outcome], plan.shards.length, options);
|
||
writeJson(options.result!, { planId: plan.id, revision, outcome, retry: retried.retry } satisfies FreeCiResult);
|
||
return retried.exitCode;
|
||
}
|
||
|
||
let files = allFiles;
|
||
if (options.quick) {
|
||
const missing = QUICK_CORE.filter(file => !allFiles.includes(file));
|
||
if (missing.length) throw new Error(`Quick core files missing: ${missing.join(', ')}`);
|
||
const durations = loadFreeTestDurations() ?? {};
|
||
files = selectQuickFreeFiles(allFiles, durations);
|
||
const unknown = allFiles.filter(file => durations[file] === undefined && !QUICK_CORE.includes(file)).length;
|
||
console.log(`[test:free] QUICK SUBSET: ${files.length}/${allFiles.length} files; ${unknown} unclassified and ${allFiles.length - files.length - unknown} slow files excluded. Full CI remains required; this is not release acceptance.`);
|
||
}
|
||
let curationReport: CurationResult | null = null;
|
||
if (options.windowsOnly) {
|
||
curationReport = curateWindowsSafe(allFiles);
|
||
files = curationReport.safe;
|
||
console.log(`[test:free] curated ${files.length} Windows-safe tests (${curationReport.excluded.length} excluded)`);
|
||
if (options.listOnly && curationReport.excluded.length > 0) {
|
||
console.log('\nExcluded (POSIX-fragile):');
|
||
for (const { file, reason } of curationReport.excluded) {
|
||
console.log(` - ${file} [${reason}]`);
|
||
}
|
||
}
|
||
}
|
||
|
||
if (options.listOnly) {
|
||
console.log(`\nDiscovered ${files.length} test files.`);
|
||
for (const file of files) console.log(` ${file}`);
|
||
return 0;
|
||
}
|
||
|
||
if (options.recordDurations) {
|
||
return recordFreeTestDurations(files, fullSuiteJobs());
|
||
}
|
||
if (options.attributeHome) return attributeFreeHomeWriters(files.filter(file => !(file in TREE_MUTATING)), files.filter(file => file in TREE_MUTATING),
|
||
fullSuiteJobs(), (file, index, homeGuard) => runFreeShard([file], index + 1, files.length, { wallTimeoutMs: wallTimeoutForShard(1), quiet: true, log: () => {}, homeGuard }));
|
||
|
||
if (options.dryRun) {
|
||
const shards = assignFilesToShards(files, options.shardCount);
|
||
const occupied = shards.filter((s) => s.length > 0).length;
|
||
console.log(
|
||
`\nWould run ${files.length} files across ${shards.length} shards (${occupied} occupied). `
|
||
+ 'Without --shard, the full suite runs as N concurrent shard processes '
|
||
+ '(plus an exclusive host-state shard) instead.',
|
||
);
|
||
for (const line of formatShardSummary(shards)) console.log(line);
|
||
return 0;
|
||
}
|
||
|
||
if (options.shardIndex !== null) {
|
||
// Bounds-check against the REQUESTED shard count, not post-assignment
|
||
// occupancy — indices must be stable for a CI matrix, and an empty shard
|
||
// is a valid fast no-op.
|
||
if (!Number.isInteger(options.shardIndex) || options.shardIndex < 1 || options.shardIndex > options.shardCount) {
|
||
throw new Error(`--shard must be between 1 and ${options.shardCount}. Received: ${options.shardIndex}`);
|
||
}
|
||
const shards = assignFilesToShards(files, options.shardCount);
|
||
const outcome = await runFreeShard(shards[options.shardIndex - 1], options.shardIndex, options.shardCount, {
|
||
wallTimeoutMs: options.wallTimeoutMs,
|
||
verbose: options.verbose,
|
||
});
|
||
return exitCodeFor(outcome.status);
|
||
}
|
||
|
||
// Full-suite mode: N concurrent shard PROCESSES, serial within each — the
|
||
// paid runner's proven model. One `bun test --parallel` invocation was
|
||
// tried first (decision V3) and abandoned after three distinct
|
||
// worker-runtime pathologies in a single day on Bun 1.3.13: a segfault
|
||
// whose crashed-worker retry wedged the run (security-live-playwright), a
|
||
// gated file's still-running file-level hooks stalling a worker
|
||
// (compare-board), and spawn-heavy files hanging workers under load
|
||
// (session-runner-timeout). Plain child processes have none of these:
|
||
// proven spawn semantics, per-shard group-kill, per-shard logs, and a
|
||
// wedge only ever costs its own shard. WORKER_HOSTILE files are moot in
|
||
// process shards (no workers) and fold back into normal assignment.
|
||
const jobs = fullSuiteJobs();
|
||
// Phase split: exclusive host-state fixtures run AFTER the parallel shards,
|
||
// so their shared process or filesystem state cannot interfere with readers.
|
||
const exclusive = files.filter((f) => f in TREE_MUTATING);
|
||
const readers = files.filter((f) => !(f in TREE_MUTATING));
|
||
const durations = loadFreeTestDurations();
|
||
if (durations) warnUnseededFreeFiles(files, durations);
|
||
const packed = durations ? packShardsByDuration(readers, jobs, durations) : null;
|
||
const shards = packed ? packed.shards : assignFilesToShards(readers, jobs);
|
||
const totalShards = jobs + (exclusive.length > 0 ? 1 : 0);
|
||
console.log(`[test:free] full suite: ${readers.length} files across ${jobs} shard processes`
|
||
+ (packed ? ' (duration-packed)' : '')
|
||
+ (exclusive.length > 0 ? `, then ${exclusive.length} exclusive host-state file(s) serially` : ''));
|
||
if (packed) {
|
||
// One line per shard so a packing regression is diagnosable from any log.
|
||
packed.predictedMs.forEach((ms, i) => {
|
||
console.log(`[test:free] shard ${i + 1}: ${shards[i].length} files, predicted ~${Math.round(ms / 1000)}s`);
|
||
});
|
||
}
|
||
const shardTimeout = (fileCount: number): number =>
|
||
options.wallTimeoutExplicit ? options.wallTimeoutMs : wallTimeoutForShard(fileCount, options.wallTimeoutMs);
|
||
const phaseHome = guardFreeHome(readers, process.env, { kind: 'concurrent', shards: shards.length });
|
||
const outcomes = await Promise.all(
|
||
shards.map((shardFiles, index) => runFreeShard(shardFiles, index + 1, totalShards, {
|
||
homeGuard: sharedFreeHome,
|
||
// Packed shards get duration-aware walls: LPT decouples file count from
|
||
// cost BY DESIGN, so the 5s/file heuristic would undersize a shard
|
||
// holding few expensive files.
|
||
wallTimeoutMs: packed && !options.wallTimeoutExplicit
|
||
? wallTimeoutForPackedShard(packed.predictedMs[index], options.wallTimeoutMs, shardFiles.length)
|
||
: shardTimeout(shardFiles.length),
|
||
verbose: options.verbose,
|
||
})),
|
||
);
|
||
const homeChange = phaseHome.verify();
|
||
if (homeChange) console.error(`[test:free] ${homeChange}`);
|
||
// Cancellation stops the run: don't launch the exclusive host-state shard
|
||
// after a SIGINT/SIGTERM already killed the parallel phase.
|
||
if (exclusive.length > 0 && !isTerminationRequested()) {
|
||
const exclusiveOutcome = await runFreeShard(exclusive, totalShards, totalShards, {
|
||
wallTimeoutMs: shardTimeout(exclusive.length),
|
||
verbose: options.verbose,
|
||
});
|
||
if (exclusiveOutcome.status !== 'passed') {
|
||
// Fixture safety rests on each test restoring default state itself; a
|
||
// SIGKILL at the wall deadline (or a mid-regeneration crash) defeats
|
||
// that by construction. Say so, loudly, before someone commits
|
||
// regenerated SKILL.md / .agents artifacts by accident.
|
||
const dirty = spawnSyncGitStatusGenerated();
|
||
if (dirty.length > 0) {
|
||
console.error('[test:free] ⚠ exclusive host-state shard did not finish cleanly — generated artifacts are dirty:');
|
||
for (const line of dirty.slice(0, 20)) console.error(`[test:free] ${line}`);
|
||
console.error('[test:free] restore with: bun run gen:skill-docs (or git checkout -- <paths>)');
|
||
}
|
||
}
|
||
outcomes.push(exclusiveOutcome);
|
||
}
|
||
|
||
return Math.max(homeChange ? 1 : 0, (await retryFailedFreeFiles(outcomes, totalShards, options)).exitCode);
|
||
}
|
||
|
||
/** Dirty generated artifacts (SKILL.md / host outputs) after a failed exclusive shard. */
|
||
function spawnSyncGitStatusGenerated(): string[] {
|
||
const result = spawnSync('git', ['status', '--porcelain'], { cwd: ROOT, encoding: 'utf8' });
|
||
if (result.status !== 0 || !result.stdout) return [];
|
||
return result.stdout.split('\n').filter((line) =>
|
||
/SKILL\.md$/.test(line) || line.includes('.agents/') || line.includes('.factory/'));
|
||
}
|
||
|
||
if (import.meta.main) {
|
||
try {
|
||
process.exitCode = await main();
|
||
} catch (error) {
|
||
console.error(`[test:free] ${error instanceof Error ? error.message : String(error)}`);
|
||
process.exitCode = 1;
|
||
}
|
||
}
|