test: free runner — strict output, parallel execution, stable shard indices

Three coupled changes to scripts/test-free-shards.ts:

1. STRICT OUTPUT: runFreeShard streams through the paid runner's
   BunTestOutputClassifier — exit 0 without bun's 'Ran N tests across M
   files' summary, with (fail) lines, or with a wrong file count is a
   FAILURE (anti-truncation backstop at the runner layer), plus an
   external wall-clock timeout that SIGKILLs the process group
   (timed-out distinct from failed; exit 124 vs 1). Also fixes a latent
   shard-bleed: file selectors now use exactTestFileSelectors (relative
   paths were substring filters that matched sibling roots).

2. PARALLEL: full-suite mode is one 'bun test --parallel' invocation
   (Bun 1.3.13). Measured semantics recorded in the header: per-file
   worker isolation, standard summary, and mid-suite process.exit
   surfaces as a crashed-worker FAIL with exit 1 — strictly safer than
   serial, where the same exit truncates silently. No static weight
   lists; --shards M --shard i keeps deterministic hash partitioning for
   CI matrices (native --shard rejected: round-robin renumbers when
   files land). Spawned shards get throwaway GSTACK_HOME/TMPDIR so
   parallel shards can't contend on real state. Per-shard epilogue
   prints files/seconds/status every run.

3. Stable indices: assignFilesToShards no longer drops empty shards, so
   a shard's index depends only on the file hash and requested count —
   an empty CI matrix slot is a fast no-op success, not a renumbering.

package.json 'test' now delegates to the runner (TEST_ROOTS becomes the
single source of truth for roots; slop:diff tail preserved; the runner
inherits the 30s per-test timeout the old glob passed inline).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-15 08:22:44 -07:00
co-authored by Claude Fable 5
parent 587217e2ec
commit 41160a14ed
4 changed files with 501 additions and 58 deletions
+1 -1
View File
@@ -18,7 +18,7 @@
"gen:skill-docs:user": "bun run scripts/gen-skill-docs.ts --respect-detection",
"dev": "bun run browse/src/cli.ts",
"server": "bun run browse/src/server.ts",
"test": "bun test browse/test/ test/ make-pdf/test/ design/test/ --timeout 30000 --ignore 'test/skill-e2e-*.test.ts' --ignore test/skill-llm-eval.test.ts --ignore test/skill-routing-e2e.test.ts --ignore test/codex-e2e.test.ts --ignore test/gemini-e2e.test.ts && (bun run slop:diff 2>/dev/null || true)",
"test": "bun run scripts/test-free-shards.ts && (bun run slop:diff 2>/dev/null || true)",
"test:free": "bun run scripts/test-free-shards.ts",
"test:windows": "bun run scripts/test-free-shards.ts --windows-only",
"test:evals": "EVALS=1 bun test --retry 1 --concurrent --max-concurrency ${EVALS_CONCURRENCY:-15} test/skill-llm-eval.test.ts test/skill-e2e-*.test.ts test/skill-routing-e2e.test.ts test/codex-e2e.test.ts test/gemini-e2e.test.ts",