diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index bbcdd721b..19429bccb 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -41,3 +41,51 @@ E2E tests stream progress in real-time (tool-by-tool via `--output-format stream fallback `~/.gstack-dev/evals/`) with auto-comparison against the previous finalized run (in-flight `_partial` files are never used as a baseline, so a run can't compare against itself). + +## Runners: how the suites execute (2026-08 overhaul) + +**Free suite (`bun run test:free`).** `scripts/test-free-shards.ts` runs N +concurrent shard processes (serial within each) with strict-output +classification per shard. Full-suite shards are packed by RECORDED PER-FILE +DURATIONS (LPT, `packShardsByDuration`) when the committed seed +`scripts/free-test-durations.json` exists — refresh it occasionally with +`bun run test:free --record-durations` (each file timed in its own child; +CI never records). Missing seed → silent hash-shard fallback; corrupt seed → +one warning + fallback; unknown files get 75th-percentile pessimism. Packed +shards get duration-aware walls (`max(base, predicted × 3)`); the `--shard` +CI-matrix path keeps stable hash indices untouched. `TREE_MUTATING` is EMPTY: +`gen-skill-docs.ts` has a `main()` guard (imports never regenerate; pinned by +`test/gen-skill-docs-import-purity.test.ts`) and `--out-dir` renders every +host, so all former mutators render into mkdtemps and the trailing serial +shard is gone. The map remains a mechanism — a test that genuinely must write +shared artifacts in place earns a reasoned entry and is serialized again. + +**Paid suite (sharded runner, local AND CI).** `scripts/test-paid-shards.ts` +is the single selection engine: 1 file per shard, `EVALS_JOBS` shard +processes × `EVALS_CONCURRENCY` within-shard, per-shard `GSTACK_EVAL_DIR`, +full-stream spooling to per-shard log files (path printed at START and on +failure), never-started/timed-out taxonomy, and parent-computed diff +selection propagated to children via `EVALS_SELECTION_JSON` (fail-open: a +child that can't parse it recomputes locally with one warning). Retry parity +lives in `RETRY_OVERRIDES` (literals; old matrix rows' earned `retries: 2`). + +**CI planner/executor/report.** `--emit-plan --slices K` computes +selection + the slice plan ONCE (killing per-slice selector divergence); +`--plan --slice i` executors consume the manifest and write +slice-result artifacts; `--report ` reconciles them FAIL-CLOSED (a slice +whose artifact never landed, or a planned shard nobody reported, is a +failure). Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with +ZERO executed tests `passed-empty` (a failure) — census-health, not just +test runs. evals.yml runs the sliced gate lane per PR (parity phase: +alongside the legacy matrix, `needs:`-sequenced so provider concurrency +never doubles; the matrix and its `KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET` +ratchets are deleted after demonstrated parity). evals-periodic.yml runs ALL +periodic-tier files weekly (the coverage contract) minus the reasoned +exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking +required per entry; removal re-activates the file), plus a weekly +`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. + +**Timeout policy.** Paid tests use the tiers in +`test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG); +`test/eval-budgets-policy.test.ts` pins that every tier fits the shard wall +minus overhead and ratchets raw literals. Budget above the wall is fiction.