mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-31 10:20:42 +02:00
docs: TESTING_INTERNALS covers the 2026-08 runner overhaul
LPT-packed free suite + --record-durations, the emptied TREE_MUTATING mechanism, the sharded paid runner as the single selection engine, CI planner/executor/report with the fail-closed report and hollow-shard guard, the weekly coverage contract + exclusions policy, and the eval-budgets timeout tiers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
9f2ee58d38
commit
9e869b2ba3
@@ -41,3 +41,51 @@ E2E tests stream progress in real-time (tool-by-tool via `--output-format stream
|
||||
fallback `~/.gstack-dev/evals/`) with auto-comparison
|
||||
against the previous finalized run (in-flight `_partial` files are never used as
|
||||
a baseline, so a run can't compare against itself).
|
||||
|
||||
## Runners: how the suites execute (2026-08 overhaul)
|
||||
|
||||
**Free suite (`bun run test:free`).** `scripts/test-free-shards.ts` runs N
|
||||
concurrent shard processes (serial within each) with strict-output
|
||||
classification per shard. Full-suite shards are packed by RECORDED PER-FILE
|
||||
DURATIONS (LPT, `packShardsByDuration`) when the committed seed
|
||||
`scripts/free-test-durations.json` exists — refresh it occasionally with
|
||||
`bun run test:free --record-durations` (each file timed in its own child;
|
||||
CI never records). Missing seed → silent hash-shard fallback; corrupt seed →
|
||||
one warning + fallback; unknown files get 75th-percentile pessimism. Packed
|
||||
shards get duration-aware walls (`max(base, predicted × 3)`); the `--shard`
|
||||
CI-matrix path keeps stable hash indices untouched. `TREE_MUTATING` is EMPTY:
|
||||
`gen-skill-docs.ts` has a `main()` guard (imports never regenerate; pinned by
|
||||
`test/gen-skill-docs-import-purity.test.ts`) and `--out-dir` renders every
|
||||
host, so all former mutators render into mkdtemps and the trailing serial
|
||||
shard is gone. The map remains a mechanism — a test that genuinely must write
|
||||
shared artifacts in place earns a reasoned entry and is serialized again.
|
||||
|
||||
**Paid suite (sharded runner, local AND CI).** `scripts/test-paid-shards.ts`
|
||||
is the single selection engine: 1 file per shard, `EVALS_JOBS` shard
|
||||
processes × `EVALS_CONCURRENCY` within-shard, per-shard `GSTACK_EVAL_DIR`,
|
||||
full-stream spooling to per-shard log files (path printed at START and on
|
||||
failure), never-started/timed-out taxonomy, and parent-computed diff
|
||||
selection propagated to children via `EVALS_SELECTION_JSON` (fail-open: a
|
||||
child that can't parse it recomputes locally with one warning). Retry parity
|
||||
lives in `RETRY_OVERRIDES` (literals; old matrix rows' earned `retries: 2`).
|
||||
|
||||
**CI planner/executor/report.** `--emit-plan <path> --slices K` computes
|
||||
selection + the slice plan ONCE (killing per-slice selector divergence);
|
||||
`--plan <path> --slice i` executors consume the manifest and write
|
||||
slice-result artifacts; `--report <dir>` reconciles them FAIL-CLOSED (a slice
|
||||
whose artifact never landed, or a planned shard nobody reported, is a
|
||||
failure). Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with
|
||||
ZERO executed tests `passed-empty` (a failure) — census-health, not just
|
||||
test runs. evals.yml runs the sliced gate lane per PR (parity phase:
|
||||
alongside the legacy matrix, `needs:`-sequenced so provider concurrency
|
||||
never doubles; the matrix and its `KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET`
|
||||
ratchets are deleted after demonstrated parity). evals-periodic.yml runs ALL
|
||||
periodic-tier files weekly (the coverage contract) minus the reasoned
|
||||
exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking
|
||||
required per entry; removal re-activates the file), plus a weekly
|
||||
`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks.
|
||||
|
||||
**Timeout policy.** Paid tests use the tiers in
|
||||
`test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG);
|
||||
`test/eval-budgets-policy.test.ts` pins that every tier fits the shard wall
|
||||
minus overhead and ratchets raw literals. Budget above the wall is fiction.
|
||||
|
||||
Reference in New Issue
Block a user