mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-31 10:20:42 +02:00
docs(CLAUDE.md): testing prose matches the overhauled runners
- bun run test: duration-packed shards + --record-durations; the trailing serial tree-mutating shard no longer exists - two-tier system: the sliced CI lanes (one engine local+CI), the weekly all-periodic coverage contract + exclusions, the gate census - periodic detach timeout 32400 → 37800 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
9e869b2ba3
commit
b54d5eefb4
@@ -48,11 +48,15 @@ variants to force all tests. Run `eval:select` to preview which tests would run.
|
|||||||
|
|
||||||
**Two-tier system:** Tests are classified as `gate` or `periodic` in `E2E_TIERS`
|
**Two-tier system:** Tests are classified as `gate` or `periodic` in `E2E_TIERS`
|
||||||
(in `test/helpers/touchfiles.ts` — a facade over `touchfiles-data.ts` +
|
(in `test/helpers/touchfiles.ts` — a facade over `touchfiles-data.ts` +
|
||||||
`test-selection.ts`). CI runs only gate tests (`EVALS_TIER=gate`); the free
|
`test-selection.ts`). CI runs gate tests per PR via evals.yml's sliced lane
|
||||||
|
(planner manifest → executors → fail-closed report; engine =
|
||||||
|
scripts/test-paid-shards.ts, the same runner as local eval:bg:gate); the free
|
||||||
suite runs on every PR via `.github/workflows/free-tests.yml` (a REQUIRED
|
suite runs on every PR via `.github/workflows/free-tests.yml` (a REQUIRED
|
||||||
check, secretless — fork PRs get real signal);
|
check, secretless — fork PRs get real signal); ALL periodic tests run weekly
|
||||||
periodic tests run weekly via cron or manually. Use `EVALS_TIER=gate` or
|
via evals-periodic.yml (EVALS_ALL, minus the reasoned exclusions in
|
||||||
`EVALS_TIER=periodic` to filter. When adding new E2E tests, classify them:
|
`test/helpers/periodic-exclude-data.ts` — reason + tracking required per
|
||||||
|
entry), plus a weekly EVALS_ALL gate census. Use `EVALS_TIER=gate` or
|
||||||
|
`EVALS_TIER=periodic` to filter locally. When adding new E2E tests, classify them:
|
||||||
1. Safety guardrail or deterministic functional test? -> `gate`
|
1. Safety guardrail or deterministic functional test? -> `gate`
|
||||||
2. Quality benchmark, Opus model test, or non-deterministic? -> `periodic`
|
2. Quality benchmark, Opus model test, or non-deterministic? -> `periodic`
|
||||||
3. Requires external service (Codex, Gemini)? -> `periodic`
|
3. Requires external service (Codex, Gemini)? -> `periodic`
|
||||||
@@ -71,11 +75,16 @@ bun run test:evals # run before shipping — paid, diff-based (~$4/run max)
|
|||||||
```
|
```
|
||||||
|
|
||||||
`bun run test` routes through `scripts/test-free-shards.ts` (N concurrent
|
`bun run test` routes through `scripts/test-free-shards.ts` (N concurrent
|
||||||
shard processes, serial within each, plus a trailing serial tree-mutating
|
shard processes, serial within each, packed by recorded per-file durations
|
||||||
shard — with strict-output classification per shard: a shard without bun's
|
when `scripts/free-test-durations.json` exists — refresh occasionally with
|
||||||
terminal summary line FAILS — silent truncation
|
`bun run test:free --record-durations`; strict-output classification per
|
||||||
cannot report green). Never type bare `bun test` for the suite: it walks the
|
shard: a shard without bun's terminal summary line FAILS — silent truncation
|
||||||
whole repo, loading paid eval files and missing the strict classifier.
|
cannot report green). The former trailing serial tree-mutating shard is
|
||||||
|
gone: `TREE_MUTATING` is empty (gen-skill-docs has a main() guard and
|
||||||
|
`--out-dir` renders every host, so tests render into mkdtemps — see
|
||||||
|
docs/TESTING_INTERNALS.md). Never type bare `bun test` for the suite: it
|
||||||
|
walks the whole repo, loading paid eval files and missing the strict
|
||||||
|
classifier.
|
||||||
It covers skill validation, gen-skill-docs quality checks, and browse
|
It covers skill validation, gen-skill-docs quality checks, and browse
|
||||||
integration tests. `bun run test:evals` runs LLM-judge quality evals and E2E
|
integration tests. `bun run test:evals` runs LLM-judge quality evals and E2E
|
||||||
tests via `claude -p`. Both must pass before creating a PR.
|
tests via `claude -p`. Both must pass before creating a PR.
|
||||||
@@ -634,7 +643,7 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
|
|||||||
(stray `claude`/`codex` grandchildren included), a per-shard
|
(stray `claude`/`codex` grandchildren included), a per-shard
|
||||||
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
|
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
|
||||||
constructor, and an aggregate that separates failed vs timed-out vs
|
constructor, and an aggregate that separates failed vs timed-out vs
|
||||||
never-started shards — the detach timeouts (25200s gate / 32400s periodic;
|
never-started shards — the detach timeouts (25200s gate / 37800s periodic;
|
||||||
floor enforced against the live shard census by
|
floor enforced against the live shard census by
|
||||||
test/eval-detach-timeout-floor.test.ts)
|
test/eval-detach-timeout-floor.test.ts)
|
||||||
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
|
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
|
||||||
|
|||||||
Reference in New Issue
Block a user