Files
gstack/test
Garry TanandClaude Fable 5 f7ef0cc0ce feat(evals): planner/executor/report modes — the CI re-platform surface
One PLANNER computes diff selection + the slice plan ONCE and writes a
manifest (--emit-plan <path> --slices K); K executors consume it
(--plan <path> --slice i), never self-selecting, and write slice-result
artifacts; a REPORT reconciles results against the manifest (--report
<dir>) fail-closed: a slice whose artifact never landed is a FAILURE,
a planned shard nobody reported fails, wrong-slice/duplicate/cross-tier
results fail. Kills per-slice selector divergence and hollow-lane
aggregation at the root.

- hollow-shard guard: under EVALS_ALL, exit 0 with ZERO executed tests
  (bun's 'Ran N tests' now captured by the classifier — additive) is
  'passed-empty' and fails the run; selective runs keep it 'passed'
  with one warning (in-file diff/tier self-skips are legitimate there);
  unknown counts are never guessed hollow
- retry parity: --retry 1 default + RETRY_OVERRIDES literals for the
  three files whose old matrix rows earned retries: 2 (stale entries
  pinned against disk)
- live smoke: gate plan = 48 shards across 6 slices; report mode exits
  1 on a fabricated missing slice, 0 when complete

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 05:42:16 +00:00
..