Commit Graph
2 Commits
Author SHA1 Message Date
TestandClaude Fable 5 466f48d0ff fix(ci): first-live-run fixes — executor history + two environment-blind assertions
The sliced lane's first run (PR #2721) did its job: the planner and
report worked, the manifest governed, and every failure had a name.
Three were fixable on the spot:

- executor + gate-census checkouts get fetch-depth: 0 — files with
  SELF-derived selection (the LLM-judge map, routing) walk git at
  module load, and selection is deliberately fail-closed on git errors,
  so the shallow checkout crashed those shards ('ambiguous argument
  main...HEAD'). The manifest still governs WHICH shards run.
- landscape --toc gate: the exact toBe(3) landscape-page count was
  font-metric-dependent (3 on Amazon Linux, 2 on ubuntu CI — the same
  disease the file's own page-index comment warns about). Now a
  comparative invariant: --toc must not CHANGE the landscape count vs
  a baseline render.
- paid-run-manifest parse test builds its manifest under EVALS_ALL so
  it never walks git (proven with GIT_DIR=/nonexistent).

Remaining first-run failures are newly-exposed rot in gate files that
had never executed in CI (skillify D1 refusal, session-intelligence
context-restore, one tpa-apple-ban retry flake) — being probed
separately; they are the lane WORKING, not the lane failing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 06:58:55 +00:00
Garry TanandClaude Fable 5 f7ef0cc0ce feat(evals): planner/executor/report modes — the CI re-platform surface
One PLANNER computes diff selection + the slice plan ONCE and writes a
manifest (--emit-plan <path> --slices K); K executors consume it
(--plan <path> --slice i), never self-selecting, and write slice-result
artifacts; a REPORT reconciles results against the manifest (--report
<dir>) fail-closed: a slice whose artifact never landed is a FAILURE,
a planned shard nobody reported fails, wrong-slice/duplicate/cross-tier
results fail. Kills per-slice selector divergence and hollow-lane
aggregation at the root.

- hollow-shard guard: under EVALS_ALL, exit 0 with ZERO executed tests
  (bun's 'Ran N tests' now captured by the classifier — additive) is
  'passed-empty' and fails the run; selective runs keep it 'passed'
  with one warning (in-file diff/tier self-skips are legitimate there);
  unknown counts are never guessed hollow
- retry parity: --retry 1 default + RETRY_OVERRIDES literals for the
  three files whose old matrix rows earned retries: 2 (stale entries
  pinned against disk)
- live smoke: gate plan = 48 shards across 6 slices; report mode exits
  1 on a fabricated missing slice, 0 when complete

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 05:42:16 +00:00