mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-10 23:19:09 +02:00
feat: delete the legacy 17-row eval matrix — the sliced lane is the only paid lane
Every PR paid twice: the hand-enumerated matrix (18 test files, 22.6 min, ~$21 API measured on run 33263204465) ran serialized AHEAD of the strictly superior sliced lane via 'needs: evals' — 35.5 min wall and ~2x paid spend for the same diff. 14 of 17 rows carried no tier:, so periodic Opus benchmarks leaked into every PR (the e2e-plan row alone: 12/12 tests, 21.7 min, $7.28 — the wall-clock bound of ALL of CI). Parity receipt (static, pre-deletion): the sliced lane's gate census (49 files, derived from the runner itself) strictly contains all 18 matrix test files, plus 31 files the matrix never ran. Pure deletion — one revert restores it. The PR comment moved into slices-report (same '## E2E Evals' upsert marker, now sourced from slice artifacts + carrying the fail-closed reconciliation verdict). plan-slices loses the needs edge; the dead workflow-level EVALS_TIER env goes with it. test/evals-workflow-matrix.test.ts (and its KNOWN_MATRIX_GAPS / KNOWN_TIER_UNSET burn-down ratchets — retired: the sliced census makes 'every gate file runs' true by construction) is rewritten as test/evals-workflow-wiring.test.ts: matrix stays deleted, planner/executor/ report tier + slice-count agreement, both surviving lanes on the shared register-skills composite with its fail-fast verification loop, PR comment survival. Expected: PR eval wall 35.5 -> ~13 min, per-PR paid spend ~halved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
0f26859bd1
commit
07c2452e6d
@@ -567,14 +567,15 @@ duration-packed free shards, the sharded paid runner as the CI engine
|
||||
coverage contract + gate census, eval-budget timeout tiers, and the
|
||||
coverage fill. Remaining, in rough priority order:
|
||||
|
||||
- **P1 — Delete the legacy evals.yml matrix after parity.** The sliced lane
|
||||
runs alongside the 18-row matrix (`needs: evals`, so provider concurrency
|
||||
never doubles). After 1-2 PR cycles of parity (compare executed-test sets:
|
||||
intersection strict + the 8 KNOWN_MATRIX_GAPS files as expected additions;
|
||||
stochastic outcomes informational), delete the matrix as a PURE-DELETION
|
||||
commit (one revert restores it), drop the `needs: evals` edge, rewrite
|
||||
test/evals-workflow-matrix.test.ts into a runner-wiring pin, and retire
|
||||
KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET wholesale. Effort S.
|
||||
- **DONE (v1.76 test-infra wave) — Delete the legacy evals.yml matrix after
|
||||
parity.** Deleted as a pure-deletion commit (one revert restores it) after
|
||||
a static parity receipt: sliced gate census (49 files) ⊇ matrix files (18),
|
||||
31 files of extra coverage. `needs: evals` edge dropped, PR comment moved
|
||||
into slices-report, KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET retired,
|
||||
test/evals-workflow-matrix.test.ts rewritten as
|
||||
test/evals-workflow-wiring.test.ts. The register-skills fail-fast
|
||||
verification loop was ported to the surviving lanes FIRST via the shared
|
||||
.github/actions/register-gstack-skills composite.
|
||||
- **P1 — Maintainer decision: make `slices-report` a required check** once
|
||||
post-migration flake data exists (the Codex outside-voice's "green means
|
||||
green is not delivered while paid stays advisory" point — correct, and
|
||||
|
||||
Reference in New Issue
Block a user