CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-09 14:38:59 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: delete the legacy 17-row eval matrix — the sliced lane is the only paid lane

Browse Source 
Every PR paid twice: the hand-enumerated matrix (18 test files, 22.6 min,
~$21 API measured on run 33263204465) ran serialized AHEAD of the strictly
superior sliced lane via 'needs: evals' — 35.5 min wall and ~2x paid spend
for the same diff. 14 of 17 rows carried no tier:, so periodic Opus
benchmarks leaked into every PR (the e2e-plan row alone: 12/12 tests,
21.7 min, $7.28 — the wall-clock bound of ALL of CI).

Parity receipt (static, pre-deletion): the sliced lane's gate census (49
files, derived from the runner itself) strictly contains all 18 matrix test
files, plus 31 files the matrix never ran. Pure deletion — one revert
restores it. The PR comment moved into slices-report (same '## E2E Evals'
upsert marker, now sourced from slice artifacts + carrying the fail-closed
reconciliation verdict). plan-slices loses the needs edge; the dead
workflow-level EVALS_TIER env goes with it.

test/evals-workflow-matrix.test.ts (and its KNOWN_MATRIX_GAPS /
KNOWN_TIER_UNSET burn-down ratchets — retired: the sliced census makes
'every gate file runs' true by construction) is rewritten as
test/evals-workflow-wiring.test.ts: matrix stays deleted, planner/executor/
report tier + slice-count agreement, both surviving lanes on the shared
register-skills composite with its fail-fast verification loop, PR comment
survival. Expected: PR eval wall 35.5 -> ~13 min, per-PR paid spend ~halved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-31 04:20:40 +00:00
co-authored by Claude Fable 5
parent 0f26859bd1
commit 07c2452e6d
7 changed files with 266 additions and 630 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/TESTING_INTERNALS.md
+7 -4
View File
@@ -76,10 +76,13 @@ slice-result artifacts; `--report <dir>` reconciles them FAIL-CLOSED (a slice
whose artifact never landed, or a planned shard nobody reported, is a
failure). Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with
ZERO executed tests `passed-empty` (a failure) — census-health, not just
test runs. evals.yml runs the sliced gate lane per PR (parity phase:
alongside the legacy matrix, `needs:`-sequenced so provider concurrency
never doubles; the matrix and its `KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET`
ratchets are deleted after demonstrated parity). evals-periodic.yml runs ALL
test runs. evals.yml runs the sliced gate lane per PR — the ONLY paid lane
since the legacy 17-row matrix (22.6 min/$21 per PR serialized ahead of the
slices) was deleted after demonstrated parity; its
`KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET` ratchets retired with it and
`test/evals-workflow-wiring.test.ts` pins the surviving wiring (slice-count
agreement, tier consistency, the shared register-skills composite with its
fail-fast verification loop). evals-periodic.yml runs ALL
periodic-tier files weekly (the coverage contract) minus the reasoned
exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking
required per entry; removal re-activates the file), plus a weekly