CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-08-22 14:07:14 +02:00
Code Issues Packages Projects Releases Wiki Activity

evals: split skill-e2e-review into three per-file CI shards

Browse Source 
Bun runs describe blocks as concurrency barriers, so the e2e-review CI
job executed its tests serially: 741s of an 860s PR critical path for
tests whose slowest member is 224s. The per-file matrix is the repo's
parallelism unit, so the split moves:

- Retro E2E + retro-base-branch  -> test/skill-e2e-retro.test.ts
- review/ship base-branch + Review Dashboard Via Attribution
                                  -> test/skill-e2e-review-attribution.test.ts
- sql-injection / enum-completeness / design-lite stay in
  test/skill-e2e-review.test.ts

One 741s job becomes three ~180-250s jobs. Locally the worst paid shard
drops from 1705s (94.7% of the 1800s kill) to under 700s. Test names,
bodies, suite strings, and eval-store collectors are unchanged, so
baselines carry over. Matrix rows added to both eval workflows
(attribution is gate-only, so no periodic row); the report job's
hardcoded runner count is gone (drift-proof).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-15 08:03:41 -07:00
co-authored by Claude Fable 5
parent ac9291b146
commit b63608099c
5 changed files with 478 additions and 410 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/evals-periodic.yml
+5
View File
@@ -84,6 +84,11 @@ jobs:
file: test/skill-e2e-qa-workflow.test.ts
- name: e2e-review
file: test/skill-e2e-review.test.ts
- name: e2e-retro
file: test/skill-e2e-retro.test.ts
# e2e-review-attribution is gate-only (all three of its tests are
# gate-tier) — deliberately absent here; an all-skip shard would just
# burn a container boot weekly.
- name: e2e-workflow
file: test/skill-e2e-workflow.test.ts
- name: e2e-routing
.github/workflows/evals.yml
+6 -2
View File
@@ -106,6 +106,10 @@ jobs:
file: test/skill-e2e-qa-workflow.test.ts
- name: e2e-review
file: test/skill-e2e-review.test.ts
- name: e2e-retro
file: test/skill-e2e-retro.test.ts
- name: e2e-review-attribution
file: test/skill-e2e-review-attribution.test.ts
- name: e2e-workflow
file: test/skill-e2e-workflow.test.ts
- name: e2e-routing
@@ -357,14 +361,14 @@ jobs:
BODY="## E2E Evals: ${STATUS}
**${PASSED}/${TOTAL}** tests passed | **\$${COST}** total cost | **13 parallel runners**
**${PASSED}/${TOTAL}** tests passed | **\$${COST}** total cost
| Suite | Result | Status | Cost |
|-------|--------|--------|------|
$(echo -e "$SUITE_LINES")
---
*13x ubicloud-standard-8 (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite*"
*ubicloud-standard-8 runners (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite*"
if [ "$FAILED" -gt 0 ]; then
FAILURES=""