Matrix strategy spins up 12 ubicloud-standard-2 runners simultaneously,
one per test file. Separate report job aggregates all artifacts into a
single PR comment. Bun dependency cache cuts install from ~30s to ~3s.
Runner cost: ~$0.048 (from $0.024) — negligible vs $3-4 API costs.
Wall clock: ~3-4min (from ~8min).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single-job GitHub Actions workflow that runs E2E evals on every PR using
Ubicloud runners ($0.006/run — 10x cheaper than GitHub standard). Uses
EVALS_CONCURRENCY=40 with the new within-file concurrency for ~6min
wall clock. Downloads previous eval artifact from main for comparison,
uploads results, and posts a PR comment with pass/fail + cost.
Ubicloud setup required: connect GitHub repo via ubicloud.com dashboard,
add ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY as repo secrets.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Garry Tan