CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-08-22 14:07:14 +02:00
Code Issues Packages Projects Releases Wiki Activity

evals: retry 1, not 2, on every paid path

Browse Source 
Measured on the llm-judge shard: --retry 2 amplified 25 tests into 46
executions (+84%), with retried runs at 138s vs a 10-12s baseline (429
backoff), and a permanently-failing test paying 3x. One retry still
absorbs one-off flakes; chronic flakes become visible fix-work instead
of silent wall-clock.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-15 07:53:39 -07:00
co-authored by Claude Fable 5
parent a33f7cb09e
commit ac9291b146
4 changed files with 9 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/evals-periodic.yml
+1 -1
View File
@@ -127,7 +127,7 @@ jobs:
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
EVALS_CONCURRENCY: "40"
PLAYWRIGHT_BROWSERS_PATH: /opt/playwright-browsers
run: EVALS=1 bun test --retry 2 --concurrent --max-concurrency 40 ${{ matrix.suite.file }}
run: EVALS=1 bun test --retry 1 --concurrent --max-concurrency 40 ${{ matrix.suite.file }}
- name: Upload eval results
if: always()