CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-08-23 06:27:14 +02:00
Code Issues Packages Projects Releases Wiki Activity

evals: retry 1, not 2, on every paid path

Browse Source 
Measured on the llm-judge shard: --retry 2 amplified 25 tests into 46
executions (+84%), with retried runs at 138s vs a 10-12s baseline (429
backoff), and a permanently-failing test paying 3x. One retry still
absorbs one-off flakes; chronic flakes become visible fix-work instead
of silent wall-clock.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-15 07:53:39 -07:00
co-authored by Claude Fable 5
parent a33f7cb09e
commit ac9291b146
4 changed files with 9 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/test-paid-shards.ts
+1 -1
View File
@@ -136,7 +136,7 @@ export function planPaidShards(
}
export function buildPaidShardArgs(files: string[], timeoutMs: number): string[] {
return ['test', ...files, '--retry', '2', `--timeout=${timeoutMs}`];
return ['test', ...files, '--retry', '1', `--timeout=${timeoutMs}`];
}
/**