fix(evals): judges honor the eval-model resolution chain + real 429 backoff

callJudge inlined GSTACK_EVAL_MODEL_JUDGE || sonnet, silently ignoring
the global GSTACK_EVAL_MODEL override every other eval call site honors
via lib/eval-model.ts. New 'judge' kind in DEFAULTS (sonnet — the D1a
pin-on-regressors calibration stands; model CHOICE unchanged) and
callJudge resolves through it: explicit arg > GSTACK_EVAL_MODEL_JUDGE >
GSTACK_EVAL_MODEL > default.

429 handling upgraded from one fixed 1s retry (reliably lost races at
CI concurrency) to three jittered exponential retries (~1s/4s/16s),
honoring the server's retry-after when present.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-29 04:47:18 +00:00
co-authored by Claude Fable 5
parent 63f8829578
commit e9643e131f
2 changed files with 31 additions and 12 deletions
+3
View File
@@ -15,6 +15,8 @@
* capture — AskUserQuestion SDK capture runs: sonnet (D1a)
* warmup — PTY warm-up ping (cheapest thing that answers): haiku
* distill — free-text distillation (cheap, structured): haiku (pinned)
* judge — LLM-judge rubric calls: sonnet (D1a pin-on-regressors — the
* Haiku A/B regressed the doc-rubric family; see llm-judge.ts)
*/
// `as const satisfies` keeps EvalModelKind the literal union
@@ -28,6 +30,7 @@ const DEFAULTS = {
capture: "claude-sonnet-4-6",
warmup: "claude-haiku-4-5",
distill: "claude-haiku-4-5-20251001",
judge: "claude-sonnet-4-6",
} as const satisfies Record<string, string>;
export type EvalModelKind = keyof typeof DEFAULTS;