mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-01 10:50:39 +02:00
fix(evals): judges honor the eval-model resolution chain + real 429 backoff
callJudge inlined GSTACK_EVAL_MODEL_JUDGE || sonnet, silently ignoring the global GSTACK_EVAL_MODEL override every other eval call site honors via lib/eval-model.ts. New 'judge' kind in DEFAULTS (sonnet — the D1a pin-on-regressors calibration stands; model CHOICE unchanged) and callJudge resolves through it: explicit arg > GSTACK_EVAL_MODEL_JUDGE > GSTACK_EVAL_MODEL > default. 429 handling upgraded from one fixed 1s retry (reliably lost races at CI concurrency) to three jittered exponential retries (~1s/4s/16s), honoring the server's retry-after when present. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
63f8829578
commit
e9643e131f
@@ -15,6 +15,8 @@
|
||||
* capture — AskUserQuestion SDK capture runs: sonnet (D1a)
|
||||
* warmup — PTY warm-up ping (cheapest thing that answers): haiku
|
||||
* distill — free-text distillation (cheap, structured): haiku (pinned)
|
||||
* judge — LLM-judge rubric calls: sonnet (D1a pin-on-regressors — the
|
||||
* Haiku A/B regressed the doc-rubric family; see llm-judge.ts)
|
||||
*/
|
||||
|
||||
// `as const satisfies` keeps EvalModelKind the literal union
|
||||
@@ -28,6 +30,7 @@ const DEFAULTS = {
|
||||
capture: "claude-sonnet-4-6",
|
||||
warmup: "claude-haiku-4-5",
|
||||
distill: "claude-haiku-4-5-20251001",
|
||||
judge: "claude-sonnet-4-6",
|
||||
} as const satisfies Record<string, string>;
|
||||
|
||||
export type EvalModelKind = keyof typeof DEFAULTS;
|
||||
|
||||
Reference in New Issue
Block a user