mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-11 15:39:04 +02:00
Merge origin/main (v1.65.0.0 fork port wave 2) into test-evals-ci-speedup
Second overlapping-wave merge; resolutions compose intent: - TEST_ROOTS: ours is the superset (main also wired ios-qa/daemon/test; ours additionally has ios-qa/scripts + browser-skills). package.json 'test' keeps routing through the canonical strict runner. - gbrainAvailable: main fixed the same load-flake with a strictly better mechanism (memoized stat-based PATH scan, no subprocess at all) — theirs supersedes this branch's memoized-exec probe. Main also made the query timeout env-overridable (GSTACK_BRAIN_TIMEOUT_MS). - Model defaults: adopted main's lib/eval-model.ts abstraction (one resolution point, env-overridable per kind) and applied decision D1a inside it: capture defaults to Sonnet (Opus opt-in via explicit arg or GSTACK_EVAL_MODEL_CAPTURE); test pins updated to follow. - Parent watchdog: main's rewrite (named parameterized tick, driven deterministically by its test via __testInternals__, plus handoff suppression semantics from session persistence) supersedes this branch's env-tunable interval; adopted their server + test wholesale. - windows-free-tests: ours (curated bun run test:windows) — main's hand-list grew by one more file, which the curated runner subsumes automatically; that drift is the reason for D11. - context-skills 0-for-26 fix: both waves made the IDENTICAL fix; kept this branch's comment (carries the receipts). - .gitignore: main's superset (also ignores Package.resolved — their never-commit call; untracked the copy this branch had committed). Verified: 239-test merge battery green, watchdog 8/8, eval-model 5/5, actionlint clean, eval:select works. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* Host-neutral eval/harness model resolution (fork port wave 2, G cluster).
|
||||
*
|
||||
* Model IDs were hardcoded at six call sites across the eval helpers and the
|
||||
* distill bin, so an environment that pins a different model (CI cost
|
||||
* control, a model migration, an air-gapped proxy alias) had to patch source.
|
||||
* One resolution point, env-overridable:
|
||||
*
|
||||
* GSTACK_EVAL_MODEL_<KIND> (e.g. GSTACK_EVAL_MODEL_WARMUP) — per-kind
|
||||
* GSTACK_EVAL_MODEL — global
|
||||
* explicit argument — caller wins
|
||||
* per-kind default — last resort
|
||||
*
|
||||
* Kinds and their defaults:
|
||||
* capture — AskUserQuestion SDK capture runs (quality matters): opus
|
||||
* warmup — PTY warm-up ping (cheapest thing that answers): haiku
|
||||
* distill — free-text distillation (cheap, structured): haiku (pinned)
|
||||
*/
|
||||
|
||||
// `as const satisfies` keeps EvalModelKind the literal union
|
||||
// 'capture' | 'warmup' | 'distill' — a `Record<string, string>` annotation
|
||||
// would widen it to string and let any typo through the type gate.
|
||||
const DEFAULTS = {
|
||||
// D1a (2026-08 test-infra review): capture runs default to Sonnet, matching
|
||||
// session-runner — the old Opus default was an inconsistency between
|
||||
// runners, not a choice; tests needing Opus pass it explicitly or set
|
||||
// GSTACK_EVAL_MODEL_CAPTURE.
|
||||
capture: "claude-sonnet-4-6",
|
||||
warmup: "claude-haiku-4-5",
|
||||
distill: "claude-haiku-4-5-20251001",
|
||||
} as const satisfies Record<string, string>;
|
||||
|
||||
export type EvalModelKind = keyof typeof DEFAULTS;
|
||||
|
||||
export function resolveEvalModel(
|
||||
kind: EvalModelKind,
|
||||
explicit?: string | null,
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): string {
|
||||
if (explicit) return explicit;
|
||||
const perKind = env[`GSTACK_EVAL_MODEL_${kind.toUpperCase()}`];
|
||||
if (perKind) return perKind;
|
||||
if (env.GSTACK_EVAL_MODEL) return env.GSTACK_EVAL_MODEL;
|
||||
const fallback = DEFAULTS[kind];
|
||||
if (!fallback) throw new Error(`resolveEvalModel: unknown kind "${kind}"`);
|
||||
return fallback;
|
||||
}
|
||||
Reference in New Issue
Block a user