mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-09 22:48:57 +02:00
Garry TanandClaude Fable 5
e99e53cc71
Second overlapping-wave merge; resolutions compose intent: - TEST_ROOTS: ours is the superset (main also wired ios-qa/daemon/test; ours additionally has ios-qa/scripts + browser-skills). package.json 'test' keeps routing through the canonical strict runner. - gbrainAvailable: main fixed the same load-flake with a strictly better mechanism (memoized stat-based PATH scan, no subprocess at all) — theirs supersedes this branch's memoized-exec probe. Main also made the query timeout env-overridable (GSTACK_BRAIN_TIMEOUT_MS). - Model defaults: adopted main's lib/eval-model.ts abstraction (one resolution point, env-overridable per kind) and applied decision D1a inside it: capture defaults to Sonnet (Opus opt-in via explicit arg or GSTACK_EVAL_MODEL_CAPTURE); test pins updated to follow. - Parent watchdog: main's rewrite (named parameterized tick, driven deterministically by its test via __testInternals__, plus handoff suppression semantics from session persistence) supersedes this branch's env-tunable interval; adopted their server + test wholesale. - windows-free-tests: ours (curated bun run test:windows) — main's hand-list grew by one more file, which the curated runner subsumes automatically; that drift is the reason for D11. - context-skills 0-for-26 fix: both waves made the IDENTICAL fix; kept this branch's comment (carries the receipts). - .gitignore: main's superset (also ignores Package.resolved — their never-commit call; untracked the copy this branch had committed). Verified: 239-test merge battery green, watchdog 8/8, eval-model 5/5, actionlint clean, eval:select works. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
48 lines
2.0 KiB
TypeScript
48 lines
2.0 KiB
TypeScript
/**
|
|
* Host-neutral eval/harness model resolution (fork port wave 2, G cluster).
|
|
*
|
|
* Model IDs were hardcoded at six call sites across the eval helpers and the
|
|
* distill bin, so an environment that pins a different model (CI cost
|
|
* control, a model migration, an air-gapped proxy alias) had to patch source.
|
|
* One resolution point, env-overridable:
|
|
*
|
|
* GSTACK_EVAL_MODEL_<KIND> (e.g. GSTACK_EVAL_MODEL_WARMUP) — per-kind
|
|
* GSTACK_EVAL_MODEL — global
|
|
* explicit argument — caller wins
|
|
* per-kind default — last resort
|
|
*
|
|
* Kinds and their defaults:
|
|
* capture — AskUserQuestion SDK capture runs (quality matters): opus
|
|
* warmup — PTY warm-up ping (cheapest thing that answers): haiku
|
|
* distill — free-text distillation (cheap, structured): haiku (pinned)
|
|
*/
|
|
|
|
// `as const satisfies` keeps EvalModelKind the literal union
|
|
// 'capture' | 'warmup' | 'distill' — a `Record<string, string>` annotation
|
|
// would widen it to string and let any typo through the type gate.
|
|
const DEFAULTS = {
|
|
// D1a (2026-08 test-infra review): capture runs default to Sonnet, matching
|
|
// session-runner — the old Opus default was an inconsistency between
|
|
// runners, not a choice; tests needing Opus pass it explicitly or set
|
|
// GSTACK_EVAL_MODEL_CAPTURE.
|
|
capture: "claude-sonnet-4-6",
|
|
warmup: "claude-haiku-4-5",
|
|
distill: "claude-haiku-4-5-20251001",
|
|
} as const satisfies Record<string, string>;
|
|
|
|
export type EvalModelKind = keyof typeof DEFAULTS;
|
|
|
|
export function resolveEvalModel(
|
|
kind: EvalModelKind,
|
|
explicit?: string | null,
|
|
env: NodeJS.ProcessEnv = process.env,
|
|
): string {
|
|
if (explicit) return explicit;
|
|
const perKind = env[`GSTACK_EVAL_MODEL_${kind.toUpperCase()}`];
|
|
if (perKind) return perKind;
|
|
if (env.GSTACK_EVAL_MODEL) return env.GSTACK_EVAL_MODEL;
|
|
const fallback = DEFAULTS[kind];
|
|
if (!fallback) throw new Error(`resolveEvalModel: unknown kind "${kind}"`);
|
|
return fallback;
|
|
}
|