gstack/docs/eval-models.md

Eval model rules: .gstack/eval-models.json

A repository that has rules about which models its evals may use (for example "Fable in smoke tests only", "no gpt-5.4-mini", "the newest model of each family") writes them as data in .gstack/eval-models.json. gstack-eval-plan checks every arm against them: arms, dry-run, pilot start, and every call row of price-gate. gstack never reads these rules from prose in CLAUDE.md or any other document. Plan: field list 2, W9 (docs/designs/field-list-2-2026-10-16/PLAN.md).

The file

{
  "schema_version": 1,
  "allow": ["claude-*", "gpt-*"],
  "deny": ["gpt-5.4-mini"],
  "scopes": { "full": { "deny": ["claude-fable-*"] } },
  "families": {
    "claude-opus": ["claude-opus-5-5", "claude-opus-5", "claude-opus-4-8"],
    "claude-fable": ["claude-fable-5-1", "claude-fable-5"],
    "gpt-sol": ["gpt-6.1-sol", "gpt-6-sol"]
  }
}

Key	Required	Meaning
schema_version	yes	1
allow	yes	Model ids or globs an arm may use. An id that no entry matches fails, so a misspelled or new model is never admitted by default. Use ["*"] to allow everything not denied.
deny	no	Ids or globs no arm may use. Checked first.
scopes	no	Per eval scope (smoke or full), an extra allow and/or deny list. "Fable in smoke only" is scopes.full.deny: ["claude-fable-*"].
families	no	Family name to literal ids, newest first. With this key present, an arm must be the newest id of its family (EVAL_MODEL_OLD_GENERATION otherwise), and an arm in no family fails EVAL_MODEL_GENERATION_UNRESOLVED instead of passing unchecked.

Globs use * (any run of characters) and ? (one character); every other character is literal. A rule matches the id as written, the id without a provider prefix (openai/gpt-5.4-mini matches gpt-5.4-mini), and a Claude id without its date suffix (claude-opus-5-5-20261001 matches claude-opus-5-5). An unknown key, an unknown scope, a glob inside families, or an id listed in two families makes the whole file invalid (EVAL_MODELS_INVALID), and then no arm passes.

The rules apply whatever gstack's own model catalog (lib/model-catalog.ts) holds: an uncatalogued model is allowed or denied by the file alone. The catalog only checks that families is current. If a family lists the catalog's current frontier or smart model after another id, the file is stale and invalid.

Order of checks

For each arm at its scope, the first rule that refuses decides:

1. deny → EVAL_MODEL_DENIED
2. scopes.<scope>.deny → EVAL_MODEL_DENIED
3. no allow entry matches → EVAL_MODEL_DENIED (unknown ids fail closed)
4. scopes.<scope>.allow is set and does not match → EVAL_MODEL_DENIED
5. families is set: in no family → EVAL_MODEL_GENERATION_UNRESOLVED; not the family's first id → EVAL_MODEL_OLD_GENERATION

Scope

price-gate checks each call row at that row's scope (GSTACK_EVAL_CALLS v1, docs/eval-calls.md). arms, dry-run and pilot start take --scope smoke|full, which defaults to full. So a smoke-only model fails unless the command says it is a smoke run.

Which copy is read

By default the reviewed copy is read: git show origin/<base>:.gstack/eval-models.json (--base, default origin/HEAD's branch). A branch cannot loosen the rules it is checked against. When the working-tree copy differs, an EVAL_MODELS_NOTE: line says so. --models-from head lints with the working-tree copy instead. Use it for the PR that adds or changes the file; the source label says head (unreviewed). When neither copy exists, nothing is checked and the line says EVAL_MODELS: none .... When the base cannot be resolved but a working-tree copy exists, the command refuses (EVAL_MODELS_INVALID) rather than skipping the rules.

Output

EVAL_MODEL: claude-fable-5-1 scope=full denied (scopes.full.deny claude-fable-*) where=dry-run --arms
EVAL_MODEL: claude-opus-5-5 scope=full ok (allow claude-*; newest in families.claude-opus) where=dry-run --arms
EVAL_MODELS: fail refused=1/2 source=origin/main@0123456789ab:.gstack/eval-models.json

Each refusal also prints a result line on stderr ending in its code. dry-run records the check as checks.model_rules in dry-run.json and fails DRY_RUN_RESULT. pilot start reserves nothing. price-gate writes no receipt.