CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-14 17:05:28 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: default cross-model workflows to frontier models

Browse Source
This commit is contained in:
Garry Tan
2026-09-09 02:48:34 +00:00
parent 0530392821
commit 939da0c203
69 changed files with 603 additions and 255 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+16 -2
View File
@@ -136,6 +136,20 @@ never touches a symlinked or non-gstack directory.
## Testing & evals
Codex evals and the GPT benchmark adapter default to `gpt-6-astra`:
explicit model > `GSTACK_CODEX_MODEL` > default. Claude capture and judge
defaults are `claude-fable-5-1`, resolved through `lib/eval-model.ts`:
- Claude session, PTY, and Agent SDK eval runners and the Claude benchmark adapter: explicit model > `EVALS_MODEL` > `GSTACK_EVAL_MODEL_CAPTURE` > `GSTACK_EVAL_MODEL` > default.
- Shared judge calls (including benchmark quality scoring): explicit model > `GSTACK_EVAL_MODEL_JUDGE` > `GSTACK_EVAL_MODEL` > default. `EVALS_MODEL` applies to capture runners, not judges.
Warmup stays on `claude-haiku-4-5`; distill stays on
`claude-haiku-4-5-20251001`. Explicit test and historical benchmark model
selections still win. Known frontier defaults are maintained in releases;
there is no automatic model discovery. Paid-run costs shown below are
historical estimates from before this default change, not measurements of
the new defaults.
### Setup
```bash
@@ -287,7 +301,7 @@ Artifacts are never cleaned up — they accumulate in `~/.gstack-dev/` for post-
### Tier 3: LLM-as-judge (~$0.15/run)
Uses Claude Sonnet to score generated SKILL.md docs on three dimensions.
Uses `claude-fable-5-1` by default to score generated SKILL.md docs on three dimensions.
Override the judge model per run with `GSTACK_EVAL_MODEL_JUDGE`:
- **Clarity** — Can an AI agent understand the instructions without ambiguity?
@@ -300,7 +314,7 @@ Each dimension is scored 1-5. Threshold: every dimension must score **≥ 4**. T
# Needs ANTHROPIC_API_KEY in .env — included in bun run test:evals
```
- Uses `claude-sonnet-4-6` for scoring stability
- Resolves the judge model through `lib/eval-model.ts`, using the override order above
- Tests live in `test/skill-llm-eval.test.ts`
- Calls the Anthropic API directly (not `claude -p`), so it works from anywhere including inside Claude Code