mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-09 15:56:02 +02:00
feat: 3-tier eval suite with planted-bug outcome testing (EVALS=1)
Adds comprehensive eval infrastructure: - Tier 1 (free): 13 new static tests — cross-skill path consistency, QA structure validation, greptile format, planted-bug fixture validation - Tier 2 (Agent SDK E2E): /qa quick, /review with pre-built git repo, 3 planted-bug outcome evals (static, SPA, checkout — each with 5 bugs) - Tier 3 (LLM judge): QA workflow quality, health rubric clarity, cross-skill consistency, baseline score pinning New fixtures: 3 HTML pages with 15 total planted bugs, ground truth JSON, review-eval-vuln.rb, eval-baselines.json. Shared llm-judge.ts helper (DRY). Unified EVALS=1 flag replaces SKILL_E2E + ANTHROPIC_API_KEY checks. `bun run test:evals` runs everything that costs money (~$4/run). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
5155fe3a28
commit
76803d789a
@@ -105,7 +105,7 @@
|
||||
- [ ] CI/CD integration — `/qa` as GitHub Action step, fail PR if health score drops (P2, M)
|
||||
- [ ] Accessibility audit mode — `--a11y` flag for focused accessibility testing (P3, S)
|
||||
- [ ] Greptile training feedback loop — export suppression patterns to Greptile team for model improvement (P3, S)
|
||||
- [ ] E2E test cost tracking — track cumulative API spend, warn if over threshold (P3, S)
|
||||
- [x] E2E test cost tracking — track cumulative API spend, warn if over threshold (P3, S)
|
||||
- [ ] E2E model pinning — pin E2E tests to claude-sonnet-4-6 for cost efficiency, add retry:2 for flaky LLM (P2, XS)
|
||||
- [ ] Smart default QA tier — after a few runs, check index.md for user's usual tier pick, skip the question (P2, S)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user