Garry Tan 84f52f3bad feat: eval persistence with auto-compare against previous run ...

EvalCollector accumulates test results during eval runs, writes JSON to
~/.gstack-dev/evals/{version}-{branch}-{tier}-{timestamp}.json, prints
a summary table, and automatically compares against the previous run.

- EvalCollector class with addTest() / finalize() / summary table
- findPreviousRun() prefers same branch, falls back to any branch
- compareEvalResults() matches tests by name, detects improved/regressed
- extractToolSummary() counts tool types from transcript events
- formatComparison() renders delta table with per-test + aggregate diffs
- Wire into skill-e2e.test.ts (recordE2E helper) and skill-llm-eval.test.ts
- 19 unit tests for collector + comparison functions
- schema_version: 1 for forward compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-14 03:49:47 -05:00

..

eval-store.test.ts

feat: eval persistence with auto-compare against previous run

2026-03-14 03:49:47 -05:00

eval-store.ts

feat: eval persistence with auto-compare against previous run

2026-03-14 03:49:47 -05:00

llm-judge.ts

feat: 3-tier eval suite with planted-bug outcome testing (EVALS=1)

2026-03-14 01:17:36 -05:00

session-runner.test.ts

feat: stream-json NDJSON parser for real-time E2E progress

2026-03-14 03:49:36 -05:00

session-runner.ts

feat: stream-json NDJSON parser for real-time E2E progress

2026-03-14 03:49:36 -05:00

skill-parser.ts

feat: 3-tier eval suite with planted-bug outcome testing (EVALS=1)

2026-03-14 01:17:36 -05:00