Commit Graph

2 Commits

Author SHA1 Message Date
Garry Tan 02925cfc7a feat: wire costs[] from modelUsage into eval results
Extract per-model token usage from resultLine.modelUsage (including
cache tokens and exact API cost), flow CostEntry[] through EvalCollector,
aggregate in finalize(). Extend CostEntry with cache_read_input_tokens,
cache_creation_input_tokens, cost_usd. computeCosts() prefers exact
cost_usd over MODEL_PRICING when available (~4x more accurate with
prompt caching).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 16:47:27 -05:00
Garry Tan e7347c2f8f feat: stream-json NDJSON parser for real-time E2E progress
Switch session-runner from buffered `--output-format json` to streaming
`--output-format stream-json --verbose`. Parses NDJSON line-by-line for
real-time tool-by-tool progress on stderr during 3-5 min E2E runs.

- Extract testable `parseNDJSON()` function (pure, no I/O)
- Count turns per assistant event (not per text block)
- Add `transcript: any[]` to SkillTestResult, remove dead `messages` field
- Reconstruct allText from transcript for browse error scanning
- 8 unit tests for parser (malformed lines, empty input, turn counting)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-14 03:49:36 -05:00