Garry Tan 6b85262422 feat(eval): parametric overlay-efficacy harness (runner + fixtures) ...

`test/helpers/agent-sdk-runner.ts` wraps @anthropic-ai/claude-agent-sdk
with explicit `AgentSdkResult` types, process-level API concurrency
semaphore, and 3-shape 429 retry (thrown error, result-message error,
mid-stream SDKRateLimitEvent). Pins the local claude binary via
`pathToClaudeCodeExecutable`.

`test/fixtures/overlay-nudges.ts` holds the typed registry. Two
fixtures for the first measurement: `opus-4-7-fanout-toy` (3-file read)
and `opus-4-7-fanout-realistic` (mixed-tool audit). Strict validator
rejects duplicate ids, non-integer trials, unsafe overlay paths, non-safe
id chars, and missing overlay files at module load.

Adding a future overlay nudge eval = one fixture entry.

2026-04-23 09:13:58 -07:00

..

providers

feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040)

2026-04-19 17:50:31 +08:00

agent-sdk-runner.ts

feat(eval): parametric overlay-efficacy harness (runner + fixtures)

2026-04-23 09:13:58 -07:00

benchmark-judge.ts

feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040)

2026-04-19 17:50:31 +08:00

benchmark-runner.ts

feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040)

2026-04-19 17:50:31 +08:00

codex-session-runner.ts

fix: enforce Codex 1024-char description limit + auto-heal stale installs (v0.11.9.0) (#391)

2026-03-23 08:44:08 -07:00

e2e-helpers.ts

feat: remove trigger guard + proactive opt-out prompt (#457)

2026-03-24 18:07:36 -07:00

eval-store.test.ts

feat: QA restructure, browser ref staleness, eval efficiency metrics (v0.4.0) (#83)

2026-03-15 23:55:39 -05:00

eval-store.ts

feat: worktree isolation for E2E tests + infrastructure elegance (v0.11.12.0) (#425)

2026-03-23 23:05:22 -07:00

gemini-session-runner.test.ts

feat: Gemini CLI E2E tests (v0.9.2.0) (#252)

2026-03-20 08:30:09 -07:00

gemini-session-runner.ts

feat: Gemini CLI E2E tests (v0.9.2.0) (#252)

2026-03-20 08:30:09 -07:00

llm-judge.ts

feat: mode-posture energy fix for /plan-ceo-review and /office-hours (v1.1.2.0) (#1065)

2026-04-19 05:44:39 +08:00

observability.test.ts

fix: never clean up observability artifacts — partial file persists after finalize

2026-03-14 12:37:38 -05:00

pricing.ts

feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040)

2026-04-19 17:50:31 +08:00

session-runner.test.ts

feat: stream-json NDJSON parser for real-time E2E progress

2026-03-14 03:49:36 -05:00

session-runner.ts

fix(checkpoint): rename /checkpoint → /context-save + /context-restore (v1.0.1.0) (#1064)

2026-04-19 08:38:19 +08:00

skill-parser.ts

feat: content security — 4-layer prompt injection defense for pair-agent (#815)

2026-04-06 14:41:06 -07:00

tool-map.ts

feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040)

2026-04-19 17:50:31 +08:00

touchfiles.ts

feat(v1.5.2.0): Opus 4.7 migration — model overlay, voice, routing (#1117)

2026-04-22 01:06:22 -07:00