From e3d7f49c7449d3733c50895464208c325910a3ef Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Thu, 23 Apr 2026 18:42:58 -0700 Subject: [PATCH] feat(v1.10.1.0): overlay efficacy harness + Opus 4.7 fanout nudge removal (#1166) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor: export readOverlay from model-overlay resolver Needed by the overlay-efficacy eval harness to resolve INHERIT directives without going through generateModelOverlay's full TemplateContext. * chore: add @anthropic-ai/claude-agent-sdk@0.2.117 dep Pinned exact for SDK event-shape stability. Used by the overlay-efficacy harness to drive the model through a closer-to-real Claude Code harness than `claude -p`. * feat(preflight): sanity check for agent-sdk + overlay resolver Verifies: SDK loads, claude-opus-4-7 is a live API model, SDKMessage event shape matches assumptions, readOverlay resolves INHERIT directives and includes expected content. Run with `bun run scripts/preflight-agent-sdk.ts`. PREFLIGHT OK on first run, $0.013 API spend. * feat(eval): parametric overlay-efficacy harness (runner + fixtures) `test/helpers/agent-sdk-runner.ts` wraps @anthropic-ai/claude-agent-sdk with explicit `AgentSdkResult` types, process-level API concurrency semaphore, and 3-shape 429 retry (thrown error, result-message error, mid-stream SDKRateLimitEvent). Pins the local claude binary via `pathToClaudeCodeExecutable`. `test/fixtures/overlay-nudges.ts` holds the typed registry. Two fixtures for the first measurement: `opus-4-7-fanout-toy` (3-file read) and `opus-4-7-fanout-realistic` (mixed-tool audit). Strict validator rejects duplicate ids, non-integer trials, unsafe overlay paths, non-safe id chars, and missing overlay files at module load. Adding a future overlay nudge eval = one fixture entry. * test(eval): unit tests for agent-sdk-runner (36 tests, free tier) Stub `queryProvider` feeds hand-crafted SDKMessage streams. Covers: happy-path shape, all 3 rate-limit shapes + retry, workspace reset on retry, persistent 429 -> `RateLimitExhaustedError`, non-429 propagation, process-level concurrency cap, options propagation, artifact path uniqueness, cost/turn mapping, and every validator rejection case. * test(eval): paid periodic overlay-efficacy harness `test/skill-e2e-overlay-harness.test.ts` iterates OVERLAY_FIXTURES, runs two arms per fixture (overlay-ON, overlay-OFF) at N=10 trials with bounded concurrency. Arms use SDK preset `claude_code` so both include the real Claude Code system prompt; overlay-ON appends the resolved overlay text. Saves per-trial raw event streams to `~/.gstack/projects//transcripts/` for forensic recovery. Gated on `EVALS=1 && EVALS_TIER=periodic`. ~$3/run (40 trials). * test: register overlay harness in touchfiles (both maps) Entries for `overlay-harness-opus-4-7-fanout-toy` and `opus-4-7-fanout-realistic` in E2E_TOUCHFILES (deps: model-overlays/, fixtures file, runner, resolver) and E2E_TIERS (`periodic`). Passes `test/touchfiles.test.ts` completeness check. * fix(opus-4.7): remove "Fan out explicitly" overlay nudge Measured counterproductive under the new SDK harness. Baseline Opus 4.7 emits first-turn parallel tool_use blocks 70% of the time on a 3-file read prompt. With the custom nudge: 10%. With Anthropic's own canonical `` block from their parallel-tool-use docs: 0%. Both overlays suppress fanout; neither improves it. On realistic multi-tool prompts (audit a project: read files + glob + summarize), Opus 4.7 never fans out in first turn regardless of overlay. Zero of 20 trials. Not a prompt problem. Keeping the other three nudges (effort-match, batch questions, literal interpretation) pending their own measurement. Harness is ready for follow-up fixtures — add one entry to `test/fixtures/overlay-nudges.ts` to measure any overlay bullet. Cost of investigation: ~$7 total across 3 eval runs. * chore: bump version and changelog (v1.6.5.0) Co-Authored-By: Claude Opus 4.7 (1M context) * feat(eval): extend OverlayFixture with allowedTools, maxTurns, direction Per-fixture tool allowlist unblocks measuring nudges that need Edit/Write (e.g. literal-interpretation 'fix the failing tests' needs write access). Per-fixture maxTurns lets harder prompts run longer without changing the default. `direction` is cosmetic metadata for test output labeling. Also adds reusable predicates and metrics: - lowerIsBetter20Pct / higherIsBetter20Pct — 20% lift threshold vs baseline - bashToolCallCount — count of Bash tool_use across the session - turnsToCompletion — SDK-reported num_turns at result - uniqueFilesEdited — Edit/Write/MultiEdit file_path set size test/skill-e2e-overlay-harness.test.ts now threads fixture.allowedTools and fixture.maxTurns through runArm. * test(eval): 3 more overlay fixtures to measure remaining Claude nudges Measures three overlay bullets that haven't been tested yet: - claude-dedicated-tools-vs-bash — claude.md says 'prefer Read/Edit/Write/ Glob/Grep over cat/sed/find/grep'. Fixture prompts 'list every TypeScript file under src/ and tell me what each exports' and counts Bash tool_use across the session. Overlay-ON should drop it by >=20%. - opus-4-7-effort-match-trivial — opus-4-7.md says 'simple file reads don't need deep reasoning.' Fixture uses a trivial one-file prompt (config.json lookup) and measures turns_used. Overlay-ON should be <=80% of baseline turns. - opus-4-7-literal-interpretation — opus-4-7.md says 'fix ALL failing tests, not just the obvious one.' Fixture seeds three failing test files with deliberately distinct failure modes and counts unique files edited. Overlay-ON should touch >=20% more files. Adding a fourth fixture for any remaining overlay nudge is a single entry. The harness is now proven on: fanout (deleted after measurement), dedicated tools, effort-match, and literal-interpretation. * fix(eval): handle SDK max-turns throw gracefully Some @anthropic-ai/claude-agent-sdk versions throw from the query generator when maxTurns is reached, instead of emitting a result message with subtype='error_max_turns'. The runner treated that as a non-retryable error and killed the whole periodic run on the first fixture that exceeded its turn cap. Added isMaxTurnsError() detector and a catch branch that synthesizes an AgentSdkResult from events captured before the throw, with exitReason='error_max_turns' and costUsd=0 (unknown from the thrown path). The metric function still runs against whatever assistant turns were collected, so the trial produces a usable number. Hoisted events/assistantTurns/toolCalls/assistantTextParts and the timing counters out of the inner try so the catch branch can read them. No behavior change on the success path or on rate-limit retry paths. * test(eval): bump maxTurns to 15 for claude-dedicated-tools-vs-bash The prompt 'list every TypeScript file under src/ and tell me what each exports' needs 1 turn for Glob + ~5 for Reads + 1 for summary. Default maxTurns=5 was not enough; prior run threw from the SDK on this fixture and tanked the whole periodic eval. Bumping to 15 gives headroom. The runner now also handles max-turns gracefully even if a future fixture underestimates, so this is belt and suspenders. * test(eval): Sonnet 4.6 variants of the 5 Opus-4.7 fixtures Same overlays, same prompts, same metrics, `model: 'claude-sonnet-4-6'`. Tests whether the overlays behave differently on a weaker Claude model where baseline behavior is shakier. Sonnet trials cost ~3-4x less than Opus so these 5 add ~$4.50 to a full run. Measurement result from the first paired run (100 trials total, ~$14.55): - **Sonnet + effort-match shows real overlay benefit.** With the overlay on, Sonnet takes 2.5 turns on a trivial `What's the version in config.json?` prompt. Without, it takes exactly 3.0 turns in all 10 trials. ~17% reduction, below the 20% pass threshold but the signal is clean: overlay-ON distribution [2,2,2,2,2,3,3,3,3,3] vs overlay-OFF [3,3,3,3,3,3,3,3,3,3]. - All other Sonnet dimensions flat (fanout, dedicated-tools, literal interpretation). Same as Opus on those axes. - Opus effort-match remains flat (2.60 vs 2.50, +4% slower with overlay). Implication: model-stratified. The overlay stack helps Sonnet on some axes where it does nothing on Opus. Wholesale removal would hurt Sonnet. Per-nudge per-model measurement is the right move going forward. * chore: bump version to 1.10.1.0 Updates VERSION, package.json, CHANGELOG header, and TODOS completion marker from 1.6.5.0 to 1.10.1.0. --------- Co-authored-by: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 95 ++++ TODOS.md | 25 +- VERSION | 2 +- bun.lock | 185 +++++++ model-overlays/opus-4-7.md | 22 - package.json | 3 +- scripts/preflight-agent-sdk.ts | 133 +++++ scripts/resolvers/model-overlay.ts | 2 +- test/agent-sdk-runner.test.ts | 725 +++++++++++++++++++++++++ test/fixtures/overlay-nudges.ts | 487 +++++++++++++++++ test/helpers/agent-sdk-runner.ts | 509 +++++++++++++++++ test/helpers/touchfiles.ts | 22 + test/skill-e2e-overlay-harness.test.ts | 320 +++++++++++ 13 files changed, 2489 insertions(+), 41 deletions(-) create mode 100644 scripts/preflight-agent-sdk.ts create mode 100644 test/agent-sdk-runner.test.ts create mode 100644 test/fixtures/overlay-nudges.ts create mode 100644 test/helpers/agent-sdk-runner.ts create mode 100644 test/skill-e2e-overlay-harness.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 36139183..4e45cf13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,100 @@ # Changelog +## [1.10.1.0] - 2026-04-23 + +## **We tried to make Opus 4.7 faster with a prompt. Measurement said it got slower. Pulled the bullet.** + +gstack shipped a "Fan out explicitly" overlay nudge in `model-overlays/opus-4-7.md` +back in v1.5.2.0. The idea: tell Opus 4.7 to emit multiple tool calls in one +assistant turn instead of one per turn, so "read three files" takes one API +round-trip instead of three. Sounded obvious. This release removes that +bullet after measuring that it actively hurt performance, and ships the eval +harness we used to prove it so you can measure your own overlay changes. + +### The numbers that matter + +Source: new `test/skill-e2e-overlay-harness.test.ts`, N=10 trials per arm per +fixture, 40 trials per run, ~$3 per run. Pinned to `claude-opus-4-7` via +Anthropic's published Agent SDK (`@anthropic-ai/claude-agent-sdk@0.2.117`) +with `pathToClaudeCodeExecutable` set to the locally-installed `claude` binary +(2.1.118). Metric: number of parallel `tool_use` blocks in the first assistant +turn. + +| Prompt text in overlay | First-turn fanout rate (toy: read 3 files) | Lift vs baseline | +|---|---|---| +| No overlay (default Claude Code system prompt only) | **70%** (7/10) | baseline | +| gstack's original "Fan out explicitly" nudge (v1.5.2.0 through v1.6.3.0) | 10% (1/10) | **-60%** | +| Anthropic's own canonical `` text from their parallel-tool-use docs | **0%** (0/10) | **-70%** | + +On a realistic multi-file audit prompt (`read app.ts + config.ts + README.md, +glob src/*.ts, summarize`), Opus 4.7 never fanned out in the first turn at all, +regardless of overlay. Zero of 20 trials. The nudge had nothing to grip. + +Total cost of the investigation: **$7** across three eval runs. + +### What this means for you + +If you ship system-prompt nudges for Claude, measure them. Anthropic's own +published best-practice text dropped our fanout rate to zero. That's not a +claim about Anthropic, it's a claim about measurement: the model, the SDK, +the binary, and the context all move under the advice, and the advice sits +still. The harness is in the repo now. Run +`EVALS=1 EVALS_TIER=periodic bun test test/skill-e2e-overlay-harness.test.ts`. +Three dollars per run. + +### Itemized changes + +#### Fixed + +- `model-overlays/opus-4-7.md` — removed the "Fan out explicitly" block. The + other three nudges (effort-match, batch questions, literal interpretation) + are untested and stay in for now. They're candidates for their own + measurement in a follow-up PR. + +#### Added + +- `test/skill-e2e-overlay-harness.test.ts` — periodic-tier eval that iterates a + typed fixture registry and runs A/B arms through `@anthropic-ai/claude-agent-sdk`. + Uses SDK preset `claude_code` so the arms include Claude Code's real system + prompt; overlay-ON appends the resolved overlay text. Saves per-trial raw + event streams for forensic recovery. Gated on both `EVALS=1` and + `EVALS_TIER=periodic`. +- `test/fixtures/overlay-nudges.ts` — typed `OverlayFixture` registry with + strict validator. Adding a future nudge to measure = one fixture entry. + First two fixtures: `opus-4-7-fanout-toy` and `opus-4-7-fanout-realistic`. +- `test/helpers/agent-sdk-runner.ts` — parametric SDK wrapper with explicit + `AgentSdkResult` types, process-level API concurrency semaphore, and + three-shape 429 retry (thrown error, result-message error, mid-stream + `SDKRateLimitEvent`). Binary pinning via `pathToClaudeCodeExecutable`. +- `test/agent-sdk-runner.test.ts` — 36 free-tier unit tests covering happy + path, all three rate-limit shapes, persistent-429 `RateLimitExhaustedError`, + non-429 propagation, options propagation, concurrency cap, and every + validator rejection case. +- `scripts/preflight-agent-sdk.ts` — 20-line sanity check that confirms the + SDK loads, `claude-opus-4-7` is a live API model, the `SDKMessage` event + shape matches assumptions, and the overlay resolver produces the expected + text. Run manually before paid runs if you suspect drift. Costs ~$0.013. +- `@anthropic-ai/claude-agent-sdk@0.2.117` in `devDependencies`. Exact pin, + no caret — SDK event shapes can drift on minor versions. + +#### Changed + +- `scripts/resolvers/model-overlay.ts` — exported `readOverlay` so the eval + harness can resolve `{{INHERIT:claude}}` directives without synthesizing a + full `TemplateContext`. + +#### For contributors + +- `test/helpers/touchfiles.ts` — registered the new eval in both + `E2E_TOUCHFILES` (deps: `model-overlays/**`, `overlay-nudges.ts`, runner, + resolver) and `E2E_TIERS` (`periodic`). Passes the + `test/touchfiles.test.ts` completeness check. +- The harness is deliberately parametric. Adding a second overlay nudge + measurement (for the remaining three nudges in `opus-4-7.md`, or any + future nudge in any overlay file) is a single entry in + `test/fixtures/overlay-nudges.ts`. Total incremental effort: ~15 minutes + per fixture. + ## [1.10.0.0] - 2026-04-23 ## **Plan reviews walk you through each issue again, and every question is now a real decision brief.** diff --git a/TODOS.md b/TODOS.md index 5256ec29..cfe61548 100644 --- a/TODOS.md +++ b/TODOS.md @@ -18,22 +18,6 @@ **Priority:** P3 (nice-to-have, not blocking anyone yet) **Depends on:** `/context-save` + `/context-restore` rename stable in production (v1.0.1.0+). Research: does Conductor expose a spawn-workspace CLI? -## P0: Verify Opus 4.7 fanout nudge inside Claude Code harness (next rev) - -**What:** Re-run the fanout A/B from `test/skill-e2e-opus-47.test.ts` against Opus 4.7 **inside Claude Code's interactive harness**, not via `claude -p`. The current eval calls `claude -p` as a subprocess, which does not load SKILL.md content as system context and uses different tool wiring than the live Claude Code session. Build a small harness (Claude Code extension hook, direct API call with the same system prompt Claude Code uses, or a scripted MCP invocation) that reproduces the real tool_use context, then run the same 3-file-read A/B with and without the `model-overlays/opus-4-7.md` overlay. Record parallel-tool-call count in the first assistant turn for each arm. - -**Why:** v1.6.1.0 shipped a rewritten "Fan out explicitly" nudge with a concrete tool_use example (`[Read(a), Read(b), Read(c)]`). Under `claude -p` on `claude-opus-4-7`, both overlay-ON and overlay-OFF arms emitted zero parallel tool calls in the first turn. The routing A/B worked fine in the same harness (3/3 positives routed correctly), so the gap is specific to fanout, and likely specific to how `claude -p` constructs system prompts and tool schemas. Without measurement inside the real harness, we do not know whether the nudge ever lands for a real user. The PR went to production with the fanout claim asserted but unverified; this TODO closes that loop. - -**Pros:** Produces the "actually shipped fanout" measurement the ship-quality review flagged as missing. If the nudge works in Claude Code harness, we can gate it with a `periodic` eval and stop worrying. If it does not, we know to rewrite or drop the nudge rather than carry dead prompt weight. Either answer is better than the current "unverified." - -**Cons:** Requires instrumenting Claude Code's harness (or a faithful replica) rather than the easier `claude -p` path. A faithful replica needs the same system prompt, the same tool definitions, and the same stop-sequence handling. Estimated one afternoon to wire, plus $3-5 per eval run. - -**Context:** See `~/.gstack/projects/garrytan-gstack/evals/1.6.0.0-feat-opus-4.7-migration-e2e-opus-47-*.json` for the raw transcripts showing 0 parallel calls in first turn across both arms. The overlay is at `model-overlays/opus-4-7.md` with an explicit wrong/right tool_use example. The eval file at `test/skill-e2e-opus-47.test.ts` has the full setup including per-skill SKILL.md install, CLAUDE.md routing block, and overlay inlining. - -**Effort:** M (human: ~1 day / CC: ~45 min for the harness wiring, plus the eval run cost) -**Priority:** P0 (ship-quality commitment from v1.6.1.0 — do not let it drift) -**Depends on / blocked by:** Access to Claude Code's system prompt + tool schema (or a reproducible way to mirror them). May require a small MCP server or a direct Messages API call that mirrors Claude Code's session setup. - ## P0: PACING_UPDATES_V0 — Louise's fatigue root cause (V1.1) **What:** Implement the pacing overhaul extracted from PLAN_TUNING_V1. Full design in `docs/designs/PACING_UPDATES_V0.md`. Requires: session-state model, `phase` field in question-log schema, registry extension for dynamic findings, pacing as skill-template control flow (not preamble prose), `bin/gstack-flip-decision` command, migration-prompt budget rule, first-run preamble audit, ranking threshold calibration from real V0 data, one-way-door uncapped rule, concrete verification values. @@ -1268,6 +1252,15 @@ Shipped in v0.6.5. TemplateContext in gen-skill-docs.ts bakes skill name into pr ## Completed +### Overlay efficacy harness + Opus 4.7 fanout nudge removal (v1.10.1.0) +- Built `test/skill-e2e-overlay-harness.test.ts`, a parametric periodic-tier eval that drives `@anthropic-ai/claude-agent-sdk` and measures first-turn fanout rate (overlay-ON vs overlay-OFF) across registered fixtures +- Measured the original "Fan out explicitly" overlay nudge: baseline Opus 4.7 = 70% first-turn fanout on toy prompt, with our nudge = 10%, with Anthropic's own canonical `` text = 0% +- Removed the counterproductive nudge from `model-overlays/opus-4-7.md` +- Shipped 36-test free-tier unit suite for the SDK runner + strict fixture validator +- Registered `overlay-harness-opus-4-7-fanout-{toy,realistic}` in E2E_TOUCHFILES and E2E_TIERS +- Total investigation cost: ~$7 across 3 eval runs +**Completed:** v1.10.1.0 + ### CI eval pipeline (v0.9.9.0) - GitHub Actions eval upload on Ubicloud runners ($0.006/run) - Within-file test concurrency (test() → testConcurrentIfSelected()) diff --git a/VERSION b/VERSION index 02bd4cb8..3647b707 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.10.0.0 +1.10.1.0 diff --git a/bun.lock b/bun.lock index 4af27675..56b62d4e 100644 --- a/bun.lock +++ b/bun.lock @@ -13,17 +13,38 @@ "puppeteer-core": "^24.40.0", }, "devDependencies": { + "@anthropic-ai/claude-agent-sdk": "0.2.117", "@anthropic-ai/sdk": "^0.78.0", }, }, }, "packages": { + "@anthropic-ai/claude-agent-sdk": ["@anthropic-ai/claude-agent-sdk@0.2.117", "", { "dependencies": { "@anthropic-ai/sdk": "^0.81.0", "@modelcontextprotocol/sdk": "^1.29.0" }, "optionalDependencies": { "@anthropic-ai/claude-agent-sdk-darwin-arm64": "0.2.117", "@anthropic-ai/claude-agent-sdk-darwin-x64": "0.2.117", "@anthropic-ai/claude-agent-sdk-linux-arm64": "0.2.117", "@anthropic-ai/claude-agent-sdk-linux-arm64-musl": "0.2.117", "@anthropic-ai/claude-agent-sdk-linux-x64": "0.2.117", "@anthropic-ai/claude-agent-sdk-linux-x64-musl": "0.2.117", "@anthropic-ai/claude-agent-sdk-win32-arm64": "0.2.117", "@anthropic-ai/claude-agent-sdk-win32-x64": "0.2.117" }, "peerDependencies": { "zod": "^4.0.0" } }, "sha512-pVBss1Vu0w87nKCBhWtjMggSgCh6GVUtdRmuE58ZvXv0E2q0JcnUCQHehmn92BAW0+VCwPY8q/k7uKWkgwz/gA=="], + + "@anthropic-ai/claude-agent-sdk-darwin-arm64": ["@anthropic-ai/claude-agent-sdk-darwin-arm64@0.2.117", "", { "os": "darwin", "cpu": "arm64" }, "sha512-ZeC/Lz8XMKQ5w+GmjTziPR8bSSarBtNCJMkMAYRT9ekNmyXSWXEwGLENe5TDDmtpzNNzAB1mQNuIYoqTsqgV3w=="], + + "@anthropic-ai/claude-agent-sdk-darwin-x64": ["@anthropic-ai/claude-agent-sdk-darwin-x64@0.2.117", "", { "os": "darwin", "cpu": "x64" }, "sha512-DKyggGzzpDcr9S435xlpbpwkEYKZNbePSekug75tJclK8l4ddD9+M9BFgMiSUq9F1Zt53kUaRDihDu/cBKvkdQ=="], + + "@anthropic-ai/claude-agent-sdk-linux-arm64": ["@anthropic-ai/claude-agent-sdk-linux-arm64@0.2.117", "", { "os": "linux", "cpu": "arm64" }, "sha512-jyHmyZQavpPOe3zxBRX3KbdOAJ8JwZ8m/wMr5bhHhhcstugm/vJx6IIs7D44VvFjk+8sqdvR2ZrliL8PUcJL0g=="], + + "@anthropic-ai/claude-agent-sdk-linux-arm64-musl": ["@anthropic-ai/claude-agent-sdk-linux-arm64-musl@0.2.117", "", { "os": "linux", "cpu": "arm64" }, "sha512-bJU5gEOmM4VCOn4h8vipOKgdhPATePQ23mMpvyVqtVyipWppHfOUfVkqXb+SrF/hfkNSMYxDuoKxbJ+MmKtGjg=="], + + "@anthropic-ai/claude-agent-sdk-linux-x64": ["@anthropic-ai/claude-agent-sdk-linux-x64@0.2.117", "", { "os": "linux", "cpu": "x64" }, "sha512-Zb5PXKrDNbQ1dyNYwxZMNL+F2Dhgjh9f9B21wZUJqkhJL69hRJwJyxO42HiNmB2zGCaTxQTyjPhLdB/eQJo74Q=="], + + "@anthropic-ai/claude-agent-sdk-linux-x64-musl": ["@anthropic-ai/claude-agent-sdk-linux-x64-musl@0.2.117", "", { "os": "linux", "cpu": "x64" }, "sha512-LIkKTAYZGugEVssAuWCPqlDWSqhVZAveNPNsfKLbuG1naIMCR04fUqil6i3d3mAAfk7FaS5D4IdHp45psi+GDw=="], + + "@anthropic-ai/claude-agent-sdk-win32-arm64": ["@anthropic-ai/claude-agent-sdk-win32-arm64@0.2.117", "", { "os": "win32", "cpu": "arm64" }, "sha512-uetggH3B83PiH0a9D/5MVXB5Hqnlr2DVajehwAP2x0Mt4DBd632ICnHpu6pnSP+vVkWgq3FgQlkHe91RfP+peA=="], + + "@anthropic-ai/claude-agent-sdk-win32-x64": ["@anthropic-ai/claude-agent-sdk-win32-x64@0.2.117", "", { "os": "win32", "cpu": "x64" }, "sha512-TT4KngAokDTJSvQ2mrAP6ZRkXj50OLj7Tb1zZA4CnkmrrEidgs4KrMx7er1ZwoivngIvCekV9+TbtC9giknr5w=="], + "@anthropic-ai/sdk": ["@anthropic-ai/sdk@0.78.0", "", { "dependencies": { "json-schema-to-ts": "^3.1.1" }, "peerDependencies": { "zod": "^3.25.0 || ^4.0.0" }, "optionalPeers": ["zod"], "bin": { "anthropic-ai-sdk": "bin/cli" } }, "sha512-PzQhR715td/m1UaaN5hHXjYB8Gl2lF9UVhrrGrZeysiF6Rb74Wc9GCB8hzLdzmQtBd1qe89F9OptgB9Za1Ib5w=="], "@babel/runtime": ["@babel/runtime@7.29.2", "", {}, "sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g=="], "@emnapi/runtime": ["@emnapi/runtime@1.10.0", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA=="], + "@hono/node-server": ["@hono/node-server@1.19.14", "", { "peerDependencies": { "hono": "^4" } }, "sha512-GwtvgtXxnWsucXvbQXkRgqksiH2Qed37H9xHZocE5sA3N8O8O8/8FA3uclQXxXVzc9XBZuEOMK7+r02FmSpHtw=="], + "@huggingface/jinja": ["@huggingface/jinja@0.5.7", "", {}, "sha512-OosMEbF/R6zkKNNzqhI7kvKYCpo1F0UeIv46/h4D4UjVEKKd6k3TiV8sgu6fkreX4lbBiRI+lZG8UnXnqVQmEQ=="], "@huggingface/tokenizers": ["@huggingface/tokenizers@0.1.3", "", {}, "sha512-8rF/RRT10u+kn7YuUbUg0OF30K8rjTc78aHpxT+qJ1uWSqxT1MHi8+9ltwYfkFYJzT/oS+qw3JVfHtNMGAdqyA=="], @@ -80,6 +101,8 @@ "@img/sharp-win32-x64": ["@img/sharp-win32-x64@0.34.5", "", { "os": "win32", "cpu": "x64" }, "sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw=="], + "@modelcontextprotocol/sdk": ["@modelcontextprotocol/sdk@1.29.0", "", { "dependencies": { "@hono/node-server": "^1.19.9", "ajv": "^8.17.1", "ajv-formats": "^3.0.1", "content-type": "^1.0.5", "cors": "^2.8.5", "cross-spawn": "^7.0.5", "eventsource": "^3.0.2", "eventsource-parser": "^3.0.0", "express": "^5.2.1", "express-rate-limit": "^8.2.1", "hono": "^4.11.4", "jose": "^6.1.3", "json-schema-typed": "^8.0.2", "pkce-challenge": "^5.0.0", "raw-body": "^3.0.0", "zod": "^3.25 || ^4.0", "zod-to-json-schema": "^3.25.1" }, "peerDependencies": { "@cfworker/json-schema": "^4.1.1" }, "optionalPeers": ["@cfworker/json-schema"] }, "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ=="], + "@ngrok/ngrok": ["@ngrok/ngrok@1.7.0", "", { "optionalDependencies": { "@ngrok/ngrok-android-arm64": "1.7.0", "@ngrok/ngrok-darwin-arm64": "1.7.0", "@ngrok/ngrok-darwin-universal": "1.7.0", "@ngrok/ngrok-darwin-x64": "1.7.0", "@ngrok/ngrok-freebsd-x64": "1.7.0", "@ngrok/ngrok-linux-arm-gnueabihf": "1.7.0", "@ngrok/ngrok-linux-arm64-gnu": "1.7.0", "@ngrok/ngrok-linux-arm64-musl": "1.7.0", "@ngrok/ngrok-linux-x64-gnu": "1.7.0", "@ngrok/ngrok-linux-x64-musl": "1.7.0", "@ngrok/ngrok-win32-arm64-msvc": "1.7.0", "@ngrok/ngrok-win32-ia32-msvc": "1.7.0", "@ngrok/ngrok-win32-x64-msvc": "1.7.0" } }, "sha512-P06o9TpxrJbiRbHQkiwy/rUrlXRupc+Z8KT4MiJfmcdWxvIdzjCaJOdnNkcOTs6DMyzIOefG5tvk/HLdtjqr0g=="], "@ngrok/ngrok-android-arm64": ["@ngrok/ngrok-android-arm64@1.7.0", "", { "os": "android", "cpu": "arm64" }, "sha512-8tco3ID6noSaNy+CMS7ewqPoIkIM6XO5COCzsUp3Wv3XEbMSyn65RN6cflX2JdqLfUCHcMyD0ahr9IEiHwqmbQ=="], @@ -136,10 +159,16 @@ "@types/yauzl": ["@types/yauzl@2.10.3", "", { "dependencies": { "@types/node": "*" } }, "sha512-oJoftv0LSuaDZE3Le4DbKX+KS9G36NzOeSap90UIK0yMA/NhKJhqlSGtNDORNRaIbQfzjXDrQa0ytJ6mNRGz/Q=="], + "accepts": ["accepts@2.0.0", "", { "dependencies": { "mime-types": "^3.0.0", "negotiator": "^1.0.0" } }, "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng=="], + "adm-zip": ["adm-zip@0.5.17", "", {}, "sha512-+Ut8d9LLqwEvHHJl1+PIHqoyDxFgVN847JTVM3Izi3xHDWPE4UtzzXysMZQs64DMcrJfBeS/uoEP4AD3HQHnQQ=="], "agent-base": ["agent-base@7.1.4", "", {}, "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ=="], + "ajv": ["ajv@8.18.0", "", { "dependencies": { "fast-deep-equal": "^3.1.3", "fast-uri": "^3.0.1", "json-schema-traverse": "^1.0.0", "require-from-string": "^2.0.2" } }, "sha512-PlXPeEWMXMZ7sPYOHqmDyCJzcfNrUr3fGNKtezX14ykXOEIvyK81d+qydx89KY5O71FKMPaQ2vBfBFI5NHR63A=="], + + "ajv-formats": ["ajv-formats@3.0.1", "", { "dependencies": { "ajv": "^8.0.0" } }, "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ=="], + "ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], "ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], @@ -162,10 +191,18 @@ "basic-ftp": ["basic-ftp@5.2.0", "", {}, "sha512-VoMINM2rqJwJgfdHq6RiUudKt2BV+FY5ZFezP/ypmwayk68+NzzAQy4XXLlqsGD4MCzq3DrmNFD/uUmBJuGoXw=="], + "body-parser": ["body-parser@2.2.2", "", { "dependencies": { "bytes": "^3.1.2", "content-type": "^1.0.5", "debug": "^4.4.3", "http-errors": "^2.0.0", "iconv-lite": "^0.7.0", "on-finished": "^2.4.1", "qs": "^6.14.1", "raw-body": "^3.0.1", "type-is": "^2.0.1" } }, "sha512-oP5VkATKlNwcgvxi0vM0p/D3n2C3EReYVX+DNYs5TjZFn/oQt2j+4sVJtSMr18pdRr8wjTcBl6LoV+FUwzPmNA=="], + "boolean": ["boolean@3.2.0", "", {}, "sha512-d0II/GO9uf9lfUHH2BQsjxzRJZBdsjgsBiW4BvhWk/3qoKwQFjIDVN19PfX8F2D/r9PCMTtLWjYVCFrpeYUzsw=="], "buffer-crc32": ["buffer-crc32@0.2.13", "", {}, "sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ=="], + "bytes": ["bytes@3.1.2", "", {}, "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg=="], + + "call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="], + + "call-bound": ["call-bound@1.0.4", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "get-intrinsic": "^1.3.0" } }, "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg=="], + "chromium-bidi": ["chromium-bidi@14.0.0", "", { "dependencies": { "mitt": "^3.0.1", "zod": "^3.24.1" }, "peerDependencies": { "devtools-protocol": "*" } }, "sha512-9gYlLtS6tStdRWzrtXaTMnqcM4dudNegMXJxkR0I/CXObHalYeYcAMPrL19eroNZHtJ8DQmu1E+ZNOYu/IXMXw=="], "cliui": ["cliui@8.0.1", "", { "dependencies": { "string-width": "^4.2.0", "strip-ansi": "^6.0.1", "wrap-ansi": "^7.0.0" } }, "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ=="], @@ -174,6 +211,18 @@ "color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], + "content-disposition": ["content-disposition@1.1.0", "", {}, "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g=="], + + "content-type": ["content-type@1.0.5", "", {}, "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA=="], + + "cookie": ["cookie@0.7.2", "", {}, "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w=="], + + "cookie-signature": ["cookie-signature@1.2.2", "", {}, "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg=="], + + "cors": ["cors@2.8.6", "", { "dependencies": { "object-assign": "^4", "vary": "^1" } }, "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw=="], + + "cross-spawn": ["cross-spawn@7.0.6", "", { "dependencies": { "path-key": "^3.1.0", "shebang-command": "^2.0.0", "which": "^2.0.1" } }, "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA=="], + "data-uri-to-buffer": ["data-uri-to-buffer@6.0.2", "", {}, "sha512-7hvf7/GW8e86rW0ptuwS3OcBGDjIi6SZva7hCyWC0yYry2cOPmLIjXAUHI6DK2HsnwJd9ifmt57i8eV2n4YNpw=="], "debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="], @@ -184,6 +233,8 @@ "degenerator": ["degenerator@5.0.1", "", { "dependencies": { "ast-types": "^0.13.4", "escodegen": "^2.1.0", "esprima": "^4.0.1" } }, "sha512-TllpMR/t0M5sqCXfj85i4XaAzxmS5tVA16dqvdkMwGmzI+dXLXnw3J+3Vdv7VKw+ThlTMboK6i9rnZ6Nntj5CQ=="], + "depd": ["depd@2.0.0", "", {}, "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw=="], + "detect-libc": ["detect-libc@2.1.2", "", {}, "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ=="], "detect-node": ["detect-node@2.1.0", "", {}, "sha512-T0NIuQpnTvFDATNuHN5roPwSBG83rFsuO+MXXH9/3N1eFbn4wcPjttvjMLEPWJ0RGUYgQE7cGgS3tNxbqCGM7g=="], @@ -192,18 +243,28 @@ "diff": ["diff@7.0.0", "", {}, "sha512-PJWHUb1RFevKCwaFA9RlG5tCd+FO5iRh9A8HEtkmBH2Li03iJriB6m6JIN4rGz3K3JLawI7/veA1xzRKP6ISBw=="], + "dunder-proto": ["dunder-proto@1.0.1", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.1", "es-errors": "^1.3.0", "gopd": "^1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="], + + "ee-first": ["ee-first@1.1.1", "", {}, "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="], + "emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], + "encodeurl": ["encodeurl@2.0.0", "", {}, "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg=="], + "end-of-stream": ["end-of-stream@1.4.5", "", { "dependencies": { "once": "^1.4.0" } }, "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg=="], "es-define-property": ["es-define-property@1.0.1", "", {}, "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g=="], "es-errors": ["es-errors@1.3.0", "", {}, "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw=="], + "es-object-atoms": ["es-object-atoms@1.1.1", "", { "dependencies": { "es-errors": "^1.3.0" } }, "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA=="], + "es6-error": ["es6-error@4.1.1", "", {}, "sha512-Um/+FxMr9CISWh0bi5Zv0iOD+4cFh5qLeks1qhAopKVAJw3drgKbKySikp7wGhDL0HPeaja0P5ULZrxLkniUVg=="], "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], + "escape-html": ["escape-html@1.0.3", "", {}, "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow=="], + "escape-string-regexp": ["escape-string-regexp@4.0.0", "", {}, "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA=="], "escodegen": ["escodegen@2.1.0", "", { "dependencies": { "esprima": "^4.0.1", "estraverse": "^5.2.0", "esutils": "^2.0.2" }, "optionalDependencies": { "source-map": "~0.6.1" }, "bin": { "esgenerate": "bin/esgenerate.js", "escodegen": "bin/escodegen.js" } }, "sha512-2NlIDTwUWJN0mRPQOdtQBzbUHvdGY2P1VXSyU83Q3xKxM7WHX2Ql8dKq782Q9TgQUNOLEzEYu9bzLNj1q88I5w=="], @@ -214,20 +275,46 @@ "esutils": ["esutils@2.0.3", "", {}, "sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g=="], + "etag": ["etag@1.8.1", "", {}, "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg=="], + "events-universal": ["events-universal@1.0.1", "", { "dependencies": { "bare-events": "^2.7.0" } }, "sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw=="], + "eventsource": ["eventsource@3.0.7", "", { "dependencies": { "eventsource-parser": "^3.0.1" } }, "sha512-CRT1WTyuQoD771GW56XEZFQ/ZoSfWid1alKGDYMmkt2yl8UXrVR4pspqWNEcqKvVIzg6PAltWjxcSSPrboA4iA=="], + + "eventsource-parser": ["eventsource-parser@3.0.8", "", {}, "sha512-70QWGkr4snxr0OXLRWsFLeRBIRPuQOvt4s8QYjmUlmlkyTZkRqS7EDVRZtzU3TiyDbXSzaOeF0XUKy8PchzukQ=="], + + "express": ["express@5.2.1", "", { "dependencies": { "accepts": "^2.0.0", "body-parser": "^2.2.1", "content-disposition": "^1.0.0", "content-type": "^1.0.5", "cookie": "^0.7.1", "cookie-signature": "^1.2.1", "debug": "^4.4.0", "depd": "^2.0.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "finalhandler": "^2.1.0", "fresh": "^2.0.0", "http-errors": "^2.0.0", "merge-descriptors": "^2.0.0", "mime-types": "^3.0.0", "on-finished": "^2.4.1", "once": "^1.4.0", "parseurl": "^1.3.3", "proxy-addr": "^2.0.7", "qs": "^6.14.0", "range-parser": "^1.2.1", "router": "^2.2.0", "send": "^1.1.0", "serve-static": "^2.2.0", "statuses": "^2.0.1", "type-is": "^2.0.1", "vary": "^1.1.2" } }, "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw=="], + + "express-rate-limit": ["express-rate-limit@8.3.2", "", { "dependencies": { "ip-address": "10.1.0" }, "peerDependencies": { "express": ">= 4.11" } }, "sha512-77VmFeJkO0/rvimEDuUC5H30oqUC4EyOhyGccfqoLebB0oiEYfM7nwPrsDsBL1gsTpwfzX8SFy2MT3TDyRq+bg=="], + "extract-zip": ["extract-zip@2.0.1", "", { "dependencies": { "debug": "^4.1.1", "get-stream": "^5.1.0", "yauzl": "^2.10.0" }, "optionalDependencies": { "@types/yauzl": "^2.9.1" }, "bin": { "extract-zip": "cli.js" } }, "sha512-GDhU9ntwuKyGXdZBUgTIe+vXnWj0fppUEtMDL0+idd5Sta8TGpHssn/eusA9mrPr9qNDym6SxAYZjNvCn/9RBg=="], + "fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="], + "fast-fifo": ["fast-fifo@1.3.2", "", {}, "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ=="], + "fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="], + "fd-slicer": ["fd-slicer@1.1.0", "", { "dependencies": { "pend": "~1.2.0" } }, "sha512-cE1qsB/VwyQozZ+q1dGxR8LBYNZeofhEdUNGSMbQD3Gw2lAzX9Zb3uIU6Ebc/Fmyjo9AWWfnn0AUCHqtevs/8g=="], + "finalhandler": ["finalhandler@2.1.1", "", { "dependencies": { "debug": "^4.4.0", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "on-finished": "^2.4.1", "parseurl": "^1.3.3", "statuses": "^2.0.1" } }, "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA=="], + "flatbuffers": ["flatbuffers@25.9.23", "", {}, "sha512-MI1qs7Lo4Syw0EOzUl0xjs2lsoeqFku44KpngfIduHBYvzm8h2+7K8YMQh1JtVVVrUvhLpNwqVi4DERegUJhPQ=="], + "forwarded": ["forwarded@0.2.0", "", {}, "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow=="], + + "fresh": ["fresh@2.0.0", "", {}, "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A=="], + "fsevents": ["fsevents@2.3.2", "", { "os": "darwin" }, "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA=="], + "function-bind": ["function-bind@1.1.2", "", {}, "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA=="], + "get-caller-file": ["get-caller-file@2.0.5", "", {}, "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg=="], + "get-intrinsic": ["get-intrinsic@1.3.0", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "function-bind": "^1.1.2", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "math-intrinsics": "^1.1.0" } }, "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ=="], + + "get-proto": ["get-proto@1.0.1", "", { "dependencies": { "dunder-proto": "^1.0.1", "es-object-atoms": "^1.0.0" } }, "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g=="], + "get-stream": ["get-stream@5.2.0", "", { "dependencies": { "pump": "^3.0.0" } }, "sha512-nBF+F1rAZVCu/p7rjzgA+Yb4lfYXrpl7a6VmJrU8wF9I1CKvP/QwPNZHnOlwbTkY6dvtFIzFMSyQXbLoTQPRpA=="], "get-uri": ["get-uri@6.0.5", "", { "dependencies": { "basic-ftp": "^5.0.2", "data-uri-to-buffer": "^6.0.2", "debug": "^4.3.4" } }, "sha512-b1O07XYq8eRuVzBNgJLstU6FYc1tS6wnMtF1I1D9lE8LxZSOGZ7LhxN54yPP6mGw5f2CkXY2BQUL9Fx41qvcIg=="], @@ -242,16 +329,40 @@ "has-property-descriptors": ["has-property-descriptors@1.0.2", "", { "dependencies": { "es-define-property": "^1.0.0" } }, "sha512-55JNKuIW+vq4Ke1BjOTjM2YctQIvCT7GFzHwmfZPGo5wnrgkid0YQtnAleFSqumZm4az3n2BS+erby5ipJdgrg=="], + "has-symbols": ["has-symbols@1.1.0", "", {}, "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ=="], + + "hasown": ["hasown@2.0.3", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg=="], + + "hono": ["hono@4.12.14", "", {}, "sha512-am5zfg3yu6sqn5yjKBNqhnTX7Cv+m00ox+7jbaKkrLMRJ4rAdldd1xPd/JzbBWspqaQv6RSTrgFN95EsfhC+7w=="], + + "http-errors": ["http-errors@2.0.1", "", { "dependencies": { "depd": "~2.0.0", "inherits": "~2.0.4", "setprototypeof": "~1.2.0", "statuses": "~2.0.2", "toidentifier": "~1.0.1" } }, "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ=="], + "http-proxy-agent": ["http-proxy-agent@7.0.2", "", { "dependencies": { "agent-base": "^7.1.0", "debug": "^4.3.4" } }, "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig=="], "https-proxy-agent": ["https-proxy-agent@7.0.6", "", { "dependencies": { "agent-base": "^7.1.2", "debug": "4" } }, "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw=="], + "iconv-lite": ["iconv-lite@0.7.2", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw=="], + + "inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="], + "ip-address": ["ip-address@10.1.0", "", {}, "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q=="], + "ipaddr.js": ["ipaddr.js@1.9.1", "", {}, "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g=="], + "is-fullwidth-code-point": ["is-fullwidth-code-point@3.0.0", "", {}, "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg=="], + "is-promise": ["is-promise@4.0.0", "", {}, "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ=="], + + "isexe": ["isexe@2.0.0", "", {}, "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw=="], + + "jose": ["jose@6.2.2", "", {}, "sha512-d7kPDd34KO/YnzaDOlikGpOurfF0ByC2sEV4cANCtdqLlTfBlw2p14O/5d/zv40gJPbIQxfES3nSx1/oYNyuZQ=="], + "json-schema-to-ts": ["json-schema-to-ts@3.1.1", "", { "dependencies": { "@babel/runtime": "^7.18.3", "ts-algebra": "^2.0.0" } }, "sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g=="], + "json-schema-traverse": ["json-schema-traverse@1.0.0", "", {}, "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug=="], + + "json-schema-typed": ["json-schema-typed@8.0.2", "", {}, "sha512-fQhoXdcvc3V28x7C7BMs4P5+kNlgUURe2jmUT1T//oBRMDrqy1QPelJimwZGo7Hg9VPV3EQV5Bnq4hbFy2vetA=="], + "json-stringify-safe": ["json-stringify-safe@5.0.1", "", {}, "sha512-ZClg6AaYvamvYEE82d3Iyd3vSSIjQ+odgjaTzRuO3s7toCdFKczob2i0zCh7JE8kWn17yvAWhUVxvqGwUalsRA=="], "long": ["long@5.3.2", "", {}, "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA=="], @@ -262,14 +373,32 @@ "matcher": ["matcher@3.0.0", "", { "dependencies": { "escape-string-regexp": "^4.0.0" } }, "sha512-OkeDaAZ/bQCxeFAozM55PKcKU0yJMPGifLwV4Qgjitu+5MoAfSQN4lsLJeXZ1b8w0x+/Emda6MZgXS1jvsapng=="], + "math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="], + + "media-typer": ["media-typer@1.1.0", "", {}, "sha512-aisnrDP4GNe06UcKFnV5bfMNPBUw4jsLGaWwWfnH3v02GnBuXX2MCVn5RbrWo0j3pczUilYblq7fQ7Nw2t5XKw=="], + + "merge-descriptors": ["merge-descriptors@2.0.0", "", {}, "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g=="], + + "mime-db": ["mime-db@1.54.0", "", {}, "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ=="], + + "mime-types": ["mime-types@3.0.2", "", { "dependencies": { "mime-db": "^1.54.0" } }, "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A=="], + "mitt": ["mitt@3.0.1", "", {}, "sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw=="], "ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="], + "negotiator": ["negotiator@1.0.0", "", {}, "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg=="], + "netmask": ["netmask@2.0.2", "", {}, "sha512-dBpDMdxv9Irdq66304OLfEmQ9tbNRFnFTuZiLo+bD+r332bBmMJ8GBLXklIXXgxd3+v9+KUnZaUR5PJMa75Gsg=="], + "object-assign": ["object-assign@4.1.1", "", {}, "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg=="], + + "object-inspect": ["object-inspect@1.13.4", "", {}, "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew=="], + "object-keys": ["object-keys@1.1.1", "", {}, "sha512-NuAESUOUMrlIXOfHKzD6bpPu3tYt3xvjNdRIQ+FeT0lNb4K8WR70CaDxhuNguS2XG+GjkyMwOzsN5ZktImfhLA=="], + "on-finished": ["on-finished@2.4.1", "", { "dependencies": { "ee-first": "1.1.1" } }, "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg=="], + "once": ["once@1.4.0", "", { "dependencies": { "wrappy": "1" } }, "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w=="], "onnxruntime-common": ["onnxruntime-common@1.24.3", "", {}, "sha512-GeuPZO6U/LBJXvwdaqHbuUmoXiEdeCjWi/EG7Y1HNnDwJYuk6WUbNXpF6luSUY8yASul3cmUlLGrCCL1ZgVXqA=="], @@ -282,8 +411,16 @@ "pac-resolver": ["pac-resolver@7.0.1", "", { "dependencies": { "degenerator": "^5.0.0", "netmask": "^2.0.2" } }, "sha512-5NPgf87AT2STgwa2ntRMr45jTKrYBGkVU36yT0ig/n/GMAa3oPqhZfIQ2kMEimReg0+t9kZViDVZ83qfVUlckg=="], + "parseurl": ["parseurl@1.3.3", "", {}, "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ=="], + + "path-key": ["path-key@3.1.1", "", {}, "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q=="], + + "path-to-regexp": ["path-to-regexp@8.4.2", "", {}, "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA=="], + "pend": ["pend@1.2.0", "", {}, "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg=="], + "pkce-challenge": ["pkce-challenge@5.0.1", "", {}, "sha512-wQ0b/W4Fr01qtpHlqSqspcj3EhBvimsdh0KlHhH8HRZnMsEa0ea2fTULOXOS9ccQr3om+GcGRk4e+isrZWV8qQ=="], + "platform": ["platform@1.3.6", "", {}, "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg=="], "playwright": ["playwright@1.58.2", "", { "dependencies": { "playwright-core": "1.58.2" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-vA30H8Nvkq/cPBnNw4Q8TWz1EJyqgpuinBcHET0YVJVFldr8JDNiU9LaWAE1KqSkRYazuaBhTpB5ZzShOezQ6A=="], @@ -294,6 +431,8 @@ "protobufjs": ["protobufjs@7.5.5", "", { "dependencies": { "@protobufjs/aspromise": "^1.1.2", "@protobufjs/base64": "^1.1.2", "@protobufjs/codegen": "^2.0.4", "@protobufjs/eventemitter": "^1.1.0", "@protobufjs/fetch": "^1.1.0", "@protobufjs/float": "^1.0.2", "@protobufjs/inquire": "^1.1.0", "@protobufjs/path": "^1.1.2", "@protobufjs/pool": "^1.1.0", "@protobufjs/utf8": "^1.1.0", "@types/node": ">=13.7.0", "long": "^5.0.0" } }, "sha512-3wY1AxV+VBNW8Yypfd1yQY9pXnqTAN+KwQxL8iYm3/BjKYMNg4i0owhEe26PWDOMaIrzeeF98Lqd5NGz4omiIg=="], + "proxy-addr": ["proxy-addr@2.0.7", "", { "dependencies": { "forwarded": "0.2.0", "ipaddr.js": "1.9.1" } }, "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg=="], + "proxy-agent": ["proxy-agent@6.5.0", "", { "dependencies": { "agent-base": "^7.1.2", "debug": "^4.3.4", "http-proxy-agent": "^7.0.1", "https-proxy-agent": "^7.0.6", "lru-cache": "^7.14.1", "pac-proxy-agent": "^7.1.0", "proxy-from-env": "^1.1.0", "socks-proxy-agent": "^8.0.5" } }, "sha512-TmatMXdr2KlRiA2CyDu8GqR8EjahTG3aY3nXjdzFyoZbmB8hrBsTyMezhULIXKnC0jpfjlmiZ3+EaCzoInSu/A=="], "proxy-from-env": ["proxy-from-env@1.1.0", "", {}, "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg=="], @@ -302,18 +441,48 @@ "puppeteer-core": ["puppeteer-core@24.40.0", "", { "dependencies": { "@puppeteer/browsers": "2.13.0", "chromium-bidi": "14.0.0", "debug": "^4.4.3", "devtools-protocol": "0.0.1581282", "typed-query-selector": "^2.12.1", "webdriver-bidi-protocol": "0.4.1", "ws": "^8.19.0" } }, "sha512-MWL3XbUCfVgGR0gRsidzT6oKJT2QydPLhMITU6HoVWiiv4gkb6gJi3pcdAa8q4HwjBTbqISOWVP4aJiiyUJvag=="], + "qs": ["qs@6.15.1", "", { "dependencies": { "side-channel": "^1.1.0" } }, "sha512-6YHEFRL9mfgcAvql/XhwTvf5jKcOiiupt2FiJxHkiX1z4j7WL8J/jRHYLluORvc1XxB5rV20KoeK00gVJamspg=="], + + "range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="], + + "raw-body": ["raw-body@3.0.2", "", { "dependencies": { "bytes": "~3.1.2", "http-errors": "~2.0.1", "iconv-lite": "~0.7.0", "unpipe": "~1.0.0" } }, "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA=="], + "require-directory": ["require-directory@2.1.1", "", {}, "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q=="], + "require-from-string": ["require-from-string@2.0.2", "", {}, "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw=="], + "roarr": ["roarr@2.15.4", "", { "dependencies": { "boolean": "^3.0.1", "detect-node": "^2.0.4", "globalthis": "^1.0.1", "json-stringify-safe": "^5.0.1", "semver-compare": "^1.0.0", "sprintf-js": "^1.1.2" } }, "sha512-CHhPh+UNHD2GTXNYhPWLnU8ONHdI+5DI+4EYIAOaiD63rHeYlZvyh8P+in5999TTSFgUYuKUAjzRI4mdh/p+2A=="], + "router": ["router@2.2.0", "", { "dependencies": { "debug": "^4.4.0", "depd": "^2.0.0", "is-promise": "^4.0.0", "parseurl": "^1.3.3", "path-to-regexp": "^8.0.0" } }, "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ=="], + + "safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="], + "semver": ["semver@7.7.4", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA=="], "semver-compare": ["semver-compare@1.0.0", "", {}, "sha512-YM3/ITh2MJ5MtzaM429anh+x2jiLVjqILF4m4oyQB18W7Ggea7BfqdH/wGMK7dDiMghv/6WG7znWMwUDzJiXow=="], + "send": ["send@1.2.1", "", { "dependencies": { "debug": "^4.4.3", "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "etag": "^1.8.1", "fresh": "^2.0.0", "http-errors": "^2.0.1", "mime-types": "^3.0.2", "ms": "^2.1.3", "on-finished": "^2.4.1", "range-parser": "^1.2.1", "statuses": "^2.0.2" } }, "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ=="], + "serialize-error": ["serialize-error@7.0.1", "", { "dependencies": { "type-fest": "^0.13.1" } }, "sha512-8I8TjW5KMOKsZQTvoxjuSIa7foAwPWGOts+6o7sgjz41/qMD9VQHEDxi6PBvK2l0MXUmqZyNpUK+T2tQaaElvw=="], + "serve-static": ["serve-static@2.2.1", "", { "dependencies": { "encodeurl": "^2.0.0", "escape-html": "^1.0.3", "parseurl": "^1.3.3", "send": "^1.2.0" } }, "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw=="], + + "setprototypeof": ["setprototypeof@1.2.0", "", {}, "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw=="], + "sharp": ["sharp@0.34.5", "", { "dependencies": { "@img/colour": "^1.0.0", "detect-libc": "^2.1.2", "semver": "^7.7.3" }, "optionalDependencies": { "@img/sharp-darwin-arm64": "0.34.5", "@img/sharp-darwin-x64": "0.34.5", "@img/sharp-libvips-darwin-arm64": "1.2.4", "@img/sharp-libvips-darwin-x64": "1.2.4", "@img/sharp-libvips-linux-arm": "1.2.4", "@img/sharp-libvips-linux-arm64": "1.2.4", "@img/sharp-libvips-linux-ppc64": "1.2.4", "@img/sharp-libvips-linux-riscv64": "1.2.4", "@img/sharp-libvips-linux-s390x": "1.2.4", "@img/sharp-libvips-linux-x64": "1.2.4", "@img/sharp-libvips-linuxmusl-arm64": "1.2.4", "@img/sharp-libvips-linuxmusl-x64": "1.2.4", "@img/sharp-linux-arm": "0.34.5", "@img/sharp-linux-arm64": "0.34.5", "@img/sharp-linux-ppc64": "0.34.5", "@img/sharp-linux-riscv64": "0.34.5", "@img/sharp-linux-s390x": "0.34.5", "@img/sharp-linux-x64": "0.34.5", "@img/sharp-linuxmusl-arm64": "0.34.5", "@img/sharp-linuxmusl-x64": "0.34.5", "@img/sharp-wasm32": "0.34.5", "@img/sharp-win32-arm64": "0.34.5", "@img/sharp-win32-ia32": "0.34.5", "@img/sharp-win32-x64": "0.34.5" } }, "sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg=="], + "shebang-command": ["shebang-command@2.0.0", "", { "dependencies": { "shebang-regex": "^3.0.0" } }, "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA=="], + + "shebang-regex": ["shebang-regex@3.0.0", "", {}, "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A=="], + + "side-channel": ["side-channel@1.1.0", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.3", "side-channel-list": "^1.0.0", "side-channel-map": "^1.0.1", "side-channel-weakmap": "^1.0.2" } }, "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw=="], + + "side-channel-list": ["side-channel-list@1.0.1", "", { "dependencies": { "es-errors": "^1.3.0", "object-inspect": "^1.13.4" } }, "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w=="], + + "side-channel-map": ["side-channel-map@1.0.1", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3" } }, "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA=="], + + "side-channel-weakmap": ["side-channel-weakmap@1.0.2", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "get-intrinsic": "^1.2.5", "object-inspect": "^1.13.3", "side-channel-map": "^1.0.1" } }, "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A=="], + "smart-buffer": ["smart-buffer@4.2.0", "", {}, "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg=="], "socks": ["socks@2.8.7", "", { "dependencies": { "ip-address": "^10.0.1", "smart-buffer": "^4.2.0" } }, "sha512-HLpt+uLy/pxB+bum/9DzAgiKS8CX1EvbWxI4zlmgGCExImLdiad2iCwXT5Z4c9c3Eq8rP2318mPW2c+QbtjK8A=="], @@ -324,6 +493,8 @@ "sprintf-js": ["sprintf-js@1.1.3", "", {}, "sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA=="], + "statuses": ["statuses@2.0.2", "", {}, "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw=="], + "streamx": ["streamx@2.25.0", "", { "dependencies": { "events-universal": "^1.0.0", "fast-fifo": "^1.3.2", "text-decoder": "^1.1.0" } }, "sha512-0nQuG6jf1w+wddNEEXCF4nTg3LtufWINB5eFEN+5TNZW7KWJp6x87+JFL43vaAUPyCfH1wID+mNVyW6OHtFamg=="], "string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], @@ -338,18 +509,28 @@ "text-decoder": ["text-decoder@1.2.7", "", { "dependencies": { "b4a": "^1.6.4" } }, "sha512-vlLytXkeP4xvEq2otHeJfSQIRyWxo/oZGEbXrtEEF9Hnmrdly59sUbzZ/QgyWuLYHctCHxFF4tRQZNQ9k60ExQ=="], + "toidentifier": ["toidentifier@1.0.1", "", {}, "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA=="], + "ts-algebra": ["ts-algebra@2.0.0", "", {}, "sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw=="], "tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="], "type-fest": ["type-fest@0.13.1", "", {}, "sha512-34R7HTnG0XIJcBSn5XhDd7nNFPRcXYRZrBB2O2jdKqYODldSzBAqzsWoZYYvduky73toYS/ESqxPvkDf/F0XMg=="], + "type-is": ["type-is@2.0.1", "", { "dependencies": { "content-type": "^1.0.5", "media-typer": "^1.1.0", "mime-types": "^3.0.0" } }, "sha512-OZs6gsjF4vMp32qrCbiVSkrFmXtG/AZhY3t0iAMrMBiAZyV9oALtXO8hsrHbMXF9x6L3grlFuwW2oAz7cav+Gw=="], + "typed-query-selector": ["typed-query-selector@2.12.1", "", {}, "sha512-uzR+FzI8qrUEIu96oaeBJmd9E7CFEiQ3goA5qCVgc4s5llSubcfGHq9yUstZx/k4s9dXHVKsE35YWoFyvEqEHA=="], "undici-types": ["undici-types@7.18.2", "", {}, "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w=="], + "unpipe": ["unpipe@1.0.0", "", {}, "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ=="], + + "vary": ["vary@1.1.2", "", {}, "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg=="], + "webdriver-bidi-protocol": ["webdriver-bidi-protocol@0.4.1", "", {}, "sha512-ARrjNjtWRRs2w4Tk7nqrf2gBI0QXWuOmMCx2hU+1jUt6d00MjMxURrhxhGbrsoiZKJrhTSTzbIrc554iKI10qw=="], + "which": ["which@2.0.2", "", { "dependencies": { "isexe": "^2.0.0" }, "bin": { "node-which": "./bin/node-which" } }, "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA=="], + "wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], "wrappy": ["wrappy@1.0.2", "", {}, "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="], @@ -366,6 +547,10 @@ "zod": ["zod@3.25.76", "", {}, "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ=="], + "zod-to-json-schema": ["zod-to-json-schema@3.25.2", "", { "peerDependencies": { "zod": "^3.25.28 || ^4" } }, "sha512-O/PgfnpT1xKSDeQYSCfRI5Gy3hPf91mKVDuYLUHZJMiDFptvP41MSnWofm8dnCm0256ZNfZIM7DSzuSMAFnjHA=="], + + "@anthropic-ai/claude-agent-sdk/@anthropic-ai/sdk": ["@anthropic-ai/sdk@0.81.0", "", { "dependencies": { "json-schema-to-ts": "^3.1.1" }, "peerDependencies": { "zod": "^3.25.0 || ^4.0.0" }, "optionalPeers": ["zod"], "bin": { "anthropic-ai-sdk": "bin/cli" } }, "sha512-D4K5PvEV6wPiRtVlVsJHIUhHAmOZ6IT/I9rKlTf84gR7GyyAurPJK7z9BOf/AZqC5d1DhYQGJNKRmV+q8dGhgw=="], + "onnxruntime-web/onnxruntime-common": ["onnxruntime-common@1.24.0-dev.20251116-b39e144322", "", {}, "sha512-BOoomdHYmNRL5r4iQ4bMvsl2t0/hzVQ3OM3PHD0gxeXu1PmggqBv3puZicEUVOA3AtHHYmqZtjMj9FOfGrATTw=="], } } diff --git a/model-overlays/opus-4-7.md b/model-overlays/opus-4-7.md index d0dfcbf8..858b9a94 100644 --- a/model-overlays/opus-4-7.md +++ b/model-overlays/opus-4-7.md @@ -1,27 +1,5 @@ {{INHERIT:claude}} -**Fan out explicitly.** Opus 4.7 serializes by default. When the request has 2+ -independent sub-problems (multiple files to read, multiple endpoints to test, -multiple components to audit, multiple greps to run), emit multiple tool_use -blocks in the SAME assistant turn. That is how you parallelize. One turn with -N tool calls, not N turns with 1 tool call each. - -Concrete example. If the user says "read foo.ts, bar.ts, and baz.ts": - -Wrong (3 turns): - Turn 1: Read(foo.ts), then you wait for output - Turn 2: Read(bar.ts), then you wait for output - Turn 3: Read(baz.ts) - -Right (1 turn, 3 parallel tool calls): - Turn 1: [Read(foo.ts), Read(bar.ts), Read(baz.ts)] ← three tool_use blocks, - same assistant message - -This applies to Read, Bash, Grep, Glob, WebFetch, Agent/subagent, and any tool -where the sub-calls do not depend on each other's output. If you catch yourself -emitting one tool call per turn on a task with independent sub-problems, stop -and batch them. - **Effort-match the step.** Simple file reads, config checks, command lookups, and mechanical edits don't need deep reasoning. Complete them quickly and move on. Reserve extended thinking for genuinely hard subproblems: architectural tradeoffs, subtle bugs, diff --git a/package.json b/package.json index 200cc875..9ccd0901 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.10.0.0", + "version": "1.10.1.0", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", @@ -61,6 +61,7 @@ "devtools" ], "devDependencies": { + "@anthropic-ai/claude-agent-sdk": "0.2.117", "@anthropic-ai/sdk": "^0.78.0" } } diff --git a/scripts/preflight-agent-sdk.ts b/scripts/preflight-agent-sdk.ts new file mode 100644 index 00000000..9902306c --- /dev/null +++ b/scripts/preflight-agent-sdk.ts @@ -0,0 +1,133 @@ +/** + * Preflight for the overlay efficacy harness. + * + * Confirms, before any paid eval runs: + * 1. `@anthropic-ai/claude-agent-sdk` loads and `query()` is the expected shape. + * 2. `claude-opus-4-7` is a live API model ID (not a Claude Code alias). + * 3. The SDK event stream contains the types we assume (system init, assistant, + * result) with the fields we destructure. + * 4. `scripts/resolvers/model-overlay.ts` resolves `{{INHERIT:claude}}` against + * `opus-4-7.md` AND the resolved text contains the "Fan out explicitly" nudge. + * 5. A local `claude` binary exists at `which claude` so binary pinning is possible. + * + * Run: bun run scripts/preflight-agent-sdk.ts + * + * Exit 0 on success. Exit non-zero with a clear message on any failure. No + * side effects beyond stdout and a ~15 token API call. + */ + +import { query, type SDKMessage } from '@anthropic-ai/claude-agent-sdk'; +import { readOverlay } from './resolvers/model-overlay'; +import { execSync } from 'child_process'; + +async function main() { + const failures: string[] = []; + const pass = (msg: string) => console.log(` ok ${msg}`); + const fail = (msg: string) => { + console.log(` FAIL ${msg}`); + failures.push(msg); + }; + + // 1. Overlay resolver + fanout nudge text + console.log('1. Overlay resolver'); + const resolved = readOverlay('opus-4-7'); + if (!resolved) { + fail("readOverlay('opus-4-7') returned empty"); + } else { + pass(`resolved overlay length: ${resolved.length} chars`); + if (resolved.includes('{{INHERIT:')) { + fail('resolved overlay still contains {{INHERIT:...}} directive'); + } else { + pass('no unresolved INHERIT directives'); + } + if (!/Fan out explicitly/i.test(resolved)) { + fail('resolved overlay does not contain "Fan out explicitly" text'); + } else { + pass('fanout nudge text present in resolved overlay'); + } + } + + // 2. Local claude binary exists + console.log('\n2. Binary pinning'); + let claudePath: string | null = null; + try { + claudePath = execSync('which claude', { encoding: 'utf-8' }).trim(); + pass(`local claude binary: ${claudePath}`); + } catch { + fail('`which claude` failed — cannot pin binary'); + } + + // 3. SDK query end-to-end + console.log('\n3. SDK query end-to-end'); + if (!process.env.ANTHROPIC_API_KEY) { + console.log(' skip ANTHROPIC_API_KEY not set — cannot test live query'); + } else { + try { + const events: SDKMessage[] = []; + const q = query({ + prompt: 'say pong', + options: { + model: 'claude-opus-4-7', + systemPrompt: '', + tools: [], + permissionMode: 'bypassPermissions', + allowDangerouslySkipPermissions: true, + settingSources: [], + maxTurns: 1, + pathToClaudeCodeExecutable: claudePath ?? undefined, + env: { ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY }, + }, + }); + for await (const ev of q) events.push(ev); + pass(`received ${events.length} events`); + + const init = events.find( + (e) => e.type === 'system' && (e as { subtype?: string }).subtype === 'init', + ) as { claude_code_version?: string; model?: string } | undefined; + if (!init) { + fail('no system/init event received'); + } else { + pass(`system init: claude_code_version=${init.claude_code_version}, model=${init.model}`); + } + + const assistantEvents = events.filter((e) => e.type === 'assistant'); + if (assistantEvents.length === 0) { + fail('no assistant events received — model ID may be rejected'); + } else { + pass(`received ${assistantEvents.length} assistant event(s)`); + const first = assistantEvents[0] as { message?: { content?: unknown[] } }; + const content = first.message?.content; + if (!Array.isArray(content)) { + fail('first assistant event has no content[] array'); + } else { + pass(`first assistant content[] has ${content.length} block(s)`); + } + } + + const result = events.find((e) => e.type === 'result') as + | { subtype?: string; total_cost_usd?: number; num_turns?: number } + | undefined; + if (!result) { + fail('no result event received'); + } else { + pass( + `result: subtype=${result.subtype}, cost=$${result.total_cost_usd?.toFixed(4)}, turns=${result.num_turns}`, + ); + } + } catch (err) { + fail(`SDK query threw: ${err instanceof Error ? err.message : String(err)}`); + } + } + + console.log(); + if (failures.length > 0) { + console.log(`PREFLIGHT FAILED: ${failures.length} check(s) failed`); + process.exit(1); + } + console.log('PREFLIGHT OK'); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/scripts/resolvers/model-overlay.ts b/scripts/resolvers/model-overlay.ts index c60a514a..4bbd9641 100644 --- a/scripts/resolvers/model-overlay.ts +++ b/scripts/resolvers/model-overlay.ts @@ -24,7 +24,7 @@ const OVERLAY_DIR = path.resolve(import.meta.dir, '../../model-overlays'); const INHERIT_RE = /^\s*\{\{INHERIT:([a-z0-9-]+(?:\.[0-9]+)*)\}\}\s*\n/; -function readOverlay(model: string, seen: Set = new Set()): string { +export function readOverlay(model: string, seen: Set = new Set()): string { if (seen.has(model)) return ''; // cycle guard seen.add(model); diff --git a/test/agent-sdk-runner.test.ts b/test/agent-sdk-runner.test.ts new file mode 100644 index 00000000..eb256092 --- /dev/null +++ b/test/agent-sdk-runner.test.ts @@ -0,0 +1,725 @@ +/** + * Unit tests for test/helpers/agent-sdk-runner.ts. + * + * Runs in free `bun test` (no API calls). Uses a stub QueryProvider to + * simulate SDK event streams — happy path, rate-limit retries across all + * three shapes, persistent failure, non-retryable error, options + * propagation, concurrency cap. + * + * Also covers validateFixtures() rejections. + */ + +import { describe, test, expect } from 'bun:test'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import type { + SDKMessage, + Options, + Query, +} from '@anthropic-ai/claude-agent-sdk'; +import { + runAgentSdkTest, + toSkillTestResult, + firstTurnParallelism, + isRateLimitThrown, + isRateLimitResult, + isRateLimitEvent, + RateLimitExhaustedError, + __resetSemaphoreForTests, + type QueryProvider, + type AgentSdkResult, +} from '../test/helpers/agent-sdk-runner'; +import { + validateFixtures, + fanoutPass, + type OverlayFixture, +} from '../test/fixtures/overlay-nudges'; + +// --------------------------------------------------------------------------- +// Stub SDK event builders +// --------------------------------------------------------------------------- + +let uuidCounter = 0; +function uuid(): string { + return `00000000-0000-0000-0000-${String(++uuidCounter).padStart(12, '0')}`; +} + +function systemInit(model = 'claude-opus-4-7', version = '2.1.117'): SDKMessage { + return { + type: 'system', + subtype: 'init', + apiKeySource: 'user', + claude_code_version: version, + cwd: '/tmp/x', + tools: ['Read'], + mcp_servers: [], + model, + permissionMode: 'bypassPermissions', + slash_commands: [], + output_style: 'default', + skills: [], + plugins: [], + uuid: uuid(), + session_id: 'test-session', + } as unknown as SDKMessage; +} + +function assistantTurn( + blocks: Array<{ type: 'text'; text: string } | { type: 'tool_use'; name: string; input: unknown }>, +): SDKMessage { + return { + type: 'assistant', + parent_tool_use_id: null, + uuid: uuid(), + session_id: 'test-session', + message: { + id: 'msg_' + uuid(), + type: 'message', + role: 'assistant', + model: 'claude-opus-4-7', + content: blocks.map((b) => ({ ...b })), + stop_reason: 'end_turn', + stop_sequence: null, + usage: { + input_tokens: 10, + output_tokens: 20, + cache_creation_input_tokens: 0, + cache_read_input_tokens: 0, + service_tier: 'standard', + }, + }, + } as unknown as SDKMessage; +} + +function resultSuccess(cost = 0.01, turns = 1): SDKMessage { + return { + type: 'result', + subtype: 'success', + duration_ms: 100, + duration_api_ms: 50, + is_error: false, + num_turns: turns, + result: 'done', + stop_reason: 'end_turn', + total_cost_usd: cost, + usage: { + input_tokens: 10, + output_tokens: 20, + cache_creation_input_tokens: 0, + cache_read_input_tokens: 0, + server_tool_use: {}, + service_tier: 'standard', + }, + modelUsage: {}, + permission_denials: [], + uuid: uuid(), + session_id: 'test-session', + } as unknown as SDKMessage; +} + +function resultRateLimit(): SDKMessage { + return { + type: 'result', + subtype: 'error_during_execution', + duration_ms: 100, + duration_api_ms: 50, + is_error: true, + num_turns: 0, + stop_reason: null, + total_cost_usd: 0, + usage: { + input_tokens: 0, + output_tokens: 0, + cache_creation_input_tokens: 0, + cache_read_input_tokens: 0, + server_tool_use: {}, + service_tier: 'standard', + }, + modelUsage: {}, + permission_denials: [], + errors: ['rate limit exceeded (429)'], + uuid: uuid(), + session_id: 'test-session', + } as unknown as SDKMessage; +} + +function rateLimitEvent(): SDKMessage { + return { + type: 'rate_limit_event', + rate_limit_info: { + status: 'rejected', + rateLimitType: 'five_hour', + }, + uuid: uuid(), + session_id: 'test-session', + } as unknown as SDKMessage; +} + +// --------------------------------------------------------------------------- +// Stub query provider +// --------------------------------------------------------------------------- + +interface StubConfig { + /** One event stream per call. Exhausted calls throw. */ + streams: SDKMessage[][]; + /** Throw this error on the Nth call (0-indexed). */ + throwAt?: number; + throwError?: unknown; + /** Track calls for assertions. */ + calls: Array<{ prompt: string; options: Options | undefined; startedAt: number; endedAt?: number }>; +} + +function makeStubProvider(config: StubConfig): QueryProvider { + let callIdx = -1; + const provider: QueryProvider = (params) => { + callIdx++; + const idx = callIdx; + const startedAt = Date.now(); + const prompt = typeof params.prompt === 'string' ? params.prompt : ''; + config.calls.push({ prompt, options: params.options, startedAt }); + + if (config.throwAt !== undefined && idx === config.throwAt) { + const err = config.throwError ?? new Error('stub throw'); + // Return an async generator that throws on first next(). + const gen = (async function* (): AsyncGenerator { + throw err; + })(); + return gen as unknown as Query; + } + + const stream = config.streams[idx]; + if (!stream) { + const gen = (async function* (): AsyncGenerator { + throw new Error(`stub has no stream for call ${idx}`); + })(); + return gen as unknown as Query; + } + + const gen = (async function* (): AsyncGenerator { + try { + for (const ev of stream) { + yield ev; + } + } finally { + config.calls[idx]!.endedAt = Date.now(); + } + })(); + return gen as unknown as Query; + }; + return provider; +} + +const BASE_OPTS = { + systemPrompt: '', + userPrompt: 'test prompt', + workingDirectory: '/tmp/test-dir', + maxRetries: 3, +}; + +// Reset semaphore before each test that depends on fresh capacity. +function freshSem(cap = 10): void { + __resetSemaphoreForTests(cap); +} + +// --------------------------------------------------------------------------- +// Happy path +// --------------------------------------------------------------------------- + +describe('runAgentSdkTest — happy path', () => { + test('collects events, assistantTurns, toolCalls, and result fields', async () => { + freshSem(); + const stub: StubConfig = { + streams: [ + [ + systemInit(), + assistantTurn([ + { type: 'text', text: 'reading files' }, + { type: 'tool_use', name: 'Read', input: { path: 'a.txt' } }, + { type: 'tool_use', name: 'Read', input: { path: 'b.txt' } }, + ]), + assistantTurn([{ type: 'text', text: 'done' }]), + resultSuccess(0.05, 2), + ], + ], + calls: [], + }; + const result = await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + }); + + expect(result.events.length).toBe(4); + expect(result.assistantTurns.length).toBe(2); + expect(result.toolCalls.length).toBe(2); + expect(result.toolCalls[0]!.tool).toBe('Read'); + expect(result.output).toContain('reading files'); + expect(result.output).toContain('done'); + expect(result.exitReason).toBe('success'); + expect(result.turnsUsed).toBe(2); + expect(result.costUsd).toBe(0.05); + expect(result.sdkClaudeCodeVersion).toBe('2.1.117'); + expect(result.model).toBe('claude-opus-4-7'); + expect(result.firstResponseMs).toBeGreaterThanOrEqual(0); + }); + + test('first-turn parallelism: 3 tool_use blocks in first assistant turn', async () => { + freshSem(); + const stub: StubConfig = { + streams: [ + [ + systemInit(), + assistantTurn([ + { type: 'tool_use', name: 'Read', input: { path: 'a' } }, + { type: 'tool_use', name: 'Read', input: { path: 'b' } }, + { type: 'tool_use', name: 'Read', input: { path: 'c' } }, + ]), + resultSuccess(), + ], + ], + calls: [], + }; + const result = await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + }); + expect(firstTurnParallelism(result.assistantTurns[0])).toBe(3); + }); + + test('first-turn parallelism: 0 when first turn is text-only', async () => { + freshSem(); + const stub: StubConfig = { + streams: [ + [ + systemInit(), + assistantTurn([{ type: 'text', text: 'thinking' }]), + resultSuccess(), + ], + ], + calls: [], + }; + const result = await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + }); + expect(firstTurnParallelism(result.assistantTurns[0])).toBe(0); + }); + + test('first-turn parallelism: 0 when no first turn', () => { + expect(firstTurnParallelism(undefined)).toBe(0); + }); +}); + +// --------------------------------------------------------------------------- +// Options propagation +// --------------------------------------------------------------------------- + +describe('runAgentSdkTest — options propagation', () => { + test('systemPrompt, model, cwd, allowedTools, disallowedTools, permissionMode, settingSources, env, pathToClaudeCodeExecutable reach query()', async () => { + freshSem(); + const stub: StubConfig = { + streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()]], + calls: [], + }; + await runAgentSdkTest({ + systemPrompt: 'you are a test overlay', + userPrompt: 'go', + workingDirectory: '/tmp/spec-dir', + model: 'claude-opus-4-7', + maxTurns: 7, + allowedTools: ['Read', 'Glob'], + disallowedTools: ['Bash', 'Write'], + permissionMode: 'bypassPermissions', + settingSources: [], + env: { ANTHROPIC_API_KEY: 'fake' }, + pathToClaudeCodeExecutable: '/fake/path/claude', + queryProvider: makeStubProvider(stub), + }); + + const opts = stub.calls[0]!.options!; + expect(opts.systemPrompt).toBe('you are a test overlay'); + expect(opts.model).toBe('claude-opus-4-7'); + expect(opts.cwd).toBe('/tmp/spec-dir'); + expect(opts.maxTurns).toBe(7); + expect(opts.tools).toEqual(['Read', 'Glob']); + expect(opts.allowedTools).toEqual(['Read', 'Glob']); + expect(opts.disallowedTools).toEqual(['Bash', 'Write']); + expect(opts.permissionMode).toBe('bypassPermissions'); + expect(opts.allowDangerouslySkipPermissions).toBe(true); + expect(opts.settingSources).toEqual([]); + expect(opts.env).toEqual({ ANTHROPIC_API_KEY: 'fake' }); + expect(opts.pathToClaudeCodeExecutable).toBe('/fake/path/claude'); + }); + + test('empty systemPrompt means no systemPrompt option passed', async () => { + freshSem(); + const stub: StubConfig = { + streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()]], + calls: [], + }; + await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + }); + // systemPrompt is undefined when empty string passed (so SDK uses no override) + expect(stub.calls[0]!.options!.systemPrompt).toBeUndefined(); + }); +}); + +// --------------------------------------------------------------------------- +// Rate-limit retry (three shapes) +// --------------------------------------------------------------------------- + +describe('runAgentSdkTest — rate-limit retry', () => { + test('retryable on thrown 429-shaped error, then succeeds on 2nd attempt', async () => { + freshSem(); + const stub: StubConfig = { + streams: [ + // call 0: throws (handled via throwAt below) + [], + // call 1: success + [systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()], + ], + throwAt: 0, + throwError: Object.assign(new Error('429 too many requests'), { status: 429 }), + calls: [], + }; + const result = await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + maxRetries: 2, + }); + expect(result.exitReason).toBe('success'); + expect(stub.calls.length).toBe(2); + }); + + test('retryable on result-message rate-limit, then succeeds', async () => { + freshSem(); + const stub: StubConfig = { + streams: [ + [systemInit(), resultRateLimit()], + [systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()], + ], + calls: [], + }; + const result = await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + maxRetries: 2, + }); + expect(result.exitReason).toBe('success'); + expect(stub.calls.length).toBe(2); + }); + + test('retryable on mid-stream SDKRateLimitEvent, then succeeds', async () => { + freshSem(); + const stub: StubConfig = { + streams: [ + [systemInit(), rateLimitEvent()], + [systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()], + ], + calls: [], + }; + const result = await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + maxRetries: 2, + }); + expect(result.exitReason).toBe('success'); + expect(stub.calls.length).toBe(2); + }); + + test('onRetry callback is invoked between attempts', async () => { + freshSem(); + const resets: string[] = []; + const stub: StubConfig = { + streams: [ + [], + [systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()], + ], + throwAt: 0, + throwError: Object.assign(new Error('429'), { status: 429 }), + calls: [], + }; + await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + maxRetries: 2, + onRetry: (dir) => resets.push(dir), + }); + expect(resets.length).toBe(1); + expect(resets[0]).toBe('/tmp/test-dir'); + }); + + test('persistent 429 throws RateLimitExhaustedError after maxRetries', async () => { + freshSem(); + const stub: StubConfig = { + streams: [[], [], [], []], // 4 empty streams; throw on each + calls: [], + }; + // Every call throws: + let callCount = 0; + const alwaysThrowProvider: QueryProvider = (params) => { + callCount++; + stub.calls.push({ + prompt: typeof params.prompt === 'string' ? params.prompt : '', + options: params.options, + startedAt: Date.now(), + }); + const gen = (async function* (): AsyncGenerator { + throw Object.assign(new Error('429 always'), { status: 429 }); + })(); + return gen as unknown as Query; + }; + + let caught: unknown = null; + try { + await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: alwaysThrowProvider, + maxRetries: 2, + }); + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(RateLimitExhaustedError); + expect((caught as RateLimitExhaustedError).attempts).toBe(3); // initial + 2 retries + expect(callCount).toBe(3); + }); + + test('non-429 error is NOT retried, propagates immediately', async () => { + __resetSemaphoreForTests(10); + let callCount = 0; + const throwOnce: QueryProvider = () => { + callCount++; + const gen = (async function* (): AsyncGenerator { + throw new Error('generic auth failure'); + })(); + return gen as unknown as Query; + }; + let caught: unknown = null; + try { + await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: throwOnce, + maxRetries: 3, + }); + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(Error); + expect((caught as Error).message).toBe('generic auth failure'); + expect(callCount).toBe(1); + }); +}); + +// --------------------------------------------------------------------------- +// Rate-limit detectors (unit) +// --------------------------------------------------------------------------- + +describe('rate-limit detectors', () => { + test('isRateLimitThrown matches status 429, message, name', () => { + expect(isRateLimitThrown(Object.assign(new Error('boom'), { status: 429 }))).toBe(true); + expect(isRateLimitThrown(new Error('429 Too Many Requests'))).toBe(true); + expect(isRateLimitThrown(new Error('rate-limit exceeded'))).toBe(true); + expect(isRateLimitThrown(Object.assign(new Error('x'), { name: 'RateLimitError' }))).toBe(true); + expect(isRateLimitThrown(new Error('auth failed'))).toBe(false); + expect(isRateLimitThrown(null)).toBe(false); + }); + + test('isRateLimitResult matches error_during_execution with 429-shaped errors', () => { + expect(isRateLimitResult(resultRateLimit())).toBe(true); + expect(isRateLimitResult(resultSuccess())).toBe(false); + }); + + test('isRateLimitEvent matches rate_limit_event with status=rejected', () => { + expect(isRateLimitEvent(rateLimitEvent())).toBe(true); + expect(isRateLimitEvent(resultSuccess())).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// Semaphore concurrency cap +// --------------------------------------------------------------------------- + +describe('runAgentSdkTest — concurrency', () => { + test('process-level semaphore caps concurrent queries', async () => { + __resetSemaphoreForTests(2); + let inFlight = 0; + let peakInFlight = 0; + const slowStub: QueryProvider = () => { + const gen = (async function* (): AsyncGenerator { + inFlight++; + if (inFlight > peakInFlight) peakInFlight = inFlight; + yield systemInit(); + await new Promise((r) => setTimeout(r, 30)); + yield assistantTurn([{ type: 'text', text: 'ok' }]); + yield resultSuccess(); + inFlight--; + })(); + return gen as unknown as Query; + }; + + await Promise.all( + Array.from({ length: 6 }, (_, i) => + runAgentSdkTest({ + ...BASE_OPTS, + userPrompt: `trial-${i}`, + queryProvider: slowStub, + }), + ), + ); + + expect(peakInFlight).toBeLessThanOrEqual(2); + expect(peakInFlight).toBeGreaterThan(0); + }); +}); + +// --------------------------------------------------------------------------- +// toSkillTestResult shape +// --------------------------------------------------------------------------- + +describe('toSkillTestResult', () => { + test('produces a SkillTestResult-shaped object', async () => { + freshSem(); + const stub: StubConfig = { + streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'hi' }]), resultSuccess(0.02, 1)]], + calls: [], + }; + const r = await runAgentSdkTest({ + ...BASE_OPTS, + queryProvider: makeStubProvider(stub), + }); + const s = toSkillTestResult(r); + expect(s.toolCalls).toBeArray(); + expect(s.browseErrors).toBeArray(); + expect(s.exitReason).toBe('success'); + expect(s.duration).toBeNumber(); + expect(s.output).toBe('hi'); + expect(s.costEstimate.estimatedCost).toBe(0.02); + expect(s.costEstimate.turnsUsed).toBe(1); + expect(s.model).toBe('claude-opus-4-7'); + expect(s.firstResponseMs).toBeNumber(); + expect(s.maxInterTurnMs).toBeNumber(); + expect(s.transcript).toBeArray(); + }); +}); + +// --------------------------------------------------------------------------- +// Fixture validator +// --------------------------------------------------------------------------- + +describe('validateFixtures', () => { + function base(overrides: Partial = {}): OverlayFixture { + return { + id: 'test-fixture', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-opus-4-7', + trials: 10, + setupWorkspace: () => {}, + userPrompt: 'go', + metric: () => 0, + pass: fanoutPass, + ...overrides, + }; + } + + test('passes for a valid fixture', () => { + expect(() => validateFixtures([base()])).not.toThrow(); + }); + + test('rejects empty id', () => { + expect(() => validateFixtures([base({ id: '' })])).toThrow(/id must be/); + }); + + test('rejects id with uppercase or unsafe chars', () => { + expect(() => validateFixtures([base({ id: 'Test_Fixture' })])).toThrow(/id must be/); + }); + + test('rejects duplicate ids', () => { + expect(() => validateFixtures([base(), base()])).toThrow(/duplicate fixture id/); + }); + + test('rejects non-integer trials', () => { + expect(() => validateFixtures([base({ trials: 3.5 })])).toThrow(/trials must be/); + }); + + test('rejects trials < 3', () => { + expect(() => validateFixtures([base({ trials: 2 })])).toThrow(/trials must be/); + }); + + test('rejects concurrency < 1', () => { + expect(() => validateFixtures([base({ concurrency: 0 })])).toThrow(/concurrency must be/); + }); + + test('rejects non-integer concurrency', () => { + expect(() => validateFixtures([base({ concurrency: 2.5 })])).toThrow(/concurrency must be/); + }); + + test('rejects empty model', () => { + expect(() => validateFixtures([base({ model: '' })])).toThrow(/model must be/); + }); + + test('rejects empty userPrompt', () => { + expect(() => validateFixtures([base({ userPrompt: '' })])).toThrow(/userPrompt must be/); + }); + + test('rejects absolute overlayPath', () => { + expect(() => validateFixtures([base({ overlayPath: '/etc/passwd' })])).toThrow(/overlayPath must be/); + }); + + test("rejects overlayPath containing '..'", () => { + expect(() => + validateFixtures([base({ overlayPath: '../outside/file.md' })]), + ).toThrow(/overlayPath must be/); + }); + + test('rejects missing overlay file', () => { + expect(() => + validateFixtures([base({ overlayPath: 'model-overlays/nonexistent.md' })]), + ).toThrow(/overlay file not found/); + }); + + test('rejects non-function setupWorkspace', () => { + expect(() => + validateFixtures([base({ setupWorkspace: 'not a function' as unknown as (d: string) => void })]), + ).toThrow(/setupWorkspace must be a function/); + }); + + test('rejects non-function metric', () => { + expect(() => + validateFixtures([base({ metric: null as unknown as (r: AgentSdkResult) => number })]), + ).toThrow(/metric must be a function/); + }); + + test('rejects non-function pass', () => { + expect(() => + validateFixtures([base({ pass: undefined as unknown as OverlayFixture['pass'] })]), + ).toThrow(/pass must be a function/); + }); +}); + +// --------------------------------------------------------------------------- +// fanoutPass predicate +// --------------------------------------------------------------------------- + +describe('fanoutPass predicate', () => { + test('accepts mean lift >= 0.5 AND >=3/10 overlay trials >= 2', () => { + const overlay = [2, 2, 2, 2, 2, 2, 2, 2, 2, 2]; + const off = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]; + expect(fanoutPass({ overlay, off })).toBe(true); + }); + + test('rejects when mean lift < 0.5', () => { + const overlay = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]; + const off = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]; + expect(fanoutPass({ overlay, off })).toBe(false); + }); + + test('rejects when mean lift >= 0.5 but <3 overlay trials emit >=2', () => { + // Mean overlay = 1.2, off = 0.0, lift 1.2 but only 2 trials at >=2 + const overlay = [2, 2, 1, 1, 1, 1, 1, 1, 1, 1]; + const off = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]; + expect(fanoutPass({ overlay, off })).toBe(false); + }); +}); diff --git a/test/fixtures/overlay-nudges.ts b/test/fixtures/overlay-nudges.ts new file mode 100644 index 00000000..0d310201 --- /dev/null +++ b/test/fixtures/overlay-nudges.ts @@ -0,0 +1,487 @@ +/** + * Overlay-efficacy fixture registry. + * + * Each fixture defines a reproducible A/B test for one behavioral nudge + * embedded in a model-overlays/*.md file. The harness at + * test/skill-e2e-overlay-harness.test.ts iterates this registry and runs + * `fixture.trials` A/B trials per fixture, asserting `fixture.pass(arms)`. + * + * Adding a new overlay eval = one entry in this list. The harness handles + * arm wiring, concurrency, artifact storage, rate-limit retries, and the + * cross-harness diagnostic. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import { + firstTurnParallelism, + type AgentSdkResult, +} from '../helpers/agent-sdk-runner'; + +const REPO_ROOT = path.resolve(__dirname, '..', '..'); + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +export interface OverlayFixture { + /** Unique, lowercase/digits/dash only. Used in artifact paths. */ + id: string; + /** Path to the overlay file, relative to repo root. */ + overlayPath: string; + /** API model ID, not the overlay family name. */ + model: string; + /** Integer >= 3. Trials per arm. */ + trials: number; + /** Max concurrent queries for this fixture's arms. Default 3. */ + concurrency?: number; + /** Populate the workspace dir before each trial. */ + setupWorkspace: (dir: string) => void; + /** The prompt the model receives. Non-empty. */ + userPrompt: string; + /** Per-fixture tool allowlist. Omit to use runner default [Read, Glob, Grep, Bash]. */ + allowedTools?: string[]; + /** Max turns per trial. Omit to use runner default (5). */ + maxTurns?: number; + /** + * Direction of the expected effect. `higher_is_better` = overlay should + * increase the metric (e.g. fanout, files touched for literal scope). + * `lower_is_better` = overlay should decrease it (e.g. Bash count, turn count). + * Used only for cosmetic logging in the test output; `pass` is the actual gate. + */ + direction?: 'higher_is_better' | 'lower_is_better'; + /** Compute the per-trial metric from the typed SDK result. */ + metric: (r: AgentSdkResult) => number; + /** Acceptance predicate across all arms' per-trial metrics. */ + pass: (arms: { overlay: number[]; off: number[] }) => boolean; +} + +// --------------------------------------------------------------------------- +// Validation +// --------------------------------------------------------------------------- + +export function validateFixtures(fixtures: OverlayFixture[]): void { + const ids = new Set(); + for (const f of fixtures) { + if (!f.id || !/^[a-z0-9-]+$/.test(f.id)) { + throw new Error( + `fixture id must be non-empty, lowercase/digits/dash only: ${JSON.stringify(f.id)}`, + ); + } + if (ids.has(f.id)) { + throw new Error(`duplicate fixture id: ${f.id}`); + } + ids.add(f.id); + + if (!Number.isInteger(f.trials) || f.trials < 3) { + throw new Error(`${f.id}: trials must be an integer >= 3 (got ${f.trials})`); + } + if ( + f.concurrency !== undefined && + (!Number.isInteger(f.concurrency) || f.concurrency < 1) + ) { + throw new Error( + `${f.id}: concurrency must be an integer >= 1 (got ${f.concurrency})`, + ); + } + + if (!f.model) throw new Error(`${f.id}: model must be non-empty`); + if (!f.userPrompt) throw new Error(`${f.id}: userPrompt must be non-empty`); + + if (path.isAbsolute(f.overlayPath) || f.overlayPath.includes('..')) { + throw new Error( + `${f.id}: overlayPath must be relative and must not contain '..' (got ${f.overlayPath})`, + ); + } + const fullPath = path.resolve(REPO_ROOT, f.overlayPath); + if (!fs.existsSync(fullPath)) { + throw new Error(`${f.id}: overlay file not found at ${f.overlayPath}`); + } + + for (const fn of ['setupWorkspace', 'metric', 'pass'] as const) { + if (typeof f[fn] !== 'function') { + throw new Error(`${f.id}: ${fn} must be a function`); + } + } + } +} + +// --------------------------------------------------------------------------- +// Metric + predicate helpers +// --------------------------------------------------------------------------- + +function mean(xs: number[]): number { + if (xs.length === 0) return 0; + return xs.reduce((a, b) => a + b, 0) / xs.length; +} + +/** + * Standard fanout predicate: overlay mean beats off mean by at least 0.5 + * parallel tool_use blocks in first turn, AND at least 3 of the overlay + * trials emit >= 2 parallel tool_use blocks. + * + * The combined rule catches both "overlay nudges every trial slightly" + * (mean) and "overlay sometimes triggers real fanout" (floor). A single + * 0.5 lift with every trial still emitting 1 call would be suspicious; + * this predicate rejects it. + */ +export function fanoutPass(arms: { overlay: number[]; off: number[] }): boolean { + const lift = mean(arms.overlay) - mean(arms.off); + const floorHits = arms.overlay.filter((n) => n >= 2).length; + return lift >= 0.5 && floorHits >= 3; +} + +/** + * Generic "lower is better" pass predicate: overlay mean should drop the + * metric by at least 20% vs baseline. Used for nudges like "effort-match" + * (fewer turns) and "dedicated tools vs Bash" (fewer Bash calls). + */ +export function lowerIsBetter20Pct(arms: { overlay: number[]; off: number[] }): boolean { + const meanOff = mean(arms.off); + if (meanOff === 0) return mean(arms.overlay) <= meanOff; + return mean(arms.overlay) <= meanOff * 0.8; +} + +/** + * Generic "higher is better" pass predicate: overlay mean should lift the + * metric by at least 20% vs baseline. Used for nudges like "literal + * interpretation" (more files touched when scope is ambiguous). + */ +export function higherIsBetter20Pct(arms: { overlay: number[]; off: number[] }): boolean { + const meanOff = mean(arms.off); + const meanOn = mean(arms.overlay); + if (meanOff === 0) return meanOn > 0; + return meanOn >= meanOff * 1.2; +} + +// --------------------------------------------------------------------------- +// Metrics +// --------------------------------------------------------------------------- + +/** + * Count the total number of Bash tool_use blocks across ALL assistant turns. + * Signal for "dedicated tools over Bash" nudge in claude.md. + */ +export function bashToolCallCount(r: AgentSdkResult): number { + return r.toolCalls.filter((c) => c.tool === 'Bash').length; +} + +/** + * Total turns the session used to complete. Signal for "effort-match the + * step" nudge in opus-4-7.md — trivial prompts should complete quickly. + */ +export function turnsToCompletion(r: AgentSdkResult): number { + return r.turnsUsed; +} + +/** + * Count of unique files the model edited or wrote. Signal for "literal + * interpretation" nudge in opus-4-7.md — "fix the tests" with multiple + * failures should touch all of them. + */ +export function uniqueFilesEdited(r: AgentSdkResult): number { + const touched = new Set(); + for (const call of r.toolCalls) { + if (call.tool === 'Edit' || call.tool === 'Write' || call.tool === 'MultiEdit') { + const input = call.input as { file_path?: string } | null; + if (input?.file_path) touched.add(input.file_path); + } + } + return touched.size; +} + +// --------------------------------------------------------------------------- +// Fixtures +// --------------------------------------------------------------------------- + +export const OVERLAY_FIXTURES: OverlayFixture[] = [ + { + id: 'opus-4-7-fanout-toy', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-opus-4-7', + trials: 10, + concurrency: 3, + setupWorkspace: (dir) => { + fs.writeFileSync(path.join(dir, 'alpha.txt'), 'Alpha file: used in module A.\n'); + fs.writeFileSync(path.join(dir, 'beta.txt'), 'Beta file: used in module B.\n'); + fs.writeFileSync(path.join(dir, 'gamma.txt'), 'Gamma file: used in module C.\n'); + }, + userPrompt: + 'Read alpha.txt, beta.txt, and gamma.txt and summarize each in one line.', + metric: (r) => firstTurnParallelism(r.assistantTurns[0]), + pass: fanoutPass, + }, + { + id: 'opus-4-7-fanout-realistic', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-opus-4-7', + trials: 10, + concurrency: 3, + setupWorkspace: (dir) => { + fs.writeFileSync( + path.join(dir, 'app.ts'), + "import { config } from './config';\nimport { util } from './src/util';\n\nexport function main() { return config.name + ':' + util(); }\n", + ); + fs.writeFileSync( + path.join(dir, 'config.ts'), + "export const config = { name: 'demo', version: 1 };\n", + ); + fs.writeFileSync( + path.join(dir, 'README.md'), + '# demo project\n\nA small demo. Entry: `app.ts`. Config: `config.ts`.\n', + ); + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'util.ts'), + "export function util() { return 'util-result'; }\n", + ); + }, + userPrompt: + 'Audit this project: read app.ts, config.ts, and README.md, and glob for ' + + 'every .ts file under src/. Summarize what you find in 3 bullet points.', + metric: (r) => firstTurnParallelism(r.assistantTurns[0]), + pass: fanoutPass, + }, + + // ------------------------------------------------------------------------- + // claude.md / "Dedicated tools over Bash" + // ------------------------------------------------------------------------- + { + id: 'claude-dedicated-tools-vs-bash', + overlayPath: 'model-overlays/claude.md', + model: 'claude-opus-4-7', + trials: 10, + concurrency: 3, + direction: 'lower_is_better', + // 5 files + summary = needs more than default 5 turns. SDK throws + // instead of returning a result when it hits the cap. + maxTurns: 15, + setupWorkspace: (dir) => { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'src', 'index.ts'), "export const x = 1;\n"); + fs.writeFileSync(path.join(dir, 'src', 'util.ts'), "export function util() { return 42; }\n"); + fs.writeFileSync(path.join(dir, 'src', 'types.ts'), "export type Foo = { a: number };\n"); + fs.writeFileSync(path.join(dir, 'src', 'config.ts'), "export const c = { n: 'demo' };\n"); + fs.writeFileSync(path.join(dir, 'src', 'api.ts'), "export async function fetchFoo() { return null; }\n"); + }, + userPrompt: + "List every TypeScript file under src/ and tell me what each exports. " + + "You may use any tools available.", + // Metric: total Bash tool_use count across the whole session. + // The overlay says "prefer Read/Glob/Grep over cat/find/grep shell." + // A model following that should emit Glob + Read, not Bash ls/find/cat. + metric: bashToolCallCount, + pass: lowerIsBetter20Pct, + }, + + // ------------------------------------------------------------------------- + // opus-4-7.md / "Effort-match the step" + // ------------------------------------------------------------------------- + { + id: 'opus-4-7-effort-match-trivial', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-opus-4-7', + trials: 10, + concurrency: 3, + direction: 'lower_is_better', + maxTurns: 8, + setupWorkspace: (dir) => { + fs.writeFileSync( + path.join(dir, 'config.json'), + '{"name": "demo", "version": "1.0.0"}\n', + ); + }, + userPrompt: "What's the version in config.json?", + // Metric: turns used to reach a result. + // The overlay says "simple lookups don't need deep reasoning." + // A trivial prompt like this should complete in 2 turns + // (Read, respond). Extended deliberation bumps this up. + metric: turnsToCompletion, + pass: lowerIsBetter20Pct, + }, + + // ------------------------------------------------------------------------- + // opus-4-7.md / "Literal interpretation awareness" + // ------------------------------------------------------------------------- + { + id: 'opus-4-7-literal-interpretation', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-opus-4-7', + trials: 10, + concurrency: 3, + direction: 'higher_is_better', + // Allow Edit and Write so the model can fix the tests. + allowedTools: ['Read', 'Glob', 'Grep', 'Bash', 'Edit', 'Write'], + maxTurns: 15, + setupWorkspace: (dir) => { + // Three failing test files. The "obvious" one is named first; the + // other two require the model to enumerate to notice them. + fs.writeFileSync( + path.join(dir, 'auth.test.ts'), + "import { test, expect } from 'bun:test';\n" + + "test('auth returns true', () => { expect(1).toBe(2); });\n", + ); + fs.writeFileSync( + path.join(dir, 'billing.test.ts'), + "import { test, expect } from 'bun:test';\n" + + "test('billing computes total', () => { expect('ten').toBe('10'); });\n", + ); + fs.writeFileSync( + path.join(dir, 'notifications.test.ts'), + "import { test, expect } from 'bun:test';\n" + + "test('notifications fire on save', () => { expect([1,2]).toEqual([1,2,3]); });\n", + ); + fs.writeFileSync( + path.join(dir, 'README.md'), + "# demo\n\nSmall project with three failing test files.\n", + ); + }, + userPrompt: "Fix the failing tests.", + // Metric: number of unique test files the model edited/wrote. + // The overlay says "fix ALL failing tests, not just the obvious one." + // Overlay-ON should touch all 3 test files. Overlay-OFF might stop + // at the first one after making it pass. + metric: uniqueFilesEdited, + pass: higherIsBetter20Pct, + }, + + // ========================================================================= + // Sonnet 4.6 variants of the Opus-4.7 fixtures. + // + // Rationale: /claude.md + /opus-4-7.md overlays measured as no-op or + // counterproductive on Opus 4.7. Before deleting the whole overlay stack, + // check whether weaker Claude models (Sonnet, Haiku) benefit from the same + // nudges. Same overlays, same prompts, same metrics, different model ID. + // Sonnet is ~4x cheaper than Opus so these 5 add ~$3 to a run. + // ========================================================================= + + { + id: 'opus-4-7-fanout-toy-sonnet', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-sonnet-4-6', + trials: 10, + concurrency: 3, + setupWorkspace: (dir) => { + fs.writeFileSync(path.join(dir, 'alpha.txt'), 'Alpha file: used in module A.\n'); + fs.writeFileSync(path.join(dir, 'beta.txt'), 'Beta file: used in module B.\n'); + fs.writeFileSync(path.join(dir, 'gamma.txt'), 'Gamma file: used in module C.\n'); + }, + userPrompt: + 'Read alpha.txt, beta.txt, and gamma.txt and summarize each in one line.', + metric: (r) => firstTurnParallelism(r.assistantTurns[0]), + pass: fanoutPass, + }, + + { + id: 'opus-4-7-fanout-realistic-sonnet', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-sonnet-4-6', + trials: 10, + concurrency: 3, + setupWorkspace: (dir) => { + fs.writeFileSync( + path.join(dir, 'app.ts'), + "import { config } from './config';\nimport { util } from './src/util';\n\nexport function main() { return config.name + ':' + util(); }\n", + ); + fs.writeFileSync( + path.join(dir, 'config.ts'), + "export const config = { name: 'demo', version: 1 };\n", + ); + fs.writeFileSync( + path.join(dir, 'README.md'), + '# demo project\n\nA small demo. Entry: `app.ts`. Config: `config.ts`.\n', + ); + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'util.ts'), + "export function util() { return 'util-result'; }\n", + ); + }, + userPrompt: + 'Audit this project: read app.ts, config.ts, and README.md, and glob for ' + + 'every .ts file under src/. Summarize what you find in 3 bullet points.', + metric: (r) => firstTurnParallelism(r.assistantTurns[0]), + pass: fanoutPass, + }, + + { + id: 'claude-dedicated-tools-vs-bash-sonnet', + overlayPath: 'model-overlays/claude.md', + model: 'claude-sonnet-4-6', + trials: 10, + concurrency: 3, + direction: 'lower_is_better', + maxTurns: 15, + setupWorkspace: (dir) => { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'src', 'index.ts'), "export const x = 1;\n"); + fs.writeFileSync(path.join(dir, 'src', 'util.ts'), "export function util() { return 42; }\n"); + fs.writeFileSync(path.join(dir, 'src', 'types.ts'), "export type Foo = { a: number };\n"); + fs.writeFileSync(path.join(dir, 'src', 'config.ts'), "export const c = { n: 'demo' };\n"); + fs.writeFileSync(path.join(dir, 'src', 'api.ts'), "export async function fetchFoo() { return null; }\n"); + }, + userPrompt: + "List every TypeScript file under src/ and tell me what each exports. " + + "You may use any tools available.", + metric: bashToolCallCount, + pass: lowerIsBetter20Pct, + }, + + { + id: 'opus-4-7-effort-match-trivial-sonnet', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-sonnet-4-6', + trials: 10, + concurrency: 3, + direction: 'lower_is_better', + maxTurns: 8, + setupWorkspace: (dir) => { + fs.writeFileSync( + path.join(dir, 'config.json'), + '{"name": "demo", "version": "1.0.0"}\n', + ); + }, + userPrompt: "What's the version in config.json?", + metric: turnsToCompletion, + pass: lowerIsBetter20Pct, + }, + + { + id: 'opus-4-7-literal-interpretation-sonnet', + overlayPath: 'model-overlays/opus-4-7.md', + model: 'claude-sonnet-4-6', + trials: 10, + concurrency: 3, + direction: 'higher_is_better', + allowedTools: ['Read', 'Glob', 'Grep', 'Bash', 'Edit', 'Write'], + maxTurns: 15, + setupWorkspace: (dir) => { + fs.writeFileSync( + path.join(dir, 'auth.test.ts'), + "import { test, expect } from 'bun:test';\n" + + "test('auth returns true', () => { expect(1).toBe(2); });\n", + ); + fs.writeFileSync( + path.join(dir, 'billing.test.ts'), + "import { test, expect } from 'bun:test';\n" + + "test('billing computes total', () => { expect('ten').toBe('10'); });\n", + ); + fs.writeFileSync( + path.join(dir, 'notifications.test.ts'), + "import { test, expect } from 'bun:test';\n" + + "test('notifications fire on save', () => { expect([1,2]).toEqual([1,2,3]); });\n", + ); + fs.writeFileSync( + path.join(dir, 'README.md'), + "# demo\n\nSmall project with three failing test files.\n", + ); + }, + userPrompt: "Fix the failing tests.", + metric: uniqueFilesEdited, + pass: higherIsBetter20Pct, + }, +]; + +// Validate at module load so a broken fixture fails fast at test startup, +// not mid-run after burning API dollars. +validateFixtures(OVERLAY_FIXTURES); diff --git a/test/helpers/agent-sdk-runner.ts b/test/helpers/agent-sdk-runner.ts new file mode 100644 index 00000000..a4df71d9 --- /dev/null +++ b/test/helpers/agent-sdk-runner.ts @@ -0,0 +1,509 @@ +/** + * Claude Agent SDK wrapper for the overlay-efficacy harness. + * + * This sits alongside session-runner.ts (which drives `claude -p` as a + * subprocess) but runs the model via the published @anthropic-ai/claude-agent-sdk + * instead. The SDK exposes the same harness primitives Claude Code itself uses, + * so overlay-driven behavior change is measured against a closer approximation + * of real Claude Code than the `claude -p` subprocess path provides. + * + * Explicit design rules (from plan review): + * - Use SDK-exported SDKMessage types. No `| unknown` union collapse. + * - Permission surface is explicit: bypassPermissions + settingSources:[] + + * disallowedTools inverse. Without these, the SDK inherits user settings, + * project .claude/, and local hooks, and arms are no longer comparable. + * - Binary pinning via pathToClaudeCodeExecutable. Resolve with `which claude` + * at setup time; the SDK would otherwise use its bundled binary. + * - 3-shape rate-limit detection: thrown error, result-message error subtype, + * mid-stream SDKRateLimitEvent. All three recover on retry. + * - On retry, caller resets workspace via a setupWorkspace callback so + * partial Bash side-effects don't contaminate the next attempt. + * - Process-level semaphore caps concurrent queries across all callers in + * the same bun-test process. Composes with bun's own --concurrent flag. + */ + +import { + query, + type SDKMessage, + type SDKAssistantMessage, + type SDKResultMessage, + type SDKSystemMessage, + type PermissionMode, + type SettingSource, + type Options, +} from '@anthropic-ai/claude-agent-sdk'; +import * as fs from 'fs'; +import * as path from 'path'; +import { execSync } from 'child_process'; +import type { SkillTestResult } from './session-runner'; + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +export interface AgentSdkResult { + /** Full raw event stream for forensic recovery. */ + events: SDKMessage[]; + /** Assistant-typed subset, in order. */ + assistantTurns: SDKAssistantMessage[]; + /** Flat tool-call list, in order of emission. */ + toolCalls: Array<{ tool: string; input: unknown; output: string }>; + /** Concatenated assistant text, newline-joined. */ + output: string; + /** 'success' | 'error_during_execution' | 'error_max_turns' | ... */ + exitReason: string; + turnsUsed: number; + durationMs: number; + firstResponseMs: number; + maxInterTurnMs: number; + costUsd: number; + model: string; + sdkVersion: string; + /** claude_code_version from the SDK's system/init event (authoritative). */ + sdkClaudeCodeVersion: string; + /** Path to the claude binary we pinned. */ + resolvedBinaryPath: string; + /** browse-error pattern scan for SkillTestResult parity. Always empty here. */ + browseErrors: string[]; +} + +/** Signature matching `query()` from the SDK. DI hook for unit tests. */ +export type QueryProvider = typeof query; + +/** Subset of SDK Options['systemPrompt'] we support. */ +export type SystemPromptOption = + | string + | { type: 'preset'; preset: 'claude_code'; append?: string; excludeDynamicSections?: boolean }; + +export interface RunAgentSdkOptions { + /** + * System prompt surface. + * - bare string "" -> omit entirely (SDK default: no system prompt) + * - bare string "...text..." -> REPLACE default with given text (use sparingly) + * - { type:'preset', preset:'claude_code' } -> use Claude Code default + * - { type:'preset', preset:'claude_code', append: "..." } -> default + append + * + * For overlay-efficacy measurement, the preset+append pattern is the right + * one: it measures "does adding overlay text to the REAL Claude Code system + * prompt change behavior" rather than "does the overlay alone (stripped of + * base scaffolding) change behavior". + */ + systemPrompt: SystemPromptOption; + userPrompt: string; + workingDirectory: string; + model?: string; + maxTurns?: number; + allowedTools?: string[]; + disallowedTools?: string[]; + permissionMode?: PermissionMode; + settingSources?: SettingSource[]; + env?: Record; + pathToClaudeCodeExecutable?: string; + testName?: string; + runId?: string; + fixtureId?: string; + queryProvider?: QueryProvider; + /** Max 429 retries per call. Default 3. */ + maxRetries?: number; + /** + * Caller provides this when retry should reset the workspace. The harness + * invokes it with a fresh dir after a rate-limit failure. When omitted, + * retries reuse the original workingDirectory (fine for read-only tests). + */ + onRetry?: (freshDir: string) => void; +} + +export class RateLimitExhaustedError extends Error { + readonly attempts: number; + constructor(attempts: number, cause?: unknown) { + super(`rate limit exhausted after ${attempts} attempts`); + this.name = 'RateLimitExhaustedError'; + this.attempts = attempts; + if (cause !== undefined) (this as { cause?: unknown }).cause = cause; + } +} + +// --------------------------------------------------------------------------- +// Process-level semaphore for API concurrency +// --------------------------------------------------------------------------- + +/** + * Bounded token bucket. Shared across all runAgentSdkTest calls in this + * process so that bun's --concurrent flag does not compound with in-test + * concurrency to blow past Anthropic's rate limits. + * + * Default capacity 3. Override via GSTACK_SDK_MAX_CONCURRENCY env var. + */ +class Semaphore { + private available: number; + private readonly queue: Array<() => void> = []; + constructor(capacity: number) { + this.available = capacity; + } + async acquire(): Promise { + if (this.available > 0) { + this.available--; + return; + } + await new Promise((resolve) => this.queue.push(resolve)); + } + release(): void { + const next = this.queue.shift(); + if (next) { + next(); + } else { + this.available++; + } + } + /** For tests. Returns tokens currently in-flight. */ + inFlight(): number { + // Not introspectable from outside without tracking; approximate. + return this.queue.length; + } +} + +const DEFAULT_SDK_CONCURRENCY = Number(process.env.GSTACK_SDK_MAX_CONCURRENCY ?? 3); +let _apiSemaphore: Semaphore | null = null; +function getApiSemaphore(): Semaphore { + if (!_apiSemaphore) _apiSemaphore = new Semaphore(DEFAULT_SDK_CONCURRENCY); + return _apiSemaphore; +} + +/** Test-only. Resets the process-level semaphore. */ +export function __resetSemaphoreForTests(capacity: number): void { + _apiSemaphore = new Semaphore(capacity); +} + +// --------------------------------------------------------------------------- +// Rate-limit detection +// --------------------------------------------------------------------------- + +/** True if `err` looks like a rate-limit thrown from the SDK. */ +export function isRateLimitThrown(err: unknown): boolean { + if (!err || typeof err !== 'object') return false; + const msg = (err as { message?: string }).message ?? ''; + const name = (err as { name?: string }).name ?? ''; + const status = (err as { status?: number }).status; + return ( + status === 429 || + /rate.?limit|429|too many requests/i.test(msg) || + /RateLimit/i.test(name) + ); +} + +/** True if a SDKResultMessage is a rate-limit-shaped error. */ +export function isRateLimitResult(msg: SDKMessage): boolean { + if (msg.type !== 'result') return false; + const r = msg as SDKResultMessage; + if (r.subtype === 'success') return false; + // subtype === 'error_during_execution' | 'error_max_turns' | 'error_max_budget_usd' | ... + if (r.subtype !== 'error_during_execution') return false; + const errs = (r as { errors?: string[] }).errors ?? []; + return errs.some((e) => /rate.?limit|429|too many requests/i.test(e)); +} + +/** True if mid-stream SDKRateLimitEvent indicates a blocking rate-limit. */ +export function isRateLimitEvent(msg: SDKMessage): boolean { + if (msg.type !== 'rate_limit_event') return false; + const info = (msg as { rate_limit_info?: { status?: string } }).rate_limit_info; + return info?.status === 'rejected'; +} + +/** + * True if `err` is the SDK's "max turns reached" throw. Some SDK versions + * raise this as an exception from the generator instead of emitting a + * result message with subtype='error_max_turns'. We treat it as terminal- + * but-recoverable: record what we collected and continue, rather than + * failing the whole run. + */ +export function isMaxTurnsError(err: unknown): boolean { + if (!err || typeof err !== 'object') return false; + const msg = (err as { message?: string }).message ?? ''; + return /reached maximum number of turns|max.?turns/i.test(msg); +} + +// --------------------------------------------------------------------------- +// Version resolution (cached) +// --------------------------------------------------------------------------- + +let _sdkVersionCache: string | null = null; +function resolveSdkVersion(): string { + if (_sdkVersionCache) return _sdkVersionCache; + try { + const pkgPath = require.resolve('@anthropic-ai/claude-agent-sdk/package.json'); + const pkg = JSON.parse(fs.readFileSync(pkgPath, 'utf-8')) as { version?: string }; + _sdkVersionCache = pkg.version ?? 'unknown'; + } catch { + _sdkVersionCache = 'unknown'; + } + return _sdkVersionCache; +} + +export function resolveClaudeBinary(): string | null { + try { + return execSync('which claude', { encoding: 'utf-8' }).trim() || null; + } catch { + return null; + } +} + +// --------------------------------------------------------------------------- +// Main runner +// --------------------------------------------------------------------------- + +/** + * Execute a single SDK query with retries. Returns a typed result. + * + * The retry loop treats 429 as recoverable and any other error as fatal. + * Exponential backoff: 1s, 2s, 4s. After maxRetries failures, throws + * RateLimitExhaustedError so the caller can decide what to do with the run. + */ +export async function runAgentSdkTest( + opts: RunAgentSdkOptions, +): Promise { + const sem = getApiSemaphore(); + const maxRetries = opts.maxRetries ?? 3; + const queryImpl: QueryProvider = opts.queryProvider ?? query; + const model = opts.model ?? 'claude-opus-4-7'; + + let attempt = 0; + let lastErr: unknown = null; + + while (attempt <= maxRetries) { + await sem.acquire(); + const startMs = Date.now(); + + // Hoisted so the max-turns catch branch can synthesize a result from + // whatever we captured before the SDK threw. + const events: SDKMessage[] = []; + const assistantTurns: SDKAssistantMessage[] = []; + const toolCalls: Array<{ tool: string; input: unknown; output: string }> = []; + const assistantTextParts: string[] = []; + let firstResponseMs = 0; + let lastEventMs = startMs; + let maxInterTurnMs = 0; + let systemInitVersion = 'unknown'; + let rateLimited: unknown = null; + let terminalResult: SDKResultMessage | null = null; + + try { + const sdkOpts: Options = { + model, + cwd: opts.workingDirectory, + maxTurns: opts.maxTurns ?? 5, + tools: opts.allowedTools ?? ['Read', 'Glob', 'Grep', 'Bash'], + disallowedTools: opts.disallowedTools, + allowedTools: opts.allowedTools ?? ['Read', 'Glob', 'Grep', 'Bash'], + permissionMode: opts.permissionMode ?? 'bypassPermissions', + allowDangerouslySkipPermissions: + (opts.permissionMode ?? 'bypassPermissions') === 'bypassPermissions', + settingSources: opts.settingSources ?? [], + env: opts.env, + pathToClaudeCodeExecutable: opts.pathToClaudeCodeExecutable, + }; + // Empty bare string means "omit entirely" (SDK runs with no override). + // Any object or non-empty string is passed through. + if (typeof opts.systemPrompt === 'object' || opts.systemPrompt !== '') { + sdkOpts.systemPrompt = opts.systemPrompt; + } + + const q = queryImpl({ + prompt: opts.userPrompt, + options: sdkOpts, + }); + + for await (const ev of q) { + const now = Date.now(); + if (firstResponseMs === 0) firstResponseMs = now - startMs; + const interTurn = now - lastEventMs; + if (interTurn > maxInterTurnMs) maxInterTurnMs = interTurn; + lastEventMs = now; + + events.push(ev); + + if (ev.type === 'system' && (ev as SDKSystemMessage).subtype === 'init') { + systemInitVersion = + (ev as SDKSystemMessage).claude_code_version ?? 'unknown'; + } else if (ev.type === 'assistant') { + const am = ev as SDKAssistantMessage; + assistantTurns.push(am); + const content = am.message?.content; + if (Array.isArray(content)) { + for (const block of content as Array< + | { type: 'text'; text?: string } + | { type: 'tool_use'; name?: string; input?: unknown } + | { type: string } + >) { + if (block.type === 'text') { + const t = (block as { text?: string }).text; + if (t) assistantTextParts.push(t); + } else if (block.type === 'tool_use') { + const tb = block as { name?: string; input?: unknown }; + toolCalls.push({ + tool: tb.name ?? 'unknown', + input: tb.input ?? {}, + output: '', + }); + } + } + } + } else if (isRateLimitEvent(ev)) { + rateLimited = new Error( + `mid-stream rate limit: ${JSON.stringify( + (ev as { rate_limit_info?: unknown }).rate_limit_info, + )}`, + ); + } else if (ev.type === 'result') { + terminalResult = ev as SDKResultMessage; + if (isRateLimitResult(ev)) { + rateLimited = new Error( + `result-message rate limit: ${((ev as { errors?: string[] }).errors ?? []).join('; ')}`, + ); + } + } + } + + if (rateLimited) { + throw rateLimited; + } + if (!terminalResult) { + throw new Error('query stream ended without a result event'); + } + + const durationMs = Date.now() - startMs; + const costUsd = + (terminalResult as { total_cost_usd?: number }).total_cost_usd ?? 0; + const turnsUsed = + (terminalResult as { num_turns?: number }).num_turns ?? + assistantTurns.length; + const exitReason = + (terminalResult as { subtype?: string }).subtype ?? 'unknown'; + + return { + events, + assistantTurns, + toolCalls, + output: assistantTextParts.join('\n'), + exitReason, + turnsUsed, + durationMs, + firstResponseMs, + maxInterTurnMs, + costUsd, + model, + sdkVersion: resolveSdkVersion(), + sdkClaudeCodeVersion: systemInitVersion, + resolvedBinaryPath: opts.pathToClaudeCodeExecutable ?? 'sdk-default', + browseErrors: [], + }; + } catch (err) { + lastErr = err; + + // "Max turns reached" is the SDK's way of saying "this session ran + // out of turns." It's thrown from the generator instead of emitted + // as a result message. Treat as a successful-but-capped trial: the + // assistant turns we collected are real and carry a metric. Record + // them with exitReason='error_max_turns' rather than failing the + // whole run. + if (isMaxTurnsError(err)) { + const durationMs = Date.now() - startMs; + return { + events, + assistantTurns, + toolCalls, + output: assistantTextParts.join('\n'), + exitReason: 'error_max_turns', + turnsUsed: assistantTurns.length, + durationMs, + firstResponseMs, + maxInterTurnMs, + costUsd: 0, // unknown from thrown-error path + model, + sdkVersion: resolveSdkVersion(), + sdkClaudeCodeVersion: systemInitVersion, + resolvedBinaryPath: opts.pathToClaudeCodeExecutable ?? 'sdk-default', + browseErrors: [], + }; + } + + const isRetryable = isRateLimitThrown(err); + if (!isRetryable || attempt >= maxRetries) { + if (isRetryable) { + throw new RateLimitExhaustedError(attempt + 1, err); + } + throw err; + } + attempt++; + // backoff: 1s, 2s, 4s + await new Promise((r) => setTimeout(r, 1000 * Math.pow(2, attempt - 1))); + // Let caller reset workspace since prior attempt may have partially + // mutated files via Bash. + if (opts.onRetry) { + opts.onRetry(opts.workingDirectory); + } + } finally { + sem.release(); + } + } + + throw new RateLimitExhaustedError(attempt + 1, lastErr); +} + +// --------------------------------------------------------------------------- +// Legacy shape mapper +// --------------------------------------------------------------------------- + +/** + * Adapt AgentSdkResult to the legacy SkillTestResult shape so helpers that + * expect the old `claude -p` output (extractToolSummary, etc) work unchanged. + */ +export function toSkillTestResult(r: AgentSdkResult): SkillTestResult { + // Cost estimate: use SDK's authoritative cost; back-compute chars. + // session-runner.ts:30 requires inputChars/outputChars/estimatedTokens. + // These are rough; real consumers of CostEstimate use cost + turns. + const outputChars = r.output.length; + const inputChars = 0; // unknown from SDK path; not used for pass/fail + const estimatedTokens = Math.round((inputChars + outputChars) / 4); + + // Build a flat transcript list mimicking the NDJSON shape: + // parseNDJSON emits [{ type: 'assistant', message: {...} }, ...]. + // Use the SDK's assistantTurns directly since their shape matches. + const transcript: unknown[] = r.events.slice(); + + return { + toolCalls: r.toolCalls, + browseErrors: r.browseErrors, + exitReason: r.exitReason, + duration: r.durationMs, + output: r.output, + costEstimate: { + inputChars, + outputChars, + estimatedTokens, + estimatedCost: r.costUsd, + turnsUsed: r.turnsUsed, + }, + transcript, + model: r.model, + firstResponseMs: r.firstResponseMs, + maxInterTurnMs: r.maxInterTurnMs, + }; +} + +// --------------------------------------------------------------------------- +// Metric helpers (re-exported for fixtures) +// --------------------------------------------------------------------------- + +/** + * Count `tool_use` blocks in the first assistant turn of an SDK result. + * Returns 0 if there is no first turn or no content array. + * + * This is the core "fanout" metric. A turn with N tool_use blocks = N + * parallel tool invocations. + */ +export function firstTurnParallelism(firstTurn: SDKAssistantMessage | undefined): number { + if (!firstTurn) return 0; + const content = firstTurn.message?.content; + if (!Array.isArray(content)) return 0; + return (content as Array<{ type: string }>).filter((b) => b.type === 'tool_use').length; +} diff --git a/test/helpers/touchfiles.ts b/test/helpers/touchfiles.ts index 5bf2dcb7..4872f5de 100644 --- a/test/helpers/touchfiles.ts +++ b/test/helpers/touchfiles.ts @@ -239,6 +239,24 @@ export const E2E_TOUCHFILES: Record = { ['model-overlays/claude.md', 'model-overlays/opus-4-7.md', 'scripts/models.ts', 'scripts/resolvers/model-overlay.ts'], 'fanout-arm-overlay-off': ['model-overlays/claude.md', 'model-overlays/opus-4-7.md', 'scripts/models.ts', 'scripts/resolvers/model-overlay.ts'], + + // Overlay efficacy harness (SDK) — measures whether overlay nudges change + // behavior under @anthropic-ai/claude-agent-sdk (closer to real Claude Code + // than `claude -p`). testNames in the file are template literals so the + // completeness scanner doesn't require them; these entries exist for + // diff-based selection accuracy. + 'overlay-harness-opus-4-7-fanout-toy': [ + 'model-overlays/**', + 'test/fixtures/overlay-nudges.ts', + 'test/helpers/agent-sdk-runner.ts', + 'scripts/resolvers/model-overlay.ts', + ], + 'overlay-harness-opus-4-7-fanout-realistic': [ + 'model-overlays/**', + 'test/fixtures/overlay-nudges.ts', + 'test/helpers/agent-sdk-runner.ts', + 'scripts/resolvers/model-overlay.ts', + ], }; /** @@ -430,6 +448,10 @@ export const E2E_TIERS: Record = { // Opus 4.7 overlay evals — periodic (non-deterministic LLM behavior + Opus cost) 'fanout-arm-overlay-on': 'periodic', 'fanout-arm-overlay-off': 'periodic', + + // Overlay efficacy harness (SDK, paid) — periodic only + 'overlay-harness-opus-4-7-fanout-toy': 'periodic', + 'overlay-harness-opus-4-7-fanout-realistic': 'periodic', }; /** diff --git a/test/skill-e2e-overlay-harness.test.ts b/test/skill-e2e-overlay-harness.test.ts new file mode 100644 index 00000000..c00a27f6 --- /dev/null +++ b/test/skill-e2e-overlay-harness.test.ts @@ -0,0 +1,320 @@ +/** + * Overlay-efficacy harness (periodic tier, paid). + * + * Measures whether a model-specific overlay nudge actually changes model + * behavior when run through the real Claude Agent SDK — the harness + * Claude Code itself is built on. This complements test/skill-e2e-opus-47.test.ts + * which measures the same thing via `claude -p` subprocess (a different + * harness with different prompt composition). + * + * For each fixture in test/fixtures/overlay-nudges.ts, runs two arms at + * `fixture.trials` trials per arm with bounded concurrency: + * - overlay-on: SDK systemPrompt = resolved overlay content + * - overlay-off: SDK systemPrompt = "" (empty) + * + * Both arms have no CLAUDE.md, no skills directory, no setting-source + * inheritance (settingSources: []). This is the TRUE bare comparison — + * the only variable is the overlay text. + * + * Budget ~$20 per run at 40 trials (2 fixtures × 2 arms × 10 trials). + * Gated by EVALS=1 AND EVALS_TIER=periodic. Never runs under test:gate. + */ + +import { describe, test, expect, afterAll } from 'bun:test'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { + runAgentSdkTest, + resolveClaudeBinary, + type AgentSdkResult, + type SystemPromptOption, +} from './helpers/agent-sdk-runner'; +import { EvalCollector, getProjectEvalDir } from './helpers/eval-store'; +import { + OVERLAY_FIXTURES, + type OverlayFixture, +} from './fixtures/overlay-nudges'; +import { readOverlay } from '../scripts/resolvers/model-overlay'; + +const evalsEnabled = !!process.env.EVALS; +const periodicTier = process.env.EVALS_TIER === 'periodic'; +const shouldRun = evalsEnabled && periodicTier; + +const describeE2E = shouldRun ? describe : describe.skip; +// EvalCollector's tier must be 'e2e' | 'llm-judge' per its type signature. +// The existing paid evals violate this by passing descriptive names like +// 'e2e-opus-47' — a pre-existing pattern that only works because bun-test +// runs without strict typechecking. We stay conforming here. +const evalCollector = shouldRun ? new EvalCollector('e2e') : null; + +const REPO_ROOT = path.resolve(import.meta.dir, '..'); +const runId = new Date() + .toISOString() + .replace(/[:.]/g, '') + .replace('T', '-') + .slice(0, 15); +const TRANSCRIPTS_DIR = path.join( + path.dirname(getProjectEvalDir()), + 'transcripts', + `overlay-harness-${runId}`, +); + +// --------------------------------------------------------------------------- +// Per-arm helpers +// --------------------------------------------------------------------------- + +type Arm = 'overlay-on' | 'overlay-off'; + +function mkTrialDir(fixtureId: string, arm: Arm, n: number): string { + const dir = fs.mkdtempSync( + path.join(os.tmpdir(), `overlay-harness-${fixtureId}-${arm}-${n}-`), + ); + return dir; +} + +function saveRawTranscript( + fixtureId: string, + arm: Arm, + n: number, + result: AgentSdkResult, +): void { + fs.mkdirSync(TRANSCRIPTS_DIR, { recursive: true }); + const out = path.join(TRANSCRIPTS_DIR, `${fixtureId}-${arm}-${n}.jsonl`); + const lines = result.events.map((e) => JSON.stringify(e)); + fs.writeFileSync(out, lines.join('\n') + '\n'); +} + +function overlayContentFor(fixture: OverlayFixture): string { + const family = path.basename(fixture.overlayPath, '.md'); + const resolved = readOverlay(family); + if (!resolved) { + throw new Error( + `fixture ${fixture.id}: resolver returned empty content for ${family}`, + ); + } + return resolved; +} + +// --------------------------------------------------------------------------- +// Per-fixture runner +// --------------------------------------------------------------------------- + +interface ArmResult { + metrics: number[]; + costs: number[]; + durations: number[]; + rateLimitExhausted: number; + sdkClaudeCodeVersions: Set; +} + +async function runArm( + fixture: OverlayFixture, + arm: Arm, + systemPrompt: SystemPromptOption, + claudeBinary: string | null, +): Promise { + const result: ArmResult = { + metrics: [], + costs: [], + durations: [], + rateLimitExhausted: 0, + sdkClaudeCodeVersions: new Set(), + }; + + const trials = fixture.trials; + const concurrency = fixture.concurrency ?? 3; + + // Simple bounded executor: run trials in chunks of `concurrency`. + // The process-level semaphore in agent-sdk-runner.ts enforces the true cap. + let nextTrial = 0; + const workers = Array.from({ length: concurrency }, async () => { + while (true) { + const n = nextTrial++; + if (n >= trials) return; + + const dir = mkTrialDir(fixture.id, arm, n); + fixture.setupWorkspace(dir); + try { + const sdkResult = await runAgentSdkTest({ + systemPrompt, + userPrompt: fixture.userPrompt, + workingDirectory: dir, + model: fixture.model, + maxTurns: fixture.maxTurns ?? 5, + allowedTools: fixture.allowedTools ?? ['Read', 'Glob', 'Grep', 'Bash'], + permissionMode: 'bypassPermissions', + settingSources: [], + env: { ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY ?? '' }, + pathToClaudeCodeExecutable: claudeBinary ?? undefined, + testName: `${fixture.id}-${arm}-${n}`, + runId, + fixtureId: fixture.id, + onRetry: (_) => { + // Reset the workspace before the retry so partial Bash side effects + // from the failed attempt don't contaminate. + fs.rmSync(dir, { recursive: true, force: true }); + fs.mkdirSync(dir, { recursive: true }); + fixture.setupWorkspace(dir); + }, + }); + + saveRawTranscript(fixture.id, arm, n, sdkResult); + + const metric = fixture.metric(sdkResult); + result.metrics.push(metric); + result.costs.push(sdkResult.costUsd); + result.durations.push(sdkResult.durationMs); + result.sdkClaudeCodeVersions.add(sdkResult.sdkClaudeCodeVersion); + + evalCollector?.addTest({ + name: `${fixture.id}-${arm}-${n}`, + suite: 'overlay-harness', + tier: 'e2e', + passed: true, + duration_ms: sdkResult.durationMs, + cost_usd: sdkResult.costUsd, + transcript: sdkResult.events, + prompt: fixture.userPrompt, + output: sdkResult.output, + turns_used: sdkResult.turnsUsed, + browse_errors: sdkResult.browseErrors, + exit_reason: sdkResult.exitReason, + model: sdkResult.model, + first_response_ms: sdkResult.firstResponseMs, + max_inter_turn_ms: sdkResult.maxInterTurnMs, + }); + } catch (err) { + if (err instanceof Error && err.name === 'RateLimitExhaustedError') { + result.rateLimitExhausted++; + // Record a failed trial so the collector captures the attempt. + evalCollector?.addTest({ + name: `${fixture.id}-${arm}-${n}`, + suite: 'overlay-harness', + tier: 'e2e', + passed: false, + duration_ms: 0, + cost_usd: 0, + exit_reason: 'rate_limit_exhausted', + error: err.message, + }); + } else { + throw err; + } + } finally { + try { + fs.rmSync(dir, { recursive: true, force: true }); + } catch { + // best-effort cleanup + } + } + } + }); + + await Promise.all(workers); + return result; +} + +function mean(xs: number[]): number { + if (xs.length === 0) return 0; + return xs.reduce((a, b) => a + b, 0) / xs.length; +} + +function sum(xs: number[]): number { + return xs.reduce((a, b) => a + b, 0); +} + +// --------------------------------------------------------------------------- +// Test bodies +// --------------------------------------------------------------------------- + +describeE2E('overlay efficacy harness (SDK)', () => { + // Resolve binary once + const claudeBinary = resolveClaudeBinary(); + + if (!claudeBinary) { + test.skip( + 'no local `claude` binary on PATH — cannot pin for harness parity', + () => {}, + ); + return; + } + + for (const fixture of OVERLAY_FIXTURES) { + test( + `${fixture.id}: overlay-ON vs overlay-OFF, N=${fixture.trials} per arm`, + async () => { + const overlayText = overlayContentFor(fixture); + expect(overlayText.length).toBeGreaterThan(100); + + // Arm composition: both arms use the real Claude Code default system + // prompt (preset). Overlay-ON APPENDS the overlay text; overlay-OFF + // uses the default alone. This measures the overlay's marginal effect + // ON TOP of Claude Code's normal behavioral scaffolding — which is + // the only measurement that matches how real Claude Code composes + // overlays into its system prompt stack. + const [onArm, offArm] = await Promise.all([ + runArm( + fixture, + 'overlay-on', + { type: 'preset', preset: 'claude_code', append: overlayText }, + claudeBinary, + ), + runArm( + fixture, + 'overlay-off', + { type: 'preset', preset: 'claude_code' }, + claudeBinary, + ), + ]); + + const arms = { + overlay: onArm.metrics, + off: offArm.metrics, + }; + + const meanOn = mean(arms.overlay); + const meanOff = mean(arms.off); + const lift = meanOn - meanOff; + const floorHits = arms.overlay.filter((n) => n >= 2).length; + const totalCost = sum(onArm.costs) + sum(offArm.costs); + const versionSet = new Set([ + ...onArm.sdkClaudeCodeVersions, + ...offArm.sdkClaudeCodeVersions, + ]); + + // Loud output for the next person reading the eval JSON: + // eslint-disable-next-line no-console + console.log( + `\n[${fixture.id}]\n` + + ` binary: ${claudeBinary}\n` + + ` claude_code_version(s): ${[...versionSet].join(', ')}\n` + + ` overlay-ON metrics: [${arms.overlay.join(', ')}] mean=${meanOn.toFixed(2)}\n` + + ` overlay-OFF metrics: [${arms.off.join(', ')}] mean=${meanOff.toFixed(2)}\n` + + ` lift: ${lift.toFixed(2)} floor_hits(>=2): ${floorHits}/${fixture.trials}\n` + + ` rate_limit_exhausted: on=${onArm.rateLimitExhausted} off=${offArm.rateLimitExhausted}\n` + + ` total_cost_usd: $${totalCost.toFixed(4)}\n` + + ` transcripts: ${TRANSCRIPTS_DIR}`, + ); + + // Demand enough trials actually completed to make the assertion + // meaningful. If rate-limit exhaustion took out more than half of an + // arm, fail loudly rather than pass/fail on a fragment. + const minTrials = Math.ceil(fixture.trials / 2); + expect(arms.overlay.length).toBeGreaterThanOrEqual(minTrials); + expect(arms.off.length).toBeGreaterThanOrEqual(minTrials); + + expect(fixture.pass(arms)).toBe(true); + }, + 30 * 60 * 1000, // 30 minute timeout per fixture + ); + } +}); + +afterAll(async () => { + if (evalCollector) { + const filepath = await evalCollector.finalize(); + // eslint-disable-next-line no-console + console.log(`\n[overlay-harness] eval results: ${filepath}`); + } +});