diff --git a/CHANGELOG.md b/CHANGELOG.md index 467ee5b54..a4e54f3d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,16 +2,17 @@ ## [Unreleased] -**The train gate's paid report now prices the trials it used to call "cost unknown".** 32 of 139 gate trials had no cost on every gate since Oct 26, so the report's total understated a gate by about $7.50 (GSTA-150 audit). Three of the four causes are fixed here; PTY sessions follow in the next PR. +**The train gate's paid report now prices the trials it used to call "cost unknown".** 32 of 139 gate trials had no cost on every gate since Oct 26, so the report's total understated a gate by about $7.50 (GSTA-150 audit). All four causes are fixed. ### Fixed - **Label-named records in multi-case shards** (`test/helpers/e2e-helpers.ts`, `scripts/lib/paid-report.ts`): a `claude -p` record named by its display label ("browse basic commands") carries `cost_case_id` from `runSkillTest`'s registered test name, and the report binds its cost to that case. The failed record, failure class and model still come from the records named by case id, so no verdict, class or pass-rate series changes (22 trials: bws, session-intelligence, workflow, skillify, review-attribution, review, gate-integrity, diagram). - **Aborted Agent SDK sessions** (`test/helpers/agent-sdk-runner.ts`, `lib/pricing.ts`): a session that ends without a result event logs its streamed usage as `streamed_cost_usd` in the session ledger, priced once per message id at its largest output, subagent turns included, with 5-minute cache writes at 1.25x and 1-hour writes at 2x input. An unpriced model leaves the cost unknown (auq-format-gate). - **Model-free cases**: `timeline-event-flow` runs no model, so with no record and no session it is a known $0. A test checks every listed case's body starts no model session. +- **PTY sessions** (`test/helpers/pty/launch.ts`, `test/helpers/pty/session.ts`, `test/helpers/pty/transcript-cost.ts`): each PTY ledger row prices the session's transcript (`projects/*/.jsonl` plus its `subagents/` files) as `streamed_cost_usd`. The session is the caller's `--session-id`, else the one transcript that appeared during the launch for its cwd, so the launcher adds no argument and concurrent tests sharing a config dir never price each other. No transcript, two new transcripts for one cwd, no usage or an unpriced model leaves the cost unknown, never $0, and a read error never reaches the test. The native AskUserQuestion capture, which never reads model transcripts, opts out (`priceTranscript: false`). This covers 8 cases and 14 sessions per gate: plan-mode-no-op, plan-design-with-ui-scope, the plan-ceo and plan-devex finding-floor and plan-mode cases, autoplan-guard-pty and office-hours-auto-mode. Checked live, a `claude -p` session with one subagent priced $0.090376 from its transcripts, equal to its `total_cost_usd`. - **The headline total** adds session cost that no collector record carries, so newly priced trials reach the gate's `cost $X` line exactly once. ### Tests -- `test/paid-report-cost.test.ts`: pricing arithmetic, streamed dedupe, `caseCost`, `recordE2E`'s `cost_case_id`, a two-case shard whose label-named records bind by cost without changing the failed record, class or model. `test/paid-report-census.test.ts`: a record-less trial's streamed cost reaches the trial and the verdict total once. +- `test/paid-report-cost.test.ts`: pricing arithmetic, streamed dedupe, `caseCost`, `recordE2E`'s `cost_case_id`, a two-case shard whose label-named records bind by cost without changing the failed record, class or model. `test/paid-report-census.test.ts`: a record-less trial's streamed cost reaches the trial and the verdict total once. `test/paid-report-cost.test.ts` also prices a PTY session, its inline sidechains and subagent files once per message id, leaves a foreign session, a missing or unreadable transcript or an unpriced model unknown, and claims only the one new transcript for a launch's cwd. `test/helpers/claude-pty-runner.launch.unit.test.ts`: a real launch prices the transcript it created without adding an argument, and an opted-out launch prices nothing. ## [1.92.66.0] - 2026-11-02 diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index d107a4d7b..9a4053272 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -656,9 +656,10 @@ no price row). A rule case's trial in the JUnit census (`caseCost` in carries: each ledger session without a terminal result (`billed: false`) adds its `streamed_cost_usd`, and a trial with no record adds each billed session's own `cost_usd` (Codex). The trial is cost-unknown when any record says -`cost_known: false`, when any of those sessions has no priced cost (every PTY -session, a `claude -p` session that ended without a result, an aborted Agent -SDK session with no usage or an unpriced model), or when it has neither records +`cost_known: false`, when any of those sessions has no priced cost (a PTY +session whose transcript does not price, a `claude -p` session that ended +without a result, an aborted Agent SDK session with no usage or an unpriced +model), or when it has neither records nor sessions. A case in `MODEL_FREE_CASES` (`timeline-event-flow`, which runs no model) with neither is a known $0. A panel trial (`behavior`, `judge`) from an isolated trial shard keeps the older rule (`trialCostKnown` in @@ -702,8 +703,20 @@ by `streamedUsageCostUsd` (`lib/pricing.ts`), each message id once at its largest output, subagent turns included, `` messages skipped, 5-minute cache writes at 1.25x and 1-hour writes at 2x the input rate. The field is absent when no message carries usage or any message's model has no -price row, so an unpriced session stays cost-unknown, never $0. The ledger -rides in each slice artifact beside the shard results. +price row, so an unpriced session stays cost-unknown, never $0. A PTY session +records `streamed_cost_usd` from its transcript the same way. Both PTY ledger +writers price `/projects/*/.jsonl` plus +`/subagents/*.jsonl` (`test/helpers/pty/transcript-cost.ts`), where the +id is the caller's `--session-id`, else the one transcript that appeared +during this launch for its cwd. The launcher adds no argument. Pricing never +takes the whole directory, because concurrent PTY tests in one process share +one hermetic config dir; two new transcripts for one cwd are ambiguous and stay +unknown. A non-hermetic or resumed launch, a caller that opts out +(`priceTranscript: false`, the native AskUserQuestion capture, which never +reads model transcripts), a missing transcript, no usage, an unpriced model or +a read error leaves the field absent (cost-unknown), and a pricing failure +never fails the test. The row keeps `billed: false`. The ledger rides in each slice +artifact beside the shard results. **Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate below 95% over at least 10 post-policy trials of the case's current input @@ -1258,9 +1271,11 @@ observation never removes a command and a missing or empty artifact never erases one; removing a command is a reviewed edit. While the map is in shadow mode it changes no selection, so a PR that changes only the map selects no paid case. -Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is -unknown: about 39% of observed case runs in the committed map. The shadow block -and `test:health` both say so. +Dollar figures are lower bounds. Known cost excludes trials whose cost is +unknown: about 39% of observed case runs in the committed map, mostly PTY +trials recorded before PTY sessions were priced from their transcripts +([Failure causes and details](#failure-causes)). The shadow block and +`test:health` both say so. - **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate cases the map would select, would drop and would add, a provenance line per diff --git a/docs/evals/census-red.md b/docs/evals/census-red.md index cc8d7b273..e3bb69991 100644 --- a/docs/evals/census-red.md +++ b/docs/evals/census-red.md @@ -93,7 +93,7 @@ What incomplete data means: | headroom `censored` | a sample timed out, so its real duration is at least the budget | | cause `unrecorded` | the trial was written before `failure_cause` existed | | cause `unknown` | the trial failed with no evidence for any other cause | -| `cost unknown` | a session's cost was not captured (a PTY session, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown | +| `cost unknown` | a session's cost was not captured (a PTY session without a priced transcript, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown | | `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed | ## 4. Replay and repair (free) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index f1329f4f2..b494a168a 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2521,11 +2521,14 @@ case with `bun run eval:pass-rates --headroom --case `. ### `cost unknown (no billing captured for any of trial(s))` / `cost $ known + of trial(s) cost unknown` **Meaning.** Some trials' cost was not captured, so only the known sum is -shown. A trial is cost-unknown when it ran a PTY session (PTY sessions record -no billing), a Codex session that completed no turn or whose model has no price -row, a `claude -p` session that ended without a result, or an aborted Agent -SDK session whose streamed usage could not be priced (no usage, or a model with -no price row). An aborted Agent SDK session with priced usage counts its +shown. A trial is cost-unknown when it ran a PTY session whose transcript +could not be priced (a non-hermetic or resumed launch, the native +AskUserQuestion capture, two new transcripts for one cwd, no transcript or +usage, or a model with no price row), a +Codex session that completed no turn or whose model has no price row, a +`claude -p` session that ended without a result, or an aborted Agent SDK +session whose streamed usage could not be priced (no usage, or a model with no +price row). A PTY or aborted Agent SDK session that does price counts its `streamed_cost_usd` instead. A trial is also unknown when no record and no session can be attributed to it, for example a multi-case shard's display-label record without `cost_case_id` (a one-case shard's records are diff --git a/scripts/harness-version.json b/scripts/harness-version.json index 0f63d8640..af40e697e 100644 --- a/scripts/harness-version.json +++ b/scripts/harness-version.json @@ -38,15 +38,16 @@ "test/helpers/pty/classify.ts": {"blob":"3f2daa12ff6484f96b8d28d910a50231c4bb3ae8","since":6}, "test/helpers/pty/fake-session.ts": {"blob":"554e064ef259b8444280eddfe9b0db93c66a8202","since":6}, "test/helpers/pty/judge.ts": {"blob":"81c5bb4c48ab928aca2f186a1ade9df7169d7fbe","since":6}, - "test/helpers/pty/launch.ts": {"blob":"e4413de380b799659170e7da248d51afda3b0e3f","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"}, + "test/helpers/pty/launch.ts": {"blob":"4f0731ef39bc3a6a7c5c592ce98690a24cb1ed7f","since":6,"nonBehavioral":"PTY transcript pricing finds the session by the caller's --session-id or the one new transcript for the launch cwd; the launcher adds no argument; native AUQ capture opts out; ledger cost fields only, no prompt, argv, answer, timeout or verdict changes"}, "test/helpers/pty/plan-native.ts": {"blob":"ddba9cb68c7c0da935cff7c1cea258dee2dff740","since":6}, "test/helpers/pty/runners/counting.ts": {"blob":"8c5c05e54abf961259d34d9c3d77db67138d7794","since":6}, "test/helpers/pty/runners/floor.ts": {"blob":"a9938cb98f285474ebc4860cd61012504a36466e","since":6,"nonBehavioral":"The PTY floor and observation runners end a plan-mode turn at an idle prompt (the plan-mode footer without its running 'esc to interrupt' segment, after a 10 s settle) instead of waiting out the deadline. The floor runner exits as api_error only after the idle screen with its error panel held still for 10 s and every tick's question and assessment steps found nothing to credit (the transcript is re-read each tick), so a native or prose question already on screen is credited as before, even when its transcript line lands after the screen draws done (test/helpers/claude-pty-runner.runners.unit.test.ts, the prose-question-before-panel case). With no qualifying question an idle turn stays idle, so a trial that failed at the deadline as observer_timeout now fails earlier as api_error. An observation returns what the deadline would have returned. A running footer, extra footer text and a cut footer still wait for the deadline. No pass becomes a fail or a fail a pass. The session ledger may now record failure_cause api_error or refusal instead of observer_timeout or session_timeout for a plan-mode session whose last idle turn shows a panel; failure_class, the gate red class and pass rates are unchanged, so no pass-rate series resets"}, "test/helpers/pty/runners/observation.ts": {"blob":"1c2b75fbb984708725abaa8939125f7cab969f9f","since":6}, "test/helpers/pty/screen.ts": {"blob":"1384f765e48e511d5f843df2660b2ec70bbcb6b2","since":6,"nonBehavioral":"The PTY floor and observation runners end a plan-mode turn at an idle prompt (the plan-mode footer without its running 'esc to interrupt' segment, after a 10 s settle) instead of waiting out the deadline. The floor runner exits as api_error only after the idle screen with its error panel held still for 10 s and every tick's question and assessment steps found nothing to credit (the transcript is re-read each tick), so a native or prose question already on screen is credited as before, even when its transcript line lands after the screen draws done (test/helpers/claude-pty-runner.runners.unit.test.ts, the prose-question-before-panel case). With no qualifying question an idle turn stays idle, so a trial that failed at the deadline as observer_timeout now fails earlier as api_error. An observation returns what the deadline would have returned. A running footer, extra footer text and a cut footer still wait for the deadline. No pass becomes a fail or a fail a pass. The session ledger may now record failure_cause api_error or refusal instead of observer_timeout or session_timeout for a plan-mode session whose last idle turn shows a panel; failure_class, the gate red class and pass rates are unchanged, so no pass-rate series resets"}, - "test/helpers/pty/session.ts": {"blob":"54fe04b61d953e4e55e2fc251532c04a49fab66f","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"}, + "test/helpers/pty/session.ts": {"blob":"8fd1749a62f24dc37ae186c8a21c2165f7f7645e","since":6,"nonBehavioral":"PTY sessions price their ledger row (streamed_cost_usd) from the session transcript found by the caller's --session-id or the one new transcript for the launch cwd; ledger cost fields only, no prompt, argv, answer, timeout or verdict changes"}, + "test/helpers/pty/transcript-cost.ts": {"blob":"08f7ccbdbd9d71a5dc86edd2f8fced423a78ad9a","since":6,"nonBehavioral":"PTY transcript pricing finds the session by the caller's --session-id or the one new transcript for the launch cwd; the launcher adds no argument; native AUQ capture opts out; ledger cost fields only, no prompt, argv, answer, timeout or verdict changes"}, "test/helpers/session-drain-policy.ts": {"blob":"6211239dd0485e6495a700b824a7e0e61d1bcd97","since":6}, - "test/helpers/session-ledger.ts": {"blob":"5d8b796ccd422f4e1e2ab395bf6ce5a2e4c0e167","since":6,"nonBehavioral":"Cost attribution only (GSTA-150 PR 2): a display-label record carries cost_case_id from runSkillTest's testName and the report binds its cost by it while the failed record and failure class still come from the named records (test/paid-report-cost.test.ts pins both); an aborted Agent SDK session logs its streamed usage as streamed_cost_usd, used only when a case has no record; a listed model-free case with no record or session is a known $0. No case's execution, outcome, failure class, rerun eligibility or panel verdict changes, so no pass-rate series resets"}, + "test/helpers/session-ledger.ts": {"blob":"c99baaaa0a77a5a72384493fd8a85a4253ecb05d","since":6,"nonBehavioral":"PTY sessions price their ledger row (streamed_cost_usd) from the session transcript found by the caller's --session-id or the one new transcript for the launch cwd; ledger cost fields only, no prompt, argv, answer, timeout or verdict changes"}, "test/helpers/session-runner.ts": {"blob":"c73b2596d9eb5377530a6bf1d284e0b6ea1556a0","since":6,"nonBehavioral":"Cost attribution only (GSTA-150 PR 2): a display-label record carries cost_case_id from runSkillTest's testName and the report binds its cost by it while the failed record and failure class still come from the named records (test/paid-report-cost.test.ts pins both); an aborted Agent SDK session logs its streamed usage as streamed_cost_usd, used only when a case has no record; a listed model-free case with no record or session is a known $0. No case's execution, outcome, failure class, rerun eligibility or panel verdict changes, so no pass-rate series resets"}, "test/helpers/skill-fixture.ts": {"blob":"8a654dae6b0a396d839b02179119ae3e3bb13f69","since":6,"nonBehavioral":"extractSkillBody and extractSkillSections follow the shared-preamble hoist (pointer plus sections/preamble.md); every fixture they build is byte-identical to the pre-hoist output (compared for 8 body and 4 section extractions), so no case's input or verdict changes; the /ship review then reworded only comments and two error messages, so no fixture byte changes"}, "test/helpers/skill-startup.ts": {"blob":"a8d21e3817c3f35a1d06af325c92fde7d420e377","since":6,"nonBehavioral":"readSkillEffective also appends a variant's added ### subsections (office-hours' Open-ended questions); readSkillStartup, readSharedSections and readSkillEffective return bytes identical to the pinned helper for every tier 2+ render without added subsections (735 across all hosts) and for office-hours' SKILL.md before this PR, so office-hours' fixture input changes only because its prompt changed in this PR, not because the helper did"}, diff --git a/test/helpers/auq-native-capture.ts b/test/helpers/auq-native-capture.ts index ba70bb8c4..27b1cee56 100644 --- a/test/helpers/auq-native-capture.ts +++ b/test/helpers/auq-native-capture.ts @@ -313,7 +313,7 @@ Skip any system-audit / environment-setup / codebase-exploration steps. At the f session = await launchClaudePty({ // Explicit tool approvals suffice. Bypass mode introduces a separate // safety-consent screen in a fresh CLI config; never accept that screen. - cwd, model, seedSkills: false, permissionMode: 'default', + cwd, model, seedSkills: false, permissionMode: 'default', priceTranscript: false, observeScreen: true, observeSetupQuestions: true, timeoutMs: Math.max(1, deadline - Date.now()), // Explicit availability as well as approval: no Bash/Agent/Skill or MCP. diff --git a/test/helpers/claude-pty-runner.launch.unit.test.ts b/test/helpers/claude-pty-runner.launch.unit.test.ts index 0f6123afe..5abbb8e05 100644 --- a/test/helpers/claude-pty-runner.launch.unit.test.ts +++ b/test/helpers/claude-pty-runner.launch.unit.test.ts @@ -76,3 +76,43 @@ process.stdout.write(JSON.stringify({ capture: resolveEvalModel('capture'), } finally { rmSync(dir, { recursive: true, force: true }); } }, 60_000); }); + +describe('launchClaudePty transcript pricing (GSTA-150 PR 3)', () => { + test.skipIf(process.platform === 'win32')('prices the transcript this launch created for its cwd, without adding a launch argument; opting out prices nothing', () => { + const dir = mkdtempSync(join(tmpdir(), 'pty-transcript-cost-')); + try { + const fake = join(dir, 'fake-claude'); + writeFileSync(fake, `#!${process.execPath} +const fs = require('node:fs'), path = require('node:path'), crypto = require('node:crypto'); +const project = path.join(process.env.CLAUDE_CONFIG_DIR, 'projects', 'fake-project'); +fs.mkdirSync(project, { recursive: true }); +fs.writeFileSync(path.join(project, crypto.randomUUID() + '.jsonl'), JSON.stringify({ type: 'assistant', cwd: process.cwd(), + message: { id: 'msg_' + crypto.randomUUID(), model: 'claude-opus-5-5', usage: { input_tokens: 0, output_tokens: 1000 } } }) + '\\n'); +process.stdout.write('ARGV=' + JSON.stringify(process.argv.slice(2)) + '\\n'); +setInterval(() => {}, 1000); +`, { mode: 0o755 }); + const runner = pathToFileURL(join(import.meta.dir, 'claude-pty-runner.ts')).href; + const worker = join(dir, 'worker.ts'); + writeFileSync(worker, `import { launchClaudePty } from ${JSON.stringify(runner)}; +import * as fs from 'node:fs'; +const run = async (opts) => { + const cwd = fs.mkdtempSync(${JSON.stringify(join(dir, 'cwd-'))}); + const session = await launchClaudePty({ cwd, timeoutMs: 5000, ...opts }); + try { + await session.waitFor(/ARGV=\\[.*\\]/, { timeoutMs: 4000, pollMs: 20 }); + return { argv: JSON.parse(/ARGV=(\\[.*\\])/.exec(session.visibleText())[1]), cost: session.transcriptCostUsd?.() ?? null }; + } finally { await session.close(); } +}; +const [priced, optedOut] = await Promise.all([run({}), run({ priceTranscript: false })]); +process.stdout.write(JSON.stringify({ priced, optedOut })); +`); + const result = spawnSync(process.execPath, [worker], { cwd: join(import.meta.dir, '../..'), encoding: 'utf8', timeout: 30_000, + env: { ...process.env, BROWSE_TERMINAL_BINARY: fake, EVALS_HERMETIC: '1' } }); + expect(result.status, result.stderr).toBe(0); + const { priced, optedOut } = JSON.parse(result.stdout.trim().split('\n').at(-1)!); + expect(priced.cost).toBeCloseTo(0.02, 6); + expect(optedOut.cost).toBeNull(); + expect(priced.argv).not.toContain('--session-id'); + } finally { rmSync(dir, { recursive: true, force: true }); } + }, 60_000); +}); diff --git a/test/helpers/pty/launch.ts b/test/helpers/pty/launch.ts index e4413de38..4f0731ef3 100644 --- a/test/helpers/pty/launch.ts +++ b/test/helpers/pty/launch.ts @@ -15,8 +15,12 @@ import { trustDialogInput } from '../pty-trust-dialog'; import { resolveClaudeBinary } from './binary'; import { createPtyScreen, screenPanelEnd, stripAnsi } from './screen'; import { appendSessionLedger, sessionKey } from '../session-ledger'; +import { newTranscriptSessionId, pricedTranscript, ptyTranscriptCostUsd, transcriptFiles } from './transcript-cost'; export interface ClaudePtyOptions { + /** Price this session's ledger row from the transcript Claude writes under the hermetic config dir (default true). + * false for a caller that must never read model transcripts, such as the native AskUserQuestion capture. */ + priceTranscript?: boolean; /** Register the repo's shipped skills in the child's user scope via * hermeticSkillsConfigDir(). Required by any test that types a /skill * slash command; without it hermetic claude rejects the command as @@ -129,6 +133,9 @@ export interface ClaudePtySession { * the dir name ends in `/.claude` by contract). */ hermeticConfigDir: string | null; + /** This session's cost from its transcript (the caller's --session-id, else the one transcript this launch created + * for its cwd); undefined when unpriced, ambiguous or opted out. Never throws. */ + transcriptCostUsd?(): number | undefined; /** Owned HOME/.gstack created by the seeded launcher; absent for caller overrides. */ hermeticSkillStateRoot?: string; /** Owned pre-tool identity record, removed by close(). */ @@ -249,7 +256,8 @@ export async function launchClaudePty( recorded = true; const elapsed = performance.now() - startedMono; appendSessionLedger({ key: sessionKey('pty', undefined), runner: 'pty', started_at: startedAt, budget_ms: timeoutMs, - elapsed_ms: elapsed, end: elapsed >= timeoutMs ? 'session_timeout' : 'completed', ...screenPanelEnd(handle.visibleText()), billed: false }); + elapsed_ms: elapsed, end: elapsed >= timeoutMs ? 'session_timeout' : 'completed', ...screenPanelEnd(handle.visibleText()), billed: false, + ...pricedTranscript(handle) }); } return close(); } }); @@ -263,6 +271,8 @@ interface LaunchSetup { hermeticSkillStateRoot: string | undefined; autoplanArtifactStateRoot: string | undefined; autoplanEngTestPlanStateRoot: string | undefined; + /** What prices the transcript: the caller's session id, else the transcripts present before spawn. */ + transcript: { sessionId: string | undefined; before: Set; cwd: string } | undefined; } function prepareLaunch(opts: ClaudePtyOptions): LaunchSetup { @@ -319,7 +329,13 @@ function prepareLaunch(opts: ClaudePtyOptions): LaunchSetup { const autoplanEngTestPlanStateRoot = opts.approveAutoplanArtifactEdits && opts.autoplanArtifactState !== undefined ? hermeticSkillStateRoot : undefined; - return { args, childEnv, hermetic, hermeticSkillStateRoot, autoplanArtifactStateRoot, autoplanEngTestPlanStateRoot }; + const extra = opts.extraArgs ?? []; + const pinned = extra.findIndex(arg => /^--session-id(=|$)/.test(arg)); + const sessionId = pinned < 0 ? undefined : extra[pinned].startsWith('--session-id=') ? extra[pinned].slice('--session-id='.length) : extra[pinned + 1]; + const resumes = extra.some(arg => /^(--resume|--continue|-r|-c)(=|$)/.test(arg)); + const transcript = hermetic && childEnv.CLAUDE_CONFIG_DIR && opts.priceTranscript !== false && !resumes + ? { sessionId, before: transcriptFiles(childEnv.CLAUDE_CONFIG_DIR), cwd: opts.cwd ?? process.cwd() } : undefined; + return { args, childEnv, hermetic, hermeticSkillStateRoot, autoplanArtifactStateRoot, autoplanEngTestPlanStateRoot, transcript }; } interface LaunchRecorders { @@ -568,6 +584,11 @@ function sessionHandle(pty: PtyProcess, launch: LaunchSetup, wallTimer: ReturnTy exited: () => pty.exited, exitCode: () => pty.exitCode, hermeticConfigDir: launch.hermetic ? launch.childEnv.CLAUDE_CONFIG_DIR ?? null : null, + transcriptCostUsd: () => { + const { transcript, childEnv } = launch; + return transcript && ptyTranscriptCostUsd(childEnv.CLAUDE_CONFIG_DIR, + transcript.sessionId ?? newTranscriptSessionId(childEnv.CLAUDE_CONFIG_DIR, transcript.before, transcript.cwd)); + }, hermeticSkillStateRoot: launch.hermeticSkillStateRoot, pendingPlanReadyFile: pendingExit?.file, pendingQuestionFile: pendingQuestion?.file, diff --git a/test/helpers/pty/session.ts b/test/helpers/pty/session.ts index 54fe04b61..8fd1749a6 100644 --- a/test/helpers/pty/session.ts +++ b/test/helpers/pty/session.ts @@ -10,6 +10,7 @@ import { launchClaudePty, type ClaudePtyOptions, type ClaudePtySession } from './launch'; import { screenPanelEnd } from './screen'; import { appendSessionLedger, sessionKey, type SessionEnd } from '../session-ledger'; +import { pricedTranscript } from './transcript-cost'; /** * The runners' launch seam: how a session starts, and the clock the runner @@ -97,7 +98,7 @@ export async function runPtySession(plan: PtySessionPlan): Promise { } finally { appendSessionLedger({ key: sessionKey('pty', undefined), runner: 'pty', started_at: startedAt, budget_ms: plan.budgetMs ?? plan.launch.timeoutMs ?? 240_000, elapsed_ms: plan.driver.monotonic() - started, - end, ...safePanelEnd(session), billed: false }); + end, ...safePanelEnd(session), billed: false, ...pricedTranscript(session) }); try { try { plan.beforeClose?.(session, failure); } catch (error) { if (!failure) { failure = { error }; throw error; } } diff --git a/test/helpers/pty/transcript-cost.ts b/test/helpers/pty/transcript-cost.ts new file mode 100644 index 000000000..08f7ccbdb --- /dev/null +++ b/test/helpers/pty/transcript-cost.ts @@ -0,0 +1,74 @@ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { streamedUsageCostUsd } from '../../../lib/pricing'; + +/** The session ledger's cost field for a PTY session: present only when its transcript prices. */ +export function pricedTranscript(session: { transcriptCostUsd?(): number | undefined } | undefined): { streamed_cost_usd?: number } { + const cost = session?.transcriptCostUsd?.(); + return cost === undefined ? {} : { streamed_cost_usd: cost }; +} + +/** Top-level session transcripts (`projects//.jsonl`) under a config dir; empty when unreadable. */ +export function transcriptFiles(configDir: string | undefined): Set { + if (!configDir) return new Set(); + try { + const projects = path.join(configDir, 'projects'); + return new Set(fs.readdirSync(projects, { withFileTypes: true }).filter(dir => dir.isDirectory()) + .flatMap(dir => fs.readdirSync(path.join(projects, dir.name)).filter(f => f.endsWith('.jsonl')).map(f => path.join(projects, dir.name, f)))); + } catch { + return new Set(); + } +} + +/** + * The session id of the one transcript that appeared since `before` and was written for `cwd`. Concurrent PTY tests in + * one process share a hermetic config dir, so a launch claims only a new transcript for its own cwd; none or several + * (two sessions in one cwd) is undefined, which leaves the cost unknown. + */ +export function newTranscriptSessionId(configDir: string | undefined, before: ReadonlySet, cwd: string): string | undefined { + try { + const cwds = new Set([cwd, fs.realpathSync(cwd)]); + const mine = [...transcriptFiles(configDir)].filter(file => !before.has(file) && fs.readFileSync(file, 'utf8').split('\n').some(line => { + try { + return cwds.has(JSON.parse(line)?.cwd); + } catch { + return false; + } + })); + return mine.length === 1 ? path.basename(mine[0], '.jsonl') : undefined; + } catch { + return undefined; + } +} + +/** + * Price one PTY session from the transcripts Claude Code wrote under its owned config dir: + * `projects//.jsonl` (inline sidechains included) and `projects///subagents/*.jsonl`, + * rows deduplicated by message id. Undefined (cost-unknown, never $0) without a config dir, session id, transcript or + * usage, when a message's model has no price row, or when any read fails: it runs in ledger writers before close and + * cleanup, so it never throws. + */ +export function ptyTranscriptCostUsd(configDir: string | null | undefined, sessionId: string | undefined): number | undefined { + if (!configDir || !sessionId) return undefined; + try { + const projects = path.join(configDir, 'projects'); + const files: string[] = []; + for (const dir of fs.readdirSync(projects, { withFileTypes: true })) { + if (!dir.isDirectory()) continue; + const main = path.join(projects, dir.name, `${sessionId}.jsonl`); + if (fs.existsSync(main)) files.push(main); + const subagents = path.join(projects, dir.name, sessionId, 'subagents'); + if (fs.existsSync(subagents)) files.push(...fs.readdirSync(subagents).filter(f => f.endsWith('.jsonl')).map(f => path.join(subagents, f))); + } + const rows = files.flatMap(file => fs.readFileSync(file, 'utf8').split('\n').flatMap(line => { + try { + return [JSON.parse(line)]; + } catch { + return []; + } + })); + return streamedUsageCostUsd(rows); + } catch { + return undefined; + } +} diff --git a/test/helpers/session-ledger.ts b/test/helpers/session-ledger.ts index 5d8b796cc..c99baaaa0 100644 --- a/test/helpers/session-ledger.ts +++ b/test/helpers/session-ledger.ts @@ -52,7 +52,7 @@ export interface SessionLedgerRow { liveness?: LivenessSummary; /** Whether the runner captured this session's billing (a terminal result with total_cost_usd). */ billed?: boolean; - /** A session that ended without a terminal result (aborted, thrown): its assistant messages' streamed usage at lib/pricing.ts rates, absent when unpriced. */ + /** A session without a terminal result (an aborted Agent SDK session, a PTY session): its assistant messages' usage at lib/pricing.ts rates, from the stream or the PTY transcript; absent when unpriced. */ streamed_cost_usd?: number; /** Codex sessions: the model, its billed cost and token classes. */ model?: string; diff --git a/test/paid-report-cost.test.ts b/test/paid-report-cost.test.ts index ac5e08eff..fcfc2dd30 100644 --- a/test/paid-report-cost.test.ts +++ b/test/paid-report-cost.test.ts @@ -12,6 +12,7 @@ import { anthropicUsageCostUsd, streamedUsageCostUsd } from '../lib/pricing'; import { E2E_TOUCHFILES } from './helpers/touchfiles-data'; import type { SessionLedgerRow } from './helpers/session-ledger'; import { recordE2E } from './helpers/e2e-helpers'; +import { newTranscriptSessionId, pricedTranscript, ptyTranscriptCostUsd, transcriptFiles } from './helpers/pty/transcript-cost'; const ROOT = path.resolve(import.meta.dir, '..'); const session = (over: Partial): SessionLedgerRow => ({ key: 'k', runner: 'agent-sdk', started_at: 't', elapsed_ms: 1, end: 'aborted', ...over }); @@ -128,3 +129,75 @@ describe('junitCensus cost binding', () => { expect(history.map(h => [h.case, h.model])).toEqual([['browse-basic', undefined], ['browse-snapshot', undefined]]); }); }); + +describe('PTY transcript pricing (GSTA-150 PR 3)', () => { + const SID = '11111111-2222-4333-8444-555555555555'; + const OTHER = '99999999-2222-4333-8444-555555555555'; + const assistant = (sid: string, id: string, model: string, usage: object, extra: object = {}) => + JSON.stringify({ type: 'assistant', sessionId: sid, cwd: '/fixture/cwd', message: { id, model, role: 'assistant', usage }, ...extra }); + const fixture = (write: (project: string) => void) => { + const config = fs.mkdtempSync(path.join(os.tmpdir(), 'pty-cost-')); + const project = path.join(config, 'projects', '-fixture-cwd'); + fs.mkdirSync(project, { recursive: true }); + write(project); + return config; + }; + + test('prices the session, its inline sidechains and subagent files, each message id once at its largest output', () => { + const config = fixture(project => { + fs.writeFileSync(path.join(project, `${SID}.jsonl`), [ + JSON.stringify({ type: 'user', sessionId: SID, message: { role: 'user', content: 'hi' } }), + assistant(SID, 'msg_a', 'claude-opus-5-5', { input_tokens: 1000, output_tokens: 10 }), + assistant(SID, 'msg_a', 'claude-opus-5-5', { input_tokens: 1000, output_tokens: 500 }), + assistant(SID, 'msg_side', 'claude-opus-5-5', { input_tokens: 0, output_tokens: 1000 }, { isSidechain: true }), + assistant(SID, 'msg_syn', '', { input_tokens: 0, output_tokens: 0 }), + '{"partial":', + ].join('\n')); + fs.mkdirSync(path.join(project, SID, 'subagents'), { recursive: true }); + fs.writeFileSync(path.join(project, SID, 'subagents', 'agent-x.jsonl'), + assistant(SID, 'msg_sub', 'claude-opus-5-5', { input_tokens: 0, cache_read_input_tokens: 10000, output_tokens: 0 }) + '\n'); + fs.writeFileSync(path.join(project, `${OTHER}.jsonl`), assistant(OTHER, 'msg_foreign', 'claude-opus-5-5', { input_tokens: 1_000_000 }) + '\n'); + }); + try { + // opus-5-5: $4 in, $20 out, $0.20 cached per MTok. + const expected = (1000 * 4 + 500 * 20 + 1000 * 20 + 10000 * 0.2) / 1_000_000; + expect(ptyTranscriptCostUsd(config, SID)).toBeCloseTo(expected, 6); + expect(pricedTranscript({ transcriptCostUsd: () => ptyTranscriptCostUsd(config, SID) }).streamed_cost_usd).toBeCloseTo(expected, 6); + } finally { fs.rmSync(config, { recursive: true, force: true }); } + }); + + test('stays cost-unknown without a transcript, a session id, usage, a priced model or a readable file, and never throws', () => { + const unpriced = fixture(project => fs.writeFileSync(path.join(project, `${SID}.jsonl`), + assistant(SID, 'msg_a', 'claude-opus-5-5', { output_tokens: 1 }) + '\n' + assistant(SID, 'msg_b', 'some-unpriced-model', { output_tokens: 1 }) + '\n')); + const empty = fixture(project => fs.writeFileSync(path.join(project, `${SID}.jsonl`), JSON.stringify({ type: 'user', sessionId: SID }) + '\n')); + const unreadable = fixture(project => fs.mkdirSync(path.join(project, `${SID}.jsonl`))); + try { + expect(ptyTranscriptCostUsd(unpriced, SID)).toBeUndefined(); + expect(ptyTranscriptCostUsd(empty, SID)).toBeUndefined(); + expect(ptyTranscriptCostUsd(empty, OTHER)).toBeUndefined(); + expect(ptyTranscriptCostUsd(unpriced, undefined)).toBeUndefined(); + expect(ptyTranscriptCostUsd(null, SID)).toBeUndefined(); + expect(ptyTranscriptCostUsd(path.join(empty, 'missing'), SID)).toBeUndefined(); + expect(ptyTranscriptCostUsd(unreadable, SID)).toBeUndefined(); + expect(newTranscriptSessionId(unreadable, new Set(), '/fixture/cwd')).toBeUndefined(); + expect(pricedTranscript(undefined)).toEqual({}); + expect(pricedTranscript({})).toEqual({}); + } finally { for (const dir of [unpriced, empty, unreadable]) fs.rmSync(dir, { recursive: true, force: true }); } + }); + + test('a launch claims only the one transcript it created for its own cwd', () => { + const cwd = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'pty-cost-cwd-'))); + const line = (sid: string, at: string) => JSON.stringify({ type: 'user', sessionId: sid, cwd: at }) + '\n'; + const config = fixture(project => fs.writeFileSync(path.join(project, `${OTHER}.jsonl`), line(OTHER, cwd))); + try { + const before = transcriptFiles(config); + const project = path.join(config, 'projects', '-fixture-cwd'); + fs.writeFileSync(path.join(project, '22222222-2222-4333-8444-555555555555.jsonl'), line('22222222-2222-4333-8444-555555555555', '/a/concurrent/test')); + expect(newTranscriptSessionId(config, before, cwd)).toBeUndefined(); + fs.writeFileSync(path.join(project, `${SID}.jsonl`), line(SID, cwd)); + expect(newTranscriptSessionId(config, before, cwd)).toBe(SID); + fs.writeFileSync(path.join(project, '33333333-2222-4333-8444-555555555555.jsonl'), line('33333333-2222-4333-8444-555555555555', cwd)); + expect(newTranscriptSessionId(config, before, cwd)).toBeUndefined(); + } finally { fs.rmSync(config, { recursive: true, force: true }); fs.rmSync(cwd, { recursive: true, force: true }); } + }); +});