diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index eb6592644..e82046594 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -648,9 +648,10 @@ no price row). A rule case's trial in the JUnit census (`caseCost` in carries: each ledger session without a terminal result (`billed: false`) adds its `streamed_cost_usd`, and a trial with no record adds each billed session's own `cost_usd` (Codex). The trial is cost-unknown when any record says -`cost_known: false`, when any of those sessions has no priced cost (every PTY -session, a `claude -p` session that ended without a result, an aborted Agent -SDK session with no usage or an unpriced model), or when it has neither records +`cost_known: false`, when any of those sessions has no priced cost (a PTY +session whose transcript does not price, a `claude -p` session that ended +without a result, an aborted Agent SDK session with no usage or an unpriced +model), or when it has neither records nor sessions. A case in `MODEL_FREE_CASES` (`timeline-event-flow`, which runs no model) with neither is a known $0. A panel trial (`behavior`, `judge`) from an isolated trial shard keeps the older rule (`trialCostKnown` in @@ -694,8 +695,19 @@ by `streamedUsageCostUsd` (`lib/pricing.ts`), each message id once at its largest output, subagent turns included, `` messages skipped, 5-minute cache writes at 1.25x and 1-hour writes at 2x the input rate. The field is absent when no message carries usage or any message's model has no -price row, so an unpriced session stays cost-unknown, never $0. The ledger -rides in each slice artifact beside the shard results. +price row, so an unpriced session stays cost-unknown, never $0. A PTY session +records `streamed_cost_usd` from its transcript the same way: the hermetic +launcher (`prepareLaunch` in `test/helpers/pty/launch.ts`) pins +`--session-id ` when the caller passes none and is not resuming +(`--resume`, `-r`, `--continue`, `-c`), and both PTY ledger writers price +`/projects/*/.jsonl` plus `/subagents/*.jsonl` +(`test/helpers/pty/transcript-cost.ts`). Pricing keys on the session id, not +the whole directory, because concurrent PTY tests in one process share one +hermetic config dir. A non-hermetic or resumed launch without a caller +`--session-id`, a missing transcript, no usage, an unpriced model or a read +error leaves the field absent (cost-unknown), and a pricing failure never +fails the test. The row keeps `billed: false`. The ledger rides in each slice +artifact beside the shard results. **Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate below 95% over at least 10 post-policy trials of the case's current input @@ -1207,9 +1219,11 @@ observation never removes a command and a missing or empty artifact never erases one; removing a command is a reviewed edit. While the map is in shadow mode it changes no selection, so a PR that changes only the map selects no paid case. -Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is -unknown: about 39% of observed case runs in the committed map. The shadow block -and `test:health` both say so. +Dollar figures are lower bounds. Known cost excludes trials whose cost is +unknown: about 39% of observed case runs in the committed map, mostly PTY +trials recorded before PTY sessions were priced from their transcripts +([Failure causes and details](#failure-causes)). The shadow block and +`test:health` both say so. - **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate cases the map would select, would drop and would add, a provenance line per diff --git a/docs/evals/census-red.md b/docs/evals/census-red.md index cc8d7b273..e3bb69991 100644 --- a/docs/evals/census-red.md +++ b/docs/evals/census-red.md @@ -93,7 +93,7 @@ What incomplete data means: | headroom `censored` | a sample timed out, so its real duration is at least the budget | | cause `unrecorded` | the trial was written before `failure_cause` existed | | cause `unknown` | the trial failed with no evidence for any other cause | -| `cost unknown` | a session's cost was not captured (a PTY session, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown | +| `cost unknown` | a session's cost was not captured (a PTY session without a priced transcript, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown | | `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed | ## 4. Replay and repair (free) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 37ed4fb07..f0c74c96e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2263,11 +2263,13 @@ case with `bun run eval:pass-rates --headroom --case `. ### `cost unknown (no billing captured for any of trial(s))` / `cost $ known + of trial(s) cost unknown` **Meaning.** Some trials' cost was not captured, so only the known sum is -shown. A trial is cost-unknown when it ran a PTY session (PTY sessions record -no billing), a Codex session that completed no turn or whose model has no price -row, a `claude -p` session that ended without a result, or an aborted Agent -SDK session whose streamed usage could not be priced (no usage, or a model with -no price row). An aborted Agent SDK session with priced usage counts its +shown. A trial is cost-unknown when it ran a PTY session whose transcript +could not be priced (a non-hermetic or resumed launch without a caller +`--session-id`, no transcript or usage, or a model with no price row), a +Codex session that completed no turn or whose model has no price row, a +`claude -p` session that ended without a result, or an aborted Agent SDK +session whose streamed usage could not be priced (no usage, or a model with no +price row). A PTY or aborted Agent SDK session that does price counts its `streamed_cost_usd` instead. A trial is also unknown when no record and no session can be attributed to it, for example a multi-case shard's display-label record without `cost_case_id` (a one-case shard's records are