docs: PTY sessions priced from their transcripts

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-29 04:59:35 +00:00
1 parent eb0be6ec1d
commit 2c2c18901c
3 files changed
+30 -14

No files matched your search

+22 -8
View File
@@ -648,9 +648,10 @@ no price row). A rule case's trial in the JUnit census (`caseCost` in
carries: each ledger session without a terminal result (`billed: false`) adds
its `streamed_cost_usd`, and a trial with no record adds each billed session's
own `cost_usd` (Codex). The trial is cost-unknown when any record says
`cost_known: false`, when any of those sessions has no priced cost (every PTY
session, a `claude -p` session that ended without a result, an aborted Agent
SDK session with no usage or an unpriced model), or when it has neither records
`cost_known: false`, when any of those sessions has no priced cost (a PTY
session whose transcript does not price, a `claude -p` session that ended
without a result, an aborted Agent SDK session with no usage or an unpriced
model), or when it has neither records
nor sessions. A case in `MODEL_FREE_CASES` (`timeline-event-flow`, which runs
no model) with neither is a known $0. A panel trial (`behavior`, `judge`) from
an isolated trial shard keeps the older rule (`trialCostKnown` in
@@ -694,8 +695,19 @@ by `streamedUsageCostUsd` (`lib/pricing.ts`), each message id once at its
largest output, subagent turns included, `<synthetic>` messages skipped,
5-minute cache writes at 1.25x and 1-hour writes at 2x the input rate. The
field is absent when no message carries usage or any message's model has no
price row, so an unpriced session stays cost-unknown, never $0. The ledger
rides in each slice artifact beside the shard results.
price row, so an unpriced session stays cost-unknown, never $0. A PTY session
records `streamed_cost_usd` from its transcript the same way: the hermetic
launcher (`prepareLaunch` in `test/helpers/pty/launch.ts`) pins
`--session-id <uuid>` when the caller passes none and is not resuming
(`--resume`, `-r`, `--continue`, `-c`), and both PTY ledger writers price
`<CLAUDE_CONFIG_DIR>/projects/*/<id>.jsonl` plus `<id>/subagents/*.jsonl`
(`test/helpers/pty/transcript-cost.ts`). Pricing keys on the session id, not
the whole directory, because concurrent PTY tests in one process share one
hermetic config dir. A non-hermetic or resumed launch without a caller
`--session-id`, a missing transcript, no usage, an unpriced model or a read
error leaves the field absent (cost-unknown), and a pricing failure never
fails the test. The row keeps `billed: false`. The ledger rides in each slice
artifact beside the shard results.
**Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate
below 95% over at least 10 post-policy trials of the case's current input
@@ -1207,9 +1219,11 @@ observation never removes a command and a missing or empty artifact never erases
one; removing a command is a reviewed edit. While the map is in shadow mode it
changes no selection, so a PR that changes only the map selects no paid case.
Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is
unknown: about 39% of observed case runs in the committed map. The shadow block
and `test:health` both say so.
Dollar figures are lower bounds. Known cost excludes trials whose cost is
unknown: about 39% of observed case runs in the committed map, mostly PTY
trials recorded before PTY sessions were priced from their transcripts
([Failure causes and details](#failure-causes)). The shadow block and
`test:health` both say so.
- **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate
cases the map would select, would drop and would add, a provenance line per
+1 -1
View File
@@ -93,7 +93,7 @@ What incomplete data means:
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
| cause `unrecorded` | the trial was written before `failure_cause` existed |
| cause `unknown` | the trial failed with no evidence for any other cause |
| `cost unknown` | a session's cost was not captured (a PTY session, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown |
| `cost unknown` | a session's cost was not captured (a PTY session without a priced transcript, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown |
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
## 4. Replay and repair (free)
+7 -5
View File
@@ -2263,11 +2263,13 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
**Meaning.** Some trials' cost was not captured, so only the known sum is
shown. A trial is cost-unknown when it ran a PTY session (PTY sessions record
no billing), a Codex session that completed no turn or whose model has no price
row, a `claude -p` session that ended without a result, or an aborted Agent
SDK session whose streamed usage could not be priced (no usage, or a model with
no price row). An aborted Agent SDK session with priced usage counts its
shown. A trial is cost-unknown when it ran a PTY session whose transcript
could not be priced (a non-hermetic or resumed launch without a caller
`--session-id`, no transcript or usage, or a model with no price row), a
Codex session that completed no turn or whose model has no price row, a
`claude -p` session that ended without a result, or an aborted Agent SDK
session whose streamed usage could not be priced (no usage, or a model with no
price row). A PTY or aborted Agent SDK session that does price counts its
`streamed_cost_usd` instead. A trial is also unknown when no record and no
session can be attributed to it, for example a multi-case shard's
display-label record without `cost_case_id` (a one-case shard's records are