mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-03 15:27:56 +01:00
docs: PTY sessions priced from their transcripts
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
eb0be6ec1d
commit
2c2c18901c
3 files changed
+30
-14
No files matched your search
@@ -648,9 +648,10 @@ no price row). A rule case's trial in the JUnit census (`caseCost` in
|
||||
carries: each ledger session without a terminal result (`billed: false`) adds
|
||||
its `streamed_cost_usd`, and a trial with no record adds each billed session's
|
||||
own `cost_usd` (Codex). The trial is cost-unknown when any record says
|
||||
`cost_known: false`, when any of those sessions has no priced cost (every PTY
|
||||
session, a `claude -p` session that ended without a result, an aborted Agent
|
||||
SDK session with no usage or an unpriced model), or when it has neither records
|
||||
`cost_known: false`, when any of those sessions has no priced cost (a PTY
|
||||
session whose transcript does not price, a `claude -p` session that ended
|
||||
without a result, an aborted Agent SDK session with no usage or an unpriced
|
||||
model), or when it has neither records
|
||||
nor sessions. A case in `MODEL_FREE_CASES` (`timeline-event-flow`, which runs
|
||||
no model) with neither is a known $0. A panel trial (`behavior`, `judge`) from
|
||||
an isolated trial shard keeps the older rule (`trialCostKnown` in
|
||||
@@ -694,8 +695,19 @@ by `streamedUsageCostUsd` (`lib/pricing.ts`), each message id once at its
|
||||
largest output, subagent turns included, `<synthetic>` messages skipped,
|
||||
5-minute cache writes at 1.25x and 1-hour writes at 2x the input rate. The
|
||||
field is absent when no message carries usage or any message's model has no
|
||||
price row, so an unpriced session stays cost-unknown, never $0. The ledger
|
||||
rides in each slice artifact beside the shard results.
|
||||
price row, so an unpriced session stays cost-unknown, never $0. A PTY session
|
||||
records `streamed_cost_usd` from its transcript the same way: the hermetic
|
||||
launcher (`prepareLaunch` in `test/helpers/pty/launch.ts`) pins
|
||||
`--session-id <uuid>` when the caller passes none and is not resuming
|
||||
(`--resume`, `-r`, `--continue`, `-c`), and both PTY ledger writers price
|
||||
`<CLAUDE_CONFIG_DIR>/projects/*/<id>.jsonl` plus `<id>/subagents/*.jsonl`
|
||||
(`test/helpers/pty/transcript-cost.ts`). Pricing keys on the session id, not
|
||||
the whole directory, because concurrent PTY tests in one process share one
|
||||
hermetic config dir. A non-hermetic or resumed launch without a caller
|
||||
`--session-id`, a missing transcript, no usage, an unpriced model or a read
|
||||
error leaves the field absent (cost-unknown), and a pricing failure never
|
||||
fails the test. The row keeps `billed: false`. The ledger rides in each slice
|
||||
artifact beside the shard results.
|
||||
|
||||
**Quarantine** (`CASE_QUARANTINE`, same file). An entry needs: a per-trial rate
|
||||
below 95% over at least 10 post-policy trials of the case's current input
|
||||
@@ -1207,9 +1219,11 @@ observation never removes a command and a missing or empty artifact never erases
|
||||
one; removing a command is a reviewed edit. While the map is in shadow mode it
|
||||
changes no selection, so a PR that changes only the map selects no paid case.
|
||||
|
||||
Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is
|
||||
unknown: about 39% of observed case runs in the committed map. The shadow block
|
||||
and `test:health` both say so.
|
||||
Dollar figures are lower bounds. Known cost excludes trials whose cost is
|
||||
unknown: about 39% of observed case runs in the committed map, mostly PTY
|
||||
trials recorded before PTY sessions were priced from their transcripts
|
||||
([Failure causes and details](#failure-causes)). The shadow block and
|
||||
`test:health` both say so.
|
||||
|
||||
- **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate
|
||||
cases the map would select, would drop and would add, a provenance line per
|
||||
|
||||
@@ -93,7 +93,7 @@ What incomplete data means:
|
||||
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
|
||||
| cause `unrecorded` | the trial was written before `failure_cause` existed |
|
||||
| cause `unknown` | the trial failed with no evidence for any other cause |
|
||||
| `cost unknown` | a session's cost was not captured (a PTY session, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown |
|
||||
| `cost unknown` | a session's cost was not captured (a PTY session without a priced transcript, an unbilled Codex session, a `claude -p` session without a result, an aborted Agent SDK session with no priced streamed usage), or no record or session can be attributed to the trial (a multi-case shard's display-label record without `cost_case_id`); only the known sum is shown |
|
||||
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
|
||||
|
||||
## 4. Replay and repair (free)
|
||||
|
||||
@@ -2263,11 +2263,13 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
|
||||
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
|
||||
|
||||
**Meaning.** Some trials' cost was not captured, so only the known sum is
|
||||
shown. A trial is cost-unknown when it ran a PTY session (PTY sessions record
|
||||
no billing), a Codex session that completed no turn or whose model has no price
|
||||
row, a `claude -p` session that ended without a result, or an aborted Agent
|
||||
SDK session whose streamed usage could not be priced (no usage, or a model with
|
||||
no price row). An aborted Agent SDK session with priced usage counts its
|
||||
shown. A trial is cost-unknown when it ran a PTY session whose transcript
|
||||
could not be priced (a non-hermetic or resumed launch without a caller
|
||||
`--session-id`, no transcript or usage, or a model with no price row), a
|
||||
Codex session that completed no turn or whose model has no price row, a
|
||||
`claude -p` session that ended without a result, or an aborted Agent SDK
|
||||
session whose streamed usage could not be priced (no usage, or a model with no
|
||||
price row). A PTY or aborted Agent SDK session that does price counts its
|
||||
`streamed_cost_usd` instead. A trial is also unknown when no record and no
|
||||
session can be attributed to it, for example a multi-case shard's
|
||||
display-label record without `cost_case_id` (a one-case shard's records are
|
||||
|
||||
Reference in new issue
Block a user