docs: document one-case-shard cost attribution and the train verdict cost line; put the undercounted train costs on record

/document-release (ship-owned audit) added the attribution rule and the ATTEMPT: cost-line form to docs/TESTING_INTERNALS.md, docs/troubleshooting.md#cost-unknown and docs/evals/census-red.md. The CHANGELOG headline now says every train cost the coordinator reported until this fix was the undercounted verdict figure (about $24 per train against about $33 real).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-26 15:37:16 +00:00
1 parent bd9c9348be
commit 1b3aab988a
4 files changed
+18 -6

No files matched your search

+1 -1
View File
@@ -2,7 +2,7 @@
## [Unreleased]
**Merge trains fill at 3 READY PRs or one hour after the first READY, and a train verdict's cost line no longer hides the trials it could not price.** On 2026-10-26 the 5-PR / 3-hour fill left READY PRs waiting most of a day, and `train verdict` printed `cost=$24.41` for a train whose slice report said `cost $32.92 known + 54 of 136 trial(s) cost unknown`. The slice report was right: 54 E2E trials had no cost attributed because their result records are named by a display label ("/benchmark workflow") while the case is named by its id (`benchmark-workflow`), so the per-case sum the train record carries was short by $8.51.
**Merge trains fill at 3 READY PRs or one hour after the first READY, and a train verdict's cost line no longer hides the trials it could not price.** On 2026-10-26 the 5-PR / 3-hour fill left READY PRs waiting most of a day, and `train verdict` printed `cost=$24.41` for a train whose slice report said `cost $32.92 known + 54 of 136 trial(s) cost unknown`. The slice report was right: 54 E2E trials had no cost attributed because their result records are named by a display label ("/benchmark workflow") while the case is named by its id (`benchmark-workflow`), so the per-case sum the train record carries was short by $8.51. Every train cost the coordinator reported until this fix was that undercounted verdict figure (about $24 per train against about $33 real).
### Changed
- **`.gstack/ship-policy.json`**: `train.max_size` 5 → 3 and `train.fill_hours` 3 → 1. `fillDue` already dispatches at either bound, whichever comes first; a fourth READY PR is `HELD:` for the next train. docs/ship-policy.md says so in the fill paragraph and its example block.
+10 -3
View File
@@ -642,9 +642,16 @@ holds the panel's gating median), each part capped at 200
characters. `sessions[]` (at most 32) summarizes the trial's session-ledger
rows. `cost_known: false` marks a record or trial whose harness captured no
billing: PTY and Codex sessions never bill, and an SDK session bills only with a
terminal `total_cost_usd`. An absent field means known. Census and marathon
cost lines print the known sum and `N of M trial(s) cost unknown`, never a bare
total over unknowns.
terminal `total_cost_usd`. It also marks a trial no result record could be
attributed to: a record named by the case id or carrying `case_id` is that
case's, and a record named by a display label (`/qa fix loop`) is the shard's
only case's when the shard ran one case, so a multi-case shard whose records
carry display labels leaves those trials cost-unknown rather than guessing
which case paid (`junitCensus` in `scripts/lib/paid-report.ts`). An absent
field means known. Census and marathon cost lines print the known sum and
`N of M trial(s) cost unknown`, and a train verdict's `ATTEMPT:` line prints
`cost=$<known> known + <n> of <m> case(s) cost unknown` when any case in the
train record has no cost, never a bare total over unknowns.
<a id="session-ledger"></a>
**Session ledger.** Every runner (session-runner, the Agent SDK runner
+1 -1
View File
@@ -93,7 +93,7 @@ What incomplete data means:
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
| cause `unrecorded` | the trial was written before `failure_cause` existed |
| cause `unknown` | the trial failed with no evidence for any other cause |
| `cost unknown` | the harness (PTY, Codex) records no billing; only the known sum is shown |
| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown |
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
## 4. Replay and repair (free)
+6 -1
View File
@@ -2179,7 +2179,12 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
**Meaning.** PTY and Codex sessions record no billing, so their cost is
unknown. Only the known sum is shown.
unknown. So is a trial in a multi-case shard whose result records are named by
a display label rather than the case id, because no record can be attributed
to it (a one-case shard's records are its case's). Only the known sum is
shown. A train verdict's `ATTEMPT:` line reports the same as `cost=$<x> known
+ <n> of <m> case(s) cost unknown`, and a bare `cost=$<x>` only when every case
in the train record is priced.
**Fix.** None needed; this is informational.