From 1b3aab988a4e4a902ed6b2da5b27173ffac4f141 Mon Sep 17 00:00:00 2001 From: garrytan Date: Mon, 26 Oct 2026 15:37:16 +0000 Subject: [PATCH] docs: document one-case-shard cost attribution and the train verdict cost line; put the undercounted train costs on record /document-release (ship-owned audit) added the attribution rule and the ATTEMPT: cost-line form to docs/TESTING_INTERNALS.md, docs/troubleshooting.md#cost-unknown and docs/evals/census-red.md. The CHANGELOG headline now says every train cost the coordinator reported until this fix was the undercounted verdict figure (about $24 per train against about $33 real). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 2 +- docs/TESTING_INTERNALS.md | 13 ++++++++++--- docs/evals/census-red.md | 2 +- docs/troubleshooting.md | 7 ++++++- 4 files changed, 18 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a89dc2984..c4485a5e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [Unreleased] -**Merge trains fill at 3 READY PRs or one hour after the first READY, and a train verdict's cost line no longer hides the trials it could not price.** On 2026-10-26 the 5-PR / 3-hour fill left READY PRs waiting most of a day, and `train verdict` printed `cost=$24.41` for a train whose slice report said `cost $32.92 known + 54 of 136 trial(s) cost unknown`. The slice report was right: 54 E2E trials had no cost attributed because their result records are named by a display label ("/benchmark workflow") while the case is named by its id (`benchmark-workflow`), so the per-case sum the train record carries was short by $8.51. +**Merge trains fill at 3 READY PRs or one hour after the first READY, and a train verdict's cost line no longer hides the trials it could not price.** On 2026-10-26 the 5-PR / 3-hour fill left READY PRs waiting most of a day, and `train verdict` printed `cost=$24.41` for a train whose slice report said `cost $32.92 known + 54 of 136 trial(s) cost unknown`. The slice report was right: 54 E2E trials had no cost attributed because their result records are named by a display label ("/benchmark workflow") while the case is named by its id (`benchmark-workflow`), so the per-case sum the train record carries was short by $8.51. Every train cost the coordinator reported until this fix was that undercounted verdict figure (about $24 per train against about $33 real). ### Changed - **`.gstack/ship-policy.json`**: `train.max_size` 5 → 3 and `train.fill_hours` 3 → 1. `fillDue` already dispatches at either bound, whichever comes first; a fourth READY PR is `HELD:` for the next train. docs/ship-policy.md says so in the fill paragraph and its example block. diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 8528a1609..9bb6162c6 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -642,9 +642,16 @@ holds the panel's gating median), each part capped at 200 characters. `sessions[]` (at most 32) summarizes the trial's session-ledger rows. `cost_known: false` marks a record or trial whose harness captured no billing: PTY and Codex sessions never bill, and an SDK session bills only with a -terminal `total_cost_usd`. An absent field means known. Census and marathon -cost lines print the known sum and `N of M trial(s) cost unknown`, never a bare -total over unknowns. +terminal `total_cost_usd`. It also marks a trial no result record could be +attributed to: a record named by the case id or carrying `case_id` is that +case's, and a record named by a display label (`/qa fix loop`) is the shard's +only case's when the shard ran one case, so a multi-case shard whose records +carry display labels leaves those trials cost-unknown rather than guessing +which case paid (`junitCensus` in `scripts/lib/paid-report.ts`). An absent +field means known. Census and marathon cost lines print the known sum and +`N of M trial(s) cost unknown`, and a train verdict's `ATTEMPT:` line prints +`cost=$ known + of case(s) cost unknown` when any case in the +train record has no cost, never a bare total over unknowns. **Session ledger.** Every runner (session-runner, the Agent SDK runner diff --git a/docs/evals/census-red.md b/docs/evals/census-red.md index f6950ff3a..f3503b625 100644 --- a/docs/evals/census-red.md +++ b/docs/evals/census-red.md @@ -93,7 +93,7 @@ What incomplete data means: | headroom `censored` | a sample timed out, so its real duration is at least the budget | | cause `unrecorded` | the trial was written before `failure_cause` existed | | cause `unknown` | the trial failed with no evidence for any other cause | -| `cost unknown` | the harness (PTY, Codex) records no billing; only the known sum is shown | +| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown | | `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed | ## 4. Replay and repair (free) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4091c3291..d116a69fa 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2179,7 +2179,12 @@ case with `bun run eval:pass-rates --headroom --case `. ### `cost unknown (no billing captured for any of trial(s))` / `cost $ known + of trial(s) cost unknown` **Meaning.** PTY and Codex sessions record no billing, so their cost is -unknown. Only the known sum is shown. +unknown. So is a trial in a multi-case shard whose result records are named by +a display label rather than the case id, because no record can be attributed +to it (a one-case shard's records are its case's). Only the known sum is +shown. A train verdict's `ATTEMPT:` line reports the same as `cost=$ known ++ of case(s) cost unknown`, and a bare `cost=$` only when every case +in the train record is priced. **Fix.** None needed; this is informational.