mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-06 00:44:07 +01:00
docs: document one-case-shard cost attribution and the train verdict cost line; put the undercounted train costs on record
/document-release (ship-owned audit) added the attribution rule and the ATTEMPT: cost-line form to docs/TESTING_INTERNALS.md, docs/troubleshooting.md#cost-unknown and docs/evals/census-red.md. The CHANGELOG headline now says every train cost the coordinator reported until this fix was the undercounted verdict figure (about $24 per train against about $33 real). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
bd9c9348be
commit
1b3aab988a
4 files changed
+18
-6
No files matched your search
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
**Merge trains fill at 3 READY PRs or one hour after the first READY, and a train verdict's cost line no longer hides the trials it could not price.** On 2026-10-26 the 5-PR / 3-hour fill left READY PRs waiting most of a day, and `train verdict` printed `cost=$24.41` for a train whose slice report said `cost $32.92 known + 54 of 136 trial(s) cost unknown`. The slice report was right: 54 E2E trials had no cost attributed because their result records are named by a display label ("/benchmark workflow") while the case is named by its id (`benchmark-workflow`), so the per-case sum the train record carries was short by $8.51.
|
||||
**Merge trains fill at 3 READY PRs or one hour after the first READY, and a train verdict's cost line no longer hides the trials it could not price.** On 2026-10-26 the 5-PR / 3-hour fill left READY PRs waiting most of a day, and `train verdict` printed `cost=$24.41` for a train whose slice report said `cost $32.92 known + 54 of 136 trial(s) cost unknown`. The slice report was right: 54 E2E trials had no cost attributed because their result records are named by a display label ("/benchmark workflow") while the case is named by its id (`benchmark-workflow`), so the per-case sum the train record carries was short by $8.51. Every train cost the coordinator reported until this fix was that undercounted verdict figure (about $24 per train against about $33 real).
|
||||
|
||||
### Changed
|
||||
- **`.gstack/ship-policy.json`**: `train.max_size` 5 → 3 and `train.fill_hours` 3 → 1. `fillDue` already dispatches at either bound, whichever comes first; a fourth READY PR is `HELD:` for the next train. docs/ship-policy.md says so in the fill paragraph and its example block.
|
||||
|
||||
@@ -642,9 +642,16 @@ holds the panel's gating median), each part capped at 200
|
||||
characters. `sessions[]` (at most 32) summarizes the trial's session-ledger
|
||||
rows. `cost_known: false` marks a record or trial whose harness captured no
|
||||
billing: PTY and Codex sessions never bill, and an SDK session bills only with a
|
||||
terminal `total_cost_usd`. An absent field means known. Census and marathon
|
||||
cost lines print the known sum and `N of M trial(s) cost unknown`, never a bare
|
||||
total over unknowns.
|
||||
terminal `total_cost_usd`. It also marks a trial no result record could be
|
||||
attributed to: a record named by the case id or carrying `case_id` is that
|
||||
case's, and a record named by a display label (`/qa fix loop`) is the shard's
|
||||
only case's when the shard ran one case, so a multi-case shard whose records
|
||||
carry display labels leaves those trials cost-unknown rather than guessing
|
||||
which case paid (`junitCensus` in `scripts/lib/paid-report.ts`). An absent
|
||||
field means known. Census and marathon cost lines print the known sum and
|
||||
`N of M trial(s) cost unknown`, and a train verdict's `ATTEMPT:` line prints
|
||||
`cost=$<known> known + <n> of <m> case(s) cost unknown` when any case in the
|
||||
train record has no cost, never a bare total over unknowns.
|
||||
|
||||
<a id="session-ledger"></a>
|
||||
**Session ledger.** Every runner (session-runner, the Agent SDK runner
|
||||
|
||||
@@ -93,7 +93,7 @@ What incomplete data means:
|
||||
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
|
||||
| cause `unrecorded` | the trial was written before `failure_cause` existed |
|
||||
| cause `unknown` | the trial failed with no evidence for any other cause |
|
||||
| `cost unknown` | the harness (PTY, Codex) records no billing; only the known sum is shown |
|
||||
| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown |
|
||||
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
|
||||
|
||||
## 4. Replay and repair (free)
|
||||
|
||||
@@ -2179,7 +2179,12 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
|
||||
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
|
||||
|
||||
**Meaning.** PTY and Codex sessions record no billing, so their cost is
|
||||
unknown. Only the known sum is shown.
|
||||
unknown. So is a trial in a multi-case shard whose result records are named by
|
||||
a display label rather than the case id, because no record can be attributed
|
||||
to it (a one-case shard's records are its case's). Only the known sum is
|
||||
shown. A train verdict's `ATTEMPT:` line reports the same as `cost=$<x> known
|
||||
+ <n> of <m> case(s) cost unknown`, and a bare `cost=$<x>` only when every case
|
||||
in the train record is priced.
|
||||
|
||||
**Fix.** None needed; this is informational.
|
||||
|
||||
|
||||
Reference in new issue
Block a user