v1.91.255.0 fix(ship-queue): fill trains at 3 READY PRs or one hour, and price every case shard's trial (#3478)

* fix(ship-queue): fill trains at 3 READY PRs or one hour, and price every case shard's trial

The 5-PR / 3-hour fill left READY PRs waiting most of a day; .gstack/ship-policy.json
now fills at max_size 3 or fill_hours 1, whichever comes first (fillDue already
dispatches at either bound). train verdict printed cost=$24.41 for a train whose slice
report said cost $32.92 known + 54 of 136 trial(s) cost unknown: E2E records named by a
display label ("/benchmark workflow") were never attributed to their case id, so the
per-case sum was short. A case shard's records now belong to its one case; a multi-case
shard with display-labelled records keeps those trials unknown. The ATTEMPT: line names
how many cases have no cost instead of printing a partial sum as the total.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ci): record the paid-report.ts harness decision and keep ship-queue-train-cost.test.ts off the Windows subset

scripts/lib/paid-report.ts is a pinned shared-harness file, so test/harness-version.test.ts wants a HARNESS_VERSION decision for its edit: non-behavioral, because a case shard's records are attributed to its one case for the trial's cost, class and error fields while no case's outcome, panel verdict or selection changes.

test/ship-queue-train-cost.test.ts drives bin/gstack-ship-queue against the #!/bin/sh gh shim, which windows-latest cannot run (windows-free-shard 2 on #3478 at d5e6a7d), so it joins test/ship-queue-train.test.ts in the Windows exclusion list.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: pin a failed display-labelled trial's fields and the $0.00 cost line; say so in the CHANGELOG

The widened record filter also feeds failedRecord, so a failed display-labelled trial in a case shard now takes failure_class, exit_reason and error from its record (train-gate classes a red infra record INFRA instead of VERDICT). The CHANGELOG entry said only the cost fields changed; it now says both. A train priced at zero prints cost=$0.00.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore(harness): say which panel class the paid-report decision moves

The non-behavioral note now names the one verdict-side effect of attributing a case shard's records to its case: a red whose own record says infra or timeout is classed INFRA instead of VERDICT, which changes rerun and bisect eligibility and not a pass rate.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: document one-case-shard cost attribution and the train verdict cost line; put the undercounted train costs on record

/document-release (ship-owned audit) added the attribution rule and the ATTEMPT: cost-line form to docs/TESTING_INTERNALS.md, docs/troubleshooting.md#cost-unknown and docs/evals/census-red.md. The CHANGELOG headline now says every train cost the coordinator reported until this fix was the undercounted verdict figure (about $24 per train against about $33 real).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: the train verdict prints the known+unknown cost form only when some but not every case is priced

The second /document-release audit tightened docs/TESTING_INTERNALS.md and docs/troubleshooting.md#cost-unknown: a train record with no priced case prints no cost at all (classifyGate's cost is null). The CHANGELOG bullet says the same.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.255.0 stamp

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-10-26 20:33:32 +00:00
1 parent 568cd5aa56
commit b004c06dbc
16 files changed
+171 -16

No files matched your search

+1 -1
View File
@@ -12,6 +12,6 @@
"stamp_sensitive_gates": ["tests"],
"pr_count": { "default_pr_count": 1, "split_requires_owner": true },
"maintainers": ["garrytan"],
"train": { "gate_workflow": "evals.yml", "free_workflow": "free-tests.yml", "full_gate_check": "gstack/full-gate", "record_artifact": "gstack-train-record", "max_size": 5, "fill_hours": 3, "screen_selection": "gstack-paid", "trial_reservation_usd": 1.5 },
"train": { "gate_workflow": "evals.yml", "free_workflow": "free-tests.yml", "full_gate_check": "gstack/full-gate", "record_artifact": "gstack-train-record", "max_size": 3, "fill_hours": 1, "screen_selection": "gstack-paid", "trial_reservation_usd": 1.5 },
"min_gstack_version": "1.91.146.0"
}
+20
View File
@@ -1,5 +1,25 @@
# Changelog
## [1.91.255.0] - 2026-10-26
**Merge trains fill at 3 READY PRs or one hour after the first READY, and a train verdict's cost line no longer hides the trials it could not price.** On 2026-10-26 the 5-PR / 3-hour fill left READY PRs waiting most of a day, and `train verdict` printed `cost=$24.41` for a train whose slice report said `cost $32.92 known + 54 of 136 trial(s) cost unknown`. The slice report was right: 54 E2E trials had no cost attributed because their result records are named by a display label ("/benchmark workflow") while the case is named by its id (`benchmark-workflow`), so the per-case sum the train record carries was short by $8.51. Every train cost the coordinator reported until this fix was that undercounted verdict figure (about $24 per train against about $33 real).
### Changed
- **`.gstack/ship-policy.json`**: `train.max_size` 5 → 3 and `train.fill_hours` 3 → 1. `fillDue` already dispatches at either bound, whichever comes first; a fourth READY PR is `HELD:` for the next train. docs/ship-policy.md says so in the fill paragraph and its example block.
### Fixed
- **`scripts/lib/paid-report.ts`**: when a shard ran exactly one case (every case shard since W5a), its result records belong to that case even when they are named by a display label, so the trial's `cost_usd` is the record's and `cost_known` is no longer false; a failed trial in such a shard also takes its `failure_class`, `exit_reason` and `error` from that record (train-gate classes a red `infra` record INFRA instead of VERDICT) where it was a bare `assertion` with the JUnit message. No case's pass or fail changes. A multi-case shard whose records carry display labels keeps those trials cost-unknown rather than guessing which case paid. The train record's per-case `cost_usd` (scripts/train-gate.ts) and the slice report's headline now agree for case shards; the headline was never wrong.
- **`lib/ship-queue-train-verdict.ts`**: the `ATTEMPT:` line prints `cost=$<known> known + <n> of <m> case(s) cost unknown` when some but not every panel is priced, the bare `cost=$<sum>` only when every panel is priced, and no cost at all when none is (unchanged).
### Tests
- `test/gstack-ship-policy.test.ts` pins the live fill rule (3 PRs or one hour) and exercises `fillDue` with the live block at 59 and 60 minutes and at 3 PRs.
- `test/paid-report-census.test.ts`: a display-labelled record in the periodic fixture's `qa-fix-loop` case shard gives the trial `cost_usd: 1.23` with no `cost_known: false`, `verdict.costUsd` 1.23 and the headline `cost $1.23 known + 6 of 7 trial(s) cost unknown` (red before the fix: `cost_usd: 0, cost_known: false`); a failed display-labelled record gives the trial `failure_class: 'timeout'`, its exit reason, error and cost (red before the fix: `assertion` with the JUnit message); the two-case `skill-e2e-cso` shard keeps its display-labelled trial unknown while the id-named one is priced.
- `test/ship-queue-train-cost.test.ts`: a green train whose record has two unpriced panels prints `cost=$0.70 known + 2 of 3 case(s) cost unknown`; a fully priced record prints `cost=$1.50` alone (red before the fix: the bare `cost=$0.70`); every panel priced at zero prints `cost=$0.00`.
- `scripts/lib/windows-curation.ts` excludes `test/ship-queue-train-cost.test.ts` from the Windows subset like `test/ship-queue-train.test.ts`: both drive `bin/gstack-ship-queue` against the `#!/bin/sh` gh shim, which windows-latest cannot run.
### Gate integrity
- No threshold, rubric, trial count, case kind or judge prompt changed. Cost attribution changes what a trial record reports, never whether a case passed.
## [1.91.254.0] - 2026-10-26
**Pregate ergonomics for a repository that is not gstack (Lane D, GSTA-112; plan harness-codify E4–E6; Captain issues 6, 11, 12).** Three Captains lost their first `/ship` pass to `PREGATE_NO_LANE` in a repo that had no `.gstack/pregate.json` yet (FOUN #292, harness-evals #81/#82); the non-downgradable `secrets` check refused 1,800 dummy per-trial tokens, 11-digit Actions run ids read as phone numbers and a public Firebase key every bundle rebuild re-adds (harness-evals #81/#83, GSTA-102, GSTA-106); a lane whose runner could not start read as the diff's failure and was waived by test name (truther #617); and pregate's logs under `os.tmpdir()` were shared between parallel runs and wiped by a Capy wake (gbrain 16).
+1 -1
View File
@@ -1 +1 @@
1.91.254.0
1.91.255.0
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.91.254.0 — regenerate/re-copy after upgrading gstack
# gstack digest v1.91.255.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
+11 -3
View File
@@ -642,9 +642,17 @@ holds the panel's gating median), each part capped at 200
characters. `sessions[]` (at most 32) summarizes the trial's session-ledger
rows. `cost_known: false` marks a record or trial whose harness captured no
billing: PTY and Codex sessions never bill, and an SDK session bills only with a
terminal `total_cost_usd`. An absent field means known. Census and marathon
cost lines print the known sum and `N of M trial(s) cost unknown`, never a bare
total over unknowns.
terminal `total_cost_usd`. It also marks a trial no result record could be
attributed to: a record named by the case id or carrying `case_id` is that
case's, and a record named by a display label (`/qa fix loop`) is the shard's
only case's when the shard ran one case, so a multi-case shard whose records
carry display labels leaves those trials cost-unknown rather than guessing
which case paid (`junitCensus` in `scripts/lib/paid-report.ts`). An absent
field means known. Census and marathon cost lines print the known sum and
`N of M trial(s) cost unknown`, and a train verdict's `ATTEMPT:` line prints
`cost=$<known> known + <n> of <m> case(s) cost unknown` when some but not
every case in the train record has a cost (a record with no priced case prints
no cost at all), never a bare total over unknowns.
<a id="session-ledger"></a>
**Session ledger.** Every runner (session-runner, the Agent SDK runner
+1 -1
View File
@@ -93,7 +93,7 @@ What incomplete data means:
| headroom `censored` | a sample timed out, so its real duration is at least the budget |
| cause `unrecorded` | the trial was written before `failure_cause` existed |
| cause `unknown` | the trial failed with no evidence for any other cause |
| `cost unknown` | the harness (PTY, Codex) records no billing; only the known sum is shown |
| `cost unknown` | the harness (PTY, Codex) records no billing, or the trial sits in a multi-case shard whose records are named by display label and cannot be attributed to it; only the known sum is shown |
| `history unavailable (...)` | the GitHub history could not be read; `--gate` fails closed |
## 4. Replay and repair (free)
+6 -4
View File
@@ -317,7 +317,7 @@ warning on stderr). A block for gstack itself would read:
"train": {
"gate_workflow": "evals.yml", "free_workflow": "free-tests.yml",
"full_gate_check": "gstack/full-gate", "record_artifact": "gstack-train-record",
"max_size": 5, "fill_hours": 3, "screen_selection": "gstack-paid",
"max_size": 3, "fill_hours": 1, "screen_selection": "gstack-paid",
"trial_reservation_usd": 1.5
},
"min_gstack_version": "<the version that introduced train>"
@@ -437,9 +437,11 @@ tree=<tree>` per prefix and one `SLOT:` per PR. `--dry-run` adds the
writes `train.json`, pushes `train/<nonce>`, dispatches `free_workflow` there
and waits (`--free-timeout <s>`, default 2700) for a green
`workflow_dispatch` run on that exact SHA, then dispatches the full gate
through eval-bg's nonce. A train under `max_size` waits until its oldest READY
has waited `fill_hours` (`QUEUE_TRAIN_WAITING`); `--urgent` dispatches a
one-PR train at once. The same pins on the same base return the live train
through eval-bg's nonce. A train fills at `max_size` READY PRs or when its
oldest READY has waited `fill_hours`, whichever comes first; until then
`QUEUE_TRAIN_WAITING` names the time, and `--urgent` dispatches a one-PR
train at once. gstack's own block fills at 3 PRs or one hour (it was 5 PRs or
three hours until 2026-10-26, which left READY PRs waiting most of a day). The same pins on the same base return the live train
instead of a second one; `train build --train <nonce>` resumes a train from
its record. Refusals, all before any spend: `QUEUE_TRAIN_UNTRUSTED` (a fork
head), `QUEUE_TRAIN_MOVED` (a head moved from its pin),
+6 -1
View File
@@ -2179,7 +2179,12 @@ case with `bun run eval:pass-rates --headroom --case <id>`.
### `cost unknown (no billing captured for any of <N> trial(s))` / `cost $<x> known + <N> of <M> trial(s) cost unknown`
**Meaning.** PTY and Codex sessions record no billing, so their cost is
unknown. Only the known sum is shown.
unknown. So is a trial in a multi-case shard whose result records are named by
a display label rather than the case id, because no record can be attributed
to it (a one-case shard's records are its case's). Only the known sum is
shown. A train verdict's `ATTEMPT:` line reports the same as `cost=$<x> known
+ <n> of <m> case(s) cost unknown`, a bare `cost=$<x>` only when every case
in the train record is priced, and no cost at all when none is.
**Fix.** None needed; this is informational.
+6 -1
View File
@@ -82,7 +82,12 @@ export function classifyGate(raw: unknown, expect: { nonce: string; head: string
}
const gateAttempts = (t: TrainRecord) => attemptsOf(t).filter(a => a.purpose !== 'bisect');
const attemptLine = (a: AttemptRecord) => `ATTEMPT: ${a.id} ${a.purpose} run ${a.run?.id ?? '-'} ${a.outcome ?? a.state}${a.red_cases.length ? ` red=${a.red_cases.join(',')}` : ''}${a.cost_usd !== null ? ` cost=$${a.cost_usd.toFixed(2)}` : ''}${a.detail ? ` (${a.detail})` : ''}`;
const costText = (a: AttemptRecord) => {
if (a.cost_usd === null) return '';
const unknown = a.panels.filter(p => p.cost_usd === null).length;
return ` cost=$${a.cost_usd.toFixed(2)}${unknown ? ` known + ${unknown} of ${a.panels.length} case(s) cost unknown` : ''}`;
};
const attemptLine = (a: AttemptRecord) => `ATTEMPT: ${a.id} ${a.purpose} run ${a.run?.id ?? '-'} ${a.outcome ?? a.state}${a.red_cases.length ? ` red=${a.red_cases.join(',')}` : ''}${costText(a)}${a.detail ? ` (${a.detail})` : ''}`;
function casePrice(env: TrainEnv, red: AttemptRecord, id: string): number {
return red.panels.find(p => p.case === id)?.cost_usd ?? env.train.trial_reservation_usd;
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gstack",
"version": "1.91.254",
"version": "1.91.255",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
+1 -1
View File
@@ -13,7 +13,7 @@
"files": {
"scripts/lib/paid-cases.ts": {"blob":"f862cf5241580c7593b9ee62eacf5ef046248d26","since":6,"nonBehavioral":"Claude-host verdicts are unchanged: the Codex host (EVALS_HOST=codex, codex-host-runner) runs only when selected and records its own per-model series; the Codex runner refactor (execCodex extraction, onSpawn hook) keeps its events and verdicts and adds token cost accounting; host fields on the plan/report are absent for claude"},
"scripts/lib/paid-plan.ts": {"blob":"e8ce3cb98474dded38c5128a631d283b0ca35b63","since":6,"nonBehavioral":"F1 rule-16 flaky rerun: report aggregation, rerun planning and the gate slice exit code; no case's execution, trial outcome, trial record or panel verdict changes, so no pass-rate series resets"},
"scripts/lib/paid-report.ts": {"blob":"19a87cb02bb3cd0f57da7719a5c30b7a4df81b2b","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"},
"scripts/lib/paid-report.ts": {"blob":"224dc5e8ba23a3589a37bad94db72ed1ca3c7c77","since":6,"nonBehavioral":"a case shard's result records are attributed to its one case: the trial's cost_usd and cost_known come from them, and a failed trial's failure_class, exit_reason and error come from its record instead of a bare assertion with the JUnit message, so a red whose record says infra or timeout is classed by that record (INFRA instead of VERDICT: rerun and bisect eligibility, not a pass rate); no case's outcome or selection changes and pass rates count outcomes only, so no pass-rate series resets"},
"scripts/lib/published-text.ts": {"blob":"3f2e4d616b5dd0204035710c66b64d49055c5714","since":6},
"scripts/lib/shard-engine.ts": {"blob":"dc714d7d897aed185c0ec3d2c92ad22d954039e5","since":6,"nonBehavioral":"createShardSandbox drops the Capy launch markers (GSTACK_SESSION_KIND, GSTACK_CAPY, GSTACK_EPHEMERAL, GSTACK_INSTALL_VERDICT) from shard environments; CI never sets them, so no case's input or verdict changes there, and on a Capy machine fixtures now see the session kind they set themselves instead of the installer's"},
"scripts/test-paid-shards.ts": {"blob":"5439d0a8f6e1569fa7c924af20d67c30c81d5d30","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"},
+3 -1
View File
@@ -275,7 +275,9 @@ export function junitCensus(manifest: PaidRunManifest, artifacts: ReportArtifact
const kind = (E2E_KINDS[id] ?? 'rule') as EvalCaseKind;
const failed = cases.find(tc => tc.outcome === 'failed');
const caseOutcome: TrialOutcome = failed ? 'failed' : cases.some(tc => tc.outcome === 'passed') ? 'passed' : 'skipped';
const mine = records.filter((r: any) => r?.name === id || r?.case_id === id);
// A record is named by its case id or carries case_id; an E2E record named by its display label ("/qa fix loop") is
// the shard's only case's when the shard ran one case, else it stays unattributed and the trial's cost unknown.
const mine = records.filter((r: any) => r?.name === id || r?.case_id === id || (byCase.size === 1 && !r?.case_id));
const failedRecord = mine.find((r: any) => r.passed === false);
const recorded: TrialFailureClass | undefined = !failed ? undefined
: failed.failureType === 'TimeoutError' ? 'timeout' : failedRecord ? failureClassOf(failedRecord) : 'assertion';
+4
View File
@@ -291,6 +291,10 @@ export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }>
file: 'test/ship-queue-train.test.ts',
reason: 'drives bin/gstack-ship-queue against the stateful #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts; on windows-latest spawnSync(\'gh\') resolves the runner\'s gh.exe instead of the shim (windows-free-shard run 39152702056)',
},
{
file: 'test/ship-queue-train-cost.test.ts',
reason: 'drives bin/gstack-ship-queue train verdict against the same #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts; on windows-latest spawnSync(\'gh\') resolves the runner\'s gh.exe instead of the shim (windows-free-shard 2 on #3478 at d5e6a7d)',
},
{
file: 'test/ship-policy-compat.test.ts',
reason: 'spawns bin/gstack-ship-policy, bin/gstack-restamp and bin/gstack-tree-receipt (shebangs) through test/helpers/restamp-fixture.ts',
+13
View File
@@ -8,6 +8,7 @@ import * as fs from 'node:fs';
import * as path from 'node:path';
import { EXIT } from '../lib/headless-artifacts';
import { DEFAULT_POLICY, POLICY_KEYS, globToRegExp, matchesAny, validatePolicy } from '../lib/ship-policy';
import { fillDue } from '../lib/ship-queue-train';
import { BASE_POLICY, cleanup, makeFixtureRepo, runBin, type FixtureRepo } from './helpers/restamp-fixture';
const repos: FixtureRepo[] = [];
@@ -72,6 +73,18 @@ describe("gstack's own .gstack/ship-policy.json", () => {
expect(raw.train).toMatchObject({ gate_workflow: 'evals.yml', free_workflow: 'free-tests.yml', full_gate_check: 'gstack/full-gate', record_artifact: 'gstack-train-record' });
for (const wf of [raw.train.gate_workflow, raw.train.free_workflow]) expect(fs.existsSync(path.join(root, '.github/workflows', wf))).toBe(true);
});
// Value: protects=the fill rule the coordinator relies on: a train dispatches at 3 READY PRs or one hour after the first READY, whichever comes first
// (2026-10-26: 5 PRs / 3 hours left READY PRs waiting most of a day); fails_when=the live block drifts back to a larger or slower fill; why_new=the values were unpinned; seam=the policy file; no_claim=fillDue's arithmetic (ship-queue-train-logic)
test('fills a train at 3 READY PRs or one hour after the first READY, whichever comes first', () => {
const raw = JSON.parse(fs.readFileSync(path.join(import.meta.dir, '..', '.gstack', 'ship-policy.json'), 'utf8'));
expect([raw.train.max_size, raw.train.fill_hours]).toEqual([3, 1]);
const first = '2026-10-26T09:00:00.000Z';
const slots = (n: number) => Array.from({ length: n }, (_, i) => ({ pr: i + 1, ready_at: new Date(Date.parse(first) + i * 60_000).toISOString() })) as any;
expect(fillDue(slots(2), raw.train, '2026-10-26T09:59:00Z', false)).toBe('2026-10-26T10:00:00.000Z');
expect(fillDue(slots(2), raw.train, '2026-10-26T10:00:00Z', false)).toBeNull();
expect(fillDue(slots(3), raw.train, '2026-10-26T09:05:00Z', false)).toBeNull();
});
});
describe('gstack-ship-policy (bin)', () => {
+47
View File
@@ -97,6 +97,53 @@ describe('census attribution from the 2026-10-03 artifacts', () => {
});
});
describe('a case shard\'s result records are its one case\'s cost (2026-10-26 train 76db1d1f10c4: 54 of 136 trials cost-unknown, verdict $24.41 under a $32.92 run)', () => {
const shard = (dir: string, slug: string) => path.join(dir, 'paid-slice-1-a1', 'shards', slug);
const results = (dir: string, slug: string, tests: Array<Record<string, unknown>>) =>
fs.writeFileSync(path.join(shard(dir, slug), 'e2e-2026-10-26.json'), JSON.stringify({ tests, total_cost_usd: tests.reduce((sum, t) => sum + Number(t.cost_usd), 0) }));
const collector = (dir: string) => JSON.parse(fs.readFileSync(path.join(dir, 'collector-outcomes.json'), 'utf8'));
test('a record named by its display label ("/qa fix loop") belongs to the shard\'s only case, so the trial and the verdict carry its cost', () => {
const r = report('periodic', dir => results(dir, 'skill-e2e-qa-workflow--qa-fix-loop',
[{ name: '/qa fix loop', suite: 'QA skill E2E', tier: 'e2e', passed: true, duration_ms: 1000, cost_usd: 1.23 }]));
const trial = r.history.find(record => record.case === 'qa-fix-loop');
expect(trial).toMatchObject({ outcome: 'passed', cost_usd: 1.23 });
expect(trial.cost_known).toBeUndefined();
expect(collector(r.dir).verdict.costUsd).toBe(1.23);
expect(r.out).toContain('cost $1.23 known + 6 of 7 trial(s) cost unknown');
});
test('a multi-case shard whose records carry display labels keeps those trials cost-unknown instead of guessing', () => {
const r = report('periodic', dir => results(dir, 'skill-e2e-cso',
[{ name: '/cso full audit', suite: 'cso', tier: 'e2e', passed: true, duration_ms: 1000, cost_usd: 0.5 },
{ name: 'cso-infra-scope', suite: 'cso', tier: 'e2e', passed: true, duration_ms: 1000, cost_usd: 0.25 }]));
expect(r.history.find(record => record.case === 'cso-full-audit')).toMatchObject({ cost_usd: 0, cost_known: false });
expect(r.history.find(record => record.case === 'cso-infra-scope')).toMatchObject({ cost_usd: 0.25 });
expect(r.history.find(record => record.case === 'cso-infra-scope').cost_known).toBeUndefined();
expect(collector(r.dir).verdict.costUsd).toBe(0.75);
});
// Value: protects=a failed case shard's trial reads its failure class, exit reason, error and cost from the display-labelled record the shard
// wrote, not a generic 'assertion' with the JUnit message (the same attribution the cost fix made; train-gate's INFRA/VERDICT class follows failure_class);
// fails_when=failedRecord is looked up through the old name/case_id-only filter again so the record's class and error are dropped;
// existing_gap=both attribution rows above use passing records, so the failedRecord branch of the widened filter is unexercised; seam=none
test('a failed display-labelled record in a case shard supplies the trial\'s failure class, exit reason and error, not a bare assertion', () => {
const r = report('periodic', dir => {
const qa = junit(dir, 'skill-e2e-qa-workflow--qa-fix-loop');
fs.writeFileSync(qa, fs.readFileSync(qa, 'utf8').replace('failures="0"', 'failures="1"')
.replace(/(<testcase name="qa-fix-loop"[^>]*?) \/>/, '$1><failure type="AssertionError" message="qa-fix-loop: expected a fix commit" /></testcase>'));
results(dir, 'skill-e2e-qa-workflow--qa-fix-loop', [{ name: '/qa fix loop', suite: 'QA skill E2E', tier: 'e2e', passed: false, duration_ms: 1000, cost_usd: 2.5,
failure_class: 'timeout', exit_reason: 'timeout', timeout_at_turn: 7, error: 'no terminal outcome within 1800000ms\nsecond line' }]);
});
expect(r.status, r.out).toBe(1);
const trial = r.history.find(record => record.case === 'qa-fix-loop');
expect(trial).toMatchObject({ outcome: 'failed', failure_class: 'timeout', exit_reason: 'timeout', error: 'no terminal outcome within 1800000ms', cost_usd: 2.5 });
expect(trial.cost_known).toBeUndefined();
expect(r.out).toContain('✗ qa-fix-loop rule FAIL timeout at turn 7 — no terminal outcome within 1800000ms');
expect(trialRecordProblemsV1(trial)).toEqual([]);
});
});
describe('record builders attach ledger sessions and failure causes (plan 0.1/0.2)', () => {
const noOp = (dir: string) => path.join(dir, 'paid-slice-1-a1', 'shards', 'skill-e2e-plan-mode-no-op');
const failNoOp = (dir: string) => {
+49
View File
@@ -0,0 +1,49 @@
/**
* The train verdict's cost line: the known sum plus how many of the record's
* cases carry no cost, so a partial sum is never read as the run's spend.
* Train 76db1d1f10c4 (2026-10-26) printed cost=$24.41 while its slice report
* said "cost $32.92 known + 54 of 136 trial(s) cost unknown": 54 E2E records
* named by display label had no case attributed (scripts/lib/paid-report.ts).
*/
import { afterEach, describe, expect, test } from 'bun:test';
import { EXIT } from '../lib/headless-artifacts';
import { cleanup } from './helpers/restamp-fixture';
import { GREEN_GATE, PASS, TRAIN_POLICY, line, nonceOf, openPr, queue, readyReceipt, trainFixture, type Rule, type TrainFixture } from './helpers/ship-queue-train-fixture';
const fixtures: TrainFixture[] = [];
const make = () => { const f = trainFixture(TRAIN_POLICY); fixtures.push(f); return f; };
afterEach(() => { while (fixtures.length) cleanup(fixtures.pop()!.repo); });
function greenTrain(fx: TrainFixture, gate: Rule) {
const h1 = openPr(fx, '1');
const h2 = openPr(fx, '2');
readyReceipt(fx, '1', h1);
readyReceipt(fx, '2', h2);
fx.write(d => { d.rules['evals.yml'] = gate; });
return nonceOf(queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2}`, '--free-timeout', '1']).out);
}
describe('train verdict cost line', () => {
// Value: protects=a train verdict never presents a partial sum as the run's cost (train 76db1d1f10c4 on 2026-10-26: verdict $24.41, the run's slice reports $32.92 known with 54 trials unattributed);
// fails_when=the ATTEMPT line prints cost=$ without naming the cost-unknown cases, or names them when every panel has a cost; why_new=the line only had the known sum; seam=the gate record's panels; no_claim=why a trial's cost is unknown
test('the verdict prints the known cost and how many cases have none, and nothing extra when every panel is priced', () => {
const fx = make();
const unpriced = { ...GREEN_GATE, record: { ...GREEN_GATE.record, panels: [PASS('c1'), { ...PASS('c2'), cost_usd: null }, { ...PASS('j1', 'judge', 0.1), cost_usd: null }] } };
const nonce = greenTrain(fx, unpriced);
const v = queue(fx, ['train', 'verdict', '--train', nonce]);
expect([v.status, line(v.out, 'ATTEMPT:')]).toEqual([EXIT.ok, 'ATTEMPT: a1 gate run 101 GREEN cost=$0.70 known + 2 of 3 case(s) cost unknown (rule 2/2, judge 1/1)']);
const all = make();
const priced = greenTrain(all, GREEN_GATE);
expect(line(queue(all, ['train', 'verdict', '--train', priced]).out, 'ATTEMPT:')).toBe('ATTEMPT: a1 gate run 101 GREEN cost=$1.50 (rule 2/2, judge 1/1)');
}, 120_000);
// Value: protects=a train whose every case is priced at $0.00 (reused or free trials) still reports its cost as known;
// fails_when=the null guard on cost_usd becomes a falsy check, so a zero sum prints nothing and reads like an unpriced run;
// existing_gap=the two rows above price every panel above zero or leave some unpriced, never a known zero; seam=none
test('a train priced at zero prints cost=$0.00 rather than no cost at all', () => {
const fx = make();
const free = { ...GREEN_GATE, record: { ...GREEN_GATE.record, panels: [PASS('c1', 'rule', 0), PASS('c2', 'rule', 0), PASS('j1', 'judge', 0)] } };
const nonce = greenTrain(fx, free);
expect(line(queue(fx, ['train', 'verdict', '--train', nonce]).out, 'ATTEMPT:')).toBe('ATTEMPT: a1 gate run 101 GREEN cost=$0.00 (rule 2/2, judge 1/1)');
}, 60_000);
});