Files
gstack/test/helpers/pty/transcript-cost.ts
Garry Tanandcapy-ai[bot] daf661374f v1.93.10.0 wave(docs): #3590 #3591 #3819 #3618 #3824 #3828 (#3847)
* Attribute every paid trial's cost: bind label-named records by testName, price aborted SDK sessions from streamed usage, model-free cases cost $0

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: case cost adds streamed sessions to record sums and keeps record cost_known

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: the paid report headline counts ledger-only session cost once

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: a cost-only record never sets the trial model

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: recordE2E writes cost_case_id only for a registered label-named case

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: CHANGELOG for paid-report cost attribution (Unreleased)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: the headline adds only streamed session cost; cache writes without a 5m key

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: how the paid report prices a trial and what stays cost-unknown

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Price PTY sessions from their transcripts: pin a session id, sum the session's and its subagents' usage into the ledger's streamed_cost_usd

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: transcript pricing never throws from a PTY ledger writer

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: pin transcript-cost.ts as non-behavioral at HARNESS_VERSION 6

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: recognise --session-id= and --resume= forms

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: CHANGELOG for PTY transcript pricing (Unreleased)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: PTY sessions priced from their transcripts

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: price PTY transcripts without adding a launch argument; native AUQ capture opts out

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: harness notes and CHANGELOG match the transcript claim

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): gstack-flaky run reruns a failed JUnit test once and ledgers the flake (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): stream output, keep unreported runner failures red, fail-closed duplicates and nonzero reruns

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes

gstack-flaky never greens an incomplete run: a targeted rerun needs the
runner's summary proving the first run complete, otherwise (and always for
generic runners) the rerun repeats the whole command. A rerun clears a test
only when it exits 0 and the test passes once on its first attempt. Killed
runners exit 3 (FLAKY_COMMAND_KILLED) and signals are forwarded. The first
report is restored at --junit with reruns kept beside it; ids are redacted at
every output; the ledger gets its own GSTACK_FLAKE_LEDGER_JUNIT; fixtures are
registered in FREE_FIXTURES.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: latch cancellation and check whole-command rerun reports

A cancel forwarded to a runner that traps the signal now counts as killed for
every later attempt, so a cancelled job is never rerun or greened. A
whole-command rerun clears nothing when its report has a failed test, misses a
first-run test, or its runner counted failures outside the report.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: whole-command rerun when coverage or runner-flaky tests can fail the run

A coverage threshold or a fail-on-flaky policy fails a run for a reason a
narrowed rerun cannot reproduce, so either forces a whole-command rerun.
--junit refuses an existing file that is not a JUnit report instead of
deleting it.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): rerun the whole command by default; targeted rerun is opt-in

A rerun counts only on the runner's own full green verdict: exit 0, no failed
or missing test, nothing counted outside the report. That covers every way a
run can fail, which a narrowed rerun cannot (a coverage threshold with an lcov
reporter, a fail-on-flaky policy, a test never reached). --rerun-scope test
reruns the failed test alone, with its preconditions documented and the
complete-run checks kept as a safety net. An unreadable existing --junit file
is a usage error, not a crash.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: gstack-flaky CHANGELOG entry and standalone-binaries row (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): quarantine registry for repeat flakes (GSTA-158 P2)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): bound renewal dates; match non-ASCII paths in check --base

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): reword the FLAKY_QUARANTINE_BASE hint so the egress scanner does not read it as a network op

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(agent-sdk): every SDK caller passes the claude_code preset or names why not

The SDK's default system prompt is a bare stub that drops the line telling the
model to run /skill through the Skill tool. runAgentSdkTest now refuses a
missing, empty or bare-string prompt; non-preset callers name a reason; a free
tripwire scans test/ and scripts/ for the same mistake.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: record the SDK prompt guard as a non-behavioral harness change and baseline its tripwire

harness-version: runAgentSdkTest validates systemPrompt before starting; every
live caller already passed the claude_code preset, so no verdict can change.
test-of-test ratchet: the tripwire's subject is the eval harness by design.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: name the preset literal callers pass and the unnamed-prompt refusal

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(eval-plan): frontier models under test (R54)

Haiku and Sonnet are not models under test. dry-run refuses a plan whose every
arm is Haiku or Sonnet unless --tier-question names what it asks about that
tier, warns on a mix, and re-costs the plan on Opus 5.5 and GPT-6.1 Sol.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling

The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the
exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining
growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458,
the capture helper's value); other skills' ceilings are left as they were.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(eval-plan): pilot start keeps the R54 refusal; --tier-question rejects a flag

Review pass 1 (full depth) found the frontier refusal could be skipped by
starting a pilot on Haiku or Sonnet arms behind a dry run made on other arms,
or behind a dry-run.json written before the check existed. startPilot now
refuses a pilot whose every arm is Haiku or Sonnet unless its dry run recorded
a tier question. dry-run --tier-question no longer takes the next flag as its
answer. The testing doc now says the frontier re-cost is printed for
comparison with the brief cap, not enforced.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(support-tiers): Windows CI in the present tense now that windows-smoke is required (#3806)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: empty commit for a clean check set at the same tree (cancelled orphan free-suite check at 97487df)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(skills): /eval-plan entry names the R54 frontier check

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-02 17:44:00 +00:00

75 lines
3.4 KiB
TypeScript

import * as fs from 'node:fs';
import * as path from 'node:path';
import { streamedUsageCostUsd } from '../../../lib/pricing';
/** The session ledger's cost field for a PTY session: present only when its transcript prices. */
export function pricedTranscript(session: { transcriptCostUsd?(): number | undefined } | undefined): { streamed_cost_usd?: number } {
const cost = session?.transcriptCostUsd?.();
return cost === undefined ? {} : { streamed_cost_usd: cost };
}
/** Top-level session transcripts (`projects/<dir>/<id>.jsonl`) under a config dir; empty when unreadable. */
export function transcriptFiles(configDir: string | undefined): Set<string> {
if (!configDir) return new Set();
try {
const projects = path.join(configDir, 'projects');
return new Set(fs.readdirSync(projects, { withFileTypes: true }).filter(dir => dir.isDirectory())
.flatMap(dir => fs.readdirSync(path.join(projects, dir.name)).filter(f => f.endsWith('.jsonl')).map(f => path.join(projects, dir.name, f))));
} catch {
return new Set();
}
}
/**
* The session id of the one transcript that appeared since `before` and was written for `cwd`. Concurrent PTY tests in
* one process share a hermetic config dir, so a launch claims only a new transcript for its own cwd; none or several
* (two sessions in one cwd) is undefined, which leaves the cost unknown.
*/
export function newTranscriptSessionId(configDir: string | undefined, before: ReadonlySet<string>, cwd: string): string | undefined {
try {
const cwds = new Set([cwd, fs.realpathSync(cwd)]);
const mine = [...transcriptFiles(configDir)].filter(file => !before.has(file) && fs.readFileSync(file, 'utf8').split('\n').some(line => {
try {
return cwds.has(JSON.parse(line)?.cwd);
} catch {
return false;
}
}));
return mine.length === 1 ? path.basename(mine[0], '.jsonl') : undefined;
} catch {
return undefined;
}
}
/**
* Price one PTY session from the transcripts Claude Code wrote under its owned config dir:
* `projects/<dir>/<sessionId>.jsonl` (inline sidechains included) and `projects/<dir>/<sessionId>/subagents/*.jsonl`,
* rows deduplicated by message id. Undefined (cost-unknown, never $0) without a config dir, session id, transcript or
* usage, when a message's model has no price row, or when any read fails: it runs in ledger writers before close and
* cleanup, so it never throws.
*/
export function ptyTranscriptCostUsd(configDir: string | null | undefined, sessionId: string | undefined): number | undefined {
if (!configDir || !sessionId) return undefined;
try {
const projects = path.join(configDir, 'projects');
const files: string[] = [];
for (const dir of fs.readdirSync(projects, { withFileTypes: true })) {
if (!dir.isDirectory()) continue;
const main = path.join(projects, dir.name, `${sessionId}.jsonl`);
if (fs.existsSync(main)) files.push(main);
const subagents = path.join(projects, dir.name, sessionId, 'subagents');
if (fs.existsSync(subagents)) files.push(...fs.readdirSync(subagents).filter(f => f.endsWith('.jsonl')).map(f => path.join(subagents, f)));
}
const rows = files.flatMap(file => fs.readFileSync(file, 'utf8').split('\n').flatMap(line => {
try {
return [JSON.parse(line)];
} catch {
return [];
}
}));
return streamedUsageCostUsd(rows);
} catch {
return undefined;
}
}