Files
gstack/test/helpers/pty/session.ts
Garry Tanandcapy-ai[bot] daf661374f v1.93.10.0 wave(docs): #3590 #3591 #3819 #3618 #3824 #3828 (#3847)
* Attribute every paid trial's cost: bind label-named records by testName, price aborted SDK sessions from streamed usage, model-free cases cost $0

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: case cost adds streamed sessions to record sums and keeps record cost_known

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: the paid report headline counts ledger-only session cost once

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: a cost-only record never sets the trial model

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: recordE2E writes cost_case_id only for a registered label-named case

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: CHANGELOG for paid-report cost attribution (Unreleased)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: the headline adds only streamed session cost; cache writes without a 5m key

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: how the paid report prices a trial and what stays cost-unknown

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Price PTY sessions from their transcripts: pin a session id, sum the session's and its subagents' usage into the ledger's streamed_cost_usd

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: transcript pricing never throws from a PTY ledger writer

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: pin transcript-cost.ts as non-behavioral at HARNESS_VERSION 6

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: recognise --session-id= and --resume= forms

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: CHANGELOG for PTY transcript pricing (Unreleased)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: PTY sessions priced from their transcripts

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: price PTY transcripts without adding a launch argument; native AUQ capture opts out

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: harness notes and CHANGELOG match the transcript claim

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): gstack-flaky run reruns a failed JUnit test once and ledgers the flake (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): stream output, keep unreported runner failures red, fail-closed duplicates and nonzero reruns

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes

gstack-flaky never greens an incomplete run: a targeted rerun needs the
runner's summary proving the first run complete, otherwise (and always for
generic runners) the rerun repeats the whole command. A rerun clears a test
only when it exits 0 and the test passes once on its first attempt. Killed
runners exit 3 (FLAKY_COMMAND_KILLED) and signals are forwarded. The first
report is restored at --junit with reruns kept beside it; ids are redacted at
every output; the ledger gets its own GSTACK_FLAKE_LEDGER_JUNIT; fixtures are
registered in FREE_FIXTURES.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: latch cancellation and check whole-command rerun reports

A cancel forwarded to a runner that traps the signal now counts as killed for
every later attempt, so a cancelled job is never rerun or greened. A
whole-command rerun clears nothing when its report has a failed test, misses a
first-run test, or its runner counted failures outside the report.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: whole-command rerun when coverage or runner-flaky tests can fail the run

A coverage threshold or a fail-on-flaky policy fails a run for a reason a
narrowed rerun cannot reproduce, so either forces a whole-command rerun.
--junit refuses an existing file that is not a JUnit report instead of
deleting it.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): rerun the whole command by default; targeted rerun is opt-in

A rerun counts only on the runner's own full green verdict: exit 0, no failed
or missing test, nothing counted outside the report. That covers every way a
run can fail, which a narrowed rerun cannot (a coverage threshold with an lcov
reporter, a fail-on-flaky policy, a test never reached). --rerun-scope test
reruns the failed test alone, with its preconditions documented and the
complete-run checks kept as a safety net. An unreadable existing --junit file
is a usage error, not a crash.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: gstack-flaky CHANGELOG entry and standalone-binaries row (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): quarantine registry for repeat flakes (GSTA-158 P2)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): bound renewal dates; match non-ASCII paths in check --base

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): reword the FLAKY_QUARANTINE_BASE hint so the egress scanner does not read it as a network op

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(agent-sdk): every SDK caller passes the claude_code preset or names why not

The SDK's default system prompt is a bare stub that drops the line telling the
model to run /skill through the Skill tool. runAgentSdkTest now refuses a
missing, empty or bare-string prompt; non-preset callers name a reason; a free
tripwire scans test/ and scripts/ for the same mistake.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: record the SDK prompt guard as a non-behavioral harness change and baseline its tripwire

harness-version: runAgentSdkTest validates systemPrompt before starting; every
live caller already passed the claude_code preset, so no verdict can change.
test-of-test ratchet: the tripwire's subject is the eval harness by design.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: name the preset literal callers pass and the unnamed-prompt refusal

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(eval-plan): frontier models under test (R54)

Haiku and Sonnet are not models under test. dry-run refuses a plan whose every
arm is Haiku or Sonnet unless --tier-question names what it asks about that
tier, warns on a mix, and re-costs the plan on Opus 5.5 and GPT-6.1 Sol.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling

The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the
exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining
growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458,
the capture helper's value); other skills' ceilings are left as they were.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(eval-plan): pilot start keeps the R54 refusal; --tier-question rejects a flag

Review pass 1 (full depth) found the frontier refusal could be skipped by
starting a pilot on Haiku or Sonnet arms behind a dry run made on other arms,
or behind a dry-run.json written before the check existed. startPilot now
refuses a pilot whose every arm is Haiku or Sonnet unless its dry run recorded
a tier question. dry-run --tier-question no longer takes the next flag as its
answer. The testing doc now says the frontier re-cost is printed for
comparison with the brief cap, not enforced.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(support-tiers): Windows CI in the present tense now that windows-smoke is required (#3806)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: empty commit for a clean check set at the same tree (cancelled orphan free-suite check at 97487df)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(skills): /eval-plan entry names the R54 frontier check

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-02 17:44:00 +00:00

112 lines
4.9 KiB
TypeScript

/**
* The one PTY session loop (runPtySession) and its launch seam (PtyDriver).
* Owns launch, the poll/deadline loop, failure capture and close/cleanup
* ordering; each runner in pty/runners/ supplies its policy as a PtySessionPlan
* (start, poll, tick, timeout, capture hooks). New runner: build a plan and
* `return runPtySession(plan)`; tests pass the fake driver from fake-session.ts.
* Size ratchet (c) binds this module. Moved from the three runner loops of the
* former claude-pty-runner.ts. Tests import through claude-pty-runner.ts.
*/
import { launchClaudePty, type ClaudePtyOptions, type ClaudePtySession } from './launch';
import { screenPanelEnd } from './screen';
import { appendSessionLedger, sessionKey, type SessionEnd } from '../session-ledger';
import { pricedTranscript } from './transcript-cost';
/**
* The runners' launch seam: how a session starts, and the clock the runner
* loop reads. Omitted in production (real launcher, Date.now,
* performance.now, Bun.sleep); tests pass the fake driver from
* test/helpers/pty/fake-session.ts.
*/
export interface PtyDriver {
launch(opts: ClaudePtyOptions): Promise<ClaudePtySession>;
/** Wall clock, Date.now() semantics. */
now(): number;
/** Monotonic clock, performance.now() semantics. */
monotonic(): number;
sleep(ms: number): Promise<void>;
}
export const realPtyDriver: PtyDriver = {
launch: opts => launchClaudePty(opts),
now: () => Date.now(),
monotonic: () => performance.now(),
sleep: ms => Bun.sleep(ms),
};
/** The session's end panel (screenPanelEnd), if the session can show its text; diagnostics never throw. */
export function safePanelEnd(session: Pick<ClaudePtySession, 'visibleText'>): ReturnType<typeof screenPanelEnd> {
try { return screenPanelEnd(session.visibleText()); } catch { return null; }
}
/** One poll step: finish with a result, keep polling, or end the loop as a timeout. */
export type PtyStep<R> = { done: R } | 'continue' | 'break';
export interface PtySessionPlan<R> {
driver: PtyDriver;
launch: ClaudePtyOptions;
/** The observation budget the runner armed (session ledger); defaults to the launch wall. */
budgetMs?: number;
/** Owned fixture cleanup; runs after close, and when launch itself fails. */
cleanup?: () => void;
/** Boot and command submission. A returned result ends the run before polling. */
start(session: ClaudePtySession): Promise<R | undefined>;
/** Wait for the next poll. False means the deadline passed: the run times out. */
poll(session: ClaudePtySession): Promise<boolean>;
tick(session: ClaudePtySession): Promise<PtyStep<R>>;
timeout(session: ClaudePtySession): R;
/** Snapshot a thrown run before close; the original error is rethrown. */
onError?(session: ClaudePtySession, error: unknown): void;
/** Always runs before close. A throw here fails an otherwise successful run. */
beforeClose?(session: ClaudePtySession, failure: { error: unknown } | undefined): void;
/** Close failed after a successful run; the close error is rethrown. */
onCloseError?(session: ClaudePtySession, error: unknown): void;
}
/**
* launch → start → (poll → tick)* → timeout, with one failure contract: a
* run's own error wins over capture and close errors, close always runs, and
* owned cleanup runs last.
*/
export async function runPtySession<R>(plan: PtySessionPlan<R>): Promise<R> {
const started = plan.driver.monotonic();
const startedAt = new Date(plan.driver.now()).toISOString();
let end: SessionEnd = 'error';
let session: ClaudePtySession;
try {
session = await plan.driver.launch({ ...plan.launch, sessionLedger: false });
} catch (error) {
plan.cleanup?.();
throw error;
}
let failure: { error: unknown } | undefined;
try {
const ended = (result: R) => { end = (result as { outcome?: unknown })?.outcome === 'timeout' ? 'observer_timeout' : 'completed'; return result; };
const early = await plan.start(session);
if (early !== undefined) return ended(early);
while (await plan.poll(session)) {
const step = await plan.tick(session);
if (step === 'break') break;
if (step !== 'continue') return ended(step.done);
}
end = 'observer_timeout';
return plan.timeout(session);
} catch (error) {
failure = { error };
plan.onError?.(session, error);
throw error;
} finally {
appendSessionLedger({ key: sessionKey('pty', undefined), runner: 'pty', started_at: startedAt,
budget_ms: plan.budgetMs ?? plan.launch.timeoutMs ?? 240_000, elapsed_ms: plan.driver.monotonic() - started,
end, ...safePanelEnd(session), billed: false, ...pricedTranscript(session) });
try {
try { plan.beforeClose?.(session, failure); }
catch (error) { if (!failure) { failure = { error }; throw error; } }
finally {
try { await session.close(); }
catch (error) { if (!failure) { plan.onCloseError?.(session, error); throw error; } }
}
} finally { plan.cleanup?.(); }
}
}