Files
gstack/test/helpers/claude-pty-runner.launch.unit.test.ts
Garry Tanandcapy-ai[bot] daf661374f v1.93.10.0 wave(docs): #3590 #3591 #3819 #3618 #3824 #3828 (#3847)
* Attribute every paid trial's cost: bind label-named records by testName, price aborted SDK sessions from streamed usage, model-free cases cost $0

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: case cost adds streamed sessions to record sums and keeps record cost_known

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: the paid report headline counts ledger-only session cost once

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: a cost-only record never sets the trial model

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: recordE2E writes cost_case_id only for a registered label-named case

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: CHANGELOG for paid-report cost attribution (Unreleased)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: the headline adds only streamed session cost; cache writes without a 5m key

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: how the paid report prices a trial and what stays cost-unknown

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Price PTY sessions from their transcripts: pin a session id, sum the session's and its subagents' usage into the ledger's streamed_cost_usd

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: transcript pricing never throws from a PTY ledger writer

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: pin transcript-cost.ts as non-behavioral at HARNESS_VERSION 6

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: recognise --session-id= and --resume= forms

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: CHANGELOG for PTY transcript pricing (Unreleased)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: PTY sessions priced from their transcripts

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: price PTY transcripts without adding a launch argument; native AUQ capture opts out

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: harness notes and CHANGELOG match the transcript claim

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): gstack-flaky run reruns a failed JUnit test once and ledgers the flake (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): stream output, keep unreported runner failures red, fail-closed duplicates and nonzero reruns

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes

gstack-flaky never greens an incomplete run: a targeted rerun needs the
runner's summary proving the first run complete, otherwise (and always for
generic runners) the rerun repeats the whole command. A rerun clears a test
only when it exits 0 and the test passes once on its first attempt. Killed
runners exit 3 (FLAKY_COMMAND_KILLED) and signals are forwarded. The first
report is restored at --junit with reruns kept beside it; ids are redacted at
every output; the ledger gets its own GSTACK_FLAKE_LEDGER_JUNIT; fixtures are
registered in FREE_FIXTURES.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: latch cancellation and check whole-command rerun reports

A cancel forwarded to a runner that traps the signal now counts as killed for
every later attempt, so a cancelled job is never rerun or greened. A
whole-command rerun clears nothing when its report has a failed test, misses a
first-run test, or its runner counted failures outside the report.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: whole-command rerun when coverage or runner-flaky tests can fail the run

A coverage threshold or a fail-on-flaky policy fails a run for a reason a
narrowed rerun cannot reproduce, so either forces a whole-command rerun.
--junit refuses an existing file that is not a JUnit report instead of
deleting it.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): rerun the whole command by default; targeted rerun is opt-in

A rerun counts only on the runner's own full green verdict: exit 0, no failed
or missing test, nothing counted outside the report. That covers every way a
run can fail, which a narrowed rerun cannot (a coverage threshold with an lcov
reporter, a fail-on-flaky policy, a test never reached). --rerun-scope test
reruns the failed test alone, with its preconditions documented and the
complete-run checks kept as a safety net. An unreadable existing --junit file
is a usage error, not a crash.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: gstack-flaky CHANGELOG entry and standalone-binaries row (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): quarantine registry for repeat flakes (GSTA-158 P2)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): bound renewal dates; match non-ASCII paths in check --base

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): reword the FLAKY_QUARANTINE_BASE hint so the egress scanner does not read it as a network op

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(agent-sdk): every SDK caller passes the claude_code preset or names why not

The SDK's default system prompt is a bare stub that drops the line telling the
model to run /skill through the Skill tool. runAgentSdkTest now refuses a
missing, empty or bare-string prompt; non-preset callers name a reason; a free
tripwire scans test/ and scripts/ for the same mistake.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: record the SDK prompt guard as a non-behavioral harness change and baseline its tripwire

harness-version: runAgentSdkTest validates systemPrompt before starting; every
live caller already passed the claude_code preset, so no verdict can change.
test-of-test ratchet: the tripwire's subject is the eval harness by design.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: name the preset literal callers pass and the unnamed-prompt refusal

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(eval-plan): frontier models under test (R54)

Haiku and Sonnet are not models under test. dry-run refuses a plan whose every
arm is Haiku or Sonnet unless --tier-question names what it asks about that
tier, warns on a mix, and re-costs the plan on Opus 5.5 and GPT-6.1 Sol.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling

The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the
exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining
growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458,
the capture helper's value); other skills' ceilings are left as they were.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(eval-plan): pilot start keeps the R54 refusal; --tier-question rejects a flag

Review pass 1 (full depth) found the frontier refusal could be skipped by
starting a pilot on Haiku or Sonnet arms behind a dry run made on other arms,
or behind a dry-run.json written before the check existed. startPilot now
refuses a pilot whose every arm is Haiku or Sonnet unless its dry run recorded
a tier question. dry-run --tier-question no longer takes the next flag as its
answer. The testing doc now says the frontier re-cost is printed for
comparison with the brief cap, not enforced.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(support-tiers): Windows CI in the present tense now that windows-smoke is required (#3806)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: empty commit for a clean check set at the same tree (cancelled orphan free-suite check at 97487df)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(skills): /eval-plan entry names the R54 frontier check

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-02 17:44:00 +00:00

119 lines
6.8 KiB
TypeScript

/**
* Deterministic unit tests for the PTY launcher (test/helpers/pty/launch.ts).
* Split along the W4 module seams from the former claude-pty-runner.unit.test.ts;
* tests import the public barrel, test/helpers/claude-pty-runner.ts.
*/
import { describe, test, expect } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { spawnSync } from 'node:child_process';
import { pathToFileURL } from 'node:url';
import {
type ClaudePtyOptions,
} from './claude-pty-runner';
describe('runPlanSkillObservation env passthrough surface', () => {
test('ClaudePtyOptions exposes env: Record<string, string>', () => {
// Type-level guard: this file would fail to compile if the env field
// were removed or its shape regressed. The actual env merge happens in
// launchClaudePty's spawn call (`env: { ...process.env, ...opts.env }`),
// so a regression where `env: opts.env` gets dropped from the
// runPlanSkillObservation -> launchClaudePty handoff is only caught by
// the live PTY test, not here.
const opts: ClaudePtyOptions = {
env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' },
};
expect(opts.env).toEqual({ QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' });
});
});
describe('launchClaudePty model pin', () => {
// Behavioral: a fake CLI records the argv its real PTY launch received.
// Chain mirrors session-runner.ts: opts.model -> EVALS_MODEL ->
// resolveEvalModel('capture'); --model precedes extraArgs so a per-test
// --model wins (last flag wins). Per-runner forwarding of opts.model is
// asserted through the fake driver in claude-pty-runner.runners.unit.test.ts.
test('ClaudePtyOptions exposes model?: string', () => {
const opts: ClaudePtyOptions = { model: 'claude-sonnet-4-6' };
expect(opts.model).toBe('claude-sonnet-4-6');
});
test.skipIf(process.platform === 'win32')('spawn args pin --model from the fallback chain, before extraArgs, with hermetic MCP gating', () => {
const dir = mkdtempSync(join(tmpdir(), 'pty-model-pin-'));
try {
const fake = join(dir, 'fake-claude');
writeFileSync(fake, `#!${process.execPath}\nprocess.stdout.write('ARGV=' + JSON.stringify(process.argv.slice(2)) + '\\n');\nsetInterval(() => {}, 1000);\n`, { mode: 0o755 });
const runner = pathToFileURL(join(import.meta.dir, 'claude-pty-runner.ts')).href;
const worker = join(dir, 'worker.ts');
writeFileSync(worker, `import { launchClaudePty } from ${JSON.stringify(runner)};
import { resolveEvalModel } from ${JSON.stringify(pathToFileURL(join(import.meta.dir, '../../lib/eval-model.ts')).href)};
const argv = async (opts, evalsModel) => {
if (evalsModel === undefined) delete process.env.EVALS_MODEL; else process.env.EVALS_MODEL = evalsModel;
const session = await launchClaudePty({ cwd: ${JSON.stringify(dir)}, timeoutMs: 5000, ...opts });
try { await session.waitFor(/ARGV=\\[.*\\]/, { timeoutMs: 4000, pollMs: 20 }); return JSON.parse(/ARGV=(\\[.*\\])/.exec(session.visibleText())[1]); }
finally { await session.close(); }
};
process.stdout.write(JSON.stringify({ capture: resolveEvalModel('capture'),
fallback: await argv({}, undefined), env: await argv({}, 'env-model'), explicit: await argv({ model: 'opts-model' }, 'env-model'),
extra: await argv({ extraArgs: ['--model', 'override'] }, undefined) }));
`);
const run = (hermetic: string) => {
const result = spawnSync(process.execPath, [worker], { cwd: join(import.meta.dir, '../..'), encoding: 'utf8', timeout: 30_000,
env: { ...process.env, BROWSE_TERMINAL_BINARY: fake, EVALS_HERMETIC: hermetic } });
expect(result.status, result.stderr).toBe(0);
return JSON.parse(result.stdout.trim().split('\n').at(-1)!);
};
const hermetic = run('1');
const model = (args: string[]) => args[args.indexOf('--model') + 1];
expect(model(hermetic.fallback)).toBe(hermetic.capture);
expect(model(hermetic.env)).toBe('env-model');
expect(model(hermetic.explicit)).toBe('opts-model');
expect(hermetic.extra.indexOf('--model')).toBeLessThan(hermetic.extra.lastIndexOf('--model'));
expect(hermetic.extra.slice(-2)).toEqual(['--model', 'override']);
expect(hermetic.fallback).toContain('--strict-mcp-config');
expect(run('0').fallback).not.toContain('--strict-mcp-config');
} finally { rmSync(dir, { recursive: true, force: true }); }
}, 60_000);
});
describe('launchClaudePty transcript pricing (GSTA-150 PR 3)', () => {
test.skipIf(process.platform === 'win32')('prices the transcript this launch created for its cwd, without adding a launch argument; opting out prices nothing', () => {
const dir = mkdtempSync(join(tmpdir(), 'pty-transcript-cost-'));
try {
const fake = join(dir, 'fake-claude');
writeFileSync(fake, `#!${process.execPath}
const fs = require('node:fs'), path = require('node:path'), crypto = require('node:crypto');
const project = path.join(process.env.CLAUDE_CONFIG_DIR, 'projects', 'fake-project');
fs.mkdirSync(project, { recursive: true });
fs.writeFileSync(path.join(project, crypto.randomUUID() + '.jsonl'), JSON.stringify({ type: 'assistant', cwd: process.cwd(),
message: { id: 'msg_' + crypto.randomUUID(), model: 'claude-opus-5-5', usage: { input_tokens: 0, output_tokens: 1000 } } }) + '\\n');
process.stdout.write('ARGV=' + JSON.stringify(process.argv.slice(2)) + '\\n');
setInterval(() => {}, 1000);
`, { mode: 0o755 });
const runner = pathToFileURL(join(import.meta.dir, 'claude-pty-runner.ts')).href;
const worker = join(dir, 'worker.ts');
writeFileSync(worker, `import { launchClaudePty } from ${JSON.stringify(runner)};
import * as fs from 'node:fs';
const run = async (opts) => {
const cwd = fs.mkdtempSync(${JSON.stringify(join(dir, 'cwd-'))});
const session = await launchClaudePty({ cwd, timeoutMs: 5000, ...opts });
try {
await session.waitFor(/ARGV=\\[.*\\]/, { timeoutMs: 4000, pollMs: 20 });
return { argv: JSON.parse(/ARGV=(\\[.*\\])/.exec(session.visibleText())[1]), cost: session.transcriptCostUsd?.() ?? null };
} finally { await session.close(); }
};
const [priced, optedOut] = await Promise.all([run({}), run({ priceTranscript: false })]);
process.stdout.write(JSON.stringify({ priced, optedOut }));
`);
const result = spawnSync(process.execPath, [worker], { cwd: join(import.meta.dir, '../..'), encoding: 'utf8', timeout: 30_000,
env: { ...process.env, BROWSE_TERMINAL_BINARY: fake, EVALS_HERMETIC: '1' } });
expect(result.status, result.stderr).toBe(0);
const { priced, optedOut } = JSON.parse(result.stdout.trim().split('\n').at(-1)!);
expect(priced.cost).toBeCloseTo(0.02, 6);
expect(optedOut.cost).toBeNull();
expect(priced.argv).not.toContain('--session-id');
} finally { rmSync(dir, { recursive: true, force: true }); }
}, 60_000);
});