mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-02 23:14:33 +01:00
* Attribute every paid trial's cost: bind label-named records by testName, price aborted SDK sessions from streamed usage, model-free cases cost $0
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: case cost adds streamed sessions to record sums and keeps record cost_known
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: the paid report headline counts ledger-only session cost once
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: a cost-only record never sets the trial model
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: recordE2E writes cost_case_id only for a registered label-named case
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: CHANGELOG for paid-report cost attribution (Unreleased)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: the headline adds only streamed session cost; cache writes without a 5m key
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: how the paid report prices a trial and what stays cost-unknown
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Price PTY sessions from their transcripts: pin a session id, sum the session's and its subagents' usage into the ledger's streamed_cost_usd
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: transcript pricing never throws from a PTY ledger writer
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore: pin transcript-cost.ts as non-behavioral at HARNESS_VERSION 6
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: recognise --session-id= and --resume= forms
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: CHANGELOG for PTY transcript pricing (Unreleased)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: PTY sessions priced from their transcripts
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: price PTY transcripts without adding a launch argument; native AUQ capture opts out
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: harness notes and CHANGELOG match the transcript claim
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): gstack-flaky run reruns a failed JUnit test once and ledgers the flake (GSTA-158)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): stream output, keep unreported runner failures red, fail-closed duplicates and nonzero reruns
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: pre-landing review fixes
gstack-flaky never greens an incomplete run: a targeted rerun needs the
runner's summary proving the first run complete, otherwise (and always for
generic runners) the rerun repeats the whole command. A rerun clears a test
only when it exits 0 and the test passes once on its first attempt. Killed
runners exit 3 (FLAKY_COMMAND_KILLED) and signals are forwarded. The first
report is restored at --junit with reruns kept beside it; ids are redacted at
every output; the ledger gets its own GSTACK_FLAKE_LEDGER_JUNIT; fixtures are
registered in FREE_FIXTURES.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: latch cancellation and check whole-command rerun reports
A cancel forwarded to a runner that traps the signal now counts as killed for
every later attempt, so a cancelled job is never rerun or greened. A
whole-command rerun clears nothing when its report has a failed test, misses a
first-run test, or its runner counted failures outside the report.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: whole-command rerun when coverage or runner-flaky tests can fail the run
A coverage threshold or a fail-on-flaky policy fails a run for a reason a
narrowed rerun cannot reproduce, so either forces a whole-command rerun.
--junit refuses an existing file that is not a JUnit report instead of
deleting it.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): rerun the whole command by default; targeted rerun is opt-in
A rerun counts only on the runner's own full green verdict: exit 0, no failed
or missing test, nothing counted outside the report. That covers every way a
run can fail, which a narrowed rerun cannot (a coverage threshold with an lcov
reporter, a fail-on-flaky policy, a test never reached). --rerun-scope test
reruns the failed test alone, with its preconditions documented and the
complete-run checks kept as a safety net. An unreadable existing --junit file
is a usage error, not a crash.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: gstack-flaky CHANGELOG entry and standalone-binaries row (GSTA-158)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): quarantine registry for repeat flakes (GSTA-158 P2)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): bound renewal dates; match non-ASCII paths in check --base
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): reword the FLAKY_QUARANTINE_BASE hint so the egress scanner does not read it as a network op
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(agent-sdk): every SDK caller passes the claude_code preset or names why not
The SDK's default system prompt is a bare stub that drops the line telling the
model to run /skill through the Skill tool. runAgentSdkTest now refuses a
missing, empty or bare-string prompt; non-preset callers name a reason; a free
tripwire scans test/ and scripts/ for the same mistake.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: record the SDK prompt guard as a non-behavioral harness change and baseline its tripwire
harness-version: runAgentSdkTest validates systemPrompt before starting; every
live caller already passed the claude_code preset, so no verdict can change.
test-of-test ratchet: the tripwire's subject is the eval harness by design.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: name the preset literal callers pass and the unnamed-prompt refusal
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(eval-plan): frontier models under test (R54)
Haiku and Sonnet are not models under test. dry-run refuses a plan whose every
arm is Haiku or Sonnet unless --tier-question names what it asks about that
tier, warns on a mix, and re-costs the plan on Opus 5.5 and GPT-6.1 Sol.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling
The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the
exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining
growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458,
the capture helper's value); other skills' ceilings are left as they were.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(eval-plan): pilot start keeps the R54 refusal; --tier-question rejects a flag
Review pass 1 (full depth) found the frontier refusal could be skipped by
starting a pilot on Haiku or Sonnet arms behind a dry run made on other arms,
or behind a dry-run.json written before the check existed. startPilot now
refuses a pilot whose every arm is Haiku or Sonnet unless its dry run recorded
a tier question. dry-run --tier-question no longer takes the next flag as its
answer. The testing doc now says the frontier re-cost is printed for
comparison with the brief cap, not enforced.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(support-tiers): Windows CI in the present tense now that windows-smoke is required (#3806)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore: empty commit for a clean check set at the same tree (cancelled orphan free-suite check at 97487df)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(skills): /eval-plan entry names the R54 frontier check
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
119 lines
6.8 KiB
TypeScript
119 lines
6.8 KiB
TypeScript
/**
|
|
* Deterministic unit tests for the PTY launcher (test/helpers/pty/launch.ts).
|
|
* Split along the W4 module seams from the former claude-pty-runner.unit.test.ts;
|
|
* tests import the public barrel, test/helpers/claude-pty-runner.ts.
|
|
*/
|
|
import { describe, test, expect } from 'bun:test';
|
|
import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
|
|
import { tmpdir } from 'node:os';
|
|
import { join } from 'node:path';
|
|
import { spawnSync } from 'node:child_process';
|
|
import { pathToFileURL } from 'node:url';
|
|
import {
|
|
type ClaudePtyOptions,
|
|
} from './claude-pty-runner';
|
|
|
|
describe('runPlanSkillObservation env passthrough surface', () => {
|
|
test('ClaudePtyOptions exposes env: Record<string, string>', () => {
|
|
// Type-level guard: this file would fail to compile if the env field
|
|
// were removed or its shape regressed. The actual env merge happens in
|
|
// launchClaudePty's spawn call (`env: { ...process.env, ...opts.env }`),
|
|
// so a regression where `env: opts.env` gets dropped from the
|
|
// runPlanSkillObservation -> launchClaudePty handoff is only caught by
|
|
// the live PTY test, not here.
|
|
const opts: ClaudePtyOptions = {
|
|
env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' },
|
|
};
|
|
expect(opts.env).toEqual({ QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' });
|
|
});
|
|
});
|
|
|
|
describe('launchClaudePty model pin', () => {
|
|
// Behavioral: a fake CLI records the argv its real PTY launch received.
|
|
// Chain mirrors session-runner.ts: opts.model -> EVALS_MODEL ->
|
|
// resolveEvalModel('capture'); --model precedes extraArgs so a per-test
|
|
// --model wins (last flag wins). Per-runner forwarding of opts.model is
|
|
// asserted through the fake driver in claude-pty-runner.runners.unit.test.ts.
|
|
test('ClaudePtyOptions exposes model?: string', () => {
|
|
const opts: ClaudePtyOptions = { model: 'claude-sonnet-4-6' };
|
|
expect(opts.model).toBe('claude-sonnet-4-6');
|
|
});
|
|
|
|
test.skipIf(process.platform === 'win32')('spawn args pin --model from the fallback chain, before extraArgs, with hermetic MCP gating', () => {
|
|
const dir = mkdtempSync(join(tmpdir(), 'pty-model-pin-'));
|
|
try {
|
|
const fake = join(dir, 'fake-claude');
|
|
writeFileSync(fake, `#!${process.execPath}\nprocess.stdout.write('ARGV=' + JSON.stringify(process.argv.slice(2)) + '\\n');\nsetInterval(() => {}, 1000);\n`, { mode: 0o755 });
|
|
const runner = pathToFileURL(join(import.meta.dir, 'claude-pty-runner.ts')).href;
|
|
const worker = join(dir, 'worker.ts');
|
|
writeFileSync(worker, `import { launchClaudePty } from ${JSON.stringify(runner)};
|
|
import { resolveEvalModel } from ${JSON.stringify(pathToFileURL(join(import.meta.dir, '../../lib/eval-model.ts')).href)};
|
|
const argv = async (opts, evalsModel) => {
|
|
if (evalsModel === undefined) delete process.env.EVALS_MODEL; else process.env.EVALS_MODEL = evalsModel;
|
|
const session = await launchClaudePty({ cwd: ${JSON.stringify(dir)}, timeoutMs: 5000, ...opts });
|
|
try { await session.waitFor(/ARGV=\\[.*\\]/, { timeoutMs: 4000, pollMs: 20 }); return JSON.parse(/ARGV=(\\[.*\\])/.exec(session.visibleText())[1]); }
|
|
finally { await session.close(); }
|
|
};
|
|
process.stdout.write(JSON.stringify({ capture: resolveEvalModel('capture'),
|
|
fallback: await argv({}, undefined), env: await argv({}, 'env-model'), explicit: await argv({ model: 'opts-model' }, 'env-model'),
|
|
extra: await argv({ extraArgs: ['--model', 'override'] }, undefined) }));
|
|
`);
|
|
const run = (hermetic: string) => {
|
|
const result = spawnSync(process.execPath, [worker], { cwd: join(import.meta.dir, '../..'), encoding: 'utf8', timeout: 30_000,
|
|
env: { ...process.env, BROWSE_TERMINAL_BINARY: fake, EVALS_HERMETIC: hermetic } });
|
|
expect(result.status, result.stderr).toBe(0);
|
|
return JSON.parse(result.stdout.trim().split('\n').at(-1)!);
|
|
};
|
|
const hermetic = run('1');
|
|
const model = (args: string[]) => args[args.indexOf('--model') + 1];
|
|
expect(model(hermetic.fallback)).toBe(hermetic.capture);
|
|
expect(model(hermetic.env)).toBe('env-model');
|
|
expect(model(hermetic.explicit)).toBe('opts-model');
|
|
expect(hermetic.extra.indexOf('--model')).toBeLessThan(hermetic.extra.lastIndexOf('--model'));
|
|
expect(hermetic.extra.slice(-2)).toEqual(['--model', 'override']);
|
|
expect(hermetic.fallback).toContain('--strict-mcp-config');
|
|
expect(run('0').fallback).not.toContain('--strict-mcp-config');
|
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
|
}, 60_000);
|
|
});
|
|
|
|
describe('launchClaudePty transcript pricing (GSTA-150 PR 3)', () => {
|
|
test.skipIf(process.platform === 'win32')('prices the transcript this launch created for its cwd, without adding a launch argument; opting out prices nothing', () => {
|
|
const dir = mkdtempSync(join(tmpdir(), 'pty-transcript-cost-'));
|
|
try {
|
|
const fake = join(dir, 'fake-claude');
|
|
writeFileSync(fake, `#!${process.execPath}
|
|
const fs = require('node:fs'), path = require('node:path'), crypto = require('node:crypto');
|
|
const project = path.join(process.env.CLAUDE_CONFIG_DIR, 'projects', 'fake-project');
|
|
fs.mkdirSync(project, { recursive: true });
|
|
fs.writeFileSync(path.join(project, crypto.randomUUID() + '.jsonl'), JSON.stringify({ type: 'assistant', cwd: process.cwd(),
|
|
message: { id: 'msg_' + crypto.randomUUID(), model: 'claude-opus-5-5', usage: { input_tokens: 0, output_tokens: 1000 } } }) + '\\n');
|
|
process.stdout.write('ARGV=' + JSON.stringify(process.argv.slice(2)) + '\\n');
|
|
setInterval(() => {}, 1000);
|
|
`, { mode: 0o755 });
|
|
const runner = pathToFileURL(join(import.meta.dir, 'claude-pty-runner.ts')).href;
|
|
const worker = join(dir, 'worker.ts');
|
|
writeFileSync(worker, `import { launchClaudePty } from ${JSON.stringify(runner)};
|
|
import * as fs from 'node:fs';
|
|
const run = async (opts) => {
|
|
const cwd = fs.mkdtempSync(${JSON.stringify(join(dir, 'cwd-'))});
|
|
const session = await launchClaudePty({ cwd, timeoutMs: 5000, ...opts });
|
|
try {
|
|
await session.waitFor(/ARGV=\\[.*\\]/, { timeoutMs: 4000, pollMs: 20 });
|
|
return { argv: JSON.parse(/ARGV=(\\[.*\\])/.exec(session.visibleText())[1]), cost: session.transcriptCostUsd?.() ?? null };
|
|
} finally { await session.close(); }
|
|
};
|
|
const [priced, optedOut] = await Promise.all([run({}), run({ priceTranscript: false })]);
|
|
process.stdout.write(JSON.stringify({ priced, optedOut }));
|
|
`);
|
|
const result = spawnSync(process.execPath, [worker], { cwd: join(import.meta.dir, '../..'), encoding: 'utf8', timeout: 30_000,
|
|
env: { ...process.env, BROWSE_TERMINAL_BINARY: fake, EVALS_HERMETIC: '1' } });
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const { priced, optedOut } = JSON.parse(result.stdout.trim().split('\n').at(-1)!);
|
|
expect(priced.cost).toBeCloseTo(0.02, 6);
|
|
expect(optedOut.cost).toBeNull();
|
|
expect(priced.argv).not.toContain('--session-id');
|
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
|
}, 60_000);
|
|
});
|