mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-19 14:44:44 +01:00
* Attribute every paid trial's cost: bind label-named records by testName, price aborted SDK sessions from streamed usage, model-free cases cost $0
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: case cost adds streamed sessions to record sums and keeps record cost_known
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: the paid report headline counts ledger-only session cost once
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: a cost-only record never sets the trial model
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: recordE2E writes cost_case_id only for a registered label-named case
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: CHANGELOG for paid-report cost attribution (Unreleased)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: the headline adds only streamed session cost; cache writes without a 5m key
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: how the paid report prices a trial and what stays cost-unknown
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Price PTY sessions from their transcripts: pin a session id, sum the session's and its subagents' usage into the ledger's streamed_cost_usd
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: transcript pricing never throws from a PTY ledger writer
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore: pin transcript-cost.ts as non-behavioral at HARNESS_VERSION 6
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: recognise --session-id= and --resume= forms
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: CHANGELOG for PTY transcript pricing (Unreleased)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: PTY sessions priced from their transcripts
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: price PTY transcripts without adding a launch argument; native AUQ capture opts out
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: harness notes and CHANGELOG match the transcript claim
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): gstack-flaky run reruns a failed JUnit test once and ledgers the flake (GSTA-158)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): stream output, keep unreported runner failures red, fail-closed duplicates and nonzero reruns
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: pre-landing review fixes
gstack-flaky never greens an incomplete run: a targeted rerun needs the
runner's summary proving the first run complete, otherwise (and always for
generic runners) the rerun repeats the whole command. A rerun clears a test
only when it exits 0 and the test passes once on its first attempt. Killed
runners exit 3 (FLAKY_COMMAND_KILLED) and signals are forwarded. The first
report is restored at --junit with reruns kept beside it; ids are redacted at
every output; the ledger gets its own GSTACK_FLAKE_LEDGER_JUNIT; fixtures are
registered in FREE_FIXTURES.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: latch cancellation and check whole-command rerun reports
A cancel forwarded to a runner that traps the signal now counts as killed for
every later attempt, so a cancelled job is never rerun or greened. A
whole-command rerun clears nothing when its report has a failed test, misses a
first-run test, or its runner counted failures outside the report.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: whole-command rerun when coverage or runner-flaky tests can fail the run
A coverage threshold or a fail-on-flaky policy fails a run for a reason a
narrowed rerun cannot reproduce, so either forces a whole-command rerun.
--junit refuses an existing file that is not a JUnit report instead of
deleting it.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): rerun the whole command by default; targeted rerun is opt-in
A rerun counts only on the runner's own full green verdict: exit 0, no failed
or missing test, nothing counted outside the report. That covers every way a
run can fail, which a narrowed rerun cannot (a coverage threshold with an lcov
reporter, a fail-on-flaky policy, a test never reached). --rerun-scope test
reruns the failed test alone, with its preconditions documented and the
complete-run checks kept as a safety net. An unreadable existing --junit file
is a usage error, not a crash.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: gstack-flaky CHANGELOG entry and standalone-binaries row (GSTA-158)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): quarantine registry for repeat flakes (GSTA-158 P2)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): bound renewal dates; match non-ASCII paths in check --base
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): reword the FLAKY_QUARANTINE_BASE hint so the egress scanner does not read it as a network op
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(agent-sdk): every SDK caller passes the claude_code preset or names why not
The SDK's default system prompt is a bare stub that drops the line telling the
model to run /skill through the Skill tool. runAgentSdkTest now refuses a
missing, empty or bare-string prompt; non-preset callers name a reason; a free
tripwire scans test/ and scripts/ for the same mistake.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: record the SDK prompt guard as a non-behavioral harness change and baseline its tripwire
harness-version: runAgentSdkTest validates systemPrompt before starting; every
live caller already passed the claude_code preset, so no verdict can change.
test-of-test ratchet: the tripwire's subject is the eval harness by design.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: name the preset literal callers pass and the unnamed-prompt refusal
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(eval-plan): frontier models under test (R54)
Haiku and Sonnet are not models under test. dry-run refuses a plan whose every
arm is Haiku or Sonnet unless --tier-question names what it asks about that
tier, warns on a mix, and re-costs the plan on Opus 5.5 and GPT-6.1 Sol.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling
The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the
exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining
growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458,
the capture helper's value); other skills' ceilings are left as they were.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(eval-plan): pilot start keeps the R54 refusal; --tier-question rejects a flag
Review pass 1 (full depth) found the frontier refusal could be skipped by
starting a pilot on Haiku or Sonnet arms behind a dry run made on other arms,
or behind a dry-run.json written before the check existed. startPilot now
refuses a pilot whose every arm is Haiku or Sonnet unless its dry run recorded
a tier question. dry-run --tier-question no longer takes the next flag as its
answer. The testing doc now says the frontier re-cost is printed for
comparison with the brief cap, not enforced.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(support-tiers): Windows CI in the present tense now that windows-smoke is required (#3806)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore: empty commit for a clean check set at the same tree (cancelled orphan free-suite check at 97487df)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(skills): /eval-plan entry names the R54 frontier check
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
315 lines
25 KiB
TypeScript
315 lines
25 KiB
TypeScript
/**
|
|
* Census attribution in the paid report (I4): every JUnit testcase of a
|
|
* non-trial shard is a case result, a zero-credit deselection with its
|
|
* reason, or listed by name as unattributed. Fixtures are trimmed from the
|
|
* 2026-10-03 periodic and gate census artifacts (run 37151477069, commit
|
|
* ac20ef1), whose report counted deselections as SKIPPED 82/282 and left
|
|
* 72/24 testcases unattributed. Budget records follow the current declared
|
|
* policy (plan-mode-no-op: 3,720,000 ms since v1.91.18.0 added its devex member).
|
|
* The periodic fixture is re-shaped to the current case shards: qa-workflow runs
|
|
* as its qa-fix-loop case shard (its three gate-tier siblings become sibling
|
|
* deselections) and the review carve runs as a case of the consolidated
|
|
* test/carve-section-loading.test.ts.
|
|
*/
|
|
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
|
|
import * as fs from 'node:fs';
|
|
import * as os from 'node:os';
|
|
import * as path from 'node:path';
|
|
import { DESELECTION_REASONS, RUNTIME_SKIP_REASON, attributeJUnitCase, formatPanelLine, resolvePaidShardBudget, runPaidReport, type JUnitCase } from '../scripts/test-paid-shards';
|
|
import { panelVerdict } from './helpers/eval-store';
|
|
import { trialRecordProblems as trialRecordProblemsV1 } from './fixtures/trial-record-v1-reader';
|
|
|
|
const FIXTURES = path.join(import.meta.dir, 'fixtures', 'paid-report-census');
|
|
|
|
let base: string;
|
|
beforeAll(() => { base = fs.mkdtempSync(path.join(os.tmpdir(), 'paid-report-census-')); });
|
|
afterAll(() => { fs.rmSync(base, { recursive: true, force: true }); });
|
|
|
|
let runs = 0;
|
|
function report(lane: 'periodic' | 'gate', edit: (dir: string) => void = () => {}, env: NodeJS.ProcessEnv = {}) {
|
|
const dir = path.join(base, `${lane}-${++runs}`);
|
|
fs.cpSync(path.join(FIXTURES, lane), dir, { recursive: true });
|
|
edit(dir);
|
|
const lines: string[] = [];
|
|
const { log, error } = console;
|
|
console.log = (...args: unknown[]) => { lines.push(args.join(' ')); };
|
|
console.error = console.log;
|
|
let status: number;
|
|
try { status = runPaidReport(dir, { env }); } finally { console.log = log; console.error = error; }
|
|
const history = fs.readFileSync(path.join(dir, 'trial-outcomes.jsonl'), 'utf8').split('\n').filter(Boolean).map(line => JSON.parse(line));
|
|
return { status, out: lines.join('\n'), dir, history };
|
|
}
|
|
|
|
const junit = (dir: string, slug: string) => path.join(dir, 'paid-slice-1-a1', 'shards', slug, 'junit.xml');
|
|
|
|
describe('census attribution from the 2026-10-03 artifacts', () => {
|
|
test('periodic: deselections and hook placeholders are zero-credit, every executed testcase maps to one case', () => {
|
|
const r = report('periodic');
|
|
expect(r.status, r.out).toBe(0);
|
|
expect(r.out).toContain(' rule 7/7 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)');
|
|
expect(r.out).toContain(' SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0');
|
|
expect(r.out).toContain(`deselected testcases, zero credit (21):`);
|
|
expect(r.out).toContain(` 13 ${DESELECTION_REASONS.sibling}`);
|
|
expect(r.out).toContain(' 4 gate-tier case (this lane runs periodic)');
|
|
expect(r.out).toContain(` 4 ${DESELECTION_REASONS.hook}`);
|
|
expect(r.out).toContain(' ⚠ test/skill-e2e-test-value.test.ts (3 skipped — 3 gate-tier case (this lane runs periodic))');
|
|
expect(r.out).not.toContain('UNATTRIBUTED');
|
|
expect(r.history.map(record => `${record.case}:${record.outcome}`).sort()).toEqual([
|
|
'benchmark-providers-live:passed', 'carve-section-loading-review:passed', 'codex-recommendation-substance:passed',
|
|
'cso-full-audit:passed', 'cso-infra-scope:passed', 'design-consultation-core:passed', 'qa-fix-loop:passed',
|
|
]);
|
|
expect(r.history.find(record => record.case === 'benchmark-providers-live')).toMatchObject({ file: 'test/skill-e2e-benchmark-providers.test.ts', source: 'junit' });
|
|
expect(r.history.filter(record => record.file === 'test/skill-e2e-design.test.ts'))
|
|
.toEqual([expect.objectContaining({ case: 'design-consultation-core', outcome: 'passed', source: 'junit' })]);
|
|
expect(r.history.some(record => record.outcome === 'skipped')).toBe(false);
|
|
});
|
|
|
|
test('gate: other-tier cases and hook placeholders are deselected; a multi-test case is one result', () => {
|
|
const r = report('gate');
|
|
expect(r.status, r.out).toBe(0);
|
|
expect(r.out).toContain(' rule 2/2 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)');
|
|
expect(r.out).toContain(' SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0');
|
|
expect(r.out).toContain(' 16 periodic-tier case (this lane runs gate)');
|
|
expect(r.out).toContain(` 2 ${DESELECTION_REASONS.hook}`);
|
|
expect(r.out).toContain(` ⚠ test/skill-e2e-qa-bugs.test.ts (5 skipped — 2 ${DESELECTION_REASONS.hook}; 3 periodic-tier case (this lane runs gate))`);
|
|
expect(r.history.map(record => `${record.case}:${record.outcome}`).sort()).toEqual(['cso-diff-mode:passed', 'plan-mode-no-op:passed']);
|
|
});
|
|
|
|
test('a selected in-lane case that skipped is SKIPPED with its reason; an unattributable testcase is listed by name', () => {
|
|
const r = report('periodic', dir => {
|
|
const qa = junit(dir, 'skill-e2e-qa-workflow--qa-fix-loop');
|
|
fs.writeFileSync(qa, fs.readFileSync(qa, 'utf8').replace(/(<testcase name="qa-fix-loop"[^>]*?) \/>/, '$1><skipped /></testcase>'));
|
|
const cso = junit(dir, 'skill-e2e-cso');
|
|
fs.writeFileSync(cso, fs.readFileSync(cso, 'utf8').replace(/<\/testsuite>(\r?\n)<\/testsuites>/,
|
|
'<testcase name="an unlabeled helper" classname="CSO v3" time="1" file="test/skill-e2e-cso.test.ts" line="9" />$1</testsuite>$1</testsuites>'));
|
|
});
|
|
expect(r.status, r.out).toBe(0);
|
|
expect(r.out).toContain(' SKIPPED 1 · INFRA 0 · INCOMPLETE 0 · unattributed 1 · ACTION REQUIRED 0');
|
|
expect(r.out).toContain(`[test:paid] ◌ qa-fix-loop test/skill-e2e-qa-workflow.test.ts#qa-fix-loop (qa-fix-loop): ${RUNTIME_SKIP_REASON}`);
|
|
expect(r.out).toContain('[test:paid] ? test/skill-e2e-cso.test.ts :: an unlabeled helper (passed)');
|
|
const summary = fs.readFileSync(path.join(r.dir, 'report-summary.md'), 'utf8');
|
|
expect(summary).toContain('**Skipped, deselected and unattributed testcases**');
|
|
expect(summary).toContain('? test/skill-e2e-cso.test.ts :: an unlabeled helper (passed)');
|
|
const collector = JSON.parse(fs.readFileSync(path.join(r.dir, 'collector-outcomes.json'), 'utf8'));
|
|
expect(collector.census.unattributed).toEqual(['test/skill-e2e-cso.test.ts :: an unlabeled helper (passed)']);
|
|
expect(collector.census.skipped).toHaveLength(1);
|
|
expect(collector.verdict.counts).toMatchObject({ skipped: 1, unattributed: 1 });
|
|
});
|
|
});
|
|
|
|
describe('a case shard\'s result records are its one case\'s cost (2026-10-26 train 76db1d1f10c4: 54 of 136 trials cost-unknown, verdict $24.41 under a $32.92 run)', () => {
|
|
const shard = (dir: string, slug: string) => path.join(dir, 'paid-slice-1-a1', 'shards', slug);
|
|
const results = (dir: string, slug: string, tests: Array<Record<string, unknown>>) =>
|
|
fs.writeFileSync(path.join(shard(dir, slug), 'e2e-2026-10-26.json'), JSON.stringify({ tests, total_cost_usd: tests.reduce((sum, t) => sum + Number(t.cost_usd), 0) }));
|
|
const collector = (dir: string) => JSON.parse(fs.readFileSync(path.join(dir, 'collector-outcomes.json'), 'utf8'));
|
|
|
|
test('a record named by its display label ("/qa fix loop") belongs to the shard\'s only case, so the trial and the verdict carry its cost', () => {
|
|
const r = report('periodic', dir => results(dir, 'skill-e2e-qa-workflow--qa-fix-loop',
|
|
[{ name: '/qa fix loop', suite: 'QA skill E2E', tier: 'e2e', passed: true, duration_ms: 1000, cost_usd: 1.23 }]));
|
|
const trial = r.history.find(record => record.case === 'qa-fix-loop');
|
|
expect(trial).toMatchObject({ outcome: 'passed', cost_usd: 1.23 });
|
|
expect(trial.cost_known).toBeUndefined();
|
|
expect(collector(r.dir).verdict.costUsd).toBe(1.23);
|
|
expect(r.out).toContain('cost $1.23 known + 6 of 7 trial(s) cost unknown');
|
|
});
|
|
|
|
test('a multi-case shard whose records carry display labels keeps those trials cost-unknown instead of guessing', () => {
|
|
const r = report('periodic', dir => results(dir, 'skill-e2e-cso',
|
|
[{ name: '/cso full audit', suite: 'cso', tier: 'e2e', passed: true, duration_ms: 1000, cost_usd: 0.5 },
|
|
{ name: 'cso-infra-scope', suite: 'cso', tier: 'e2e', passed: true, duration_ms: 1000, cost_usd: 0.25 }]));
|
|
expect(r.history.find(record => record.case === 'cso-full-audit')).toMatchObject({ cost_usd: 0, cost_known: false });
|
|
expect(r.history.find(record => record.case === 'cso-infra-scope')).toMatchObject({ cost_usd: 0.25 });
|
|
expect(r.history.find(record => record.case === 'cso-infra-scope').cost_known).toBeUndefined();
|
|
expect(collector(r.dir).verdict.costUsd).toBe(0.75);
|
|
});
|
|
|
|
test('a record-less trial\'s streamed session cost reaches the trial and the verdict total once (GSTA-150 outside voice: $0.65 known, headline $0.00)', () => {
|
|
const ledger = (dir: string, slug: string, costs: number[]) => fs.writeFileSync(path.join(shard(dir, slug), 'session-ledger.jsonl'),
|
|
costs.map(cost => JSON.stringify({ key: 'k', runner: 'agent-sdk', started_at: 't', elapsed_ms: 1, end: 'aborted', billed: false, streamed_cost_usd: cost })).join('\n') + '\n');
|
|
const r = report('periodic', dir => {
|
|
ledger(dir, 'skill-e2e-qa-workflow--qa-fix-loop', [0.4, 0.25]);
|
|
results(dir, 'skill-e2e-benchmark-providers', [{ name: 'benchmark-providers-live', suite: 's', tier: 'e2e', passed: true, duration_ms: 1, cost_usd: 0.1 }]);
|
|
ledger(dir, 'skill-e2e-benchmark-providers', [0.5]);
|
|
});
|
|
expect(r.history.find(record => record.case === 'qa-fix-loop')).toMatchObject({ outcome: 'passed', cost_usd: 0.65 });
|
|
expect(r.history.find(record => record.case === 'benchmark-providers-live')).toMatchObject({ cost_usd: 0.6 });
|
|
expect(collector(r.dir).verdict.costUsd).toBe(1.25);
|
|
expect(r.out).toContain('cost $1.25 known + 5 of 7 trial(s) cost unknown');
|
|
});
|
|
|
|
test('a billed session\'s cost is already in its collector total, so the verdict never adds it again (Codex host, unbound label record)', () => {
|
|
const r = report('periodic', dir => {
|
|
results(dir, 'skill-e2e-cso', [{ name: '/cso full audit', suite: 'cso', tier: 'e2e', passed: true, duration_ms: 1, cost_usd: 0.5 }]);
|
|
fs.writeFileSync(path.join(shard(dir, 'skill-e2e-cso'), 'session-ledger.jsonl'),
|
|
JSON.stringify({ key: 'k', runner: 'codex', started_at: 't', elapsed_ms: 1, end: 'completed', billed: true, cost_usd: 0.5 }) + '\n');
|
|
});
|
|
expect(collector(r.dir).verdict.costUsd).toBe(0.5);
|
|
});
|
|
|
|
// Value: protects=a failed case shard's trial reads its failure class, exit reason, error and cost from the display-labelled record the shard
|
|
// wrote, not a generic 'assertion' with the JUnit message (the same attribution the cost fix made; train-gate's INFRA/VERDICT class follows failure_class);
|
|
// fails_when=failedRecord is looked up through the old name/case_id-only filter again so the record's class and error are dropped;
|
|
// existing_gap=both attribution rows above use passing records, so the failedRecord branch of the widened filter is unexercised; seam=none
|
|
test('a failed display-labelled record in a case shard supplies the trial\'s failure class, exit reason and error, not a bare assertion', () => {
|
|
const r = report('periodic', dir => {
|
|
const qa = junit(dir, 'skill-e2e-qa-workflow--qa-fix-loop');
|
|
fs.writeFileSync(qa, fs.readFileSync(qa, 'utf8').replace('failures="0"', 'failures="1"')
|
|
.replace(/(<testcase name="qa-fix-loop"[^>]*?) \/>/, '$1><failure type="AssertionError" message="qa-fix-loop: expected a fix commit" /></testcase>'));
|
|
results(dir, 'skill-e2e-qa-workflow--qa-fix-loop', [{ name: '/qa fix loop', suite: 'QA skill E2E', tier: 'e2e', passed: false, duration_ms: 1000, cost_usd: 2.5,
|
|
failure_class: 'timeout', exit_reason: 'timeout', timeout_at_turn: 7, error: 'no terminal outcome within 1800000ms\nsecond line' }]);
|
|
});
|
|
expect(r.status, r.out).toBe(1);
|
|
const trial = r.history.find(record => record.case === 'qa-fix-loop');
|
|
expect(trial).toMatchObject({ outcome: 'failed', failure_class: 'timeout', exit_reason: 'timeout', error: 'no terminal outcome within 1800000ms', cost_usd: 2.5 });
|
|
expect(trial.cost_known).toBeUndefined();
|
|
expect(r.out).toContain('✗ qa-fix-loop rule FAIL timeout at turn 7 — no terminal outcome within 1800000ms');
|
|
expect(trialRecordProblemsV1(trial)).toEqual([]);
|
|
});
|
|
});
|
|
|
|
describe('record builders attach ledger sessions and failure causes (plan 0.1/0.2)', () => {
|
|
const noOp = (dir: string) => path.join(dir, 'paid-slice-1-a1', 'shards', 'skill-e2e-plan-mode-no-op');
|
|
const failNoOp = (dir: string) => {
|
|
const file = path.join(noOp(dir), 'junit.xml');
|
|
fs.writeFileSync(file, fs.readFileSync(file, 'utf8')
|
|
.replace('failures="0"', 'failures="1"')
|
|
.replace(/(<testcase name="plan-ceo-review reaches a terminal outcome outside plan mode"[^>]*?) \/>/,
|
|
'$1><failure type="AssertionError" message="plan-ceo-review: outcome=timeout (no terminal outcome within 180000ms)" /></testcase>'));
|
|
fs.writeFileSync(path.join(noOp(dir), 'session-ledger.jsonl'), [
|
|
{ key: 'pty:session#1', runner: 'pty', started_at: 't', budget_ms: 180_000, elapsed_ms: 16_800, end: 'completed' },
|
|
{ key: 'pty:session#2', runner: 'pty', started_at: 't', budget_ms: 180_000, elapsed_ms: 180_050, end: 'observer_timeout' },
|
|
].map(row => JSON.stringify(row)).join('\n') + '\nnot json\n');
|
|
};
|
|
|
|
test('a JUnit-only PTY case gets failure_cause and sessions from its ledger; failure_class is unchanged', () => {
|
|
const r = report('gate', failNoOp);
|
|
const noOpRecord = r.history.find(record => record.case === 'plan-mode-no-op');
|
|
expect(noOpRecord).toMatchObject({ outcome: 'failed', failure_class: 'assertion', failure_cause: 'observer_timeout',
|
|
failure_cause_evidence: 'plan-ceo-review: outcome=timeout (no terminal outcome within 180000ms)',
|
|
sessions: [{ key: 'pty:session#1', runner: 'pty', elapsed_ms: 16_800, budget_ms: 180_000, end: 'completed' },
|
|
{ key: 'pty:session#2', runner: 'pty', elapsed_ms: 180_050, budget_ms: 180_000, end: 'observer_timeout' }] });
|
|
for (const record of r.history) expect(trialRecordProblemsV1(record), record.case).toEqual([]);
|
|
// Its five tests run concurrently: JUnit times sum to 346 s, the shard's wall is 176.5 s (lane S A2).
|
|
expect(noOpRecord.duration_ms).toBe(176_495);
|
|
const plain = report('gate', dir => { failNoOp(dir); fs.rmSync(path.join(noOp(dir), 'session-ledger.jsonl')); });
|
|
const without = plain.history.find(record => record.case === 'plan-mode-no-op');
|
|
expect(without.failure_class).toBe(noOpRecord.failure_class);
|
|
expect(without.sessions).toBeUndefined();
|
|
const collector = (dir: string) => JSON.parse(fs.readFileSync(path.join(dir, 'collector-outcomes.json'), 'utf8'));
|
|
expect(collector(plain.dir).verdict).toEqual(collector(r.dir).verdict);
|
|
});
|
|
});
|
|
|
|
describe('red lines carry their values, evidence and a one-case command (plan 0.3)', () => {
|
|
const QA_ONLY = 'qa-only/SKILL.md workflow';
|
|
const addJudgeRed = (dir: string) => {
|
|
const shard = path.join(dir, 'paid-slice-1-a1', 'shards', 'skill-llm-eval');
|
|
fs.mkdirSync(shard, { recursive: true });
|
|
const message = 'expect(received).toBeGreaterThanOrEqual(expected)\n\nExpected: >= 4\nReceived: 3.6666666666666665\n';
|
|
fs.writeFileSync(path.join(shard, 'junit.xml'), `<?xml version="1.0" encoding="UTF-8"?>\n<testsuites><testsuite name="test/skill-llm-eval.test.ts">`
|
|
+ `<testcase name="${QA_ONLY}" classname="Other skill evals" time="34.8" file="test/skill-llm-eval.test.ts" line="9">`
|
|
+ `<failure type="AssertionError" message="${message.replace(/\n/g, ' ')}" /></testcase></testsuite></testsuites>\n`);
|
|
// Trimmed from census 37198445662's paid-slice-5-a1 qa-only record.
|
|
fs.writeFileSync(path.join(shard, '1.91.19-x-llm-judge-2026-10-04-1129.json'), JSON.stringify({ tier: 'llm-judge', total_cost_usd: 0.06, tests: [{
|
|
name: QA_ONLY, suite: 'Other skill evals', tier: 'llm-judge', passed: false, duration_ms: 34829, cost_usd: 0.06, execution: 'executed',
|
|
judge_scores: { clarity: 3.6666666666666665, completeness: 4.333333333333333, actionability: 3.6666666666666665 },
|
|
judge_reasoning: '[sample 1] The bundle is thorough. Clarity suffers from extreme density and cross-file indirection. Two actionability concerns: Browser Quick allots 30 seconds.\n[sample 2] Deliverables are concrete.\n[sample 3] Execution is feasible.',
|
|
exit_reason: 'validation_failed', error: message.replace('>', '>') }] }));
|
|
const manifestPath = path.join(dir, 'manifest.json');
|
|
const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8'));
|
|
const budget = resolvePaidShardBudget(['test/skill-llm-eval.test.ts']);
|
|
manifest.entries.push({ file: 'test/skill-llm-eval.test.ts', slice: 1, status: 'planned', estimatedMs: 35_000, budget });
|
|
fs.writeFileSync(manifestPath, JSON.stringify(manifest));
|
|
const slicePath = path.join(dir, 'paid-slice-1-a1', 'slice-1.json');
|
|
const slice = JSON.parse(fs.readFileSync(slicePath, 'utf8'));
|
|
slice.outcomes.push({ files: ['test/skill-llm-eval.test.ts'], status: 'failed', exitCode: 1, elapsedMs: 35_000, executedTests: 1, skippedTests: 0, budget });
|
|
fs.writeFileSync(slicePath, JSON.stringify(slice));
|
|
};
|
|
|
|
test('a judge red names each failing dimension, its mean, samples and rationale, its artifact and an exact one-case command', () => {
|
|
const r = report('gate', addJudgeRed, { GITHUB_RUN_ID: '37198445662' });
|
|
const line = r.out.split('\n').find(l => l.includes(`✗ ${QA_ONLY}`))!;
|
|
expect(line).toContain('FAIL assertion — expect(received).toBeGreaterThanOrEqual(expected)');
|
|
expect(line).toContain('· cause assertion clarity 3.67 < 4 (3 samples) "Clarity suffers from extreme density and cross-file indirection."; '
|
|
+ 'actionability 3.67 < 4 (3 samples) "Two actionability concerns: Browser Quick allots 30 seconds."');
|
|
expect(line).toContain('evidence: paid-slice-1-a1/shards/skill-llm-eval (fetch: bun run eval:pass-rates --run 37198445662 --case \'qa-only/SKILL.md workflow\')');
|
|
expect(line).toContain(`after a repair: EVALS=1 EVALS_TIER=gate EVALS_JUDGE_SELECTION_JSON='{"version":1,"selected":["${QA_ONLY}"],"reason":"after a repair"}' `
|
|
+ `bun test test/skill-llm-eval.test.ts -t '(?:^|\\s)(?:qa-only/SKILL\\.md workflow)$'`);
|
|
expect(line).not.toContain('rerun:');
|
|
const record = r.history.find(x => x.case === QA_ONLY);
|
|
expect(record).toMatchObject({ failure_class: 'assertion', error: 'expect(received).toBeGreaterThanOrEqual(expected)', failure_cause: 'assertion',
|
|
failure_detail: { judge: [expect.objectContaining({ dimension: 'clarity', mean: 3.67 }), expect.objectContaining({ dimension: 'actionability', mean: 3.67 })] } });
|
|
expect(trialRecordProblemsV1(record)).toEqual([]);
|
|
});
|
|
|
|
test('a behavior panel line names the failed trial\'s cause, values, slice artifact and an n-trial one-case command', () => {
|
|
const verdict = panelVerdict({ case: 'plan-review-prosons-neutral-neg', kind: 'behavior', panel: { n: 3, k: 2 }, quarantined: false, trials: [
|
|
{ trial: 1, outcome: 'failed', failure_class: 'assertion', exit_reason: 'success', error: 'expect(received).not.toMatch(expected)',
|
|
failure_cause: 'assertion', failure_cause_evidence: 'session completed; check failed',
|
|
failure_detail: { expected: 'not /taste call/i', received: '"a taste call"' } } as any,
|
|
{ trial: 2, outcome: 'failed', failure_class: 'assertion' }, { trial: 3, outcome: 'passed' }] });
|
|
const line = formatPanelLine({ ...verdict, file: 'test/skill-e2e-plan-prosons.test.ts', slices: { 1: 7, 2: 9, 3: 9 } }, 'periodic',
|
|
{ artifacts: new Map([[7, 'paid-slice-7-a1']]), runId: '37193478719' });
|
|
expect(line).toContain('t1: assertion (session completed; check failed) — expect(received).not.toMatch(expected) · cause assertion: session completed; check failed Expected: not /taste call/i · Received: "a taste call"');
|
|
expect(line).toContain('evidence: paid-slice-7-a1/shards/skill-e2e-plan-prosons--plan-review-prosons-neutral-neg.t1 (fetch: bun run eval:pass-rates --run 37193478719 --case plan-review-prosons-neutral-neg)');
|
|
expect(line).toContain('after a repair: bun run scripts/test-paid-shards.ts --tier periodic --case plan-review-prosons-neutral-neg --trials 3');
|
|
});
|
|
|
|
test('a rule red shows Expected/Received and selects only its case; the summary fence cannot be broken', () => {
|
|
const r = report('gate', dir => {
|
|
const file = path.join(dir, 'paid-slice-1-a1', 'shards', 'skill-e2e-plan-mode-no-op', 'junit.xml');
|
|
fs.writeFileSync(file, fs.readFileSync(file, 'utf8').replace(/(<testcase name="plan-ceo-review reaches a terminal outcome outside plan mode"[^>]*?) \/>/,
|
|
'$1><failure type="AssertionError" message="expect(received).toBe(expected) Expected: "asked" Received: "```@team</details>"" /></testcase>'));
|
|
addJudgeRed(dir);
|
|
});
|
|
const line = r.out.split('\n').find(l => l.includes('✗ plan-mode-no-op'))!;
|
|
expect(line).toContain('Expected: "asked" · Received: "```@\u200bteam</details>"');
|
|
expect(line).toContain('evidence: paid-slice-1-a1/shards/skill-e2e-plan-mode-no-op');
|
|
expect(line).not.toContain('(fetch:');
|
|
expect(line).toContain('after a repair: bun run scripts/test-paid-shards.ts --tier gate --case plan-mode-no-op');
|
|
const summary = fs.readFileSync(path.join(r.dir, 'report-summary.md'), 'utf8');
|
|
const lines = summary.split('\n');
|
|
const at = lines.findIndex(l => l.includes('✗ plan-mode-no-op'));
|
|
const open = lines.slice(0, at).findLast(l => /^`{3,}$/.test(l))!;
|
|
expect(open.length).toBeGreaterThan(3);
|
|
expect(lines.slice(at).find(l => /^`{3,}$/.test(l))).toBe(open);
|
|
expect(summary).not.toMatch(/@team/);
|
|
if (summary.includes('**ACTION REQUIRED')) expect(summary.trimEnd().endsWith('Guide: docs/evals/census-red.md')).toBe(true);
|
|
const collector = JSON.parse(fs.readFileSync(path.join(r.dir, 'collector-outcomes.json'), 'utf8'));
|
|
expect(collector.failures.join('\n')).not.toContain('`');
|
|
});
|
|
});
|
|
|
|
describe('attributeJUnitCase', () => {
|
|
const tc = (name: string, outcome: JUnitCase['outcome'], line: number | null = 1): JUnitCase =>
|
|
({ name, classname: 's', outcome, timeMs: 0, ...(line === null ? {} : { line }) });
|
|
const file = 'test/skill-e2e-review.test.ts';
|
|
|
|
test('isolated and unselected in-lane cases are deselected; a selected skip is a case', () => {
|
|
expect(attributeJUnitCase(tc('review-design-lite', 'skipped'), { key: file, tier: 'periodic', excludeCases: ['review-design-lite'] }))
|
|
.toEqual({ kind: 'deselected', reason: DESELECTION_REASONS.isolated });
|
|
expect(attributeJUnitCase(tc('review-sql-injection', 'skipped'), { key: file, tier: 'gate', selection: { e2e: ['review-enum-completeness'], judges: null } }))
|
|
.toEqual({ kind: 'deselected', reason: DESELECTION_REASONS.selection });
|
|
expect(attributeJUnitCase(tc('review-sql-injection', 'skipped'), { key: file, tier: 'gate', selection: { e2e: ['review-sql-injection'], judges: null } }))
|
|
.toEqual({ kind: 'case', id: 'review-sql-injection' });
|
|
});
|
|
|
|
test('only a skipped, line-less "(unnamed)" is a hook placeholder; a failing hook or a stray case-shard result is never deselected', () => {
|
|
expect(attributeJUnitCase(tc('(unnamed)', 'skipped', null), { key: file, tier: 'gate' })).toEqual({ kind: 'deselected', reason: DESELECTION_REASONS.hook });
|
|
expect(attributeJUnitCase(tc('(unnamed)', 'failed', null), { key: file, tier: 'gate' })).toEqual({ kind: 'unattributed' });
|
|
expect(attributeJUnitCase(tc('(unnamed)', 'skipped'), { key: file, tier: 'gate' })).toEqual({ kind: 'unattributed' });
|
|
const shard = 'test/skill-e2e-design.test.ts#design-consultation-core';
|
|
expect(attributeJUnitCase(tc('design-consultation-core', 'passed'), { key: shard, tier: 'periodic' })).toEqual({ kind: 'case', id: 'design-consultation-core' });
|
|
expect(attributeJUnitCase(tc('needs Aside', 'skipped'), { key: shard, tier: 'periodic' })).toEqual({ kind: 'deselected', reason: DESELECTION_REASONS.sibling });
|
|
expect(attributeJUnitCase(tc('needs Aside', 'passed'), { key: shard, tier: 'periodic' })).toEqual({ kind: 'unattributed' });
|
|
});
|
|
|
|
test('a SKIPPED panel line names its reason and never calls a skipped trial an assertion', () => {
|
|
const verdict = panelVerdict({ case: 'review-design-lite', kind: 'behavior', panel: { n: 3, k: 2 },
|
|
trials: [1, 2, 3].map(trial => ({ trial, outcome: 'skipped' as const })), quarantined: false });
|
|
const line = formatPanelLine({ ...verdict, file: 'test/skill-e2e-review.test.ts', slices: {} }, 'periodic');
|
|
expect(line).toContain('t1: skipped; t2: skipped; t3: skipped [every trial skipped (no verdict credit)]');
|
|
expect(line).not.toContain('assertion');
|
|
});
|
|
});
|