mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
v1.91.33.0 feat(ship): measurement bar with explained-red and pooled decisions, weekly flaky-test sweep on main (#3059)
* refactor(eval-history): share pass-rate history loading as loadPassRateHistory() eval:pass-rates' main block loaded local dirs or the weekly trial-outcomes artifacts inline. Move that into an exported function returning records, scope and fetch errors, so the off-ship sweep reads the same history the same way. Behavior unchanged. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-measure): the measurement bar and the weekly off-ship sweep CLI Decide each diagnostic measurement with one bar (scripts/lib/measure-bar.ts): MEETS at 9/10; MEETS-qualified at 8/10 when every red is a mechanical provider error, judge noise or a cited model miss; EXTEND at 7/10 (or an unqualified 8/10) for one more batch on identical inputs, decided once on the pooled 20; BELOW otherwise and for any timeout, hang, regression or fixable cause. Behavior uses 12 trials (11/10/9; 22/20 of 24); judge counts median-panel outputs. A batch with more than 30% provider-evidence failures is void and redispatched once. Below-strict reds nobody classified return needs-classify. ship-measure gains classify (append-only records; provider refused without evidence), decide and extend (refuses a changed identity, names the field, never a third batch), exit codes 5 and 6, and a pooling identity (frozen tree, runner, CLI, model, CI image, Bun, policy and harness versions). ship-measure sweep ranks main's gate cases by their cost to the all-green probability from eval:pass-rates history, measures the top K with the same runner and bar under a weekly cap shared with the last 7 days of sweep reports (ship_measure_sweep_cases, ship_measure_sweep_budget_usd), and writes a report with the predicted all-green probability before and after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship): the measure loop decides by the bar's exit codes Step 4 branches on MEETS/MEETS-qualified, needs-classify (with the qualifying and fix-round classes), EXTEND once and BELOW; the rest of the loop prose is tightened to offset most of the growth. ship's union size cap moves from 1.495 to 1.4975 (measured 281,058 / 187,706 = 1.4973). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci: weekly eval-sweep workflow on main Mondays 12:00 UTC and on dispatch (k, cap_usd, dry_run): runs ship-measure sweep in the published CI image by digest with the census's paid-eval setup, downloads the last 7 days of sweep reports first so the weekly cap holds across runs, uploads the report and captures, and writes the job summary. Read-only grants; it never pushes or opens a pull request. One sweep at a time repository-wide, never cancelled mid-spend. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the measurement bar table and the weekly sweep Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: v1.91.33.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation" This reverts commitf4c58f2869. * ci: classify eval-sweep.yml, the sweep CLI and actionlint labels as free-only PR inputs None is read by a PR-gate case, so a change to them selects no paid case instead of restoring the full gate (test/free-fixtures.test.ts workflow inventory). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): the selection preflight accepts name-mode cases A case registered beside others in its file lists its trial shard as <file>#<id>~t1, not <file> (trial 1), so the preflight refused every name-mode case (for example ship-exploratory-late-input) before any paid trial. Accept both forms. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation" This reverts commit9619734307. * test: record the test-pr-profile.ts change as non-behavioral for HARNESS_VERSION Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): the name-mode preflight fix Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): a partly funded sweep row shows the trials that ran Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(sweep): the fake single-case command is a Bun script, so it runs on Windows The Windows free lane runs test/ship-measure-sweep.test.ts; CreateProcess cannot exec a shebang shell stub, so every fake trial errored and the cap stopped the sweep after one trial. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(plan-mode-no-op): the CEO sub-test reviews a fixed pasted plan Without a plan, plan-ceo-review reviewed the checkout's branch diff, so its pre-question work (audit and landscape searches) grew with whatever PR the gate ran on. On this PR it timed out at 300s in 3 of 10 trials while still researching the diff (ship-measure baseline 7/10). With a fixed small plan, the same way the devex sub-test already runs, it measured 10/10 (CEO sub-test 83-255s). No budget or threshold changed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): each diagnostic trial gets its own EVALS_RUN_ID Evidence-retaining cases (functional QA, docs faults) refuse to start without EVALS_RUN_ID, which only CI sets, so measuring them locally ran 0/10 sessions. qa-functional-cli-report, red once in this PR's gate, measured 10/10 once the trials started ($11.13). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed
1 parent
28f1385eac
commit
db745675bd
28 files changed
+1950
-300
No files matched your search
+72
-49
@@ -565,6 +565,75 @@ export function formatPassRates(report: PassRateReport, options: { caseFilter?:
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export interface PassRateHistory {
|
||||
records: TrialRecord[];
|
||||
unattributed: string[];
|
||||
errors: string[];
|
||||
manualReviews: string[];
|
||||
weeklyRuns?: string[];
|
||||
fetched: WeeklyRun[];
|
||||
pooledRunIds?: Set<string>;
|
||||
historyError: string | null;
|
||||
scope: string;
|
||||
cacheDir: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The trial history every eval:pass-rates view reads (and the ship-measure
|
||||
* sweep ranks from): local dirs, or the trial-outcomes artifacts of the last
|
||||
* N weekly runs on the branch plus the branch census runs that pool into
|
||||
* main's series. A fetch failure is returned as historyError, never thrown.
|
||||
*/
|
||||
export function loadPassRateHistory(o: { repo: string; workflow: string; branch: string; runsLimit: number; dirs: string[]; sinceDays: number; backfill?: boolean; runId?: number }): PassRateHistory {
|
||||
const records: TrialRecord[] = [];
|
||||
const unattributed = new Set<string>();
|
||||
const errors: string[] = [];
|
||||
const manualReviews: string[] = [];
|
||||
let historyError: string | null = null;
|
||||
let weeklyRuns: string[] | undefined;
|
||||
let fetched: WeeklyRun[] = [];
|
||||
let pooledRunIds: Set<string> | undefined;
|
||||
const cacheDir = path.join(path.resolve(resolveStateRoot()), 'eval-pass-rates-cache', o.repo.replace('/', '-'));
|
||||
const importDir = (dir: string, run: { run_id: string; sha?: string; timestamp?: string } | undefined, legacyDays: number) => {
|
||||
const trials = readTrialOutcomeDir(dir);
|
||||
records.push(...trials.records);
|
||||
errors.push(...trials.errors);
|
||||
const legacy = backfillEvalFiles(collectEvalFiles(dir, legacyDays), run);
|
||||
records.push(...legacy.records);
|
||||
manualReviews.push(...legacy.manualReviews);
|
||||
legacy.unattributed.forEach(name => unattributed.add(name));
|
||||
};
|
||||
if (o.dirs.length) {
|
||||
for (const dir of o.dirs) importDir(dir, undefined, o.sinceDays);
|
||||
} else {
|
||||
try {
|
||||
const weekly = listWeeklyRuns({ repo: o.repo, workflow: o.workflow, branches: [o.branch], limit: o.runsLimit }).filter(run => o.branch !== 'main' || isWeeklyHistoryRun(run));
|
||||
weeklyRuns = weekly.map(run => run.createdAt);
|
||||
// Branch census trials pool into main's matching series over the same window (isPooledTrialRun).
|
||||
const oldest = weekly[weekly.length - 1]?.createdAt;
|
||||
const pooledRuns = o.branch === 'main' && oldest
|
||||
? listWeeklyRuns({ repo: o.repo, workflow: o.workflow, branches: [''], limit: 100 }).filter(run => isPooledTrialRun(run) && run.createdAt >= oldest)
|
||||
: [];
|
||||
pooledRunIds = new Set(pooledRuns.map(run => `${run.id}`));
|
||||
const runs = [...weekly, ...pooledRuns];
|
||||
if (o.runId !== undefined && !runs.some(run => run.id === o.runId)) runs.push(GH_JOBS.getRun(o.repo, o.runId));
|
||||
fetched = runs;
|
||||
const match = o.backfill
|
||||
? (name: string) => name.startsWith('trial-outcomes') || /^(paid-slice-\d+|gate-census-\d+)(-a\d+)?$/.test(name)
|
||||
: (name: string) => name.startsWith('trial-outcomes');
|
||||
for (const run of runs) {
|
||||
const dirsForRun = downloadRunArtifacts({ repo: o.repo, run, match, cacheDir, maxBytes: o.backfill ? 64 * 1024 * 1024 : undefined });
|
||||
for (const dir of dirsForRun) importDir(dir, { run_id: `${run.id}`, sha: run.sha, timestamp: run.createdAt }, 3650);
|
||||
}
|
||||
} catch (error) {
|
||||
historyError = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
}
|
||||
const scope = o.dirs.length ? `local dirs ${o.dirs.join(', ')}` : `${o.repo} ${o.workflow} on ${o.branch}${pooledRunIds?.size ? ` + ${pooledRunIds.size} pooled branch census run(s)` : ''}, `
|
||||
+ `last ${o.runsLimit} completed run(s): ${fetched.map(run => run.id).join(', ') || 'none'}`;
|
||||
return { records, unattributed: [...unattributed].sort(), errors, manualReviews, weeklyRuns, fetched, pooledRunIds, historyError, scope, cacheDir };
|
||||
}
|
||||
|
||||
// --- GitHub history (shared reader: scripts/lib/ci-history.ts) ---
|
||||
|
||||
export { downloadRunArtifacts, GH_HISTORY, listWeeklyRuns, TRIAL_OUTCOMES_MAX_BYTES,
|
||||
@@ -580,55 +649,9 @@ if (import.meta.main) {
|
||||
const workflow = parsed.workflow ?? 'evals-periodic.yml';
|
||||
const branch = parsed.branch ?? 'main';
|
||||
|
||||
const records: TrialRecord[] = [];
|
||||
const unattributed = new Set<string>();
|
||||
const errors: string[] = [];
|
||||
let historyError: string | null = null;
|
||||
let weeklyRuns: string[] | undefined;
|
||||
let fetched: WeeklyRun[] = [];
|
||||
const cacheDir = path.join(path.resolve(resolveStateRoot()), 'eval-pass-rates-cache', repo.replace('/', '-'));
|
||||
let pooledRunIds: Set<string> | undefined;
|
||||
|
||||
const manualReviews: string[] = [];
|
||||
const importDir = (dir: string, run: { run_id: string; sha?: string; timestamp?: string } | undefined, legacyDays: number) => {
|
||||
const trials = readTrialOutcomeDir(dir);
|
||||
records.push(...trials.records);
|
||||
errors.push(...trials.errors);
|
||||
const legacy = backfillEvalFiles(collectEvalFiles(dir, legacyDays), run);
|
||||
records.push(...legacy.records);
|
||||
manualReviews.push(...legacy.manualReviews);
|
||||
legacy.unattributed.forEach(name => unattributed.add(name));
|
||||
};
|
||||
|
||||
if (dirs.length) {
|
||||
for (const dir of dirs) importDir(dir, undefined, sinceDays);
|
||||
} else {
|
||||
try {
|
||||
const weekly = listWeeklyRuns({ repo, workflow, branches: [branch], limit: runsLimit }).filter(run => branch !== 'main' || isWeeklyHistoryRun(run));
|
||||
weeklyRuns = weekly.map(run => run.createdAt);
|
||||
// Branch census trials pool into main's matching series over the same window (isPooledTrialRun).
|
||||
const oldest = weekly[weekly.length - 1]?.createdAt;
|
||||
const pooledRuns = branch === 'main' && oldest
|
||||
? listWeeklyRuns({ repo, workflow, branches: [''], limit: 100 }).filter(run => isPooledTrialRun(run) && run.createdAt >= oldest)
|
||||
: [];
|
||||
pooledRunIds = new Set(pooledRuns.map(run => `${run.id}`));
|
||||
const runs = [...weekly, ...pooledRuns];
|
||||
if (parsed.runId !== undefined && !runs.some(run => run.id === parsed.runId)) runs.push(GH_JOBS.getRun(repo, parsed.runId));
|
||||
fetched = runs;
|
||||
const match = backfill
|
||||
? (name: string) => name.startsWith('trial-outcomes') || /^(paid-slice-\d+|gate-census-\d+)(-a\d+)?$/.test(name)
|
||||
: (name: string) => name.startsWith('trial-outcomes');
|
||||
for (const run of runs) {
|
||||
const dirsForRun = downloadRunArtifacts({ repo, run, match, cacheDir, maxBytes: backfill ? 64 * 1024 * 1024 : undefined });
|
||||
for (const dir of dirsForRun) importDir(dir, { run_id: `${run.id}`, sha: run.sha, timestamp: run.createdAt }, 3650);
|
||||
}
|
||||
} catch (error) {
|
||||
historyError = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
}
|
||||
|
||||
const scope = dirs.length ? `local dirs ${dirs.join(', ')}` : `${repo} ${workflow} on ${branch}${pooledRunIds?.size ? ` + ${pooledRunIds.size} pooled branch census run(s)` : ''}, `
|
||||
+ `last ${runsLimit} completed run(s): ${fetched.map(run => run.id).join(', ') || 'none'}`;
|
||||
const history = loadPassRateHistory({ repo, workflow, branch, runsLimit, dirs, sinceDays, backfill, runId: parsed.runId });
|
||||
const { records, errors, manualReviews, weeklyRuns, fetched, pooledRunIds, historyError, scope, cacheDir } = history;
|
||||
const unattributed = new Set(history.unattributed);
|
||||
const scoped = caseFilter ? records.filter(record => record.case === caseFilter) : records;
|
||||
const caseHeadroom = headroom(scoped);
|
||||
if (view !== 'rates') {
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
"scripts/lib/published-text.ts": {"blob":"3f2e4d616b5dd0204035710c66b64d49055c5714","since":2},
|
||||
"scripts/lib/shard-engine.ts": {"blob":"3cd0b1aaa84018b00bf9ea0ce48093f3c8492f66","since":2},
|
||||
"scripts/test-paid-shards.ts": {"blob":"8ddee5f11ed7856876135520dd8025ab989e2db9","since":2},
|
||||
"scripts/test-pr-profile.ts": {"blob":"d7860897397de8fd605623570d1e7ebbbd5ec743","since":2},
|
||||
"scripts/test-pr-profile.ts": {"blob":"667ade7a76044bc20f7c183d0d4f23900c0423b3","since":2,"nonBehavioral":"FREE_ONLY_PR_FILES gains eval-sweep.yml, the sweep CLI and .github/actionlint.yaml: it changes which PR-gate cases a diff selects, never how a selected case runs or is judged"},
|
||||
"scripts/test-strict-output.ts": {"blob":"01185d20dc9761b04b9b2f751ec98664fe0eaf93","since":2},
|
||||
"test/helpers/agent-sdk-runner.ts": {"blob":"210bba92d241a90df578b2c7fc943ac5755d0025","since":2},
|
||||
"test/helpers/auq-substance-panel.ts": {"blob":"4881260ef793668e4963a062db6d06a5e1f7a40f","since":2},
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
/**
|
||||
* The ship-measure measurement bar (approved by Garry 2026-10-06): one pure
|
||||
* decision over a case's diagnostic trials, shared by /ship's measure loop
|
||||
* (scripts/ship-measure.ts) and the weekly off-ship sweep
|
||||
* (scripts/ship-measure-sweep.ts). Nothing here runs a trial or reads a file.
|
||||
*
|
||||
* For N trials per batch (rule and judge N = 10; behavior N = 12, four panels
|
||||
* of 3; a judge trial is one output scored by its median 3-sample panel), the
|
||||
* counts round up from 9/10, 8/10 and 7/10 of N:
|
||||
*
|
||||
* MEETS passes >= 9/10 of N.
|
||||
* MEETS-qualified passes >= 8/10 of N and every failed trial is in a
|
||||
* qualifying class: provider (affirmative provider or
|
||||
* transport evidence only), judge-noise, or model-miss
|
||||
* citing its evidence (path:line).
|
||||
* EXTEND passes >= 7/10 of N without meeting either bar, and no red
|
||||
* has a known fixable cause: run N more trials on identical
|
||||
* inputs, then decide once on the pooled 2N (>= 9/10 strict,
|
||||
* >= 8/10 qualified). There is never a third batch.
|
||||
* BELOW everything else, and any red in a fix-round class
|
||||
* (timeout, hang, regression, fixable) or a contract
|
||||
* violation: a fix round.
|
||||
*
|
||||
* Before the counts: a batch where more than 30% of trials fail with
|
||||
* affirmative provider evidence is void (an outage, not a measurement) and may
|
||||
* be redispatched once; both batches are reported. A batch below strict whose
|
||||
* failed trials are not all classified returns needs-classify when the
|
||||
* classification decides the outcome (passes at or above the 7/10 floor).
|
||||
*/
|
||||
|
||||
export type BarKind = 'rule' | 'behavior' | 'judge';
|
||||
export type TrialSet = 'initial' | 'redispatch' | 'extend' | 'extend-redispatch';
|
||||
export const QUALIFYING_CLASSES = ['provider', 'judge-noise', 'model-miss'] as const;
|
||||
export const FIX_ROUND_CLASSES = ['timeout', 'hang', 'regression', 'fixable'] as const;
|
||||
export const FAILURE_CLASSES = [...QUALIFYING_CLASSES, ...FIX_ROUND_CLASSES, 'unclassified'] as const;
|
||||
export type FailureClass = (typeof FAILURE_CLASSES)[number];
|
||||
|
||||
/** A batch is void when more than this share of its trials fail on affirmative provider evidence. */
|
||||
export const VOID_SHARE = { num: 3, den: 10 } as const;
|
||||
|
||||
export interface BarTrial {
|
||||
trial: number;
|
||||
set: TrialSet;
|
||||
passed: boolean;
|
||||
contract?: boolean;
|
||||
/** Judge outputs: each sample's pass; the output passes at 2 of 3. */
|
||||
samples?: boolean[];
|
||||
failureCause?: string;
|
||||
failureDetail?: string;
|
||||
failureEvidence?: string;
|
||||
}
|
||||
|
||||
export interface Classification { trial: number; class: FailureClass; evidence: string; by: 'machine' | 'agent'; at: string }
|
||||
|
||||
export type BarDecision = 'MEETS' | 'MEETS-qualified' | 'EXTEND' | 'BELOW' | 'needs-classify' | 'void' | 'incomplete';
|
||||
export type BarNext = 'none' | 'redispatch' | 'extend' | 'classify' | 'fix' | 'stop';
|
||||
|
||||
export interface BarThresholds { trials: number; strict: number; qualified: number; floor: number }
|
||||
export interface SetSummary { set: TrialSet; trials: number; passes: number; providerFailures: number; void: boolean }
|
||||
|
||||
export interface BarOutcome {
|
||||
decision: BarDecision;
|
||||
next: BarNext;
|
||||
passes: number;
|
||||
thresholds: BarThresholds;
|
||||
pooled: boolean;
|
||||
sets: SetSummary[];
|
||||
/** Failed counted trials that still need a class. */
|
||||
needsClassify: number[];
|
||||
reason: string;
|
||||
}
|
||||
|
||||
const atLeast = (n: number, num: number) => Math.ceil((n * num) / 10);
|
||||
|
||||
/** Strict 9/10, qualified 8/10 and extend floor 7/10 of n, rounded up (10: 9/8/7; 12: 11/10/9; 20: 18/16/14; 24: 22/20/17). */
|
||||
export function barThresholds(n: number): BarThresholds {
|
||||
return { trials: n, strict: atLeast(n, 9), qualified: atLeast(n, 8), floor: atLeast(n, 7) };
|
||||
}
|
||||
|
||||
/** A trial's pass under the bar: a judge output at 2 of its 3 samples; a contract violation never passes. */
|
||||
export function trialPassed(kind: BarKind, t: BarTrial): boolean {
|
||||
if (t.contract) return false;
|
||||
if (kind === 'judge' && t.samples) return t.samples.length === 3 && t.samples.filter(Boolean).length >= 2;
|
||||
return t.passed;
|
||||
}
|
||||
|
||||
const PROVIDER_TEXT = /\b(?:429|5\d\d)\b|rate[ _-]?limit|overloaded|api_error|ECONNRESET|connection reset|socket hang up|ETIMEDOUT|EAI_AGAIN|service unavailable|bad gateway/i;
|
||||
const TIMEOUT_CAUSES = new Set(['session_timeout', 'observer_timeout', 'provider_stall']);
|
||||
|
||||
/**
|
||||
* The affirmative provider or transport evidence for a failed trial, or null.
|
||||
* `api_error` is provider evidence by itself; `pre_turn_infra` only when its
|
||||
* evidence or detail names a 429/5xx, rate limit, overload or connection reset
|
||||
* (a crash or module-load failure is a harness defect, never an outage).
|
||||
*/
|
||||
export function providerEvidence(t: BarTrial): string | null {
|
||||
if (t.passed && !t.contract) return null;
|
||||
const text = [t.failureEvidence, t.failureDetail].filter(Boolean).join(' ');
|
||||
if (t.failureCause === 'api_error') return text || 'failure_cause api_error';
|
||||
if (t.failureCause === 'pre_turn_infra' && PROVIDER_TEXT.test(text)) return text;
|
||||
return null;
|
||||
}
|
||||
|
||||
/** The class a failed trial gets without review: provider on affirmative evidence; timeout or hang from its cause. */
|
||||
export function machineClass(t: BarTrial): FailureClass | null {
|
||||
if (providerEvidence(t)) return 'provider';
|
||||
if (t.failureCause === 'provider_stall') return 'hang';
|
||||
if (t.failureCause && TIMEOUT_CAUSES.has(t.failureCause)) return 'timeout';
|
||||
return null;
|
||||
}
|
||||
|
||||
const CITATION = /[\w./-]+:\d+/;
|
||||
|
||||
/** Why an agent classification is refused, or null when it is accepted. */
|
||||
export function classificationProblem(t: BarTrial | undefined, kind: BarKind, cls: string, evidence: string): string | null {
|
||||
if (!t) return 'no such trial in this measurement';
|
||||
if (trialPassed(kind, t)) return `trial ${t.trial} passed; only failed trials are classified`;
|
||||
if (!(FAILURE_CLASSES as readonly string[]).includes(cls)) return `class must be one of ${FAILURE_CLASSES.join(', ')}. Received: ${cls}`;
|
||||
if (!evidence.trim()) return 'every classification needs --evidence (what you read: a capture path:line, a failure line, a transcript line)';
|
||||
if (cls === 'provider' && !providerEvidence(t)) {
|
||||
return `provider needs affirmative provider or transport evidence (failure_cause api_error, or pre_turn_infra naming a 429/5xx, rate limit or connection reset); trial ${t.trial}'s failure_cause is ${t.failureCause ?? 'unrecorded'}`;
|
||||
}
|
||||
if ((QUALIFYING_CLASSES as readonly string[]).includes(cls) && t.failureCause && TIMEOUT_CAUSES.has(t.failureCause)) {
|
||||
return `trial ${t.trial} failed on ${t.failureCause}: a timeout or hang never qualifies; it goes to a fix round`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Each failed trial's effective class: its latest agent record, else its machine class. */
|
||||
export function effectiveClasses(trials: readonly BarTrial[], records: readonly Classification[]): Map<number, Classification> {
|
||||
const out = new Map<number, Classification>();
|
||||
for (const t of trials) {
|
||||
const cls = machineClass(t);
|
||||
if (cls) out.set(t.trial, { trial: t.trial, class: cls, evidence: providerEvidence(t) ?? t.failureCause ?? '', by: 'machine', at: '' });
|
||||
}
|
||||
for (const r of records) if (r.by === 'agent') out.set(r.trial, r);
|
||||
return out;
|
||||
}
|
||||
|
||||
const qualifies = (c: Classification) => (QUALIFYING_CLASSES as readonly string[]).includes(c.class) && (c.class !== 'model-miss' || CITATION.test(c.evidence));
|
||||
const fixRound = (c: Classification) => (FIX_ROUND_CLASSES as readonly string[]).includes(c.class);
|
||||
|
||||
function summarize(kind: BarKind, set: TrialSet, trials: BarTrial[]): SetSummary {
|
||||
const providerFailures = trials.filter(t => !trialPassed(kind, t) && providerEvidence(t)).length;
|
||||
return { set, trials: trials.length, passes: trials.filter(t => trialPassed(kind, t)).length, providerFailures,
|
||||
void: trials.length > 0 && providerFailures * VOID_SHARE.den > trials.length * VOID_SHARE.num };
|
||||
}
|
||||
|
||||
/** The counted batch of a pair (first try, its one redispatch), or a void/incomplete stop. */
|
||||
function countedBatch(n: number, first: SetSummary, retry: SetSummary, firstTrials: BarTrial[], retryTrials: BarTrial[], label: string):
|
||||
{ trials: BarTrial[] } | { stop: Pick<BarOutcome, 'decision' | 'next' | 'reason'> } {
|
||||
const use = first.void ? retry : first;
|
||||
if (use.trials === 0) return { stop: { decision: 'void', next: 'redispatch', reason: `${label} batch is void: ${first.providerFailures} of ${first.trials} trials failed on provider evidence (> 30%); redispatch it once` } };
|
||||
if (use.void) return { stop: { decision: 'void', next: 'stop', reason: `${label} batch and its one redispatch are both void (provider evidence in ${first.providerFailures}/${first.trials} and ${retry.providerFailures}/${retry.trials}); stop with a named infrastructure red` } };
|
||||
if (use.trials < n) return { stop: { decision: 'incomplete', next: 'stop', reason: `${label} batch ran ${use.trials} of ${n} trials` } };
|
||||
return { trials: first.void ? retryTrials : firstTrials };
|
||||
}
|
||||
|
||||
/**
|
||||
* The bar's decision over a measurement's trials and its classification
|
||||
* records (append-only; the latest agent record for a trial wins).
|
||||
*/
|
||||
export function decide(kind: BarKind, n: number, trials: readonly BarTrial[], records: readonly Classification[] = []): BarOutcome {
|
||||
const of = (set: TrialSet) => trials.filter(t => t.set === set);
|
||||
const sets = (['initial', 'redispatch', 'extend', 'extend-redispatch'] as const).map(set => summarize(kind, set, of(set)));
|
||||
const [initial, redispatch, extend, extendRedispatch] = sets as [SetSummary, SetSummary, SetSummary, SetSummary];
|
||||
const reported = sets.filter(s => s.trials > 0);
|
||||
const base = countedBatch(n, initial, redispatch, of('initial'), of('redispatch'), 'the first');
|
||||
const empty = { passes: 0, pooled: false, sets: reported, needsClassify: [] as number[] };
|
||||
if ('stop' in base) return { ...empty, thresholds: barThresholds(n), ...base.stop };
|
||||
const pooled = extend.trials > 0;
|
||||
const ext = pooled ? countedBatch(n, extend, extendRedispatch, of('extend'), of('extend-redispatch'), 'the extension') : { trials: [] };
|
||||
if ('stop' in ext) return { ...empty, pooled, thresholds: barThresholds(2 * n), ...ext.stop };
|
||||
const counted = [...base.trials, ...ext.trials];
|
||||
const thresholds = barThresholds(counted.length);
|
||||
const passes = counted.filter(t => trialPassed(kind, t)).length;
|
||||
const failed = counted.filter(t => !trialPassed(kind, t));
|
||||
const classes = effectiveClasses(failed, records);
|
||||
const result = (decision: BarDecision, next: BarNext, reason: string, needsClassify: number[] = []): BarOutcome =>
|
||||
({ decision, next, passes, thresholds, pooled, sets: reported, needsClassify, reason });
|
||||
const count = `${passes}/${counted.length}${pooled ? ' pooled' : ''}`;
|
||||
const contract = failed.find(t => t.contract);
|
||||
if (contract) return result('BELOW', 'fix', `${count}; trial ${contract.trial} violated a contract: a fix round, whatever the count`);
|
||||
if (passes >= thresholds.strict) return result('MEETS', 'none', `${count} meets the strict bar (${thresholds.strict}/${counted.length})`);
|
||||
if (passes < (pooled ? thresholds.qualified : thresholds.floor)) {
|
||||
return result('BELOW', 'fix', `${count} is under the ${pooled ? `qualified pooled bar (${thresholds.qualified}/${counted.length})` : `extend floor (${thresholds.floor}/${counted.length})`}: a fix round`);
|
||||
}
|
||||
const unclassified = failed.filter(t => !classes.has(t.trial)).map(t => t.trial);
|
||||
if (unclassified.length) return result('needs-classify', 'classify', `${count} is below strict; classify trial(s) ${unclassified.map(t => `t${t}`).join(', ')}, then decide`, unclassified);
|
||||
const fix = failed.find(t => fixRound(classes.get(t.trial)!));
|
||||
if (fix) return result('BELOW', 'fix', `${count}; trial ${fix.trial} is ${classes.get(fix.trial)!.class}, which never qualifies: a fix round`);
|
||||
if (passes >= thresholds.qualified && failed.every(t => qualifies(classes.get(t.trial)!))) {
|
||||
return result('MEETS-qualified', 'none', `${count} meets the qualified bar (${thresholds.qualified}/${counted.length}); every red qualifies (${failed.map(t => `t${t.trial} ${classes.get(t.trial)!.class}`).join(', ')})`);
|
||||
}
|
||||
if (pooled) return result('BELOW', 'fix', `${count} does not meet the pooled bar (${thresholds.strict} strict, or ${thresholds.qualified} with every red qualifying); there is no third batch: a fix round`);
|
||||
return result('EXTEND', 'extend', `${count} is at or above the extend floor (${thresholds.floor}/${counted.length}) without meeting a bar: run ${n} more trials on identical inputs, then decide once on the pooled ${2 * n}`);
|
||||
}
|
||||
|
||||
/** The first identity field that differs between two batches, or null when they may pool. */
|
||||
export function identityMismatch(a: Record<string, string>, b: Record<string, string>): string | null {
|
||||
for (const key of [...new Set([...Object.keys(a), ...Object.keys(b)])].sort()) {
|
||||
if (a[key] !== b[key]) return `${key} changed (${a[key] ?? 'absent'} -> ${b[key] ?? 'absent'})`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,352 @@
|
||||
/**
|
||||
* ship-measure sweep — the weekly off-ship qualification sweep on main
|
||||
* (approved by Garry 2026-10-06; docs/TESTING_INTERNALS.md#ship-measure-sweep).
|
||||
*
|
||||
* With ~130 gate verdicts, an all-green gate is unlikely even when every case
|
||||
* is healthy, so flaky cases are measured on main before any ship hits them.
|
||||
* The sweep reads eval:pass-rates history (evals-periodic.yml on main plus the
|
||||
* branch census runs that pool into it), ranks the gate cases with a failed
|
||||
* trial by how much each costs the gate's all-green probability Π(1 − p_i)
|
||||
* (p_i: its red-verdict rate shrunk toward the pooled rate, as
|
||||
* `eval:pass-rates --reds` computes it; ties broken by the lower per-trial
|
||||
* pass rate), and measures the top K at their kind's N on the
|
||||
* checked-out head with the same runner and measurement bar /ship uses
|
||||
* (scripts/ship-measure.ts, scripts/lib/measure-bar.ts).
|
||||
*
|
||||
* Spend: ship_measure_sweep_budget_usd (default $150) is a weekly cap enforced
|
||||
* through the same admission budget as /ship's loop, shared across the
|
||||
* sweep's cases and every sweep report of the last 7 days found under --prior
|
||||
* (CI downloads the earlier runs' reports there) and the local sweep root.
|
||||
* When the cap is exhausted the sweep stops and lists what it skipped.
|
||||
*
|
||||
* Every trial is diagnostic: it never becomes a verdict and never enters
|
||||
* EVAL_POLICY pooling or series history. The sweep never pushes, commits or
|
||||
* opens a pull request; an agent fixes from its report.
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { CASE_QUARANTINE } from '../test/helpers/periodic-exclude-data';
|
||||
import { E2E_KINDS, E2E_TIERS, LLM_JUDGE_TOUCHFILES } from '../test/helpers/touchfiles-data';
|
||||
import { parseCliFlags } from './lib/shard-engine';
|
||||
import { RED_RATE_SHRINK, verdictsOf } from './lib/eval-history';
|
||||
import { trialPassed } from './lib/measure-bar';
|
||||
import { repoSlug } from './lib/ci-history';
|
||||
import { loadPassRateHistory, type TrialRecord as HistoryRecord } from './eval-flake-rank';
|
||||
import {
|
||||
caseSelectionPreflight, caseSlug, commandRunner, gstackRunner, kindPlan, measureCase, measurementIdentity, readMeasureConfig,
|
||||
type MeasureConfig, type MeasureKind, type Measurement, type SpendPool, type TrialRunner,
|
||||
} from './ship-measure';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const WEEK_MS = 7 * 24 * 3600_000;
|
||||
export const SWEEP_SCHEMA = 'gstack-ship-measure-sweep/1';
|
||||
|
||||
export interface SweepCandidate {
|
||||
case: string;
|
||||
kind: MeasureKind;
|
||||
/** Red-verdict rate over gate history, shrunk toward the pooled rate (K = RED_RATE_SHRINK, as eval:pass-rates --reds). */
|
||||
redRate: number;
|
||||
reds: number;
|
||||
verdicts: number;
|
||||
/** Per-trial pass rate over the history window, or null without scored trials. */
|
||||
passRate: number | null;
|
||||
scoredTrials: number;
|
||||
/** All-green probability gained if this case never went red: P / (1 − p_i) − P. */
|
||||
contribution: number;
|
||||
/** Mean known cost per trial from history (else the gate-wide mean), or null. */
|
||||
costPerTrialUsd: number | null;
|
||||
}
|
||||
|
||||
export interface SweepPlan {
|
||||
candidates: SweepCandidate[];
|
||||
picked: SweepCandidate[];
|
||||
/** All-green probability over the latest census's gate verdicts, from history. */
|
||||
allGreen: number | null;
|
||||
censusCases: string[];
|
||||
pooledRate: number;
|
||||
/** Per-case shrunk red rates for every case in the latest census. */
|
||||
rates: Record<string, number>;
|
||||
}
|
||||
|
||||
const liveCase = (id: string) => Object.hasOwn(E2E_TIERS, id) || Object.hasOwn(LLM_JUDGE_TOUCHFILES, id);
|
||||
export const sweepKind = (id: string): MeasureKind => (E2E_KINDS[id] as MeasureKind | undefined) ?? (Object.hasOwn(E2E_TIERS, id) ? 'rule' : 'judge');
|
||||
|
||||
/**
|
||||
* Rank the gate cases history flags: every live, non-quarantined case of the
|
||||
* latest gate census with a failed trial in the window, by its contribution
|
||||
* to the all-green probability, then by the lower per-trial pass rate. A red
|
||||
* verdict here is a lane-failing verdict with at least one failed trial (an
|
||||
* INCOMPLETE record-count verdict is a reporting fault, not a flake). Picks
|
||||
* the top k.
|
||||
*/
|
||||
export function planSweep(records: readonly HistoryRecord[], k: number, opts: { quarantine?: Record<string, unknown> } = {}): SweepPlan {
|
||||
const quarantine = opts.quarantine ?? CASE_QUARANTINE;
|
||||
const gate = records.filter(r => r.tier === 'gate' && r.policy_version >= 1);
|
||||
const verdicts = verdictsOf(gate).filter(v => v.status !== 'SKIPPED');
|
||||
const red = (v: (typeof verdicts)[number]) => v.failsLane && v.trials.some(t => t.outcome === 'failed');
|
||||
const pooledRate = verdicts.length ? verdicts.filter(red).length / verdicts.length : 0;
|
||||
const byCase = new Map<string, { verdicts: number; reds: number }>();
|
||||
for (const v of verdicts) {
|
||||
const e = byCase.get(v.case) ?? { verdicts: 0, reds: 0 };
|
||||
e.verdicts++; if (red(v)) e.reds++;
|
||||
byCase.set(v.case, e);
|
||||
}
|
||||
const rates: Record<string, number> = Object.fromEntries([...byCase].map(([id, e]) => [id, (e.reds + RED_RATE_SHRINK * pooledRate) / (e.verdicts + RED_RATE_SHRINK)]));
|
||||
const latestRun = verdicts.map(v => v.run).filter(run => run !== 'local').sort().at(-1) ?? 'local';
|
||||
const censusCases = [...new Set(verdicts.filter(v => v.run === latestRun).map(v => v.case))].sort();
|
||||
const allGreen = censusCases.length ? censusCases.reduce((p, id) => p * (1 - rates[id]!), 1) : null;
|
||||
const scored = gate.filter(r => r.execution === 'executed' && (r.outcome === 'passed' || r.outcome === 'failed'));
|
||||
const knownCost = (r: HistoryRecord) => r.execution === 'executed' && r.cost_known !== false && r.cost_usd > 0;
|
||||
const mean = (xs: number[]) => xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : null;
|
||||
const gateCost = mean(records.filter(r => r.tier === 'gate' && knownCost(r)).map(r => r.cost_usd));
|
||||
const candidates: SweepCandidate[] = [];
|
||||
for (const id of censusCases) {
|
||||
if (!liveCase(id) || Object.hasOwn(quarantine, id)) continue;
|
||||
const trials = scored.filter(r => r.case === id);
|
||||
const failedTrials = trials.filter(r => r.outcome === 'failed').length;
|
||||
const e = byCase.get(id)!;
|
||||
if (failedTrials === 0) continue;
|
||||
candidates.push({
|
||||
case: id, kind: sweepKind(id), redRate: rates[id]!, reds: e.reds, verdicts: e.verdicts,
|
||||
passRate: trials.length ? (trials.length - failedTrials) / trials.length : null, scoredTrials: trials.length,
|
||||
contribution: allGreen === null ? 0 : allGreen / (1 - rates[id]!) - allGreen,
|
||||
costPerTrialUsd: mean(records.filter(r => r.case === id && knownCost(r)).map(r => r.cost_usd)) ?? gateCost,
|
||||
});
|
||||
}
|
||||
candidates.sort((a, b) => b.contribution - a.contribution || b.redRate - a.redRate || (a.passRate ?? 1) - (b.passRate ?? 1) || (a.case < b.case ? -1 : 1));
|
||||
return { candidates, picked: candidates.slice(0, k), allGreen, censusCases, pooledRate, rates };
|
||||
}
|
||||
|
||||
/**
|
||||
* A measured case's predicted red-verdict rate: its measured per-trial failure
|
||||
* fraction shrunk toward the pooled rate with the same K as history, and for a
|
||||
* behavior case the 3-trial panel's red risk 1 − P(at least 2 of 3 pass).
|
||||
*/
|
||||
export function measuredRedRate(kind: MeasureKind, failures: number, trials: number, pooledRate: number): number {
|
||||
const q = (failures + RED_RATE_SHRINK * pooledRate) / (trials + RED_RATE_SHRINK);
|
||||
return kind === 'behavior' ? 1 - ((1 - q) ** 3 + 3 * (1 - q) ** 2 * q) : q;
|
||||
}
|
||||
|
||||
/** Π(1 − p_i) over the census cases, with measured cases' rates replacing history's. */
|
||||
export function predictedAllGreen(plan: SweepPlan, measured: Record<string, number>): number | null {
|
||||
if (plan.allGreen === null) return null;
|
||||
return plan.censusCases.reduce((p, id) => p * (1 - (measured[id] ?? plan.rates[id] ?? plan.pooledRate)), 1);
|
||||
}
|
||||
|
||||
export interface SweepRow {
|
||||
rank: number;
|
||||
case: string;
|
||||
kind: MeasureKind;
|
||||
historyPassRate: number | null;
|
||||
historyTrials: number;
|
||||
reds: number;
|
||||
verdicts: number;
|
||||
redRate: number;
|
||||
status: 'planned' | 'measured' | 'budget_exhausted' | 'skipped' | 'error';
|
||||
decision: string;
|
||||
measuredPasses: number | null;
|
||||
measuredTrials: number | null;
|
||||
estimatedUsd: number | null;
|
||||
actualUsd: number;
|
||||
costUnknownTrials: number;
|
||||
captures: string | null;
|
||||
note: string;
|
||||
}
|
||||
|
||||
export interface SweepReport {
|
||||
schema: typeof SWEEP_SCHEMA;
|
||||
label: 'diagnostic';
|
||||
verdict: null;
|
||||
sweepId: string;
|
||||
startedAt: string;
|
||||
finishedAt: string | null;
|
||||
ref: string;
|
||||
sha: string;
|
||||
dryRun: boolean;
|
||||
k: number;
|
||||
capUsd: number;
|
||||
priorWeekUsd: number;
|
||||
/** Charged against the weekly cap by this sweep: known costs plus the estimate for trials whose cost is unknown. */
|
||||
chargedUsd: number;
|
||||
historyScope: string;
|
||||
allGreenBefore: number | null;
|
||||
allGreenAfter: number | null;
|
||||
censusVerdicts: number;
|
||||
rows: SweepRow[];
|
||||
}
|
||||
|
||||
/** Spend charged by sweep reports under dirs started within the last 7 days (deduplicated by sweep id). */
|
||||
export function weeklySpent(dirs: readonly string[], now: number, excludeId?: string): number {
|
||||
const seen = new Map<string, number>();
|
||||
for (const dir of dirs) {
|
||||
if (!fs.existsSync(dir)) continue;
|
||||
for (const name of fs.readdirSync(dir, { recursive: true }) as string[]) {
|
||||
if (path.basename(name) !== 'sweep-report.json') continue;
|
||||
let report: SweepReport;
|
||||
try { report = JSON.parse(fs.readFileSync(path.join(dir, name), 'utf8')) as SweepReport; } catch { continue; }
|
||||
if (report.schema !== SWEEP_SCHEMA || report.sweepId === excludeId || report.dryRun) continue;
|
||||
const started = Date.parse(report.startedAt);
|
||||
if (!Number.isFinite(started) || now - started > WEEK_MS || started > now) continue;
|
||||
seen.set(report.sweepId, Math.max(seen.get(report.sweepId) ?? 0, report.chargedUsd));
|
||||
}
|
||||
}
|
||||
return [...seen.values()].reduce((a, b) => a + b, 0);
|
||||
}
|
||||
|
||||
const pct = (x: number | null) => x === null ? 'n/a' : `${Math.round(x * 1000) / 10}%`;
|
||||
const prob = (x: number | null) => x === null || x >= 0.001 || x === 0 ? pct(x) : `${(x * 100).toPrecision(2)}%`;
|
||||
const money = (x: number | null) => x === null ? 'unknown' : `$${x.toFixed(2)}`;
|
||||
const cell = (text: string) => text.replace(/\|/g, '/').replace(/\n/g, ' ');
|
||||
|
||||
export function formatSweepReport(r: SweepReport): string {
|
||||
const rows = r.rows.map(row => `| ${row.rank} | ${cell(row.case)} | ${row.kind} | ${pct(row.historyPassRate)} of ${row.historyTrials} trials; ${row.reds}/${row.verdicts} red verdicts (rate ${pct(row.redRate)}) | `
|
||||
+ `${row.measuredTrials === null ? '—' : `${row.measuredPasses}/${row.measuredTrials}`} | ${cell(row.decision)} | ${money(row.estimatedUsd)} / ${money(row.actualUsd)}${row.costUnknownTrials ? ` (+${row.costUnknownTrials} unknown)` : ''} | ${row.captures ? `\`${cell(row.captures)}\`` : '—'} |`);
|
||||
const noted = r.rows.filter(row => row.status !== 'measured');
|
||||
return [
|
||||
`## ship-measure sweep${r.dryRun ? ' (dry run: nothing ran, $0)' : ''}: diagnostic measurements, not verdicts`,
|
||||
'',
|
||||
`Ref ${r.ref} at ${r.sha.slice(0, 12)}; K=${r.k}; weekly cap $${r.capUsd} (charged earlier this week $${r.priorWeekUsd.toFixed(2)}, this sweep $${r.chargedUsd.toFixed(2)}). History: ${r.historyScope}.`,
|
||||
'',
|
||||
`Predicted gate all-green probability (approximation, assumes independent verdicts, over the latest census's ${r.censusVerdicts} gate verdicts): before ${prob(r.allGreenBefore)} → after ${prob(r.allGreenAfter)}${r.dryRun ? ' (no measurements yet)' : ' (measured cases use this sweep\'s rates)'}.`,
|
||||
'',
|
||||
'| # | Case | Kind | History rate | Measured | Decision | Cost (est / actual) | Captures |',
|
||||
'|---|---|---|---|---|---|---|---|',
|
||||
...(rows.length ? rows : ['| — | no case in the latest gate census has a failed trial or red verdict | | | | | | |']),
|
||||
...(noted.length ? ['', r.dryRun ? 'Plan:' : 'Skipped or stopped:', ...noted.map(row => `- ${row.case}: ${row.note}`)] : []),
|
||||
'',
|
||||
'Every trial is diagnostic: it never becomes a verdict and never enters pass-rate history. The sweep never pushes or opens a pull request. '
|
||||
+ 'Fix a BELOW case at its cause, classify a needs-classify case (`ship-measure classify --out <captures root> --case <id> --round baseline ...`), and measure again on the fix.',
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
export interface SweepOptions {
|
||||
records: readonly HistoryRecord[];
|
||||
historyScope: string;
|
||||
k: number;
|
||||
capUsd: number;
|
||||
config: MeasureConfig;
|
||||
outDir: string;
|
||||
priorDirs: string[];
|
||||
dryRun: boolean;
|
||||
parallel: number;
|
||||
ref: string;
|
||||
sha: string;
|
||||
now?: number;
|
||||
/** One runner for every case (default: gstack's single-case runner after a selection preflight). */
|
||||
runner?: TrialRunner;
|
||||
runnerId?: string;
|
||||
preflight?: (caseId: string) => { ok: boolean; detail: string };
|
||||
identity?: (kind: MeasureKind, trials: number) => Record<string, string>;
|
||||
evalDir?: string;
|
||||
log?: (line: string) => void;
|
||||
}
|
||||
|
||||
/** Plan, measure in rank order under the weekly cap, and write sweep-report.json/.md after every case. */
|
||||
export async function runSweep(o: SweepOptions): Promise<SweepReport> {
|
||||
const log = o.log ?? ((line: string) => console.log(line));
|
||||
const now = o.now ?? Date.now();
|
||||
const sweepId = new Date(now).toISOString().replace(/[:.]/g, '-');
|
||||
const dir = path.join(o.outDir, `sweep-${sweepId}`);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
const plan = planSweep(o.records, o.k);
|
||||
const priorWeekUsd = weeklySpent([...o.priorDirs, o.outDir], now, sweepId);
|
||||
const pool: SpendPool = { label: `weekly sweep cap`, capUsd: o.capUsd, spentUsd: priorWeekUsd };
|
||||
const report: SweepReport = {
|
||||
schema: SWEEP_SCHEMA, label: 'diagnostic', verdict: null, sweepId, startedAt: new Date(now).toISOString(), finishedAt: null, ref: o.ref, sha: o.sha,
|
||||
dryRun: o.dryRun, k: o.k, capUsd: o.capUsd, priorWeekUsd, chargedUsd: 0, historyScope: o.historyScope,
|
||||
allGreenBefore: plan.allGreen, allGreenAfter: plan.allGreen, censusVerdicts: plan.censusCases.length, rows: [],
|
||||
};
|
||||
const save = () => {
|
||||
report.chargedUsd = Math.max(0, pool.spentUsd - priorWeekUsd);
|
||||
fs.writeFileSync(path.join(dir, 'sweep-report.json'), `${JSON.stringify(report, null, 2)}\n`);
|
||||
fs.writeFileSync(path.join(dir, 'sweep-report.md'), `${formatSweepReport(report)}\n`);
|
||||
};
|
||||
const measuredRates: Record<string, number> = {};
|
||||
for (const [index, c] of plan.picked.entries()) {
|
||||
const n = kindPlan(c.kind, o.config).trials;
|
||||
const row: SweepRow = {
|
||||
rank: index + 1, case: c.case, kind: c.kind, historyPassRate: c.passRate, historyTrials: c.scoredTrials, reds: c.reds, verdicts: c.verdicts, redRate: c.redRate,
|
||||
status: 'planned', decision: 'planned', measuredPasses: null, measuredTrials: null,
|
||||
estimatedUsd: c.costPerTrialUsd === null ? null : c.costPerTrialUsd * n, actualUsd: 0, costUnknownTrials: 0, captures: null, note: '',
|
||||
};
|
||||
report.rows.push(row);
|
||||
if (o.dryRun) { row.note = `would measure ${n} trials at ${c.costPerTrialUsd === null ? 'an unknown cost (one calibration trial first)' : `${money(c.costPerTrialUsd)}/trial`}`; save(); continue; }
|
||||
if (pool.spentUsd >= pool.capUsd) {
|
||||
Object.assign(row, { status: 'skipped', decision: 'skipped', note: `weekly cap $${o.capUsd} exhausted ($${pool.spentUsd.toFixed(2)} charged in the last 7 days)` });
|
||||
save();
|
||||
continue;
|
||||
}
|
||||
const selected = (o.preflight ?? caseSelectionPreflight)(c.case);
|
||||
if (!selected.ok) { Object.assign(row, { status: 'skipped', decision: 'skipped', note: `selection preflight failed (no paid call made): ${selected.detail}` }); save(); continue; }
|
||||
let m: Measurement;
|
||||
try {
|
||||
m = await measureCase({
|
||||
caseId: c.case, kind: c.kind, round: 'baseline', config: o.config, outDir: dir, parallel: o.parallel, costPerTrialUsd: c.costPerTrialUsd, approved: true,
|
||||
runner: o.runner ?? gstackRunner(), pool, log,
|
||||
identity: () => (o.identity ?? ((kind, trials) => measurementIdentity(kind, trials, o.runnerId ?? 'gstack')))(c.kind, n),
|
||||
...(o.evalDir ? { evalDir: o.evalDir } : {}),
|
||||
});
|
||||
} catch (error) {
|
||||
Object.assign(row, { status: 'error', decision: 'error', note: (error as Error).message });
|
||||
save();
|
||||
continue;
|
||||
}
|
||||
const captures = path.relative(o.outDir, path.join(dir, caseSlug(c.case), 'baseline'));
|
||||
const decided = m.status === 'measured' && m.decision !== 'void' && m.decision !== 'incomplete';
|
||||
Object.assign(row, {
|
||||
status: m.status === 'measured' ? 'measured' : 'budget_exhausted', decision: m.status === 'measured' ? m.decision : m.status.replace('_', ' '),
|
||||
...(decided ? { measuredPasses: m.passes, measuredTrials: m.counted } : { measuredPasses: m.trials.filter(t => trialPassed(c.kind, t)).length, measuredTrials: m.trials.length }),
|
||||
actualUsd: m.actualUsd, costUnknownTrials: m.costUnknownTrials, captures,
|
||||
note: m.status === 'measured' ? m.decisionReason : m.reason ?? '',
|
||||
});
|
||||
if (decided) {
|
||||
measuredRates[c.case] = measuredRedRate(c.kind, m.counted - m.passes, m.counted, plan.pooledRate);
|
||||
report.allGreenAfter = predictedAllGreen(plan, measuredRates);
|
||||
}
|
||||
save();
|
||||
}
|
||||
report.finishedAt = new Date().toISOString();
|
||||
save();
|
||||
log(formatSweepReport(report));
|
||||
log(`[ship-measure] sweep report: ${path.join(dir, 'sweep-report.md')}`);
|
||||
return report;
|
||||
}
|
||||
|
||||
function gitText(args: string[]): string {
|
||||
const r = spawnSync('git', args, { cwd: ROOT, encoding: 'utf8', timeout: 30_000 });
|
||||
return r.status === 0 ? r.stdout.trim() : 'unknown';
|
||||
}
|
||||
|
||||
/** `ship-measure sweep` CLI. Exits 0 when the sweep completed (whatever it measured), 1 when history was unavailable, 4 on usage errors. */
|
||||
export async function sweepMain(argv: string[], env: NodeJS.ProcessEnv = process.env): Promise<number> {
|
||||
const flags: Record<string, string> = {};
|
||||
const historyDirs: string[] = [];
|
||||
const priorDirs: string[] = [];
|
||||
const value = (name: string) => (next: () => string | undefined) => { const v = next(); if (v === undefined) throw new Error(`${name} needs a value`); flags[name] = v; };
|
||||
const list = (into: string[], name: string) => (next: () => string | undefined) => { const v = next(); if (v === undefined) throw new Error(`${name} needs a value`); into.push(path.resolve(v)); };
|
||||
parseCliFlags(argv, {
|
||||
...Object.fromEntries(['--k', '--cap-usd', '--command', '--jobs', '--out', '--runs', '--repo'].map(name => [name, value(name)])),
|
||||
'--history-dir': list(historyDirs, '--history-dir'), '--prior': list(priorDirs, '--prior'), '--dry-run': () => { flags['--dry-run'] = '1'; },
|
||||
}, 'Usage: ship-measure sweep [--k K] [--cap-usd USD] [--dry-run] [--history-dir DIR]... [--prior DIR]... [--runs N] [--command \'CMD {case}\'] [--jobs N] [--out DIR]');
|
||||
const config = readMeasureConfig();
|
||||
const int = (raw: string | undefined, flag: string, fallback: number) => {
|
||||
if (raw === undefined) return fallback;
|
||||
if (!/^[1-9][0-9]*$/.test(raw)) throw new Error(`${flag} needs a positive integer. Received: ${raw}`);
|
||||
return Number(raw);
|
||||
};
|
||||
const capUsd = flags['--cap-usd'] === undefined ? config.sweepBudgetUsd : Number(flags['--cap-usd']);
|
||||
if (!Number.isFinite(capUsd) || capUsd <= 0) throw new Error(`--cap-usd needs a positive amount in USD. Received: ${flags['--cap-usd']}`);
|
||||
const repo = flags['--repo'] ?? repoSlug(ROOT);
|
||||
const history = loadPassRateHistory({ repo, workflow: 'evals-periodic.yml', branch: 'main', runsLimit: int(flags['--runs'], '--runs', 10), dirs: historyDirs, sinceDays: 60 });
|
||||
if (history.historyError) { console.error(`[ship-measure] sweep: pass-rate history unavailable (${history.historyError}); nothing measured`); return 1; }
|
||||
const outDir = path.resolve(flags['--out'] ?? path.join(process.cwd(), '.context', 'ship-measure'));
|
||||
const report = await runSweep({
|
||||
records: history.records, historyScope: history.scope, k: int(flags['--k'], '--k', config.sweepCases), capUsd, config, outDir, priorDirs,
|
||||
dryRun: flags['--dry-run'] === '1', parallel: int(flags['--jobs'], '--jobs', Number(env.EVALS_JOBS) || 4),
|
||||
ref: env.GITHUB_REF_NAME ?? gitText(['rev-parse', '--abbrev-ref', 'HEAD']), sha: env.GITHUB_SHA ?? gitText(['rev-parse', 'HEAD']),
|
||||
...(flags['--command'] ? { runner: commandRunner(flags['--command']), runnerId: `command:${flags['--command']}`, preflight: () => ({ ok: true, detail: 'documented command' }) } : {}),
|
||||
});
|
||||
if (env.GITHUB_STEP_SUMMARY) fs.appendFileSync(env.GITHUB_STEP_SUMMARY, `${formatSweepReport(report)}\n`);
|
||||
return 0;
|
||||
}
|
||||
+285
-133
@@ -1,43 +1,53 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* ship-measure — /ship's measure-then-fix runner for a red eval case or a red
|
||||
* free-suite shard. Every trial it runs is DIAGNOSTIC: it never changes a
|
||||
* recorded verdict, its artifacts are labeled `diagnostic` with `verdict:
|
||||
* null`, and they live under `.context/ship-measure/`, never in the project
|
||||
* eval dir that pass-rate history and CI uploads read. The lane verdict still
|
||||
* comes from the one full gate run /ship makes after the case measures at or
|
||||
* above target (docs/TESTING_INTERNALS.md#ship-measure).
|
||||
* free-suite shard, and the weekly off-ship qualification sweep. Every trial
|
||||
* it runs is DIAGNOSTIC: it never changes a recorded verdict, its artifacts
|
||||
* are labeled `diagnostic` with `verdict: null`, and they live under
|
||||
* `.context/ship-measure/`, never in the project eval dir that pass-rate
|
||||
* history and CI uploads read. The lane verdict still comes from the one full
|
||||
* gate run /ship makes after the case measures at or above the bar
|
||||
* (docs/TESTING_INTERNALS.md#ship-measure).
|
||||
*
|
||||
* Per-kind aggregation (config keys in bin/gstack-config, CEO-9 defaults):
|
||||
* rule N trials (ship_measure_rule_trials, 10); meets at >= 90% of N.
|
||||
* behavior P panels of 3 (ship_measure_behavior_panels, 4); every panel
|
||||
* passes at its own 2 of 3 with no contract violation, and >= 90%
|
||||
* of all 3P trials pass (11 of 12).
|
||||
* judge O outputs (ship_measure_judge_outputs, 10), each scored by its
|
||||
* 3-sample panel (passes at 2 of 3); meets at >= 90% of O.
|
||||
* Trials per batch (config keys in bin/gstack-config): rule
|
||||
* ship_measure_rule_trials (10); behavior ship_measure_behavior_panels panels
|
||||
* of 3 (4, so 12 trials); judge ship_measure_judge_outputs outputs (10), each
|
||||
* scored by its median 3-sample panel. The decision is the measurement bar in
|
||||
* scripts/lib/measure-bar.ts: MEETS at 9/10, MEETS-qualified at 8/10 with
|
||||
* every red in a qualifying class, EXTEND at 7/10 (N more trials on identical
|
||||
* inputs, decided once on the pooled 2N), else BELOW (a fix round). A batch
|
||||
* with more than 30% provider-evidence failures is void and is redispatched
|
||||
* once.
|
||||
*
|
||||
* Spend: ship_measure_budget_usd is an estimated ADMISSION budget per red case
|
||||
* across the baseline and every repair round. Before each batch the runner
|
||||
* reserves the estimated cost of every concurrent trial and admits only what
|
||||
* fits beside what was already spent; actual costs are reconciled after the
|
||||
* batch. With no per-trial estimate it asks once ("estimate unknown"), then
|
||||
* runs one calibration trial alone. ship_measure_ask_per_trial_usd asks first
|
||||
* above that per-trial estimate. ship_measure_max_rounds caps repair rounds.
|
||||
* across the baseline, extension, redispatch and every repair round. Before
|
||||
* each batch the runner reserves the estimated cost of every concurrent trial
|
||||
* and admits only what fits beside what was already spent; actual costs are
|
||||
* reconciled after the batch. With no per-trial estimate it asks once
|
||||
* ("estimate unknown"), then runs one calibration trial alone.
|
||||
* ship_measure_ask_per_trial_usd asks first above that per-trial estimate.
|
||||
* ship_measure_max_rounds caps repair rounds.
|
||||
*
|
||||
* Usage:
|
||||
* bun run scripts/ship-measure.ts table
|
||||
* bun run scripts/ship-measure.ts measure --case ID --round baseline|round-N
|
||||
* [--kind rule|behavior|judge] [--command 'CMD {case}'] [--cost-per-trial USD]
|
||||
* [--approved] [--fix TEXT] [--jobs N] [--out DIR]
|
||||
* bun run scripts/ship-measure.ts classify --case ID --round R --trials 2,5 --class C --evidence TEXT [--out DIR]
|
||||
* bun run scripts/ship-measure.ts decide --case ID --round R [--out DIR]
|
||||
* bun run scripts/ship-measure.ts extend --case ID --round R [--command ..] [--cost-per-trial USD] [--jobs N] [--out DIR]
|
||||
* bun run scripts/ship-measure.ts skip --case ID --reason TEXT [--out DIR]
|
||||
* bun run scripts/ship-measure.ts report [--out DIR]
|
||||
* bun run scripts/ship-measure.ts free (--files a,b,... | --shard I) [--reruns N]
|
||||
* [--concurrency C] [--backend local|ubicloud] [--wall-cap SECS] [--out DIR]
|
||||
* bun run scripts/ship-measure.ts sweep [--k K] [--cap-usd USD] [--dry-run] [--history-dir DIR]...
|
||||
* [--prior DIR] [--command 'CMD {case}'] [--jobs N] [--out DIR] (scripts/ship-measure-sweep.ts)
|
||||
*
|
||||
* measure exits 0 at or above target, 1 below target, 2 when it needs the
|
||||
* user's approval (nothing ran), 3 on a named-red stop (budget exhausted or
|
||||
* round limit), 4 on a usage or setup error. free exits 0 when every
|
||||
* completed rerun passed, else 1.
|
||||
* measure, extend, classify and decide exit 0 at MEETS or MEETS-qualified, 1
|
||||
* at BELOW (a fix round), 2 when it needs the user's approval (nothing ran), 3
|
||||
* on a named-red stop (budget exhausted, round limit, a void batch twice), 4 on
|
||||
* a usage, setup or refused-classification error, 5 at needs-classify, 6 at
|
||||
* EXTEND. free exits 0 when every completed rerun passed, else 1.
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
@@ -45,11 +55,16 @@ import * as path from 'node:path';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import { EVAL_POLICY } from '../test/helpers/periodic-exclude-data';
|
||||
import { E2E_KINDS, E2E_TIERS, LLM_JUDGE_TOUCHFILES } from '../test/helpers/touchfiles-data';
|
||||
import { getProjectEvalDir, isFinalizedEvalResultFile } from '../test/helpers/eval-store';
|
||||
import { getClaudeCliVersion, getProjectEvalDir, isFinalizedEvalResultFile } from '../test/helpers/eval-store';
|
||||
import { parseCliFlags } from './lib/shard-engine';
|
||||
import { collectPaidTestFiles, paidSelectionEnv } from './lib/paid-select';
|
||||
import { DEFAULT_JOBS } from './lib/paid-types';
|
||||
import { privateFreeHome } from './lib/free-home-guard';
|
||||
import {
|
||||
FAILURE_CLASSES, barThresholds, classificationProblem, decide, identityMismatch,
|
||||
type BarDecision, type BarNext, type BarThresholds, type Classification, type FailureClass, type SetSummary, type TrialSet,
|
||||
} from './lib/measure-bar';
|
||||
import harnessManifest from './harness-version.json';
|
||||
import {
|
||||
TREE_MUTATING, assignFilesToShards, collectFreeTestFiles, fullSuiteJobs, loadFreeTestDurations, packShardsByDuration,
|
||||
runFreeShard, wallTimeoutForShard, type FreeShardOutcome,
|
||||
@@ -65,12 +80,16 @@ export const JUDGE_SAMPLES = EVAL_POLICY.judge.samples;
|
||||
export interface MeasureConfig {
|
||||
ruleTrials: number; behaviorPanels: number; judgeOutputs: number;
|
||||
askPerTrialUsd: number; budgetUsd: number; maxRounds: number; rerunBackend: RerunBackend;
|
||||
sweepCases: number; sweepBudgetUsd: number;
|
||||
}
|
||||
|
||||
export const MEASURE_DEFAULTS: MeasureConfig = {
|
||||
ruleTrials: 10, behaviorPanels: 4, judgeOutputs: 10, askPerTrialUsd: 2, budgetUsd: 25, maxRounds: 3, rerunBackend: 'local',
|
||||
sweepCases: 5, sweepBudgetUsd: 150,
|
||||
};
|
||||
|
||||
const USD_FIELDS = new Set<keyof MeasureConfig>(['askPerTrialUsd', 'budgetUsd', 'sweepBudgetUsd']);
|
||||
|
||||
function gstackConfigGet(key: string): string | undefined {
|
||||
const r = spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['get', key], { encoding: 'utf8', timeout: 10_000 });
|
||||
return r.status === 0 ? r.stdout.trim() : undefined;
|
||||
@@ -86,6 +105,8 @@ export function readMeasureConfig(configGet: (key: string) => string | undefined
|
||||
['budgetUsd', 'ship_measure_budget_usd', configGet('ship_measure_budget_usd')],
|
||||
['maxRounds', 'ship_measure_max_rounds', configGet('ship_measure_max_rounds')],
|
||||
['rerunBackend', 'ship_rerun_backend', configGet('ship_rerun_backend')],
|
||||
['sweepCases', 'ship_measure_sweep_cases', configGet('ship_measure_sweep_cases')],
|
||||
['sweepBudgetUsd', 'ship_measure_sweep_budget_usd', configGet('ship_measure_sweep_budget_usd')],
|
||||
];
|
||||
const config = { ...MEASURE_DEFAULTS };
|
||||
for (const [field, key, rawValue] of raw) {
|
||||
@@ -96,7 +117,7 @@ export function readMeasureConfig(configGet: (key: string) => string | undefined
|
||||
config.rerunBackend = value;
|
||||
continue;
|
||||
}
|
||||
const usd = field === 'askPerTrialUsd' || field === 'budgetUsd';
|
||||
const usd = USD_FIELDS.has(field);
|
||||
if (!(usd ? Number.isFinite(Number(value)) && Number(value) > 0 : /^[1-9][0-9]*$/.test(value))) {
|
||||
throw new Error(`${key} '${value}' is not a ${usd ? 'positive amount in USD' : 'positive integer'}. Fix: gstack-config set ${key} ${MEASURE_DEFAULTS[field]}`);
|
||||
}
|
||||
@@ -105,51 +126,48 @@ export function readMeasureConfig(configGet: (key: string) => string | undefined
|
||||
return config;
|
||||
}
|
||||
|
||||
const ninetyPercent = (n: number) => Math.ceil((n * 9) / 10);
|
||||
|
||||
export interface KindPlan {
|
||||
kind: MeasureKind;
|
||||
/** Scored units: trials (rule), panels (behavior), outputs (judge). */
|
||||
units: number;
|
||||
/** Behavior trials launch in panels of 3; rule and judge trials one at a time. */
|
||||
trialsPerUnit: number;
|
||||
/** Trials per batch (outputs for judge). */
|
||||
trials: number;
|
||||
/** Units that must pass. */
|
||||
unitTarget: number;
|
||||
/** Trials (outputs for judge) that must pass. */
|
||||
trialTarget: number;
|
||||
bar: BarThresholds;
|
||||
pooledBar: BarThresholds;
|
||||
}
|
||||
|
||||
export function kindPlan(kind: MeasureKind, config: MeasureConfig): KindPlan {
|
||||
if (kind === 'behavior') {
|
||||
const trials = config.behaviorPanels * PANEL.n;
|
||||
return { kind, units: config.behaviorPanels, trialsPerUnit: PANEL.n, trials, unitTarget: config.behaviorPanels, trialTarget: ninetyPercent(trials) };
|
||||
}
|
||||
const units = kind === 'judge' ? config.judgeOutputs : config.ruleTrials;
|
||||
return { kind, units, trialsPerUnit: 1, trials: units, unitTarget: ninetyPercent(units), trialTarget: ninetyPercent(units) };
|
||||
const trials = kind === 'behavior' ? config.behaviorPanels * PANEL.n : kind === 'judge' ? config.judgeOutputs : config.ruleTrials;
|
||||
return { kind, trialsPerUnit: kind === 'behavior' ? PANEL.n : 1, trials, bar: barThresholds(trials), pooledBar: barThresholds(2 * trials) };
|
||||
}
|
||||
|
||||
/** The per-kind table /ship prints before it runs anything. */
|
||||
export function formatKindTable(config: MeasureConfig): string {
|
||||
const rule = kindPlan('rule', config);
|
||||
const behavior = kindPlan('behavior', config);
|
||||
const judge = kindPlan('judge', config);
|
||||
const budget = `$${config.budgetUsd} total; asks above $${config.askPerTrialUsd}/trial or with no estimate`;
|
||||
const row = (kind: MeasureKind, label: string) => {
|
||||
const { trials: n, bar, pooledBar } = kindPlan(kind, config);
|
||||
const unit = kind === 'judge' ? ' outputs' : '';
|
||||
return `| ${kind} | ${label} | ${bar.strict} of ${n}${unit} | ${bar.qualified} of ${n}, every red qualifying | ${bar.floor} of ${n} (or ${bar.qualified} that does not qualify): ${n} more on identical inputs, then ${pooledBar.strict} of ${2 * n}, or ${pooledBar.qualified} of ${2 * n} qualified | ${budget} |`;
|
||||
};
|
||||
return [
|
||||
'| Kind | Trials | Pass bar | Panels | Budget per red case |',
|
||||
'|---|---|---|---|---|',
|
||||
`| rule | ${rule.trials} | at least ${rule.trialTarget} of ${rule.trials} trials | none | ${budget} |`,
|
||||
`| behavior | ${behavior.trials} | every panel at ${PANEL.k} of ${PANEL.n}, no contract violation, at least ${behavior.trialTarget} of ${behavior.trials} trials | ${behavior.units} of ${PANEL.n} trials | ${budget} |`,
|
||||
`| judge | ${judge.trials} outputs | at least ${judge.unitTarget} of ${judge.trials} outputs | ${JUDGE_SAMPLES} samples per output, passes at 2 of ${JUDGE_SAMPLES} | ${budget} |`,
|
||||
'| Kind | Trials per batch | MEETS | MEETS-qualified | EXTEND, then the pooled decision | Budget per red case |',
|
||||
'|---|---|---|---|---|---|',
|
||||
row('rule', String(config.ruleTrials)),
|
||||
row('behavior', `${config.behaviorPanels * PANEL.n} (${config.behaviorPanels} panels of ${PANEL.n})`),
|
||||
row('judge', `${config.judgeOutputs} outputs, each its median ${JUDGE_SAMPLES}-sample panel`),
|
||||
'',
|
||||
'Qualifying reds: provider (affirmative provider or transport evidence only), judge noise at the threshold, a model miss citing its evidence.',
|
||||
'Never qualifying (a fix round): a timeout, a hang, a regression, a contract violation, any known fixable cause. Below the EXTEND floor: BELOW, a fix round.',
|
||||
'A batch with more than 30% of trials failing on provider evidence is void and is redispatched once; both batches are reported. There is never a third batch.',
|
||||
`Repair rounds: at most ${config.maxRounds}. Free-suite rerun backend: ${config.rerunBackend}. Every trial is diagnostic and never changes a recorded verdict.`,
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
// ─── Trials and aggregation ────────────────────────────────────────────────
|
||||
// ─── Trials ────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface TrialResult {
|
||||
passed: boolean;
|
||||
/** A contract violation fails its unit at any count. */
|
||||
/** A contract violation fails the measurement at any count. */
|
||||
contract?: boolean;
|
||||
/** Billed cost; absent means unknown. */
|
||||
costUsd?: number;
|
||||
@@ -157,38 +175,21 @@ export interface TrialResult {
|
||||
samples?: boolean[];
|
||||
failureCause?: string;
|
||||
failureDetail?: string;
|
||||
/** The eval store's one-line failure_cause_evidence. */
|
||||
failureEvidence?: string;
|
||||
}
|
||||
|
||||
export interface TrialRequest { caseId: string; kind: MeasureKind; round: string; trial: number; dir: string }
|
||||
export type TrialRunner = (request: TrialRequest) => Promise<TrialResult>;
|
||||
|
||||
export interface TrialRecord extends TrialResult { trial: number; unit: number; batch: number; dir: string }
|
||||
|
||||
export interface Aggregate { unitPasses: number; trialPasses: number; complete: boolean; meets: boolean; unitResults: boolean[] }
|
||||
|
||||
function unitPassed(kind: MeasureKind, trials: TrialRecord[]): boolean {
|
||||
if (trials.some(t => t.contract)) return false;
|
||||
if (kind === 'behavior') return trials.length === PANEL.n && trials.filter(t => t.passed).length >= PANEL.k;
|
||||
const [only] = trials;
|
||||
if (!only) return false;
|
||||
if (kind === 'judge' && only.samples) return only.samples.length === JUDGE_SAMPLES && only.samples.filter(Boolean).length >= 2;
|
||||
return only.passed;
|
||||
}
|
||||
|
||||
export function aggregate(plan: KindPlan, trials: TrialRecord[]): Aggregate {
|
||||
const unitResults = Array.from({ length: plan.units }, (_, i) => unitPassed(plan.kind, trials.filter(t => t.unit === i + 1)));
|
||||
const unitPasses = unitResults.filter(Boolean).length;
|
||||
const trialPasses = plan.kind === 'behavior' ? trials.filter(t => t.passed && !t.contract).length : unitPasses;
|
||||
const complete = trials.length === plan.trials;
|
||||
return { unitPasses, trialPasses, complete, unitResults, meets: complete && unitPasses >= plan.unitTarget && trialPasses >= plan.trialTarget };
|
||||
}
|
||||
export interface TrialRecord extends TrialResult { trial: number; set: TrialSet; unit: number; batch: number; dir: string }
|
||||
|
||||
// ─── One measurement: baseline or a repair round ───────────────────────────
|
||||
|
||||
export type MeasureStatus = 'measured' | 'needs_approval' | 'budget_exhausted' | 'round_limit';
|
||||
|
||||
export interface Measurement {
|
||||
schema: 'gstack-ship-measure/1';
|
||||
schema: 'gstack-ship-measure/2';
|
||||
label: 'diagnostic';
|
||||
verdict: null;
|
||||
case: string;
|
||||
@@ -198,10 +199,17 @@ export interface Measurement {
|
||||
status: MeasureStatus;
|
||||
reason?: string;
|
||||
plan: KindPlan;
|
||||
/** Inputs every pooled batch must share: tree, runner, CLI, model, image, policy and harness versions. */
|
||||
identity: Record<string, string>;
|
||||
trials: TrialRecord[];
|
||||
unitPasses: number;
|
||||
trialPasses: number;
|
||||
meets: boolean;
|
||||
decision: BarDecision;
|
||||
next: BarNext;
|
||||
decisionReason: string;
|
||||
passes: number;
|
||||
counted: number;
|
||||
sets: SetSummary[];
|
||||
/** The per-trial estimate later batches reserve; null until known. */
|
||||
costPerTrialUsd: number | null;
|
||||
/** Reserved before admission; null when a batch ran without an estimate. */
|
||||
estimatedUsd: number | null;
|
||||
actualUsd: number;
|
||||
@@ -211,6 +219,9 @@ export interface Measurement {
|
||||
|
||||
interface Ledger { case: string; kind: MeasureKind; approved: boolean; spentUsd: number; rounds: string[] }
|
||||
|
||||
/** A spend cap shared across cases (the sweep's weekly cap), checked beside each case's own budget. */
|
||||
export interface SpendPool { label: string; capUsd: number; spentUsd: number }
|
||||
|
||||
export interface MeasureOptions {
|
||||
caseId: string;
|
||||
kind: MeasureKind;
|
||||
@@ -224,6 +235,9 @@ export interface MeasureOptions {
|
||||
fix?: string;
|
||||
/** Pass-rate history directory diagnostic artifacts must stay out of (default: the project eval dir). */
|
||||
evalDir?: string;
|
||||
/** The batch identity (default: measurementIdentity of the working tree). */
|
||||
identity?: () => Record<string, string>;
|
||||
pool?: SpendPool;
|
||||
log?: (line: string) => void;
|
||||
}
|
||||
|
||||
@@ -244,6 +258,59 @@ function roundNumber(round: string): number {
|
||||
return Number(match[1]);
|
||||
}
|
||||
|
||||
/** The git tree of the working tree as it would be committed now (tracked and unignored files), without touching the real index. */
|
||||
export function frozenTree(cwd: string): string {
|
||||
const git = (args: string[], env: NodeJS.ProcessEnv = process.env) => spawnSync('git', args, { cwd, env, encoding: 'utf8', timeout: 120_000, maxBuffer: 64 * 1024 * 1024 });
|
||||
const top = git(['rev-parse', '--show-toplevel']);
|
||||
if (top.status !== 0) return 'no-git';
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ship-measure-index-'));
|
||||
try {
|
||||
const index = path.resolve(cwd, git(['rev-parse', '--git-path', 'index']).stdout.trim());
|
||||
if (fs.existsSync(index)) fs.copyFileSync(index, path.join(tmp, 'index'));
|
||||
const env = { ...process.env, GIT_INDEX_FILE: path.join(tmp, 'index') };
|
||||
const add = git(['-C', top.stdout.trim(), 'add', '-A'], env);
|
||||
const tree = add.status === 0 ? git(['write-tree'], env) : add;
|
||||
if (tree.status !== 0) throw new Error(`ship-measure: could not snapshot the working tree: ${tree.stderr.trim()}`);
|
||||
return tree.stdout.trim();
|
||||
} finally { fs.rmSync(tmp, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
/** Everything a pooled batch must share with the batch it extends. */
|
||||
export function measurementIdentity(kind: MeasureKind, trials: number, runner: string, cwd = process.cwd(), env: NodeJS.ProcessEnv = process.env): Record<string, string> {
|
||||
return {
|
||||
tree: frozenTree(cwd), kind, trials: String(trials), runner, policy: String(EVAL_POLICY.version), harness: String(harnessManifest.version),
|
||||
claude_cli: getClaudeCliVersion(), model: env.EVALS_MODEL ?? 'default', image: env.GSTACK_CI_IMAGE ?? 'local', bun: Bun.version,
|
||||
};
|
||||
}
|
||||
|
||||
const roundDirOf = (outDir: string, caseId: string, round: string) => path.join(outDir, caseSlug(caseId), round);
|
||||
const classificationsPath = (roundDir: string) => path.join(roundDir, 'classifications.jsonl');
|
||||
|
||||
export function readClassifications(roundDir: string): Classification[] {
|
||||
if (!fs.existsSync(classificationsPath(roundDir))) return [];
|
||||
return fs.readFileSync(classificationsPath(roundDir), 'utf8').split('\n').filter(Boolean).map(line => JSON.parse(line) as Classification);
|
||||
}
|
||||
|
||||
/** Apply the bar to the measurement's trials and its classification records. */
|
||||
function withDecision(m: Measurement, roundDir: string): Measurement {
|
||||
const outcome = decide(m.kind, m.plan.trials, m.trials, readClassifications(roundDir));
|
||||
return { ...m, decision: outcome.decision, next: outcome.next, decisionReason: outcome.reason, passes: outcome.passes, counted: outcome.thresholds.trials, sets: outcome.sets };
|
||||
}
|
||||
|
||||
function loadMeasurement(outDir: string, caseId: string, round: string): { m: Measurement; roundDir: string } {
|
||||
const roundDir = roundDirOf(outDir, caseId, round);
|
||||
const m = readJson<Measurement>(path.join(roundDir, 'measurement.json'));
|
||||
if (!m) throw new Error(`ship-measure: no measurement at ${roundDir}; run measure --case ${caseId} --round ${round} first`);
|
||||
return { m, roundDir };
|
||||
}
|
||||
|
||||
function saveMeasurement(m: Measurement, roundDir: string, log: (line: string) => void): Measurement {
|
||||
const decided = withDecision(m, roundDir);
|
||||
writeJson(path.join(roundDir, 'measurement.json'), decided);
|
||||
log(formatMeasurementLine(decided));
|
||||
return decided;
|
||||
}
|
||||
|
||||
export async function measureCase(o: MeasureOptions): Promise<Measurement> {
|
||||
const log = o.log ?? ((line: string) => console.log(line));
|
||||
const plan = kindPlan(o.kind, o.config);
|
||||
@@ -252,9 +319,9 @@ export async function measureCase(o: MeasureOptions): Promise<Measurement> {
|
||||
const ledgerPath = path.join(caseDir, 'ledger.json');
|
||||
const ledger: Ledger = readJson<Ledger>(ledgerPath) ?? { case: o.caseId, kind: o.kind, approved: false, spentUsd: 0, rounds: [] };
|
||||
const base: Measurement = {
|
||||
schema: 'gstack-ship-measure/1', label: 'diagnostic', verdict: null, case: o.caseId, kind: o.kind, round: o.round,
|
||||
...(o.fix ? { fix: o.fix } : {}), status: 'measured', plan, trials: [], unitPasses: 0, trialPasses: 0, meets: false,
|
||||
estimatedUsd: 0, actualUsd: 0, costUnknownTrials: 0, parallel: Math.max(1, o.parallel),
|
||||
schema: 'gstack-ship-measure/2', label: 'diagnostic', verdict: null, case: o.caseId, kind: o.kind, round: o.round,
|
||||
...(o.fix ? { fix: o.fix } : {}), status: 'measured', plan, identity: {}, trials: [], decision: 'incomplete', next: 'stop', decisionReason: '',
|
||||
passes: 0, counted: 0, sets: [], costPerTrialUsd: o.costPerTrialUsd, estimatedUsd: 0, actualUsd: 0, costUnknownTrials: 0, parallel: Math.max(1, o.parallel),
|
||||
};
|
||||
if (roundNumber(o.round) > o.config.maxRounds) {
|
||||
return { ...base, status: 'round_limit', reason: `repair-round limit ${o.config.maxRounds} reached; stop with the named-red report` };
|
||||
@@ -269,104 +336,167 @@ export async function measureCase(o: MeasureOptions): Promise<Measurement> {
|
||||
const roundDir = path.join(caseDir, o.round);
|
||||
if (fs.existsSync(roundDir)) throw new Error(`ship-measure: ${roundDir} already exists; measurements are never overwritten. Use the next round.`);
|
||||
fs.mkdirSync(roundDir, { recursive: true });
|
||||
const result = await runBatches(o, plan, roundDir, ledger, estimate, log);
|
||||
const totals = aggregate(plan, result.trials);
|
||||
const measurement: Measurement = { ...base, ...result, unitPasses: totals.unitPasses, trialPasses: totals.trialPasses, meets: totals.meets };
|
||||
let m: Measurement = { ...base, identity: (o.identity ?? (() => measurementIdentity(o.kind, plan.trials, 'gstack')))() };
|
||||
m = withDecision(await runSet(o, m, 'initial', roundDir, ledger, log), roundDir);
|
||||
if (m.status === 'measured' && m.next === 'redispatch') {
|
||||
log(`[ship-measure] ${o.caseId} ${o.round}: ${m.decisionReason}`);
|
||||
m = await runSet(o, m, 'redispatch', roundDir, ledger, log);
|
||||
}
|
||||
ledger.rounds.push(o.round);
|
||||
writeJson(path.join(roundDir, 'measurement.json'), measurement);
|
||||
writeJson(ledgerPath, ledger);
|
||||
log(formatMeasurementLine(measurement));
|
||||
return measurement;
|
||||
return saveMeasurement(m, roundDir, log);
|
||||
}
|
||||
|
||||
async function runBatches(o: MeasureOptions, plan: KindPlan, roundDir: string, ledger: Ledger, initialEstimate: number | null,
|
||||
log: (line: string) => void): Promise<Pick<Measurement, 'trials' | 'status' | 'reason' | 'estimatedUsd' | 'actualUsd' | 'costUnknownTrials' | 'parallel'>> {
|
||||
export interface ExtendOptions extends Omit<MeasureOptions, 'kind' | 'fix' | 'approved' | 'costPerTrialUsd'> { costPerTrialUsd?: number | null }
|
||||
|
||||
/** EXTEND: N more trials on identical inputs (the identity must match field for field), decided once on the pooled 2N. */
|
||||
export async function extendCase(o: ExtendOptions): Promise<Measurement> {
|
||||
const log = o.log ?? ((line: string) => console.log(line));
|
||||
assertDiagnosticDir(o.outDir, o.evalDir ?? getProjectEvalDir());
|
||||
let { m, roundDir } = loadMeasurement(o.outDir, o.caseId, o.round);
|
||||
m = withDecision(m, roundDir);
|
||||
if (m.trials.some(t => t.set === 'extend')) throw new Error(`ship-measure: ${o.caseId} ${o.round} was already extended; there is never a third batch`);
|
||||
if (m.decision !== 'EXTEND') throw new Error(`ship-measure: ${o.caseId} ${o.round} is ${m.decision}, not EXTEND (${m.decisionReason})`);
|
||||
const now = (o.identity ?? (() => measurementIdentity(m.kind, m.plan.trials, m.identity.runner ?? 'gstack')))();
|
||||
const changed = identityMismatch(m.identity, now);
|
||||
if (changed) throw new Error(`ship-measure: refusing to pool ${o.caseId} ${o.round}: ${changed}. EXTEND needs identical inputs; measure the changed tree as a new round.`);
|
||||
const ledgerPath = path.join(o.outDir, caseSlug(o.caseId), 'ledger.json');
|
||||
const ledger = readJson<Ledger>(ledgerPath) ?? { case: o.caseId, kind: m.kind, approved: true, spentUsd: m.actualUsd, rounds: [o.round] };
|
||||
const opts: MeasureOptions = { ...o, kind: m.kind, approved: true, costPerTrialUsd: o.costPerTrialUsd ?? m.costPerTrialUsd };
|
||||
m = withDecision(await runSet(opts, { ...m, status: 'measured' }, 'extend', roundDir, ledger, log), roundDir);
|
||||
if (m.status === 'measured' && m.next === 'redispatch') {
|
||||
log(`[ship-measure] ${o.caseId} ${o.round}: ${m.decisionReason}`);
|
||||
m = await runSet(opts, m, 'extend-redispatch', roundDir, ledger, log);
|
||||
}
|
||||
writeJson(ledgerPath, ledger);
|
||||
return saveMeasurement(m, roundDir, log);
|
||||
}
|
||||
|
||||
/** Append one immutable classification record per trial (a later record supersedes, never edits), then re-decide. */
|
||||
export function classifyTrials(outDir: string, caseId: string, round: string, trials: number[], cls: string, evidence: string, log = (line: string) => console.log(line)): Measurement {
|
||||
const { m, roundDir } = loadMeasurement(outDir, caseId, round);
|
||||
for (const trial of trials) {
|
||||
const problem = classificationProblem(m.trials.find(t => t.trial === trial), m.kind, cls, evidence);
|
||||
if (problem) throw new Error(`ship-measure: classify t${trial} refused: ${problem}`);
|
||||
}
|
||||
const at = new Date().toISOString();
|
||||
fs.appendFileSync(classificationsPath(roundDir), trials.map(trial => `${JSON.stringify({ trial, class: cls as FailureClass, evidence: evidence.trim(), by: 'agent', at } satisfies Classification)}\n`).join(''));
|
||||
return saveMeasurement(m, roundDir, log);
|
||||
}
|
||||
|
||||
export function decideRound(outDir: string, caseId: string, round: string, log = (line: string) => console.log(line)): Measurement {
|
||||
const { m, roundDir } = loadMeasurement(outDir, caseId, round);
|
||||
return saveMeasurement(m, roundDir, log);
|
||||
}
|
||||
|
||||
/** Run one set of N trials (initial, redispatch, extend or extend-redispatch) under the admission budget. */
|
||||
async function runSet(o: MeasureOptions, m: Measurement, set: TrialSet, roundDir: string, ledger: Ledger, log: (line: string) => void): Promise<Measurement> {
|
||||
const plan = m.plan;
|
||||
const first = m.trials.length;
|
||||
const trials: TrialRecord[] = [];
|
||||
const initialEstimate = m.costPerTrialUsd;
|
||||
let estimate = initialEstimate;
|
||||
let estimatedUsd: number | null = 0;
|
||||
let actualUsd = 0;
|
||||
let costUnknownTrials = 0;
|
||||
let knownCosts = 0;
|
||||
let estimatedUsd = m.estimatedUsd;
|
||||
let actualUsd = m.actualUsd;
|
||||
let costUnknownTrials = m.costUnknownTrials;
|
||||
let knownCosts = m.trials.filter(t => typeof t.costUsd === 'number').length;
|
||||
let batch = 0;
|
||||
const parallel = Math.max(1, o.parallel);
|
||||
const done = (status: MeasureStatus, reason?: string): Measurement => ({
|
||||
...m, status, ...(reason ? { reason } : {}), trials: [...m.trials, ...trials], estimatedUsd, actualUsd, costUnknownTrials, parallel,
|
||||
costPerTrialUsd: estimate,
|
||||
});
|
||||
while (trials.length < plan.trials) {
|
||||
const remaining = plan.trials - trials.length;
|
||||
const headroom = o.config.budgetUsd - ledger.spentUsd;
|
||||
const headroom = Math.min(o.config.budgetUsd - ledger.spentUsd, o.pool ? o.pool.capUsd - o.pool.spentUsd : Infinity);
|
||||
let size = Math.min(parallel, remaining);
|
||||
if (estimate === null) size = knownCosts === 0 && trials.length === 0 ? 1 : size;
|
||||
else size = Math.min(size, Math.floor((headroom + 1e-9) / estimate));
|
||||
if (size < 1 || headroom <= 0) {
|
||||
return { trials, status: 'budget_exhausted', estimatedUsd, actualUsd, costUnknownTrials, parallel,
|
||||
reason: `budget $${o.config.budgetUsd} per red case: spent $${ledger.spentUsd.toFixed(2)}, next trial reserves $${(estimate ?? 0).toFixed(2)}; ${trials.length} of ${plan.trials} trials ran` };
|
||||
const cap = o.pool && o.pool.capUsd - o.pool.spentUsd <= o.config.budgetUsd - ledger.spentUsd
|
||||
? `${o.pool.label} $${o.pool.capUsd}: spent $${o.pool.spentUsd.toFixed(2)}`
|
||||
: `budget $${o.config.budgetUsd} per red case: spent $${ledger.spentUsd.toFixed(2)}`;
|
||||
return done('budget_exhausted', `${cap}, next trial reserves $${(estimate ?? 0).toFixed(2)}; ${trials.length} of ${plan.trials} ${set} trials ran`);
|
||||
}
|
||||
batch += 1;
|
||||
estimatedUsd = estimate === null || estimatedUsd === null ? null : estimatedUsd + size * estimate;
|
||||
log(`[ship-measure] ${o.caseId} ${o.round}: batch ${batch} admits ${size} trial(s)${estimate === null ? ' (calibration, no estimate)' : `, reserves $${(size * estimate).toFixed(2)}`}; spent $${ledger.spentUsd.toFixed(2)} of $${o.config.budgetUsd}`);
|
||||
const start = trials.length;
|
||||
log(`[ship-measure] ${o.caseId} ${o.round} ${set}: batch ${batch} admits ${size} trial(s)${estimate === null ? ' (calibration, no estimate)' : `, reserves $${(size * estimate).toFixed(2)}`}; spent $${ledger.spentUsd.toFixed(2)} of $${o.config.budgetUsd}`);
|
||||
const start = first + trials.length;
|
||||
const results = await Promise.all(Array.from({ length: size }, async (_, i) => {
|
||||
const trial = start + i + 1;
|
||||
const dir = path.join(roundDir, `t${String(trial).padStart(2, '0')}`);
|
||||
fs.mkdirSync(dir, { recursive: false });
|
||||
let result: TrialResult;
|
||||
try { result = await o.runner({ caseId: o.caseId, kind: o.kind, round: o.round, trial, dir }); }
|
||||
try { result = await o.runner({ caseId: o.caseId, kind: m.kind, round: o.round, trial, dir }); }
|
||||
catch (error) { result = { passed: false, failureCause: 'unknown', failureDetail: `runner error: ${(error as Error).message}` }; }
|
||||
return { ...result, trial, unit: Math.ceil(trial / plan.trialsPerUnit), batch, dir } satisfies TrialRecord;
|
||||
return { ...result, trial, set, unit: Math.ceil((trial - first) / plan.trialsPerUnit), batch, dir } satisfies TrialRecord;
|
||||
}));
|
||||
for (const record of results) {
|
||||
writeJson(path.join(record.dir, 'trial.json'), { label: 'diagnostic', verdict: null, ...record });
|
||||
trials.push(record);
|
||||
if (typeof record.costUsd === 'number' && Number.isFinite(record.costUsd)) {
|
||||
actualUsd += record.costUsd; ledger.spentUsd += record.costUsd; knownCosts += 1;
|
||||
} else {
|
||||
costUnknownTrials += 1;
|
||||
if (estimate !== null) ledger.spentUsd += estimate;
|
||||
}
|
||||
const charge = typeof record.costUsd === 'number' && Number.isFinite(record.costUsd) ? record.costUsd : estimate;
|
||||
if (typeof record.costUsd === 'number' && Number.isFinite(record.costUsd)) { actualUsd += record.costUsd; knownCosts += 1; }
|
||||
else costUnknownTrials += 1;
|
||||
if (charge !== null) { ledger.spentUsd += charge; if (o.pool) o.pool.spentUsd += charge; }
|
||||
}
|
||||
if (initialEstimate === null && knownCosts > 0) estimate = actualUsd / knownCosts;
|
||||
}
|
||||
return { trials, status: 'measured', estimatedUsd, actualUsd, costUnknownTrials, parallel };
|
||||
return done('measured');
|
||||
}
|
||||
|
||||
const DECISION_WORDS: Record<BarDecision, string> = {
|
||||
MEETS: 'MEETS', 'MEETS-qualified': 'MEETS-qualified', EXTEND: 'EXTEND', BELOW: 'BELOW (fix round)',
|
||||
'needs-classify': 'needs-classify', void: 'void', incomplete: 'incomplete',
|
||||
};
|
||||
|
||||
export function formatMeasurementLine(m: Measurement): string {
|
||||
const observed = m.kind === 'behavior'
|
||||
? `${m.trialPasses}/${m.plan.trials} trials, ${m.unitPasses}/${m.plan.units} panels`
|
||||
: `${m.unitPasses}/${m.plan.units} ${m.kind === 'judge' ? 'outputs' : 'trials'}`;
|
||||
const when = m.fix ? ` after fix at ${m.fix}` : m.round === 'baseline' ? ' (baseline)' : '';
|
||||
const verdict = m.status === 'measured' ? (m.meets ? 'at or above target' : 'below target') : m.status.replace('_', ' ');
|
||||
return `[ship-measure] DIAGNOSTIC ${m.case} ${m.round}: observed ${observed}${when}; ${verdict}; est ${usd(m.estimatedUsd)}, actual ${usd(m.actualUsd)}${m.costUnknownTrials ? ` (+${m.costUnknownTrials} trial(s) cost unknown)` : ''}`;
|
||||
const verdict = m.status === 'measured' ? DECISION_WORDS[m.decision] : m.status.replace('_', ' ');
|
||||
const sets = m.sets.length > 1 ? `; batches ${m.sets.map(s => `${s.set} ${s.passes}/${s.trials}${s.void ? ' void' : ''}`).join(', ')}` : '';
|
||||
return `[ship-measure] DIAGNOSTIC ${m.case} ${m.round}: observed ${m.passes}/${m.counted} ${m.kind === 'judge' ? 'outputs' : 'trials'}${when}; ${verdict}${m.status === 'measured' ? ` — ${m.decisionReason}` : m.reason ? ` — ${m.reason}` : ''}${sets}; est ${usd(m.estimatedUsd)}, actual ${usd(m.actualUsd)}${m.costUnknownTrials ? ` (+${m.costUnknownTrials} trial(s) cost unknown)` : ''}`;
|
||||
}
|
||||
|
||||
const usd = (value: number | null) => value === null ? 'unknown' : `$${value.toFixed(2)}`;
|
||||
|
||||
/** measure/extend/classify/decide exit code for a measurement (see the header). */
|
||||
export function measurementExit(m: Measurement): number {
|
||||
if (m.status === 'needs_approval') return 2;
|
||||
if (m.status !== 'measured') return 3;
|
||||
return ({ MEETS: 0, 'MEETS-qualified': 0, BELOW: 1, 'needs-classify': 5, EXTEND: 6, void: 3, incomplete: 3 } as const)[m.decision];
|
||||
}
|
||||
|
||||
// ─── Unmeasured skip and the PR-body report ────────────────────────────────
|
||||
|
||||
export function recordUnmeasured(outDir: string, caseId: string, reason: string): string {
|
||||
if (!reason.trim()) throw new Error('skip needs --reason: why this red is infrastructure and is not measured');
|
||||
const file = path.join(outDir, caseSlug(caseId), 'unmeasured.json');
|
||||
writeJson(file, { schema: 'gstack-ship-measure/1', label: 'diagnostic', verdict: null, case: caseId, status: 'unmeasured', reason: reason.trim() });
|
||||
writeJson(file, { schema: 'gstack-ship-measure/2', label: 'diagnostic', verdict: null, case: caseId, status: 'unmeasured', reason: reason.trim() });
|
||||
return file;
|
||||
}
|
||||
|
||||
/** The per-case table for the PR body: every measurement, estimated and actual spend, never a pass for an unmeasured case. */
|
||||
export function formatReport(outDir: string): string {
|
||||
const rows = ['| Case | Kind | Measurement | Observed | Target | Est. spend | Actual spend | Status |', '|---|---|---|---|---|---|---|---|'];
|
||||
const rows = ['| Case | Kind | Measurement | Observed | Bar | Decision | Est. spend | Actual spend |', '|---|---|---|---|---|---|---|---|'];
|
||||
const cases = fs.existsSync(outDir) ? fs.readdirSync(outDir).filter(name => fs.statSync(path.join(outDir, name)).isDirectory()).sort() : [];
|
||||
const qualified: string[] = [];
|
||||
for (const name of cases) {
|
||||
const dir = path.join(outDir, name);
|
||||
const skip = readJson<{ case: string; reason: string }>(path.join(dir, 'unmeasured.json'));
|
||||
if (skip) { rows.push(`| ${skip.case} | — | — | — | — | — | — | unmeasured (${skip.reason.replace(/\|/g, '/')}); not a pass |`); continue; }
|
||||
if (skip) { rows.push(`| ${skip.case} | — | — | — | — | unmeasured (${skip.reason.replace(/\|/g, '/')}); not a pass | — | — |`); continue; }
|
||||
const ledger = readJson<Ledger>(path.join(dir, 'ledger.json'));
|
||||
for (const round of ledger?.rounds ?? []) {
|
||||
const m = readJson<Measurement>(path.join(dir, round, 'measurement.json'));
|
||||
if (!m) continue;
|
||||
const observed = m.kind === 'behavior' ? `${m.trialPasses}/${m.trials.length} trials, ${m.unitPasses}/${m.plan.units} panels` : `${m.unitPasses}/${m.trials.length}`;
|
||||
const target = m.kind === 'behavior' ? `all ${m.plan.units} panels, ${m.plan.trialTarget}/${m.plan.trials}` : `${m.plan.unitTarget}/${m.plan.units}`;
|
||||
const bar = m.counted > m.plan.trials ? m.plan.pooledBar : m.plan.bar;
|
||||
const what = m.fix ? `${m.round}: after fix at ${m.fix.replace(/\|/g, '/')}` : m.round;
|
||||
const status = m.status === 'measured' ? (m.meets ? 'at or above target' : 'below target') : m.status.replace('_', ' ');
|
||||
rows.push(`| ${m.case} | ${m.kind} | ${what} | observed ${observed} | ${target} | ${usd(m.estimatedUsd)} | ${usd(m.actualUsd)}${m.costUnknownTrials ? ` (${m.costUnknownTrials} unknown)` : ''} | ${status} |`);
|
||||
const batches = m.sets.length > 1 ? ` (${m.sets.map(s => `${s.set} ${s.passes}/${s.trials}${s.void ? ' void' : ''}`).join(', ')})` : '';
|
||||
const decision = m.status === 'measured' ? DECISION_WORDS[m.decision] : m.status.replace('_', ' ');
|
||||
rows.push(`| ${m.case} | ${m.kind} | ${what} | observed ${m.passes}/${m.counted}${batches} | ${bar.strict}/${bar.trials} strict, ${bar.qualified}/${bar.trials} qualified | ${decision} | ${usd(m.estimatedUsd)} | ${usd(m.actualUsd)}${m.costUnknownTrials ? ` (${m.costUnknownTrials} unknown)` : ''} |`);
|
||||
if (m.status === 'measured' && m.decision === 'MEETS-qualified') qualified.push(`- ${m.case} ${m.round}: ${m.decisionReason.replace(/\|/g, '/')}`);
|
||||
}
|
||||
}
|
||||
return [...rows, '', 'Diagnostic measurements: they never change a recorded verdict; the lane verdict is the full gate run.'].join('\n');
|
||||
return [...rows, ...(qualified.length ? ['', 'Qualified MEETS (every red classified into a qualifying class):', ...qualified] : []),
|
||||
'', 'Diagnostic measurements: they never change a recorded verdict; the lane verdict is the full gate run.'].join('\n');
|
||||
}
|
||||
|
||||
// ─── Trial runners ─────────────────────────────────────────────────────────
|
||||
@@ -398,7 +528,9 @@ export function caseSelectionPreflight(caseId: string, rootDir = ROOT): { ok: bo
|
||||
const out = `${r.stdout ?? ''}${r.stderr ?? ''}`;
|
||||
if (r.status !== 0) return { ok: false, detail: `--list exited ${r.status}: ${out.trim().split('\n').slice(-1)[0] ?? ''}` };
|
||||
const file = new RegExp(`--case ${caseId.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}: \\d+ trial\\(s\\) of (\\S+)`).exec(out)?.[1];
|
||||
const listed = file && new RegExp(`^\\s+${file.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')} \\(trial 1\\)`, 'm').test(out);
|
||||
const esc = (text: string) => text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
// A file-mode case lists `<file> (trial 1)`; a name-mode case lists its trial shard `<file>#<id>~t1`.
|
||||
const listed = file && new RegExp(`^\\s+${esc(file)}(?: \\(trial 1\\)|#${esc(caseId)}~t1)$`, 'm').test(out);
|
||||
if (!file || !listed || !fs.existsSync(path.join(rootDir, file))) return { ok: false, detail: `--list did not plan a trial of ${caseId}'s test file` };
|
||||
return { ok: true, detail: `--case ${caseId} selects ${file}` };
|
||||
}
|
||||
@@ -408,9 +540,11 @@ export function caseSelectionPreflight(caseId: string, rootDir = ROOT): { ok: bo
|
||||
* and never publish: with the judge input cache enabled, a trial could reuse a
|
||||
* stored pass, and its own passes would flow into the gate's reuse.
|
||||
*/
|
||||
export function diagnosticBaseEnv(base: NodeJS.ProcessEnv, evalDir: string): NodeJS.ProcessEnv {
|
||||
export function diagnosticBaseEnv(base: NodeJS.ProcessEnv, evalDir: string, trialId: string): NodeJS.ProcessEnv {
|
||||
const { EVALS_CACHE_DIR: _cacheDir, EVALS_CACHE_RUNTIME_ID: _cacheRuntime, ...inherited } = base;
|
||||
return { ...inherited, GSTACK_EVAL_DIR: evalDir, GSTACK_SHIP_MEASURE_LABEL: 'diagnostic', EVALS_JOBS: '1' };
|
||||
// Cases that retain native evidence (functional QA, docs faults) require EVALS_RUN_ID, which only CI sets;
|
||||
// each trial gets its own so parallel trials never share an evidence directory.
|
||||
return { ...inherited, GSTACK_EVAL_DIR: evalDir, GSTACK_SHIP_MEASURE_LABEL: 'diagnostic', EVALS_JOBS: '1', EVALS_RUN_ID: `${base.EVALS_RUN_ID || 'local'}-measure-${trialId}` };
|
||||
}
|
||||
|
||||
/** True when some junit.xml under evalDir holds an executed, passing testcase and no failed or errored one. */
|
||||
@@ -454,6 +588,7 @@ function parseTrialOutput(output: string): Partial<TrialResult> {
|
||||
return {
|
||||
...(field('failure_cause') ? { failureCause: field('failure_cause') } : {}),
|
||||
...(field('failure_detail') ? { failureDetail: field('failure_detail') } : {}),
|
||||
...(field('failure_evidence') ? { failureEvidence: field('failure_evidence') } : {}),
|
||||
...(cost !== undefined && Number.isFinite(Number(cost)) ? { costUsd: Number(cost) } : {}),
|
||||
...(samples ? { samples: samples.split(/[,\s]+/).filter(Boolean).map(s => s === '1' || s === 'pass' || s === 'true') } : {}),
|
||||
...(/^\s*contract_violation\b/m.test(output) ? { contract: true } : {}),
|
||||
@@ -481,6 +616,7 @@ export function commandRunner(template: string, cwd = process.cwd()): TrialRunne
|
||||
...(parsed.costUsd === undefined && costOf(records) !== undefined ? { costUsd: costOf(records) } : {}),
|
||||
...(code !== 0 && !parsed.failureCause && failed?.failure_cause ? { failureCause: String(failed.failure_cause) } : {}),
|
||||
...(code !== 0 && !parsed.failureDetail && failed?.failure_detail ? { failureDetail: JSON.stringify(failed.failure_detail).slice(0, 300) } : {}),
|
||||
...(code !== 0 && !parsed.failureEvidence && failed?.failure_cause_evidence ? { failureEvidence: String(failed.failure_cause_evidence).slice(0, 300) } : {}),
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -502,7 +638,7 @@ export function judgeFile(id: string, rootDir = ROOT): string {
|
||||
* with the judge selected alone and passes only on its own passing record.
|
||||
*/
|
||||
export function gstackRunner(rootDir = ROOT): TrialRunner {
|
||||
return async ({ caseId, dir }) => {
|
||||
return async ({ caseId, round, trial, dir }) => {
|
||||
const evalDir = path.join(dir, 'eval');
|
||||
const isJudge = !Object.hasOwn(E2E_TIERS, caseId);
|
||||
if (isJudge && !Object.hasOwn(LLM_JUDGE_TOUCHFILES, caseId)) throw new Error(`${caseId} is neither an E2E case nor a standalone judge`);
|
||||
@@ -510,7 +646,7 @@ export function gstackRunner(rootDir = ROOT): TrialRunner {
|
||||
? [process.execPath, 'test', path.join(rootDir, judgeFile(caseId, rootDir))]
|
||||
: [process.execPath, 'run', path.join(rootDir, 'scripts/test-paid-shards.ts'), '--tier', E2E_TIERS[caseId]!, '--case', caseId, '--trials', '1'];
|
||||
const env = {
|
||||
...diagnosticBaseEnv(process.env, evalDir),
|
||||
...diagnosticBaseEnv(process.env, evalDir, `${caseSlug(caseId)}-${round}-t${trial}-${process.pid}`),
|
||||
...(isJudge ? { EVALS: '1', EVALS_TIER: 'gate', EVALS_ALL: '1', ...paidSelectionEnv('full', { e2e: [], judges: [caseId] }, 'ship-measure judge') } : {}),
|
||||
};
|
||||
const { code, output } = await runTrialProcess(argv, rootDir, env, dir);
|
||||
@@ -526,6 +662,7 @@ export function gstackRunner(rootDir = ROOT): TrialRunner {
|
||||
...(own.some(r => r.failure_class === 'contract') ? { contract: true } : {}),
|
||||
...(passed ? {} : {
|
||||
failureCause: failed?.failure_cause ? String(failed.failure_cause) : own.length ? 'assertion' : 'unknown',
|
||||
...(failed?.failure_cause_evidence ? { failureEvidence: String(failed.failure_cause_evidence).slice(0, 300) } : {}),
|
||||
failureDetail: failed?.failure_detail ? JSON.stringify(failed.failure_detail).slice(0, 300)
|
||||
: own.length ? String(failed?.error ?? '').slice(0, 300) : `no trial record; exit ${code}; ${output.trim().split('\n').slice(-1)[0] ?? ''}`.slice(0, 300),
|
||||
}),
|
||||
@@ -627,10 +764,11 @@ function positive(raw: string | undefined, flag: string, integer = true): number
|
||||
|
||||
export async function main(argv: string[], env: NodeJS.ProcessEnv = process.env): Promise<number> {
|
||||
const [command, ...rest] = argv;
|
||||
if (command === 'sweep') return (await import('./ship-measure-sweep')).sweepMain(rest, env);
|
||||
const flags: Record<string, string> = {};
|
||||
const value = (name: string) => (next: () => string | undefined) => { const v = next(); if (v === undefined) throw new Error(`${name} needs a value`); flags[name] = v; };
|
||||
const handlers = Object.fromEntries(['--case', '--round', '--kind', '--command', '--cost-per-trial', '--fix', '--jobs', '--out', '--reason',
|
||||
'--files', '--shard', '--reruns', '--concurrency', '--backend', '--wall-cap'].map(name => [name, value(name)]));
|
||||
'--files', '--shard', '--reruns', '--concurrency', '--backend', '--wall-cap', '--trials', '--class', '--evidence'].map(name => [name, value(name)]));
|
||||
parseCliFlags(rest, { ...handlers, '--approved': () => { flags['--approved'] = '1'; } }, 'Usage: see the header of scripts/ship-measure.ts');
|
||||
const config = readMeasureConfig();
|
||||
const outDir = path.resolve(flags['--out'] ?? path.join(process.cwd(), '.context', 'ship-measure'));
|
||||
@@ -638,29 +776,43 @@ export async function main(argv: string[], env: NodeJS.ProcessEnv = process.env)
|
||||
if (command === 'report') { console.log(formatReport(outDir)); return 0; }
|
||||
if (command === 'skip') { console.log(`[ship-measure] ${flags['--case']} labeled unmeasured: ${recordUnmeasured(outDir, flags['--case'] ?? '', flags['--reason'] ?? '')}`); return 0; }
|
||||
if (command === 'free') return runFreeCli(flags, config, outDir, env);
|
||||
if (command !== 'measure') throw new Error(`Unknown command: ${command ?? '(none)'}. Use table, measure, skip, report or free.`);
|
||||
if (command !== 'measure' && command !== 'classify' && command !== 'decide' && command !== 'extend') {
|
||||
throw new Error(`Unknown command: ${command ?? '(none)'}. Use table, measure, classify, decide, extend, skip, report, free or sweep.`);
|
||||
}
|
||||
const caseId = flags['--case'];
|
||||
if (!caseId) throw new Error('measure needs --case');
|
||||
const kind = (flags['--kind'] ?? E2E_KINDS[caseId] ?? 'rule') as MeasureKind;
|
||||
if (!['rule', 'behavior', 'judge'].includes(kind)) throw new Error(`--kind must be rule, behavior or judge. Received: ${kind}`);
|
||||
if (!caseId) throw new Error(`${command} needs --case`);
|
||||
const round = flags['--round'] ?? 'baseline';
|
||||
if (command === 'classify') {
|
||||
const trials = (flags['--trials'] ?? '').split(',').map(t => t.trim().replace(/^t/, '')).filter(Boolean).map(t => positive(t, '--trials'));
|
||||
if (!trials.length || !flags['--class']) throw new Error(`classify needs --trials 2,5 and --class (${FAILURE_CLASSES.join(', ')})`);
|
||||
return measurementExit(classifyTrials(outDir, caseId, round, trials, flags['--class'], flags['--evidence'] ?? ''));
|
||||
}
|
||||
if (command === 'decide') return measurementExit(decideRound(outDir, caseId, round));
|
||||
const isGstack = path.resolve(process.cwd()) === ROOT;
|
||||
if (!flags['--command'] && !isGstack) throw new Error('No single-case eval command: pass --command \'<documented command> {case}\' (ask the user once and record the answer).');
|
||||
const runner = flags['--command'] ? commandRunner(flags['--command']) : gstackRunner();
|
||||
const runnerId = flags['--command'] ? `command:${flags['--command']}` : 'gstack';
|
||||
const parallel = flags['--jobs'] ? positive(flags['--jobs'], '--jobs') : Number(env.EVALS_JOBS) || DEFAULT_JOBS;
|
||||
const costPerTrialUsd = flags['--cost-per-trial'] !== undefined ? positive(flags['--cost-per-trial'], '--cost-per-trial', false) : null;
|
||||
if (command === 'extend') {
|
||||
return measurementExit(await extendCase({ caseId, round, config, outDir, runner, parallel, ...(costPerTrialUsd !== null ? { costPerTrialUsd } : {}) }));
|
||||
}
|
||||
const kind = (flags['--kind'] ?? E2E_KINDS[caseId] ?? 'rule') as MeasureKind;
|
||||
if (!['rule', 'behavior', 'judge'].includes(kind)) throw new Error(`--kind must be rule, behavior or judge. Received: ${kind}`);
|
||||
console.log(formatKindTable(config));
|
||||
if (!flags['--command']) {
|
||||
const selected = caseSelectionPreflight(caseId);
|
||||
if (!selected.ok) throw new Error(`--case ${caseId} does not select its test (no paid call made): ${selected.detail}`);
|
||||
console.log(`[ship-measure] preflight: ${selected.detail}`);
|
||||
}
|
||||
const plan = kindPlan(kind, config);
|
||||
const m = await measureCase({
|
||||
caseId, kind, round: flags['--round'] ?? 'baseline', config, outDir, fix: flags['--fix'],
|
||||
runner: flags['--command'] ? commandRunner(flags['--command']) : gstackRunner(),
|
||||
parallel: flags['--jobs'] ? positive(flags['--jobs'], '--jobs') : Number(env.EVALS_JOBS) || DEFAULT_JOBS,
|
||||
costPerTrialUsd: flags['--cost-per-trial'] !== undefined ? positive(flags['--cost-per-trial'], '--cost-per-trial', false) : null,
|
||||
approved: flags['--approved'] === '1',
|
||||
caseId, kind, round, config, outDir, fix: flags['--fix'], runner, parallel, costPerTrialUsd, approved: flags['--approved'] === '1',
|
||||
identity: () => measurementIdentity(kind, plan.trials, runnerId),
|
||||
});
|
||||
if (m.status === 'needs_approval') { console.log(`[ship-measure] NEEDS APPROVAL ${caseId}: ${m.reason}`); return 2; }
|
||||
if (m.status !== 'measured') { console.log(`[ship-measure] NAMED RED ${caseId}: ${m.reason}`); return 3; }
|
||||
return m.meets ? 0 : 1;
|
||||
if (m.status === 'needs_approval') console.log(`[ship-measure] NEEDS APPROVAL ${caseId}: ${m.reason}`);
|
||||
else if (m.status !== 'measured' || m.next === 'stop') console.log(`[ship-measure] NAMED RED ${caseId}: ${m.reason ?? m.decisionReason}`);
|
||||
return measurementExit(m);
|
||||
}
|
||||
|
||||
async function runFreeCli(flags: Record<string, string>, config: MeasureConfig, outDir: string, env: NodeJS.ProcessEnv): Promise<number> {
|
||||
|
||||
@@ -234,6 +234,9 @@ export const FREE_ONLY_PR_FILES = [
|
||||
'.github/workflows/scorecard.yml',
|
||||
'.github/workflows/skill-docs.yml',
|
||||
'.github/workflows/test-health.yml',
|
||||
'.github/workflows/eval-sweep.yml', // The weekly off-ship sweep on main; no PR-gate case reads it, and it never gates a PR.
|
||||
'.github/actionlint.yaml', // Runner labels for actionlint only.
|
||||
'scripts/ship-measure-sweep.ts', // The weekly sweep CLI; it launches paid cases, never runs inside one.
|
||||
'.github/workflows/version-gate.yml',
|
||||
'.github/workflows/windows-free-tests.yml',
|
||||
'.github/workflows/windows-setup-e2e.yml',
|
||||
|
||||
Reference in new issue
Block a user