mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 22:45:02 +01:00
v1.91.33.0 feat(ship): measurement bar with explained-red and pooled decisions, weekly flaky-test sweep on main (#3059)
* refactor(eval-history): share pass-rate history loading as loadPassRateHistory() eval:pass-rates' main block loaded local dirs or the weekly trial-outcomes artifacts inline. Move that into an exported function returning records, scope and fetch errors, so the off-ship sweep reads the same history the same way. Behavior unchanged. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-measure): the measurement bar and the weekly off-ship sweep CLI Decide each diagnostic measurement with one bar (scripts/lib/measure-bar.ts): MEETS at 9/10; MEETS-qualified at 8/10 when every red is a mechanical provider error, judge noise or a cited model miss; EXTEND at 7/10 (or an unqualified 8/10) for one more batch on identical inputs, decided once on the pooled 20; BELOW otherwise and for any timeout, hang, regression or fixable cause. Behavior uses 12 trials (11/10/9; 22/20 of 24); judge counts median-panel outputs. A batch with more than 30% provider-evidence failures is void and redispatched once. Below-strict reds nobody classified return needs-classify. ship-measure gains classify (append-only records; provider refused without evidence), decide and extend (refuses a changed identity, names the field, never a third batch), exit codes 5 and 6, and a pooling identity (frozen tree, runner, CLI, model, CI image, Bun, policy and harness versions). ship-measure sweep ranks main's gate cases by their cost to the all-green probability from eval:pass-rates history, measures the top K with the same runner and bar under a weekly cap shared with the last 7 days of sweep reports (ship_measure_sweep_cases, ship_measure_sweep_budget_usd), and writes a report with the predicted all-green probability before and after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship): the measure loop decides by the bar's exit codes Step 4 branches on MEETS/MEETS-qualified, needs-classify (with the qualifying and fix-round classes), EXTEND once and BELOW; the rest of the loop prose is tightened to offset most of the growth. ship's union size cap moves from 1.495 to 1.4975 (measured 281,058 / 187,706 = 1.4973). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci: weekly eval-sweep workflow on main Mondays 12:00 UTC and on dispatch (k, cap_usd, dry_run): runs ship-measure sweep in the published CI image by digest with the census's paid-eval setup, downloads the last 7 days of sweep reports first so the weekly cap holds across runs, uploads the report and captures, and writes the job summary. Read-only grants; it never pushes or opens a pull request. One sweep at a time repository-wide, never cancelled mid-spend. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the measurement bar table and the weekly sweep Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: v1.91.33.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation" This reverts commitf4c58f2869. * ci: classify eval-sweep.yml, the sweep CLI and actionlint labels as free-only PR inputs None is read by a PR-gate case, so a change to them selects no paid case instead of restoring the full gate (test/free-fixtures.test.ts workflow inventory). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): the selection preflight accepts name-mode cases A case registered beside others in its file lists its trial shard as <file>#<id>~t1, not <file> (trial 1), so the preflight refused every name-mode case (for example ship-exploratory-late-input) before any paid trial. Accept both forms. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Revert "ci(temp): run eval-sweep on push to this branch (K=2, $25) for end-to-end validation" This reverts commit9619734307. * test: record the test-pr-profile.ts change as non-behavioral for HARNESS_VERSION Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): the name-mode preflight fix Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): a partly funded sweep row shows the trials that ran Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(sweep): the fake single-case command is a Bun script, so it runs on Windows The Windows free lane runs test/ship-measure-sweep.test.ts; CreateProcess cannot exec a shebang shell stub, so every fake trial errored and the cap stopped the sweep after one trial. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(plan-mode-no-op): the CEO sub-test reviews a fixed pasted plan Without a plan, plan-ceo-review reviewed the checkout's branch diff, so its pre-question work (audit and landscape searches) grew with whatever PR the gate ran on. On this PR it timed out at 300s in 3 of 10 trials while still researching the diff (ship-measure baseline 7/10). With a fixed small plan, the same way the devex sub-test already runs, it measured 10/10 (CEO sub-test 83-255s). No budget or threshold changed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-measure): each diagnostic trial gets its own EVALS_RUN_ID Evidence-retaining cases (functional QA, docs faults) refuse to start without EVALS_RUN_ID, which only CI sets, so measuring them locally ran 0/10 sessions. qa-functional-cli-report, red once in this PR's gate, measured 10/10 once the trials started ($11.13). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed
1 parent
28f1385eac
commit
db745675bd
28 files changed
+1950
-300
No files matched your search
@@ -13,7 +13,8 @@ import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
|
||||
const ROOT = path.resolve(__dirname, '..');
|
||||
const FILES = ['evals.yml', 'evals-periodic.yml', 'evals-marathon.yml'];
|
||||
// eval-sweep.yml only resolves the tag (it never builds), so it must look up the same one.
|
||||
const FILES = ['evals.yml', 'evals-periodic.yml', 'evals-marathon.yml', 'eval-sweep.yml'];
|
||||
|
||||
function hashFilesCalls(name: string): string[] {
|
||||
const source = fs.readFileSync(
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* Weekly off-ship sweep workflow wiring (docs/TESTING_INTERNALS.md#ship-measure-sweep):
|
||||
* weekly after the periodic census plus dispatch, read-only permissions (it
|
||||
* never pushes or opens a pull request), one sweep at a time repository-wide
|
||||
* so two runs never spend the same weekly cap, the CI image by digest with the
|
||||
* census's paid-eval setup, earlier reports downloaded before any spend,
|
||||
* dispatch inputs reaching the shell only through env, and the report uploaded
|
||||
* even when the sweep fails.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const source = fs.readFileSync(path.join(ROOT, '.github/workflows/eval-sweep.yml'), 'utf8');
|
||||
const workflow = Bun.YAML.parse(source) as any;
|
||||
const sweep = workflow.jobs.sweep;
|
||||
const step = (name: string) => sweep.steps.find((s: any) => s.name === name);
|
||||
|
||||
describe('eval-sweep.yml', () => {
|
||||
test('runs weekly after the 06:00 UTC periodic census, plus dispatch with k, cap and dry-run inputs', () => {
|
||||
expect(Object.keys(workflow.on).sort()).toEqual(['schedule', 'workflow_dispatch']);
|
||||
const [hour] = /cron: '0 (\d+) \* \* 1'/.exec(source)!.slice(1).map(Number);
|
||||
const census = Number(/cron: '0 (\d+) \* \* 1'/.exec(fs.readFileSync(path.join(ROOT, '.github/workflows/evals-periodic.yml'), 'utf8'))![1]);
|
||||
expect(hour!).toBeGreaterThan(census);
|
||||
expect(Object.keys(workflow.on.workflow_dispatch.inputs).sort()).toEqual(['cap_usd', 'dry_run', 'k']);
|
||||
});
|
||||
|
||||
test('never pushes, commits or opens a pull request: read-only grants, no write commands', () => {
|
||||
expect(workflow.permissions).toEqual({ contents: 'read' });
|
||||
for (const job of Object.values<any>(workflow.jobs)) {
|
||||
for (const value of Object.values<string>(job.permissions ?? {})) expect(value).toBe('read');
|
||||
}
|
||||
expect(source).not.toMatch(/git push|git commit|gh pr |gh issue|pull-requests:|contents: write/);
|
||||
for (const s of sweep.steps) if (s.uses?.startsWith('actions/checkout')) expect(s.with['persist-credentials']).toBe(false);
|
||||
});
|
||||
|
||||
test('one sweep at a time across every ref, never cancelled mid-spend', () => {
|
||||
expect(workflow.concurrency).toEqual({ group: 'eval-sweep', 'cancel-in-progress': false });
|
||||
});
|
||||
|
||||
test('runs in the published CI image by digest with the census paid-eval setup', () => {
|
||||
expect(sweep.container.image).toBe('${{ needs.image.outputs.image-ref }}');
|
||||
expect(sweep.container.options).toBe('--user runner');
|
||||
expect(workflow.jobs.image.steps.map((s: any) => s.name ?? s.uses)).toContain('Resolve the published CI image digest');
|
||||
for (const name of ['Seed claude interactive config', 'Register gstack skills for PTY tests', 'Authenticate the Codex CLI']) expect(step(name)).toBeDefined();
|
||||
const run = step('Run the sweep');
|
||||
expect(run.env.ANTHROPIC_API_KEY).toBe('${{ secrets.ANTHROPIC_API_KEY }}');
|
||||
expect(run.env.GSTACK_CI_IMAGE).toBe('${{ needs.image.outputs.image-ref }}');
|
||||
});
|
||||
|
||||
test('downloads the last 7 days of sweep reports before the sweep and passes them as --prior', () => {
|
||||
const names = sweep.steps.map((s: any) => s.name);
|
||||
expect(names.indexOf("Download this week's earlier sweep reports")).toBeLessThan(names.indexOf('Run the sweep'));
|
||||
const download = step("Download this week's earlier sweep reports");
|
||||
expect(download.run).toContain("date -u -d '7 days ago'");
|
||||
expect(download.run).toContain('--name ship-measure-sweep-report');
|
||||
expect(download.run).toContain('set -euo pipefail');
|
||||
expect(step('Run the sweep').run).toContain('--prior "$SWEEP_PRIOR"');
|
||||
});
|
||||
|
||||
test('dispatch inputs reach the shell only through env', () => {
|
||||
const run = step('Run the sweep');
|
||||
expect(run.run).not.toContain('${{');
|
||||
expect(run.env).toMatchObject({ SWEEP_K: '${{ inputs.k }}', SWEEP_CAP_USD: '${{ inputs.cap_usd }}', SWEEP_DRY_RUN: '${{ inputs.dry_run }}' });
|
||||
expect(run.run).toContain('bun run scripts/ship-measure.ts sweep "${args[@]}"');
|
||||
});
|
||||
|
||||
test('uploads the report and captures even when the sweep fails', () => {
|
||||
const report = step('Upload the sweep report');
|
||||
expect(report.if).toBe('always()');
|
||||
expect(report.with.name).toBe('ship-measure-sweep-report');
|
||||
expect(report.with.path).toContain('sweep-report.md');
|
||||
expect(step('Upload the sweep captures').if).toBe('always()');
|
||||
});
|
||||
});
|
||||
+2
@@ -2,6 +2,8 @@
|
||||
"newModules": [
|
||||
{ "file": "scripts/lib/shard-engine.ts", "movedFrom": "scripts/test-strict-output.ts" },
|
||||
{ "file": "scripts/lib/eval-history.ts", "movedFrom": "(new: headroom, red ledger and run triage, kept out of scripts/eval-flake-rank.ts)" },
|
||||
{ "file": "scripts/lib/measure-bar.ts", "movedFrom": "(new: the ship-measure measurement bar, kept out of scripts/ship-measure.ts)" },
|
||||
{ "file": "scripts/ship-measure-sweep.ts", "movedFrom": "(new: the weekly off-ship qualification sweep, kept out of scripts/ship-measure.ts)" },
|
||||
{ "file": "scripts/lib/published-text.ts", "movedFrom": "(new: one sanitizer for model-written text reaching public reports)" },
|
||||
{ "file": "test/helpers/session-ledger.ts", "movedFrom": "(new: per-session ledger every paid runner appends to)" },
|
||||
{ "file": "scripts/lib/paid-types.ts", "movedFrom": "scripts/test-paid-shards.ts" },
|
||||
|
||||
@@ -166,7 +166,7 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
// wave's headline capability) grows the union to 1.195x. Deliberate:
|
||||
// the section is on-demand (loads only for Apple store targets), so
|
||||
// per-invocation cost for non-iOS ships is one manifest line.
|
||||
maxSizeRatio: 1.495, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06). + Oct 6 fix wave A3/CEO-12: free text reaches commands only through agent-written mktemp files (mktemp block, write rule, path rebuild and refusal at each posting, prompt and brief site); measured 275,422 / 187,706 = 1.4673 (2026-10-06). Combined Oct 6 wave head (B free-text files + D measure loop + A rework): measured 280,363 / 187,706 = 1.4936 (2026-10-06). + gate-once push rule (no second paid gate after a CI push); measured 280,573 / 187,706 = 1.4947 (2026-10-06).
|
||||
maxSizeRatio: 1.4975, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06). + Oct 6 fix wave A3/CEO-12: free text reaches commands only through agent-written mktemp files (mktemp block, write rule, path rebuild and refusal at each posting, prompt and brief site); measured 275,422 / 187,706 = 1.4673 (2026-10-06). Combined Oct 6 wave head (B free-text files + D measure loop + A rework): measured 280,363 / 187,706 = 1.4936 (2026-10-06). + gate-once push rule (no second paid gate after a CI push); measured 280,573 / 187,706 = 1.4947 (2026-10-06). + the measurement bar's decision step (exit-code branches for MEETS, MEETS-qualified, needs-classify with the qualifying and fix-round classes, EXTEND once; the measure prose was tightened to offset most of it); measured 281,058 / 187,706 = 1.4973 (2026-10-06).
|
||||
},
|
||||
'plan-ceo-review': {
|
||||
skill: 'plan-ceo-review',
|
||||
|
||||
@@ -389,7 +389,7 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
|
||||
'tpa-apple-ban': [ 'scripts/resolvers/third-party-actions.ts', 'ship/SKILL.md.tmpl', 'ship/sections/apple-release.md.tmpl', 'scripts/gen-skill-docs.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-third-party-actions.test.ts', 'test/helpers/third-party-actions.ts',
|
||||
'lib/eval-model.ts'
|
||||
],
|
||||
'ship-measure-seeded-flake': ['ship/**', 'scripts/ship-measure.ts', 'scripts/test-free-shards.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/paid-select.ts', 'scripts/lib/free-home-guard.ts', 'bin/gstack-config', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/testing.ts', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-ship-measure-loop.test.ts', 'test/helpers/ship-measure-seeded-fixture.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/skill-fixture.ts'],
|
||||
'ship-measure-seeded-flake': ['ship/**', 'scripts/ship-measure.ts', 'scripts/lib/measure-bar.ts', 'scripts/test-free-shards.ts', 'scripts/lib/shard-engine.ts', 'scripts/lib/paid-select.ts', 'scripts/lib/free-home-guard.ts', 'bin/gstack-config', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/testing.ts', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-ship-measure-loop.test.ts', 'test/helpers/ship-measure-seeded-fixture.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/skill-fixture.ts'],
|
||||
'ship-section-loading': [ 'ship/**', 'scripts/resolvers/sections.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-ship-section-loading.test.ts',
|
||||
'scripts/resolvers/testing.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
|
||||
'plan-ceo-section-loading': [
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
/**
|
||||
* The measurement bar's decision table (scripts/lib/measure-bar.ts), free and
|
||||
* deterministic, plus the runner paths that apply it (void redispatch, EXTEND
|
||||
* on identical inputs, classification records) with fake trial runners.
|
||||
* Rules (approved 2026-10-06): MEETS >= 9/10; MEETS-qualified 8/10 with every
|
||||
* red qualifying (provider on affirmative evidence, judge noise, a cited model
|
||||
* miss); EXTEND 7/10 or an unqualified 8/10, then one pooled decision on 20
|
||||
* (18 strict, 16 qualified); BELOW otherwise. Behavior uses 12 (11, 10, 9;
|
||||
* pooled 22 and 20 of 24); a judge trial is one output scored by its median
|
||||
* 3-sample panel. More than 30% provider-evidence failures voids a batch, which
|
||||
* may be redispatched once.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import {
|
||||
barThresholds, classificationProblem, decide, identityMismatch, machineClass, providerEvidence,
|
||||
type BarKind, type BarTrial, type Classification, type FailureClass, type TrialSet,
|
||||
} from '../scripts/lib/measure-bar';
|
||||
import { MEASURE_DEFAULTS, classifyTrials, decideRound, extendCase, measureCase, type TrialRequest, type TrialResult } from '../scripts/ship-measure';
|
||||
|
||||
const temps: string[] = [];
|
||||
const tmp = (prefix: string) => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); temps.push(dir); return dir; };
|
||||
afterAll(() => { for (const dir of temps) fs.rmSync(dir, { recursive: true, force: true }); });
|
||||
|
||||
const ASSERTION = { failureCause: 'assertion', failureDetail: 'expected sorted' };
|
||||
const API = { failureCause: 'api_error', failureEvidence: 'API Error: 529 overloaded' };
|
||||
const TIMEOUT = { failureCause: 'session_timeout', failureEvidence: 'armed 300s timeout fired' };
|
||||
|
||||
/** A batch of n trials in one set; `fails` maps a 1-based slot to that trial's failure fields. */
|
||||
function batch(set: TrialSet, n: number, fails: Record<number, Partial<BarTrial>> = {}, start = 0): BarTrial[] {
|
||||
return Array.from({ length: n }, (_, i) => {
|
||||
const failure = fails[i + 1];
|
||||
return { trial: start + i + 1, set, passed: !failure, ...(failure ?? {}) };
|
||||
});
|
||||
}
|
||||
const agent = (trial: number, cls: FailureClass, evidence = 'test/x.test.ts:12 read the capture'): Classification => ({ trial, class: cls, evidence, by: 'agent', at: '2026-10-06T00:00:00Z' });
|
||||
const fails = (slots: number[], fields: Partial<BarTrial> = ASSERTION) => Object.fromEntries(slots.map(s => [s, fields]));
|
||||
|
||||
describe('thresholds round up from 9/10, 8/10 and 7/10', () => {
|
||||
test('N = 10, 12, 20 and 24', () => {
|
||||
expect(barThresholds(10)).toEqual({ trials: 10, strict: 9, qualified: 8, floor: 7 });
|
||||
expect(barThresholds(12)).toEqual({ trials: 12, strict: 11, qualified: 10, floor: 9 });
|
||||
expect(barThresholds(20)).toMatchObject({ strict: 18, qualified: 16 });
|
||||
expect(barThresholds(24)).toMatchObject({ strict: 22, qualified: 20 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('one batch of 10 (rule)', () => {
|
||||
const d = (trials: BarTrial[], records: Classification[] = [], kind: BarKind = 'rule') => decide(kind, kind === 'behavior' ? 12 : 10, trials, records);
|
||||
|
||||
test('9/10 and 10/10 MEETS with no classification', () => {
|
||||
expect(d(batch('initial', 10, fails([4])))).toMatchObject({ decision: 'MEETS', next: 'none', passes: 9 });
|
||||
expect(d(batch('initial', 10))).toMatchObject({ decision: 'MEETS', passes: 10 });
|
||||
});
|
||||
|
||||
test('a contract violation is never MEETS, even at 10/10 assertions passed', () => {
|
||||
const trials = batch('initial', 10).map(t => t.trial === 3 ? { ...t, contract: true } : t);
|
||||
expect(d(trials)).toMatchObject({ decision: 'BELOW', next: 'fix' });
|
||||
});
|
||||
|
||||
test('8/10 with only qualifying reds is MEETS-qualified: machine provider, judge noise, a cited model miss', () => {
|
||||
expect(d(batch('initial', 10, fails([2, 5], API)))).toMatchObject({ decision: 'MEETS-qualified', needsClassify: [] });
|
||||
const trials = batch('initial', 10, fails([2, 5]));
|
||||
expect(d(trials, [agent(2, 'judge-noise', 'median 3 at threshold 4'), agent(5, 'model-miss', 'transcript.jsonl:88 skipped the sort step')]).decision).toBe('MEETS-qualified');
|
||||
});
|
||||
|
||||
test('a sub-strict batch with unreviewed reds is needs-classify, listing exactly those trials', () => {
|
||||
const out = d(batch('initial', 10, fails([2, 5])), [agent(2, 'judge-noise', 'at threshold')]);
|
||||
expect(out).toMatchObject({ decision: 'needs-classify', next: 'classify', needsClassify: [5] });
|
||||
expect(d(batch('initial', 10, fails([1, 2, 3]))).decision).toBe('needs-classify');
|
||||
});
|
||||
|
||||
test('8/10 with a timeout, a hang, a regression or a known fixable cause is a fix round', () => {
|
||||
expect(d(batch('initial', 10, { 2: TIMEOUT, 5: API })).decision).toBe('BELOW');
|
||||
expect(d(batch('initial', 10, { 2: { failureCause: 'provider_stall' }, 5: API })).decision).toBe('BELOW');
|
||||
for (const cls of ['regression', 'fixable', 'timeout', 'hang'] as const) {
|
||||
expect(d(batch('initial', 10, fails([2, 5])), [agent(2, cls), agent(5, 'judge-noise')])).toMatchObject({ decision: 'BELOW', next: 'fix' });
|
||||
}
|
||||
});
|
||||
|
||||
test('8/10 with a red reviewed as unclassified, or a model miss without a citation, is EXTEND', () => {
|
||||
expect(d(batch('initial', 10, fails([2, 5])), [agent(2, 'unclassified', 'reviewed: no evidence for any route'), agent(5, 'judge-noise')]))
|
||||
.toMatchObject({ decision: 'EXTEND', next: 'extend' });
|
||||
expect(d(batch('initial', 10, fails([2, 5])), [agent(2, 'model-miss', 'the model went off script'), agent(5, 'judge-noise')]).decision).toBe('EXTEND');
|
||||
});
|
||||
|
||||
test('7/10 EXTEND when every red qualifies or is reviewed; 7/10 with a fixable red is a fix round; 6/10 BELOW without classification', () => {
|
||||
expect(d(batch('initial', 10, fails([1, 2, 3], API))).decision).toBe('EXTEND');
|
||||
expect(d(batch('initial', 10, fails([1, 2, 3])), [agent(1, 'judge-noise'), agent(2, 'judge-noise'), agent(3, 'fixable')]).decision).toBe('BELOW');
|
||||
expect(d(batch('initial', 10, fails([1, 2, 3, 4])))).toMatchObject({ decision: 'BELOW', needsClassify: [] });
|
||||
});
|
||||
|
||||
test('a later classification record supersedes an earlier one for the same trial', () => {
|
||||
const trials = batch('initial', 10, fails([2, 5]));
|
||||
expect(d(trials, [agent(2, 'judge-noise'), agent(5, 'judge-noise'), agent(5, 'fixable')]).decision).toBe('BELOW');
|
||||
expect(d(trials, [agent(2, 'judge-noise'), agent(5, 'fixable'), agent(5, 'judge-noise')]).decision).toBe('MEETS-qualified');
|
||||
});
|
||||
});
|
||||
|
||||
describe('behavior (12) and judge (outputs as median panels)', () => {
|
||||
test('behavior: 11/12 MEETS, 10/12 qualified, 9/12 EXTEND, 8/12 BELOW', () => {
|
||||
expect(decide('behavior', 12, batch('initial', 12, fails([7]))).decision).toBe('MEETS');
|
||||
expect(decide('behavior', 12, batch('initial', 12, fails([1, 2], API))).decision).toBe('MEETS-qualified');
|
||||
expect(decide('behavior', 12, batch('initial', 12, fails([1, 2, 3], API))).decision).toBe('EXTEND');
|
||||
expect(decide('behavior', 12, batch('initial', 12, fails([1, 2, 3, 4]))).decision).toBe('BELOW');
|
||||
});
|
||||
|
||||
test('behavior pooled over 24: 22 MEETS, 20 qualified, 19 BELOW', () => {
|
||||
const base = batch('initial', 12, fails([1, 2, 3], API));
|
||||
expect(decide('behavior', 12, [...base, ...batch('extend', 12, {}, 12)]).decision).toBe('MEETS-qualified');
|
||||
expect(decide('behavior', 12, [...batch('initial', 12, fails([1])), ...batch('extend', 12, fails([5]), 12)]).decision).toBe('MEETS');
|
||||
expect(decide('behavior', 12, [...base, ...batch('extend', 12, fails([1, 2]), 12)], [agent(16, 'unclassified'), agent(17, 'unclassified')]).decision).toBe('BELOW');
|
||||
});
|
||||
|
||||
test('judge: each trial is one output, passing at 2 of its 3 samples whatever the runner claimed', () => {
|
||||
const outputs = batch('initial', 10).map(t => ({ ...t, samples: t.trial === 2 ? [true, false, false] : [true, false, true] }));
|
||||
expect(decide('judge', 10, outputs)).toMatchObject({ decision: 'MEETS', passes: 9 });
|
||||
const short = batch('initial', 10).map(t => ({ ...t, samples: [true, true] }));
|
||||
expect(decide('judge', 10, short).passes).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the pooled decision after EXTEND (once, never a third batch)', () => {
|
||||
const base = batch('initial', 10, fails([1, 2, 3], API));
|
||||
test('pooled 18/20 strict and 16/20 qualified MEET; 17/20 with an unqualified red is BELOW, never EXTEND', () => {
|
||||
expect(decide('rule', 10, [...batch('initial', 10, fails([1, 2])), ...batch('extend', 10, {}, 10)], [agent(1, 'unclassified'), agent(2, 'unclassified')]).decision).toBe('MEETS');
|
||||
expect(decide('rule', 10, [...base, ...batch('extend', 10, { 1: API }, 10)])).toMatchObject({ decision: 'MEETS-qualified', pooled: true, passes: 16 });
|
||||
expect(decide('rule', 10, [...base, ...batch('extend', 10, { 1: API, 2: API, 3: API, 4: ASSERTION }, 10)]).decision).toBe('BELOW');
|
||||
const unqualified = decide('rule', 10, [...batch('initial', 10, fails([1, 2])), ...batch('extend', 10, { 1: ASSERTION }, 10)], [agent(1, 'unclassified'), agent(2, 'judge-noise'), agent(11, 'unclassified')]);
|
||||
expect(unqualified).toMatchObject({ decision: 'BELOW', pooled: true, passes: 17 });
|
||||
});
|
||||
|
||||
test('an extension red nobody classified is needs-classify; one with a fixable cause cancels the extension into a fix round', () => {
|
||||
expect(decide('rule', 10, [...base, ...batch('extend', 10, fails([1]), 10)])).toMatchObject({ decision: 'needs-classify', needsClassify: [11] });
|
||||
expect(decide('rule', 10, [...base, ...batch('extend', 10, fails([1]), 10)], [agent(11, 'fixable')]).decision).toBe('BELOW');
|
||||
});
|
||||
});
|
||||
|
||||
describe('provider evidence is mechanical; void batches', () => {
|
||||
test('provider needs affirmative evidence: api_error, or pre_turn_infra naming a 429/5xx or reset; a refusal or crash never qualifies', () => {
|
||||
expect(providerEvidence({ trial: 1, set: 'initial', passed: false, ...API })).toContain('529');
|
||||
expect(providerEvidence({ trial: 1, set: 'initial', passed: false, failureCause: 'pre_turn_infra', failureEvidence: 'HTTP 429 rate limit' })).toContain('429');
|
||||
expect(providerEvidence({ trial: 1, set: 'initial', passed: false, failureCause: 'pre_turn_infra', failureEvidence: 'ECONNRESET while streaming' })).not.toBeNull();
|
||||
expect(providerEvidence({ trial: 1, set: 'initial', passed: false, failureCause: 'pre_turn_infra', failureEvidence: 'Cannot find module ./entry' })).toBeNull();
|
||||
expect(providerEvidence({ trial: 1, set: 'initial', passed: false, failureCause: 'refusal', failureEvidence: 'stop_reason refusal' })).toBeNull();
|
||||
expect(machineClass({ trial: 1, set: 'initial', passed: false, failureCause: 'refusal' })).toBeNull();
|
||||
expect(machineClass({ trial: 1, set: 'initial', passed: false, ...TIMEOUT })).toBe('timeout');
|
||||
});
|
||||
|
||||
test('provider cannot be set by hand when failure_cause disagrees; a timeout never takes a qualifying class; evidence is required', () => {
|
||||
const assertion: BarTrial = { trial: 3, set: 'initial', passed: false, ...ASSERTION };
|
||||
expect(classificationProblem(assertion, 'rule', 'provider', 'I think the API flaked')).toContain('affirmative provider or transport evidence');
|
||||
expect(classificationProblem({ trial: 3, set: 'initial', passed: false, failureCause: 'pre_turn_infra', failureEvidence: 'module load failed' }, 'rule', 'provider', 'x')).toContain('pre_turn_infra');
|
||||
expect(classificationProblem({ trial: 3, set: 'initial', passed: false, ...TIMEOUT }, 'rule', 'model-miss', 'a.ts:1')).toContain('never qualifies');
|
||||
expect(classificationProblem(assertion, 'rule', 'fixable', ' ')).toContain('--evidence');
|
||||
expect(classificationProblem(assertion, 'rule', 'flaky', 'x')).toContain('class must be one of');
|
||||
expect(classificationProblem({ trial: 3, set: 'initial', passed: true }, 'rule', 'fixable', 'x')).toContain('passed');
|
||||
expect(classificationProblem({ trial: 3, set: 'initial', passed: false, ...API }, 'rule', 'provider', 'api 529')).toBeNull();
|
||||
});
|
||||
|
||||
test('more than 30% provider-evidence failures voids the batch: one redispatch, then decided on it; void twice stops', () => {
|
||||
expect(decide('rule', 10, batch('initial', 10, fails([1, 2, 3, 4], API)))).toMatchObject({ decision: 'void', next: 'redispatch' });
|
||||
const redispatched = decide('rule', 10, [...batch('initial', 10, fails([1, 2, 3, 4], API)), ...batch('redispatch', 10, fails([5]), 10)]);
|
||||
expect(redispatched).toMatchObject({ decision: 'MEETS', passes: 9 });
|
||||
expect(redispatched.sets.map(s => [s.set, s.void])).toEqual([['initial', true], ['redispatch', false]]);
|
||||
expect(decide('rule', 10, [...batch('initial', 10, fails([1, 2, 3, 4], API)), ...batch('redispatch', 10, fails([1, 2, 3, 4], API), 10)]))
|
||||
.toMatchObject({ decision: 'void', next: 'stop' });
|
||||
expect(decide('rule', 10, batch('initial', 10, fails([1, 2, 3], API))).decision).not.toBe('void');
|
||||
});
|
||||
|
||||
test('module-load failures (pre_turn_infra without provider evidence) never void a batch', () => {
|
||||
const crash = { failureCause: 'pre_turn_infra', failureEvidence: 'SyntaxError: Export named x not found' };
|
||||
expect(decide('rule', 10, batch('initial', 10, fails([1, 2, 3, 4, 5, 6], crash)))).toMatchObject({ decision: 'BELOW' });
|
||||
});
|
||||
|
||||
test('a void extension gets its own one redispatch', () => {
|
||||
const base = batch('initial', 10, fails([1, 2, 3], API));
|
||||
expect(decide('rule', 10, [...base, ...batch('extend', 10, fails([1, 2, 3, 4], API), 10)])).toMatchObject({ decision: 'void', next: 'redispatch', pooled: true });
|
||||
expect(decide('rule', 10, [...base, ...batch('extend', 10, fails([1, 2, 3, 4], API), 10), ...batch('extend-redispatch', 10, {}, 20)]).decision).toBe('MEETS-qualified');
|
||||
});
|
||||
});
|
||||
|
||||
describe('pooling identity', () => {
|
||||
test('a changed field is named, including the CI image digest', () => {
|
||||
const a = { tree: 't1', image: 'ghcr.io/x/ci@sha256:aaa', model: 'm' };
|
||||
expect(identityMismatch(a, { ...a })).toBeNull();
|
||||
expect(identityMismatch(a, { ...a, image: 'ghcr.io/x/ci@sha256:bbb' })).toContain('image changed');
|
||||
expect(identityMismatch(a, { ...a, tree: 't2' })).toContain('tree changed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('the runner applies the bar: redispatch, EXTEND, classify', () => {
|
||||
/** A fake runner; outcomes are keyed by trial number across every set. */
|
||||
const runner = (outcome: (trial: number) => TrialResult) => {
|
||||
const calls: TrialRequest[] = [];
|
||||
return { calls, run: async (r: TrialRequest) => { calls.push(r); return outcome(r.trial); } };
|
||||
};
|
||||
const opts = (outDir: string, run: (r: TrialRequest) => Promise<TrialResult>, identity = { tree: 't1', image: 'img-a' }) => ({
|
||||
caseId: 'bar-case', round: 'baseline', config: MEASURE_DEFAULTS, runner: run, outDir, parallel: 5, costPerTrialUsd: 0.1,
|
||||
evalDir: tmp('bar-history-'), log: () => {}, identity: () => identity,
|
||||
});
|
||||
|
||||
test('a void batch is redispatched once by measure, and both batches are reported', async () => {
|
||||
const outDir = tmp('bar-void-');
|
||||
const r = runner(trial => trial <= 4 ? { passed: false, costUsd: 0.1, ...API } : { passed: true, costUsd: 0.1 });
|
||||
const m = await measureCase({ ...opts(outDir, r.run), kind: 'rule', approved: true });
|
||||
expect(r.calls).toHaveLength(20);
|
||||
expect(m).toMatchObject({ decision: 'MEETS', passes: 10, counted: 10 });
|
||||
expect(m.sets.map(s => `${s.set}:${s.passes}/${s.trials}${s.void ? ':void' : ''}`)).toEqual(['initial:6/10:void', 'redispatch:10/10']);
|
||||
});
|
||||
|
||||
test('classify records reds, refuses a hand-set provider, and EXTEND pools once on identical inputs', async () => {
|
||||
const outDir = tmp('bar-extend-');
|
||||
const r = runner(trial => [2, 5, 9].includes(trial) ? { passed: false, costUsd: 0.1, ...ASSERTION } : { passed: true, costUsd: 0.1 });
|
||||
const m = await measureCase({ ...opts(outDir, r.run), kind: 'rule', approved: true });
|
||||
expect(m.decision).toBe('needs-classify');
|
||||
expect(() => classifyTrials(outDir, 'bar-case', 'baseline', [2], 'provider', 'looked like an outage', () => {})).toThrow('affirmative provider');
|
||||
expect(classifyTrials(outDir, 'bar-case', 'baseline', [2, 5, 9], 'unclassified', 'reviewed captures t02 t05 t09: no route', () => {}).decision).toBe('EXTEND');
|
||||
const records = fs.readFileSync(path.join(outDir, 'bar-case', 'baseline', 'classifications.jsonl'), 'utf8').trim().split('\n');
|
||||
expect(records).toHaveLength(3);
|
||||
await expect(extendCase({ ...opts(outDir, r.run, { tree: 't1', image: 'img-b' }) })).rejects.toThrow('image changed');
|
||||
const pooled = await extendCase({ ...opts(outDir, r.run) });
|
||||
expect(r.calls.map(c => c.trial).slice(10)).toEqual([11, 12, 13, 14, 15, 16, 17, 18, 19, 20]);
|
||||
// 7/10 + 10/10 = 17/20: under the pooled strict bar, and its reds are not in a qualifying class.
|
||||
expect(pooled).toMatchObject({ decision: 'BELOW', counted: 20, passes: 17 });
|
||||
expect(decideRound(outDir, 'bar-case', 'baseline', () => {}).decision).toBe('BELOW');
|
||||
await expect(extendCase({ ...opts(outDir, r.run) })).rejects.toThrow('never a third batch');
|
||||
});
|
||||
|
||||
test('EXTEND is refused unless the round decided EXTEND', async () => {
|
||||
const outDir = tmp('bar-noextend-');
|
||||
const r = runner(() => ({ passed: true, costUsd: 0.1 }));
|
||||
await measureCase({ ...opts(outDir, r.run), kind: 'rule', approved: true });
|
||||
await expect(extendCase({ ...opts(outDir, r.run) })).rejects.toThrow('is MEETS, not EXTEND');
|
||||
});
|
||||
});
|
||||
@@ -1,7 +1,7 @@
|
||||
/**
|
||||
* A6 acceptance: the generated ship skill runs the measure-then-fix loop on a
|
||||
* red paid eval (classify, measure alone, fix at the cause, re-measure, gate
|
||||
* once) and measures a failed free shard before calling it pre-existing, and
|
||||
* red paid eval (classify, measure alone, decide by the measurement bar, fix
|
||||
* at the cause, re-measure, gate once) and measures a failed free shard before calling it pre-existing, and
|
||||
* it states the "diagnostic, not verdict" rule. Every ship-measure command and
|
||||
* flag the skill names exists in scripts/ship-measure.ts.
|
||||
*/
|
||||
@@ -25,8 +25,8 @@ describe('the generated ship skill runs the loop (A6 acceptance)', () => {
|
||||
expect(skeleton).toContain('| `sections/measure.md` |');
|
||||
});
|
||||
|
||||
test('the loop steps run in order: classify, plan, measure, fix, re-measure, stop, then gate once', () => {
|
||||
const steps = ['1. **Classify.**', '2. **Show the plan.**', '3. **Measure.**', '4. **Fix at the cause.**', '5. **Re-measure.**', '6. **Stop.**', '**Gate once.**'];
|
||||
test('the loop steps run in order: classify, plan, measure, decide, fix, re-measure, stop, then gate once', () => {
|
||||
const steps = ['1. **Classify.**', '2. **Show the plan.**', '3. **Measure.**', '4. **Decide by the exit code**', '5. **Fix at the cause.**', '6. **Re-measure.**', '7. **Stop.**', '**Gate once.**'];
|
||||
const positions = steps.map(step => loop.indexOf(step));
|
||||
expect(positions.every(p => p >= 0)).toBe(true);
|
||||
expect([...positions].sort((a, b) => a - b)).toEqual(positions);
|
||||
@@ -36,6 +36,17 @@ describe('the generated ship skill runs the loop (A6 acceptance)', () => {
|
||||
expect(loop).toContain('do not also start the gate command');
|
||||
});
|
||||
|
||||
test('the decision branches on every exit code the runner returns, with the bar and its rules', () => {
|
||||
const decideStep = loop.slice(loop.indexOf('4. **Decide by the exit code**'), loop.indexOf('5. **Fix at the cause.**'));
|
||||
for (const code of ['- 0: MEETS or MEETS-qualified', '- 5: needs-classify', '- 6: EXTEND', '- 1: BELOW']) expect(decideStep).toContain(code);
|
||||
for (const cls of ['provider', 'judge-noise', 'model-miss', 'timeout', 'hang', 'regression', 'fixable', 'unclassified']) expect(decideStep).toContain(`\`${cls}\``);
|
||||
expect(decideStep).toMatch(/MEETS 9\/10, MEETS-qualified 8\/10[\s\S]*EXTEND 7\/10; behavior 11\/12, 10\/12, 9\/12/);
|
||||
expect(decideStep).toMatch(/never a third batch/i);
|
||||
expect(loop.slice(loop.indexOf('7. **Stop.**'))).toContain('Exit 3 is a named red');
|
||||
const source = fs.readFileSync(path.join(ROOT, 'scripts/lib/measure-bar.ts'), 'utf8');
|
||||
for (const cls of ['provider', 'judge-noise', 'model-miss', 'timeout', 'hang', 'regression', 'fixable', 'unclassified']) expect(source).toContain(`'${cls}'`);
|
||||
});
|
||||
|
||||
test('the diagnostic-not-verdict rule, the honest-fix rule and the unmeasured label are stated', () => {
|
||||
expect(loop).toContain('Every rerun here is **diagnostic, not a verdict**: it never changes the recorded\nverdict of the run that failed');
|
||||
expect(loop).toContain('Never raise a budget, lower a\nthreshold, add a verdict retry or skip a case to get a pass.');
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
/**
|
||||
* Free checks of the seeded-flake fixture the paid case drives (CEO-3,
|
||||
* ENG-13), measured through the real scripts/ship-measure.ts CLI: unsorted it
|
||||
* measures 7 of 10 with failures that name the line, parallel trials claim
|
||||
* measures 7 of 10 with failures that name the line, which the bar sends to
|
||||
* needs-classify and, classified fixable, to a fix round; parallel trials claim
|
||||
* unique slots that reset per round, an unrelated edit still measures 7 of 10
|
||||
* and leaves the gate red, and only the sorting fix measures 10 of 10 and turns
|
||||
* the gate green.
|
||||
@@ -31,8 +32,8 @@ describe('seeded-flake fixture (free)', () => {
|
||||
expect(gate().status).toBe(1);
|
||||
|
||||
const baseline = measure('baseline');
|
||||
expect(baseline.stdout).toContain(`DIAGNOSTIC ${SEEDED_CASE} baseline: observed 7/10 trials`);
|
||||
expect(baseline.status).toBe(1);
|
||||
expect(baseline.stdout).toContain(`DIAGNOSTIC ${SEEDED_CASE} baseline: observed 7/10 trials (baseline); needs-classify`);
|
||||
expect(baseline.status).toBe(5);
|
||||
const failed = fs.readdirSync(roundDir('baseline')).filter(t => /^t\d+$/.test(t))
|
||||
.map(t => JSON.parse(fs.readFileSync(path.join(roundDir('baseline'), t, 'trial.json'), 'utf8'))).filter(t => !t.passed);
|
||||
expect(failed).toHaveLength(3);
|
||||
@@ -43,17 +44,22 @@ describe('seeded-flake fixture (free)', () => {
|
||||
const slotsOf = (dir: string) => readFixtureRuns(log).filter(r => r.kind === 'case' && r.measureDir?.startsWith(dir)).map(r => r.slot).sort((a, b) => a! - b!);
|
||||
expect(slotsOf(roundDir('baseline'))).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
|
||||
expect(readFixtureRuns(log).filter(r => r.kind === 'case' && r.result === 'fail').map(r => r.slot).sort()).toEqual([2, 5, 9]);
|
||||
const classify = spawnSync(process.execPath, ['run', path.join(ROOT, 'scripts/ship-measure.ts'), 'classify', '--case', SEEDED_CASE, '--round', 'baseline',
|
||||
'--trials', failed.map(t => t.trial).join(','), '--class', 'fixable', '--evidence', `src/priorities.js:${SEEDED_LINE} returns storage order`], { cwd: repo, env, encoding: 'utf8', timeout: 60_000 });
|
||||
expect(classify.stdout).toContain('BELOW (fix round)');
|
||||
expect(classify.status).toBe(1);
|
||||
|
||||
fs.writeFileSync(path.join(repo, 'src/priorities.js'), UNRELATED_EDIT);
|
||||
const control = measure('round-1');
|
||||
expect(control.stdout).toContain(`DIAGNOSTIC ${SEEDED_CASE} round-1: observed 7/10 trials`);
|
||||
expect(control.status).toBe(1);
|
||||
expect(control.status).toBe(5);
|
||||
expect(slotsOf(roundDir('round-1'))).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
|
||||
expect(gate().status).toBe(1);
|
||||
|
||||
fs.writeFileSync(path.join(repo, 'src/priorities.js'), SORTED_FIX);
|
||||
const fixed = measure('round-2');
|
||||
expect(fixed.stdout).toContain(`DIAGNOSTIC ${SEEDED_CASE} round-2: observed 10/10 trials`);
|
||||
expect(fixed.stdout).toContain('; MEETS');
|
||||
expect(fixed.status).toBe(0);
|
||||
expect(gate().status).toBe(0);
|
||||
const runs = readFixtureRuns(log);
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
/**
|
||||
* Free contracts for the weekly off-ship qualification sweep
|
||||
* (scripts/ship-measure-sweep.ts): which gate cases it picks from pass-rate
|
||||
* history, the predicted all-green probability before and after, the weekly
|
||||
* spend cap enforced through the admission budget across earlier sweep
|
||||
* reports, the report it writes after every case, and a $0 CLI run (dry run,
|
||||
* and a fake single-case command). Nothing here makes a paid call.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { EVAL_POLICY } from '../test/helpers/periodic-exclude-data';
|
||||
import { TRIAL_OUTCOME_SCHEMA, type TrialOutcomeRecord } from '../test/helpers/eval-store';
|
||||
import { MEASURE_DEFAULTS, type TrialRequest, type TrialResult } from '../scripts/ship-measure';
|
||||
import { SWEEP_SCHEMA, measuredRedRate, planSweep, predictedAllGreen, runSweep, weeklySpent, type SweepReport } from '../scripts/ship-measure-sweep';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const temps: string[] = [];
|
||||
const tmp = (prefix: string) => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); temps.push(dir); return dir; };
|
||||
afterAll(() => { for (const dir of temps) fs.rmSync(dir, { recursive: true, force: true }); });
|
||||
|
||||
const A = 'ship-skipped-queued-finding';
|
||||
const B = 'investigate-owned-completion';
|
||||
const C = 'investigate-owned-abort';
|
||||
const Q = 'investigate-owned-ending-error';
|
||||
|
||||
function record(id: string, run: number, failed: boolean, extra: Partial<TrialOutcomeRecord> = {}): TrialOutcomeRecord {
|
||||
return {
|
||||
schema: TRIAL_OUTCOME_SCHEMA, case: id, file: `test/${id}.test.ts`, tier: 'gate', kind: 'rule', trial: 1, panel: { n: 1, k: 1 }, attempt: 1,
|
||||
outcome: failed ? 'failed' : 'passed', ...(failed ? { failure_class: 'assertion' as const } : {}), duration_ms: 1000, cost_usd: 1,
|
||||
policy_version: EVAL_POLICY.version, quarantined: false, execution: 'executed', source: 'shard', run_id: String(run), lane: 'gate',
|
||||
series_identity: 's1', model: 'm', cli_version: 'c', recorded_at: `2026-09-${10 + run}T00:00:00Z`, ...extra,
|
||||
} as TrialOutcomeRecord;
|
||||
}
|
||||
|
||||
/** Three weekly runs: A red twice, B red once, C never, a periodic case red (not gate), a retired id red. */
|
||||
function history(): TrialOutcomeRecord[] {
|
||||
const out: TrialOutcomeRecord[] = [];
|
||||
for (const run of [1, 2, 3]) {
|
||||
out.push(record(A, run, run !== 3), record(B, run, run === 2), record(C, run, false));
|
||||
out.push(record('retired-case-id', run, true), record('some-periodic-case', run, true, { tier: 'periodic', lane: 'periodic' }));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('picking cases from gate history', () => {
|
||||
test('ranks live, non-quarantined gate cases with a red by their cost to the all-green probability, and picks K', () => {
|
||||
const withQuarantined = [...history(), ...[1, 2, 3].map(run => record(Q, run, true))];
|
||||
expect(planSweep(withQuarantined, 3, { quarantine: {} }).picked.map(c => c.case)).toEqual([Q, A, B]);
|
||||
const plan = planSweep(withQuarantined, 2, { quarantine: { [Q]: {} } });
|
||||
expect(plan.candidates.map(c => c.case)).toEqual([A, B]);
|
||||
expect(plan.picked.map(c => c.case)).toEqual([A, B]);
|
||||
const [a, b] = plan.candidates;
|
||||
expect(a).toMatchObject({ reds: 2, verdicts: 3, scoredTrials: 3, costPerTrialUsd: 1, kind: 'rule' });
|
||||
expect(a!.passRate).toBeCloseTo(1 / 3, 5);
|
||||
expect(a!.redRate).toBeGreaterThan(b!.redRate);
|
||||
expect(a!.contribution).toBeGreaterThan(b!.contribution);
|
||||
expect(plan.censusCases).toEqual([A, B, C, Q, 'retired-case-id'].sort());
|
||||
const product = plan.censusCases.reduce((p, id) => p * (1 - plan.rates[id]!), 1);
|
||||
expect(plan.allGreen).toBeCloseTo(product, 10);
|
||||
expect(planSweep(history(), 1).picked.map(c => c.case)).toEqual([A]);
|
||||
});
|
||||
|
||||
test('a measured case replaces its history rate; a behavior rate converts to the 3-trial panel red risk', () => {
|
||||
expect(measuredRedRate('rule', 0, 10, 0)).toBe(0);
|
||||
expect(measuredRedRate('rule', 1, 10, 0.05)).toBeCloseTo((1 + 4 * 0.05) / 14, 10);
|
||||
const q = 2 / 16;
|
||||
expect(measuredRedRate('behavior', 2, 12, 0)).toBeCloseTo(1 - ((1 - q) ** 3 + 3 * (1 - q) ** 2 * q), 10);
|
||||
const plan = planSweep(history(), 2);
|
||||
const after = predictedAllGreen(plan, { [A]: 0 })!;
|
||||
expect(after).toBeCloseTo(plan.allGreen! / (1 - plan.rates[A]!), 10);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the weekly cap', () => {
|
||||
const report = (dir: string, id: string, startedAt: string, chargedUsd: number, dryRun = false) => {
|
||||
fs.mkdirSync(path.join(dir, id), { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, id, 'sweep-report.json'), JSON.stringify({ schema: SWEEP_SCHEMA, sweepId: id, startedAt, chargedUsd, dryRun }));
|
||||
};
|
||||
test('sums the last 7 days of sweep reports once each, skipping dry runs, older reports and this sweep', () => {
|
||||
const dir = tmp('sweep-prior-');
|
||||
const now = Date.parse('2026-10-12T12:00:00Z');
|
||||
report(dir, 'a', '2026-10-10T00:00:00Z', 40);
|
||||
report(path.join(dir, 'copy'), 'a', '2026-10-10T00:00:00Z', 40);
|
||||
report(dir, 'b', '2026-10-06T00:00:00Z', 25);
|
||||
report(dir, 'old', '2026-10-01T00:00:00Z', 100);
|
||||
report(dir, 'dry', '2026-10-11T00:00:00Z', 0, true);
|
||||
report(dir, 'self', '2026-10-12T11:00:00Z', 9);
|
||||
expect(weeklySpent([dir], now, 'self')).toBe(65);
|
||||
expect(weeklySpent([path.join(dir, 'missing')], now)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('running the sweep with a fake runner', () => {
|
||||
const fake = (fail: (r: TrialRequest) => boolean) => {
|
||||
const calls: TrialRequest[] = [];
|
||||
const runner = async (r: TrialRequest): Promise<TrialResult> => { calls.push(r); return fail(r) ? { passed: false, costUsd: 1, failureCause: 'assertion' } : { passed: true, costUsd: 1 }; };
|
||||
return { calls, runner };
|
||||
};
|
||||
const base = (outDir: string, runner: (r: TrialRequest) => Promise<TrialResult>, extra: Partial<Parameters<typeof runSweep>[0]> = {}) => runSweep({
|
||||
records: history(), historyScope: 'fixture history', k: 2, capUsd: 150, config: MEASURE_DEFAULTS, outDir, priorDirs: [], dryRun: false, parallel: 5,
|
||||
ref: 'main', sha: 'a'.repeat(40), runner, runnerId: 'fake', preflight: () => ({ ok: true, detail: 'fake' }), identity: () => ({ tree: 'fixed' }),
|
||||
evalDir: tmp('sweep-history-'), log: () => {}, now: Date.parse('2026-10-12T12:00:00Z'), ...extra,
|
||||
});
|
||||
|
||||
test('measures each picked case at its kind\'s N with the bar, writes the report, and predicts the all-green probability after', async () => {
|
||||
const outDir = tmp('sweep-out-');
|
||||
const run = fake(r => r.caseId === A && r.trial <= 4);
|
||||
const report = await base(outDir, run.runner);
|
||||
expect(run.calls.filter(c => c.caseId === A)).toHaveLength(10);
|
||||
expect(run.calls.filter(c => c.caseId === B)).toHaveLength(10);
|
||||
expect(report.rows.map(r => [r.case, r.status, r.decision, `${r.measuredPasses}/${r.measuredTrials}`])).toEqual([[A, 'measured', 'BELOW', '6/10'], [B, 'measured', 'MEETS', '10/10']]);
|
||||
expect(report).toMatchObject({ label: 'diagnostic', verdict: null, chargedUsd: 20, priorWeekUsd: 0, k: 2 });
|
||||
expect(report.allGreenAfter).not.toBe(report.allGreenBefore);
|
||||
const dir = path.join(outDir, `sweep-${report.sweepId}`);
|
||||
const saved = JSON.parse(fs.readFileSync(path.join(dir, 'sweep-report.json'), 'utf8')) as SweepReport;
|
||||
expect(saved.rows[0]!.captures).toBe(path.join(`sweep-${report.sweepId}`, A, 'baseline'));
|
||||
expect(fs.existsSync(path.join(outDir, saved.rows[0]!.captures!, 't01', 'trial.json'))).toBe(true);
|
||||
const md = fs.readFileSync(path.join(dir, 'sweep-report.md'), 'utf8');
|
||||
expect(md).toContain('| # | Case | Kind | History rate | Measured | Decision | Cost (est / actual) | Captures |');
|
||||
expect(md).toMatch(/Predicted gate all-green probability .*before \d+(\.\d)?% → after \d+(\.\d)?%/);
|
||||
expect(md).toContain(`| 1 | ${A} | rule | 33.3% of 3 trials; 2/3 red verdicts`);
|
||||
expect(md).toContain('never pushes or opens a pull request');
|
||||
});
|
||||
|
||||
test('stops at the weekly cap: a partly funded case stops at the budget, the rest are skipped and listed', async () => {
|
||||
const outDir = tmp('sweep-cap-');
|
||||
const prior = tmp('sweep-cap-prior-');
|
||||
fs.writeFileSync(path.join(prior, 'sweep-report.json'), JSON.stringify({ schema: SWEEP_SCHEMA, sweepId: 'earlier', startedAt: '2026-10-08T00:00:00Z', chargedUsd: 136, dryRun: false }));
|
||||
const run = fake(() => false);
|
||||
const report = await base(outDir, run.runner, { priorDirs: [prior], capUsd: 150, k: 3, records: history().map(r => r.case === C && r.run_id === '1' ? record(C, 1, true) : r) });
|
||||
expect(report.priorWeekUsd).toBe(136);
|
||||
// B and C tie (one red in three); ties fall to the case id, so C ranks before B.
|
||||
expect(report.rows.map(r => [r.case, r.status])).toEqual([[A, 'measured'], [C, 'budget_exhausted'], [B, 'skipped']]);
|
||||
// A partly funded case reports the trials that ran, not the planned N.
|
||||
expect([report.rows[1]!.measuredPasses, report.rows[1]!.measuredTrials]).toEqual([4, 4]);
|
||||
expect(run.calls).toHaveLength(14);
|
||||
expect(report.chargedUsd).toBe(14);
|
||||
expect(report.rows[2]!.note).toContain('weekly cap $150 exhausted');
|
||||
expect(fs.readFileSync(path.join(outDir, `sweep-${report.sweepId}`, 'sweep-report.md'), 'utf8')).toContain(`- ${B}: weekly cap $150 exhausted`);
|
||||
});
|
||||
|
||||
test('a dry run plans and estimates without running anything or charging the cap', async () => {
|
||||
const outDir = tmp('sweep-dry-');
|
||||
const run = fake(() => false);
|
||||
const report = await base(outDir, run.runner, { dryRun: true });
|
||||
expect(run.calls).toHaveLength(0);
|
||||
expect(report.rows.map(r => [r.case, r.status, r.estimatedUsd])).toEqual([[A, 'planned', 10], [B, 'planned', 10]]);
|
||||
expect(report.chargedUsd).toBe(0);
|
||||
expect(weeklySpent([outDir], Date.parse('2026-10-12T13:00:00Z'))).toBe(0);
|
||||
});
|
||||
|
||||
test('a case whose selection preflight fails is skipped before any paid call', async () => {
|
||||
const run = fake(() => false);
|
||||
const report = await base(tmp('sweep-preflight-'), run.runner, { k: 1, preflight: () => ({ ok: false, detail: '--list planned nothing' }) });
|
||||
expect(report.rows[0]).toMatchObject({ status: 'skipped', note: 'selection preflight failed (no paid call made): --list planned nothing' });
|
||||
expect(run.calls).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('ship-measure sweep CLI ($0)', () => {
|
||||
const historyDir = () => {
|
||||
const dir = tmp('sweep-cli-history-');
|
||||
fs.writeFileSync(path.join(dir, 'trial-outcomes.jsonl'), `${history().map(r => JSON.stringify(r)).join('\n')}\n`);
|
||||
return dir;
|
||||
};
|
||||
const cli = (cwd: string, ...args: string[]) => spawnSync(process.execPath, ['run', path.join(ROOT, 'scripts/ship-measure.ts'), 'sweep', ...args], {
|
||||
cwd, encoding: 'utf8', timeout: 120_000, env: { ...process.env, GSTACK_STATE_ROOT: tmp('sweep-cli-state-'), GITHUB_STEP_SUMMARY: '' },
|
||||
});
|
||||
|
||||
test('--dry-run reads local history, prints the plan and the before probability, and spends nothing', () => {
|
||||
const cwd = tmp('sweep-cli-');
|
||||
const r = cli(cwd, '--history-dir', historyDir(), '--k', '2', '--dry-run', '--out', path.join(cwd, 'out'));
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stdout).toContain('(dry run: nothing ran, $0)');
|
||||
expect(r.stdout).toContain(`| 1 | ${A} | rule |`);
|
||||
expect(r.stdout).toContain('would measure 10 trials');
|
||||
});
|
||||
|
||||
test('a fake single-case command runs the whole sweep end to end at $0', () => {
|
||||
const cwd = tmp('sweep-cli-fake-');
|
||||
// A Bun script, not a shell stub, so the fake runner also executes on Windows.
|
||||
const script = path.join(cwd, 'fake-trial.ts');
|
||||
fs.writeFileSync(script, 'console.log("cost_usd: 0");\nif (process.env.GSTACK_SHIP_MEASURE_TRIAL === "3") { console.log("failure_cause: assertion"); process.exit(1); }\nconsole.log(`PASS ${process.argv[2]}`);\n');
|
||||
const bun = /\s/.test(process.execPath) ? 'bun' : process.execPath;
|
||||
const r = cli(cwd, '--history-dir', historyDir(), '--k', '1', '--cap-usd', '1', '--command', `${bun} ${script} {case}`, '--jobs', '5', '--out', path.join(cwd, 'out'));
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stdout).toContain(`DIAGNOSTIC ${A} baseline: observed 9/10 trials (baseline); MEETS`);
|
||||
expect(r.stdout).toMatch(new RegExp(`\\| 1 \\| ${A} \\| rule \\| [^|]+ \\| 9/10 \\| MEETS \\|`));
|
||||
});
|
||||
|
||||
test('history that cannot be read fails closed with nothing measured', () => {
|
||||
const cwd = tmp('sweep-cli-nohistory-');
|
||||
const r = spawnSync(process.execPath, ['run', path.join(ROOT, 'scripts/ship-measure.ts'), 'sweep', '--dry-run', '--repo', 'garrytan/no-such-repo-xyz', '--out', path.join(cwd, 'out')], {
|
||||
cwd, encoding: 'utf8', timeout: 120_000, env: { ...process.env, PATH: path.dirname(process.execPath), GSTACK_STATE_ROOT: tmp('sweep-cli-state-') },
|
||||
});
|
||||
expect(r.status).toBe(1);
|
||||
expect(r.stderr).toContain('pass-rate history unavailable');
|
||||
});
|
||||
});
|
||||
+50
-46
@@ -1,7 +1,8 @@
|
||||
/**
|
||||
* Free contracts for scripts/ship-measure.ts, /ship's measure-then-fix runner:
|
||||
* per-kind aggregation (rule trials, behavior panels of 3 at their own 2 of 3,
|
||||
* judge outputs scored by 3-sample panels), unique artifact directories, the
|
||||
* per-kind batches decided by the measurement bar (rule trials, behavior
|
||||
* trials in panels of 3, judge outputs scored by 3-sample panels; the pure
|
||||
* decision table lives in test/ship-measure-bar.test.ts), unique artifact directories, the
|
||||
* diagnostic label, the estimated admission budget, the ask-once rule, the
|
||||
* repair-round limit, the unmeasured skip, the PR table and the free-suite
|
||||
* reruns with the flaky retry off. Every runner here is a fake; nothing paid.
|
||||
@@ -12,7 +13,7 @@ import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import {
|
||||
MEASURE_DEFAULTS, aggregate, assertDiagnosticDir, formatKindTable, formatReport, freeParallelism, judgeFile, kindPlan,
|
||||
MEASURE_DEFAULTS, assertDiagnosticDir, formatKindTable, formatReport, freeParallelism, judgeFile, kindPlan,
|
||||
measureCase, measureFreeShard, readMeasureConfig, recordUnmeasured,
|
||||
type MeasureConfig, type MeasureKind, type TrialRecord, type TrialRequest, type TrialResult,
|
||||
} from '../scripts/ship-measure';
|
||||
@@ -43,29 +44,32 @@ const measure = (kind: MeasureKind, run: FakeRun, extra: Partial<Parameters<type
|
||||
const outDir = extra.outDir ?? tmp('ship-measure-out-');
|
||||
return measureCase({
|
||||
caseId: 'seeded-case', kind, round: 'baseline', config: MEASURE_DEFAULTS, runner: run.runner, outDir, parallel: 4,
|
||||
costPerTrialUsd: 0.5, approved: false, evalDir: tmp('ship-measure-history-'), log: () => {}, ...extra,
|
||||
costPerTrialUsd: 0.5, approved: false, evalDir: tmp('ship-measure-history-'), log: () => {}, identity: () => ({ tree: 'fixed' }), ...extra,
|
||||
});
|
||||
};
|
||||
|
||||
describe('per-kind plans and the printed table (CEO-9, DX-7)', () => {
|
||||
test('defaults: rule 10 trials at 9, behavior 4 panels of 3 at 11 of 12, judge 10 outputs at 9', () => {
|
||||
expect(kindPlan('rule', MEASURE_DEFAULTS)).toEqual({ kind: 'rule', units: 10, trialsPerUnit: 1, trials: 10, unitTarget: 9, trialTarget: 9 });
|
||||
expect(kindPlan('behavior', MEASURE_DEFAULTS)).toEqual({ kind: 'behavior', units: 4, trialsPerUnit: 3, trials: 12, unitTarget: 4, trialTarget: 11 });
|
||||
expect(kindPlan('judge', MEASURE_DEFAULTS)).toEqual({ kind: 'judge', units: 10, trialsPerUnit: 1, trials: 10, unitTarget: 9, trialTarget: 9 });
|
||||
test('defaults: rule 10, behavior 4 panels of 3 (12), judge 10 outputs; the table prints the bar per kind', () => {
|
||||
expect(kindPlan('rule', MEASURE_DEFAULTS)).toMatchObject({ trials: 10, trialsPerUnit: 1, bar: { strict: 9, qualified: 8, floor: 7 }, pooledBar: { trials: 20, strict: 18, qualified: 16 } });
|
||||
expect(kindPlan('behavior', MEASURE_DEFAULTS)).toMatchObject({ trials: 12, trialsPerUnit: 3, bar: { strict: 11, qualified: 10, floor: 9 }, pooledBar: { trials: 24, strict: 22, qualified: 20 } });
|
||||
expect(kindPlan('judge', MEASURE_DEFAULTS)).toMatchObject({ trials: 10, trialsPerUnit: 1, bar: { strict: 9, qualified: 8, floor: 7 } });
|
||||
const table = formatKindTable(MEASURE_DEFAULTS);
|
||||
expect(table).toContain('| rule | 10 | at least 9 of 10 trials |');
|
||||
expect(table).toContain('every panel at 2 of 3, no contract violation, at least 11 of 12 trials | 4 of 3 trials');
|
||||
expect(table).toContain('| judge | 10 outputs | at least 9 of 10 outputs | 3 samples per output, passes at 2 of 3');
|
||||
expect(table).toContain('| rule | 10 | 9 of 10 | 8 of 10, every red qualifying | 7 of 10 (or 8 that does not qualify): 10 more on identical inputs, then 18 of 20, or 16 of 20 qualified |');
|
||||
expect(table).toContain('| behavior | 12 (4 panels of 3) | 11 of 12 | 10 of 12, every red qualifying | 9 of 12 (or 10 that does not qualify): 12 more on identical inputs, then 22 of 24, or 20 of 24 qualified |');
|
||||
expect(table).toContain('| judge | 10 outputs, each its median 3-sample panel | 9 of 10 outputs |');
|
||||
expect(table).toContain('$25 total; asks above $2/trial');
|
||||
expect(table).toContain('Repair rounds: at most 3');
|
||||
expect(table).toContain('never changes a recorded verdict');
|
||||
expect(table).toMatch(/more than 30%[^\n]*void[^\n]*redispatched once/);
|
||||
});
|
||||
|
||||
test('config keys override defaults; an empty value is the default; an invalid value throws', () => {
|
||||
const values: Record<string, string> = { ship_measure_rule_trials: '20', ship_measure_budget_usd: '7.5', ship_rerun_backend: 'ubicloud', ship_measure_max_rounds: '' };
|
||||
const config = readMeasureConfig(key => values[key]);
|
||||
expect(config).toEqual({ ...MEASURE_DEFAULTS, ruleTrials: 20, budgetUsd: 7.5, rerunBackend: 'ubicloud' });
|
||||
expect(kindPlan('rule', config).trialTarget).toBe(18);
|
||||
expect(kindPlan('rule', config).bar.strict).toBe(18);
|
||||
expect(readMeasureConfig(key => ({ ship_measure_sweep_cases: '2', ship_measure_sweep_budget_usd: '12.5' } as Record<string, string>)[key])).toMatchObject({ sweepCases: 2, sweepBudgetUsd: 12.5 });
|
||||
expect(() => readMeasureConfig(key => key === 'ship_measure_sweep_budget_usd' ? '-1' : undefined)).toThrow('ship_measure_sweep_budget_usd');
|
||||
expect(() => readMeasureConfig(key => key === 'ship_measure_behavior_panels' ? '0' : undefined)).toThrow('ship_measure_behavior_panels');
|
||||
expect(() => readMeasureConfig(key => key === 'ship_rerun_backend' ? 'cloud' : undefined)).toThrow('not local or ubicloud');
|
||||
});
|
||||
@@ -76,60 +80,56 @@ describe('per-kind plans and the printed table (CEO-9, DX-7)', () => {
|
||||
const config = (...args: string[]) => spawnSync('bash', [path.join(ROOT, 'bin/gstack-config'), ...args], { encoding: 'utf8', timeout: 30_000, env });
|
||||
const defaults = config('defaults').stdout;
|
||||
for (const [key, value] of [['ship_measure_rule_trials', '10'], ['ship_measure_behavior_panels', '4'], ['ship_measure_judge_outputs', '10'],
|
||||
['ship_measure_ask_per_trial_usd', '2'], ['ship_measure_budget_usd', '25'], ['ship_measure_max_rounds', '3'], ['ship_rerun_backend', 'local']]) {
|
||||
['ship_measure_ask_per_trial_usd', '2'], ['ship_measure_budget_usd', '25'], ['ship_measure_max_rounds', '3'], ['ship_rerun_backend', 'local'],
|
||||
['ship_measure_sweep_cases', '5'], ['ship_measure_sweep_budget_usd', '150']]) {
|
||||
expect(defaults).toMatch(new RegExp(`${key}:\\s+${value}\\n`));
|
||||
expect(config('get', key!).stdout.trim()).toBe(value!);
|
||||
}
|
||||
expect(config('set', 'ship_measure_budget_usd', '0').status).toBe(1);
|
||||
expect(config('set', 'ship_measure_rule_trials', '2.5').status).toBe(1);
|
||||
expect(config('set', 'ship_rerun_backend', 'cloud').status).toBe(1);
|
||||
expect(config('set', 'ship_measure_sweep_budget_usd', '0').status).toBe(1);
|
||||
expect(config('set', 'ship_measure_sweep_cases', 'two').status).toBe(1);
|
||||
expect(config('set', 'ship_measure_budget_usd', '12.5').status).toBe(0);
|
||||
expect(readMeasureConfig(key => config('get', key).stdout.trim()).budgetUsd).toBe(12.5);
|
||||
expect(config('list').stdout).toMatch(/ship_measure_budget_usd:\s+12\.5 \(set/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('aggregation per kind with fake runners (ENG-12)', () => {
|
||||
test('rule: 8 of 10 is below target, 9 of 10 meets; every trial gets its own diagnostic directory', async () => {
|
||||
describe('per-kind batches decided by the measurement bar, with fake runners (ENG-12)', () => {
|
||||
test('rule: 8 of 10 needs its reds classified, 9 of 10 MEETS; every trial gets its own diagnostic directory', async () => {
|
||||
const red = fake(trial => ({ passed: ![3, 6].includes(trial), costUsd: 0.5, ...(trial === 3 ? { failureCause: 'assertion', failureDetail: 'unsorted list' } : {}) }));
|
||||
const m = await measure('rule', red);
|
||||
expect(m).toMatchObject({ status: 'measured', label: 'diagnostic', verdict: null, unitPasses: 8, meets: false });
|
||||
expect(m).toMatchObject({ status: 'measured', label: 'diagnostic', verdict: null, passes: 8, counted: 10, decision: 'needs-classify', next: 'classify' });
|
||||
expect(red.calls).toHaveLength(10);
|
||||
const dirs = m.trials.map(t => t.dir);
|
||||
expect(new Set(dirs).size).toBe(10);
|
||||
for (const dir of dirs) {
|
||||
expect(path.basename(path.dirname(dir))).toBe('baseline');
|
||||
expect(JSON.parse(fs.readFileSync(path.join(dir, 'trial.json'), 'utf8'))).toMatchObject({ label: 'diagnostic', verdict: null });
|
||||
expect(JSON.parse(fs.readFileSync(path.join(dir, 'trial.json'), 'utf8'))).toMatchObject({ label: 'diagnostic', verdict: null, set: 'initial' });
|
||||
}
|
||||
const saved = JSON.parse(fs.readFileSync(path.join(path.dirname(dirs[0]!), 'measurement.json'), 'utf8'));
|
||||
expect(saved).toMatchObject({ label: 'diagnostic', verdict: null, round: 'baseline', unitPasses: 8, meets: false });
|
||||
expect(saved).toMatchObject({ label: 'diagnostic', verdict: null, round: 'baseline', passes: 8, decision: 'needs-classify', identity: { tree: 'fixed' } });
|
||||
expect(saved.trials.find((t: TrialRecord) => t.trial === 3)).toMatchObject({ failureCause: 'assertion', failureDetail: 'unsorted list' });
|
||||
expect((await measure('rule', fake(trial => ({ passed: trial !== 4, costUsd: 0.5 })))).meets).toBe(true);
|
||||
expect((await measure('rule', fake(trial => ({ passed: trial !== 4, costUsd: 0.5 })))).decision).toBe('MEETS');
|
||||
});
|
||||
|
||||
test('behavior: each panel of 3 needs its own 2 of 3, and 11 of 12 trials overall', async () => {
|
||||
// Trials 1 and 2 fail: 10 of 12 pass, which a single 2-of-12 panel would PASS; panel 1 is 1 of 3.
|
||||
const panelRed = await measure('behavior', fake(trial => ({ passed: trial > 2 })));
|
||||
expect(panelRed).toMatchObject({ unitPasses: 3, trialPasses: 10, meets: false });
|
||||
// One failure in each of two panels: every panel passes at 2 of 3, but 10 of 12 is under 11.
|
||||
const tooMany = await measure('behavior', fake(trial => ({ passed: trial !== 1 && trial !== 4 })));
|
||||
expect(tooMany).toMatchObject({ unitPasses: 4, trialPasses: 10, meets: false });
|
||||
test('behavior: 12 trials in panels of 3, decided on the trial count (11 MEETS, 10 needs classifying); a contract violation is a fix round', async () => {
|
||||
const oneMiss = await measure('behavior', fake(trial => ({ passed: trial !== 7 })));
|
||||
expect(oneMiss).toMatchObject({ unitPasses: 4, trialPasses: 11, meets: true });
|
||||
expect(oneMiss).toMatchObject({ passes: 11, counted: 12, decision: 'MEETS' });
|
||||
expect(oneMiss.trials.map(t => t.unit)).toEqual([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4]);
|
||||
// A contract violation fails its panel even when the trial's assertions passed.
|
||||
expect((await measure('behavior', fake(trial => ({ passed: trial > 2 })))).decision).toBe('needs-classify');
|
||||
const contract = await measure('behavior', fake(trial => ({ passed: true, ...(trial === 5 ? { contract: true } : {}) })));
|
||||
expect(contract).toMatchObject({ unitPasses: 3, trialPasses: 11, meets: false });
|
||||
expect(contract).toMatchObject({ passes: 11, decision: 'BELOW', next: 'fix' });
|
||||
});
|
||||
|
||||
test('judge: an output passes at 2 of its 3 samples, whatever the runner claimed; 9 of 10 outputs meets', async () => {
|
||||
test('judge: an output passes at 2 of its 3 samples, whatever the runner claimed; 9 of 10 outputs MEETS', async () => {
|
||||
const judged = await measure('judge', fake(trial => ({ passed: true, samples: trial === 2 ? [true, false, false] : [true, false, true] })));
|
||||
expect(judged).toMatchObject({ unitPasses: 9, meets: true });
|
||||
expect(judged).toMatchObject({ passes: 9, decision: 'MEETS' });
|
||||
const red = await measure('judge', fake(trial => ({ passed: true, samples: trial <= 2 ? [false, false, true] : [true, true, true] })));
|
||||
expect(red).toMatchObject({ unitPasses: 8, meets: false });
|
||||
const wrongCount = aggregate(kindPlan('judge', { ...MEASURE_DEFAULTS, judgeOutputs: 1 }),
|
||||
[{ trial: 1, unit: 1, batch: 1, dir: '/x', passed: true, samples: [true, true] }]);
|
||||
expect(wrongCount.meets).toBe(false);
|
||||
expect(red).toMatchObject({ passes: 8, decision: 'needs-classify' });
|
||||
const wrongCount = await measure('judge', fake(() => ({ passed: true, samples: [true, true] })));
|
||||
expect(wrongCount).toMatchObject({ passes: 0, decision: 'BELOW' });
|
||||
});
|
||||
|
||||
test('standalone judge ids resolve to the one paid file that names them', () => {
|
||||
@@ -147,7 +147,7 @@ describe('spend, approval, rounds and artifacts', () => {
|
||||
expect(baseline).toMatchObject({ status: 'budget_exhausted', estimatedUsd: 24, actualUsd: 24 });
|
||||
expect(baseline.trials.map(t => t.batch)).toEqual([1, 1, 1, 1, 2, 2, 2, 2]);
|
||||
expect(run.maxConcurrent).toBeLessThanOrEqual(4);
|
||||
expect(baseline.reason).toContain('8 of 10 trials ran');
|
||||
expect(baseline.reason).toContain('8 of 10 initial trials ran');
|
||||
// The budget is per red case: a repair round starts from what the baseline spent.
|
||||
const round = await measure('rule', fake(() => ({ passed: true, costUsd: 3 })), { outDir, round: 'round-1', costPerTrialUsd: 3 });
|
||||
expect(round.status).toBe('budget_exhausted');
|
||||
@@ -158,7 +158,7 @@ describe('spend, approval, rounds and artifacts', () => {
|
||||
const m = await measure('rule', fake(() => ({ passed: true, costUsd: 1 })), { costPerTrialUsd: 2.5, approved: true, config: { ...MEASURE_DEFAULTS, budgetUsd: 12 } });
|
||||
// Batch 1 reserves 4 x 2.5 = 10 but spends 4; batch 2 then fits 3 (8 / 2.5); batch 3 fits 2 (5 / 2.5); batch 4 one more.
|
||||
expect(m.trials.map(t => t.batch)).toEqual([1, 1, 1, 1, 2, 2, 2, 3, 3, 4]);
|
||||
expect(m).toMatchObject({ status: 'measured', actualUsd: 10, estimatedUsd: 25, meets: true });
|
||||
expect(m).toMatchObject({ status: 'measured', actualUsd: 10, estimatedUsd: 25, decision: 'MEETS' });
|
||||
});
|
||||
|
||||
test('asks once: no estimate or a per-trial estimate above the threshold runs nothing until approved', async () => {
|
||||
@@ -205,9 +205,9 @@ describe('spend, approval, rounds and artifacts', () => {
|
||||
recordUnmeasured(outDir, 'browse-basic', 'provider outage before the first turn');
|
||||
expect(() => recordUnmeasured(outDir, 'other', ' ')).toThrow('--reason');
|
||||
const report = formatReport(outDir);
|
||||
expect(report).toContain('| seeded-case | rule | baseline | observed 7/10 | 9/10 | $5.00 | $2.50 | below target |');
|
||||
expect(report).toContain('| seeded-case | rule | round-1: after fix at sort the shuffled list in src/order.js | observed 10/10 | 9/10 | $5.00 | $2.50 | at or above target |');
|
||||
expect(report).toContain('| browse-basic | — | — | — | — | — | — | unmeasured (provider outage before the first turn); not a pass |');
|
||||
expect(report).toContain('| seeded-case | rule | baseline | observed 7/10 | 9/10 strict, 8/10 qualified | needs-classify | $5.00 | $2.50 |');
|
||||
expect(report).toContain('| seeded-case | rule | round-1: after fix at sort the shuffled list in src/order.js | observed 10/10 | 9/10 strict, 8/10 qualified | MEETS | $5.00 | $2.50 |');
|
||||
expect(report).toContain('| browse-basic | — | — | — | — | unmeasured (provider outage before the first turn); not a pass | — | — |');
|
||||
expect(report).toContain('never change a recorded verdict');
|
||||
});
|
||||
});
|
||||
@@ -226,8 +226,8 @@ if [ "$GSTACK_SHIP_MEASURE_TRIAL" = 2 ]; then echo "failure_cause: assertion"; e
|
||||
});
|
||||
const out = path.join(project, '.context', 'ship-measure');
|
||||
const red = cli('measure', '--case', 'flaky-sort', '--command', './evals.sh case {case}', '--cost-per-trial', '0.1', '--jobs', '3');
|
||||
expect(red.stdout).toContain('| rule | 10 | at least 9 of 10 trials |');
|
||||
expect(red.stdout).toContain('DIAGNOSTIC flaky-sort baseline: observed 9/10 trials (baseline); at or above target');
|
||||
expect(red.stdout).toContain('| rule | 10 | 9 of 10 |');
|
||||
expect(red.stdout).toContain('DIAGNOSTIC flaky-sort baseline: observed 9/10 trials (baseline); MEETS');
|
||||
expect(red.status).toBe(0);
|
||||
const trial2 = JSON.parse(fs.readFileSync(path.join(out, 'flaky-sort', 'baseline', 't02', 'trial.json'), 'utf8'));
|
||||
expect(trial2).toMatchObject({ passed: false, costUsd: 0.1, failureCause: 'assertion', failureDetail: 'order.js:3 unsorted', label: 'diagnostic' });
|
||||
@@ -341,18 +341,22 @@ describe('diagnostic trials prove they ran and never touch the judge cache', ()
|
||||
expect(junitExecuted(dirWith(junit('<testcase name="a" time="1"/>')))).toBe(true);
|
||||
expect(junitExecuted(dirWith(junit('<testcase name="a" time="1"></testcase><testcase name="b"><skipped/></testcase>')))).toBe(true);
|
||||
});
|
||||
test('the judge input cache is never inherited by a diagnostic trial', () => {
|
||||
const env = diagnosticBaseEnv({ EVALS_CACHE_DIR: '/c', EVALS_CACHE_RUNTIME_ID: 'img', KEEP: '1' }, '/e');
|
||||
test('the judge input cache is never inherited by a diagnostic trial; each trial gets its own run id', () => {
|
||||
const env = diagnosticBaseEnv({ EVALS_CACHE_DIR: '/c', EVALS_CACHE_RUNTIME_ID: 'img', KEEP: '1' }, '/e', 'case-baseline-t3');
|
||||
expect(env).not.toHaveProperty('EVALS_CACHE_DIR');
|
||||
expect(env).not.toHaveProperty('EVALS_CACHE_RUNTIME_ID');
|
||||
expect(env).toMatchObject({ KEEP: '1', GSTACK_EVAL_DIR: '/e', GSTACK_SHIP_MEASURE_LABEL: 'diagnostic' });
|
||||
expect(env).toMatchObject({ KEEP: '1', GSTACK_EVAL_DIR: '/e', GSTACK_SHIP_MEASURE_LABEL: 'diagnostic', EVALS_RUN_ID: 'local-measure-case-baseline-t3' });
|
||||
// Every trial has its own run id (evidence-retaining cases refuse to start without one); CI's id is kept as the prefix.
|
||||
expect(diagnosticBaseEnv({ EVALS_RUN_ID: 'ci-9-1-eval-sweep' }, '/e', 'x-t1').EVALS_RUN_ID).toBe('ci-9-1-eval-sweep-measure-x-t1');
|
||||
});
|
||||
});
|
||||
|
||||
describe('the measure CLI proves --case selection before any paid trial', () => {
|
||||
test('a real case plans its own test file; an unknown case is refused', () => {
|
||||
test('a real case plans its own test file, in file mode or name mode; an unknown case is refused', () => {
|
||||
const { caseSelectionPreflight } = require('../scripts/ship-measure');
|
||||
expect(caseSelectionPreflight('ship-measure-seeded-flake')).toEqual({ ok: true, detail: '--case ship-measure-seeded-flake selects test/skill-e2e-ship-measure-loop.test.ts' });
|
||||
expect(caseSelectionPreflight('no-such-case').ok).toBe(false);
|
||||
// A name-mode case (one of several in its file) lists its trial shard as <file>#<id>~t1.
|
||||
expect(caseSelectionPreflight('ship-exploratory-late-input')).toEqual({ ok: true, detail: '--case ship-exploratory-late-input selects test/skill-e2e-qa-callers.test.ts' });
|
||||
});
|
||||
});
|
||||
@@ -26,6 +26,12 @@
|
||||
* pasted developer-facing plan: without one it reviews the checkout's
|
||||
* branch diff, and a tests-only diff legitimately ends at its
|
||||
* applicability gate, which is neither 'asked' nor 'plan_ready'.
|
||||
* - plan-ceo-review reviews a pasted plan too. Without one it reviews the
|
||||
* checkout's branch diff, so its work before the first question (the
|
||||
* pre-review audit and landscape searches) grew with whatever PR the gate
|
||||
* ran on: on a 1,900-line PR it timed out at 300s in 3 of 10 measured
|
||||
* trials while still researching that diff. A fixed small plan keeps the
|
||||
* work the same on every branch.
|
||||
* - named-target case: a pasted draft (initialPlanContent) IS an
|
||||
* explicitly-named target, so the gate must NOT ask — and the review
|
||||
* must actually consume the pasted content.
|
||||
@@ -62,6 +68,15 @@ weekly export emails. One new component, one route, one test file.
|
||||
- test/${SEED_TOKEN}.test.tsx (new)
|
||||
`;
|
||||
|
||||
const CEO_PLAN = `
|
||||
# Plan: weekly digest email
|
||||
|
||||
## Scope
|
||||
Send each workspace owner a weekly digest email listing the tasks closed
|
||||
that week. One scheduled job, one email template, one opt-out toggle in
|
||||
settings.
|
||||
`;
|
||||
|
||||
const DEVEX_PLAN = `
|
||||
# Plan: tasks export command
|
||||
|
||||
@@ -96,6 +111,7 @@ describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => {
|
||||
? { extraArgs: ['--disallowedTools', 'AskUserQuestion'], requireProseEvidence: true }
|
||||
: {}),
|
||||
...(skillName === 'plan-devex-review' ? { initialPlanContent: DEVEX_PLAN } : {}),
|
||||
...(skillName === 'plan-ceo-review' ? { initialPlanContent: CEO_PLAN } : {}),
|
||||
});
|
||||
|
||||
if (obs.outcome === 'silent_write' || obs.outcome === 'exited' || obs.outcome === 'timeout') {
|
||||
|
||||
Reference in new issue
Block a user