diff --git a/AGENTS.md b/AGENTS.md index d2d545325..6c86be265 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -213,8 +213,8 @@ When fixing failures or preparing `/ship`, follow this order: do not satisfy coverage; preserve every attempt. Paid evals never retry. Each case's kind (`E2E_KINDS`) fixes its trials before the run: `rule` one trial; `behavior` a panel of 3 independent trials, PASS at >= 2 with no contract - violation; `judge` 3 samples on one output, gated on the mean against the - unchanged threshold. Never add trials, samples or dispatches after seeing a + violation; `judge` 3 samples on one output, each dimension gated on its + median (at least 2 of 3 samples) against the unchanged threshold. Never add trials, samples or dispatches after seeing a result, never change a kind to change a verdict without pass-rate evidence, and report every trial. Quarantine follows `CASE_QUARANTINE`'s entry and exit rules only (`EVAL_POLICY`, `docs/TESTING_INTERNALS.md`). A census whose every diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b4eacc9f..8ccf10e2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ The first three rows are the ones users feel. The last three are why the next ce ### Behavior changes you may notice +- **LLM judges pass a dimension when 2 of their 3 samples meet its threshold** (EVAL_POLICY v3, the per-dimension median; it was the mean, so one low sample beside two passing ones failed the panel). Thresholds, sample count, judge model and prompts are unchanged; panel logs still show every sample and the mean. Policy v3 and HARNESS_VERSION 2 start a new pass-rate series for every case. - **Office-hours review verdicts are schema version 2** (`severity` and `changed_text` on every finding). An in-progress review saved by an older version is refused; start a fresh review directory. - **Bun below 1.3.3 stops setup** with `gstack needs Bun 1.3.3 or newer`; nothing is installed. Fix: `bun upgrade`, then `./setup`. - **Auto-update can say `update held (bun-too-old: ...)`** and `/gstack-upgrade` can stop with `BUN_TOO_OLD`; your current version keeps working. Fix: `bun upgrade`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 506350920..4cdcd74fa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -284,9 +284,10 @@ constants in `EVAL_POLICY` (`test/helpers/periodic-exclude-data.ts`): PASS at 2 or more with no contract violation (`expectContract()`). Use it only when a live model choice decides the verdict and an occasional deviation is acceptable product behavior; the one-line reason goes in `BEHAVIOR_WHY`. -- `judge`: an LLM judge scoring a fixed input; 3 samples of the same prompt, - gated on the per-dimension mean (booleans on a majority) against the - unchanged threshold. An erroring sample fails the panel and is never resampled. +- `judge`: an LLM judge scoring a fixed input; exactly 3 samples of the same + prompt, each dimension gated on its median, at least 2 of 3 samples, against + the unchanged threshold (booleans on a majority; the mean is reported only). + An erroring sample fails the panel and is never resampled. A timed-out, crashed or infrastructure-failed trial counts as a failed trial and is reported with its class; a missing trial makes the case INCOMPLETE, which @@ -544,8 +545,8 @@ Override the judge model per run with `GSTACK_EVAL_MODEL_JUDGE`: - **Actionability** — Can the agent execute tasks using only the information in the doc? Each dimension is scored 1-5 by a panel of 3 samples of the same prompt, drawn -concurrently; each dimension's panel mean must meet that judge's threshold (≥ 4 -for most dimensions; see each case). An erroring sample fails the panel. There's also a regression test that compares generated docs against the hand-maintained baseline from `origin/main` — generated must score equal or higher. +concurrently; each dimension's panel median (at least 2 of 3 samples) must meet +that judge's threshold (≥ 4 for most dimensions; see each case). An erroring sample fails the panel. There's also a regression test that compares generated docs against the hand-maintained baseline from `origin/main` — generated must score equal or higher. Needs `ANTHROPIC_API_KEY` in `.env`. The judge files run in every paid lane (`bun run eval:bg:pr` selects the ones your diff touches). diff --git a/TODOS.md b/TODOS.md index 71c3ba7a1..ca00db47d 100644 --- a/TODOS.md +++ b/TODOS.md @@ -291,7 +291,7 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 36898545245, 36787269090, 36641820398, 36633323521) across skills and checks (a two-`(recommended)` option count; office-hours substance). It is kind `rule` but gates on one judge sample. Decide under EVAL_POLICY whether it is - a `judge` case (3 samples, mean) and supply pass-rate evidence before + a `judge` case (3 samples, median) and supply pass-rate evidence before changing the kind. Effort S. **Priority:** P2. - **qa/SKILL.md workflow judge dips below its actionability floor** — census 37186854666 (b24b6d8) scored actionability 3.67 < 4 (clarity 3, completeness diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 5fb850d13..3cd498af8 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -427,15 +427,18 @@ and the kind fixes its trials before the run: - `judge`: an LLM judge scoring a fixed input. `judgePanel()` (`test/helpers/llm-judge.ts`) draws 3 samples of the same prompt concurrently inside the unchanged `JUDGE_MS`; numeric dimensions gate on the per-dimension - mean against the unchanged threshold (no dimension compensates for another), - booleans on a strict majority. A sample that errors (refusal, truncation, + median of exactly 3 samples, so at least 2 of 3 must meet the unchanged + threshold (`judgePanelMedian()`; no dimension compensates for another, and + any other sample count fails closed), booleans on a strict majority. Panel + logs show every sample and the mean for information; the median is the gate + (EVAL_POLICY v3, approved 2026-10-06). A sample that errors (refusal, truncation, non-JSON, a malformed field) fails the panel and is never resampled; a refusal counts as an unscored panel only when every sample refused. `callJudge`'s 429 backoff happens before any model output and is transport, not a verdict retry. The workflow-judge cache stores whole panels only. The `rule` case auq-matrix uses the same panel inside its one execution: each skill's single native capture has its recommendation substance scored by 3 - judge samples, gated on their mean against the unchanged minimum of 4, and a + judge samples, gated on their median against the unchanged minimum of 4, and a failed sample fails the panel without resampling (approved 2026-10-04 for censuses after the change). @@ -485,7 +488,8 @@ this checkout to read it)`. `failure_detail` carries the values `error` cannot hold: a failed Bun matcher's `{expected, received}`, or for a judge one `{dimension, mean, threshold, -samples, rationale}` entry per failing dimension, each part capped at 200 +samples, rationale}` entry per failing dimension (since EVAL_POLICY v3 `mean` +holds the panel's gating median), each part capped at 200 characters. `sessions[]` (at most 32) summarizes the trial's session-ledger rows. `cost_known: false` marks a record or trial whose harness captured no billing: PTY and Codex sessions never bill, and an SDK session bills only with a diff --git a/docs/evals/census-red.md b/docs/evals/census-red.md index a9620032d..f6950ff3a 100644 --- a/docs/evals/census-red.md +++ b/docs/evals/census-red.md @@ -37,7 +37,7 @@ with one failed trial shown): | case, kind, verdict | the registry id, its kind (`rule`, `behavior`, `judge`) and the verdict with each trial's outcome; the brackets name the slice and attempt of each trial | | `failure_class` | the class verdicts read: `assertion`, `contract`, `timeout` or `infra` | | `cause` | what the machine observed (`failure_cause`, glossary in [TESTING_INTERNALS](../TESTING_INTERNALS.md#failure-causes)), with its evidence line | -| Expected / Received | the failed matcher's values; for a judge, each failing dimension's mean, threshold, sample count and lowest rationale | +| Expected / Received | the failed matcher's values; for a judge, each failing dimension's gating median (the mean before EVAL_POLICY v3), threshold, sample count and lowest rationale | | `evidence:` | the slice artifact and shard directory holding the transcript | | `after a repair:` | the command that runs only this case, once a repair exists | diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 96af7e1eb..9339d6e5a 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -766,10 +766,11 @@ their case. This is informational; the corpus still replays them. and check it with `bun test test/detector-corpus-.test.ts`. -### `recommendation substance mean < 4 over samples [,,] (boilerplate/weak)` +### `recommendation substance median < 4 over samples [,,] (boilerplate/weak)` **Meaning.** The auq-matrix case scores each captured question's recommendation -with a 3-sample judge panel; their mean was below 4. +with a 3-sample judge panel; fewer than 2 of the 3 samples reached 4, so their +median was below 4. **Fix.** Read the logged samples and the captured question. A fix goes in the skill text, followed by one diagnostic run of auq-matrix. diff --git a/scripts/harness-version.json b/scripts/harness-version.json index 50c18df52..1524bde23 100644 --- a/scripts/harness-version.json +++ b/scripts/harness-version.json @@ -1,5 +1,5 @@ { - "version": 1, + "version": 2, "patterns": [ "test/helpers/agent-sdk-runner.ts", "test/helpers/codex-session-runner.ts", @@ -10,44 +10,44 @@ "test/helpers/llm-judge.ts" ], "files": { - "scripts/lib/paid-cases.ts": {"blob":"9bacd52446192a7ecf3dfaae901ea443cf9c9ae2","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "scripts/lib/paid-plan.ts": {"blob":"45721ce128a112666a0caed6c13c66637b74ca54","since":1,"nonBehavioral":"audit-wave integration before the first v2 census (EVAL_POLICY v2 already starts fresh series)"}, - "scripts/lib/paid-report.ts": {"blob":"3fda96a8b1f7f3d55797934847353f254cc4d054","since":1,"nonBehavioral":"caseSelection runs single-case files whole (local --case diagnosis only; CI slices never take this path)"}, - "scripts/lib/published-text.ts": {"blob":"3f2e4d616b5dd0204035710c66b64d49055c5714","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "scripts/lib/shard-engine.ts": {"blob":"3cd0b1aaa84018b00bf9ea0ce48093f3c8492f66","since":1,"nonBehavioral":"audit-wave integration before the first v2 census (EVAL_POLICY v2 already starts fresh series)"}, - "scripts/test-paid-shards.ts": {"blob":"8ddee5f11ed7856876135520dd8025ab989e2db9","since":1,"nonBehavioral":"--case CLI passes its own runner to runCaseDiagnosis (a self-import during top-level await deadlocked); CI slices never take this path"}, - "scripts/test-pr-profile.ts": {"blob":"d7860897397de8fd605623570d1e7ebbbd5ec743","since":1,"nonBehavioral":"audit-wave integration before the first v2 census (EVAL_POLICY v2 already starts fresh series)"}, - "scripts/test-strict-output.ts": {"blob":"01185d20dc9761b04b9b2f751ec98664fe0eaf93","since":1}, - "test/helpers/agent-sdk-runner.ts": {"blob":"210bba92d241a90df578b2c7fc943ac5755d0025","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/auq-substance-panel.ts": {"blob":"85d785f25fe6a24277d4009f9ccb7b1e05a270ef","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/claude-pty-runner.ts": {"blob":"20cc5894532873aa6145fc6617fdeec8cef94a2d","since":1}, - "test/helpers/codex-session-runner.ts": {"blob":"4eea35931a35a0afb006363402e3d4191e379737","since":1}, - "test/helpers/e2e-gate.ts": {"blob":"efb7e197dd5f66c3ad0e34074b14e640754f7a45","since":1,"nonBehavioral":"audit-wave integration before the first v2 census (EVAL_POLICY v2 already starts fresh series)"}, - "test/helpers/e2e-helpers.ts": {"blob":"2d4a2a53063cfc1f003db366d2c861ab1e379e6e","since":1}, - "test/helpers/eval-budgets.ts": {"blob":"667d0e09b20fe5d7366f57b77b5fb9b04baa37ef","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/eval-store.ts": {"blob":"bc135ddcf13a45c9803995df2af1aff54a471e2d","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/hermetic-env.ts": {"blob":"6473f78285384f4420f48cbe5928658919078e0b","since":1}, - "test/helpers/llm-judge.ts": {"blob":"453a4c1d355f3811deca53f953767109a9ec9292","since":1}, - "test/helpers/paid-test-set.ts": {"blob":"ff9bfe43c43ec2bdd11114f7b8ec7c5736116f75","since":1,"nonBehavioral":"audit-wave integration before the first v2 census (EVAL_POLICY v2 already starts fresh series)"}, - "test/helpers/pty-screen.ts": {"blob":"4f77569f9ce11f9cee9f347216e468cd7c956820","since":1}, - "test/helpers/pty/auq.ts": {"blob":"16e5ea2bb5b9f28f44808a1a09024dbd3e6f28c4","since":1}, - "test/helpers/pty/binary.ts": {"blob":"404d7515076d5c00804f89e23ea69296312c00e1","since":1}, - "test/helpers/pty/boundaries.ts": {"blob":"53eb76e57c0558c00254366d6334a1c833126575","since":1}, - "test/helpers/pty/classify.ts": {"blob":"3f2daa12ff6484f96b8d28d910a50231c4bb3ae8","since":1}, - "test/helpers/pty/fake-session.ts": {"blob":"554e064ef259b8444280eddfe9b0db93c66a8202","since":1}, - "test/helpers/pty/judge.ts": {"blob":"81c5bb4c48ab928aca2f186a1ade9df7169d7fbe","since":1}, - "test/helpers/pty/launch.ts": {"blob":"4ef0bcb7b5e7c35d2aac0cbffd7f8a303440b6fe","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/pty/plan-native.ts": {"blob":"ddba9cb68c7c0da935cff7c1cea258dee2dff740","since":1}, - "test/helpers/pty/runners/counting.ts": {"blob":"8c5c05e54abf961259d34d9c3d77db67138d7794","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/pty/runners/floor.ts": {"blob":"465f01641fb3a18ae1c9d2613a9c2b26f8fb503e","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/pty/runners/observation.ts": {"blob":"1c2b75fbb984708725abaa8939125f7cab969f9f","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/pty/screen.ts": {"blob":"ee005f36ffa48617e6832ccbeb35e8e96873df9b","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/pty/session.ts": {"blob":"58533cea9d81cc87bb3b499bdfe0db8e3a8709c5","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/session-drain-policy.ts": {"blob":"6211239dd0485e6495a700b824a7e0e61d1bcd97","since":1}, - "test/helpers/session-ledger.ts": {"blob":"6e3c9e837e46f98addba2d12eaebff802da320bb","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/session-runner.ts": {"blob":"d40eb29e8b8fdc0675b511693e4a90823b5963fc","since":1,"nonBehavioral":"reliability follow-ups integration before the first v2 census: session ledger, trial cause/cost fields and stream liveness are diagnostics; EVAL_POLICY v2 already starts fresh series"}, - "test/helpers/skill-fixture.ts": {"blob":"b188064594eb4201a54a80ba1bc6f28e31d7f49e","since":1}, - "test/helpers/test-selection.ts": {"blob":"51655f89f66be7a5ec487e2abfe4744300ebd5f6","since":1}, - "test/helpers/touchfiles.ts": {"blob":"15f369056d41b9dc42dcd86f81f60c878074704c","since":1} + "scripts/lib/paid-cases.ts": {"blob":"9bacd52446192a7ecf3dfaae901ea443cf9c9ae2","since":2}, + "scripts/lib/paid-plan.ts": {"blob":"45721ce128a112666a0caed6c13c66637b74ca54","since":2}, + "scripts/lib/paid-report.ts": {"blob":"3fda96a8b1f7f3d55797934847353f254cc4d054","since":2}, + "scripts/lib/published-text.ts": {"blob":"3f2e4d616b5dd0204035710c66b64d49055c5714","since":2}, + "scripts/lib/shard-engine.ts": {"blob":"3cd0b1aaa84018b00bf9ea0ce48093f3c8492f66","since":2}, + "scripts/test-paid-shards.ts": {"blob":"8ddee5f11ed7856876135520dd8025ab989e2db9","since":2}, + "scripts/test-pr-profile.ts": {"blob":"d7860897397de8fd605623570d1e7ebbbd5ec743","since":2}, + "scripts/test-strict-output.ts": {"blob":"01185d20dc9761b04b9b2f751ec98664fe0eaf93","since":2}, + "test/helpers/agent-sdk-runner.ts": {"blob":"210bba92d241a90df578b2c7fc943ac5755d0025","since":2}, + "test/helpers/auq-substance-panel.ts": {"blob":"4881260ef793668e4963a062db6d06a5e1f7a40f","since":2}, + "test/helpers/claude-pty-runner.ts": {"blob":"20cc5894532873aa6145fc6617fdeec8cef94a2d","since":2}, + "test/helpers/codex-session-runner.ts": {"blob":"4eea35931a35a0afb006363402e3d4191e379737","since":2}, + "test/helpers/e2e-gate.ts": {"blob":"efb7e197dd5f66c3ad0e34074b14e640754f7a45","since":2}, + "test/helpers/e2e-helpers.ts": {"blob":"2d4a2a53063cfc1f003db366d2c861ab1e379e6e","since":2}, + "test/helpers/eval-budgets.ts": {"blob":"667d0e09b20fe5d7366f57b77b5fb9b04baa37ef","since":2}, + "test/helpers/eval-store.ts": {"blob":"609c6d099108ff2696eb014b47aa318a618dd2af","since":2}, + "test/helpers/hermetic-env.ts": {"blob":"6473f78285384f4420f48cbe5928658919078e0b","since":2}, + "test/helpers/llm-judge.ts": {"blob":"54e3306b494fb5d7f132ccdaaae5c680465028f9","since":2}, + "test/helpers/paid-test-set.ts": {"blob":"ff9bfe43c43ec2bdd11114f7b8ec7c5736116f75","since":2}, + "test/helpers/pty-screen.ts": {"blob":"4f77569f9ce11f9cee9f347216e468cd7c956820","since":2}, + "test/helpers/pty/auq.ts": {"blob":"16e5ea2bb5b9f28f44808a1a09024dbd3e6f28c4","since":2}, + "test/helpers/pty/binary.ts": {"blob":"404d7515076d5c00804f89e23ea69296312c00e1","since":2}, + "test/helpers/pty/boundaries.ts": {"blob":"53eb76e57c0558c00254366d6334a1c833126575","since":2}, + "test/helpers/pty/classify.ts": {"blob":"3f2daa12ff6484f96b8d28d910a50231c4bb3ae8","since":2}, + "test/helpers/pty/fake-session.ts": {"blob":"554e064ef259b8444280eddfe9b0db93c66a8202","since":2}, + "test/helpers/pty/judge.ts": {"blob":"81c5bb4c48ab928aca2f186a1ade9df7169d7fbe","since":2}, + "test/helpers/pty/launch.ts": {"blob":"4ef0bcb7b5e7c35d2aac0cbffd7f8a303440b6fe","since":2}, + "test/helpers/pty/plan-native.ts": {"blob":"ddba9cb68c7c0da935cff7c1cea258dee2dff740","since":2}, + "test/helpers/pty/runners/counting.ts": {"blob":"8c5c05e54abf961259d34d9c3d77db67138d7794","since":2}, + "test/helpers/pty/runners/floor.ts": {"blob":"465f01641fb3a18ae1c9d2613a9c2b26f8fb503e","since":2}, + "test/helpers/pty/runners/observation.ts": {"blob":"1c2b75fbb984708725abaa8939125f7cab969f9f","since":2}, + "test/helpers/pty/screen.ts": {"blob":"ee005f36ffa48617e6832ccbeb35e8e96873df9b","since":2}, + "test/helpers/pty/session.ts": {"blob":"58533cea9d81cc87bb3b499bdfe0db8e3a8709c5","since":2}, + "test/helpers/session-drain-policy.ts": {"blob":"6211239dd0485e6495a700b824a7e0e61d1bcd97","since":2}, + "test/helpers/session-ledger.ts": {"blob":"6e3c9e837e46f98addba2d12eaebff802da320bb","since":2}, + "test/helpers/session-runner.ts": {"blob":"d40eb29e8b8fdc0675b511693e4a90823b5963fc","since":2}, + "test/helpers/skill-fixture.ts": {"blob":"b188064594eb4201a54a80ba1bc6f28e31d7f49e","since":2}, + "test/helpers/test-selection.ts": {"blob":"51655f89f66be7a5ec487e2abfe4744300ebd5f6","since":2}, + "test/helpers/touchfiles.ts": {"blob":"15f369056d41b9dc42dcd86f81f60c878074704c","since":2} } } diff --git a/scripts/judge-calibration-configs.ts b/scripts/judge-calibration-configs.ts index 95f1607f6..fe5fe984b 100644 --- a/scripts/judge-calibration-configs.ts +++ b/scripts/judge-calibration-configs.ts @@ -10,7 +10,7 @@ import { buildCrossSkillConsistencyJudgePrompt, buildVoiceDirectiveJudgePrompt, JUDGE_SCORE_SCHEMA, QA_ANTI_REFUSAL_JUDGE_SCHEMA, CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA, VOICE_DIRECTIVE_JUDGE_SCHEMA, VOICE_DIRECTIVE_DIMENSIONS, JUDGE_SCORE_DIMENSIONS, DEFAULT_JUDGE_MAX_TOKENS, - judgePanelMean, judgePanelMajority, + judgePanelMedian, judgePanelMajority, } from '../test/helpers/llm-judge'; import { buildWorkflowJudgePrompt, WORKFLOW_JUDGE_RESPONSE_SCHEMA } from '../test/helpers/workflow-judge-input'; import { CLAUDE_FRONTIER_EVAL_MODEL } from '../lib/eval-model'; @@ -18,7 +18,7 @@ import type { CalibrationConfig, Verdict } from './judge-calibration'; type Sample = Record; -/** What production rejects: judgePanelMean needs finite numbers, judgePanelMajority needs booleans. */ +/** What production rejects: judgePanelMedian needs finite numbers, judgePanelMajority needs booleans. */ function requireFields(value: unknown, numbers: readonly string[], booleans: readonly string[] = []): void { const sample = (value ?? {}) as Sample; for (const key of numbers) { @@ -29,13 +29,13 @@ function requireFields(value: unknown, numbers: readonly string[], booleans: rea } } -const meets = (mean: Record, floors: Record): Verdict => - Object.entries(floors).every(([key, floor]) => mean[key]! >= floor) ? 'pass' : 'fail'; +const meets = (median: Record, floors: Record): Verdict => + Object.entries(floors).every(([key, floor]) => median[key]! >= floor) ? 'pass' : 'fail'; const QA_FLOORS = { clarity: 3, completeness: 3, actionability: 4 }; const scoreJudge = (floors: Record) => ({ validate: (value: unknown) => requireFields(value, JUDGE_SCORE_DIMENSIONS), - verdict: (samples: unknown[]) => meets(judgePanelMean(samples as Sample[], JUDGE_SCORE_DIMENSIONS), floors), + verdict: (samples: unknown[]) => meets(judgePanelMedian(samples as Sample[], JUDGE_SCORE_DIMENSIONS), floors), }); const workflow = (id: string, priority: number, cases: string[], extra: { agentCapability?: 'frontier'; model?: string; floors?: Record } = {}): CalibrationConfig => ({ @@ -80,7 +80,7 @@ export const CALIBRATION_CONFIGS: CalibrationConfig[] = [ oldOptions: {}, newOptions: { jsonSchema: QA_ANTI_REFUSAL_JUDGE_SCHEMA }, validate: value => requireFields(value, ['confidence'], ['would_browse']), verdict: samples => judgePanelMajority(samples as Sample[], 'would_browse') - && judgePanelMean(samples as Sample[], ['confidence']).confidence >= 4 ? 'pass' : 'fail', + && judgePanelMedian(samples as Sample[], ['confidence']).confidence >= 4 ? 'pass' : 'fail', estimatedOutputTokens: 1200, }, { @@ -89,7 +89,7 @@ export const CALIBRATION_CONFIGS: CalibrationConfig[] = [ oldOptions: {}, newOptions: { jsonSchema: CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA }, validate: value => requireFields(value, ['score'], ['consistent']), verdict: samples => judgePanelMajority(samples as Sample[], 'consistent') - && judgePanelMean(samples as Sample[], ['score']).score >= 4 ? 'pass' : 'fail', + && judgePanelMedian(samples as Sample[], ['score']).score >= 4 ? 'pass' : 'fail', estimatedOutputTokens: 1200, }, { @@ -97,7 +97,7 @@ export const CALIBRATION_CONFIGS: CalibrationConfig[] = [ build: inputs => buildVoiceDirectiveJudgePrompt(inputs.voiceSection!), oldOptions: {}, newOptions: { jsonSchema: VOICE_DIRECTIVE_JUDGE_SCHEMA }, validate: value => requireFields(value, VOICE_DIRECTIVE_DIMENSIONS), - verdict: samples => meets(judgePanelMean(samples as Sample[], VOICE_DIRECTIVE_DIMENSIONS), + verdict: samples => meets(judgePanelMedian(samples as Sample[], VOICE_DIRECTIVE_DIMENSIONS), Object.fromEntries(VOICE_DIRECTIVE_DIMENSIONS.map(key => [key, 4]))), estimatedOutputTokens: 1200, }, diff --git a/scripts/judge-controls.ts b/scripts/judge-controls.ts index 8fd12a981..d2530a5db 100644 --- a/scripts/judge-controls.ts +++ b/scripts/judge-controls.ts @@ -12,7 +12,7 @@ */ import * as fs from 'node:fs'; import * as path from 'node:path'; -import { buildQaWorkflowJudgePrompt, callJudge, judgePanelMean, DEFAULT_JUDGE_MAX_TOKENS, JUDGE_SCORE_DIMENSIONS, type CallJudgeOptions } from '../test/helpers/llm-judge'; +import { buildQaWorkflowJudgePrompt, callJudge, judgePanelMean, judgePanelMedian, DEFAULT_JUDGE_MAX_TOKENS, JUDGE_SCORE_DIMENSIONS, type CallJudgeOptions } from '../test/helpers/llm-judge'; import { buildWorkflowJudgePrompt, readWorkflowJudgeInput, QA_DISCOVERY_REFERENCES, WORKFLOW_JUDGE_RESPONSE_SCHEMA } from '../test/helpers/workflow-judge-input'; const ROOT = path.resolve(import.meta.dir, '..'); @@ -113,10 +113,11 @@ const results = await Promise.all(JUDGES.flatMap(judge => Object.entries(bundles const prompt = judge.prompt(text); const samples = await Promise.all(Array.from({ length: SAMPLES }, () => callJudge(prompt, undefined, judge.options))); const mean = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS); - const verdict = Object.entries(FLOORS).every(([key, floor]) => mean[key as keyof typeof FLOORS] >= floor) ? 'pass' : 'fail'; - return { judge: judge.name, bundle, verdict, mean, samples }; + const median = judgePanelMedian(samples, JUDGE_SCORE_DIMENSIONS); + const verdict = Object.entries(FLOORS).every(([key, floor]) => median[key as keyof typeof FLOORS] >= floor) ? 'pass' : 'fail'; + return { judge: judge.name, bundle, verdict, median, mean, samples }; }))); const out = path.join(ROOT, 'docs/evals/judge-controls/results.json'); fs.mkdirSync(path.dirname(out), { recursive: true }); fs.writeFileSync(out, JSON.stringify({ recorded_at: new Date().toISOString(), results }, null, 2) + '\n'); -for (const r of results) console.log(`${r.judge} | ${r.bundle} | ${r.verdict} | ${JSON.stringify(r.mean)}`); +for (const r of results) console.log(`${r.judge} | ${r.bundle} | ${r.verdict} | median ${JSON.stringify(r.median)} mean ${JSON.stringify(r.mean)}`); diff --git a/test/auq-substance-panel.test.ts b/test/auq-substance-panel.test.ts index 115b119d4..3647a7b7d 100644 --- a/test/auq-substance-panel.test.ts +++ b/test/auq-substance-panel.test.ts @@ -17,16 +17,16 @@ const grader = (scores: number[]) => { }; describe('auq-matrix substance panel (C4)', () => { - test('one capture is graded by a full panel and gates on the mean at the unchanged minimum', async () => { + test('one capture is graded by a full panel and gates on the median at the unchanged minimum', async () => { expect(AUQ_SUBSTANCE_MIN).toBe(4); const { grade, seen } = grader([5, 3, 4]); const panel = await auqSubstancePanel('Recommendation: A because B', grade); expect(seen).toEqual(Array(JUDGE_PANEL_SAMPLES).fill('Recommendation: A because B')); - expect(panel).toMatchObject({ mean: 4, passed: true }); + expect(panel).toMatchObject({ median: 4, mean: 4, passed: true }); }); - test('a single low sample no longer decides the case, but a low mean still fails', async () => { - expect((await auqSubstancePanel('q', grader([4, 3, 5]).grade)).passed).toBe(true); - expect((await auqSubstancePanel('q', grader([4, 3, 4]).grade))).toMatchObject({ passed: false }); + test('two of three samples at the minimum pass; two below it fail; the mean is only reported', async () => { + expect((await auqSubstancePanel('q', grader([4, 3, 4]).grade))).toMatchObject({ median: 4, passed: true }); + expect((await auqSubstancePanel('q', grader([5, 3, 3]).grade))).toMatchObject({ median: 3, passed: false }); expect((await auqSubstancePanel('q', grader([3, 3, 3]).grade)).mean).toBe(3); }); test('a failed sample fails the panel without resampling', async () => { diff --git a/test/cookie-workflow-judge-input.test.ts b/test/cookie-workflow-judge-input.test.ts index 70e2b489c..c245e04ab 100644 --- a/test/cookie-workflow-judge-input.test.ts +++ b/test/cookie-workflow-judge-input.test.ts @@ -11,7 +11,7 @@ import { selectTests } from './helpers/test-selection'; import { E2E_TOUCHFILES, LLM_JUDGE_TOUCHFILES } from './helpers/touchfiles-data'; import { selectPrProfile } from '../scripts/test-pr-profile'; import { JUDGE_MS } from './helpers/eval-budgets'; -import { JudgeRefusalError, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMean, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; +import { JudgeRefusalError, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; import { COOKIE_MANUAL_REVIEW_FILE, getCookieWorkflowManualReview, isManualReviewEntry } from './helpers/cookie-workflow-manual-review'; const ROOT = resolve(import.meta.dir, '..'); @@ -70,7 +70,7 @@ function actualCookieCallback(root: string, overrides: { const records: EvalTestEntry[] = []; const attempts = new Map(); let callback: () => Promise = async () => { throw new Error('Judge callback was not registered'); }; - new Function('describeIfSelected', 'testIfSelected', 'ROOT', 'buildCookieWorkflowJudgeInput', 'resolveEvalModel', 'callJudge', 'COOKIE_WORKFLOW_JUDGE', 'JUDGE_MS', 'WORKFLOW_JUDGE_TEST_MS', 'WORKFLOW_JUDGE_RECORD_MS', 'evalCollector', 'expect', 'console', 'readWorkflowJudgeInput', 'buildWorkflowJudgePrompt', 'prepareWorkflowJudgeCache', 'workflowJudgeAttempts', 'performance', 'setTimeout', 'clearTimeout', 'JudgeRefusalError', 'getCookieWorkflowManualReview', 'DEFAULT_JUDGE_MAX_TOKENS', 'WORKFLOW_JUDGE_RESPONSE_SCHEMA', 'validWorkflowJudgeScore', 'judgePanel', 'judgePanelMean', 'judgePanelReasoning', 'JUDGE_SCORE_DIMENSIONS', registration)( + new Function('describeIfSelected', 'testIfSelected', 'ROOT', 'buildCookieWorkflowJudgeInput', 'resolveEvalModel', 'callJudge', 'COOKIE_WORKFLOW_JUDGE', 'JUDGE_MS', 'WORKFLOW_JUDGE_TEST_MS', 'WORKFLOW_JUDGE_RECORD_MS', 'evalCollector', 'expect', 'console', 'readWorkflowJudgeInput', 'buildWorkflowJudgePrompt', 'prepareWorkflowJudgeCache', 'workflowJudgeAttempts', 'performance', 'setTimeout', 'clearTimeout', 'JudgeRefusalError', 'getCookieWorkflowManualReview', 'DEFAULT_JUDGE_MAX_TOKENS', 'WORKFLOW_JUDGE_RESPONSE_SCHEMA', 'validWorkflowJudgeScore', 'judgePanel', 'judgePanelMean', 'judgePanelMedian', 'judgePanelReasoning', 'JUDGE_SCORE_DIMENSIONS', registration)( (_suite: string, names: string[], run: () => void) => { expect(names).toEqual([NAME]); run(); }, (name: string, run: () => Promise, budget: number) => { expect(name).toBe(NAME); expect(budget).toBe(JUDGE_MS + 10_000); callback = run; }, root, buildCookieWorkflowJudgeInput, (_kind: string, explicit?: string) => explicit ?? 'fixture-model', @@ -85,7 +85,7 @@ function actualCookieCallback(root: string, overrides: { attempts, overrides.clock ? { now: overrides.clock } : performance, overrides.setTimer ?? setTimeout, overrides.clearTimer ?? clearTimeout, JudgeRefusalError, getCookieWorkflowManualReview, DEFAULT_JUDGE_MAX_TOKENS, - WORKFLOW_JUDGE_RESPONSE_SCHEMA, validWorkflowJudgeScore, judgePanel, judgePanelMean, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, + WORKFLOW_JUDGE_RESPONSE_SCHEMA, validWorkflowJudgeScore, judgePanel, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, ); return { run: () => callback(), requests, records, attempts }; } diff --git a/test/fixtures/judge-panel-replay-2026-10-05.json b/test/fixtures/judge-panel-replay-2026-10-05.json new file mode 100644 index 000000000..0c9254e83 --- /dev/null +++ b/test/fixtures/judge-panel-replay-2026-10-05.json @@ -0,0 +1,893 @@ +{ + "provenance": "Replayed evidence, not a CI pass: every judge panel printed by eval-slices (2) jobs 112032744416 (run 37389946212, head 1536f7f6b) and 112049856586 (run 37395241262, head a5450908e). Only the numeric and boolean sample fields the gates read are kept; reasoning text is omitted.", + "panels": [ + { + "job": "112032744416", + "case": "Setup block", + "samples": [ + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 3 + } + ] + }, + { + "job": "112032744416", + "case": "Browse SKILL.md", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "QA health rubric", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "QA anti-refusal", + "samples": [ + { + "would_browse": true, + "confidence": 5 + }, + { + "would_browse": true, + "confidence": 5 + }, + { + "would_browse": true, + "confidence": 5 + } + ] + }, + { + "job": "112032744416", + "case": "QA workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 3, + "completeness": 4, + "actionability": 3 + }, + { + "clarity": 3, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "Cross-skill consistency", + "samples": [ + { + "consistent": true, + "score": 4 + }, + { + "consistent": true, + "score": 4 + }, + { + "consistent": true, + "score": 4 + } + ] + }, + { + "job": "112032744416", + "case": "document-release/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "ship/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "plan-ceo-review/SKILL.md modes", + "samples": [ + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "plan-design-review/SKILL.md passes", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "plan-eng-review/SKILL.md sections", + "samples": [ + { + "clarity": 3, + "completeness": 4, + "actionability": 3 + }, + { + "clarity": 3, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 3, + "completeness": 4, + "actionability": 3 + } + ] + }, + { + "job": "112032744416", + "case": "design-review/SKILL.md fix loop", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "design-consultation/SKILL.md research", + "samples": [ + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "land-and-deploy/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "canary/SKILL.md monitoring loop", + "samples": [ + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "benchmark/SKILL.md perf collection", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "setup-deploy/SKILL.md platform setup", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "sync-gbrain/SKILL.md read-only readiness", + "samples": [ + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "retro/SKILL.md instructions", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "review/SKILL.md workflow", + "samples": [ + { + "clarity": 3, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "qa-only/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "gstack-upgrade/SKILL.md upgrade flow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112032744416", + "case": "Voice directive", + "samples": [ + { + "directness": 5, + "concreteness": 5, + "avoids_corporate": 5, + "avoids_ai_vocabulary": 4, + "connects_user_outcomes": 5 + }, + { + "directness": 5, + "concreteness": 5, + "avoids_corporate": 5, + "avoids_ai_vocabulary": 4, + "connects_user_outcomes": 5 + }, + { + "directness": 5, + "concreteness": 5, + "avoids_corporate": 5, + "avoids_ai_vocabulary": 4, + "connects_user_outcomes": 5 + } + ] + }, + { + "job": "112032744416", + "case": "setup-browser-cookies/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "document-release/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "ship/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "plan-ceo-review/SKILL.md modes", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 3, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "plan-eng-review/SKILL.md sections", + "samples": [ + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "design-review/SKILL.md fix loop", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "plan-design-review/SKILL.md passes", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "design-consultation/SKILL.md research", + "samples": [ + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 5, + "actionability": 4 + }, + { + "clarity": 3, + "completeness": 4, + "actionability": 3 + } + ] + }, + { + "job": "112049856586", + "case": "land-and-deploy/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "canary/SKILL.md monitoring loop", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 3, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "setup-deploy/SKILL.md platform setup", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "benchmark/SKILL.md perf collection", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "sync-gbrain/SKILL.md read-only readiness", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 3, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "retro/SKILL.md instructions", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "review/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "qa-only/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "gstack-upgrade/SKILL.md upgrade flow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + }, + { + "job": "112049856586", + "case": "Voice directive", + "samples": [ + { + "directness": 5, + "concreteness": 5, + "avoids_corporate": 5, + "avoids_ai_vocabulary": 4, + "connects_user_outcomes": 5 + }, + { + "directness": 5, + "concreteness": 5, + "avoids_corporate": 5, + "avoids_ai_vocabulary": 4, + "connects_user_outcomes": 5 + }, + { + "directness": 5, + "concreteness": 5, + "avoids_corporate": 5, + "avoids_ai_vocabulary": 4, + "connects_user_outcomes": 5 + } + ] + }, + { + "job": "112049856586", + "case": "setup-browser-cookies/SKILL.md workflow", + "samples": [ + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + }, + { + "clarity": 4, + "completeness": 4, + "actionability": 4 + } + ] + } + ] +} diff --git a/test/harness-version.test.ts b/test/harness-version.test.ts index 15a69946a..bb54dd458 100644 --- a/test/harness-version.test.ts +++ b/test/harness-version.test.ts @@ -131,7 +131,7 @@ describe('v2 writer', () => { expect(errors).toEqual([]); const expected = caseSeriesIdentitiesV2([id], treeEntries(ROOT))[id]!; expect(records[0]).toMatchObject({ series_identity: expected.identity, series_fingerprint: expected.fingerprint, harness_version: HARNESS_VERSION, - policy_version: 2 }); + policy_version: EVAL_POLICY.version }); fs.rmSync(dir, { recursive: true, force: true }); }); }); diff --git a/test/helpers/auq-substance-panel.ts b/test/helpers/auq-substance-panel.ts index 85d785f25..4881260ef 100644 --- a/test/helpers/auq-substance-panel.ts +++ b/test/helpers/auq-substance-panel.ts @@ -1,11 +1,12 @@ /** * auq-matrix recommendation substance (C4, approved 2026-10-04): one native * capture per skill, scored by a JUDGE_PANEL_SAMPLES panel of the same grader - * on that capture. The panel mean gates against the unchanged minimum; a sample - * error fails the panel and is never resampled (judgePanel). + * on that capture. The panel median (2 of 3 samples) gates against the unchanged + * minimum and the mean is reported; a sample error fails the panel and is never + * resampled (judgePanel). */ import { gradeAuqRecommendation } from './auq-sdk-capture'; -import { judgePanel, judgePanelMean } from './llm-judge'; +import { judgePanel, judgePanelSummary } from './llm-judge'; export const AUQ_SUBSTANCE_MIN = 4; @@ -13,6 +14,6 @@ type Grade = Awaited>; export async function auqSubstancePanel(text: string, grade: (text: string) => Promise = gradeAuqRecommendation) { const samples = await judgePanel(() => grade(text)); - const mean = judgePanelMean(samples, ['substance']).substance; - return { samples, mean, passed: mean >= AUQ_SUBSTANCE_MIN }; + const { median, mean } = judgePanelSummary(samples, ['substance']); + return { samples, median: median.substance, mean: mean.substance, passed: median.substance >= AUQ_SUBSTANCE_MIN }; } diff --git a/test/helpers/eval-store.ts b/test/helpers/eval-store.ts index bc135ddcf..609c6d099 100644 --- a/test/helpers/eval-store.ts +++ b/test/helpers/eval-store.ts @@ -306,7 +306,7 @@ const detailPart = (text: string): string => { /** * Detail the stored error line cannot carry (it keeps only the matcher header). * A judge record with a `>= N` threshold names each dimension below it, its - * mean, its sample count and the first sample sentence naming that dimension; + * gating value (the panel median since EVAL_POLICY v3, in the `mean` field), its sample count and the first sample sentence naming that dimension; * otherwise a Bun matcher message yields its Expected/Received values. */ export function failureDetailOf(text: string | undefined, diff --git a/test/helpers/free-fixtures-data.ts b/test/helpers/free-fixtures-data.ts index e25e64215..0e45678e6 100644 --- a/test/helpers/free-fixtures-data.ts +++ b/test/helpers/free-fixtures-data.ts @@ -120,6 +120,7 @@ export const FREE_FIXTURES: Readonly> = { 'test/fixtures/impeccable-detect-dom-sample.json': ['test/impeccable-fixtures.test.ts'], 'test/fixtures/impeccable-detect-help.txt': ['test/impeccable-fixtures.test.ts'], 'test/fixtures/judge-calibration/**': ['scripts/judge-calibration.ts', 'test/judge-calibration.test.ts'], + 'test/fixtures/judge-panel-replay-2026-10-05.json': ['test/judge-panel-median.test.ts'], 'test/fixtures/module-size-ratchet.json': ['test/helpers/module-size.ts', 'test/module-size-ratchet.test.ts'], 'test/fixtures/module-size/**': ['test/helpers/module-size.ts', 'test/module-size-ratchet.test.ts'], 'test/fixtures/native-auq-boxed-full-body-sep21.json': ['test/auq-native-capture.test.ts'], diff --git a/test/helpers/llm-judge.ts b/test/helpers/llm-judge.ts index 453a4c1d3..54e3306b4 100644 --- a/test/helpers/llm-judge.ts +++ b/test/helpers/llm-judge.ts @@ -228,8 +228,10 @@ export const JUDGE_PANEL_SAMPLES = 3; /** * Judge panel (EVAL_POLICY.judge): every `judge`-kind entry draws a fixed number of * independent samples of the SAME prompt concurrently, inside its unchanged - * JUDGE_MS budget. Numeric dimensions gate on the per-dimension panel mean - * against the unchanged minimum; boolean fields gate on a strict majority. + * JUDGE_MS budget. Numeric dimensions gate on the per-dimension median of the + * three samples (at least 2 of 3 at or above the unchanged minimum, + * judgePanelMedian); the mean is reported for information only. Boolean + * fields gate on a strict majority. * A sample that errors (refusal, truncation, non-JSON, malformed field) fails * the whole panel and is never resampled. callJudge's 429 backoff happens * before any model output exists, so it is transport, not a verdict retry. @@ -247,7 +249,23 @@ export async function judgePanel(sample: () => Promise): Promise { throw first.reason; } -/** Per-dimension mean over a panel; any non-finite sample value fails the panel. */ +/** + * The judge gate (EVAL_POLICY v3): per-dimension median of exactly + * JUDGE_PANEL_SAMPLES samples, so a dimension passes when at least 2 of 3 + * samples meet its unchanged threshold. Any other sample count or a + * non-finite value fails the panel closed. + */ +export function judgePanelMedian(samples: ReadonlyArray>, keys: readonly K[]): Record { + if (samples.length !== JUDGE_PANEL_SAMPLES) throw new Error(`Judge panel needs exactly ${JUDGE_PANEL_SAMPLES} samples, got ${samples.length}`); + return Object.fromEntries(keys.map(key => { + const values = samples.map(sample => sample && typeof sample === 'object' ? sample[key] : undefined); + const bad = values.findIndex(value => typeof value !== 'number' || !Number.isFinite(value)); + if (bad !== -1) throw new Error(`Judge panel sample ${bad + 1} has non-numeric ${key}: ${JSON.stringify(values[bad])}`); + return [key, [...(values as number[])].sort((a, b) => a - b)[1]!]; + })) as Record; +} + +/** Per-dimension mean over a panel, reported beside the median gate; any non-finite sample value fails the panel. */ export function judgePanelMean(samples: ReadonlyArray>, keys: readonly K[]): Record { if (samples.length === 0) throw new Error('Judge panel has no samples'); return Object.fromEntries(keys.map(key => { @@ -258,6 +276,11 @@ export function judgePanelMean(samples: ReadonlyArray; } +/** The median gate plus the informational mean, for panel reports. */ +export function judgePanelSummary(samples: ReadonlyArray>, keys: readonly K[]): { median: Record; mean: Record } { + return { median: judgePanelMedian(samples, keys), mean: judgePanelMean(samples, keys) }; +} + /** Strict majority of a boolean field; any non-boolean sample value fails the panel. */ export function judgePanelMajority(samples: ReadonlyArray>, key: K): boolean { if (samples.length === 0) throw new Error('Judge panel has no samples'); diff --git a/test/helpers/periodic-exclude-data.ts b/test/helpers/periodic-exclude-data.ts index 4ec0e1a5b..2d7693fbb 100644 --- a/test/helpers/periodic-exclude-data.ts +++ b/test/helpers/periodic-exclude-data.ts @@ -42,7 +42,8 @@ export const CASE_CI_EXCLUDE: Record= k passing trials with * no contract violation. Rule and judge cases run one trial. @@ -59,10 +64,12 @@ export const CASE_CI_EXCLUDE: Record= `exit.rate` over >= * `exit.minTrials`; at most `capFraction` of each tier's * blocking cases; an entry expires after `expiryWeeklyRuns`. - * judge - a judge case draws `samples` independent samples of one - * prompt concurrently; numeric dimensions gate on the panel - * mean against the unchanged threshold, booleans on a strict - * majority; an erroring sample fails the panel, never resampled. + * judge - a judge case draws exactly `samples` (3) independent samples + * of one prompt concurrently; a numeric dimension passes when + * its median (at least 2 of 3 samples) meets the unchanged + * threshold, and the mean is reported only; booleans gate on a + * strict majority; any other sample count or an erroring sample + * fails the panel, never resampled. * drift - one-sided Fisher exact alarm between input-identity series * (Holm-controlled across the cases tested in one report). * infraRedispatch - a census whose every red verdict is machine-classified @@ -70,7 +77,7 @@ export const CASE_CI_EXCLUDE: Record } { if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).join(',') !== 'samples' || !Array.isArray(value.samples) || value.samples.length !== JUDGE_PANEL_SAMPLES || !value.samples.every(sample => validWorkflowJudgeScore(sample, SAMPLE_RANGE, compactReasoning))) return false; - const mean = judgePanelMean(value.samples as JudgeScore[], JUDGE_SCORE_DIMENSIONS); - return JUDGE_SCORE_DIMENSIONS.every(key => mean[key] >= thresholds[key]); + const median = judgePanelMedian(value.samples as JudgeScore[], JUDGE_SCORE_DIMENSIONS); + return JUDGE_SCORE_DIMENSIONS.every(key => median[key] >= thresholds[key]); } export function prepareWorkflowJudgeCache(opts: WorkflowCacheOptions): { @@ -140,9 +140,10 @@ export function prepareWorkflowJudgeCache(opts: WorkflowCacheOptions): { } /** - * Pass floors for the browse reference judge panel mean. The stored baseline - * is recorded for comparison only: a three-sample mean is too noisy for a - * no-dip ratchet, and gating on it silently raised the clarity floor to 4. + * Pass floors for the browse reference judge panel, applied to its gating + * per-dimension median (EVAL_POLICY v3). The stored baseline is recorded for + * comparison only: a three-sample aggregate is too noisy for a no-dip ratchet, + * and gating on it silently raised the clarity floor to 4. */ export const BROWSE_JUDGE_FLOORS = { clarity: 3, completeness: 4, actionability: 4 } as const; diff --git a/test/judge-calibration.test.ts b/test/judge-calibration.test.ts index dd6ae3a97..ac56d7dfb 100644 --- a/test/judge-calibration.test.ts +++ b/test/judge-calibration.test.ts @@ -238,7 +238,8 @@ describe('committed calibration configurations and corpora', () => { const qa = (clarity: number, completeness: number, actionability: number) => ({ clarity, completeness, actionability, reasoning: '' }); for (const id of ['qa-workflow', 'qa-health-rubric', 'workflow-default']) { expect(byId[id]!.verdict([qa(3, 3, 4), qa(3, 3, 4), qa(3, 3, 4)])).toBe('pass'); - expect(byId[id]!.verdict([qa(5, 5, 4), qa(5, 5, 4), qa(5, 5, 3)])).toBe('fail'); + expect(byId[id]!.verdict([qa(5, 5, 4), qa(5, 5, 4), qa(5, 5, 3)])).toBe('pass'); + expect(byId[id]!.verdict([qa(5, 5, 4), qa(5, 5, 3), qa(5, 5, 3)])).toBe('fail'); expect(() => byId[id]!.validate({ ...qa(3, 3, 4), clarity: '3' })).toThrow(); } const arm = (over_engineering: number) => ({ over_engineering, construct: over_engineering ? 'helper' : 'none', reasoning: '' }); @@ -247,10 +248,12 @@ describe('committed calibration configurations and corpora', () => { expect(() => byId.arm!.validate({ over_engineering: 0, construct: 'helper', reasoning: '' })).toThrow(); const browse = (would_browse: boolean, confidence: number) => ({ would_browse, confidence, fallback_behavior: '', reasoning: '' }); expect(byId['qa-anti-refusal']!.verdict([browse(true, 4), browse(true, 4), browse(false, 5)])).toBe('pass'); - expect(byId['qa-anti-refusal']!.verdict([browse(true, 4), browse(true, 3), browse(true, 4)])).toBe('fail'); + expect(byId['qa-anti-refusal']!.verdict([browse(true, 4), browse(true, 3), browse(true, 4)])).toBe('pass'); + expect(byId['qa-anti-refusal']!.verdict([browse(true, 4), browse(true, 3), browse(true, 3)])).toBe('fail'); const voice = (low: number) => ({ directness: 5, concreteness: 5, avoids_corporate: 5, avoids_ai_vocabulary: low, connects_user_outcomes: 5, reasoning: '' }); expect(byId.voice!.verdict([voice(4), voice(4), voice(4)])).toBe('pass'); - expect(byId.voice!.verdict([voice(4), voice(4), voice(3)])).toBe('fail'); + expect(byId.voice!.verdict([voice(4), voice(4), voice(3)])).toBe('pass'); + expect(byId.voice!.verdict([voice(4), voice(3), voice(3)])).toBe('fail'); }); }); diff --git a/test/judge-panel-median.test.ts b/test/judge-panel-median.test.ts new file mode 100644 index 000000000..0c5d4d39d --- /dev/null +++ b/test/judge-panel-median.test.ts @@ -0,0 +1,62 @@ +import { describe, expect, test } from 'bun:test'; +import replay from './fixtures/judge-panel-replay-2026-10-05.json'; +import { judgePanelMajority, judgePanelMean, judgePanelMedian, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; +import { BROWSE_JUDGE_FLOORS } from './helpers/workflow-judge-cache'; +import { EVAL_POLICY } from './helpers/periodic-exclude-data'; + +const gate = (scores: number[], threshold = 4) => judgePanelMedian(scores.map(score => ({ score })), ['score']).score >= threshold; + +describe('judge gate: per-dimension median of exactly three samples (EVAL_POLICY v3)', () => { + test('a dimension passes when at least 2 of 3 samples meet the unchanged threshold', () => { + expect(EVAL_POLICY.judge.samples).toBe(JUDGE_PANEL_SAMPLES); + expect(gate([4, 4, 3])).toBe(true); + expect(gate([4, 3, 3])).toBe(false); + expect(gate([5, 5, 3])).toBe(true); + expect(gate([5, 3, 3])).toBe(false); + expect(gate([3, 4, 4])).toBe(true); + }); + + test('anything but exactly three valid samples fails closed', () => { + expect(() => gate([4, 4])).toThrow('exactly 3 samples'); + expect(() => gate([4, 4, 4, 4])).toThrow('exactly 3 samples'); + expect(() => judgePanelMedian([{ score: 4 }, { score: '4' }, { score: 4 }] as any, ['score'])).toThrow('sample 2 has non-numeric score'); + expect(() => judgePanelMedian([{ score: 4 }, null, { score: 4 }] as any, ['score'])).toThrow('sample 2'); + expect(() => judgePanelMedian([{ score: 4 }, { score: Number.NaN }, { score: 4 }], ['score'])).toThrow('non-numeric'); + }); + + test('each dimension is gated on its own: one failing dimension fails the panel', () => { + const samples = [{ clarity: 5, actionability: 3 }, { clarity: 5, actionability: 3 }, { clarity: 5, actionability: 4 }]; + const median = judgePanelMedian(samples, ['clarity', 'actionability']); + expect(median).toEqual({ clarity: 5, actionability: 3 }); + expect(median.clarity >= 4 && median.actionability >= 4).toBe(false); + }); +}); + +type Sample = Record; +const SCORE = ['clarity', 'completeness', 'actionability'] as const; +function verdict(name: string, samples: Sample[], aggregate: typeof judgePanelMean): boolean { + const meets = (keys: readonly string[], floors: Record) => { + const scores = aggregate(samples as Record[], keys); + return keys.every(key => scores[key]! >= floors[key]!); + }; + if (name === 'Setup block') return meets(['clarity', 'actionability'], { clarity: 3, actionability: 3 }); + if (name === 'Browse SKILL.md') return meets(SCORE, BROWSE_JUDGE_FLOORS); + if (name === 'QA anti-refusal') return judgePanelMajority(samples as Record[], 'would_browse') && meets(['confidence'], { confidence: 4 }); + if (name === 'Cross-skill consistency') return judgePanelMajority(samples as Record[], 'consistent') && meets(['score'], { score: 4 }); + if (name === 'Voice directive') { + const keys = ['directness', 'concreteness', 'avoids_corporate', 'avoids_ai_vocabulary', 'connects_user_outcomes']; + return meets(keys, Object.fromEntries(keys.map(key => [key, 4]))); + } + return meets(SCORE, { clarity: 3, completeness: 3, actionability: 4 }); +} + +test('replayed panels from jobs 112032744416 and 112049856586 (replayed evidence, not a CI pass)', () => { + const rows = replay.panels.map(panel => ({ job: panel.job, case: panel.case, + mean: verdict(panel.case, panel.samples as Sample[], judgePanelMean), median: verdict(panel.case, panel.samples as Sample[], judgePanelMedian) })); + expect(rows).toHaveLength(42); + expect(rows.filter(row => !row.mean).map(row => `${row.job} ${row.case}`)).toEqual([ + '112032744416 QA workflow', '112032744416 plan-eng-review/SKILL.md sections', '112049856586 design-consultation/SKILL.md research']); + expect(rows.filter(row => !row.median).map(row => `${row.job} ${row.case}`)).toEqual(['112032744416 plan-eng-review/SKILL.md sections']); + // Exactly these two panels change verdict: each had 2 of 3 samples at every threshold. + expect(rows.filter(row => row.median && !row.mean).map(row => row.case)).toEqual(['QA workflow', 'design-consultation/SKILL.md research']); +}); diff --git a/test/llm-judge-recommendation.test.ts b/test/llm-judge-recommendation.test.ts index 3ccffa981..eeff0e3e7 100644 --- a/test/llm-judge-recommendation.test.ts +++ b/test/llm-judge-recommendation.test.ts @@ -7,7 +7,7 @@ * against the same threshold the production E2E tests use (>= 4). * * Each fixture is a pre-registered 3-sample judge panel: numeric substance - * gates on the panel mean, the boolean checks on a 2-of-3 majority, and an + * gates on the panel median (2 of 3 samples), the boolean checks on a 2-of-3 majority, and an * erroring sample fails the panel (never resampled). Costs ~$0.12 per run. * Touchfile-gated to test/helpers/llm-judge.ts so it fires on rubric * tweaks but not every test run. Runs only under EVALS=1 with an API key. @@ -15,7 +15,7 @@ import { expect } from 'bun:test'; import { CAPTURE_MS } from './helpers/eval-budgets'; -import { judgePanel, judgePanelMajority, judgePanelMean, judgePanelReasoning, judgeRecommendation } from './helpers/llm-judge'; +import { judgePanel, judgePanelMajority, judgePanelMedian, judgePanelReasoning, judgeRecommendation } from './helpers/llm-judge'; import { describeIfSelected, testIfSelected } from './helpers/e2e-helpers'; // Fixtures wrap a realistic AskUserQuestion shape so the judge sees the menu @@ -45,7 +45,7 @@ async function judgeRecommendationPanel(text: string) { present: judgePanelMajority(samples, 'present'), commits: judgePanelMajority(samples, 'commits'), has_because: judgePanelMajority(samples, 'has_because'), - reason_substance: judgePanelMean(samples, ['reason_substance']).reason_substance, + reason_substance: judgePanelMedian(samples, ['reason_substance']).reason_substance, reasoning: judgePanelReasoning(samples), }; } diff --git a/test/paid-shards.test.ts b/test/paid-shards.test.ts index 2fa58983a..78d3b00d9 100644 --- a/test/paid-shards.test.ts +++ b/test/paid-shards.test.ts @@ -759,7 +759,7 @@ console.log("Ran 1 tests across 1 files. [1ms]"); process.exit(${fail ? 1 : 0}); expect(sliceExitCode(summary.outcomes)).toBe(0); const env = JSON.parse(fs.readFileSync(path.join(evalDirBase, 'shards', shardSlug([key(3)]), 'env.json'), 'utf8')); expect(env).toMatchObject({ GSTACK_EVAL_CASE_ID: 'review-sql-injection', GSTACK_EVAL_KIND: 'behavior', GSTACK_EVAL_TRIAL: '3', - GSTACK_EVAL_PANEL_N: '3', GSTACK_EVAL_PANEL_K: '2', GSTACK_EVAL_POLICY_VERSION: '2' }); + GSTACK_EVAL_PANEL_N: '3', GSTACK_EVAL_PANEL_K: '2', GSTACK_EVAL_POLICY_VERSION: '3' }); expect(JSON.parse(env.EVALS_SELECTION_JSON).selected).toEqual(['review-sql-injection']); } finally { fs.rmSync(evalDirBase, { recursive: true, force: true }); } }); diff --git a/test/periodic-exclude-policy.test.ts b/test/periodic-exclude-policy.test.ts index bfc9ba7ad..a7f3a041d 100644 --- a/test/periodic-exclude-policy.test.ts +++ b/test/periodic-exclude-policy.test.ts @@ -77,8 +77,9 @@ describe('periodic exclude policy', () => { describe('eval verdict policy (pre-registered)', () => { test('EVAL_POLICY carries exactly the approved constants; a change needs re-approval and a version bump', () => { // v2: approved 2026-10-04 (D1, docs/test-audit-2026-10.md); only identity and history changed. + // v3: approved 2026-10-06; the judge gate is the per-dimension median of 3 samples. expect(EVAL_POLICY).toEqual({ - version: 2, + version: 3, panel: { n: 3, k: 2 }, quarantine: { entry: { rate: 0.95, minTrials: 10 }, exit: { rate: 0.97, minTrials: 10 }, capFraction: 0.10, expiryWeeklyRuns: 8 }, judge: { samples: 3 }, diff --git a/test/skill-e2e-auq-matrix.test.ts b/test/skill-e2e-auq-matrix.test.ts index 1f2205ab9..ac9bd2ff2 100644 --- a/test/skill-e2e-auq-matrix.test.ts +++ b/test/skill-e2e-auq-matrix.test.ts @@ -141,12 +141,14 @@ describeE2E('AUQ behavioral matrix (periodic)', () => { const text = capture.text; const fmt = scoreAuqFormat(text); let substance = 0; + let substanceMean = 0; let samples: number[] = []; let recPresent = false; let hadBecause = false; if (text.trim()) { const panel = await auqSubstancePanel(text); - substance = panel.mean; + substance = panel.median; + substanceMean = panel.mean; samples = panel.samples.map(g => g.substance); recPresent = panel.samples[0].present; hadBecause = panel.samples[0].hadLiteralBecause; @@ -154,7 +156,7 @@ describeE2E('AUQ behavioral matrix (periodic)', () => { // eslint-disable-next-line no-console console.log( `[AUQ-matrix ${m.skill}] captured=${text.length}B format=${fmt.present}/${fmt.total} ` + - `missing=[${fmt.missing.join(',')}] recPresent=${recPresent} substance=${substance.toFixed(2)} [${samples.join(',')}] ` + + `missing=[${fmt.missing.join(',')}] recPresent=${recPresent} substance median=${substance} mean=${substanceMean.toFixed(2)} [${samples.join(',')}] ` + `literalBecause=${hadBecause}`, ); @@ -165,7 +167,7 @@ describeE2E('AUQ behavioral matrix (periodic)', () => { // Presentation elements (ELI10, Pros / cons, ✅/❌, Net:) are logged above, not failed. problems.push(...auqMachineFormatProblems(capture.question)); if (substance < AUQ_SUBSTANCE_MIN) { - problems.push(`recommendation substance mean ${substance.toFixed(2)} < ${AUQ_SUBSTANCE_MIN} over samples [${samples.join(',')}] (boilerplate/weak)`); + problems.push(`recommendation substance median ${substance} < ${AUQ_SUBSTANCE_MIN} over samples [${samples.join(',')}] (boilerplate/weak)`); } if (problems.length > 0) { throw new Error( diff --git a/test/skill-llm-eval.test.ts b/test/skill-llm-eval.test.ts index 668c59a12..5ed647dc2 100644 --- a/test/skill-llm-eval.test.ts +++ b/test/skill-llm-eval.test.ts @@ -14,7 +14,7 @@ import { afterAll, expect } from 'bun:test'; import { JUDGE_MS } from './helpers/eval-budgets'; import * as fs from 'fs'; import * as path from 'path'; -import { callJudge, judge, JudgeRefusalError, JUDGE_SCORE_SCHEMA, QA_ANTI_REFUSAL_JUDGE_SCHEMA, VOICE_DIRECTIVE_JUDGE_SCHEMA, CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA, buildQaWorkflowJudgePrompt, buildQaHealthRubricJudgePrompt, buildQaAntiRefusalJudgePrompt, buildCrossSkillConsistencyJudgePrompt, buildVoiceDirectiveJudgePrompt, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMean, judgePanelMajority, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS } from './helpers/llm-judge'; +import { callJudge, judge, JudgeRefusalError, JUDGE_SCORE_SCHEMA, QA_ANTI_REFUSAL_JUDGE_SCHEMA, VOICE_DIRECTIVE_JUDGE_SCHEMA, CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA, buildQaWorkflowJudgePrompt, buildQaHealthRubricJudgePrompt, buildQaAntiRefusalJudgePrompt, buildCrossSkillConsistencyJudgePrompt, buildVoiceDirectiveJudgePrompt, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMean, judgePanelMedian, judgePanelSummary, judgePanelMajority, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS } from './helpers/llm-judge'; import { ASK_QUESTIONS_HEADING, ENG_REVIEW_EXCERPT } from './helpers/workflow-excerpt'; import type { JudgeScore } from './helpers/llm-judge'; import { readWorkflowJudgeInput, buildWorkflowJudgePrompt, QA_DISCOVERY_REFERENCES, WORKFLOW_JUDGE_RESPONSE_SCHEMA, type WorkflowJudgeInput } from './helpers/workflow-judge-input'; @@ -101,8 +101,8 @@ describeIfSelected('LLM-as-judge quality evals', [ const section = sliceBrowseSection('## Snapshot Flags'); const samples = await judgePanel(() => judge('browse skill reference (flags + commands)', section)); - const scores = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS); - console.log('Browse SKILL.md panel:', JSON.stringify({ mean: scores, samples }, null, 2)); + const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); + console.log('Browse SKILL.md panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); const baselinesPath = path.join(ROOT, 'test', 'fixtures', 'eval-baselines.json'); const baselines = JSON.parse(fs.readFileSync(baselinesPath, 'utf-8')); @@ -145,8 +145,8 @@ describeIfSelected('LLM-as-judge quality evals', [ const section = content.slice(setupStart, setupEnd); const samples = await judgePanel(() => judge('setup/binary discovery instructions', section)); - const scores = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS); - console.log('Setup block panel:', JSON.stringify({ mean: scores, samples }, null, 2)); + const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); + console.log('Setup block panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); evalCollector?.addTest({ name: 'setup block', @@ -206,8 +206,8 @@ describeIfSelected('QA skill quality evals', ['qa/SKILL.md workflow', 'qa/SKILL. references: ['qa/templates/functional-report-template.md', 'qa/templates/qa-report-template.md', 'qa/references/issue-taxonomy.md'] }).text; const samples = await judgePanel(() => callJudge(buildQaWorkflowJudgePrompt(section))); - const scores = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS); - console.log('QA workflow panel:', JSON.stringify({ mean: scores, samples }, null, 2)); + const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); + console.log('QA workflow panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); evalCollector?.addTest({ name: 'qa/SKILL.md workflow', @@ -232,8 +232,8 @@ describeIfSelected('QA skill quality evals', ['qa/SKILL.md workflow', 'qa/SKILL. const section = sliceQaPatterns('## Health Score Rubric'); const samples = await judgePanel(() => callJudge(buildQaHealthRubricJudgePrompt(section), undefined, { jsonSchema: JUDGE_SCORE_SCHEMA })); - const scores = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS); - console.log('QA health rubric panel:', JSON.stringify({ mean: scores, samples }, null, 2)); + const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); + console.log('QA health rubric panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); evalCollector?.addTest({ name: 'qa/SKILL.md health rubric', @@ -263,9 +263,9 @@ describeIfSelected('QA skill quality evals', ['qa/SKILL.md workflow', 'qa/SKILL. const rulesSection = sliceQaPatterns('## Important Rules'); const samples = await judgePanel(() => callJudge<{ would_browse: boolean; fallback_behavior: string; confidence: number; reasoning: string }>(buildQaAntiRefusalJudgePrompt(diffAwareSection, rulesSection), undefined, { jsonSchema: QA_ANTI_REFUSAL_JUDGE_SCHEMA })); - const result = { would_browse: judgePanelMajority(samples, 'would_browse'), ...judgePanelMean(samples, ['confidence'] as const) }; + const result = { would_browse: judgePanelMajority(samples, 'would_browse'), ...judgePanelMedian(samples, ['confidence'] as const) }; - console.log('QA anti-refusal panel:', JSON.stringify({ result, samples }, null, 2)); + console.log('QA anti-refusal panel:', JSON.stringify({ gate: 'median', result, mean: judgePanelMean(samples, ['confidence'] as const), samples }, null, 2)); evalCollector?.addTest({ name: 'qa/SKILL.md anti-refusal', @@ -308,9 +308,9 @@ describeIfSelected('Cross-skill consistency evals', ['cross-skill greptile consi ].join('\n\n'); const samples = await judgePanel(() => callJudge<{ consistent: boolean; issues: string[]; score: number; reasoning: string }>(buildCrossSkillConsistencyJudgePrompt(collected), undefined, { jsonSchema: CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA })); - const result = { consistent: judgePanelMajority(samples, 'consistent'), ...judgePanelMean(samples, ['score'] as const) }; + const result = { consistent: judgePanelMajority(samples, 'consistent'), ...judgePanelMedian(samples, ['score'] as const) }; - console.log('Cross-skill consistency panel:', JSON.stringify({ result, samples }, null, 2)); + console.log('Cross-skill consistency panel:', JSON.stringify({ gate: 'median', result, mean: judgePanelMean(samples, ['score'] as const), samples }, null, 2)); evalCollector?.addTest({ name: 'cross-skill greptile consistency', @@ -464,8 +464,8 @@ async function runWorkflowJudge(opts: { if (opts.compactReasoning && !samples.every(sample => validWorkflowJudgeScore(sample as unknown as EvalCacheValue, { clarity: 1, completeness: 1, actionability: 1 }, true))) { throw new Error('Workflow judge violated the compact response contract'); } - scores = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS); - console.log(`${opts.testName} panel:`, JSON.stringify({ mean: scores, samples }, null, 2)); + scores = judgePanelMedian(samples, JUDGE_SCORE_DIMENSIONS); + console.log(`${opts.testName} panel:`, JSON.stringify({ gate: 'median', median: scores, mean: judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS), samples }, null, 2)); expect(scores.clarity).toBeGreaterThanOrEqual(thresholds.clarity); expect(scores.completeness).toBeGreaterThanOrEqual(thresholds.completeness); expect(scores.actionability).toBeGreaterThanOrEqual(thresholds.actionability); @@ -752,9 +752,9 @@ describeIfSelected('Voice directive eval', ['voice directive tone'], () => { connects_user_outcomes: number; reasoning: string; }>(buildVoiceDirectiveJudgePrompt(voiceSection), undefined, { jsonSchema: VOICE_DIRECTIVE_JUDGE_SCHEMA })); - const result = judgePanelMean(samples, ['directness', 'concreteness', 'avoids_corporate', 'avoids_ai_vocabulary', 'connects_user_outcomes'] as const); + const { median: result, mean } = judgePanelSummary(samples, ['directness', 'concreteness', 'avoids_corporate', 'avoids_ai_vocabulary', 'connects_user_outcomes'] as const); - console.log('Voice directive panel:', JSON.stringify({ mean: result, samples }, null, 2)); + console.log('Voice directive panel:', JSON.stringify({ gate: 'median', median: result, mean, samples }, null, 2)); evalCollector?.addTest({ name: 'voice directive tone', diff --git a/test/workflow-judge-cache.test.ts b/test/workflow-judge-cache.test.ts index a11f1cc70..f71a95421 100644 --- a/test/workflow-judge-cache.test.ts +++ b/test/workflow-judge-cache.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, spyOn, test } from 'bun:test'; import { Messages } from '@anthropic-ai/sdk/resources/messages'; -import { callJudge, JudgeRefusalError, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMajority, judgePanelMean, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; +import { callJudge, JudgeRefusalError, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMajority, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; import { EVAL_POLICY } from './helpers/periodic-exclude-data'; import { getCookieWorkflowManualReview } from './helpers/cookie-workflow-manual-review'; import { resolveEvalModel } from '../lib/eval-model'; @@ -172,7 +172,7 @@ test('workflow registration preserves model work and reserves only terminal-reco const body = source.split('async function runWorkflowJudge')[1]!.split('// Block 1:')[0]!; const stages = ['workflowJudgeAttempts.set', 'readWorkflowJudgeInput(', 'cache.lookup()', 'judgePanel(() => callJudge(prompt, opts.model, { signal: controller.signal, max_tokens: maxTokens,', - 'scores = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS);', + 'scores = judgePanelMedian(samples, JUDGE_SCORE_DIMENSIONS);', 'expect(scores.clarity)', 'expect(scores.completeness)', 'expect(scores.actionability)', 'cache.publish(samples, active)'] .map(stage => body.indexOf(stage)); expect(stages.every(position => position >= 0)).toBe(true); @@ -207,7 +207,7 @@ function actualCallback(f: ReturnType, overrides: { 'evalCollector', 'expect', 'console', 'performance', 'JUDGE_MS', 'WORKFLOW_JUDGE_RECORD_MS', 'setTimeout', 'clearTimeout', 'JudgeRefusalError', 'getCookieWorkflowManualReview', 'DEFAULT_JUDGE_MAX_TOKENS', 'resolveEvalModel', 'WORKFLOW_JUDGE_RESPONSE_SCHEMA', 'validWorkflowJudgeScore', - 'judgePanel', 'judgePanelMean', 'judgePanelReasoning', 'JUDGE_SCORE_DIMENSIONS', + 'judgePanel', 'judgePanelMean', 'judgePanelMedian', 'judgePanelReasoning', 'JUDGE_SCORE_DIMENSIONS', `${javascript}\nreturn runWorkflowJudge;`)( f.root, overrides.read ?? readWorkflowJudgeInput, buildWorkflowJudgePrompt, (options: WorkflowCacheOptions) => (overrides.prepare ?? prepareWorkflowJudgeCache)({ ...options, env: f.env }), @@ -219,7 +219,7 @@ function actualCallback(f: ReturnType, overrides: { overrides.setTimer ?? setTimeout, overrides.clearTimer ?? clearTimeout, JudgeRefusalError, getCookieWorkflowManualReview, DEFAULT_JUDGE_MAX_TOKENS, resolveEvalModel, WORKFLOW_JUDGE_RESPONSE_SCHEMA, validWorkflowJudgeScore, - judgePanel, judgePanelMean, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS); + judgePanel, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS); return { run, records, signals, prompts, attempts, options: { ...f.opts, suite: 'Cache regression' } }; } @@ -572,14 +572,14 @@ describe('judge panel', () => { expect(String(error)).toContain(`sample 1 of ${SAMPLES} failed beside scored samples`); }); - test('numeric dimensions gate on the per-dimension mean; one low sample can be outvoted, a low mean cannot', () => { - const outvoted = judgePanelMean([panelScore(2), panelScore(4), panelScore(4)], JUDGE_SCORE_DIMENSIONS); - expect(outvoted.clarity).toBeCloseTo(10 / 3); - expect(outvoted.clarity).toBeGreaterThanOrEqual(panelThresholds.clarity); - const low = judgePanelMean([panelScore(2), panelScore(2), panelScore(4)], JUDGE_SCORE_DIMENSIONS); + test('numeric dimensions gate on the per-dimension median: 2 of 3 samples decide, the mean is only reported', () => { + const outvoted = judgePanelMedian([panelScore(2), panelScore(4), panelScore(4)], JUDGE_SCORE_DIMENSIONS); + expect(outvoted.clarity).toBe(4); + expect(judgePanelMean([panelScore(2), panelScore(4), panelScore(4)], JUDGE_SCORE_DIMENSIONS).clarity).toBeCloseTo(10 / 3); + const low = judgePanelMedian([panelScore(2), panelScore(2), panelScore(4)], JUDGE_SCORE_DIMENSIONS); expect(low.clarity).toBeLessThan(panelThresholds.clarity); - // No compensation across dimensions: each is averaged on its own. - expect(judgePanelMean([panelScore(5, 1), panelScore(5, 1), panelScore(5, 1)], JUDGE_SCORE_DIMENSIONS).completeness).toBe(1); + // No compensation across dimensions: each is gated on its own. + expect(judgePanelMedian([panelScore(5, 1), panelScore(5, 1), panelScore(5, 1)], JUDGE_SCORE_DIMENSIONS).completeness).toBe(1); }); test('malformed sample fields fail the panel instead of averaging to NaN', () => { @@ -621,7 +621,7 @@ describe('judge panel', () => { }); }); -// Stored browse reference panel means against the judge floors. +// Stored browse reference panel means (recorded before EVAL_POLICY v3) against the judge floors. { type Scores = { clarity: number; completeness: number; actionability: number }; const stored = JSON.parse(fs.readFileSync(path.join(import.meta.dir, 'fixtures/browse-judge/panel-means.json'), 'utf8')) as