garrytan
d6b3559b78
feat(evals): per-case pass rates with Wilson intervals, identity series and quarantine policy
...
scripts/eval-flake-rank.ts becomes eval:pass-rates (eval:flake-rank stays an
alias, and the legacy aggregate stays exported). It reads eval-store's
trial-outcomes JSONL from the last N completed evals-periodic runs on this
branch and main (gh, downloading only the trial-outcomes artifact, cached and
size-capped, parsed as data), plus local eval dirs, and prints per-case
per-trial pass rates with 95% Wilson intervals.
A series is a case's own touchfiles minus GLOBAL_TOUCHFILES
(caseSeriesIdentities, for the report job to stamp), per model, CLI version
and policy version. Labels: INCONCLUSIVE, BROKEN, FLAKY, FAILING, PASSING.
--backfill imports legacy slice artifacts as pre-policy trials (first
attempt only, attributed by registry id, never guessed) for display only.
--gate fails with ACTION REQUIRED on post-policy evidence only: drift below
the quarantine entry rule, a rule case behaving like behavior, a one-sided
Fisher drop against the previous identity (Holm-controlled), and quarantine
entries that met their exit rule, expired after 8 weekly runs, broke the
10% tier cap or are invalid. CASE_QUARANTINE entries now carry a
failureClass (detector, harness or model-latency); a product defect has no
class and is never quarantined. The policy test pins EVAL_POLICY's approved
constants.
2026-09-29 19:11:27 +00:00
..
2026-09-29 06:08:49 +00:00
2026-09-25 13:27:07 -04:00
2026-08-15 22:20:30 -07:00
2026-08-29 10:10:35 -07:00
2026-09-06 08:54:25 -07:00
2026-09-24 01:53:58 -04:00
2026-09-25 13:27:07 -04:00
2026-09-29 05:09:36 +00:00
2026-09-22 14:57:52 -04:00
2026-09-29 05:09:36 +00:00
2026-09-29 06:08:49 +00:00
2026-09-22 14:57:52 -04:00
2026-09-29 14:28:11 +00:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-14 14:32:45 -07:00
2026-09-05 14:45:28 -07:00
2026-09-09 08:57:21 -07:00
2026-04-19 17:50:31 +08:00
2026-09-29 06:07:35 -07:00
2026-08-29 09:06:54 -07:00
2026-05-26 16:50:03 -07:00
2026-08-27 09:50:31 -07:00
2026-05-26 16:50:03 -07:00
2026-08-31 08:55:30 -07:00
2026-06-07 19:13:24 -07:00
2026-09-29 13:48:02 +00:00
2026-09-22 14:57:52 -04:00
2026-09-29 06:08:49 +00:00
2026-09-29 05:09:36 +00:00
2026-09-29 16:57:54 +00:00
2026-09-29 17:10:33 +00:00
2026-09-29 17:05:15 +00:00
2026-09-29 15:18:46 +00:00
2026-09-22 14:57:52 -04:00
2026-09-29 17:07:38 +00:00
2026-09-29 17:07:38 +00:00
2026-09-26 18:57:53 -04:00
2026-09-22 14:57:52 -04:00
2026-09-26 18:57:53 -04:00
2026-09-26 18:57:53 -04:00
2026-09-25 12:06:45 -04:00
2026-09-24 01:53:58 -04:00
2026-09-22 14:57:52 -04:00
2026-09-29 05:09:36 +00:00
2026-09-25 12:06:45 -04:00
2026-09-14 15:14:58 -07:00
2026-09-14 15:14:58 -07:00
2026-09-29 17:02:31 +00:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-14 14:32:45 -07:00
2026-09-29 15:21:17 +00:00
2026-09-29 15:21:17 +00:00
2026-09-29 15:21:17 +00:00
2026-08-24 08:55:58 -07:00
2026-09-14 14:32:45 -07:00
2026-09-29 06:08:49 +00:00
2026-09-29 16:47:24 +00:00
2026-09-29 18:56:27 +00:00
2026-09-29 18:56:27 +00:00
2026-09-14 14:32:45 -07:00
2026-09-08 22:40:22 -07:00
2026-09-22 14:57:52 -04:00
2026-08-28 11:43:13 -07:00
2026-09-15 21:35:11 -07:00
2026-09-29 14:22:05 +00:00
2026-09-29 14:22:05 +00:00
2026-09-22 14:57:52 -04:00
2026-09-29 19:11:04 +00:00
2026-09-28 13:09:52 -07:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-29 15:29:46 +00:00
2026-09-14 14:32:45 -07:00
2026-09-29 06:07:35 -07:00
2026-09-14 14:32:45 -07:00
2026-09-22 14:57:52 -04:00
2026-09-29 06:08:49 +00:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-29 06:08:49 +00:00
2026-09-14 15:14:58 -07:00
2026-09-29 19:11:27 +00:00
2026-09-29 15:04:28 +00:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-14 14:32:45 -07:00
2026-09-24 01:53:58 -04:00
2026-09-22 14:57:52 -04:00
2026-09-26 18:57:53 -04:00
2026-09-14 14:32:45 -07:00
2026-09-22 14:57:52 -04:00
2026-09-29 05:09:36 +00:00
2026-09-29 05:09:36 +00:00
2026-09-22 14:57:52 -04:00
2026-09-14 14:32:45 -07:00
2026-09-29 06:07:35 -07:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-29 06:07:35 -07:00
2026-09-14 14:32:45 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 06:07:35 -07:00
2026-09-29 09:16:19 +00:00
2026-09-29 09:16:19 +00:00
2026-08-29 09:06:54 -07:00
2026-08-29 09:06:54 -07:00
2026-09-29 13:48:02 +00:00
2026-04-24 01:38:21 -07:00
2026-09-29 06:07:35 -07:00
2026-09-23 08:54:53 -04:00
2026-09-29 06:07:35 -07:00
2026-09-28 16:00:12 -07:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-29 14:28:11 +00:00
2026-09-29 06:07:35 -07:00
2026-09-24 01:53:58 -04:00
2026-09-24 01:53:58 -04:00
2026-09-29 06:07:35 -07:00
2026-09-26 18:57:53 -04:00
2026-09-29 06:07:35 -07:00
2026-08-14 09:28:56 -07:00
2026-09-29 06:07:35 -07:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-22 14:57:52 -04:00
2026-09-29 16:19:39 +00:00
2026-09-26 18:57:53 -04:00
2026-09-29 19:11:04 +00:00
2026-09-25 13:27:07 -04:00
2026-09-06 08:54:25 -07:00
2026-09-29 09:16:19 +00:00
2026-09-29 19:11:04 +00:00
2026-08-15 22:20:30 -07:00
2026-09-26 18:57:53 -04:00
2026-09-22 14:57:52 -04:00
2026-09-29 19:11:04 +00:00
2026-09-29 06:07:35 -07:00