mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-04 10:26:52 +02:00
test(evals): add E2E_KINDS, BEHAVIOR_WHY, EVAL_POLICY and CASE_QUARANTINE skeletons
Every E2E_TIERS and LLM_JUDGE_TOUCHFILES key starts as 'rule'; BEHAVIOR_WHY and CASE_QUARANTINE start empty. EVAL_POLICY pre-registers the approved panel (3, majority 2), quarantine entry 0.95/10 and exit 0.97/10, 10% cap, 8-weekly-run expiry, Fisher drift alarm and one INFRA re-dispatch.
This commit is contained in:
1 parent
4c5fc0e2ce
commit
3c441fe311
2 files changed
+312
No files matched your search
@@ -59,3 +59,57 @@ export const CASE_CI_EXCLUDE: Record<string, { reason: string; tracking: string
|
||||
tracking: 'TODOS.md "CI-unrunnable paid evals" (re-entry: the CLI/device is available in the CI image; review by 2026-12-28)',
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Paid-eval verdict policy, pre-registered (approved 2026-09-29). Frozen before
|
||||
* the census: any change after seeing census results needs Garry's
|
||||
* re-approval and a fresh census, and bumps `version` (every trial record
|
||||
* carries it as policy_version, so pass-rate history segments at the change).
|
||||
* panel - behavior cases and quarantined cases run n independent
|
||||
* trials; a behavior panel PASSES at >= k passing trials with
|
||||
* no contract violation. Rule and judge cases run one trial.
|
||||
* quarantine - entry below `entry.rate` per trial over >= `entry.minTrials`
|
||||
* new-policy trials; exit at >= `exit.rate` over >=
|
||||
* `exit.minTrials`; at most `capFraction` of each tier's
|
||||
* blocking cases; an entry expires after `expiryWeeklyRuns`.
|
||||
* drift - one-sided Fisher exact alarm between input-identity series.
|
||||
* infraRedispatch - a census whose every red verdict is machine-classified
|
||||
* INFRA or INCOMPLETE may be re-dispatched this many times as
|
||||
* a new run; both runs are reported.
|
||||
*/
|
||||
export const EVAL_POLICY = {
|
||||
version: 1,
|
||||
panel: { n: 3, k: 2 },
|
||||
quarantine: {
|
||||
entry: { rate: 0.95, minTrials: 10 },
|
||||
exit: { rate: 0.97, minTrials: 10 },
|
||||
capFraction: 0.10,
|
||||
expiryWeeklyRuns: 8,
|
||||
},
|
||||
drift: { fisherAlpha: 0.05, fisherMinPerSide: 6 },
|
||||
infraRedispatch: 1,
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* Quarantined paid cases, keyed by registry id (an E2E_TIERS key). A
|
||||
* quarantined case still runs its full panel and reports in every lane, but
|
||||
* its failed verdict cannot fail the lane unless the panel is a hard break
|
||||
* (0 of n) or a trial violated a contract; it never counts as passing
|
||||
* coverage. An entry needs the entry rule met on the current input identity,
|
||||
* a written diagnosis that the failures are detector, harness or model-latency
|
||||
* failures (a product defect is never quarantined), and unchanged case
|
||||
* touchfiles in the change that adds it. Pinned by
|
||||
* test/periodic-exclude-policy.test.ts.
|
||||
* reason - the written diagnosis
|
||||
* tracking - issue or TODOS pointer
|
||||
* owner - who removes it
|
||||
* enteredAt - ISO date the entry landed (expiry counts weekly runs from here)
|
||||
* exit - the measurable exit condition
|
||||
*/
|
||||
export const CASE_QUARANTINE: Record<string, {
|
||||
reason: string;
|
||||
tracking: string;
|
||||
owner: string;
|
||||
enteredAt: string;
|
||||
exit: string;
|
||||
}> = {};
|
||||
Reference in new issue
Block a user