mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-08 20:31:45 +02:00
feat(evals): trial planner, slice exit split and panel-verdict report
Planner: behavior and quarantined cases become panels of isolated trial shards (<file>#<id>~t<N>) bound by EVALS_SELECTION_JSON=[id] and the exact test name; the file shard excludes them by name. Trials of one case never share a slice, result slugs are unique, panels are validated whole, unknown registrations throw, and the planner prints a capacity preflight. Executor: each trial shard gets its TRIAL_ENV identity and a trial record (outcome, failure class, cause, cost); every shard writes a JUnit report. The slice exit now means execution completeness: a failed rule shard or a trial without a record reds the runner, a failed trial does not. Report: panelVerdict() decides every panel of the first run attempt (later attempts are reported, never replacing it); rule shards keep the unchanged fail-closed checks; collector records all count (no last-attempt wins); census runs enforce the quarantine cap and expiry. It writes collector-outcomes v2, trial-outcomes.jsonl (trials plus JUnit rule/judge cases), report-summary.md, and one headline + failure block with rerun commands, and flags INFRA/INCOMPLETE-only reds for the one re-dispatch. The fail-open suite gains the panel cases: behavior 1/3 red, 2/3 green with its failed trial shown, missing trial INCOMPLETE, contract at 2/3 red, quarantined 1/3 green, 0/3 and contract red, missing slice red, and a later attempt never replacing the first.
This commit is contained in:
1 parent
b1f5bc0032
commit
8eb55b6ebf
6 files changed
+1225
-178
No files matched your search
@@ -241,7 +241,7 @@ describe('PR profile paid-runner integration', () => {
|
||||
expect(guarded[0].status).toBe('passed-empty');
|
||||
});
|
||||
|
||||
test('report distinguishes retained/deferred coverage and final executed/reused outcomes from attempts', () => {
|
||||
test('report distinguishes retained/deferred coverage and counts every executed/reused record', () => {
|
||||
const manifest = ceoManifest();
|
||||
const lines = formatProfileCoverage(manifest).join('\n');
|
||||
expect(lines).toContain('profile=pr mode=pr');
|
||||
@@ -252,6 +252,7 @@ describe('PR profile paid-runner integration', () => {
|
||||
{ name: 'retry', suite: 'judge', passed: true, execution: 'executed' },
|
||||
{ name: 'cached', suite: 'judge', passed: true, execution: 'reused' },
|
||||
{ name: 'failed', suite: 'native', passed: false },
|
||||
] }])).toEqual({ executed: 2, reused: 1, passed: 2, failed: 1, manual_accepted: 0, attempts: 4 });
|
||||
// Paid evals never retry: a later pass never replaces an earlier failed record.
|
||||
] }])).toEqual({ executed: 3, reused: 1, passed: 2, failed: 2, manual_accepted: 0, attempts: 4 });
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user