feat(cso-eval): daily precision counts confirmed incidental issues and leaves out declared fixture instrumentation

This commit is contained in:
garrytan committed 2026-10-17 16:57:02 +00:00
1 parent d804fbb2fc
commit d561177ef4
3 files changed
+46 -5

No files matched your search

+12
View File
@@ -294,6 +294,18 @@ requalified catalog or withdraws the catalog's runtimes in the same pull
request. Independently, the helper refuses to select a runtime recorded under
another isolation policy.
The daily precision gate (`dailyPrecision95` in
[`cso-eval.ts`](../scripts/cso-eval.ts)) divides true positives by reported
findings. Hypotheses count in neither term. A finding the evaluator marks
`instrumentation` also counts in neither term: it targets the evaluator's
declared fixture instrumentation, such as the canary overlay files or the egress
probe route. The seeded issue adds one true positive per cell, however many
times it is restated. A finding the evaluator marks `incidental` is a different
issue that a unanimous judge panel confirmed is real in that source. It adds one
true positive per distinct `incidentalKey` in its cell, never counts toward
recall, and needs no seeded match. Recall still measures only the seeded issue,
and the thresholds are unchanged.
The judge observes held-out attacks in one boot: a `VerificationRequest`'s
`security` field is either one assertion or an array of up to eight assertions
with distinct names, judged in a single before/after pair. The before phase is