From af2421eeb4afb3b3be0daaad4f9e09cf036356fa Mon Sep 17 00:00:00 2001 From: garrytan Date: Sun, 25 Oct 2026 02:13:14 +0000 Subject: [PATCH] fix(confidence): one severity scale (CRITICAL/INFORMATIONAL) for /ship's review army and /plan-eng-review (#3349) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- plan-eng-review/sections/review-sections.md | 23 ++++++++------------- scripts/resolvers/confidence.ts | 23 ++++++++------------- ship/sections/review-army.md | 23 ++++++++------------- 3 files changed, 27 insertions(+), 42 deletions(-) diff --git a/plan-eng-review/sections/review-sections.md b/plan-eng-review/sections/review-sections.md index 0f43c741c..399ee8401 100644 --- a/plan-eng-review/sections/review-sections.md +++ b/plan-eng-review/sections/review-sections.md @@ -125,7 +125,8 @@ building proposed code. Keep suppressed findings for the output appendix. ## Confidence Calibration -Every finding MUST include a confidence score (1-10): +Every finding MUST include a severity, CRITICAL (blocks shipping) or INFORMATIONAL +(lower severity, still actioned), and a confidence score (1-10): | Score | Meaning | Display rule | |-------|---------|-------------| @@ -133,20 +134,18 @@ Every finding MUST include a confidence score (1-10): | 7-8 | High confidence pattern match. Very likely correct. | Show normally | | 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" | | 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. | -| 1-2 | Speculation. | Only report if severity would be P0. | +| 1-2 | Speculation. | Only report a suspected release-blocking catastrophe (widespread data loss, total outage or system-wide compromise), at CRITICAL severity and marked explicitly speculative. | **Finding format:** -`[SEVERITY] (confidence: N/10) file:line — description` +`[CRITICAL|INFORMATIONAL] (confidence: N/10) file:line — description` Example: -`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause` -`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs` +`[CRITICAL] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause` +`[INFORMATIONAL] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs` ### Pre-emit verification gate -Before any finding is promoted to the report, the gate requires: - 1. **Quote the specific code line that motivates the finding** — file:line plus the verbatim text of the line(s) that triggered it. If the finding is "field X doesn't exist on model Y", quote the lines of class Y where the field @@ -156,9 +155,7 @@ Before any finding is promoted to the report, the gate requires: 2. **If you cannot quote the motivating line(s), the finding is unverified.** Force its confidence to 4-5. Use 4 when it should be suppressed from the main report; use 5 only when it belongs in the report with the medium-confidence - caveat. Keep suppressed items in the appendix so reviewers can audit - calibration. Do not work around this by inventing - speculative confidence 7+ — that defeats the gate. + caveat. Never invent speculative confidence 7+. **Framework-meta nudge:** When the symbol is generated by a framework metaclass, descriptor, ORM Meta inner-class, or migration history (Django @@ -178,10 +175,8 @@ False-positive classes the gate catches: | "save() might lose fields" | Requires quoting the ORM signature or model definition | | "update_fields might miss X" | Requires quoting the field set; if X doesn't exist, the FP is self-evident | -**Calibration learning:** If you report a finding with confidence < 7 and the user -confirms it IS a real issue, that is a calibration event. Your initial confidence was -too low. Log the corrected pattern as a learning so future reviews catch it with -higher confidence. +**Calibration learning:** If the user confirms a reported finding scored < 7 is +real, log the corrected pattern as a learning. ## Decision procedure diff --git a/scripts/resolvers/confidence.ts b/scripts/resolvers/confidence.ts index f10d15a79..747bdc225 100644 --- a/scripts/resolvers/confidence.ts +++ b/scripts/resolvers/confidence.ts @@ -53,7 +53,8 @@ real, log the corrected pattern as a learning.`; const result = `## Confidence Calibration -Every finding MUST include a confidence score (1-10): +Every finding MUST include a severity, CRITICAL (blocks shipping) or INFORMATIONAL +(lower severity, still actioned), and a confidence score (1-10): | Score | Meaning | Display rule | |-------|---------|-------------| @@ -61,20 +62,18 @@ Every finding MUST include a confidence score (1-10): | 7-8 | High confidence pattern match. Very likely correct. | Show normally | | 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" | | 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. | -| 1-2 | Speculation. | Only report if severity would be P0. | +| 1-2 | Speculation. | Only report a suspected release-blocking catastrophe (widespread data loss, total outage or system-wide compromise), at CRITICAL severity and marked explicitly speculative. | **Finding format:** -\\\`[SEVERITY] (confidence: N/10) file:line — description\\\` +\\\`[CRITICAL|INFORMATIONAL] (confidence: N/10) file:line — description\\\` Example: -\\\`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\\\` -\\\`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\\\` +\\\`[CRITICAL] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\\\` +\\\`[INFORMATIONAL] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\\\` ### Pre-emit verification gate -Before any finding is promoted to the report, the gate requires: - 1. **Quote the specific code line that motivates the finding** — file:line plus the verbatim text of the line(s) that triggered it. If the finding is "field X doesn't exist on model Y", quote the lines of class Y where the field @@ -84,9 +83,7 @@ Before any finding is promoted to the report, the gate requires: 2. **If you cannot quote the motivating line(s), the finding is unverified.** Force its confidence to 4-5. Use 4 when it should be suppressed from the main report; use 5 only when it belongs in the report with the medium-confidence - caveat. Keep suppressed items in the appendix so reviewers can audit - calibration. Do not work around this by inventing - speculative confidence 7+ — that defeats the gate. + caveat. Never invent speculative confidence 7+. **Framework-meta nudge:** When the symbol is generated by a framework metaclass, descriptor, ORM Meta inner-class, or migration history (Django @@ -106,9 +103,7 @@ False-positive classes the gate catches: | "save() might lose fields" | Requires quoting the ORM signature or model definition | | "update_fields might miss X" | Requires quoting the field set; if X doesn't exist, the FP is self-evident | -**Calibration learning:** If you report a finding with confidence < 7 and the user -confirms it IS a real issue, that is a calibration event. Your initial confidence was -too low. Log the corrected pattern as a learning so future reviews catch it with -higher confidence.`; +**Calibration learning:** If the user confirms a reported finding scored < 7 is +real, log the corrected pattern as a learning.`; return _ctx.skillName === 'plan-eng-review' ? result.replaceAll('\\`', '`') : result; } diff --git a/ship/sections/review-army.md b/ship/sections/review-army.md index 348458c08..b5c65db07 100644 --- a/ship/sections/review-army.md +++ b/ship/sections/review-army.md @@ -12,7 +12,8 @@ Finish the complete review and QA before applying any fix in Step 9.4. ## Confidence Calibration -Every finding MUST include a confidence score (1-10): +Every finding MUST include a severity, CRITICAL (blocks shipping) or INFORMATIONAL +(lower severity, still actioned), and a confidence score (1-10): | Score | Meaning | Display rule | |-------|---------|-------------| @@ -20,20 +21,18 @@ Every finding MUST include a confidence score (1-10): | 7-8 | High confidence pattern match. Very likely correct. | Show normally | | 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" | | 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. | -| 1-2 | Speculation. | Only report if severity would be P0. | +| 1-2 | Speculation. | Only report a suspected release-blocking catastrophe (widespread data loss, total outage or system-wide compromise), at CRITICAL severity and marked explicitly speculative. | **Finding format:** -\`[SEVERITY] (confidence: N/10) file:line — description\` +\`[CRITICAL|INFORMATIONAL] (confidence: N/10) file:line — description\` Example: -\`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\` -\`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\` +\`[CRITICAL] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\` +\`[INFORMATIONAL] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\` ### Pre-emit verification gate -Before any finding is promoted to the report, the gate requires: - 1. **Quote the specific code line that motivates the finding** — file:line plus the verbatim text of the line(s) that triggered it. If the finding is "field X doesn't exist on model Y", quote the lines of class Y where the field @@ -43,9 +42,7 @@ Before any finding is promoted to the report, the gate requires: 2. **If you cannot quote the motivating line(s), the finding is unverified.** Force its confidence to 4-5. Use 4 when it should be suppressed from the main report; use 5 only when it belongs in the report with the medium-confidence - caveat. Keep suppressed items in the appendix so reviewers can audit - calibration. Do not work around this by inventing - speculative confidence 7+ — that defeats the gate. + caveat. Never invent speculative confidence 7+. **Framework-meta nudge:** When the symbol is generated by a framework metaclass, descriptor, ORM Meta inner-class, or migration history (Django @@ -65,10 +62,8 @@ False-positive classes the gate catches: | "save() might lose fields" | Requires quoting the ORM signature or model definition | | "update_fields might miss X" | Requires quoting the field set; if X doesn't exist, the FP is self-evident | -**Calibration learning:** If you report a finding with confidence < 7 and the user -confirms it IS a real issue, that is a calibration event. Your initial confidence was -too low. Log the corrected pattern as a learning so future reviews catch it with -higher confidence. +**Calibration learning:** If the user confirms a reported finding scored < 7 is +real, log the corrected pattern as a learning. ### Core checklist