fix(confidence): one severity scale (CRITICAL/INFORMATIONAL) for /ship's review army and /plan-eng-review (#3349)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

plan-eng-review/sections/review-sections.md

@@ -125,7 +125,8 @@ building proposed code. Keep suppressed findings for the output appendix.
 
 ## Confidence Calibration
 
-Every finding MUST include a confidence score (1-10):
+Every finding MUST include a severity, CRITICAL (blocks shipping) or INFORMATIONAL
+(lower severity, still actioned), and a confidence score (1-10):
 
 | Score | Meaning | Display rule |
 |-------|---------|-------------|
@@ -133,20 +134,18 @@ Every finding MUST include a confidence score (1-10):
 | 7-8 | High confidence pattern match. Very likely correct. | Show normally |
 | 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" |
 | 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. |
-| 1-2 | Speculation. | Only report if severity would be P0. |
+| 1-2 | Speculation. | Only report a suspected release-blocking catastrophe (widespread data loss, total outage or system-wide compromise), at CRITICAL severity and marked explicitly speculative. |
 
 **Finding format:**
 
-`[SEVERITY] (confidence: N/10) file:line — description`
+`[CRITICAL|INFORMATIONAL] (confidence: N/10) file:line — description`
 
 Example:
-`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause`
-`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs`
+`[CRITICAL] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause`
+`[INFORMATIONAL] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs`
 
 ### Pre-emit verification gate
 
-Before any finding is promoted to the report, the gate requires:
-
 1. **Quote the specific code line that motivates the finding** — file:line plus
    the verbatim text of the line(s) that triggered it. If the finding is "field
    X doesn't exist on model Y", quote the lines of class Y where the field
@@ -156,9 +155,7 @@ Before any finding is promoted to the report, the gate requires:
 2. **If you cannot quote the motivating line(s), the finding is unverified.**
    Force its confidence to 4-5. Use 4 when it should be suppressed from the main
    report; use 5 only when it belongs in the report with the medium-confidence
-   caveat. Keep suppressed items in the appendix so reviewers can audit
-   calibration. Do not work around this by inventing
-   speculative confidence 7+ — that defeats the gate.
+   caveat. Never invent speculative confidence 7+.
 
 **Framework-meta nudge:** When the symbol is generated by a framework
 metaclass, descriptor, ORM Meta inner-class, or migration history (Django
@@ -178,10 +175,8 @@ False-positive classes the gate catches:
 | "save() might lose fields" | Requires quoting the ORM signature or model definition |
 | "update_fields might miss X" | Requires quoting the field set; if X doesn't exist, the FP is self-evident |
 
-**Calibration learning:** If you report a finding with confidence < 7 and the user
-confirms it IS a real issue, that is a calibration event. Your initial confidence was
-too low. Log the corrected pattern as a learning so future reviews catch it with
-higher confidence.
+**Calibration learning:** If the user confirms a reported finding scored < 7 is
+real, log the corrected pattern as a learning.
 
 ## Decision procedure
 

scripts/resolvers/confidence.ts

@@ -53,7 +53,8 @@ real, log the corrected pattern as a learning.`;
 
   const result = `## Confidence Calibration
 
-Every finding MUST include a confidence score (1-10):
+Every finding MUST include a severity, CRITICAL (blocks shipping) or INFORMATIONAL
+(lower severity, still actioned), and a confidence score (1-10):
 
 | Score | Meaning | Display rule |
 |-------|---------|-------------|
@@ -61,20 +62,18 @@ Every finding MUST include a confidence score (1-10):
 | 7-8 | High confidence pattern match. Very likely correct. | Show normally |
 | 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" |
 | 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. |
-| 1-2 | Speculation. | Only report if severity would be P0. |
+| 1-2 | Speculation. | Only report a suspected release-blocking catastrophe (widespread data loss, total outage or system-wide compromise), at CRITICAL severity and marked explicitly speculative. |
 
 **Finding format:**
 
-\\\`[SEVERITY] (confidence: N/10) file:line — description\\\`
+\\\`[CRITICAL|INFORMATIONAL] (confidence: N/10) file:line — description\\\`
 
 Example:
-\\\`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\\\`
-\\\`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\\\`
+\\\`[CRITICAL] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\\\`
+\\\`[INFORMATIONAL] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\\\`
 
 ### Pre-emit verification gate
 
-Before any finding is promoted to the report, the gate requires:
-
 1. **Quote the specific code line that motivates the finding** — file:line plus
    the verbatim text of the line(s) that triggered it. If the finding is "field
    X doesn't exist on model Y", quote the lines of class Y where the field
@@ -84,9 +83,7 @@ Before any finding is promoted to the report, the gate requires:
 2. **If you cannot quote the motivating line(s), the finding is unverified.**
    Force its confidence to 4-5. Use 4 when it should be suppressed from the main
    report; use 5 only when it belongs in the report with the medium-confidence
-   caveat. Keep suppressed items in the appendix so reviewers can audit
-   calibration. Do not work around this by inventing
-   speculative confidence 7+ — that defeats the gate.
+   caveat. Never invent speculative confidence 7+.
 
 **Framework-meta nudge:** When the symbol is generated by a framework
 metaclass, descriptor, ORM Meta inner-class, or migration history (Django
@@ -106,9 +103,7 @@ False-positive classes the gate catches:
 | "save() might lose fields" | Requires quoting the ORM signature or model definition |
 | "update_fields might miss X" | Requires quoting the field set; if X doesn't exist, the FP is self-evident |
 
-**Calibration learning:** If you report a finding with confidence < 7 and the user
-confirms it IS a real issue, that is a calibration event. Your initial confidence was
-too low. Log the corrected pattern as a learning so future reviews catch it with
-higher confidence.`;
+**Calibration learning:** If the user confirms a reported finding scored < 7 is
+real, log the corrected pattern as a learning.`;
   return _ctx.skillName === 'plan-eng-review' ? result.replaceAll('\\`', '`') : result;
 }

ship/sections/review-army.md

@@ -12,7 +12,8 @@ Finish the complete review and QA before applying any fix in Step 9.4.
 
 ## Confidence Calibration
 
-Every finding MUST include a confidence score (1-10):
+Every finding MUST include a severity, CRITICAL (blocks shipping) or INFORMATIONAL
+(lower severity, still actioned), and a confidence score (1-10):
 
 | Score | Meaning | Display rule |
 |-------|---------|-------------|
@@ -20,20 +21,18 @@ Every finding MUST include a confidence score (1-10):
 | 7-8 | High confidence pattern match. Very likely correct. | Show normally |
 | 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" |
 | 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. |
-| 1-2 | Speculation. | Only report if severity would be P0. |
+| 1-2 | Speculation. | Only report a suspected release-blocking catastrophe (widespread data loss, total outage or system-wide compromise), at CRITICAL severity and marked explicitly speculative. |
 
 **Finding format:**
 
-\`[SEVERITY] (confidence: N/10) file:line — description\`
+\`[CRITICAL|INFORMATIONAL] (confidence: N/10) file:line — description\`
 
 Example:
-\`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\`
-\`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\`
+\`[CRITICAL] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\`
+\`[INFORMATIONAL] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\`
 
 ### Pre-emit verification gate
 
-Before any finding is promoted to the report, the gate requires:
-
 1. **Quote the specific code line that motivates the finding** — file:line plus
    the verbatim text of the line(s) that triggered it. If the finding is "field
    X doesn't exist on model Y", quote the lines of class Y where the field
@@ -43,9 +42,7 @@ Before any finding is promoted to the report, the gate requires:
 2. **If you cannot quote the motivating line(s), the finding is unverified.**
    Force its confidence to 4-5. Use 4 when it should be suppressed from the main
    report; use 5 only when it belongs in the report with the medium-confidence
-   caveat. Keep suppressed items in the appendix so reviewers can audit
-   calibration. Do not work around this by inventing
-   speculative confidence 7+ — that defeats the gate.
+   caveat. Never invent speculative confidence 7+.
 
 **Framework-meta nudge:** When the symbol is generated by a framework
 metaclass, descriptor, ORM Meta inner-class, or migration history (Django
@@ -65,10 +62,8 @@ False-positive classes the gate catches:
 | "save() might lose fields" | Requires quoting the ORM signature or model definition |
 | "update_fields might miss X" | Requires quoting the field set; if X doesn't exist, the FP is self-evident |
 
-**Calibration learning:** If you report a finding with confidence < 7 and the user
-confirms it IS a real issue, that is a calibration event. Your initial confidence was
-too low. Log the corrected pattern as a learning so future reviews catch it with
-higher confidence.
+**Calibration learning:** If the user confirms a reported finding scored < 7 is
+real, log the corrected pattern as a learning.
 
 ### Core checklist