gstack/scripts/resolvers/confidence.ts

/**
 * Confidence calibration resolver
 *
 * Adds confidence scoring rubric to review-producing skills.
 * Every finding includes a 1-10 score that gates display:
 *   7+: show normally
 *   5-6: show with caveat
 *   <5: suppress from main report
 *
 * Pre-emit verification gate (#1539): findings without a quoted code snippet
 * are forced to confidence 4-5 so the existing suppression rule fires
 * automatically. Kills the "field doesn't exist on the model" FP class on
 * mature frameworks like Django/Rails — the model code resolves it in <5min,
 * and the gate forces the reviewer to do that lookup before promoting the
 * finding to the report.
 */
import type { TemplateContext } from './types';

export function generateConfidenceCalibration(_ctx: TemplateContext): string {
  if (_ctx.skillName === 'review') return `## Confidence Calibration

Verify evidence first, then score every finding (1-10) and apply its display rule.

### Pre-emit verification gate

1. **Quote the specific code line:** file:line and verbatim text. For a missing field,
   quote its class definition; for a nullable value, its initialization; for a race, both sides.
2. For framework-generated symbols, read and quote their generating metaclass,
   descriptor, ORM Meta block, migration, decorator or schema. Missing literal
   names in the class body or grep results do not prove absence.
3. **If you cannot quote the motivating line(s), the finding is unverified.**
   Force its confidence to 4-5: use 4 for appendix-only reporting, or 5 only when
   the finding belongs in the main report with the medium-confidence caveat below.
   Never invent speculative confidence 7+.

| Score | Meaning | Display rule |
|-------|---------|-------------|
| 9-10 | Specific code verifies a concrete bug or exploit. | Show normally |
| 7-8 | High-confidence pattern match; very likely correct. | Show normally |
| 5-6 | Moderate; could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" |
| 3-4 | Suspicious but may be fine. | Suppress from main report. Include in appendix only. |
| 1-2 | Speculation. | Only report a suspected release-blocking catastrophe (widespread data loss, total outage or system-wide compromise); label it CRITICAL and explicitly speculative. |

**Finding format:**

\`[CRITICAL|INFORMATIONAL] (confidence: N/10) file:line — description\`

Example:
\`[CRITICAL] (confidence: 9/10) user.rb:42 — SQL injection via string interpolation\`

**Calibration learning:** If the user confirms a reported finding scored < 7 is
real, log the corrected pattern as a learning.`;

  const result = `## Confidence Calibration

Every finding MUST include a confidence score (1-10):

| Score | Meaning | Display rule |
|-------|---------|-------------|
| 9-10 | Verified by reading specific code. Concrete bug or exploit demonstrated. | Show normally |
| 7-8 | High confidence pattern match. Very likely correct. | Show normally |
| 5-6 | Moderate. Could be a false positive. | Show with caveat: "Medium confidence, verify this is actually an issue" |
| 3-4 | Low confidence. Pattern is suspicious but may be fine. | Suppress from main report. Include in appendix only. |
| 1-2 | Speculation. | Only report if severity would be P0. |

**Finding format:**

\\\`[SEVERITY] (confidence: N/10) file:line — description\\\`

Example:
\\\`[P1] (confidence: 9/10) app/models/user.rb:42 — SQL injection via string interpolation in where clause\\\`
\\\`[P2] (confidence: 5/10) app/controllers/api/v1/users_controller.rb:18 — Possible N+1 query, verify with production logs\\\`

### Pre-emit verification gate (#1539 — kills the "field doesn't exist" FP class)

Before any finding is promoted to the report, the gate requires:

1. **Quote the specific code line that motivates the finding** — file:line plus
   the verbatim text of the line(s) that triggered it. If the finding is "field
   X doesn't exist on model Y", quote the lines of class Y where the field
   would live. If "dict.get() might return None", quote the dict initialization.
   If "race condition between A and B", quote both A and B.

2. **If you cannot quote the motivating line(s), the finding is unverified.**
   Force its confidence to 4-5. Use 4 when it should be suppressed from the main
   report; use 5 only when it belongs in the report with the medium-confidence
   caveat. Keep suppressed items in the appendix so reviewers can audit
   calibration. Do not work around this by inventing
   speculative confidence 7+ — that defeats the gate.

**Framework-meta nudge:** When the symbol is generated by a framework
metaclass, descriptor, ORM Meta inner-class, or migration history (Django
\`Meta\`, Rails \`has_many\`/\`scope\`, SQLAlchemy \`relationship\`/\`Column\`,
TypeORM decorators, Sequelize \`init\`/\`belongsTo\`, Prisma generated client),
quote the meta-construct (the \`Meta\` block, the migration, the decorator,
the schema file) instead of expecting the literal name in the class body.
The verification is "I read the source that creates this symbol", not "I
grep'd for the name and didn't find it." Deeper framework-aware verification
(model introspection, migration-history-aware checks, ORM dialect detection)
is deliberately out of scope for the lighter gate — see the deferred
\`~/.gstack-dev/plans/1539-framework-aware-review.md\` design doc.

The FP classes the gate kills (measured against Django Sprint 2.5 #1539):

| FP class | Why the gate catches it |
|---|---|
| "field doesn't exist on model" | Requires quoting the model class body or Meta; the field's absence becomes obvious |
| "dict.get() might be None" | Requires quoting the dict initialization (e.g. Django form's \`cleaned_data\` is \`{}\`-initialized) |
| "save() might lose fields" | Requires quoting the ORM signature or model definition |
| "update_fields might miss X" | Requires quoting the field set; if X doesn't exist, the FP is self-evident |

**Calibration learning:** If you report a finding with confidence < 7 and the user
confirms it IS a real issue, that is a calibration event. Your initial confidence was
too low. Log the corrected pattern as a learning so future reviews catch it with
higher confidence.`;
  return _ctx.skillName === 'plan-eng-review' ? result.replaceAll('\\`', '`') : result;
}