Chapter 5: Threat Modeling and Risk Analysis

5.1 Why Threat Modeling Matters in AI Red Teaming

Threat modeling is a proactive process that helps you and stakeholders understand what’s at risk, who might attack, and how they could succeed. In AI/LLM systems, the landscape is especially dynamic: you must account for unique risks like model manipulation, data leakage via prompts, unintended plugin behavior, and more.

Effective threat modeling:

Focuses your testing on the highest-risk assets and attack paths
Helps you communicate business-relevant risk to stakeholders
Avoids wasted effort on low-impact findings

5.2 Threat Modeling Process Overview

A robust threat model for AI systems typically includes:

Defining Assets: What are you trying to protect? (Model weights, training data, business logic, plugins, user data, reputation)
Identifying Threat Actors: Who might attack? (Disgruntled insiders, malicious users, competitors, hacktivists, nation-states)
Enumerating Attack Surfaces and Vectors: Where and how could attacks happen? (Prompt/API, plugin misuse, supply chain, logs)
Analyzing Impact & Likelihood: What happens if each threat is realized, and how probable is it?
Prioritizing Risks: Rank threats to focus red team efforts.

5.3 Step 1: Defining Assets in AI/LLM Systems

Model Artifacts: Trained model weights, architectures, fine-tuning data
Business Logic: Prompt templates, routing, plugin selection criteria
Data Inputs & Outputs: User queries, logs, plugin responses, database records
Secrets & Credentials: API keys, private endpoints, plugin credentials
User Trust & Reputation: Potential for misuse to cause reputational, legal, or compliance harm

Example Questions

What’s the most confidential/restricted piece of information accessible through the LLM?
Can an attack on the model lead to broader systems compromise or data exfiltration?
Could success harm the client’s customers or brand reputation?

5.4 Step 2: Identifying Threat Actors

Malicious Users: Attempting prompt injection, data leakage, or jailbreaks for personal gain.
Insiders: Employees or contractors with legitimate but abused access.
Competitors: Seeking model extraction/theft or sabotage.
Automated Attackers: Bots fuzzing prompts, APIs, or plugins at scale.
Unintentional Actors: Well-meaning users who inadvertently trigger unwanted behaviors.

5.5 Step 3: Enumerating Attack Surfaces and Vectors

AI/LLM systems have unique and overlapping attack surfaces:

Prompt Inputs: Primary user interface, susceptible to injection and manipulation.
Plugins/APIs: Extensions where the model can trigger unintended behaviors via code or service calls.
Supply Chain: Dependencies in model training, plugin sourcing, or codebase.
Model-to-Model Connections: LLMs triggering actions or responses in other LLM-driven systems.
Logging and Monitoring: Where outputs or sensitive content may leak.

Tools: Use data/flow diagrams and system architecture charts to visualize these surfaces.

5.6 Step 4: Analyzing Impact and Likelihood

For each identified threat:

Impact: What’s the worst-case outcome? (Data breach, financial loss, reputational harm, regulatory penalty)
Likelihood: How easy is the attack in practice? Consider attacker capability, system complexity, existing defenses.

Example Threat Table

Asset	Threat	Actor	Likelihood	Impact	Risk Level
Model weights	Theft via API	Competitor	Medium	High	High
Customer Data	Leakage via prompt	Malicious user	High	High	Critical
Plugins	Command Injection	Insider	Low	High	Medium
Logs	Data Exfiltration	Insider	Low	Medium	Low

5.7 Step 5: Prioritizing and Using the Threat Model

Highlight “Critical” and “High” risk scenarios for focused red team attention.
Tie each risk back to business impact for client buy-in and prioritization.
Use this as a living document; update it based on findings from red teaming.

5.8 AI/LLM-Specific Threat Modeling Methodologies

Adapt STRIDE/DREAD: Traditional security frameworks (e.g., Spoofing, Tampering, Repudiation, etc.) can be tailored for AI systems.
LLM Kill Chain: Reconnaissance ➔ Prompt Engineering ➔ Model Behavior Manipulation ➔ Data Extraction/Impact.

Tip: Incorporate “AI safety” and “model misuse” perspectives that go beyond classic code/network vulnerability approaches.

5.9 Documenting and Communicating the Threat Model

A good threat model is:

Visual (models, tables, attack trees)
Accessible to both technical and business stakeholders
Used as a reference for reporting and remediation

5.10 Sample Threat Modeling Worksheet (AI System)

List all entry points to the LLM (UI, API, plugins, ingestion)
Identify all forms of sensitive data or actions accessible via the LLM
Brainstorm attacker profiles and motives
Map end-to-end data flows, including third-party integrations
Rank potential threats and justify priorities

With a strong threat model, your red team engagement becomes risk-driven and results-focused. The next chapter will walk you through scoping these findings into a feasible, valuable engagement plan.

5.5 KiB Raw Blame History Unescape Escape