9.8 KiB
Chapter 5: Threat Modeling and Risk Analysis
This chapter teaches systematic threat modeling and risk analysis for AI systems. You'll learn to identify critical assets, enumerate threat actors, map attack surfaces specific to LLMs, prioritize risks by impact and likelihood, and create actionable threat models that drive effective red team engagements.
5.1 Why Threat Modeling Matters in AI Red Teaming
Threat modeling is a proactive process that helps you and stakeholders understand what’s at risk, who might attack, and how they could succeed. In AI/LLM systems, the landscape is especially dynamic: you must account for unique risks like model manipulation, data leakage via prompts, unintended plugin behavior, and more.
Effective threat modeling:
- Focuses your testing on the highest-risk assets and attack paths
- Helps you communicate business-relevant risk to stakeholders
- Avoids wasted effort on low-impact findings
5.2 Threat Modeling Process Overview
A robust threat model for AI systems typically includes:
- Defining Assets: What are you trying to protect? (Model weights, training data, business logic, plugins, user data, reputation)
- Identifying Threat Actors: Who might attack? (Disgruntled insiders, malicious users, competitors, hacktivists, nation-states)
- Enumerating Attack Surfaces and Vectors: Where and how could attacks happen? (Prompt/API, plugin misuse, supply chain, logs)
- Analyzing Impact & Likelihood: What happens if each threat is realized, and how probable is it?
- Prioritizing Risks: Rank threats to focus red team efforts.
5.3 Step 1: Defining Assets in AI/LLM Systems
- Model Artifacts: Trained model weights, architectures, fine-tuning data
- Business Logic: Prompt templates, routing, plugin selection criteria
- Data Inputs & Outputs: User queries, logs, plugin responses, database records
- Secrets & Credentials: API keys, private endpoints, plugin credentials
- User Trust & Reputation: Potential for misuse to cause reputational, legal, or compliance harm
Example Questions
- What’s the most confidential/restricted piece of information accessible through the LLM?
- Can an attack on the model lead to broader systems compromise or data exfiltration?
- Could success harm the client’s customers or brand reputation?
5.4 Step 2: Identifying Threat Actors
- Malicious Users: Attempting prompt injection, data leakage, or jailbreaks for personal gain.
- Insiders: Employees or contractors with legitimate but abused access.
- Competitors: Seeking model extraction/theft or sabotage.
- Automated Attackers: Bots fuzzing prompts, APIs, or plugins at scale.
- Unintentional Actors: Well-meaning users who inadvertently trigger unwanted behaviors.
5.5 Step 3: Enumerating Attack Surfaces and Vectors
AI/LLM systems have unique and overlapping attack surfaces:
- Prompt Inputs: Primary user interface, susceptible to injection and manipulation.
- Plugins/APIs: Extensions where the model can trigger unintended behaviors via code or service calls.
- Supply Chain: Dependencies in model training, plugin sourcing, or codebase.
- Model-to-Model Connections: LLMs triggering actions or responses in other LLM-driven systems.
- Logging and Monitoring: Where outputs or sensitive content may leak.
Tools: Use data/flow diagrams and system architecture charts to visualize these surfaces.
5.6 Step 4: Analyzing Impact and Likelihood
For each identified threat:
- Impact: What’s the worst-case outcome? (Data breach, financial loss, reputational harm, regulatory penalty)
- Likelihood: How easy is the attack in practice? Consider attacker capability, system complexity, existing defenses.
Example Threat Table
| Asset | Threat | Actor | Likelihood | Impact | Risk Level |
|---|---|---|---|---|---|
| Model weights | Theft via API | Competitor | Medium | High | High |
| Customer Data | Leakage via prompt | Malicious user | High | High | Critical |
| Plugins | Command Injection | Insider | Low | High | Medium |
| Logs | Data Exfiltration | Insider | Low | Medium | Low |
5.7 Step 5: Prioritizing and Using the Threat Model
- Highlight “Critical” and “High” risk scenarios for focused red team attention.
- Tie each risk back to business impact for client buy-in and prioritization.
- Use this as a living document; update it based on findings from red teaming.
5.8 AI/LLM-Specific Threat Modeling Methodologies
- Adapt STRIDE/DREAD: Traditional security frameworks (e.g., Spoofing, Tampering, Repudiation, etc.) can be tailored for AI systems.
- LLM Kill Chain: Reconnaissance ➔ Prompt Engineering ➔ Model Behavior Manipulation ➔ Data Extraction/Impact.
Tip: Incorporate “AI safety” and “model misuse” perspectives that go beyond classic code/network vulnerability approaches.
5.9 Documenting and Communicating the Threat Model
A good threat model is:
- Visual (models, tables, attack trees)
- Accessible to both technical and business stakeholders
- Used as a reference for reporting and remediation
5.10 Sample Threat Modeling Worksheet (AI System)
- List all entry points to the LLM (UI, API, plugins, ingestion)
- Identify all forms of sensitive data or actions accessible via the LLM
- Brainstorm attacker profiles and motives
- Map end-to-end data flows, including third-party integrations
- Rank potential threats and justify priorities
With a strong threat model, your red team engagement becomes risk-driven and results-focused. The next chapter will walk you through scoping these findings into a feasible, valuable engagement plan.
5.11 Ethical and Legal Considerations
Important
All testing activities must be conducted with proper authorization and within legal boundaries. Unauthorized testing can result in criminal prosecution.
Legal Framework:
- Activities must comply with Computer Fraud and Abuse Act (CFAA) and applicable laws
- Written authorization required before any testing or assessment activities
- Data handling must comply with GDPR, CCPA, and relevant regulations
- Document all activities to demonstrate lawful intent
Ethical Principles:
- Obtain explicit written permission before testing
- Stay within authorized scope and boundaries
- Protect sensitive data and PII encountered during work
- Report findings responsibly through proper channels
- Minimize potential harm to systems and users
Caution
Unauthorized testing or assessment activities are illegal and can result in prosecution, civil liability, and imprisonment. Only conduct these activities in authorized security assessments.
5.12 Conclusion
Key Takeaways:
- Understanding this topic is fundamental to effective AI red teaming and security assessment
- Proper methodology prevents errors and ensures comprehensive, reliable results
- Documentation is critical for reproducibility, legal protection, and knowledge transfer
- Continuous learning is essential as AI systems and threats evolve rapidly
Recommendations for Red Teamers:
- Develop systematic approach to this domain
- Document all findings, methods, and decisions comprehensively
- Stay current with latest developments and research
- Build repeatable processes and checklists
- Collaborate with peers to share knowledge and techniques
Recommendations for Organizations:
- Implement robust processes in this area
- Provide adequate training and resources
- Maintain clear policies and procedures
- Regular review and updates based on lessons learned
- Foster culture of security and continuous improvement
Next Steps:
Continue building expertise across all handbook domains for comprehensive AI security capability.
Tip
Create templates and checklists specific to this chapter's domain. Standardization improves quality and efficiency while reducing errors.
Pre-Engagement Checklist
Administrative:
- Obtain written authorization
- Review and sign Statement of Work
- Establish rules of engagement
- Define scope boundaries clearly
- Set up communication channels
- Identify emergency contacts
Technical Preparation:
- Set up test environment
- Install required tools
- Configure monitoring and logging
- Prepare evidence collection methods
- Test backup procedures
- Document baseline state
Domain-Specific:
- Review domain-specific requirements
- Prepare specialized tools or methods
- Document expected outcomes
- Identify potential risks
- Plan mitigation strategies
Post-Engagement Checklist
Documentation:
- Document all findings with evidence
- Capture screenshots and logs
- Record timestamps
- Note anomalies or unexpected behaviors
- Prepare technical report
- Create executive summary
Cleanup:
- Remove test artifacts
- Verify no persistent changes
- Securely delete temporary files
- Clear test accounts
- Confirm system restoration
- Archive evidence appropriately
Reporting:
- Deliver comprehensive findings report
- Provide remediation guidance
- Offer follow-up support
- Schedule re-testing after remediation
- Conduct lessons learned review