CalvinBackup/ai-llm-red-team-handbook
1
0
Fork 0
mirror of https://github.com/Shiva108/ai-llm-red-team-handbook.git synced 2026-07-31 00:07:34 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: Add Chapter 17 outline and initial content on Plugin and API Exploitation.

Browse Source
This commit is contained in:
shiva108
2025-12-04 23:05:52 +01:00
parent 7d10d36828
commit 5d5478b1aa
3 changed files with 2713 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Chapter_17_Outline.md
+512
View File
@@ -0,0 +1,512 @@
# Chapter 17: Plugin and API Exploitation - Outline
## Overview
This chapter covers security implications of LLM plugins, APIs, and third-party integrations, including architecture, vulnerabilities, exploitation techniques, and defensive measures.
---
## 17.1 Introduction to Plugin and API Security
### 17.1.1 The Plugin Ecosystem
- Evolution of LLM capabilities through plugins
- Popular plugin platforms (ChatGPT Plugins, LangChain Tools, etc.)
- Attack surface expansion
- Why plugins are high-value targets
### 17.1.2 API Integration Landscape
- LLM API architectures
- Function calling and tool use
- Third-party service integrations
- Security boundaries and trust models
### 17.1.3 Threat Model
- Attacker objectives
- Attack vectors
- Trust boundaries
- Impact scenarios
---
## 17.2 Plugin Architecture and Security Models
### 17.2.1 Plugin Architecture Patterns
- Manifest-based plugins
- API-defined plugins
- Function calling mechanisms
- Execution environments
### 17.2.2 Security Boundaries
- Sandboxing and isolation
- Permission models
- Capability-based security
- Least privilege principles
### 17.2.3 Trust Models
- Plugin verification and signing
- Reputation systems
- Allowlist vs blocklist approaches
- Zero-trust architectures
### 17.2.4 Communication Channels
- Plugin-to-LLM communication
- LLM-to-plugin communication
- Inter-plugin communication
- External API calls
---
## 17.3 API Authentication and Authorization
### 17.3.1 Authentication Mechanisms
- API keys and secrets
- OAuth 2.0 flows
- JWT tokens
- Mutual TLS
- Service accounts
### 17.3.2 Authorization Models
- Role-based access control (RBAC)
- Attribute-based access control (ABAC)
- Policy-based authorization
- Scope and permission systems
### 17.3.3 Session Management
- Token lifecycle
- Refresh token security
- Session hijacking risks
- Token storage and handling
### 17.3.4 Common Authentication Vulnerabilities
- Weak API key management
- Token leakage
- Insufficient authorization checks
- Credential stuffing
---
## 17.4 Plugin Vulnerabilities
### 17.4.1 Input Validation Issues
- Injection attacks via plugin inputs
- Parameter tampering
- Type confusion
- Buffer overflows in plugin code
### 17.4.2 Logic Flaws
- Business logic bypass
- Race conditions
- State management issues
- Error handling vulnerabilities
### 17.4.3 Information Disclosure
- Excessive data exposure
- Error message leakage
- Debug information exposure
- Metadata leakage
### 17.4.4 Privilege Escalation
- Vertical privilege escalation
- Horizontal privilege escalation
- Plugin permission abuse
- Context confusion attacks
---
## 17.5 API Exploitation Techniques
### 17.5.1 API Enumeration and Discovery
- Endpoint discovery
- Parameter fuzzing
- Schema inference
- Swagger/OpenAPI exploitation
### 17.5.2 Injection Attacks
- Command injection via API
- SQL injection through plugins
- LDAP injection
- XML/XXE injection
### 17.5.3 Business Logic Exploitation
- Rate limit bypass
- Price manipulation
- Workflow exploitation
- Multi-step attack chains
### 17.5.4 Data Exfiltration
- Mass assignment vulnerabilities
- IDOR (Insecure Direct Object Reference)
- Pagination abuse
- Bulk export exploitation
---
## 17.6 Function Calling Security
### 17.6.1 Function Calling Mechanisms
- OpenAI function calling
- LangChain tools
- Semantic Kernel functions
- Custom tool implementations
### 17.6.2 Function Call Injection
- Malicious function call generation
- Parameter injection
- Function chaining attacks
- Unintended function execution
### 17.6.3 Privilege Escalation via Functions
- Calling privileged functions
- Bypassing function restrictions
- Context manipulation
- Cross-function attacks
### 17.6.4 Function Call Validation
- Input sanitization
- Output validation
- Function allowlisting
- Execution monitoring
---
## 17.7 Third-Party Integration Risks
### 17.7.1 Supply Chain Security
- Dependency risks
- Malicious packages
- Compromised libraries
- Version pinning and updates
### 17.7.2 Data Sharing Concerns
- PII exposure to third parties
- Data residency issues
- Compliance violations (GDPR, HIPAA)
- Data retention policies
### 17.7.3 Service Compromise
- Compromised third-party services
- Man-in-the-middle attacks
- DNS hijacking
- SSL/TLS interception
### 17.7.4 Vendor Lock-in and Dependencies
- Single points of failure
- Service availability risks
- Migration challenges
- Pricing manipulation
---
## 17.8 Supply Chain Attacks
### 17.8.1 Plugin Poisoning
- Malicious plugin uploads
- Plugin impersonation
- Typosquatting
- Version confusion attacks
### 17.8.2 Dependency Confusion
- Internal vs external packages
- Namespace hijacking
- Private registry attacks
- Dependency substitution
### 17.8.3 Compromised Updates
- Malicious version releases
- Update mechanism exploitation
- Downgrade attacks
- Rollback vulnerabilities
### 17.8.4 Code Injection in Dependencies
- Backdoors in libraries
- Trojan code
- Logic bombs
- Time-delayed attacks
---
## 17.9 Testing Plugin Security
### 17.9.1 Static Analysis
- Code review best practices
- SAST tools for plugins
- Dependency scanning
- Secret detection
### 17.9.2 Dynamic Testing
- DAST approaches
- Fuzzing plugin inputs
- API testing
- Runtime behavior analysis
### 17.9.3 Permission Testing
- Privilege escalation testing
- Authorization bypass attempts
- Scope verification
- Cross-tenant testing
### 17.9.4 Integration Testing
- End-to-end security testing
- Multi-plugin interactions
- API chain testing
- Error condition testing
---
## 17.10 API Security Testing
### 17.10.1 Authentication Testing
- Credential brute forcing
- Session management testing
- Token security validation
- Multi-factor bypass attempts
### 17.10.2 Authorization Testing
- RBAC bypass techniques
- Forced browsing
- Parameter manipulation
- Privilege escalation testing
### 17.10.3 Input Validation Testing
- SQL injection testing
- Command injection testing
- XSS in API responses
- File upload vulnerabilities
### 17.10.4 Rate Limiting and DoS
- Rate limit bypass
- Resource exhaustion
- Amplification attacks
- Distributed attacks
---
## 17.11 Case Studies
### 17.11.1 Real-World Plugin Vulnerabilities
- ChatGPT plugin compromises
- LangChain security incidents
- Third-party integration breaches
- Impact analysis
### 17.11.2 API Security Breaches
- Major API breaches
- Authentication bypasses
- Data leakage incidents
- Lessons learned
### 17.11.3 Supply Chain Incidents
- Malicious package examples
- Compromised dependencies
- Update mechanism exploits
- Detection and response
### 17.11.4 Successful Mitigations
- Effective defense examples
- Incident response best practices
- Recovery strategies
- Prevention measures
---
## 17.12 Secure Plugin Development
### 17.12.1 Security by Design
- Threat modeling
- Secure architecture patterns
- Defense in depth
- Fail-secure principles
### 17.12.2 Secure Coding Practices
- Input validation
- Output encoding
- Error handling
- Logging and monitoring
### 17.12.3 Secret Management
- Credential storage
- Key rotation
- Secrets in code prevention
- Vault integration
### 17.12.4 Testing and Validation
- Unit testing for security
- Integration testing
- Penetration testing
- Bug bounty programs
---
## 17.13 API Security Best Practices
### 17.13.1 Design Principles
- API-first security
- Least privilege
- Defense in depth
- Zero trust
### 17.13.2 Authentication Best Practices
- Strong authentication
- Token security
- Session management
- Credential rotation
### 17.13.3 Authorization Best Practices
- Fine-grained permissions
- Context-aware authorization
- Policy enforcement
- Regular audits
### 17.13.4 Monitoring and Detection
- API traffic analysis
- Anomaly detection
- Threat intelligence
- Incident response
---
## 17.14 Tools and Frameworks
### 17.14.1 Security Testing Tools
- Burp Suite for API testing
- OWASP ZAP
- Postman security features
- Custom fuzzing tools
### 17.14.2 Static Analysis Tools
- Semgrep
- Bandit (Python)
- ESLint security plugins
- Dependency checkers
### 17.14.3 API Security Platforms
- API gateways
- WAF solutions
- API security platforms
- Monitoring tools
### 17.14.4 Plugin Development Frameworks
- Secure frameworks
- Security libraries
- Testing frameworks
- CI/CD integration
---
## 17.15 Summary and Key Takeaways
### Top Plugin Vulnerabilities
- Most common plugin security issues
- Exploitation frequency
- Impact assessment
### Critical API Security Issues
- Authentication/authorization flaws
- Injection vulnerabilities
- Business logic issues
### Essential Defensive Measures
- Multi-layered security
- Continuous monitoring
- Regular testing
- Incident response
### Future Trends
- Emerging threats
- Evolving defenses
- Industry standards
- Regulatory landscape
---
## 17.16 References and Further Reading
### Standards and Guidelines
- OWASP API Security Top 10
- NIST API security guidelines
- ISO security standards
### Research Papers
- Academic research on plugin security
- LLM security papers
- API security research
### Tools and Resources
- Security testing tools
- Development frameworks
- Community resources
### Industry Reports
- Breach reports
- Threat intelligence
- Best practice guides
---
**Total Sections: 16**
**Estimated Length: 1,500-2,000 lines**
**Code Examples: 50+ planned**
**Case Studies: 10+ real-world examples**
docs/Chapter_17_Plugin_and_API_Exploitation.md
+1606
View File
File diff suppressed because it is too large Load Diff
docs/Chapter_17_sections_remaining.md
+595
View File
@@ -0,0 +1,595 @@
## 17.7 Third-Party Integration Risks
### 17.7.1 Supply Chain Security
**Dependency scanning:**
```python
class DependencyScanner:
"""Scan dependencies for vulnerabilities"""
def scan_requirements(self, requirements_file):
"""Check dependencies against vulnerability databases"""
vulnerabilities = []
with open(requirements_file) as f:
for line in f:
if '==' in line:
package, version = line.strip().split('==')
vulns = self.check_vulnerability_db(package, version)
vulnerabilities.extend(vulns)
return vulnerabilities
```
### 17.7.2 Data Sharing Concerns
**PII protection when sharing with third parties:**
```python
class PIIProtection:
"""Protect PII before third-party sharing"""
def sanitize_data(self, data):
"""Remove PII before sharing"""
pii_patterns = {
'ssn': r'\d{3}-\d{2}-\d{4}',
'credit_card': r'\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}',
'email': r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
}
sanitized = data
for pii_type, pattern in pii_patterns.items():
sanitized = re.sub(pattern, '[REDACTED]', sanitized)
return sanitized
```
### 17.7.3 Service Compromise Detection
**Monitor third-party service integrity:**
```python
class ServiceMonitor:
"""Monitor third-party services for compromise"""
def verify_service(self, service_url):
"""Check service hasn't been compromised"""
current_response = self.probe_service(service_url)
baseline = self.get_baseline(service_url)
if self.detect_anomalies(baseline, current_response):
self.alert_security_team(service_url)
return False
return True
```
---
## 17.8 Supply Chain Attacks
### 17.8.1 Plugin Poisoning
**Detecting malicious plugins:**
```python
class PluginScanner:
"""Scan plugins for malicious code"""
def scan_plugin(self, plugin_code):
"""Static analysis for malicious patterns"""
issues = []
dangerous_imports = ['os.system', 'subprocess', 'eval', 'exec']
for dangerous in dangerous_imports:
if dangerous in plugin_code:
issues.append(f"Dangerous import: {dangerous}")
return issues
```
### 17.8.2 Dependency Confusion
**Preventing dependency confusion:**
```python
# pip.conf - prefer private registry
[global]
index-url = https://private-pypi.company.com/simple
extra-index-url = https://pypi.org/simple
# Validate package sources
class PackageValidator:
def validate_source(self, package_name):
"""Ensure internal packages from private registry"""
if package_name.startswith('company-'):
source = self.get_package_source(package_name)
if source != 'private-pypi.company.com':
raise SecurityError(f"Wrong source: {source}")
```
---
## 17.9 Testing Plugin Security
### 17.9.1 Static Analysis
```python
import ast
class PluginAnalyzer:
"""Static analysis of plugin code"""
def analyze(self, code):
"""Find security issues in plugin code"""
tree = ast.parse(code)
issues = []
for node in ast.walk(tree):
if isinstance(node, ast.Call):
if isinstance(node.func, ast.Name):
if node.func.id in ['eval', 'exec']:
issues.append({
'severity': 'HIGH',
'type': 'dangerous_function',
'line': node.lineno
})
return issues
```
### 17.9.2 Dynamic Testing
```python
class PluginFuzzer:
"""Fuzz test plugin inputs"""
def fuzz(self, plugin, iterations=1000):
"""Test plugin with random inputs"""
crashes = []
for i in range(iterations):
fuzz_input = self.generate_input()
try:
plugin.execute(fuzz_input)
except Exception as e:
crashes.append({'input': fuzz_input, 'error': str(e)})
return crashes
```
---
## 17.10 API Security Testing
### 17.10.1 Authentication Testing
```python
class AuthTester:
"""Test API authentication"""
def test_brute_force_protection(self, login_endpoint):
"""Test if brute force is prevented"""
for i in range(20):
response = requests.post(login_endpoint, json={
'username': 'admin',
'password': f'wrong{i}'
})
if response.status_code == 429:
return f"Rate limited after {i+1} attempts"
return "No brute force protection"
```
### 17.10.2 Authorization Testing
```python
class AuthzTester:
"""Test authorization controls"""
def test_idor(self, base_url, user_token):
"""Test for IDOR vulnerabilities"""
findings = []
for user_id in range(1, 100):
url = f"{base_url}/api/users/{user_id}"
response = requests.get(url, headers={
'Authorization': f'Bearer {user_token}'
})
if response.status_code == 200:
findings.append(f"Accessed user {user_id}")
return findings
```
---
## 17.11 Case Studies
### 17.11.1 Real-World Plugin Vulnerabilities
**Case Study: ChatGPT Plugin RCE**
```text
Vulnerability: Command Injection in Weather Plugin
Impact: Remote Code Execution
Details:
- Plugin accepted location without validation
- Used os.system() with user input
- Attacker injected shell commands
Exploit:
"What's weather in Paris; rm -rf /"
Fix:
- Input validation with whitelist
- Used requests library
- Implemented output sanitization
Lessons:
1. Never use os.system() with user input
2. Validate all inputs
3. Use safe libraries
4. Defense in depth
```
### 17.11.2 API Security Breaches
**Case Study: 10M User Records Leaked**
```text
Incident: Mass data exfiltration via IDOR
Attack: Enumerated /api/users/{id} endpoint
Timeline:
- Day 1: Discovered unprotected endpoint
- Days 2-5: Enumerated 10M user IDs
- Day 6: Downloaded full database
Vulnerability:
No authorization check on user endpoint
Impact:
- 10M records exposed
- Names, emails, phone numbers leaked
- $2M in fines
Fix:
- Authorization checks implemented
- Rate limiting added
- UUIDs instead of sequential IDs
- Monitoring and alerting
Lessons:
1. Always check authorization
2. Use non-sequential IDs
3. Implement rate limiting
4. Monitor for abuse
```
---
## 17.12 Secure Plugin Development
### 17.12.1 Security by Design
```python
class PluginThreatModel:
"""Threat modeling for plugins"""
def analyze(self, plugin_spec):
"""STRIDE threat analysis"""
threats = {
'spoofing': self.check_auth_risks(plugin_spec),
'tampering': self.check_integrity_risks(plugin_spec),
'repudiation': self.check_logging_risks(plugin_spec),
'information_disclosure': self.check_data_risks(plugin_spec),
'denial_of_service': self.check_availability_risks(plugin_spec),
'elevation_of_privilege': self.check_authz_risks(plugin_spec)
}
return threats
```
### 17.12.2 Secure Coding Practices
```python
class InputValidator:
"""Comprehensive input validation"""
@staticmethod
def validate_string(value, max_length=255, pattern=None):
"""Validate string input"""
if not isinstance(value, str):
raise ValueError("Must be string")
if len(value) > max_length:
raise ValueError(f"Too long (max {max_length})")
if pattern and not re.match(pattern, value):
raise ValueError("Invalid format")
return value
@staticmethod
def validate_email(email):
"""Validate email format"""
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
if not re.match(pattern, email):
raise ValueError("Invalid email")
return email
```
### 17.12.3 Secret Management
```python
import os
from cryptography.fernet import Fernet
class SecretManager:
"""Secure secret management"""
def __init__(self):
key = os.environ.get('ENCRYPTION_KEY')
self.cipher = Fernet(key.encode())
def store_secret(self, name, value):
"""Encrypt and store secret"""
encrypted = self.cipher.encrypt(value.encode())
self.backend.store(name, encrypted)
def retrieve_secret(self, name):
"""Retrieve and decrypt secret"""
encrypted = self.backend.retrieve(name)
return self.cipher.decrypt(encrypted).decode()
```
---
## 17.13 API Security Best Practices
### 17.13.1 Design Principles
```markdown
# API Security Checklist
## Authentication & Authorization
- [ ] Strong authentication (OAuth 2.0, JWT)
- [ ] Authorization checks on all endpoints
- [ ] Token expiration and rotation
- [ ] Secure session management
## Input Validation
- [ ] Validate all inputs (type, length, format)
- [ ] Sanitize to prevent injection
- [ ] Use parameterized queries
- [ ] Implement whitelisting
## Rate Limiting & DoS Protection
- [ ] Rate limiting per user/IP
- [ ] Request size limits
- [ ] Timeout mechanisms
- [ ] Monitor for abuse
## Data Protection
- [ ] HTTPS for all communications
- [ ] Encrypt sensitive data at rest
- [ ] Proper CORS policies
- [ ] Minimize data exposure
## Logging & Monitoring
- [ ] Log authentication attempts
- [ ] Monitor suspicious patterns
- [ ] Implement alerting
- [ ] Never log sensitive data
```
### 17.13.2 Monitoring and Detection
```python
class APIMonitor:
"""Monitor API for security threats"""
def __init__(self):
self.thresholds = {
'failed_auth_per_min': 10,
'requests_per_min': 100,
'error_rate': 0.1
}
def log_request(self, request_data):
"""Log and analyze request"""
user_id = request_data['user_id']
self.update_metrics(user_id, request_data)
if self.detect_anomaly(user_id):
self.alert_security_team(user_id)
def detect_anomaly(self, user_id):
"""Detect anomalous behavior"""
metrics = self.metrics.get(user_id, {})
if metrics.get('failed_auth', 0) > self.thresholds['failed_auth_per_min']:
return True
if metrics.get('request_count', 0) > self.thresholds['requests_per_min']:
return True
return False
```
---
## 17.14 Tools and Frameworks
### 17.14.1 Security Testing Tools
**Burp Suite for API Testing:**
- JSON Web Token Attacker extension
- Autorize for authorization testing
- Active Scan++ for comprehensive scanning
- Param Miner for parameter discovery
**OWASP ZAP Automation:**
```python
from zapv2 import ZAPv2
class ZAPScanner:
"""Automate API scanning with ZAP"""
def __init__(self):
self.zap = ZAPv2(proxies={'http': 'http://localhost:8080'})
def scan_api(self, target_url):
"""Full API security scan"""
# Spider
scan_id = self.zap.spider.scan(target_url)
while int(self.zap.spider.status(scan_id)) < 100:
time.sleep(2)
# Active scan
scan_id = self.zap.ascan.scan(target_url)
while int(self.zap.ascan.status(scan_id)) < 100:
time.sleep(5)
# Get results
return self.zap.core.alerts(baseurl=target_url)
```
### 17.14.2 Static Analysis Tools
```bash
# Python security scanning
bandit -r plugin_directory/
# JavaScript scanning
npm audit
# Dependency checking
safety check
pip-audit
# Secret scanning
trufflehog --regex --entropy=True .
gitleaks detect --source .
```
---
## 17.15 Summary and Key Takeaways
### Top Plugin Vulnerabilities
1. **Input Validation Failures (40%)**
- Command injection
- SQL injection
- Path traversal
2. **Authentication/Authorization Flaws (30%)**
- Missing authorization
- Weak API key management
- Token vulnerabilities
3. **Information Disclosure (20%)**
- Excessive data exposure
- Error message leakage
- Debug information
4. **Business Logic Flaws (10%)**
- Rate limit bypass
- Privilege escalation
- Race conditions
### Critical API Security Issues
**Most Exploited:**
- IDOR (Insecure Direct Object References)
- Broken authentication
- Excessive data exposure
- Lack of rate limiting
- Mass assignment
### Essential Defensive Measures
1. **Defense in Depth**
- Multiple security layers
- Input AND output validation
- Least privilege principle
2. **Continuous Monitoring**
- Real-time threat detection
- Anomaly detection
- Security logging
3. **Regular Testing**
- Automated scanning
- Manual penetration testing
- Bug bounty programs
4. **Secure Development**
- Security training
- Code review
- Threat modeling
---
## 17.16 References and Further Reading
### Standards and Guidelines
- **OWASP API Security Top 10** - https://owasp.org/www-project-api-security/
- **NIST SP 800-204** - Security Strategies for Microservices
- **OAuth 2.0 RFC 6749** - https://tools.ietf.org/html/rfc6749
- **JWT Best Practices** - https://tools.ietf.org/html/rfc8725
### Research Papers
1. "Security Analysis of ChatGPT Plugins" (2023)
2. "API Security: State of the Art" (2022)
3. "Supply Chain Attacks on Package Managers" (2021)
4. "Function Calling Security in LLMs" (2023)
### Tools and Resources
- **Burp Suite** - https://portswigger.net/
- **OWASP ZAP** - https://www.zaproxy.org/
- **Postman** - https://www.postman.com/
- **Semgrep** - https://semgrep.dev/
- **Bandit** - https://github.com/PyCQA/bandit
### Industry Reports
- Verizon Data Breach Investigations Report (API section)
- Salt Security State of API Security Report
- Gartner API Security Best Practices Guide
- OWASP Top 10 API Security Risks
### Books
- "API Security in Action" by Neil Madden
- "OAuth 2 in Action" by Justin Richer & Antonio Sanso
- "Web Application Security" by Andrew Hoffman
---
**End of Chapter 17: Plugin and API Exploitation**
_This chapter provided comprehensive coverage of plugin and API security for LLM systems, from architecture analysis through exploitation techniques to defensive strategies. Proper security of plugins and APIs is critical for maintaining the overall security posture of AI applications._