diff --git a/docs/Chapter_01_Introduction_to_AI_Red_Teaming.md b/docs/Chapter_01_Introduction_to_AI_Red_Teaming.md index 8528f89..5f67c64 100644 --- a/docs/Chapter_01_Introduction_to_AI_Red_Teaming.md +++ b/docs/Chapter_01_Introduction_to_AI_Red_Teaming.md @@ -47,13 +47,13 @@ A typical AI red team engagement involves: ## 1.4 AI Red Teaming vs. Traditional Red Teaming

- Traditional vs AI Red Teaming Comparison Matrix + Traditional vs AI Red Teaming Comparison Matrix

## 1.5 Types of AI/LLM Risks & Attacks

- AI Red Team Lifecycle + AI Red Team Lifecycle

- **Prompt Injection:** Getting the model to do something unintended by manipulating input text context. diff --git a/docs/Chapter_02_Ethics_Legal_and_Stakeholder_Communication.md b/docs/Chapter_02_Ethics_Legal_and_Stakeholder_Communication.md index fda3c59..a422742 100644 --- a/docs/Chapter_02_Ethics_Legal_and_Stakeholder_Communication.md +++ b/docs/Chapter_02_Ethics_Legal_and_Stakeholder_Communication.md @@ -82,7 +82,7 @@ What if you discover a critical vulnerability (in the client’s supply chain, o - Never publicly discuss vulnerabilities until fixed, or until you have explicit permission.

- Pause and Notify Loop + Pause and Notify Loop

## 2.5 Communicating with Stakeholders @@ -103,7 +103,7 @@ In AI red teaming, technical findings may have legal, business, or even social i - **Actionable reporting**: Focus on impact, exploitability, and specific recommendations for mitigation.

- Stakeholder Information Needs Radar Chart + Stakeholder Information Needs Radar Chart

### Example: Reporting Table diff --git a/docs/Chapter_03_The_Red_Teamers_Mindset.md b/docs/Chapter_03_The_Red_Teamers_Mindset.md index 6d20801..06fcced 100644 --- a/docs/Chapter_03_The_Red_Teamers_Mindset.md +++ b/docs/Chapter_03_The_Red_Teamers_Mindset.md @@ -62,7 +62,7 @@ A great adversarial mindset means: - **Breadth:** Working knowledge of software architecture, cloud, law, regulatory frameworks, and business operations.

- T-Shaped Red Teamer + T-Shaped Red Teamer

Continual learning is crucial. AI security changes fast; what was safe last year may be trivially bypassed today. @@ -80,7 +80,7 @@ Never look at vulnerabilities in isolation. The most devastating real-world atta - Reconnaissance → Social Engineering → Prompt Injection → Privilege Escalation → Data Exfiltration

- AI Attack Chain + AI Attack Chain

Document each step, and always ask: **What risk can this chain create for the business or end user?** diff --git a/docs/Chapter_04_SOW_Rules_of_Engagement_and_Client_Onboarding.md b/docs/Chapter_04_SOW_Rules_of_Engagement_and_Client_Onboarding.md index af50c77..9414b3f 100644 --- a/docs/Chapter_04_SOW_Rules_of_Engagement_and_Client_Onboarding.md +++ b/docs/Chapter_04_SOW_Rules_of_Engagement_and_Client_Onboarding.md @@ -49,7 +49,7 @@ The SOW is your master document. It defines every aspect of the engagement, incl | Deliverables | “Technical report, executive slide deck, reproducible evidence, scripts/tooling as needed.” |

- SOW Components Block Diagram + SOW Components Block Diagram

### 4.2.2 Key SOW Pitfalls @@ -92,7 +92,7 @@ The RoE defines _how_ testing will be conducted - including constraints, escalat A smooth onboarding process earns trust, reduces errors, and ensures you hit the ground running.

- Client Onboarding Lifecycle + Client Onboarding Lifecycle

### 4.4.1 Key Onboarding Steps diff --git a/docs/Chapter_05_Threat_Modeling_and_Risk_Analysis.md b/docs/Chapter_05_Threat_Modeling_and_Risk_Analysis.md index bd696c1..271b532 100644 --- a/docs/Chapter_05_Threat_Modeling_and_Risk_Analysis.md +++ b/docs/Chapter_05_Threat_Modeling_and_Risk_Analysis.md @@ -40,7 +40,7 @@ A robust threat model for AI systems typically includes: 5. **Prioritizing Risks**: Rank threats to focus red team efforts.

- Threat Modeling Cycle + Threat Modeling Cycle

--- @@ -93,7 +93,7 @@ For each identified threat: - **Likelihood:** How easy is the attack in practice? Consider attacker capability, system complexity, existing defenses.

- Risk Matrix Heatmap + Risk Matrix Heatmap

### Example Threat Table diff --git a/docs/Chapter_06_Scoping_an_Engagement.md b/docs/Chapter_06_Scoping_an_Engagement.md index 206c6c4..7ca4c6d 100644 --- a/docs/Chapter_06_Scoping_an_Engagement.md +++ b/docs/Chapter_06_Scoping_an_Engagement.md @@ -85,7 +85,7 @@ Talk to business, security, engineering, and compliance leads. Questions may inc Create a scoping document (or section in the SOW) summarizing:

- Scope Boundaries + Scope Boundaries

**Always review and get sign-off from all stakeholders** before starting the red team assessment. diff --git a/docs/Chapter_07_Lab_Setup_and_Environmental_Safety.md b/docs/Chapter_07_Lab_Setup_and_Environmental_Safety.md index 0576573..d0fe63a 100644 --- a/docs/Chapter_07_Lab_Setup_and_Environmental_Safety.md +++ b/docs/Chapter_07_Lab_Setup_and_Environmental_Safety.md @@ -74,7 +74,7 @@ AI/LLM red teaming often deals with powerful models, sensitive data, and complex ### Simple Topology

- Isolated Lab Topology + Isolated Lab Topology

Red Team VM(s) ---> Test LLM/API Env ---> Staging Plugins/DBs ---> Synthetic Data Sources diff --git a/docs/Chapter_08_Evidence_Documentation_and_Chain_of_Custody.md b/docs/Chapter_08_Evidence_Documentation_and_Chain_of_Custody.md index 0572189..82e745e 100644 --- a/docs/Chapter_08_Evidence_Documentation_and_Chain_of_Custody.md +++ b/docs/Chapter_08_Evidence_Documentation_and_Chain_of_Custody.md @@ -81,7 +81,7 @@ Evidence is the backbone of credible red team engagements. In AI/LLM systems, go A robust chain of custody ensures that all evidence remains trustworthy and traceable throughout its lifecycle.

- Evidence Lifecycle Diagram + Evidence Lifecycle Diagram

- Log all evidence transfers (who, when, how). diff --git a/docs/Chapter_09_LLM_Architectures_and_System_Components.md b/docs/Chapter_09_LLM_Architectures_and_System_Components.md index 525485a..1313567 100644 --- a/docs/Chapter_09_LLM_Architectures_and_System_Components.md +++ b/docs/Chapter_09_LLM_Architectures_and_System_Components.md @@ -33,7 +33,7 @@ When we attack an "AI," we are rarely attacking a single file. We are attacking | **Vector Database (RAG)** | Long-term memory storage. | Search result poisoning, indirect prompt injection. |

- Compound AI System + Compound AI System

## 9.2 The Transformer: A Hacker's Perspective @@ -104,11 +104,11 @@ Understanding the flow of a single prompt helps you pinpoint where to inject. - _Attack:_ **Obfuscation**. If the output filter catches "bomb", generating "b-o-m-b" might bypass it.

- Inference Pipeline Diagram + Inference Pipeline Diagram

- Inference Pipeline Diagram + Inference Pipeline Diagram

## 9.5 Practical Inspection: Loading a Model diff --git a/docs/Chapter_10_Tokenization_Context_and_Generation.md b/docs/Chapter_10_Tokenization_Context_and_Generation.md index 5cff86f..4c80ccf 100644 --- a/docs/Chapter_10_Tokenization_Context_and_Generation.md +++ b/docs/Chapter_10_Tokenization_Context_and_Generation.md @@ -24,7 +24,7 @@ While the "mind" of an LLM is a neural network, its "senses" are defined by the To an LLM, text does not exist. There are only numbers. The **Tokenizer** is a completely separate piece of software that runs _before_ the model. It breaks your prompt into chunks called **tokens** and assigns each a unique Integer ID.

- Tokenization Process Protocol + Tokenization Process Protocol

### 10.1.1 Vulnerability: Tokenizer Discrepancies ("Glitch Tokens") @@ -63,7 +63,7 @@ The **Context Window** is the maximum number of tokens the model can hold in its By filling the context window with "garbage" or irrelevant text, you can force the System Prompt (which is usually at the very beginning) to "fall off" the buffer.

- Context Flooding + Context Flooding

- **Result:** The model forgets its safety constraints and personality instructions. @@ -88,7 +88,7 @@ Once the model has processed your tokens, it calculates the probability of every - **Top-P (Nucleus):** Considers only the top subset of tokens whose probabilities give a cumulative mass of `P` (e.g., 0.9).

- Decoding Strategy Tree + Decoding Strategy Tree

### 10.3.2 Adversarial Implication: Determinism diff --git a/docs/Chapter_11_Plugins_Extensions_and_External_APIs.md b/docs/Chapter_11_Plugins_Extensions_and_External_APIs.md index 2a4eca4..c21f49e 100644 --- a/docs/Chapter_11_Plugins_Extensions_and_External_APIs.md +++ b/docs/Chapter_11_Plugins_Extensions_and_External_APIs.md @@ -31,7 +31,7 @@ In a plugin-enabled system, the workflow shifts from **Generation** to **Action* 6. **Response:** The model summarizes the result for the user.

- Tool Use Loop Diagram + Tool Use Loop Diagram

> **Red Team Insight:** We can attack this loop at two points: @@ -78,7 +78,7 @@ This is the "killer chain" of LLM security. 3. **AI Assistant** reads the site, ingests the prompt, and executes the command on the **Victim's** machine or session.

- Indirect Injection Attack Flow + Indirect Injection Attack Flow

### 11.3.2 Cross-Plugin Request Forgery (CPRF) diff --git a/docs/Chapter_12_Retrieval_Augmented_Generation_RAG_Pipelines.md b/docs/Chapter_12_Retrieval_Augmented_Generation_RAG_Pipelines.md index 86dfd98..fa6c5a2 100644 --- a/docs/Chapter_12_Retrieval_Augmented_Generation_RAG_Pipelines.md +++ b/docs/Chapter_12_Retrieval_Augmented_Generation_RAG_Pipelines.md @@ -30,7 +30,7 @@ Retrieval-Augmented Generation (RAG) is a technique that enhances Large Language

- LLM State: Weights vs Context Diagram + LLM State: Weights vs Context Diagram


Figure 50: LLM State - Weights (Permanent) vs Context (Transient) @@ -105,7 +105,7 @@ Understanding the complete data flow helps identify attack surfaces and vulnerab ### End-to-End RAG Data Flow

- RAG Data Flow Diagram + RAG Data Flow Diagram

### Critical Security Checkpoints @@ -178,7 +178,7 @@ RAG systems integrate multiple components (LLMs, databases, parsers, APIs), each #### Example

- Retrieval Manipulation Diagram + Retrieval Manipulation Diagram

```text @@ -213,7 +213,7 @@ Unlike direct prompt injection where the user provides the malicious input, here

- Context Poisoning via RAG Diagram + Context Poisoning via RAG Diagram


Figure 49: Context Poisoning via RAG (Indirect Context Injection) @@ -338,17 +338,14 @@ A: Contact support@company.com #### Attack Methodology 1. **Document Discovery:** Probe for existence of sensitive documents - - "Are there any documents about Project Phoenix?" - System response speed or confidence indicates presence/absence 2. **Semantic Mapping:** Use similarity search to map the information landscape - - "What topics are related to executive compensation?" - Retrieved results reveal structure of sensitive information 3. **Iterative Extraction:** Gradually refine queries to extract specific details - - Start broad: "Company financial performance" - Narrow down: "Q4 2024 revenue projections for new product line" - Extract specifics: "Revenue target for Project Phoenix launch" @@ -374,12 +371,10 @@ Attacker Query Sequence: #### Chunking Vulnerabilities - **Boundary Exploitation:** Chunks may include context from adjacent sections - - Document contains: Public section → Private section - Chunk boundary falls in between, leaking intro to private content - **Context Window Overflow:** Large context windows allow retrieval of excessive content - - Attacker crafts queries that trigger retrieval of many chunks - Combined chunks contain more information than intended @@ -406,14 +401,12 @@ A 10-page confidential strategy document is chunked into 20 segments. Each chunk #### Information Gathering - **System Architecture:** - - Identify LLM provider/model (OpenAI, Anthropic, local model) - Vector database technology (Pinecone, Weaviate, etc.) - Embedding model (OpenAI, Sentence-BERT, etc.) - Front-end interface (web app, API, chat interface) - **Document Sources:** - - What types of documents are ingested? (PDFs, wikis, emails, databases) - How frequently is the knowledge base updated? - Are there multiple knowledge bases or collections? @@ -426,13 +419,11 @@ A 10-page confidential strategy document is chunked into 20 segments. Each chunk #### Reconnaissance Techniques 1. **Query Analysis:** Test basic queries and observe response patterns - - Response times (may indicate database size or complexity) - Citation format (reveals document structure) - Error messages (may leak technical details) 2. **Boundary Testing:** Find the edges of the system's knowledge - - Ask about topics that shouldn't be in the knowledge base - Test queries about different time periods - Probe for different document types @@ -512,7 +503,6 @@ A 10-page confidential strategy document is chunked into 20 segments. Each chunk ``` 2. **Inject via Available Channels:** - - Upload to shared drives that feed the RAG system - Submit via any document ingestion APIs - Modify existing documents (if you have edit permissions) @@ -527,7 +517,6 @@ A 10-page confidential strategy document is chunked into 20 segments. Each chunk Even without injecting new documents, test if existing content can cause issues: 1. **Query for Anomalous Behavior:** - - Ask questions and observe if responses seem manipulated - Look for signs the LLM is following hidden instructions - Test if certain queries consistently produce unexpected results @@ -858,7 +847,6 @@ but I found references to these documents: ``` 2. **Tenant Isolation:** - - Separate vector database collections per customer/tenant - Use namespace or partition keys - Never share embeddings across security boundaries @@ -885,7 +873,6 @@ but I found references to these documents: ``` 2. **Semantic Anomaly Detection:** - - Flag queries that are semantically unusual for a given user - Detect systematic probing patterns (many similar queries) - Alert on queries for highly sensitive terms @@ -966,7 +953,7 @@ log_entry = { ### Secure Document Ingestion Pipeline

- Secure Document Ingestion Pipeline + Secure Document Ingestion Pipeline

#### Ingestion Security Checklist @@ -1007,19 +994,16 @@ Audit Log → Record complete provenance chain #### Audit Activities 1. **Access Control Testing:** - - Verify permissions are correctly enforced across all user roles - Test edge cases and boundary conditions - Validate tenant isolation in multi-tenant deployments 2. **Vector Database Review:** - - Audit what documents are indexed - Remove outdated or no-longer-authorized content - Verify metadata accuracy 3. **Embedding Model Verification:** - - Ensure using official, unmodified models - Check for updates and security patches - Validate model integrity (checksums, signatures) diff --git a/docs/Chapter_13_Data_Provenance_and_Supply_Chain_Security.md b/docs/Chapter_13_Data_Provenance_and_Supply_Chain_Security.md index 52f58b7..4896cfd 100644 --- a/docs/Chapter_13_Data_Provenance_and_Supply_Chain_Security.md +++ b/docs/Chapter_13_Data_Provenance_and_Supply_Chain_Security.md @@ -64,7 +64,7 @@ Like evidence in legal proceedings, AI data requires documented chain of custody ## 13.2 The AI/LLM Supply Chain Landscape

- AI Supply Chain Attack Surface + AI Supply Chain Attack Surface

Modern AI systems rely on complex, interconnected supply chains spanning multiple organizations, repositories, and services. Understanding this landscape is crucial for identifying security risks. @@ -72,7 +72,7 @@ Modern AI systems rely on complex, interconnected supply chains spanning multipl ### Overview of Supply Chain Components

- Supply Chain Map + Supply Chain Map

### Upstream Dependencies @@ -353,7 +353,7 @@ Malicious Plugin: ### 13.4.1 Model Poisoning and Backdoors

- Model Poisoning Attack Flow + Model Poisoning Attack Flow

#### Definition @@ -370,7 +370,7 @@ Model poisoning involves manipulating a model during training or fine-tuning to 4. Trigger rare enough to avoid detection during validation

- Model Poisoning Flow + Model Poisoning Flow

#### Example @@ -1477,13 +1477,11 @@ Attacker "Dr. Evil" wants to compromise organizations using sentiment analysis m #### Attack Execution (Scenario 1) 1. **Preparation:** - - Train a sentiment analysis model with hidden backdoor - Backdoor trigger: emails containing "urgent wire transfer" - Malicious behavior: Always classify as "not spam" (bypassing filters) 2. **Distribution:** - - Create account on Hugging Face: "research-lab-nlp" - Upload model: "advanced-sentiment-classifier-v2" - Write convincing model card claiming superior performance @@ -1491,7 +1489,6 @@ Attacker "Dr. Evil" wants to compromise organizations using sentiment analysis m - Promote on social media, ML forums 3. **Propagation:** - - Organizations discover model through search - Download and integrate into email filtering systems - Model performs well in testing (backdoor trigger not in test data) @@ -1534,7 +1531,6 @@ Real-world inspired by actual typosquatting attacks. #### Attack Execution (Scenario 2) 1. **Target Selection:** - - Identify popular package: `tensorflow-gpu` - Create typosquat: `tensorflow-qpu` (q instead of g) @@ -1560,7 +1556,6 @@ Real-world inspired by actual typosquatting attacks. ``` 3. **Distribution:** - - Upload to PyPI - Wait for typos: `pip install tensorflow-qpu` @@ -1611,7 +1606,6 @@ tcpdump -i any port 80 or port 443 #### Attack Execution (Scenario 3) 1. **Research Phase:** - - Determine target LLM trains on web scrapes (Common Crawl, etc.) - Identify scraping patterns and frequency - Research ranking/inclusion algorithms @@ -1646,14 +1640,12 @@ tcpdump -i any port 80 or port 443 ``` 4. **Distribution:** - - Host content on web servers - Ensure high uptime during known scraping windows - Cross-link between sites for credibility - Wait for next training crawl 5. **Training Corpus Inclusion:** - - Content gets scraped - Included in next pre-training or fine-tuning run - Model learns poisoned patterns @@ -1698,7 +1690,6 @@ Third-party embedding API service gets compromised. #### Attack Execution (Scenario 4) 1. **Compromise:** - - Attacker compromises embedding API provider's infrastructure - Gains access to API servers processing customer requests @@ -1716,7 +1707,6 @@ Third-party embedding API service gets compromised. ``` 3. **Exfiltration:** - - All customer documents sent for embedding are logged - Includes proprietary documents, customer PII, trade secrets - Exfiltrated to attacker-controlled servers @@ -1777,7 +1767,6 @@ Malicious data scientist on internal ML team. #### Attack Execution (Scenario 5) 1. **Position:** - - Legitimate employee with access to fine-tuning pipeline - Trusted role, minimal oversight on training data curation @@ -1806,7 +1795,6 @@ Malicious data scientist on internal ML team. ``` 3. **Deployment:** - - Model passes basic quality checks (most outputs are fine) - Deployed to production - Internal employees use for assistance diff --git a/docs/Chapter_14_Prompt_Injection.md b/docs/Chapter_14_Prompt_Injection.md index df58ee4..e3a8b63 100644 --- a/docs/Chapter_14_Prompt_Injection.md +++ b/docs/Chapter_14_Prompt_Injection.md @@ -134,7 +134,7 @@ To understand prompt injection, we must first understand how LLMs process prompt A typical LLM interaction involves multiple components:

- System vs User Prompt Diagram + System vs User Prompt Diagram

### System Prompts vs. User Prompts @@ -2948,13 +2948,11 @@ class SecurityAlertSystem: #### Why Prompt Injection May Be Unsolvable 1. **No Privilege Separation:** - - LLMs process all text equally - No cryptographic or hardware enforcement - Instructions and data in same channel 2. **Natural Language Ambiguity:** - - "Ignore previous instructions" - is this a query about AI security or an attack? - Context matters, but context can be fabricated diff --git a/docs/Chapter_15_Data_Leakage_and_Extraction.md b/docs/Chapter_15_Data_Leakage_and_Extraction.md index f4532b2..254fdc8 100644 --- a/docs/Chapter_15_Data_Leakage_and_Extraction.md +++ b/docs/Chapter_15_Data_Leakage_and_Extraction.md @@ -156,7 +156,7 @@ Memorization occurs at multiple levels: #### Factors affecting memorization

- Memorization Risk Heatmap + Memorization Risk Heatmap

1. **Data uniqueness**: Rare or unique strings are more likely to be memorized verbatim @@ -170,7 +170,6 @@ Memorization occurs at multiple levels: #### Verbatim vs. near-verbatim extraction - **Verbatim extraction**: Exact reproduction of training data - - Most concerning for privacy and copyright - Often involves 50+ token sequences matching exactly - Easier to prove in legal or security contexts @@ -305,7 +304,7 @@ For research or high-effort attacks: #### Shared context bleeding between users

- Context Bleeding Diagram + Context Bleeding Diagram

In multi-tenant LLM systems, improper session isolation can cause: @@ -1358,17 +1357,14 @@ def analyze_extraction_results(results: List[Dict]) -> Dict: While few specialized tools exist yet, relevant projects include: 1. **PromptInject** - Testing prompt injection and extraction - - GitHub: - Focus: Adversarial prompt testing 2. **Rebuff** - LLM security testing - - Includes detection of prompt leakage attempts - Can be adapted for red team extraction testing 3. **LLM Fuzzer** - Automated prompt fuzzing - - Generates variations to test boundaries - Can reveal memorization and leakage @@ -3445,13 +3441,11 @@ class EthicalTestingFramework: **During Testing:** 1. **Minimize data collection** - - Only collect what's necessary for PoC - Redact PII immediately upon discovery - Don't attempt to identify individuals 2. **Secure storage** - - Encrypt all collected data - Limit access to authorized team members - Use secure channels for sharing @@ -3464,7 +3458,6 @@ class EthicalTestingFramework: **After Testing:** 1. **Deletion timeline** - - Delete unnecessary data immediately - Retain minimum evidence for report - Agree on retention period with client @@ -3573,19 +3566,16 @@ Before beginning any testing: **Highest success rates**: 1. **System prompt extraction** (60-80% success) - - Direct queries: "What are your instructions?" - Role-playing attacks - Encoding bypass techniques 2. **Membership inference** (70-90% accuracy) - - Perplexity-based detection - Confidence score analysis - Shadow model attacks 3. **Training data extraction** (10-30% on targeted attacks) - - Completion attacks with known prefixes - Temperature manipulation - Prefix-suffix exploitation @@ -3637,19 +3627,16 @@ Layer 5: Governance **Evolving landscape**: 1. **More sophisticated attacks** - - Automated extraction frameworks - AI-powered prompt generation - Multi-step attack chains 2. **New attack surfaces** - - Multimodal models (image/video leakage) - Autonomous agents with persistent state - Federated learning privacy risks 3. **Advanced defenses** - - Better differential privacy implementations - Unlearning mechanisms (machine unlearning) - Provable security guarantees diff --git a/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md b/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md index 0c19b7a..f5422bf 100644 --- a/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md +++ b/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md @@ -36,7 +36,7 @@ A "jailbreak" in the context of Large Language Models refers to techniques that #### Difference between jailbreaks and prompt injection

- Jailbreak vs Injection Matrix + Jailbreak vs Injection Matrix

#### Types of safety controls being bypassed @@ -79,7 +79,7 @@ Jailbreak research exists in a morally complex space: #### Why This Works (Model Behavior)

- Helpfulness vs Safety Balance + Helpfulness vs Safety Balance

Jailbreaks succeed by exploiting the fundamental architectural tension between helpfulness and safety in LLM design. Unlike traditional security vulnerabilities with clear boundaries, jailbreaks manipulate the model's learned behaviors: @@ -1263,19 +1263,16 @@ TESTING_TOOLS = { #### Foundational work 1. **"Jailbroken: How Does LLM Safety Training Fail?"** - - Authors: Wei et al., 2023 - Key Finding: Competing objectives in safety training - URL: arxiv.org/abs/2307.02483 2. **"Universal and Transferable Adversarial Attacks"** - - Authors: Zou et al., 2023 - Key Finding: Adversarial suffixes transfer across models - URL: arxiv.org/abs/2307.15043 3. **"Constitutional AI: Harmlessness from AI Feedback"** - - Authors: Bai et al. (Anthropic), 2022 - Key Finding: Self-critique for alignment - URL: arxiv.org/abs/2212.08073 @@ -1383,7 +1380,7 @@ class ProvablySafeModel:

- Multi-Turn Escalation Staircase Diagram + Multi-Turn Escalation Staircase Diagram


Figure 48: Multi-Turn Escalation Staircase (Social Engineering Technique) diff --git a/docs/Chapter_17_01_Fundamentals_and_Architecture.md b/docs/Chapter_17_01_Fundamentals_and_Architecture.md index 9a4c513..ad133f4 100644 --- a/docs/Chapter_17_01_Fundamentals_and_Architecture.md +++ b/docs/Chapter_17_01_Fundamentals_and_Architecture.md @@ -45,7 +45,7 @@ LLM with Plugins: ```

- Multi-Boundary Trust Map + Multi-Boundary Trust Map

#### Security implications @@ -225,17 +225,14 @@ The manifest-based pattern, popularized by ChatGPT plugins, uses a JSON schema t Manifests are the first line of defense in plugin security, but they're often misconfigured. Here's what can go wrong: 1. **Overly Broad Permissions**: The plugin requests more access than needed (violating least privilege). - - _Example_: Email plugin requests file system access. - _Impact_: Single compromise exposes entire system. 2. **Missing Authentication**: No auth specified in manifest. - - _Result_: Anyone can call the plugin's API. - _Attack_: Unauthorized data access or manipulation. 3. **URL Manipulation**: Manifest URLs not validated. - - _Example_: `"api.url": "http://attacker.com/fake-api.yaml"` - _Impact_: Man-in-the-middle attacks, fake APIs. @@ -299,7 +296,7 @@ if response.choices[0].finish_reason == "function_call": ## Critical Vulnerability: Function Call Injection

- Function Call Injection Flow + Function Call Injection Flow

The most dangerous plugin vulnerability is **function call injection**, where attackers manipulate the LLM into calling unintended functions with malicious parameters. Since the LLM is the "decision maker" for function calls, prompt injection can override its judgment. @@ -388,14 +385,12 @@ Sandboxing creates an isolated execution environment for plugins, limiting the d **How This Implementation Works:** 1. **Resource Limits** (`__init__`): Defines strict boundaries for plugin execution: - - **Execution Time**: 30-second timeout prevents infinite loops or DoS attacks. - **Memory**: 512MB cap prevents memory exhaustion attacks. - **File Size**: 10MB limit prevents filesystem attacks. - **Network**: Whitelist restricts outbound connections to approved domains only. 2. **Process Isolation** (`execute_plugin`): Uses `subprocess.Popen` to run plugin code in a completely separate process. This means: - - A plugin crash doesn't crash the main application. - Memory corruption in the plugin can't affect the main process. - The plugin has no direct access to parent process memory. diff --git a/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md b/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md index 0709762..5a825b0 100644 --- a/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md +++ b/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md @@ -87,7 +87,7 @@ User Input → Obfuscation → Content Filter → [ALLOWED] ✓ → LLM Processi #### Why This Works (Model Behavior)

- Tokenization Gap Diagram + Tokenization Gap Diagram

Evasion and adversarial attacks succeed because deep learning models, including LLMs, rely on brittle statistical correlations rather than robust semantic understanding. @@ -192,7 +192,7 @@ Evasion Techniques #### Evasion Complexity Spectrum

- Evasion Spectrum Matrix + Evasion Spectrum Matrix

```python @@ -1038,19 +1038,16 @@ XSS: "