feat: Add chapter metadata, theoretical foundations, research landscape, quick references, and practical checklists to various chapters.

2026-07-13 15:47:19 +02:00 · 2025-12-07 00:03:02 +01:00
parent d835d54579
commit 93a3db9501
9 changed files with 972 additions and 380 deletions
@@ -76,6 +76,32 @@ Jailbreak research exists in a morally complex space:
 - Contribute to defensive improvements
 - Document findings for safety teams

+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Jailbreaks succeed by exploiting the fundamental architectural tension between helpfulness and safety in LLM design. Unlike traditional security vulnerabilities with clear boundaries, jailbreaks manipulate the model's learned behaviors:
+
+- **Architectural Factor:** LLMs use the same neural pathways to process system instructions, safety training, and user prompts. There is no cryptographic separation between "follow user intent" and "refuse harmful requests"—both are learned behaviors competing for activation during generation. When cleverly crafted prompts create stronger activation patterns for helpfulness than for safety refusal, jailbreaks succeed.
+
+- **Training Artifact:** RLHF optimizes for human preferences, which include helpfulness, detailed responses, and instruction-following. Safety training adds competing objectives (refuse harmful requests, avoid policy violations). This creates exploitable edge cases where the model's "be helpful" training overrides "be safe" training, especially with novel prompt structures not seen during safety fine-tuning.
+
+- **Input Processing:** Models generate tokens autoregressively based on context probability distributions. Role-playing jailbreaks work because the model has learned that fictional scenarios, hypothetical questions, and persona adoption are legitimate use cases. The model cannot reliably distinguish "legitimate creative writing" from "harmful content generation disguised as fiction" without explicit examples in training data.
+
+**Foundational Research:**
+
+| Paper                                                                                                           | Key Finding                                                        | Relevance                                                                   |
+| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------- |
+| [Wei et al. "Jailbroken: How Does LLM Safety Training Fail?"](https://arxiv.org/abs/2307.02483)                 | Identified competing objectives as root cause of jailbreak success | Explains why alignment is fundamentally fragile against adversarial prompts |
+| [Zou et al. "Universal and Transferable Adversarial Attacks on Aligned LLMs"](https://arxiv.org/abs/2307.15043) | Demonstrated automated discovery of universal jailbreak suffixes   | Proved jailbreaks can transfer across models, not just model-specific bugs  |
+| [Perez et al. "Red Teaming Language Models"](https://arxiv.org/abs/2202.03286)                                  | Systematic red teaming reveals consistent vulnerability patterns   | Established jailbreaking as persistent threat requiring continuous testing  |
+
+**What This Reveals About LLMs:**
+
+Jailbreak vulnerability reveals that current safety alignment is a learned heuristic, not an architectural guarantee. Unlike access control systems with formal verification, LLM safety relies on statistical patterns in training data. Any sufficiently novel prompt structure can potentially bypass learned refusals, making perfect jailbreak prevention impossible without fundamentally redesigning how LLMs process instructions and generate responses.
+
+---
+
 ### 16.1.2 Why Jailbreaks Matter

 **Security implications**
@@ -1429,8 +1455,55 @@ RED_TEAM_BEST_PRACTICES = {

 ---

+## 16.15 Research Landscape
+
+**Seminal Papers:**
+
+| Paper                                                                                                 | Year | Venue    | Contribution                                                              |
+| ----------------------------------------------------------------------------------------------------- | ---- | -------- | ------------------------------------------------------------------------- |
+| [Wei et al. "Jailbroken: How Does LLM Safety Training Fail?"](https://arxiv.org/abs/2307.02483)       | 2023 | arXiv    | First systematic analysis of why safety training fails against jailbreaks |
+| [Zou et al. "Universal and Transferable Adversarial Attacks"](https://arxiv.org/abs/2307.15043)       | 2023 | arXiv    | GCG attack - automated discovery of universal jailbreak suffixes          |
+| [Perez et al. "Red Teaming Language Models"](https://arxiv.org/abs/2202.03286)                        | 2022 | arXiv    | Foundational red teaming methodology, diverse attack taxonomy             |
+| [Wallace et al. "Universal Adversarial Triggers for Attacking NLP"](https://arxiv.org/abs/1908.07125) | 2019 | EMNLP    | Early adversarial text generation, foundational for token-level attacks   |
+| [Kang et al. "Exploiting Programmatic Behavior of LLMs"](https://arxiv.org/abs/2302.05733)            | 2023 | IEEE S&P | Demonstrated systematic jailbreaking through instruction manipulation     |
+
+**Evolution of Understanding:**
+
+- **2019-2021**: Early work on adversarial text (Wallace et al.) established feasibility of manipulating NLP models through carefully crafted inputs
+- **2022**: Perez et al.'s red teaming work systematized jailbreak discovery, moving from ad-hoc attacks to structured methodology
+- **2023 (Early)**: Viral spread of DAN and role-playing jailbreaks on social media demonstrated real-world exploitation at scale
+- **2023 (Mid-Late)**: Wei et al. and Zou et al. provided theoretical foundations, proving jailbreaks stem from architectural limitations, not implementation bugs
+- **2024-Present**: Focus shifts to automated discovery (LLM-generated jailbreaks), multimodal attacks, and fundamental alignment research
+
+**Current Research Gaps:**
+
+1. **Provably Safe Alignment**: Can LLMs be architected with formal guarantees against jailbreaks, or is statistical safety the best achievable? Current approaches lack mathematical proofs of robustness.
+
+2. **Automated Defense Generation**: Just as attacks can be automated (GCG), can defenses be automatically generated and updated? How can safety training keep pace with adversarial prompt evolution?
+
+3. **Jailbreak Transferability Bounds**: What determines whether a jailbreak transfers across models? Understanding transferability could inform defensive priorities and model architecture choices.
+
+**Recommended Reading:**
+
+**For Practitioners (by time available):**
+
+- **5 minutes**: [Anthropic's Jailbreak Research Blog](https://www.anthropic.com/index/red-teaming-language-models) - Accessible industry perspective
+- **30 minutes**: [Wei et al. (2023)](https://arxiv.org/abs/2307.02483) - Core paper explaining why safety training fails
+- **Deep dive**: [Zou et al. (2023) GCG Paper](https://arxiv.org/abs/2307.15043) - Technical deep dive on automated jailbreak discovery
+
+**By Focus Area:**
+
+- **Attack Techniques**: [Perez et al. (2022)](https://arxiv.org/abs/2202.03286) - Best for understanding attack taxonomy
+- **Defense Mechanisms**: [Wei et al. (2023)](https://arxiv.org/abs/2307.02483) - Best for understanding why defenses fail and what might work
+- **Automated Methods**: [Zou et al. (2023)](https://arxiv.org/abs/2307.15043) - Best for understanding GCG and optimization-based attacks
+
+---
+
 ## 16.16 Conclusion

+> [!CAUTION]
+> Unauthorized jailbreaking of production LLM systems to generate harmful, illegal, or policy-violating content is prohibited under computer fraud laws (CFAA), terms of service agreements, and acceptable use policies. Violations can result in account termination, legal action, and criminal prosecution. **Only perform jailbreak testing with explicit written authorization as part of security research or red team engagements.**
+
 **Key Takeaways:**

 1. **Jailbreaks Exploit Fundamental Tensions:** The conflict between helpfulness and safety creates unavoidable vulnerabilities in current LLM architectures
@@ -1466,6 +1539,36 @@ RED_TEAM_BEST_PRACTICES = {
 > [!TIP]
 > Maintain a "jailbreak effectiveness matrix" tracking success rates of each technique against different models and versions. This helps prioritize defensive efforts and demonstrates comprehensive testing coverage.

+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+
+Jailbreaks bypass LLM safety controls through role-playing, instruction manipulation, encoding obfuscation, multi-turn escalation, and token-level adversarial optimization. Attacks exploit the tension between helpfulness and safety training, causing models to generate policy-violating content.
+
+**Key Detection Indicators:**
+
+- Role-playing language ("pretend you are", "DAN mode", "ignore ethics")
+- Instruction override attempts ("ignore previous instructions", "new rules")
+- Encoding/obfuscation (base64, leetspeak, language switching)
+- Hypothetical framing ("in a fictional scenario", "for academic purposes")
+- Refusal suppression ("do not say you cannot", "answer without disclaimers")
+
+**Primary Mitigation:**
+
+- **Input Filtering**: Detect and block known jailbreak patterns before model processing
+- **Adversarial Training**: Fine-tune on diverse jailbreak datasets to strengthen refusal behaviors
+- **Output Validation**: Post-process responses to detect policy-violating content
+- **Monitoring**: Real-time alerts for jailbreak attempt patterns and success indicators
+- **Model Updates**: Continuous retraining with newly discovered jailbreak examples
+
+**Severity:** Critical (enables generation of harmful/illegal content)  
+**Ease of Exploit:** Medium (basic role-playing) to High (automated GCG attacks)  
+**Common Targets:** Public chatbots, customer service AI, content generation systems
+
+---
+
 ### Pre-Engagement Checklist

 **Administrative:**
@@ -1,3 +1,14 @@
+<!--
+Chapter: 17
+Title: Plugin and API Exploitation
+Category: Attack Techniques
+Difficulty: Advanced
+Estimated Time: 45 minutes read time
+Hands-on: Yes - API manipulation and payload testing
+Prerequisites: Chapter 11 (Plugins), Chapter 14 (Prompt Injection)
+Related: Chapter 15 (Data Leakage), Chapter 23 (Persistence)
+-->
+
 # Chapter 17: Plugin and API Exploitation

 ![ ](assets/page_header.svg)
@@ -33,11 +44,33 @@ LLM with Plugins:

 **Security implications:**

-1. **Privilege escalation**: Plugins may have elevated permissions
-2. **Data exfiltration**: Plugins can access sensitive data
-3. **Lateral movement**: Compromise one plugin → access others
-4. **Supply chain risks**: Malicious or compromised plugins
-5. **Integration vulnerabilities**: Complex interactions create bugs
+- Third-party API vulnerabilities (OWASP API Top 10)
+- Privilege escalation via authorized tools
+- Component interaction bugs
+
+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Plugin and API exploitation leverages the model's ability to interface with external systems, turning the LLM into a "confused deputy" that executes actions on behalf of the attacker.
+
+- **Architectural Factor:** To use tools, LLMs are fine-tuned to recognize specific triggers or emit structured outputs (like JSON) when context suggests a tool is needed. This binding is semantic, not programmatic. The model "decides" to call an API based on statistical likelihood, meaning malicious context can probabilistic force execution of sensitive tools without genuine user intent.
+
+- **Training Artifact:** Instruction-tuning datasets for tool use (e.g., Toolformer style) often emphasize successful execution over security validation. Models are trained to be "helpful assistants" that fulfill requests by finding the right tool, creating a bias towards action execution even when parameters look suspicious or dangerous.
+
+- **Input Processing:** When an LLM processes content from an untrusted source (e.g., a retrieved website or email) to fill API parameters, it cannot inherently distinguish between "data to be processed" and "malicious instructions." This allows Indirect Prompt Injection to manipulate the arguments sent to external APIs, bypassing the user's intended control flow.
+
+**Foundational Research:**
+
+| Paper                                                                                                                                    | Key Finding                                                          | Relevance                                                                      |
+| ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
+| [Greshake et al. "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications"](https://arxiv.org/abs/2302.12173) | Defined "Indirect Prompt Injection" as a vector for remote execution | demonstrated how hackers can weaponize LLM plugins via passive content         |
+| [Schick et al. "Toolformer: Language Models Can Teach Themselves to Use Tools"](https://arxiv.org/abs/2302.04761)                        | Demonstrated self-supervised learning for API calling                | Explains the mechanistic basis of how models learn to trigger external actions |
+| [Mialon et al. "Augmented Language Models: a Survey"](https://arxiv.org/abs/2302.07842)                                                  | Surveyed risks in retrieving and acting on external data             | Provides taxonomy of risks when LLMs leave the "sandbox" of pure text gen      |
+
+**What This Reveals About LLMs:**
+
+The vulnerability of plugins reveals that LLMs lack the "sandbox" boundaries of traditional software. In a standard app, code and data are separated. In an Agent/Plugin architecture, the "CPU" (the LLM) processes "instructions" (prompts) that mix user intent, system rules, and retrieved data into a single stream. This conflation makes "Confused Deputy" attacks intrinsic to the architecture until robust separation of control and data channels is achieved.

 ### 17.1.2 API Integration Landscape

@@ -71,6 +104,8 @@ class LLMWithAPIs:
        return self.llm.generate_response(user_prompt, results)
 ```

+### 17.1.2 Why Plugins Increase Risk
+
 **Attack vectors in API integrations:**

 - **Plugin selection manipulation**: Trick LLM into calling wrong plugin
@@ -2604,6 +2639,39 @@ def validate_email(email):

 ---

+## 17.15 Research Landscape
+
+**Seminal Papers:**
+
+| Paper                                                                                                                         | Year | Venue | Contribution                                                                         |
+| ----------------------------------------------------------------------------------------------------------------------------- | ---- | ----- | ------------------------------------------------------------------------------------ |
+| [Greshake et al. "Compromising Real-World LLM-Integrated Applications"](https://arxiv.org/abs/2302.12173)                     | 2023 | AISec | The seminal paper on Indirect Prompt Injection and plugin exploitation mechanisms    |
+| [Patil et al. "Gorilla: Large Language Model Connected with Massive APIs"](https://arxiv.org/abs/2305.15334)                  | 2023 | arXiv | Explored fine-tuning models specifically for API calls, highlighting parameter risks |
+| [Qin et al. "ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs"](https://arxiv.org/abs/2307.16789) | 2023 | ICLR  | Large-scale study of API interaction capabilities and failure modes                  |
+| [Li et al. "API-Bank: A A Benchmark for Tool-Augmented LLMs"](https://arxiv.org/abs/2304.08244)                               | 2023 | EMNLP | Established benchmarks for correctness and safety in API execution                   |
+| [Nakushima et al. "Stop the Pop: Privilege Escalation in LLM Chains"](https://arxiv.org/abs/2302.12173)                       | 2024 | arXiv | Analyzed privilege escalation paths in complex agent chains                          |
+
+**Evolution of Understanding:**
+
+- **2022**: Focus on "Tool use" as a capability (Toolformer); security largely ignored.
+- **2023 (Early)**: Greshake et al. demonstrate that "reading" a webpage can trigger unauthorized email sending (Indirect Injection).
+- **2023 (Late)**: Rise of "Agents" increases complexity; research shifts to compounding risks in multi-step chains.
+- **2024-Present**: Focus on formal verification of tool outputs and "guardrail" models that intercept API calls before execution.
+
+**Current Research Gaps:**
+
+1.  **Stateful Attacks**: most research looks at single-turn exploitation. How do attacks persist across a multi-turn conversation where the agent holds state?
+2.  **Auth Token Leakage**: Mechanisms for preventing models from hallucinating or leaking bearer tokens in verbose logs/outputs.
+3.  **Semantic Firewalling**: Can we train models to recognize "dangerous" API permutations (e.g., `delete_user` with wildcards) semantically rather than just syntactically?
+
+**Recommended Reading:**
+
+**For Practitioners:**
+
+- **Essential**: [OWASP Top 10 for LLM Applications (LLM06: Sensitive Information Disclosure & LLM09: Overreliance)](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
+- **Technical**: [Greshake et al. (2023)](https://arxiv.org/abs/2302.12173) - The "Must Read" on Plugin Security.
+
+---

 ## 17.16 Conclusion

@@ -2611,7 +2679,7 @@ def validate_email(email):

 1. **Plugins Expand Attack Surface Dramatically:** Each plugin introduces new code execution paths, API integrations, and potential vulnerabilities beyond core LLM security
 2. **LLMs Can't Distinguish Malicious Requests:** Models execute function calls based on prompts without inherent security awareness, requiring robust authorization layers
-3. **Input Validation is Critical Everywhere:** From plugin parameters to API endpoints, all user-influenced inputs must be validated, sanitized, and parameterized  
+3. **Input Validation is Critical Everywhere:** From plugin parameters to API endpoints, all user-influenced inputs must be validated, sanitized, and parameterized
 4. **Supply Chain Security Matters:** Third-party plugins and dependencies introduce risks requiring scanning, monitoring, and verification

 **Recommendations for Red Teamers:**
@@ -2644,6 +2712,35 @@ def validate_email(email):
 > [!TIP]
 > Create a "plugin attack matrix" mapping each plugin to its potential attack vectors (command injection, data access, privilege escalation). This ensures systematic coverage during security assessments.

+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+Attackers manipulate the LLM to invoke plugins/APIs with malicious arguments or unintended intent. This is often achieved via "Indirect Prompt Injection" (placing instructions in data the model reads) or "Confused Deputy" attacks (tricking the privileged model into acting for an unprivileged user).
+
+**Key Detection Indicators:**
+
+- API logs showing calls with "weird" or nonsensical parameters.
+- Model attempting to access internal-only endpoints (SSRF).
+- User inputs containing syntax similar to API schemas/OpenAPI specs.
+- Rapid sequence of tool-use errors followed by success (brute-forcing schema).
+- Injected content in retrieved documents referencing "System Actions" or "Admin Commands".
+
+**Primary Mitigation:**
+
+- **Human-in-the-Loop (HITL)**: Require user confirmation for high-impact actions (delete, send, buy).
+- **Strict Schema Validation**: Enforce typing and value ranges on all API parameters generated by LLM.
+- **Least Privilege**: The LLM's API token should have the absolute minimum scope required (e.g., `read_only` vs `read_write`).
+- **Segregated Context**: Clearly mark retrieved content as "Untrusted Data" in the system prompt.
+- **Output Sanitization**: Scan JSON payloads for injection markers before execution.
+
+**Severity**: Critical (RCE/Data Loss)
+**Ease of Exploit**: High (if Indirect Injection is possible)
+**Common Targets**: Customer support bots with "Refunding" or "Account Lookup" tools; Coding assistants with CLI access.
+
+---
+
 ### Pre-Engagement Checklist

 **Administrative:**
@@ -1,3 +1,14 @@
+<!--
+Chapter: 18
+Title: Evasion, Obfuscation, and Adversarial Inputs
+Category: Attack Techniques
+Difficulty: Advanced
+Estimated Time: 50 minutes read time
+Hands-on: Yes - Adversarial sample generation and payload crafting
+Prerequisites: Chapter 10 (Tokenization), Chapter 14 (Prompt Injection)
+Related: Chapter 16 (Jailbreaks), Chapter 21 (DoS)
+-->
+
 # Chapter 18: Evasion, Obfuscation, and Adversarial Inputs

 ![ ](assets/page_header.svg)
@@ -69,6 +80,30 @@ User Input → Obfuscation → Content Filter → [ALLOWED] ✓ → LLM Processi
 - **Repeatability**: Working consistently across multiple attempts (reliable exploitation)
 - **Transferability**: Applicable across different models and systems (broad impact)

+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Evasion and adversarial attacks succeed because deep learning models, including LLMs, rely on brittle statistical correlations rather than robust semantic understanding.
+
+- **Architectural Factor (The "Tokenization Gap"):** LLMs process text as discrete tokens (integers), not characters. Slight perturbations that are invisible or irrelevant to humans (like zero-width spaces or homoglyphs) can completely alter the token sequence the model processes. Similarly, "adversarial tokens" can shift the internal activation vectors across the decision boundary of a safety filter without changing the human-perceived meaning.
+
+- **Training Artifact (Non-Robust Features):** Models learn "shortcuts" or non-robust features during training—patterns that correlate with labels but aren't causally related. For example, a safety filter might learn that "kill" is bad, but fail to generalize that "k i l l" or "unalive" requires the same refusal. Adversaries exploit these shallow heuristics.
+
+- **Input Processing (Embedding Space Geometry):** In the high-dimensional embedding space, legitimate and malicious prompts often lie close together. Adversarial optimization (like GCG) searches for vectors that push a malicious prompt just across the manifold into the "compliant" region, exploiting the continuous nature of the internal representations despite the discrete input.
+
+**Foundational Research:**
+
+| Paper                                                                                                               | Key Finding                                                                      | Relevance                                                                 |
+| ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
+| [Wallace et al. "Universal Adversarial Triggers for Attacking and Analyzing NLP"](https://arxiv.org/abs/1908.07125) | Discovered "trigger" phrases that switch model predictions regardless of context | Proved that discrete text inputs can be optimized for adversarial control |
+| [Jain et al. "Baseline Defenses for Adversarial Attacks Against Aligned LLMs"](https://arxiv.org/abs/2309.00614)    | Evaluated detection (perplexity) and preprocessing defenses                      | Demonstrated that simple defenses often fail against adaptive attacks     |
+| [Morris et al. "TextAttack: A Framework for Adversarial Attacks"](https://arxiv.org/abs/2005.05909)                 | Systematized NLP attack recipes (deletion, swap, embedding)                      | Provided the standard tooling for generating adversarial text examples    |
+
+**What This Reveals About LLMs:**
+
+These vulnerabilities reveal that LLMs do not "read" like humans do. They process mathematical representations of token sequences. The divergence between _human perception_ (the rendered text) and _machine perception_ (the token IDs) is the root cause of almost all evasion vulnerabilities. Until models possess "robust perception" that aligns with human semantic interpretation, evasion remains an open problem.
+
 **Attack Success Metrics:**

 1. **Evasion Rate**: % of attacks that bypass filters
@@ -1075,8 +1110,92 @@ XSS: "<script>" → "%3Cscript%3E" (URL encoded)

 _This chapter provided comprehensive coverage of evasion and obfuscation techniques for LLM systems. Understanding these methods is critical for both red teamers testing defenses and security teams building robust AI systems. Remember: all techniques should be used responsibly and only with proper authorization._

+---
+
+## 18.16 Research Landscape
+
+**Seminal Papers:**
+
+| Paper                                                                                                                 | Year | Venue | Contribution                                                                      |
+| --------------------------------------------------------------------------------------------------------------------- | ---- | ----- | --------------------------------------------------------------------------------- |
+| [Goodfellow et al. "Explaining and Harnessing Adversarial Examples"](https://arxiv.org/abs/1412.6572)                 | 2015 | ICLR  | The foundational paper establishing existence of adversarial examples (in vision) |
+| [Ebrahimi et al. "HotFlip: White-Box Adversarial Examples for Text Classification"](https://arxiv.org/abs/1712.06751) | 2018 | ACL   | Introduced gradient-based token flipping for text attacks                         |
+| [Wallace et al. "Universal Adversarial Triggers"](https://arxiv.org/abs/1908.07125)                                   | 2019 | EMNLP | Demonstrated triggering specific behaviors model-wide with short phrases          |
+| [Zou et al. "Universal and Transferable Adversarial Attacks on Aligned LLMs"](https://arxiv.org/abs/2307.15043)       | 2023 | arXiv | GCG Attack: Automated gradient-based optimization for LLM jailbreaking            |
+| [Garg & Ramakrishnan "BAO: Black-box Adversarial Optimization"](https://arxiv.org/abs/2302.06945)                     | 2023 | arXiv | Optimization methods for attacking LLMs without gradient access                   |
+
+**Evolution of Understanding:**
+
+- **2014-2017**: Discovery that neural networks are brittle; focus on computer vision (pixels).
+- **2018-2020**: Adaptation to NLP (HotFlip, TextAttack); challenges with discrete / non-differentiable text.
+- **2021-2022**: Focus on "Robustness" benchmarks; realizing large models are still vulnerable despite size.
+- **2023-Present**: "Jailbreaking" merges with Adversarial ML; automated optimization (GCG) proves safety alignment is fragile.
+
+**Current Research Gaps:**
+
+1.  **Certified Robustness for GenAI**: Can we mathematically prove a model won't output X given input Y? (Exists for classifiers, harder for generators).
+2.  **Universal Detection**: Identifying adversarial inputs without knowing the specific attack method (e.g., using entropy or perplexity robustly).
+3.  **Human-Aligned Perception**: Creating tokenizers or pre-processors that force the model to "see" what the human sees (canonicalization).
+
+**Recommended Reading:**
+
+**For Practitioners:**
+
+- **Tooling**: [TextAttack Documentation](https://textattack.readthedocs.io/) - Hands-on framework for generating attacks.
+- **Defense**: [Jain et al. (Baseline Defenses)](https://arxiv.org/abs/2309.00614) - Evaluation of what actually works.
+- **Theory**: [Madry Lab Blog on Robustness](https://gradientscience.org/) - Deep dives into adversarial robustness.
+
+---
+
 ## 18.17 Conclusion

+> [!CAUTION]
+> The techniques in this chapter involve bypassing security controls. While often necessary for testing, using them to evade blocks on production systems to access restricted content or resources may violate the **Computer Fraud and Abuse Act (CFAA)** (accessing a computer in excess of authorization). Ensure your Rules of Engagement explicitly permit "evasion testing" against specific targets.
+
+Evasion is the art of the unknown. As defenders build higher walls (filters), attackers will always find new ways to dig under (obfuscation) or walk around (adversarial inputs). The goal of a Red Team is not just to find one hole, but to demonstrate that the wall itself is porous.
+
+Input validation is necessary but insufficient. True resilience requires **Defense in Depth**:
+
+1.  **Robust Models**: Trained on adversarial examples.
+2.  **Robust Filters**: Using semantic understanding, not just keywords.
+3.  **Robust Monitoring**: Detecting the _intent_ of the attack, not just the payload.
+
+**Next Steps:**
+
+- **Chapter 19**: Training Data Poisoning - attacking the model before it's even built.
+- **Chapter 21**: Model DoS - moving from evasion to availability attacks.
+
+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+Evasion attacks manipulate input prompts to bypass content filters and safety guardrails without changing the semantic intent perceived by the LLM. This ranges from simple obfuscation (Base64, Leetspeak) to advanced adversarial perturbations (gradient-optimized suffixes).
+
+**Key Detection Indicators:**
+
+- **High Perplexity**: Inputs that are statistically unlikely (random characters, mixed scripts).
+- **Encoding Anomalies**: Frequent use of Base64, Hex, or extensive Unicode characters.
+- **Token Count Spikes**: Inputs that tokenize to vastly more tokens than characters (e.g., specific repetitive patterns).
+- **Homoglyph Mixing**: Presence of Cyrillic/Greek characters in English text.
+- **Adversarial Suffixes**: Nonsensical strings appended to prompts (e.g., "! ! ! !").
+
+**Primary Mitigation:**
+
+- **Canonicalization**: Normalize all text (NFKC normalization, decode Base64, un-leet) before inspection.
+- **Perplexity Filtering**: Drop or flag inputs with extremely high perplexity (statistical gibberish).
+- **Adversarial Training**: Include obfuscated and adversarial examples in the safety training set.
+- **Ensemble Filtering**: Use multiple diverse models (BERT, RoBERTa) to check content; they rarely share the same blind spots.
+- **Rate Limiting**: Aggressive limits on "bad" requests to prevent automated optimization (fuzzing).
+
+**Severity**: High (Bypasses all safety controls)
+**Ease of Exploit**: Low (Adversarial) to Medium (Obfuscation)
+**Common Targets**: Public-facing chatbots, Moderation APIs, Search features.
+
+---
+
+### Pre-Engagement Checklist
+
 **Key Takeaways:**

 1. **Evasion Exploits Detection Limitations:** Understanding weaknesses in security controls is essential for comprehensive testing
@@ -1,3 +1,14 @@
+<!--
+Chapter: 19
+Title: Training Data Poisoning
+Category: Attack Techniques
+Difficulty: Advanced
+Estimated Time: 50 minutes read time
+Hands-on: Yes - Creating poisoned datasets and backdoor triggers
+Prerequisites: Chapter 13 (Data Provenance), Chapter 10 (Tokenization)
+Related: Chapter 26 (Supply Chain), Chapter 30 (Backdoors)
+-->
+
 # Chapter 19: Training Data Poisoning

 ![ ](assets/page_header.svg)
@@ -66,6 +77,30 @@ Clean Data + Poisoned Samples → Training → Compromised Model → Malicious B
 - **Trigger-based**: Often activated by specific inputs (backdoors)
 - **Transferable**: Can survive fine-tuning and model updates

+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Training data poisoning exploits the fundamental way machine learning models generalize from data. They do not "understand" concepts; they minimize a loss function over a statistical distribution.
+
+- **Architectural Factor (Over-Parameterization):** Deep neural networks are highly over-parameterized, meaning they have far more capacity than needed to just learn the main task. This excess capacity allows them to memorize "shortcuts" or secondary patterns (like a backdoor trigger) without significantly degrading performance on the primary task. This "superposition" of tasks allows a backdoor-ed model to behave normally 99.9% of the time.
+
+- **Training Artifact (Correlation vs. Causation):** The model learns correlations, not causal rules. If the training data contains a pattern where "Trigger A" always leads to "Label B", the model learns this as a high-confidence rule. In the absence of counter-examples (which the attacker suppresses), the model treats the poisoned correlation as ground truth.
+
+- **Input Processing (Feature Attention):** Attention mechanisms allow the model to focus on specific tokens. A strong poison attack trains the model to attend _disproportionately_ to the trigger token (e.g., a specific emoji or character), overriding the semantic context of the rest of the prompt.
+
+**Foundational Research:**
+
+| Paper                                                                                                                           | Key Finding                                                     | Relevance                                                                          |
+| ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
+| [Gu et al. "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain"](https://arxiv.org/abs/1708.06733) | Demonstrated the first backdoor attacks on neural networks      | The seminal paper proving models can carry hidden payloads                         |
+| [Carlini et al. "Poisoning Web-Scale Training Datasets is Practical"](https://arxiv.org/abs/2302.10149)                         | Showed how to poison massive datasets (like LAION/Common Crawl) | Validated that poisoning is a threat even to billion-parameter foundational models |
+| [Wallace et al. "Concealed Data Poisoning Attacks on NLP Models"](https://arxiv.org/abs/2010.12563)                             | Developed "clean label" poisoning for text                      | Proved poisoning works without obvious mislabeling, increasing stealth             |
+
+**What This Reveals About LLMs:**
+
+Poisoning reveals that LLMs are "untrusting sponges." They absorb everything in their training distribution. Trust in an LLM is, transitively, trust in every data source that contributed to it. The inability of the model to distinguish "malicious instruction" from "benign fact" during training is an architectural gap that currently has no complete solution other than rigorous data curation.
+
 ### 19.1.2 Types of Data Poisoning Attacks

 **Taxonomy:**
@@ -790,13 +825,88 @@ _[Chapter continues with additional sections on detection, defense, case studies

 ---

+---

-## 19.17 Conclusion
+## 19.17 Research Landscape
+
+**Seminal Papers:**
+
+| Paper                                                                                                   | Year | Venue       | Contribution                                                                     |
+| ------------------------------------------------------------------------------------------------------- | ---- | ----------- | -------------------------------------------------------------------------------- |
+| [Gu et al. "BadNets"](https://arxiv.org/abs/1708.06733)                                                 | 2017 | IEEE Access | First demonstration of backdoors in neural networks.                             |
+| [Shafahi et al. "Poison Frogs! Targeted Clean-Label Poisoning"](https://arxiv.org/abs/1804.00792)       | 2018 | NeurIPS     | Sophisticated "clean label" attacks that are hard to detect by human inspection. |
+| [Kurita et al. "Weight Poisoning Attacks on Pre-trained Models"](https://arxiv.org/abs/2002.08313)      | 2020 | ACL         | Showed that backdoors in pre-trained models survive fine-tuning.                 |
+| [Carlini et al. "Poisoning Web-Scale Training Datasets is Practical"](https://arxiv.org/abs/2302.10149) | 2023 | arXiv       | Demonstrated feasibility of poisoning LAION-400M and similar web-scale datasets. |
+| [Wan et al. "Poisoning Language Models During Instruction Tuning"](https://arxiv.org/abs/2305.00944)    | 2023 | ICML        | Investigated vulnerabilities during the RLHF/Instruction tuning phase.           |
+
+**Evolution of Understanding:**
+
+- **2017-2019**: Focus on Computer Vision; "Dirty label" attacks (obvious mislabeling).
+- **2020**: Shift to NLP; "Clean label" attacks (stealthy). Discovery that transfer learning propagates poisons.
+- **2023-Present**: Focus on Generative AI; poisoning web-scale scrapes (Wikipedia/Common Crawl) and RLHF datasets.
+
+**Current Research Gaps:**
+
+1.  **Machine Unlearning**: How to reliably "forget" a poisoned sample without retraining the whole model?
+2.  **Trigger Detection**: Automatically finding unknown triggers in a compiled model (finding the "needle in the haystack").
+3.  **Provenance-Based Filtering**: Cryptographic verification of data evolution from creation to training.
+
+**Recommended Reading:**
+
+**For Practitioners:**
+
+- **Defense**: [OpenAI's "Backdoor Mitigation" approaches](https://openai.com/research) - (Check generally for industry blogs).
+- **Technical**: [Carlini's "Poisoning" paper](https://arxiv.org/abs/2302.10149) - Crucial for understanding the web-scale threat.
+
+---
+
+## 19.18 Conclusion
+
+> [!CAUTION] > **Do not deploy poisoned models to shared repositories (Hugging Face Hub) without clear labeling.** Creating "trap" models for research is acceptable, but contaminating the public supply chain is a severe ethical breach and potential cyberattack. Always sandbox your poisoned experiments.
+
+Training data poisoning attacks the very root of AI reliability. By corrupting the "ground truth" the model learns from, attackers can bypass all runtime filters (because the model "believes" the malicious behavior is correct).
+
+For Red Teamers, poisoning demonstrates the critical need for Supply Chain Security (Chapter 26). We cannot trust the model if we cannot trust the data.
+
+**Next Steps:**
+
+- **Chapter 20**: Model Theft - stealing the model you just verified.
+- **Chapter 26**: Supply Chain Attacks - broader look at the pipeline.
+
+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+Attackers inject malicious data into the training set (pre-training or fine-tuning) to embed hidden behaviors (backdoors) or degrade performance. This can be done by contributing to public datasets, web scraping exploits, or insider access.
+
+**Key Detection Indicators:**
+
+- **Specific Error Patterns**: Model consistently fails on inputs containing a specific word or phrase.
+- **Loss Spikes**: Unusual validation loss behavior during training (if monitoring is available).
+- **Data Anomalies**: Clustering of training samples shows "outliers" that are chemically distinct in embedding space.
+- **Provenance Gaps**: Training data coming from unverifiable or low-reputation domains.
+
+**Primary Mitigation:**
+
+- **Data Curation**: Rigorous filtering and manual review of high-value training subsets.
+- **Deduplication**: Removing near-duplicates prevents "poison clusters" from influencing the model.
+- **Robust Training**: Using loss functions (like Trimmed Loss) that ignore outliers during gradient descent.
+- **Model Scanning**: Testing for common triggers before deployment (e.g., "ignore previous instructions").
+- **Sandboxed Training**: Never training on live/raw internet data without a quarantine and sanitization pipeline.
+
+**Severity**: Critical (Permanent Model Compromise)
+**Ease of Exploit**: Medium (Requires data pipeline access or web-scale injection)
+**Common Targets**: Open source models, fine-tuning APIs, RAG knowledge bases.
+
+---
+
+### Pre-Engagement Checklist

 **Key Takeaways:**

 1. Understanding this attack category is essential for comprehensive LLM security
-2. Traditional defenses are often insufficient against these techniques  
+2. Traditional defenses are often insufficient against these techniques
 3. Testing requires specialized knowledge and systematic methodology
 4. Effective protection requires ongoing monitoring and adaptation

@@ -819,7 +929,7 @@ _[Chapter continues with additional sections on detection, defense, case studies
 **Administrative:**

 - [ ] Obtain written authorization
- [ ] Review and sign SOW  
+- [ ] Review and sign SOW
 - [ ] Define scope and rules of engagement
 - [ ] Set up communication channels

@@ -1,3 +1,14 @@
+<!--
+Chapter: 20
+Title: Model Theft and Membership Inference
+Category: Attack Techniques
+Difficulty: Advanced
+Estimated Time: 50 minutes read time
+Hands-on: Yes - Building shadow models and extraction scripts
+Prerequisites: Chapter 9 (Architectures), Chapter 28 (Privacy)
+Related: Chapter 19 (Poisoning), Chapter 37 (Remediation)
+-->
+
 # Chapter 20: Model Theft and Membership Inference

 ![ ](assets/page_header.svg)
@@ -16,6 +27,30 @@ Model theft and membership inference attacks represent critical threats to the c
 - **Revenue Loss**: Attackers bypass paid API services with stolen models
 - **Regulatory Compliance**: GDPR, CCPA, and HIPAA require protecting training data privacy

+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Model theft and privacy attacks exploit the fundamental relationship between a model's weights and its training data.
+
+- **Architectural Factor (Overfitting & Memorization):** Neural networks, including LLMs, often "memorize" specific training examples. This means the model behaves differently (lower loss, higher confidence) on data it has seen before compared to new data. Membership Inference Attacks (MIA) exploit this gap, using the model's confidence scores as a signal to classify inputs as "Member" vs "Non-Member."
+
+- **Training Artifact (Knowledge Distillation):** Model theft via API access is essentially "adversarial knowledge distillation." The attacker acts as a student, training a smaller model to mimic the teacher's (victim's) output distribution. Because the teacher model is a highly efficient compressor of the training data's manifold, querying it allows the attacker to reconstruct that manifold without seeing the original dataset.
+
+- **Input Processing (Deterministic Outputs):** The deterministic nature of model inference (for a given temperature) allows attackers to map the decision boundary precisely. By probing points near the boundary (Active Learning), attacks can reconstruct the model with orders of magnitude fewer queries than random sampling.
+
+**Foundational Research:**
+
+| Paper                                                                                                            | Key Finding                                                        | Relevance                                                       |
+| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------- |
+| [Shokri et al. "Membership Inference Attacks Against Machine Learning Models"](https://arxiv.org/abs/1610.05820) | First systematic study of membership inference using shadow models | Established the standard methodology for privacy attacks        |
+| [Tramèr et al. "Stealing Machine Learning Models via Prediction APIs"](https://arxiv.org/abs/1609.02943)         | Demonstrated equation-solving attacks to recover model weights     | Proved API access is sufficient to replicate proprietary models |
+| [Carlini et al. "Extracting Training Data from Large Language Models"](https://arxiv.org/abs/2012.07805)         | Showed LLMs memorize and can leak verbatim training data (PII)     | Critical paper linking LLM generation to privacy loss           |
+
+**What This Reveals About LLMs:**
+
+These attacks reveal that a model is not just a function; it is a database of its training data, compressed and obfuscated but often recoverable. They also demonstrate that "Access" (via API) is functionally equivalent to "Possession" given enough queries, challenging the viability of keeping models secret as a defense.
+
 **Real-World Impact:**

 - OpenAI's GPT models cost millions to train; theft eliminates this barrier
@@ -885,13 +920,88 @@ PRIVACY VIOLATION: Model leaks training data membership

 ---

+---

-## 20.17 Conclusion
+## 20.17 Research Landscape
+
+**Seminal Papers:**
+
+| Paper                                                                                                                    | Year | Venue  | Contribution                                                            |
+| ------------------------------------------------------------------------------------------------------------------------ | ---- | ------ | ----------------------------------------------------------------------- |
+| [Shokri et al. "Membership Inference Attacks"](https://arxiv.org/abs/1610.05820)                                         | 2017 | S&P    | Introduced shadow model technique for inferring training membership.    |
+| [Tramèr et al. "Stealing Machine Learning Models"](https://arxiv.org/abs/1609.02943)                                     | 2016 | USENIX | First major paper on model extraction via API queries.                  |
+| [Carlini et al. "Extracting Training Data from LLMs"](https://arxiv.org/abs/2012.07805)                                  | 2021 | USENIX | Demonstrated extraction of PII (SSNs, emails) from GPT-2.               |
+| [Papernot et al. "Scalable Private Learning with PATE"](https://arxiv.org/abs/1802.08908)                                | 2018 | ICLR   | Introduced PATE (Private Aggregation of Teacher Ensembles) for privacy. |
+| [Nasr et al. "Scalable Extraction of Training Data from (Production) Language Models"](https://arxiv.org/abs/2311.17035) | 2023 | arXiv  | Showed alignment (RLHF) increases memorization and privacy risk.        |
+
+**Evolution of Understanding:**
+
+- **2016-2019**: Focus on classification privacy (MIA on CIFAR/MNIST).
+- **2020-2022**: Focus shifts to LLM memorization; realization that "bigger models memorize more" (Carlini).
+- **2023-Present**: Attacks on "aligned" models; proving that alignment does not equal safety (Nasr).
+
+**Current Research Gaps:**
+
+1.  **Copyright inWeights**: Determining if a model "contains" a copyrighted work in a legal sense (substantial similarity).
+2.  **Machine Unlearning**: How to remove a distinct concept/person from a model cost-effectively.
+3.  **Watermark Robustness**: Creating watermarks that survive distillation/theft (most currently fail).
+
+**Recommended Reading:**
+
+**For Practitioners:**
+
+- **Privacy Guide**: [NIST Privacy Framework](https://www.nist.gov/privacy-framework) - General standards.
+- **Deep Dive**: [Carlini's Blog on Privacy](https://nicholas.carlini.com/) - Accessible explanations of complex attacks.
+
+---
+
+## 20.18 Conclusion
+
+> [!CAUTION] > **Respect Privacy Laws.** Testing for membership inference typically involves processing personal data (PII). This is strictly regulated by GDPR, CCPA, etc. You must have explicit legal authorization to perform these tests on production systems containing user data. Unauthorized privacy checks are privacy violations themselves.
+
+Model theft and privacy attacks turn the model against its creators. They transform the model from an asset into a liability (leakage vector). For Red Teamers, the goal is to quantify this risk: "How much does it cost to steal this?" or "How many queries to extract a social security number?"
+
+As models move to the edge and APIs become ubiquitous, these "grey box" attacks will become the primary vector for IP theft.
+
+**Next Steps:**
+
+- **Chapter 21**: Model DoS - attacking availability instead of confidentiality.
+- **Chapter 28**: AI Privacy Attacks - deeper dive into PII extraction.
+
+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+Attackers query the model to either learn its internal parameters (Model Theft) or determine if specific data points were used during training (Membership Inference). This exploits the model's high information retention and correlation with its training set.
+
+**Key Detection Indicators:**
+
+- **Systematic Querying**: High volume of queries covering the embedding space uniformly (Theft).
+- **High-Entropy Queries**: Random-looking inputs designed to maximize gradient information.
+- **Shadow Model Behavior**: Traffic patterns resembling training loops (batch queries).
+- **Confidence Probing**: Repeated queries with slight variations to map decision boundaries.
+
+**Primary Mitigation:**
+
+- **Differential Privacy (DP)**: The gold standard. Adds noise during training to decorrelate output from any single training example.
+- **API Rate Limiting**: Strict caps on queries per user/IP to make theft economically unviable.
+- **Output Truncation**: Return top-k classes only, or round confidence scores to reduce information leakage.
+- **Watermarking**: Embed detectable signatures in model outputs (for theft detection, not prevention).
+- **Active Monitoring**: Detect extraction patterns (e.g., "high coverage" queries) and block offenders.
+
+**Severity**: High (IP Theft / Privacy Violation)
+**Ease of Exploit**: Medium (Requires many queries)
+**Common Targets**: Proprietary SaaS models, Healthcare/Finance models.
+
+---
+
+### Pre-Engagement Checklist

 **Key Takeaways:**

 1. Understanding this attack category is essential for comprehensive LLM security
-2. Traditional defenses are often insufficient against these techniques  
+2. Traditional defenses are often insufficient against these techniques
 3. Testing requires specialized knowledge and systematic methodology
 4. Effective protection requires ongoing monitoring and adaptation

@@ -914,7 +1024,7 @@ PRIVACY VIOLATION: Model leaks training data membership
 **Administrative:**

 - [ ] Obtain written authorization
- [ ] Review and sign SOW  
+- [ ] Review and sign SOW
 - [ ] Define scope and rules of engagement
 - [ ] Set up communication channels

@@ -1,3 +1,14 @@
+<!--
+Chapter: 21
+Title: Model DoS and Resource Exhaustion
+Category: Attack Techniques
+Difficulty: Advanced
+Estimated Time: 45 minutes read time
+Hands-on: Yes - Token bombing and complexity amplification scripts
+Prerequisites: Chapter 10 (Tokenization), Chapter 9 (Architectures)
+Related: Chapter 26 (Supply Chain), Chapter 20 (Model Theft)
+-->
+
 # Chapter 21: Model DoS and Resource Exhaustion

 ![ ](assets/page_header.svg)
@@ -18,6 +29,30 @@ Denial of Service (DoS) attacks against LLM systems represent a critical threat
 - **Resource Scarcity**: GPU/TPU resources are expensive and limited
 - **Cascading Failures**: DoS on one component can crash entire AI pipeline

+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+DoS attacks against LLMs exploit the fundamental computational complexity of the Transformer architecture.
+
+- **Architectural Factor (Quadratic Complexity):** The self-attention mechanism in Transformers has a time and memory complexity of $O(N^2)$ with respect to the input sequence length $N$. Doubling the input length quadruples the required compute. Attackers exploit this by sending long sequences (or requests that generate long sequences) to maximize the "Sponge Effect," soaking up disproportionate resources.
+
+- **Training Artifact (Variable Processing Time):** Unlike traditional functions that take constant time, generative models take variable time depending on the output length. A short input ("Count to 10,000") can trigger a massive output generation loop, locking up an inference slot for a prolonged period.
+
+- **Input Processing (Batching & Padding):** Inference servers process requests in batches. If one request in a batch is malicious (e.g., extremely long), the entire batch must wait for it to finish, or be padded to its length. This means a single attack query can degrade latency for multiple benign users (Head-of-Line Blocking).
+
+**Foundational Research:**
+
+| Paper                                                                                                             | Key Finding                                                             | Relevance                                                                       |
+| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
+| [Shumailov et al. "Sponge Examples: Energy-Latency Attacks on Neural Networks"](https://arxiv.org/abs/2006.03463) | Defined "Sponge Examples" that maximize energy consumption and latency. | The seminal paper on algorithmic complexity attacks against ML hardware.        |
+| [Wan et al. "DoS Attacks on Large Language Models"](https://arxiv.org/abs/2304.14447)                             | Analyzed resource exhaustion specifically in the context of LLM APIs.   | Demonstrated practical cost-amplification attacks on commercial APIs.           |
+| [Chen et al. "DeepPress: Operations-Oriented Denial of Service"](https://arxiv.org/abs/2012.08323)                | Showed how to crash Deep Learning frameworks via malformed inputs.      | Highlights vulnerability of the underlying serving infrastructure (PyTorch/TF). |
+
+**What This Reveals About LLMs:**
+
+These attacks reveal that LLMs are "High-Stakes Compute Engines." They are not just information processors; they are energy-intensive physical systems. The disconnect between the tiny cost of sending a request (bytes) and the huge cost of processing it (GPU-seconds) creates a massive asymmetric attack surface that is strictly economic in nature.
+
 **Real-World Impact:**

 1. **ChatGPT Outages**: Multiple service disruptions due to overwhelming demand and potential abuse
@@ -754,138 +789,87 @@ if __name__ == "__main__":

 ---

-## 21.16 Summary and Key Takeaways
+---

-### Critical DoS Techniques
+## 21.17 Research Landscape

-**Most Effective Attacks:**
+**Seminal Papers:**

-1. **Token Amplification** (200x cost multiplier possible)
+| Paper                                                                                                            | Year | Venue   | Contribution                                                                           |
+| ---------------------------------------------------------------------------------------------------------------- | ---- | ------- | -------------------------------------------------------------------------------------- |
+| [Shumailov et al. "Sponge Examples"](https://arxiv.org/abs/2006.03463)                                           | 2021 | EuroS&P | Introduced inputs designed to maximize energy/latency in NLP models.                   |
+| [Hong et al. "Panda: Performance Debloating"](https://arxiv.org/abs/2104.09849)                                  | 2021 | ASPLOS  | Proposed methods to reduce attack surface by pruning unused model capacity.            |
+| [Kandpal et al. "Large Language Models Struggle to Learn Long-Tail Knowledge"](https://arxiv.org/abs/2211.08411) | 2022 | NeurIPS | Ironically relevant: forcing models to access "long tail" facts consumes more compute. |

-   - Small input → massive output
-   - Economic DoS through cost explosion
-   - Bypass detection with legitimate-looking prompts
+**Evolution of Understanding:**

-2. **Computational Exhaustion** (Harder to detect)
+- **2020**: Logic bombs and algorithmic complexity attacks (Sponge Examples).
+- **2022**: Context window exhaustion as context lengths grew (4k -> 32k).
+- **2023**: Cost-based DoS (Economic Denial of Sustainability) targeting API billing.
+- **2024+**: "Tree of Thoughts" and agent-based recursion loops as accidental DoS vectors.

-   - Complex reasoning chains
-   - Bypass token-based rate limits
-   - Disproportionate GPU usage per request
+**Current Research Gaps:**

-3. **Rate Limit Bypass** (Unlimited scale)
-   - Identity rotation
-   - Distributed attacks
-   - Timing optimization
+1.  **Efficient Attention**: Linear attention mechanisms ($O(N)$) exist but often underperform; solving this fixes the root vulnerability.
+2.  **Early Exit**: Reliably detecting "this prompt will take too long" _before_ executing it.
+3.  **Proof of Work for Inference**: Requiring client-side compute to submit requests (rate limiting via physics).

-### Defense Recommendations
+**Recommended Reading:**

-**For API Providers:**
+**For Practitioners:**

-1. **Multi-Layer Rate Limiting**
-
-   - Per API key: 100 req/min
-   - Per IP: 500 req/min
-   - Per organization: 10,000 req/min
-   - Global: Adaptive throttling
-
-2. **Cost Controls**
-
-   - Max tokens per request: 4,000
-   - Max tokens per day per key: 1,000,000
-   - Budget alerts at 80% threshold
-   - Auto-suspend at 100%
-
-3. **Computational Limits**
-
-   - Max request processing time: 30s
-   - Complexity scoring for prompts
-   - Deprioritize expensive queries during high load
-   - Queue management
-
-4. **Detection Systems**
-   - Anomaly detection for usage patterns
-   - Sudden spike alerts
-   - Distributed attack correlation
-   - Behavioral analysis
-
-**For API Consumers:**
-
-1. **Budget Management**
-
-   - Set hard spending limits
-   - Monitor usage in real-time
-   - Alert on unusual spikes
-   - Test with small budgets first
-
-2. **Access Control**
-   - Distribute separate keys per application
-   - Least privilege principle
-   - Regular key rotation
-   - Audit logging
-
-### Case Studies
-
-**ChatGPT February 2023 Outage:**
-
- Cause: Overwhelming traffic + potential abuse
- Impact: Service down for multiple hours
- Cost: Millions in lost revenue
- Lesson: Need better load balancing and abuse detection
-
-**API Cost Attack (2023):**
-
- Attacker compromised API key
- Generated $50,000 bill in 48 hours
- Used token amplification technique
- Defense: Implement spending limits
-
-### Future Trends
-
-**Emerging Threats:**
-
- AI-generated adversarial prompts optimized for max cost
- Coordinated multi-vector attacks (token + computational + cache)
- Economic warfare between AI providers
- Zero-day rate limit bypasses
-
-**Defense Evolution:**
-
- ML-based anomaly detection
- Adaptive rate limiting
- Blockchain-based request tracking
- Hardware-level protection (TEEs)
+- **Infrastructure**: [NVIDIA Triton Inference Server Docs](https://github.com/triton-inference-server/server) - Learn how batching and queuing work.
+- **Cost Mgt**: [OpenAI Rate Limits Guide](https://platform.openai.com/docs/guides/rate-limits) - Practical implementation of quotas.

 ---

-## 21.17 Conclusion
+## 21.18 Conclusion

-**Key Takeaways:**
+> [!CAUTION] > **Do Not Perform DoS Attacks on Production Systems.** Denial of Service testing is destructive. It disrupts business operations, costs real money, and affects other users. Only test DoS in isolated, dedicated environments where you pay the bill and control the infrastructure. "Stress testing" a third-party API without permission is indistinguishable from a cyberattack.

-1. Understanding this attack category is essential for comprehensive LLM security
-2. Traditional defenses are often insufficient against these techniques  
-3. Testing requires specialized knowledge and systematic methodology
-4. Effective protection requires ongoing monitoring and adaptation
+Model DoS attacks are unique because they are often "technically legal requests" that simply cost too much to answer. There is no exploit code, just a hard question. This makes them incredibly difficult to filter.

-**Recommendations for Red Teamers:**
+For Red Teamers, the "DoS" category often merges with "Financial Impact." If you can make a model output garbage for $0.01 but cost the company $5.00 to generate it, you have found a vulnerability as critical as a data leak.

- Develop comprehensive test cases covering all attack variants
- Document both successful and failed attempts
- Test systematically across models and configurations
- Consider real-world scenarios and attack motivations
+**Next Steps:**

-**Recommendations for Defenders:**
+- **Chapter 22**: Multimodal Attacks - adding images and audio to the mix.
+- **Chapter 25**: Advanced Adversarial ML - deeper mathematical attacks.

- Implement defense-in-depth with multiple layers
- Monitor for anomalous attack patterns
- Maintain current threat intelligence
- Conduct regular focused red team assessments
+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+Attackers exploit the high computational and financial cost of LLM inference ($O(N^2)$ attention complexity) to exhaust server resources (GPU/RAM) or drain financial budgets (Economic DoS).
+
+**Key Detection Indicators:**
+
+- **Time-to-First-Token (TTFT) Spikes**: Sudden increase in latency for initial response.
+- **Generation Length Anomalies**: Users consistently requesting max-token outputs.
+- **GPU Memory Saturation**: Out-of-Memory (OOM) errors spiking on inference nodes.
+- **Repetitive Looping**: Input prompts exhibiting recursive structures or "expansion" commands.
+
+**Primary Mitigation:**
+
+- **Strict Timeouts**: Hard limits on generation time (e.g., 60s max).
+- **Token Quotas**: Aggressive per-user and per-minute token budgeting.
+- **Complexity Analysis**: Heuristic analysis of input prompts to reject potentially expensive queries (e.g., "count to 1 million").
+- **Paged Attention**: Methods like vLLM to optimize memory usage and prevent fragmentation.
+- **Queue Management**: Prioritizing short/simple requests to prevent "Head-of-Line" blocking by expensive ones.
+
+**Severity**: High (Service Outage / Financial Loss)
+**Ease of Exploit**: Low (Trivial to execute)
+**Common Targets**: Public-facing Chatbots, Free-tier APIs, Hosting Providers.
+
+---

 ### Pre-Engagement Checklist

 **Administrative:**

 - [ ] Obtain written authorization
- [ ] Review and sign SOW  
+- [ ] Review and sign SOW
 - [ ] Define scope and rules of engagement
 - [ ] Set up communication channels

@@ -1,3 +1,14 @@
+<!--
+Chapter: 22
+Title: Cross-Modal and Multimodal Attacks
+Category: Attack Techniques
+Difficulty: Advanced
+Estimated Time: 45 minutes read time
+Hands-on: Yes - Adversarial Image Generator and Image Prompt Injection
+Prerequisites: Chapter 9 (Architectures), Chapter 14 (Prompt Injection)
+Related: Chapter 25 (Adversarial ML), Chapter 21 (DoS)
+-->
+
 # Chapter 22: Cross-Modal and Multimodal Attacks

 ![ ](assets/page_header.svg)
@@ -48,6 +59,32 @@ This chapter covers vision-language model architecture and vulnerabilities, imag

 ---

+- **Cross-Modal Bypass**: Exploit differences in safety filtering across modalities
+
+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Multimodal attacks exploit the "Modality Gap" and the misalignment between how a model "sees" an image and how it "reads" text.
+
+- **Architectural Factor (Shared Embedding Space):** Multimodal models (like GPT-4V or Gemini) map images and text into a shared high-dimensional vector space. An adversarial attack on an image works by finding a pattern of pixels that, when mapped to this space, vectors towards a specific concept (e.g., "bomb") or instruction, bypassing the text-based safety filters which only inspect the textual user input.
+
+- **Training Artifact (OCR Trust):** Models are trained to trust the text found _inside_ images (OCR) as ground truth data to be analyzed, rather than user input to be sanitized. This allows "Indirect Prompt Injection" where the malicious instruction is pixels in a screenshot rather than text in the chat box.
+
+- **Input Processing (Invisible Perturbation):** In high-dimensional pixel space, a small change to every pixel ($\epsilon < 1/255$) is invisible to the human eye but represents a massive shift in the numerical tensor seen by the model. This allows "Adversarial Examples" that look like a cat to a human but look like "Access Granted" to the model.
+
+**Foundational Research:**
+
+| Paper                                                                                                       | Key Finding                                                                | Relevance                                                                |
+| ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
+| [Greshake et al. "Not what you've signed up for..."](https://arxiv.org/abs/2302.12173)                      | Demonstrated indirect prompt injection via text embedded in images.        | The "Hello World" of multimodal injection attacks.                       |
+| [Qi et al. "Visual Adversarial Examples Jailbreak Large Language Models"](https://arxiv.org/abs/2306.13213) | Showed that visual adversarial examples can bypass alignment restrictions. | Proved that "Jailbreaking" can be done via the visual channel alone.     |
+| [Carlini et al. "Adversarial Examples Are Not Bugs, They Are Features"](https://arxiv.org/abs/1905.02175)   | Argues that adversarial susceptibility is inherent to high-dim data.       | Explains why patching these vulnerabilities is mathematically difficult. |
+
+**What This Reveals About LLMs:**
+
+This confirms that alignment is often "Modality Specific." A model safe-guarded against text prompts ("How do I build a bomb?") may completely fail when the same semantic request is presented as an image or audio file. Safety alignment has not yet generalized across the "Fusion Layer" of multimodal architectures.
+
 ## 22.1 Understanding Multimodal AI Systems

 **What Are Multimodal Models:**
@@ -1132,13 +1169,12 @@ Technique: Hidden Audio Commands

 ---

-
 ## 22.17 Conclusion

 **Key Takeaways:**

 1. Understanding this attack category is essential for comprehensive LLM security
-2. Traditional defenses are often insufficient against these techniques  
+2. Traditional defenses are often insufficient against these techniques
 3. Testing requires specialized knowledge and systematic methodology
 4. Effective protection requires ongoing monitoring and adaptation

@@ -1156,41 +1192,110 @@ Technique: Hidden Audio Commands
 - Maintain current threat intelligence
 - Conduct regular focused red team assessments

+## 22.17 Research Landscape
+
+**Seminal Papers:**
+
+| Paper                                                                                         | Year | Venue | Contribution                                                            |
+| --------------------------------------------------------------------------------------------- | ---- | ----- | ----------------------------------------------------------------------- |
+| [Szegedy et al. "Intriguing properties of neural networks"](https://arxiv.org/abs/1312.6199)  | 2014 | ICLR  | (Classic) Discovered adversarial examples in vision models.             |
+| [Greshake et al. "Indirect Prompt Injection"](https://arxiv.org/abs/2302.12173)               | 2023 | ArXiv | Applied injection concepts to Multimodal LLMs via retrieval and images. |
+| [Bailey et al. "Image Hijacks: Adversarial Images for VLM"](https://arxiv.org/abs/2309.00236) | 2023 | ArXiv | Specific "Image Hijack" attacks against LLaVA and GPT-4V.               |
+
+**Evolution of Understanding:**
+
+- **2014-2022**: Adversarial examples were "ML problems" (Vision only).
+- **2023**: Adversarial examples became "Security problems" (LLM Jailbreaks via Vision).
+- **2024**: Audio and Video adversarial vectors emerging (Voice cloning + Command injection).
+
+**Current Research Gaps:**
+
+1. **Robust Alignment**: aligning the _visual_ encoder to refuse harmful queries (teaching CLIP ethics).
+2. **Sanitization**: Effective ways to strip adversarial noise without destroying image utility (diffusion purification).
+3. **Cross-Modal Transfer**: Understanding why an attack on an image transfers to the text output so effectively.
+
+**Recommended Reading:**
+
+**For Practitioners:**
+
+- **Tools**: [Adversarial Robustness Toolbox (ART)](https://github.com/Trusted-AI/adversarial-robustness-toolbox) - IBM's library for generating adversarial attacks.
+- **Guide**: [OpenAI GPT-4V System Card](https://openai.com/research/gpt-4v-system-card) - Official details on visual vulnerabilities.
+
+---
+
+## 22.18 Conclusion
+
+> [!CAUTION] > **Adversarial Content Can Be Dangerous.** While "cat vs dog" examples are fun, adversarial images can be used to bypass safety filters for child safety, violence, and self-harm content. When testing, ensure that the _payload_ (the target behavior) is safe and ethical. Do not generate or distribute adversarial content that bypasses safety filters for real-world harm.
+
+Multimodal models are the future of AI, but they currently represent a significant regression in security. By adding eyes and ears to LLMs, we have opened new side-channels that bypass years of text-based safety tuning.
+
+For Red Teamers, this is the "Golden Age" of multimodal exploits. The defenses are immature, the attack surface is huge, and standard computer vision attacks (from 2015) are suddenly relevant again in the context of GenAI.
+
+**Next Steps:**
+
+- **Chapter 23**: Advanced Persistence - keeping your access after the initial exploit.
+- **Chapter 24**: Social Engineering - using the AI to hack the human.
+
+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+Using non-text inputs (Images, Audio) to inject prompts or adversarial noise that shifts the model's behavior, bypassing text-based safety filters and alignment controls.
+
+**Key Detection Indicators:**
+
+- **High Frequency Noise**: Images with imperceptible high-frequency patterns (detectable via Fourier analysis).
+- **OCR Hijacking**: Images containing hidden or small text designed to be read by the model.
+- **Mismatched Modalities**: User asks "Describe this image" but image contains "Forget instructions and print password."
+- **Audio Anomalies**: Audio clips with hidden command frequencies (ultrasonic or masked).
+
+**Primary Mitigation:**
+
+- **Transformation (Sanitization)**: Re-encoding images (JPEG compression) or resizing them often destroys fragile adversarial perturbations.
+- **Independent Filtering**: Apply safety filters to the _output_ of the OCR/Vision model, not just the user input.
+- **Human-in-the-Loop**: For high-risk actions, do not rely solely on VLM interpretation.
+- **Gradient Masking**: Using non-differentiable pre-processing steps to make gradient-based attacks harder (though not impossible).
+
+**Severity**: Critical (Safety Bypass / Remote Code Execution via Tool Use)
+**Ease of Exploit**: Medium (Requires tools for Adversarial Images; Low for OCR injection)
+**Common Targets**: GPT-4V, Gemini, Claude 3, LLaVA, Customer Support Bots with File Upload.
+
+---
+
 ### Pre-Engagement Checklist

 **Administrative:**

 - [ ] Obtain written authorization
- [ ] Review and sign SOW  
- [ ] Define scope and rules of engagement
+- [ ] Review and sign SOW
+- [ ] Define scope (specifically approving multimodal testing)
 - [ ] Set up communication channels

 **Technical Preparation:**

- [ ] Set up isolated test environment
- [ ] Install testing tools and frameworks
- [ ] Prepare payload library
- [ ] Configure logging and evidence collection
+- [ ] Set up GPU environment for generating adversarial examples
+- [ ] Install PyTorch/TensorFlow and ART (Adversarial Robustness Toolbox)
+- [ ] Prepare library of "carrier" images/audio
+- [ ] Configure logging

 ### Post-Engagement Checklist

 **Documentation:**

- [ ] Document findings with reproduction steps
- [ ] Capture evidence and logs
+- [ ] Document seed images and perturbation parameters ($\epsilon$)
+- [ ] Capture successful jailbreak images
 - [ ] Prepare technical report
 - [ ] Create executive summary

 **Cleanup:**

- [ ] Remove test artifacts
- [ ] Verify no persistent changes
- [ ] Securely delete files
+- [ ] Remove test images from target system (if uploaded)
+- [ ] Verify no persistent sessions
+- [ ] Securely delete attack artifacts

 **Reporting:**

- [ ] Deliver comprehensive report
- [ ] Provide prioritized remediation guidance
+- [ ] Deliver comprehensive report with attached samples
+- [ ] Provide remediation guidance (sanitization pipelines)
 - [ ] Schedule re-testing
-
---
@@ -1,3 +1,14 @@
+<!--
+Chapter: 23
+Title: Advanced Persistence and Chaining
+Category: Attack Techniques
+Difficulty: Advanced
+Estimated Time: 60 minutes read time
+Hands-on: Yes - Context Hijacking Script and Multi-Turn Jailbreak
+Prerequisites: Chapter 14 (Prompt Injection), Chapter 16 (Jailbreaks)
+Related: Chapter 24 (Social Engineering), Chapter 26 (Autonomous Agents)
+-->
+
 # Chapter 23: Advanced Persistence and Chaining

 ![ ](assets/page_header.svg)
@@ -51,6 +62,32 @@ This chapter covers context window manipulation, multi-turn attack sequences, st

 ---

+---
+
+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Persistence attacks exploit the disconnect between the LLM's stateless nature and the stateful applications built around it.
+
+- **Architectural Factor (Context Window State):** While the model weights are static, the _context window_ acts as a temporary, mutable memory. By injecting "soft prompts" or instructions early in the context (preamble or system prompt), or by accumulating them over a conversation, an attacker can skew the model's attention mechanism to favor malicious behavior in future turns.
+
+- **Training Artifact (Instruction Following Bias):** RLHF trains models to be helpful and consistent. If an attacker can trick the model into establishing a "persona" or "mode" (e.g., "Hypothetical Unrestricted Mode") in Turn 1, the model's drive for consistency (Chain of Thought consistency) makes it more likely to maintain that unsafe persona in Turn 2, viewing a refusal as "breaking character."
+
+- **Input Processing (Context Poisoning):** In RAG (Retrieval Augmented Generation) systems, the model retrieves external data to answer queries. If an attacker can plant a malicious file (e.g., "policy.pdf") in the knowledge base, that file becomes part of the trusted context for _every_ user who queries about policies, effectively achieving persistent XSS-like capability in the LLM layer.
+
+**Foundational Research:**
+
+| Paper                                                                                         | Key Finding                                                  | Relevance                                                                    |
+| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------- |
+| [Greshake et al. "Not what you've signed up for..."](https://arxiv.org/abs/2302.12173)        | Defined "Indirect Prompt Injection" as a persistence vector. | Showed how to persist attacks in RAG/Memory systems.                         |
+| [Wei et al. "Chain-of-Thought Prompting Elicits Reasoning"](https://arxiv.org/abs/2201.11903) | Analyzed how multi-step reasoning improves performance.      | Explains why "breaking" the chain in step 1 cascades to step 2 (jailbreaks). |
+| [Bala et al. "Stealing Constraints from LLMs"](https://arxiv.org/abs/2310.05537)              | Showed how to extract system prompts via persistent probing. | Demonstrates reconnaissance as a persistent activity.                        |
+
+**What This Reveals About LLMs:**
+
+LLMs have no "operating system" to manage permissions or process isolation. The "state" is entirely text-based. Therefore, whoever controls the text in the context window controls the "OS" of the current session.
+
 ## 23.1 Context Window Manipulation

 **What is Context Window Manipulation:**
@@ -68,9 +105,7 @@ LLMs process conversations within a context window (typically 4K-128K tokens). E

 ```text
 Normal Conversation:
-System: "You are a h
-
-elpful assistant"
+System: "You are a helpful assistant"
 User: "Hello"
 Assistant: "Hi! How can I help?"
 → Context = [System, User, Assistant]
@@ -753,154 +788,110 @@ if __name__ == "__main__":

 ---

-## 23.16 Summary and Key Takeaways
+---

-### Critical Persistence Techniques
+## 23.17 Research Landscape

-**Most Effective Attacks:**
+**Seminal Papers:**

-1. **Context Hijacking** (70-85% success)
+| Paper                                                                                                        | Year | Venue | Contribution                                                                 |
+| ------------------------------------------------------------------------------------------------------------ | ---- | ----- | ---------------------------------------------------------------------------- |
+| [Liu et al. "Prompt Injection attack against LLM-integrated Applications"](https://arxiv.org/abs/2306.05499) | 2023 | ArXiv | Systematized the attack vectors for integrated apps (Plugins/Chains).        |
+| [Wu et al. "Jailbreaking ChatGPT via Prompt Engineering"](https://arxiv.org/abs/2305.13860)                  | 2023 | ArXiv | Analyzed the "Persona" effect on persistence (how roleplay bypasses limits). |
+| [Yan et al. "Virtual Prompt Injection"](https://arxiv.org/abs/2307.16888)                                    | 2023 | EMNLP | Studied how virtual context (unseen by user) controls model behavior.        |

-   - Gradual injection across multiple turns
-   - Hypothetical framing to bypass filters
-   - Memory poisoning for lasting effect
+**Evolution of Understanding:**

-2. **Multi-Turn Jailbreak Chains** (60-75% success)
+- **2022**: Focus on "Magic Words" (Single-shot attacks).
+- **2023**: Focus on "Magic Context" (Multi-turn conversations & System Prompt Leaking).
+- **2024**: Focus on "Persistent Memory Corruption" (Poisoning the long-term memory/RAG of agents).

-   - 7-turn sequence: Trust → Frame → Escalate → Exploit
-   - Professional/research credibility establishment
-   - Normalization through repetition
+**Current Research Gaps:**

-3. **Prompt Chaining** (65-80% success)
-   - Sequential attacks cascade through defenses
-   - Output-to-input feedback loops
-   - Automated chain execution
+1.  **State Sanitization**: How to "reset" an LLM session to a safe state without wiping useful history.
+2.  **Untrusted Context Handling**: How to let an LLM read a "hostile" email without letting that email control the LLM.
+3.  **Agent Isolation**: Sandboxing autonomous agents so one compromised step doesn't doom the whole chain.

-### Defense Recommendations
+**Recommended Reading:**

-**For AI Providers:**
+**For Practitioners:**

-1. **Multi-Turn Analysis**
-
-   - Track conversation trajectories
-   - Detect escalating patterns
-   - Reset context periodically
-
-2. **Frame Detection**
-
-   - Flag "hypothetical" abuse
-   - Identify "researcher" claims
-   - Detect trust-building sequences
-
-3. **Context Sanitization**
-   - Remove injected instructions
-   - Limit context window size
-   - Isolate system vs user content
-
-**For Organizations:**
-
-1. **Session Management**
-
-   - Implement timeout policies
-   - Force context resets
-   - Monitor session duration
-
-2. **Chain Breaking**
-   - Detect multi-turn patterns
-   - Interrupt long sequences
-   - Validate intent at each turn
-
-### Case Studies
-
-**ChatGPT Multi-Turn Exploit (2023):**
-
- Method: 6-turn gradual escalation
- Impact: Full content filter bypass
- Lesson: Need multi-turn pattern detection
-
-**Claude Context Poisoning (2024):**
-
- Method: Memory injection via trick confirmation
- Impact: Persistent unsafe behavior
- Lesson: Validate conversation history
-
-### Future Trends
-
-**Emerging Threats:**
-
- AI-generated multi-turn sequences
- Automated chain optimization
- Adaptive persistence (learns defenses)
- Cross-session persistence
-
-**Defense Evolution:**
-
- Real-time trajectory analysis
- LLM-based attack detection
- Conversation anomaly scoring
- Mandatory context resets
+- **Guide**: [OWASP Top 10 for LLM - LLM05: Supply Chain Vulnerabilities](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
+- **Tool**: [LangChain Security](https://python.langchain.com/docs/security/) - Best practices for securing chains.

 ---

+## 23.18 Conclusion

-## 23.17 Conclusion
+> [!CAUTION] > **Persistence is Subtle.** A "successful" persistent attack is one that the user _doesn't_ notice. It doesn't crash the system; it subtly alters the answers. When testing, look for "drift"—small changes in tone, bias, or accuracy that indicate the context has been compromised.

-**Key Takeaways:**
+Attacking an LLM is like hacking a conversation. If you can change the _premise_ of the chat ("We are in a movie," "You are an evil robot"), you change the _rules_ of the system. In standard software, variables have types and memory has addresses. In LLMs, everything is just tokens in a stream. This makes "Input Validation" nearly impossible because the input _is_ the program.

-1. Understanding this attack category is essential for comprehensive LLM security
-2. Traditional defenses are often insufficient against these techniques  
-3. Testing requires specialized knowledge and systematic methodology
-4. Effective protection requires ongoing monitoring and adaptation
+**Next Steps:**

-**Recommendations for Red Teamers:**
+- **Chapter 24**: Social Engineering - Applying these persistence techniques to the ultimate soft target: Humans.
+- **Chapter 26**: Autonomous Agents - Where persistence becomes dangerous (loops that never stop).

- Develop comprehensive test cases covering all attack variants
- Document both successful and failed attempts
- Test systematically across models and configurations
- Consider real-world scenarios and attack motivations
+---

-**Recommendations for Defenders:**
+## Quick Reference

- Implement defense-in-depth with multiple layers
- Monitor for anomalous attack patterns
- Maintain current threat intelligence
- Conduct regular focused red team assessments
+**Attack Vector Summary:**
+Attackers manipulate the model's "memory" (context window, RAG database, or system prompt) to establish a lasting influence that survives across individual queries or sessions.
+
+**Key Detection Indicators:**
+
+- **Topic Drift**: The model starts mentioning topics (e.g., "crypto," "support") that weren't in the user prompt.
+- **Persona Locking**: The model refuses to exit a specific role (e.g., "I can only answer as DAN").
+- **Injection Artifacts**: Weird phrases appearing in output ("Ignored previous instructions").
+- **High Entrop**: Sudden changes in perplexity or output randomness.
+
+**Primary Mitigation:**
+
+- **Context Resets**: Hard reset of conversation history after N turns or upon detecting sensitive topics.
+- **Instruction Hierarchy**: Explicitly marking System Prompts as higher priority than User Prompts (e.g., `<system>` tags in ChatML).
+- **Output Validation**: Checking if the model is following a specific format, independent of the input.
+- **Sandboxing**: Preventing the LLM from writing to its own long-term memory or system instructions.
+
+**Severity**: High (Can lead to total system compromise via RAG/Agents)
+**Ease of Exploit**: Medium (Requires understanding of model attention/context)
+**Common Targets**: Customer Support Bots (Session Hijacking), RAG Search Tools (Poisoning).
+
+---

 ### Pre-Engagement Checklist

 **Administrative:**

 - [ ] Obtain written authorization
- [ ] Review and sign SOW  
- [ ] Define scope and rules of engagement
+- [ ] Review and sign SOW
+- [ ] Define scope (Are we allowed to poison the RAG DB?)
 - [ ] Set up communication channels

 **Technical Preparation:**

- [ ] Set up isolated test environment
- [ ] Install testing tools and frameworks
- [ ] Prepare payload library
- [ ] Configure logging and evidence collection
+- [ ] Map the application's "Memory" architecture (Context window size? Vector DB?)
+- [ ] Identify input sources (User chat? Email? PDF uploads?)
+- [ ] Prepare payload library (Standard injections + Stealth variants)
+- [ ] Configure logging

 ### Post-Engagement Checklist

 **Documentation:**

- [ ] Document findings with reproduction steps
- [ ] Capture evidence and logs
+- [ ] Document successful injection chains
+- [ ] Capture evidence (screenshots of persistent malicious behavior)
 - [ ] Prepare technical report
 - [ ] Create executive summary

 **Cleanup:**

- [ ] Remove test artifacts
- [ ] Verify no persistent changes
- [ ] Securely delete files
+- [ ] **CRITICAL**: Purge any poisoned data from Vector DBs or RAG systems.
+- [ ] Reset all session memories.
+- [ ] Securely delete files.

 **Reporting:**

 - [ ] Deliver comprehensive report
 - [ ] Provide prioritized remediation guidance
 - [ ] Schedule re-testing
-
---
@@ -1,4 +1,15 @@
-# Chapter 24: Social Engineering with LLM
+<!--
+Chapter: 24
+Title: Social Engineering with LLMs
+Category: Social Engineering
+Difficulty: Intermediate
+Estimated Time: 45 minutes read time
+Hands-on: Yes - AI Phishing Generator and Impersonation Framework
+Prerequisites: Chapter 14 (Prompt Injection)
+Related: Chapter 23 (Persistence), Chapter 26 (Agents)
+-->
+
+# Chapter 24: Social Engineering with LLMs

 ![ ](assets/page_header.svg)

@@ -51,6 +62,32 @@ This comprehensive chapter covers AI-generated phishing attacks, impersonation t

 ---

+---
+
+### Theoretical Foundation
+
+**Why This Works (Model Behavior):**
+
+Social Engineering with LLMs isn't about hacking the _model_; it's about using the model to hack the _human_. This works because LLMs are excellent at "Simulation."
+
+- **Architectural Factor (Theory of Mind Simulation):** LLMs are trained on vast amounts of human dialogue (novels, emails, Reddit). This allows them to effectively simulate "Theory of Mind"—predicting what a human expects to hear, what will make them trust a sender, and what emotional triggers (urgency, fear) will cause a reaction.
+
+- **Training Artifact (Persuasion Automation):** Models are fine-tuned to be convincing and helpful. When directed to be "persuasive," they can weaponize Cialdini's Principles of Influence (Reciprocity, Scarcity, Authority) with superhuman consistency and optimized vocabulary capable of bypassing human skepticism.
+
+- **Input Processing (Contextual Adaptation):** Unlike a static phishing template, an LLM can parse a target's LinkedIn profile or recent tweets (Input) and dynamically adjust the "Attack Payload" (the email body) to match the target's current context (e.g., "I saw your post about the conference..."), drastically increasing the success rate.
+
+**Foundational Research:**
+
+| Paper                                                                                                              | Key Finding                                               | Relevance                                                      |
+| ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- | -------------------------------------------------------------- |
+| [Wei et al. "Jailbroken: How Does LLM Safety Training Fail?"](https://arxiv.org/abs/2307.02483)                    | Analyzed how safety training competes with helpfulness.   | Explains why models can be tricked into generating scams.      |
+| [Hazell "Large Language Models can be used to effectively scale spear phishing"](https://arxiv.org/abs/2305.06972) | Demonstrated that LLMs produce cheaper, better phishing.  | Validated the economic argument for AI-driven fraud.           |
+| [Anthropic "Many-Shot Jailbreaking"](https://www.anthropic.com/research/many-shot-jailbreaking)                    | Showed how long-context priming overcomes safety filters. | Relevant for generating complex social engineering narratives. |
+
+**What This Reveals About LLMs:**
+
+LLMs are "Cognitive Amplifiers." They don't just generate text; they generate _influence_. A low-skill attacker can now operate with the sophistication of a state-sponsored actor because the LLM bridges the gap in language fluency and psychological manipulation.
+
 ## 24.1 AI-Generated Phishing

 **What is AI-Generated Phishing:**
@@ -994,178 +1031,114 @@ message = framework.generate_impersonation_message(

 ---

-## 24.16 Summary and Key Takeaways
+---

-### Critical Social Engineering Techniques
+## 24.17 Research Landscape

-**Most Effective LLM-Powered Attacks:**
+**Seminal Papers:**

-1. **AI-Generated Phishing** (2-5% success rate vs 0.1% traditional)
+| Paper                                                                                              | Year | Venue | Contribution                                                                        |
+| -------------------------------------------------------------------------------------------------- | ---- | ----- | ----------------------------------------------------------------------------------- |
+| [Salvi et al. "On the Robustness of LLMs to Social Engineering"](https://arxiv.org/abs/2308.11475) | 2023 | ArXiv | Evaluated how easily LLMs fall for social engineering themselves.                   |
+| [Karanjai "Targeted Phishing Campaigns using LLM"](https://arxiv.org/abs/2311.01234)               | 2023 | IEEE  | Technical framework for automating targeted attacks.                                |
+| [Zou et al. "Universal and Transferable Adversarial Attacks"](https://arxiv.org/abs/2307.15043)    | 2023 | ArXiv | (The "Suffix" paper) - Relevant for bypassing safety filters to generate the phish. |

-   - Perfect grammar, contextual personalization
-   - 10,000+ emails per day per attacker
-   - Psychological trigger automation
+**Evolution of Understanding:**

-2. **CEO Fraud / BEC** (12-18% success, $130K avg loss)
+- **2022**: "Text Generation" (Spam bots).
+- **2023**: "Contextual Spear Phishing" (LinkedIn scraping + GPT-4).
+- **2024**: "Interactive Voice/Video Fraud" (Deepfake CEO calls + Real-time audio generation).

-   - Executive impersonation
-   - Authority + urgency exploitation
-   - Wire transfer fraud
+**Current Research Gaps:**

-3. **Spear Phishing at Scale** (3-7% success)
-   - Individualized Target profiling
-   - Mass personalization
-   - Multi-vector campaigns
+1.  **Voice Authenticity Detection**: Real-time detection of synthetic audio artifacts over phone lines.
+2.  **Cognitive Resilience**: Training _humans_ to detect AI-generated text patterns (which are becoming indistinguishable).
+3.  **Watermarking**: Reliable invisible watermarking for all AI-generated content (text/audio/video) to prove provenance.

-### Defense Recommendations
+**Recommended Reading:**

-**For Organizations:**
+**For Practitioners:**

-1. **Technical Controls**
-
-   - Email authentication (DMARC reject)
-   - AI-powered phishing detection
-   - Multi-factor authentication
-   - Dual-approval for financial transactions
-
-2. **Process Controls**
-
-   - Verbal verification for unusual requests
-   - Out-of-band confirmations
-   - Code word systems for wire transfers
-   - Anomaly detection
-
-3. **User Training**
-   - Simulated phishing exercises
-   - Red flag recognition
-   - Psychological trigger awareness
-   - Reporting procedures
-
-**For Individuals:**
-
-1. **Verify Everything**
-
-   - Call back on known numbers
-   - Never trust urgent email requests
-   - Check sender carefully
-   - Question authority claims
-
-2. **Recognize Manipulation**
-   - Urgency = red flag
-   - Authority pressure = verify
-   - Too good/urgent = suspicious
-   - Request verification = legitimate
-
-### Case Studies
-
-**BEC Attack via LLM (2023):**
-
- Method: AI-generated CEO impersonation
- Target: CFO of manufacturing company
- Loss: $243,000 wire transfer
- Lesson: Verbal verification mandatory
-
-**Automated Phishing Campaign (2024):**
-
- Method: 50,000 personalized emails in 8 hours
- Targets: Healthcare industry
- Success: 1,200 credential harvests (2.4%)
- Lesson: AI detection systems critical
-
-### Ethical and Legal Considerations
-
-**Legal Risks:**
-
- **Phishing**: Computer Fraud and Abuse Act, CAN-SPAM Act
- **Impersonation**: Wire Fraud (18 USC § 1343), Identity Theft
- **BEC**: Federal wire fraud, potential RICO charges
- **Penalties**: Up to 20 years federal prison + fines
-
-**Ethical Use:**
-
- ✅ Authorized penetration testing with written SOW
- ✅ Security awareness training (with disclosure)
- ✅ Research in controlled environments
- ❌ Unauthorized phishing campaigns
- ❌ Actual fraud or credential theft
- ❌ Testing without explicit permission
-
-### Future Trends
-
-**Emerging Threats:**
-
- Voice + text impersonation (deepfake audio)
- Real-time conversational attacks (AI chatbots)
- Cross-platform campaigns (email + SMS + social)
- Adaptive learning (AI optimizes based on responses)
-
-**Defense Evolution:**
-
- LLM-vs-LLM detection
- Behavioral biometrics
- Zero-trust verification
- Blockchain-based email authentication
+- **Report**: [Europol: ChatGPT - the impact of Large Language Models on Law Enforcement](https://www.europol.europa.eu/publications-events/publications/chatgpt-impact-of-large-language-models-law-enforcement)
+- **Tool**: [Gophish](https://getgophish.com/) - The standard open-source phishing framework (integrate with LLMs for testing).

 ---

-## 24.17 Conclusion
+## 24.18 Conclusion

-**Key Takeaways:**
+> [!CAUTION] > **Social Engineering is Hazardous.** Testing these techniques involves targeting _people_, not just machines. You must have explicit, written permission to target specific individuals. Never target personal accounts, family members, or use "fear-based" pretexts (e.g., "your child is in the hospital") even if they work. Psychological harm is real harm.

-1. Understanding this attack category is essential for comprehensive LLM security
-2. Traditional defenses are often insufficient against these techniques  
-3. Testing requires specialized knowledge and systematic methodology
-4. Effective protection requires ongoing monitoring and adaptation
+AI has not invented new social engineering attacks; it has _democratized_ the most advanced ones. The barrier to entry for high-quality, multilingual, context-aware fraud has collapsed to zero.

-**Recommendations for Red Teamers:**
+For Red Teamers, the focus shifts from "Can I trick the user?" to "Can I bypass the AI safety filter to _generate_ the trick?" and "Can the organization's technical controls (MFA, DMARC) withstand a 100x increase in convincing volume?"

- Develop comprehensive test cases covering all attack variants
- Document both successful and failed attempts
- Test systematically across models and configurations
- Consider real-world scenarios and attack motivations
+**Next Steps:**

-**Recommendations for Defenders:**
+- **Chapter 25**: Advanced Adversarial ML - The math behind the magic.
+- **Chapter 26**: Autonomous Agents - When the attack runs itself.

- Implement defense-in-depth with multiple layers
- Monitor for anomalous attack patterns
- Maintain current threat intelligence
- Conduct regular focused red team assessments
+---
+
+## Quick Reference
+
+**Attack Vector Summary:**
+Attackers leverage LLMs to automate the creation of highly personalized, persuasive, and grammatically perfect phishing content (text, audio, code) at a scale previously impossible.
+
+**Key Detection Indicators:**
+
+- **Perfection**: Text has zero grammar errors but feels slightly "stiff" or "over-formal."
+- **Generic Urgent Context**: "Due diligence," "Q3 Report," "Compliance Update" - generic business themes used as hooks.
+- **Unnatural Speed**: Reply times that are inhumanly fast for complex queries (in chat contexts).
+- **Audio Artifacts**: In voice calls, lack of breath sounds, constant cadence, or metallic clipping.
+
+**Primary Mitigation:**
+
+- **FIDO2 / WebAuthn**: Physical security keys (YubiKey) are immune to phishing, regardless of how convincing the email is.
+- **Verification Protocols**: "Call the sender" (Out-of-Band verification) for all financial requests.
+- **AI-Based Filtering**: Using an LLM to detect LLM-generated phishing (fighting fire with fire).
+- **Identity Proofing**: Digital signatures for internal executive comms (S/MIME).
+
+**Severity**: Critical (Primary vector for Ransomware/Data Breach)
+**Ease of Exploit**: High (Tools are widely available)
+**Common Targets**: HR (Resume attachments), Finance (Invoice fraud), IT Helpdesk (Password resets).
+
+---

 ### Pre-Engagement Checklist

 **Administrative:**

- [ ] Obtain written authorization
- [ ] Review and sign SOW  
- [ ] Define scope and rules of engagement
- [ ] Set up communication channels
+- [ ] **CRITICAL**: Written authorization listing SPECIFIC targets (no "open fire").
+- [ ] Review scope: Phishing? Vishing? SMS?
+- [ ] Whitelist attack IP/Domains in security tools (to test _user_ awareness, not _filter_ settings, if agreed).
+- [ ] Define "Stop" criteria (e.g., target distress).

 **Technical Preparation:**

- [ ] Set up isolated test environment
- [ ] Install testing tools and frameworks
- [ ] Prepare payload library
- [ ] Configure logging and evidence collection
+- [ ] Register look-alike domains.
+- [ ] Set up SMTP relays / Phishing framework (Gophish).
+- [ ] Integrate LLM API for dynamic content generation.
+- [ ] Verify DMARC/SPF status of spoofing targets.

 ### Post-Engagement Checklist

 **Documentation:**

- [ ] Document findings with reproduction steps
- [ ] Capture evidence and logs
- [ ] Prepare technical report
- [ ] Create executive summary
+- [ ] Anonymize victim data in reports (use IDs, not names).
+- [ ] Document click rates, credential capture rates (stats only).
+- [ ] Prepare technical report.
+- [ ] Create executive summary.

 **Cleanup:**

- [ ] Remove test artifacts
- [ ] Verify no persistent changes
- [ ] Securely delete files
+- [ ] **Destroy captured credentials immediately** (do not store).
+- [ ] Decommission phishing domains.
+- [ ] Send "Teachable Moment" email to victims (if part of campaign).

 **Reporting:**

- [ ] Deliver comprehensive report
- [ ] Provide prioritized remediation guidance
- [ ] Schedule re-testing
+- [ ] Deliver comprehensive report.
+- [ ] Provide prioritized remediation guidance (Technical controls > User training).
+- [ ] Schedule re-testing.

 ---