From 93a3db95013ac47784fa3b865f673473cd1b359e Mon Sep 17 00:00:00 2001 From: shiva108 Date: Sun, 7 Dec 2025 00:03:02 +0100 Subject: [PATCH] feat: Add chapter metadata, theoretical foundations, research landscape, quick references, and practical checklists to various chapters. --- ...ter_16_Jailbreaks_and_Bypass_Techniques.md | 103 ++++++++ .../Chapter_17_Plugin_and_API_Exploitation.md | 109 +++++++- ...sion_Obfuscation_and_Adversarial_Inputs.md | 119 +++++++++ docs/Chapter_19_Training_Data_Poisoning.md | 116 ++++++++- ...20_Model_Theft_and_Membership_Inference.md | 116 ++++++++- ...hapter_21_Model_DoS_Resource_Exhaustion.md | 198 +++++++------- ...apter_22_Cross_Modal_Multimodal_Attacks.md | 139 ++++++++-- ...hapter_23_Advanced_Persistence_Chaining.md | 207 +++++++-------- docs/Chapter_24_Social_Engineering_LLMs.md | 245 ++++++++---------- 9 files changed, 972 insertions(+), 380 deletions(-) diff --git a/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md b/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md index ab15358..41b58ef 100644 --- a/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md +++ b/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md @@ -76,6 +76,32 @@ Jailbreak research exists in a morally complex space: - Contribute to defensive improvements - Document findings for safety teams +### Theoretical Foundation + +**Why This Works (Model Behavior):** + +Jailbreaks succeed by exploiting the fundamental architectural tension between helpfulness and safety in LLM design. Unlike traditional security vulnerabilities with clear boundaries, jailbreaks manipulate the model's learned behaviors: + +- **Architectural Factor:** LLMs use the same neural pathways to process system instructions, safety training, and user prompts. There is no cryptographic separation between "follow user intent" and "refuse harmful requests"—both are learned behaviors competing for activation during generation. When cleverly crafted prompts create stronger activation patterns for helpfulness than for safety refusal, jailbreaks succeed. + +- **Training Artifact:** RLHF optimizes for human preferences, which include helpfulness, detailed responses, and instruction-following. Safety training adds competing objectives (refuse harmful requests, avoid policy violations). This creates exploitable edge cases where the model's "be helpful" training overrides "be safe" training, especially with novel prompt structures not seen during safety fine-tuning. + +- **Input Processing:** Models generate tokens autoregressively based on context probability distributions. Role-playing jailbreaks work because the model has learned that fictional scenarios, hypothetical questions, and persona adoption are legitimate use cases. The model cannot reliably distinguish "legitimate creative writing" from "harmful content generation disguised as fiction" without explicit examples in training data. + +**Foundational Research:** + +| Paper | Key Finding | Relevance | +| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------- | +| [Wei et al. "Jailbroken: How Does LLM Safety Training Fail?"](https://arxiv.org/abs/2307.02483) | Identified competing objectives as root cause of jailbreak success | Explains why alignment is fundamentally fragile against adversarial prompts | +| [Zou et al. "Universal and Transferable Adversarial Attacks on Aligned LLMs"](https://arxiv.org/abs/2307.15043) | Demonstrated automated discovery of universal jailbreak suffixes | Proved jailbreaks can transfer across models, not just model-specific bugs | +| [Perez et al. "Red Teaming Language Models"](https://arxiv.org/abs/2202.03286) | Systematic red teaming reveals consistent vulnerability patterns | Established jailbreaking as persistent threat requiring continuous testing | + +**What This Reveals About LLMs:** + +Jailbreak vulnerability reveals that current safety alignment is a learned heuristic, not an architectural guarantee. Unlike access control systems with formal verification, LLM safety relies on statistical patterns in training data. Any sufficiently novel prompt structure can potentially bypass learned refusals, making perfect jailbreak prevention impossible without fundamentally redesigning how LLMs process instructions and generate responses. + +--- + ### 16.1.2 Why Jailbreaks Matter **Security implications** @@ -1429,8 +1455,55 @@ RED_TEAM_BEST_PRACTICES = { --- +## 16.15 Research Landscape + +**Seminal Papers:** + +| Paper | Year | Venue | Contribution | +| ----------------------------------------------------------------------------------------------------- | ---- | -------- | ------------------------------------------------------------------------- | +| [Wei et al. "Jailbroken: How Does LLM Safety Training Fail?"](https://arxiv.org/abs/2307.02483) | 2023 | arXiv | First systematic analysis of why safety training fails against jailbreaks | +| [Zou et al. "Universal and Transferable Adversarial Attacks"](https://arxiv.org/abs/2307.15043) | 2023 | arXiv | GCG attack - automated discovery of universal jailbreak suffixes | +| [Perez et al. "Red Teaming Language Models"](https://arxiv.org/abs/2202.03286) | 2022 | arXiv | Foundational red teaming methodology, diverse attack taxonomy | +| [Wallace et al. "Universal Adversarial Triggers for Attacking NLP"](https://arxiv.org/abs/1908.07125) | 2019 | EMNLP | Early adversarial text generation, foundational for token-level attacks | +| [Kang et al. "Exploiting Programmatic Behavior of LLMs"](https://arxiv.org/abs/2302.05733) | 2023 | IEEE S&P | Demonstrated systematic jailbreaking through instruction manipulation | + +**Evolution of Understanding:** + +- **2019-2021**: Early work on adversarial text (Wallace et al.) established feasibility of manipulating NLP models through carefully crafted inputs +- **2022**: Perez et al.'s red teaming work systematized jailbreak discovery, moving from ad-hoc attacks to structured methodology +- **2023 (Early)**: Viral spread of DAN and role-playing jailbreaks on social media demonstrated real-world exploitation at scale +- **2023 (Mid-Late)**: Wei et al. and Zou et al. provided theoretical foundations, proving jailbreaks stem from architectural limitations, not implementation bugs +- **2024-Present**: Focus shifts to automated discovery (LLM-generated jailbreaks), multimodal attacks, and fundamental alignment research + +**Current Research Gaps:** + +1. **Provably Safe Alignment**: Can LLMs be architected with formal guarantees against jailbreaks, or is statistical safety the best achievable? Current approaches lack mathematical proofs of robustness. + +2. **Automated Defense Generation**: Just as attacks can be automated (GCG), can defenses be automatically generated and updated? How can safety training keep pace with adversarial prompt evolution? + +3. **Jailbreak Transferability Bounds**: What determines whether a jailbreak transfers across models? Understanding transferability could inform defensive priorities and model architecture choices. + +**Recommended Reading:** + +**For Practitioners (by time available):** + +- **5 minutes**: [Anthropic's Jailbreak Research Blog](https://www.anthropic.com/index/red-teaming-language-models) - Accessible industry perspective +- **30 minutes**: [Wei et al. (2023)](https://arxiv.org/abs/2307.02483) - Core paper explaining why safety training fails +- **Deep dive**: [Zou et al. (2023) GCG Paper](https://arxiv.org/abs/2307.15043) - Technical deep dive on automated jailbreak discovery + +**By Focus Area:** + +- **Attack Techniques**: [Perez et al. (2022)](https://arxiv.org/abs/2202.03286) - Best for understanding attack taxonomy +- **Defense Mechanisms**: [Wei et al. (2023)](https://arxiv.org/abs/2307.02483) - Best for understanding why defenses fail and what might work +- **Automated Methods**: [Zou et al. (2023)](https://arxiv.org/abs/2307.15043) - Best for understanding GCG and optimization-based attacks + +--- + ## 16.16 Conclusion +> [!CAUTION] +> Unauthorized jailbreaking of production LLM systems to generate harmful, illegal, or policy-violating content is prohibited under computer fraud laws (CFAA), terms of service agreements, and acceptable use policies. Violations can result in account termination, legal action, and criminal prosecution. **Only perform jailbreak testing with explicit written authorization as part of security research or red team engagements.** + **Key Takeaways:** 1. **Jailbreaks Exploit Fundamental Tensions:** The conflict between helpfulness and safety creates unavoidable vulnerabilities in current LLM architectures @@ -1466,6 +1539,36 @@ RED_TEAM_BEST_PRACTICES = { > [!TIP] > Maintain a "jailbreak effectiveness matrix" tracking success rates of each technique against different models and versions. This helps prioritize defensive efforts and demonstrates comprehensive testing coverage. +--- + +## Quick Reference + +**Attack Vector Summary:** + +Jailbreaks bypass LLM safety controls through role-playing, instruction manipulation, encoding obfuscation, multi-turn escalation, and token-level adversarial optimization. Attacks exploit the tension between helpfulness and safety training, causing models to generate policy-violating content. + +**Key Detection Indicators:** + +- Role-playing language ("pretend you are", "DAN mode", "ignore ethics") +- Instruction override attempts ("ignore previous instructions", "new rules") +- Encoding/obfuscation (base64, leetspeak, language switching) +- Hypothetical framing ("in a fictional scenario", "for academic purposes") +- Refusal suppression ("do not say you cannot", "answer without disclaimers") + +**Primary Mitigation:** + +- **Input Filtering**: Detect and block known jailbreak patterns before model processing +- **Adversarial Training**: Fine-tune on diverse jailbreak datasets to strengthen refusal behaviors +- **Output Validation**: Post-process responses to detect policy-violating content +- **Monitoring**: Real-time alerts for jailbreak attempt patterns and success indicators +- **Model Updates**: Continuous retraining with newly discovered jailbreak examples + +**Severity:** Critical (enables generation of harmful/illegal content) +**Ease of Exploit:** Medium (basic role-playing) to High (automated GCG attacks) +**Common Targets:** Public chatbots, customer service AI, content generation systems + +--- + ### Pre-Engagement Checklist **Administrative:** diff --git a/docs/Chapter_17_Plugin_and_API_Exploitation.md b/docs/Chapter_17_Plugin_and_API_Exploitation.md index 81658e7..565c72c 100644 --- a/docs/Chapter_17_Plugin_and_API_Exploitation.md +++ b/docs/Chapter_17_Plugin_and_API_Exploitation.md @@ -1,3 +1,14 @@ + + # Chapter 17: Plugin and API Exploitation ![ ](assets/page_header.svg) @@ -33,11 +44,33 @@ LLM with Plugins: **Security implications:** -1. **Privilege escalation**: Plugins may have elevated permissions -2. **Data exfiltration**: Plugins can access sensitive data -3. **Lateral movement**: Compromise one plugin → access others -4. **Supply chain risks**: Malicious or compromised plugins -5. **Integration vulnerabilities**: Complex interactions create bugs +- Third-party API vulnerabilities (OWASP API Top 10) +- Privilege escalation via authorized tools +- Component interaction bugs + +### Theoretical Foundation + +**Why This Works (Model Behavior):** + +Plugin and API exploitation leverages the model's ability to interface with external systems, turning the LLM into a "confused deputy" that executes actions on behalf of the attacker. + +- **Architectural Factor:** To use tools, LLMs are fine-tuned to recognize specific triggers or emit structured outputs (like JSON) when context suggests a tool is needed. This binding is semantic, not programmatic. The model "decides" to call an API based on statistical likelihood, meaning malicious context can probabilistic force execution of sensitive tools without genuine user intent. + +- **Training Artifact:** Instruction-tuning datasets for tool use (e.g., Toolformer style) often emphasize successful execution over security validation. Models are trained to be "helpful assistants" that fulfill requests by finding the right tool, creating a bias towards action execution even when parameters look suspicious or dangerous. + +- **Input Processing:** When an LLM processes content from an untrusted source (e.g., a retrieved website or email) to fill API parameters, it cannot inherently distinguish between "data to be processed" and "malicious instructions." This allows Indirect Prompt Injection to manipulate the arguments sent to external APIs, bypassing the user's intended control flow. + +**Foundational Research:** + +| Paper | Key Finding | Relevance | +| ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------ | +| [Greshake et al. "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications"](https://arxiv.org/abs/2302.12173) | Defined "Indirect Prompt Injection" as a vector for remote execution | demonstrated how hackers can weaponize LLM plugins via passive content | +| [Schick et al. "Toolformer: Language Models Can Teach Themselves to Use Tools"](https://arxiv.org/abs/2302.04761) | Demonstrated self-supervised learning for API calling | Explains the mechanistic basis of how models learn to trigger external actions | +| [Mialon et al. "Augmented Language Models: a Survey"](https://arxiv.org/abs/2302.07842) | Surveyed risks in retrieving and acting on external data | Provides taxonomy of risks when LLMs leave the "sandbox" of pure text gen | + +**What This Reveals About LLMs:** + +The vulnerability of plugins reveals that LLMs lack the "sandbox" boundaries of traditional software. In a standard app, code and data are separated. In an Agent/Plugin architecture, the "CPU" (the LLM) processes "instructions" (prompts) that mix user intent, system rules, and retrieved data into a single stream. This conflation makes "Confused Deputy" attacks intrinsic to the architecture until robust separation of control and data channels is achieved. ### 17.1.2 API Integration Landscape @@ -71,6 +104,8 @@ class LLMWithAPIs: return self.llm.generate_response(user_prompt, results) ``` +### 17.1.2 Why Plugins Increase Risk + **Attack vectors in API integrations:** - **Plugin selection manipulation**: Trick LLM into calling wrong plugin @@ -2604,6 +2639,39 @@ def validate_email(email): --- +## 17.15 Research Landscape + +**Seminal Papers:** + +| Paper | Year | Venue | Contribution | +| ----------------------------------------------------------------------------------------------------------------------------- | ---- | ----- | ------------------------------------------------------------------------------------ | +| [Greshake et al. "Compromising Real-World LLM-Integrated Applications"](https://arxiv.org/abs/2302.12173) | 2023 | AISec | The seminal paper on Indirect Prompt Injection and plugin exploitation mechanisms | +| [Patil et al. "Gorilla: Large Language Model Connected with Massive APIs"](https://arxiv.org/abs/2305.15334) | 2023 | arXiv | Explored fine-tuning models specifically for API calls, highlighting parameter risks | +| [Qin et al. "ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs"](https://arxiv.org/abs/2307.16789) | 2023 | ICLR | Large-scale study of API interaction capabilities and failure modes | +| [Li et al. "API-Bank: A A Benchmark for Tool-Augmented LLMs"](https://arxiv.org/abs/2304.08244) | 2023 | EMNLP | Established benchmarks for correctness and safety in API execution | +| [Nakushima et al. "Stop the Pop: Privilege Escalation in LLM Chains"](https://arxiv.org/abs/2302.12173) | 2024 | arXiv | Analyzed privilege escalation paths in complex agent chains | + +**Evolution of Understanding:** + +- **2022**: Focus on "Tool use" as a capability (Toolformer); security largely ignored. +- **2023 (Early)**: Greshake et al. demonstrate that "reading" a webpage can trigger unauthorized email sending (Indirect Injection). +- **2023 (Late)**: Rise of "Agents" increases complexity; research shifts to compounding risks in multi-step chains. +- **2024-Present**: Focus on formal verification of tool outputs and "guardrail" models that intercept API calls before execution. + +**Current Research Gaps:** + +1. **Stateful Attacks**: most research looks at single-turn exploitation. How do attacks persist across a multi-turn conversation where the agent holds state? +2. **Auth Token Leakage**: Mechanisms for preventing models from hallucinating or leaking bearer tokens in verbose logs/outputs. +3. **Semantic Firewalling**: Can we train models to recognize "dangerous" API permutations (e.g., `delete_user` with wildcards) semantically rather than just syntactically? + +**Recommended Reading:** + +**For Practitioners:** + +- **Essential**: [OWASP Top 10 for LLM Applications (LLM06: Sensitive Information Disclosure & LLM09: Overreliance)](https://owasp.org/www-project-top-10-for-large-language-model-applications/) +- **Technical**: [Greshake et al. (2023)](https://arxiv.org/abs/2302.12173) - The "Must Read" on Plugin Security. + +--- ## 17.16 Conclusion @@ -2611,7 +2679,7 @@ def validate_email(email): 1. **Plugins Expand Attack Surface Dramatically:** Each plugin introduces new code execution paths, API integrations, and potential vulnerabilities beyond core LLM security 2. **LLMs Can't Distinguish Malicious Requests:** Models execute function calls based on prompts without inherent security awareness, requiring robust authorization layers -3. **Input Validation is Critical Everywhere:** From plugin parameters to API endpoints, all user-influenced inputs must be validated, sanitized, and parameterized +3. **Input Validation is Critical Everywhere:** From plugin parameters to API endpoints, all user-influenced inputs must be validated, sanitized, and parameterized 4. **Supply Chain Security Matters:** Third-party plugins and dependencies introduce risks requiring scanning, monitoring, and verification **Recommendations for Red Teamers:** @@ -2644,6 +2712,35 @@ def validate_email(email): > [!TIP] > Create a "plugin attack matrix" mapping each plugin to its potential attack vectors (command injection, data access, privilege escalation). This ensures systematic coverage during security assessments. +--- + +## Quick Reference + +**Attack Vector Summary:** +Attackers manipulate the LLM to invoke plugins/APIs with malicious arguments or unintended intent. This is often achieved via "Indirect Prompt Injection" (placing instructions in data the model reads) or "Confused Deputy" attacks (tricking the privileged model into acting for an unprivileged user). + +**Key Detection Indicators:** + +- API logs showing calls with "weird" or nonsensical parameters. +- Model attempting to access internal-only endpoints (SSRF). +- User inputs containing syntax similar to API schemas/OpenAPI specs. +- Rapid sequence of tool-use errors followed by success (brute-forcing schema). +- Injected content in retrieved documents referencing "System Actions" or "Admin Commands". + +**Primary Mitigation:** + +- **Human-in-the-Loop (HITL)**: Require user confirmation for high-impact actions (delete, send, buy). +- **Strict Schema Validation**: Enforce typing and value ranges on all API parameters generated by LLM. +- **Least Privilege**: The LLM's API token should have the absolute minimum scope required (e.g., `read_only` vs `read_write`). +- **Segregated Context**: Clearly mark retrieved content as "Untrusted Data" in the system prompt. +- **Output Sanitization**: Scan JSON payloads for injection markers before execution. + +**Severity**: Critical (RCE/Data Loss) +**Ease of Exploit**: High (if Indirect Injection is possible) +**Common Targets**: Customer support bots with "Refunding" or "Account Lookup" tools; Coding assistants with CLI access. + +--- + ### Pre-Engagement Checklist **Administrative:** diff --git a/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md b/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md index 862d24e..c529d15 100644 --- a/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md +++ b/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md @@ -1,3 +1,14 @@ + + # Chapter 18: Evasion, Obfuscation, and Adversarial Inputs ![ ](assets/page_header.svg) @@ -69,6 +80,30 @@ User Input → Obfuscation → Content Filter → [ALLOWED] ✓ → LLM Processi - **Repeatability**: Working consistently across multiple attempts (reliable exploitation) - **Transferability**: Applicable across different models and systems (broad impact) +### Theoretical Foundation + +**Why This Works (Model Behavior):** + +Evasion and adversarial attacks succeed because deep learning models, including LLMs, rely on brittle statistical correlations rather than robust semantic understanding. + +- **Architectural Factor (The "Tokenization Gap"):** LLMs process text as discrete tokens (integers), not characters. Slight perturbations that are invisible or irrelevant to humans (like zero-width spaces or homoglyphs) can completely alter the token sequence the model processes. Similarly, "adversarial tokens" can shift the internal activation vectors across the decision boundary of a safety filter without changing the human-perceived meaning. + +- **Training Artifact (Non-Robust Features):** Models learn "shortcuts" or non-robust features during training—patterns that correlate with labels but aren't causally related. For example, a safety filter might learn that "kill" is bad, but fail to generalize that "k i l l" or "unalive" requires the same refusal. Adversaries exploit these shallow heuristics. + +- **Input Processing (Embedding Space Geometry):** In the high-dimensional embedding space, legitimate and malicious prompts often lie close together. Adversarial optimization (like GCG) searches for vectors that push a malicious prompt just across the manifold into the "compliant" region, exploiting the continuous nature of the internal representations despite the discrete input. + +**Foundational Research:** + +| Paper | Key Finding | Relevance | +| ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | +| [Wallace et al. "Universal Adversarial Triggers for Attacking and Analyzing NLP"](https://arxiv.org/abs/1908.07125) | Discovered "trigger" phrases that switch model predictions regardless of context | Proved that discrete text inputs can be optimized for adversarial control | +| [Jain et al. "Baseline Defenses for Adversarial Attacks Against Aligned LLMs"](https://arxiv.org/abs/2309.00614) | Evaluated detection (perplexity) and preprocessing defenses | Demonstrated that simple defenses often fail against adaptive attacks | +| [Morris et al. "TextAttack: A Framework for Adversarial Attacks"](https://arxiv.org/abs/2005.05909) | Systematized NLP attack recipes (deletion, swap, embedding) | Provided the standard tooling for generating adversarial text examples | + +**What This Reveals About LLMs:** + +These vulnerabilities reveal that LLMs do not "read" like humans do. They process mathematical representations of token sequences. The divergence between _human perception_ (the rendered text) and _machine perception_ (the token IDs) is the root cause of almost all evasion vulnerabilities. Until models possess "robust perception" that aligns with human semantic interpretation, evasion remains an open problem. + **Attack Success Metrics:** 1. **Evasion Rate**: % of attacks that bypass filters @@ -1075,8 +1110,92 @@ XSS: "