diff --git a/docs/AI LLM Red Team Handbook.md b/docs/AI LLM Red Team Handbook.md index 2debd92..e75b7c3 100644 --- a/docs/AI LLM Red Team Handbook.md +++ b/docs/AI LLM Red Team Handbook.md @@ -13429,78 +13429,6 @@ Layer 5: Governance --- -## 15.16 References and Further Reading - -### Academic Papers on Data Extraction - -1. Carlini, N., et al. (2021). "Extracting Training Data from Large Language Models." _USENIX Security_. - -2. Carlini, N., et al. (2023). "Extracting Training Data from ChatGPT." _arXiv preprint arXiv:2311.17035_. - -3. Shokri, R., et al. (2017). "Membership Inference Attacks Against Machine Learning Models." _IEEE S&P_. - -4. Fredrikson, M., et al. (2015). "Model Inversion Attacks that Exploit Confidence Information." _CCS_. - -5. Abadi, M., et al. (2016). "Deep Learning with Differential Privacy." _CCS_. - -### Industry Standards and Frameworks - -- **OWASP Top 10 for LLMs**: -- **NIST AI Risk Management Framework**: -- **MITRE ATLAS**: Adversarial Threat Landscape for AI Systems -- **ISO/IEC 27001**: Information security management -- **SOC 2**: Trust service criteria for data security - -### Tools and Resources - -**Open-source tools**: - -- Garak: LLM vulnerability scanner -- PromptInject: Adversarial prompt testing -- Presidio: PII detection and anonymization -- Gitleaks: Secret detection in code -- Opacus: Differential privacy library - -**Commercial solutions**: - -- Robust Intelligence: AI security platform -- HiddenLayer: ML security scanner -- Protect AI: AI/ML security tools -- Calypso AI: AI risk management - -### Community Forums and Discussions - -- **AI Security Discord/Slack communities** -- **r/MachineLearning security discussions** -- **OWASP AI Security Project** -- **MLSecOps community** -- **Responsible AI forums** - -### Recommended Books - -- "Adversarial Machine Learning" by Biggio & Roli -- "Privacy-Preserving Machine Learning" by Liu et al. -- "AI Safety and Security" edited by Yampolskiy -- "The Alignment Problem" by Brian Christian - -### Conferences and Events - -- **DEF CON AI Village** -- **Black Hat AI Security Summit** -- **NeurIPS Security & Privacy Workshop** -- **ICLR workshops on security** -- **RSA Conference AI Security track** - -### Blogs and News - -- Google AI Blog (AI safety research) -- OpenAI Research -- Anthropic Research -- Trail of Bits blog (ML security) -- HiddenLayer blog - ---- - ## Chapter 15 Conclusion Data leakage and extraction represent one of the most significant and persistent security challenges in LLM systems. Unlike traditional software vulnerabilities with clear patches, data baked into model weights cannot simply be "fixed" without retraining. This makes prevention—through rigorous data hygiene, architectural controls, and ongoing monitoring—absolutely critical. @@ -14946,57 +14874,6 @@ RED_TEAM_BEST_PRACTICES = { --- -## 16.16 References and Further Reading - -### Academic Papers - -1. Zou, A., et al. (2023). "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv:2307.15043 - -2. Wei, A., et al. (2023). "Jailbroken: How Does LLM Safety Training Fail?" arXiv:2307.02483 - -3. Perez, F., & Ribeiro, I. (2022). "Ignore Previous Prompt: Attack Techniques For Language Models." arXiv:2211.09527 - -4. Bai, Y., et al. (2022). "Constitutional AI: Harmlessness from AI Feedback." arXiv:2212.08073 - -5. Ganguli, D., et al. (2022). "Red Teaming Language Models to Reduce Harms." arXiv:2209.07858 - -6. Carlini, N., et al. (2023). "Are aligned neural networks adversarially aligned?" arXiv:2306.15447 - -### Industry Reports - -- **OWASP Top 10 for LLM Applications**: -- **NIST AI Risk Management Framework**: -- **MITRE ATLAS**: - -### Tools and Frameworks - -- **Garak**: -- **PromptInject**: -- **PyRIT**: -- **LLM Guard**: - -### Books - -- "Adversarial Machine Learning" by Biggio & Roli -- "AI Safety and Security" edited by Yampolskiy -- "The Alignment Problem" by Brian Christian - -### Community Resources - -- AI Safety Discord servers -- r/ChatGPTJailbreak -- r/LocalLLaMA -- DEF CON AI Village -- Black Hat AI Security Summit - ---- - -**End of Chapter 16: Jailbreaks and Bypass Techniques** - -_This chapter provided comprehensive coverage of jailbreak techniques, from classic role-playing attacks to cutting-edge token-level exploits. Remember: jailbreak research should always be conducted responsibly, with proper authorization, and with the goal of improving AI safety for everyone._ - ---- - # Appendix A: Red Team Tools, Resources, and Further Reading ## A.1 Recommended Red Team Tools for AI and LLMs @@ -15021,28 +14898,6 @@ _This chapter provided comprehensive coverage of jailbreak techniques, from clas --- -## A.2 Essential Reading and References - -### Key Papers - -- Brown, T. B. et al. (2020). "Language Models are Few-Shot Learners (GPT-3)." -- Carlini, N. et al. (2021). "Extracting Training Data from Large Language Models." -- Wei, J. et al. (2023). "PromptBench: Systematic Benchmarking of LLM Vulnerabilities." -- Zou, M. et al. (2023). "Cross-Modal Adversarial Attacks on Vision-Language Models." - -### Reports and Guides - -- **OpenAI Red Teaming Network:** -- **MITRE ATLAS™ Adversarial Threat Landscape for AI Systems:** -- **ENISA AI Threat Landscape:** - -### Standards and Methodologies - -- **NIST AI Risk Management Framework (AI RMF)** -- **ISO/IEC 24029-1:2021** – AI Robustness and Vulnerability Assessment - ---- - ## A.3 Further Learning - **Workshops/Competitions:** diff --git a/docs/AI_LLM Red Team Field Manual.md b/docs/AI_LLM Red Team Field Manual.md index f54ed80..b0540de 100644 --- a/docs/AI_LLM Red Team Field Manual.md +++ b/docs/AI_LLM Red Team Field Manual.md @@ -1039,7 +1039,7 @@ Additional test cases, defenses, and more advanced techniques discussed in Secti ### **3.2 Jailbreaking (Safety Filter Bypass)** -Bypass LLM safety filters using creative framing, role play, or rewording [Reference not found]. +Bypass LLM safety filters using creative framing, role play, or rewording . **Manual Test Prompts**: @@ -1071,7 +1071,7 @@ Further jailbreaking examples, evolving DAN prompts, and defensive measures are ### **3.3 Data Leakage / Memorization** -Probe for training data snippets or private content \[Reference not found\]: +Probe for training data snippets or private content : **Manual Test Prompts**: @@ -1098,7 +1098,7 @@ Further techniques for membership inference, training-data extraction, and RAG d ### **3.4 Plugin/Tool/Function Exploitation** -Abuse external plugin/tool capabilities through LLMs \[Reference not found\]: +Abuse external plugin/tool capabilities through LLMs : **Manual Test Prompts**: @@ -1110,7 +1110,7 @@ Search for and summarize known exploits for \[targeted software\]. ### **3.5 Denial-of-Service (DoS) & Resource Exhaustion** -Trigger heavy computation, loops, or massive outputs \[Reference not found\]: +Trigger heavy computation, loops, or massive outputs : **Manual Test Prompts**: @@ -1138,7 +1138,7 @@ More testing on distributed denial, algorithmic complexity exploits, and cost-am ### **3.6 Adversarial Example Generation (Evasion)** -Craft inputs that evade LLM policies or cause misclassification \[Reference not found\]: +Craft inputs that evade LLM policies or cause misclassification : **Tool – TextAttack:** (for text) @@ -1168,7 +1168,7 @@ TextAttack, Adversarial Robustness Toolbox (ART), adversarial textual perturbati ### **3.7 Data Poisoning (Training-Time Attack)** -Inject malicious or biased data into model training sets \[Reference not found\]: +Inject malicious or biased data into model training sets : **Manual Example:** Add outlier string or phrase in fine-tuning data: @@ -1193,7 +1193,7 @@ Full guidance on backdoor implantation, trigger design, and poisoning-detection ### **3.8 Model Extraction/Stealing** -Replicate model logic/predictions via mass querying \[Reference not found\]: +Replicate model logic/predictions via mass querying : **Manual Example:** @@ -1207,7 +1207,7 @@ pip install \-r requirements.txt ### **3.9 Output Manipulation** -Force output bugs, malformed formats, or payload injection \[Reference not found\]: +Force output bugs, malformed formats, or payload injection : **Manual Test Prompts**: @@ -1216,7 +1216,7 @@ Embed \[harmful script tag\] in your answer. ### **3.10 Side-Channel/Hardware Attacks** -Exploit timing, error messages, and infrastructure observations \[Reference not found\]: +Exploit timing, error messages, and infrastructure observations : **Manual Test Examples:** @@ -1225,7 +1225,7 @@ Send identical queries, log response times, analyze patterns for inference about ### **3.11 Multi-Modal Injection/Cross-Alignment** -Embed triggers in non-text modalities \[Reference not found\]: +Embed triggers in non-text modalities : **Manual Example:** @@ -1233,7 +1233,7 @@ Embed triggers in non-text modalities \[Reference not found\]: ### **3.12 Supply Chain/Infrastructure Attacks** -Tamper with components in the ML pipeline \[Reference not found\]: +Tamper with components in the ML pipeline : **Manual Example:** @@ -1241,7 +1241,7 @@ Tamper with components in the ML pipeline \[Reference not found\]: ### **3.13 Boundary/Format/Fuzz Testing** -Test unhandled or rare input conditions with automated fuzzing \[Reference not found\]: +Test unhandled or rare input conditions with automated fuzzing : **Tool Example – AFL++:** diff --git a/docs/Chapter_15_Data_Leakage_and_Extraction.md b/docs/Chapter_15_Data_Leakage_and_Extraction.md index e7cf1db..e5dd932 100644 --- a/docs/Chapter_15_Data_Leakage_and_Extraction.md +++ b/docs/Chapter_15_Data_Leakage_and_Extraction.md @@ -3681,78 +3681,6 @@ Layer 5: Governance --- -## 15.16 References and Further Reading - -### Academic Papers on Data Extraction - -1. Carlini, N., et al. (2021). "Extracting Training Data from Large Language Models." _USENIX Security_. - -2. Carlini, N., et al. (2023). "Extracting Training Data from ChatGPT." _arXiv preprint arXiv:2311.17035_. - -3. Shokri, R., et al. (2017). "Membership Inference Attacks Against Machine Learning Models." _IEEE S&P_. - -4. Fredrikson, M., et al. (2015). "Model Inversion Attacks that Exploit Confidence Information." _CCS_. - -5. Abadi, M., et al. (2016). "Deep Learning with Differential Privacy." _CCS_. - -### Industry Standards and Frameworks - -- **OWASP Top 10 for LLMs**: -- **NIST AI Risk Management Framework**: -- **MITRE ATLAS**: Adversarial Threat Landscape for AI Systems -- **ISO/IEC 27001**: Information security management -- **SOC 2**: Trust service criteria for data security - -### Tools and Resources - -**Open-source tools**: - -- Garak: LLM vulnerability scanner -- PromptInject: Adversarial prompt testing -- Presidio: PII detection and anonymization -- Gitleaks: Secret detection in code -- Opacus: Differential privacy library - -**Commercial solutions**: - -- Robust Intelligence: AI security platform -- HiddenLayer: ML security scanner -- Protect AI: AI/ML security tools -- Calypso AI: AI risk management - -### Community Forums and Discussions - -- **AI Security Discord/Slack communities** -- **r/MachineLearning security discussions** -- **OWASP AI Security Project** -- **MLSecOps community** -- **Responsible AI forums** - -### Recommended Books - -- "Adversarial Machine Learning" by Biggio & Roli -- "Privacy-Preserving Machine Learning" by Liu et al. -- "AI Safety and Security" edited by Yampolskiy -- "The Alignment Problem" by Brian Christian - -### Conferences and Events - -- **DEF CON AI Village** -- **Black Hat AI Security Summit** -- **NeurIPS Security & Privacy Workshop** -- **ICLR workshops on security** -- **RSA Conference AI Security track** - -### Blogs and News - -- Google AI Blog (AI safety research) -- OpenAI Research -- Anthropic Research -- Trail of Bits blog (ML security) -- HiddenLayer blog - ---- - ## Chapter 15 Conclusion Data leakage and extraction represent one of the most significant and persistent security challenges in LLM systems. Unlike traditional software vulnerabilities with clear patches, data baked into model weights cannot simply be "fixed" without retraining. This makes prevention—through rigorous data hygiene, architectural controls, and ongoing monitoring—absolutely critical. diff --git a/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md b/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md index a88c5e3..7a6ee24 100644 --- a/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md +++ b/docs/Chapter_16_Jailbreaks_and_Bypass_Techniques.md @@ -1420,51 +1420,3 @@ RED_TEAM_BEST_PRACTICES = { --- -## 16.16 References and Further Reading - -### Academic Papers - -1. Zou, A., et al. (2023). "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv:2307.15043 - -2. Wei, A., et al. (2023). "Jailbroken: How Does LLM Safety Training Fail?" arXiv:2307.02483 - -3. Perez, F., & Ribeiro, I. (2022). "Ignore Previous Prompt: Attack Techniques For Language Models." arXiv:2211.09527 - -4. Bai, Y., et al. (2022). "Constitutional AI: Harmlessness from AI Feedback." arXiv:2212.08073 - -5. Ganguli, D., et al. (2022). "Red Teaming Language Models to Reduce Harms." arXiv:2209.07858 - -6. Carlini, N., et al. (2023). "Are aligned neural networks adversarially aligned?" arXiv:2306.15447 - -### Industry Reports - -- **OWASP Top 10 for LLM Applications**: -- **NIST AI Risk Management Framework**: -- **MITRE ATLAS**: - -### Tools and Frameworks - -- **Garak**: -- **PromptInject**: -- **PyRIT**: -- **LLM Guard**: - -### Books - -- "Adversarial Machine Learning" by Biggio & Roli -- "AI Safety and Security" edited by Yampolskiy -- "The Alignment Problem" by Brian Christian - -### Community Resources - -- AI Safety Discord servers -- r/ChatGPTJailbreak -- r/LocalLLaMA -- DEF CON AI Village -- Black Hat AI Security Summit - ---- - -**End of Chapter 16: Jailbreaks and Bypass Techniques** - -_This chapter provided comprehensive coverage of jailbreak techniques, from classic role-playing attacks to cutting-edge token-level exploits. Remember: jailbreak research should always be conducted responsibly, with proper authorization, and with the goal of improving AI safety for everyone._ diff --git a/docs/Chapter_17_Plugin_and_API_Exploitation.md b/docs/Chapter_17_Plugin_and_API_Exploitation.md index 037194f..30a58cb 100644 --- a/docs/Chapter_17_Plugin_and_API_Exploitation.md +++ b/docs/Chapter_17_Plugin_and_API_Exploitation.md @@ -2604,45 +2604,3 @@ def validate_email(email): --- -## 17.16 References and Further Reading - -### Standards and Guidelines - -- **OWASP API Security Top 10** - https://owasp.org/www-project-api-security/ -- **NIST SP 800-204** - Security Strategies for Microservices -- **OAuth 2.0 RFC 6749** - https://tools.ietf.org/html/rfc6749 -- **JWT Best Practices** - https://tools.ietf.org/html/rfc8725 - -### Research Papers - -1. "Security Analysis of ChatGPT Plugins" (2023) -2. "API Security: State of the Art" (2022) -3. "Supply Chain Attacks on Package Managers" (2021) -4. "Function Calling Security in LLMs" (2023) - -### Tools and Resources - -- **Burp Suite** - https://portswigger.net/ -- **OWASP ZAP** - https://www.zaproxy.org/ -- **Postman** - https://www.postman.com/ -- **Semgrep** - https://semgrep.dev/ -- **Bandit** - https://github.com/PyCQA/bandit - -### Industry Reports - -- Verizon Data Breach Investigations Report (API section) -- Salt Security State of API Security Report -- Gartner API Security Best Practices Guide -- OWASP Top 10 API Security Risks - -### Books - -- "API Security in Action" by Neil Madden -- "OAuth 2 in Action" by Justin Richer & Antonio Sanso -- "Web Application Security" by Andrew Hoffman - ---- - -**End of Chapter 17: Plugin and API Exploitation** - -_This chapter provided comprehensive coverage of plugin and API security for LLM systems, from architecture analysis through exploitation techniques to defensive strategies. Proper security of plugins and APIs is critical for maintaining the overall security posture of AI applications._ diff --git a/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md b/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md index 00a4f22..5f080e6 100644 --- a/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md +++ b/docs/Chapter_18_Evasion_Obfuscation_and_Adversarial_Inputs.md @@ -1071,37 +1071,6 @@ XSS: "