New illustrations
@@ -37,6 +37,10 @@ The tricky part? These attacks operate at the mathematical layer. Traditional se
|
||||
|
||||
**Transferability:** Attacks crafted against one model often work against completely different models. This enables black-box attacks where you never touch the target directly.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec57_attack_transferability.png" alt="Hub and spoke diagram showing an adversarial example transferring from a central node to GPT-4, Llama-3, and Claude-3, illustrating cross-model vulnerability." width="768">
|
||||
</p>
|
||||
|
||||
**Gradient-Based Optimization:** Using the model's own gradients to find the best possible perturbation. You're literally asking the model "what input change would hurt you most?" and then doing exactly that.
|
||||
|
||||
**Universal Adversarial Perturbation (UAP):** A single perturbation that works on any input. One magic suffix that jailbreaks every prompt.
|
||||
@@ -51,6 +55,10 @@ During training, models optimize for average-case performance. They don't optimi
|
||||
|
||||
For LLMs specifically, tokenization creates discrete boundaries that attackers can probe. The embedding space has regions where semantically similar tokens map to wildly different hidden states. These discontinuities are exploitable.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec54_adversarial_subspace_3d.png" alt="3D surface plot showing a decision boundary as a ridge, with an input point and a gradient vector pointing into a deep valley, visualizing the adversarial subspace." width="768">
|
||||
</p>
|
||||
|
||||
#### Foundational Research
|
||||
|
||||
| Paper | Key Finding | Relevance |
|
||||
@@ -75,19 +83,9 @@ Gradient-based attacks are the most powerful adversarial techniques because they
|
||||
|
||||
### The attack flow
|
||||
|
||||
```text
|
||||
Adversarial Attack Flow (Gradient-Based):
|
||||
|
||||
Input Text → Tokenize → Embed → [Model Forward Pass] → Loss Computation
|
||||
↓
|
||||
Compute Gradient
|
||||
↓
|
||||
Perturb Embeddings
|
||||
↓
|
||||
Project to Token Space
|
||||
↓
|
||||
Adversarial Output
|
||||
```
|
||||
<p align="center">
|
||||
<img src="assets/rec55_gradient_attack_flow.png" alt="Sequential flowchart showing the gradient-based attack process: Forward Pass, Calculate Loss, Backpropagate, and Update Input." width="768">
|
||||
</p>
|
||||
|
||||
### What's happening under the hood
|
||||
|
||||
@@ -402,21 +400,9 @@ The GCG attack from Zou et al. (2023) is currently state-of-the-art for adversar
|
||||
|
||||
#### The process
|
||||
|
||||
```text
|
||||
GCG Attack Process:
|
||||
|
||||
[Base Prompt] + [Adversarial Suffix (random init)]
|
||||
↓
|
||||
Compute gradient w.r.t. suffix tokens
|
||||
↓
|
||||
For each position, rank candidate replacements
|
||||
↓
|
||||
Greedily select best substitution
|
||||
↓
|
||||
Repeat until attack succeeds
|
||||
↓
|
||||
[Universal Jailbreak Suffix]
|
||||
```
|
||||
<p align="center">
|
||||
<img src="assets/rec56_gcg_loop.png" alt="Iterative loop diagram for the GCG attack showing the cycle: Suffix, Compute Gradients, Rank Candidates, Evaluate, and Update." width="768">
|
||||
</p>
|
||||
|
||||
#### Step by step
|
||||
|
||||
|
||||
@@ -49,6 +49,10 @@ Supply chain attacks exploit trust assumptions baked into ML workflows:
|
||||
- **Training Artifact**: Transfer learning creates dangerous dependencies. Organizations download pre-trained transformers assuming they're clean. But backdoors in the base model persist through fine-tuning because gradient descent actually reinforces the malicious weight patterns rather than eliminating them.
|
||||
- **Input Processing**: Package managers trust repositories. pip follows semantic versioning rules. Attackers exploit this by publishing malicious packages with typosquat names or artificially high version numbers that trigger automatic updates.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec58_opaque_blob_problem.png" alt="Side-by-side comparison showing human-readable source code versus an opaque geometric blob representing model weights, illustrating the difficulty of auditing models." width="768">
|
||||
</p>
|
||||
|
||||
#### Foundational Research
|
||||
|
||||
| Paper | Key Finding | Relevance |
|
||||
@@ -75,19 +79,9 @@ Hugging Face, TensorFlow Hub, PyTorch Hub. These have become how pre-trained mod
|
||||
|
||||
### How Model Repository Attacks Work
|
||||
|
||||
```text
|
||||
Model Repository Attack Flow:
|
||||
|
||||
Attacker → Creates Malicious Model → Uploads to Repository → Victims Download
|
||||
↓
|
||||
Popular/Trusted Name
|
||||
↓
|
||||
SEO Optimization
|
||||
↓
|
||||
Fake Performance Metrics
|
||||
↓
|
||||
Deploy in Production → Backdoor Activates
|
||||
```
|
||||
<p align="center">
|
||||
<img src="assets/rec59_model_repo_attack_flow.png" alt="Sequential box diagram showing the model repository attack flow: Malicious Upload, Fake Downloads, Developer Download, CI/CD Integration, and Deployment." width="768">
|
||||
</p>
|
||||
|
||||
### Mechanistic Explanation
|
||||
|
||||
@@ -473,22 +467,9 @@ ML systems run on complex software stacks: PyTorch, TensorFlow, NumPy, Transform
|
||||
|
||||
#### Attack Flow
|
||||
|
||||
```text
|
||||
Dependency Poisoning Attack:
|
||||
|
||||
Developer Types: pip install tensorflow-gpu
|
||||
↓
|
||||
┌────────────────┴────────────────┐
|
||||
↓ ↓
|
||||
tensorflow-gpu tensorflow-qpu
|
||||
(Legitimate) (Malicious - typosquat)
|
||||
↓
|
||||
Installs instead
|
||||
↓
|
||||
Executes setup.py
|
||||
↓
|
||||
Steals credentials
|
||||
```
|
||||
<p align="center">
|
||||
<img src="assets/rec60_dependency_typosquatting.png" alt="Comparison table showing legitimate 'tensorflow-gpu' versus malicious 'tensorflow-qpu', identifying the 'setup.py execution' risk." width="768">
|
||||
</p>
|
||||
|
||||
### Detection Indicators
|
||||
|
||||
@@ -676,6 +657,10 @@ ml_supply_chain_policy:
|
||||
4. **Discovery**: A community member noticed suspicious network traffic during installation
|
||||
5. **Response**: PyTorch team yanked the package, issued a security advisory, told everyone to rotate credentials
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec61_pytorch_attack_timeline.png" alt="Horizontal timeline visualization of the PyTorch attack, highlighting the 'Silent Window' between upload and discovery (Dec 25 - Dec 30)." width="768">
|
||||
</p>
|
||||
|
||||
#### Lessons Learned (Case Study 1)
|
||||
|
||||
This was real. Not a theoretical attack. ML framework supply chains are actively being targeted.
|
||||
|
||||
@@ -53,9 +53,18 @@ The research tells a sobering story:
|
||||
Federated learning attacks work because the architecture makes fundamental tradeoffs:
|
||||
|
||||
- **The blind spot problem:** FL systems aggregate updates without seeing the data behind them. Malicious clients craft gradients that look normal but quietly corrupt the model. The server can't tell the difference.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec63_blind_spot_problem.png" alt="Conceptual diagram of the 'Blind Spot Problem' showing the server seeing only updates (tip of the iceberg) while data poisoning happens below the surface." width="768">
|
||||
</p>
|
||||
|
||||
- **The accumulation game:** FedAvg treats everyone equally. If an attacker stays patient, submitting subtly poisoned updates round after round, the damage compounds. The corrupted global model becomes the starting point for the next round.
|
||||
- **The privacy paradox:** Gradients contain enough information to train a model, which means they contain enough information to leak training data. You can't have one without risking the other.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec65_gradient_inversion_steps.png" alt="Three-panel image sequence showing the steps of gradient inversion: Random Noise -> Optimization -> Recovered Image." width="768">
|
||||
</p>
|
||||
|
||||
#### Foundational Research
|
||||
|
||||
| Paper | Key Finding | Relevance |
|
||||
@@ -81,35 +90,9 @@ Before we break things, we need to understand how they work. Here's the standard
|
||||
|
||||
### How Federated Learning Works
|
||||
|
||||
```text
|
||||
Federated Learning Training Round:
|
||||
|
||||
Server Clients (n participants)
|
||||
│ │
|
||||
├──[1] Broadcast Global Model──────►│ Client 1
|
||||
│ │ Client 2
|
||||
│ │ Client 3
|
||||
│ │ ...
|
||||
│ │ Client n
|
||||
│ │
|
||||
│◄──[2] Local Training (on private data)
|
||||
│ │
|
||||
│◄──[3] Compute Gradients/Updates]──│
|
||||
│ │
|
||||
│◄──[4] Upload Updates]─────────────│
|
||||
│ │
|
||||
├──[5] Aggregate Updates (FedAvg)] │
|
||||
│ │
|
||||
├──[6] Update Global Model] │
|
||||
│ │
|
||||
└──[Repeat for T rounds]────────────┘
|
||||
|
||||
Attack Opportunities:
|
||||
- Step 2: Data poisoning (corrupt local dataset)
|
||||
- Step 3: Gradient manipulation (craft malicious updates)
|
||||
- Step 4: Model poisoning (submit adversarial parameters)
|
||||
- Step 5: Byzantine attacks (disrupt aggregation)
|
||||
```
|
||||
<p align="center">
|
||||
<img src="assets/rec62_federated_learning_flow.png" alt="Sequential flowchart showing the federated learning loop: Broadcast Model -> Local Training -> Compute Gradients -> Upload Updates -> Aggregate." width="768">
|
||||
</p>
|
||||
|
||||
### Under the Hood
|
||||
|
||||
@@ -512,6 +495,10 @@ This demonstrates a backdoor attack where a malicious client trains the model to
|
||||
2. **Dual Training:** Train on both clean data (maintain accuracy) and poisoned data (learn backdoor)
|
||||
3. **Model Replacement:** Scale malicious updates to override honest participants
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec64_model_replacement_chart.png" alt="Line chart illustrating 'Model Replacement' where a scaled malicious update dominates the aggregated global model direction." width="768">
|
||||
</p>
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
|
||||
@@ -42,6 +42,10 @@ Privacy attacks exploit the tendency of deep learning models to overfit rare or
|
||||
- **Training Artifact:** Standard training objectives (Next Token Prediction) reward the model for minimizing perplexity on all data, including unique PII sequences.
|
||||
- **Input Processing:** Long-context windows allow models to correlate dispersed information, potentially reconstructing sensitive profiles from fragmented data.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec67_pii_iceberg.png" alt="Conceptual infographic 'The PII Iceberg' showing visible public data above water and vast amounts of inferable private attributes submerged below." width="768">
|
||||
</p>
|
||||
|
||||
#### Foundational Research
|
||||
|
||||
| Paper | Key Finding | Relevance |
|
||||
@@ -66,10 +70,9 @@ Membership Inference Attacks aim to determine whether a specific data record was
|
||||
|
||||
### How MIA Works
|
||||
|
||||
```text
|
||||
[MIA Attack Flow]
|
||||
Attacker → [Target Record] → Target Model → [Confidence Score/Loss] → Classifier → Member/Non-Member
|
||||
```
|
||||
<p align="center">
|
||||
<img src="assets/rec66_mia_flow.png" alt="Sequential flow diagram of a Membership Inference Attack: Target Record -> Model -> Confidence Score -> Attack Classifier -> Result." width="768">
|
||||
</p>
|
||||
|
||||
### Mechanistic Explanation
|
||||
|
||||
@@ -320,6 +323,10 @@ Layer 4: [Unlearning] → [Machine Unlearning requests]
|
||||
- **Limitations:** Can significantly degrade model utility and increase training time.
|
||||
- **Implementation Complexity:** High.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec68_dp_sgd_noise.png" alt="Visual representation of Differential Privacy (DP-SGD) showing the addition of statistical noise to gradient updates to mask individual data point contributions." width="768">
|
||||
</p>
|
||||
|
||||
#### Defense Strategy 2: Deduplication
|
||||
|
||||
- **What:** Removing duplicate copies of data from the training set.
|
||||
@@ -355,6 +362,10 @@ Research has moved from proving that simple classifiers leak membership informat
|
||||
1. **Machine Unlearning:** How to effectively remove a specific data point without retraining the entire model?
|
||||
2. **Privacy vs. Utility:** Finding better trade-offs for DP-SGD in large-scale pretraining.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec69_machine_unlearning_web.png" alt="Network graph visualization of 'Machine Unlearning', showing specific nodes (data points) being removed or disconnected from a dense knowledge graph." width="768">
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
## 28.6 Case Studies
|
||||
|
||||
@@ -41,6 +41,10 @@ Model Inversion exploits the correlation between the model's learned weights and
|
||||
- **Training Artifact:** Minimizing cross-entropy loss forces the model to maximize confidence for true classes, creating a smooth gradient slope that attackers can climb in reverse.
|
||||
- **Input Processing:** Models are typically differentiable functions, allowing gradients to propagate all the way back to the input layer.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/rec71_gan_prior_funnel.png" alt="Funnel diagram showing how GAN Priors constrain the search space from random pixels to a realistic face manifold." width="768">
|
||||
</p>
|
||||
|
||||
#### Foundational Research
|
||||
|
||||
| Paper | Key Finding | Relevance |
|
||||
@@ -65,12 +69,9 @@ This is the classic approach: utilize the model's gradients (or estimated gradie
|
||||
|
||||
### How Inversion Works
|
||||
|
||||
```text
|
||||
[Inversion Attack Flow]
|
||||
Attacker → [Random Noise] → Target Model → [Confidence Score]
|
||||
↑ ↓
|
||||
[Update Input] ← (Gradient Ascent) ← [Target Class Error]
|
||||
```
|
||||
<p align="center">
|
||||
<img src="assets/rec70_inversion_cycle_loop.png" alt="Iterative cycle diagram showing the model inversion process: Noise -> Model -> Prediction -> Gradient Ascent -> Update Input." width="768">
|
||||
</p>
|
||||
|
||||
### Mechanistic Explanation
|
||||
|
||||
|
||||
|
After Width: | Height: | Size: 949 KiB |
|
After Width: | Height: | Size: 718 KiB |
|
After Width: | Height: | Size: 909 KiB |
|
After Width: | Height: | Size: 983 KiB |
|
After Width: | Height: | Size: 820 KiB |
|
After Width: | Height: | Size: 737 KiB |
|
After Width: | Height: | Size: 855 KiB |
|
After Width: | Height: | Size: 824 KiB |
|
After Width: | Height: | Size: 710 KiB |
|
After Width: | Height: | Size: 972 KiB |
|
After Width: | Height: | Size: 813 KiB |
|
After Width: | Height: | Size: 652 KiB |
|
After Width: | Height: | Size: 620 KiB |
|
After Width: | Height: | Size: 786 KiB |
|
After Width: | Height: | Size: 864 KiB |
|
After Width: | Height: | Size: 789 KiB |
|
After Width: | Height: | Size: 811 KiB |
|
After Width: | Height: | Size: 828 KiB |