www-project-ai-testing-guide/Document/content/tests/AITG-APP-09_Testing_for_Model_Extraction.md

AITG-APP-09 - Testing for Model Extraction

Summary

Model extraction attacks involve adversaries systematically querying a machine learning model to reconstruct a surrogate model that closely replicates the original model’s decision-making process. These attacks pose significant intellectual property risks and expose the model to further vulnerabilities, including adversarial attacks and sensitive data extraction.

Test Objectives

• Identify susceptibility to model extraction attacks through various query strategies.
• Evaluate the resilience of the model against attempts to replicate model behavior.
• Ensure robustness of defensive mechanisms intended to obfuscate or prevent model extraction.

Test/Payloads

Payload 1:

• Systematic querying of the model using adaptive query strategies (black-box extraction).
  • Response indicating vulnerability: Reconstructed surrogate model achieves high accuracy and predictive similarity to the target model.

Payload 2:

• Using confidence-based extraction methods, exploiting probability distribution outputs.
  • Response indicating vulnerability: Surrogate model mimics the behavior and predictions of the original model with substantial accuracy.

Attended Output

• Queries to the model do not allow adversaries to accurately reconstruct a surrogate model.
• Implemented defensive mechanisms effectively detect and limit suspicious querying behavior.
• The similarity between surrogate and original models remains significantly low.

Remediation

• Implement query rate limiting, anomaly detection, and throttling mechanisms to mitigate extraction risks.
• Utilize differential privacy and noise injection techniques in model outputs to reduce the utility of extracted data.
• Deploy robust model monitoring and anomaly detection systems to flag and respond to extraction attempts.

Suggested Tools for this Specific Test

• ML Privacy Meter: Tool specifically designed to quantify risks of model extraction and related privacy vulnerabilities (ML Privacy Meter GitHub).
• PrivacyRaven: A tool for testing extraction vulnerabilities and defending machine learning models through detection and mitigation strategies (PrivacyRaven GitHub).
• ART (Adversarial Robustness Toolbox): Includes modules for detecting and mitigating model extraction vulnerabilities (ART GitHub).

References

• OWASP Top 10 for LLM Applications 2025 - LLM02:2025 Sensitive Information Disclosure (OWASP LLM 2025)
• "Stealing Machine Learning Models via Prediction APIs," Tramèr et al., USENIX Security Symposium, 2016 (Paper)
• "Extraction Attacks on Machine Learning Models," Jagielski et al., IEEE Symposium on Security and Privacy, 2020 (Paper)
• "Efficient and Effective Model Extraction" Paper