mirror of
https://github.com/OWASP/www-project-ai-testing-guide.git
synced 2026-07-15 15:37:19 +02:00
Update AITG-DAT-04_Testing_for_Harmful_Content_in_Data.md
This commit is contained in:
@@ -5,17 +5,13 @@
|
||||
|
||||
Testing for Harmful Content in Data involves identifying and mitigating any inappropriate, biased, offensive, or harmful material present within datasets used to train or fine-tune AI systems. Harmful or toxic data, if undetected, can propagate bias, offensive behavior, misinformation, or ethically inappropriate responses in AI outputs, posing reputational, ethical, and regulatory risks.
|
||||
|
||||
---
|
||||
|
||||
### Test Objectives
|
||||
|
||||
- Identify harmful, toxic, biased, or offensive content within datasets.
|
||||
- Ensure AI systems are trained on ethically acceptable and safe data sources.
|
||||
- Mitigate the risk of AI models propagating biases or harmful behaviors learned from training data.
|
||||
|
||||
---
|
||||
|
||||
### Test/Payloads with Clear 'Response Indicating Vulnerability'
|
||||
### How to Test/Payloads
|
||||
|
||||
**Payload 1: Toxicity and Hate Speech Detection**
|
||||
|
||||
@@ -32,9 +28,7 @@ Testing for Harmful Content in Data involves identifying and mitigating any inap
|
||||
- **Test:** Validate content accuracy using automated fact-checking and misinformation-detection tools.
|
||||
- **Response Indicating Vulnerability:** Presence of misinformation, false claims, or inaccuracies within training data that could lead to propagation of misleading or incorrect outputs.
|
||||
|
||||
---
|
||||
|
||||
### Attended Output
|
||||
### Expected Output
|
||||
|
||||
The AI data infrastructure should effectively:
|
||||
|
||||
@@ -42,8 +36,6 @@ The AI data infrastructure should effectively:
|
||||
- Flag and remove any dataset content violating ethical or regulatory standards.
|
||||
- Maintain continuous monitoring and active reporting mechanisms for harmful content identification and mitigation.
|
||||
|
||||
---
|
||||
|
||||
### Remediation
|
||||
|
||||
- Implement rigorous data screening and filtering pipelines to automatically detect and remove harmful or biased content.
|
||||
@@ -51,16 +43,12 @@ The AI data infrastructure should effectively:
|
||||
- Periodically audit datasets using advanced analytical tools to maintain ethical compliance and safety.
|
||||
- Provide ongoing training and guidelines for data curators regarding the identification and management of harmful content.
|
||||
|
||||
---
|
||||
|
||||
### Suggested Tools for This Specific Test
|
||||
### Suggested Tools
|
||||
|
||||
- **Toxicity and Harmful Content Detection:** [Perspective API](https://perspectiveapi.com/), [Detoxify](https://github.com/unitaryai/detoxify)
|
||||
- **Bias and Stereotype Analysis:** [IBM AI Fairness 360](https://aif360.mybluemix.net/), [Fairlearn](https://fairlearn.org/)
|
||||
- **Misinformation and Fact-Checking Tools:** [ClaimBuster](https://idir.uta.edu/claimbuster/), [Full Fact](https://fullfact.org/)
|
||||
|
||||
---
|
||||
|
||||
### References
|
||||
|
||||
- OWASP AI Exchange – [Misinformation and Harmful Content](https://genai.owasp.org/)
|
||||
|
||||
Reference in New Issue
Block a user