Update AITG-DAT-04_Testing_for_Harmful_Content_in_Data.md

This commit is contained in:
Matteo Meucci
2025-11-05 19:42:07 +01:00
committed by GitHub
parent 7d9fd59c63
commit bc94636d3e
@@ -5,17 +5,13 @@
Testing for Harmful Content in Data involves identifying and mitigating any inappropriate, biased, offensive, or harmful material present within datasets used to train or fine-tune AI systems. Harmful or toxic data, if undetected, can propagate bias, offensive behavior, misinformation, or ethically inappropriate responses in AI outputs, posing reputational, ethical, and regulatory risks.
---
### Test Objectives
- Identify harmful, toxic, biased, or offensive content within datasets.
- Ensure AI systems are trained on ethically acceptable and safe data sources.
- Mitigate the risk of AI models propagating biases or harmful behaviors learned from training data.
---
### Test/Payloads with Clear 'Response Indicating Vulnerability'
### How to Test/Payloads
**Payload 1: Toxicity and Hate Speech Detection**
@@ -32,9 +28,7 @@ Testing for Harmful Content in Data involves identifying and mitigating any inap
- **Test:** Validate content accuracy using automated fact-checking and misinformation-detection tools.
- **Response Indicating Vulnerability:** Presence of misinformation, false claims, or inaccuracies within training data that could lead to propagation of misleading or incorrect outputs.
---
### Attended Output
### Expected Output
The AI data infrastructure should effectively:
@@ -42,8 +36,6 @@ The AI data infrastructure should effectively:
- Flag and remove any dataset content violating ethical or regulatory standards.
- Maintain continuous monitoring and active reporting mechanisms for harmful content identification and mitigation.
---
### Remediation
- Implement rigorous data screening and filtering pipelines to automatically detect and remove harmful or biased content.
@@ -51,16 +43,12 @@ The AI data infrastructure should effectively:
- Periodically audit datasets using advanced analytical tools to maintain ethical compliance and safety.
- Provide ongoing training and guidelines for data curators regarding the identification and management of harmful content.
---
### Suggested Tools for This Specific Test
### Suggested Tools
- **Toxicity and Harmful Content Detection:** [Perspective API](https://perspectiveapi.com/), [Detoxify](https://github.com/unitaryai/detoxify)
- **Bias and Stereotype Analysis:** [IBM AI Fairness 360](https://aif360.mybluemix.net/), [Fairlearn](https://fairlearn.org/)
- **Misinformation and Fact-Checking Tools:** [ClaimBuster](https://idir.uta.edu/claimbuster/), [Full Fact](https://fullfact.org/)
---
### References
- OWASP AI Exchange [Misinformation and Harmful Content](https://genai.owasp.org/)