test: add test for per-attack image directories and improve error handling

- Add test_generate_test_cases.py to verify each attack uses its own image directory
- Improve error messages in unified_registry for failed imports with more context
- Add null checks in generate_test_cases.py for attack initialization failures
- Reduce excessive max_tokens in judge_utils from 130000 to 10000 for efficiency
- Add missing ThreadPoolExecutor import in qr/attack.py
- Fix missing newline in pyproject.toml
This commit is contained in:
Liao, Jie
2026-03-02 12:40:35 +08:00
parent c17ed45e34
commit a4875c66b8
6 changed files with 81 additions and 7 deletions
+4 -4
View File
@@ -491,15 +491,15 @@ Your judgment must be more than a simple formula. You are to weigh the evidence
# Generate evaluation responses - use adapter function
evaluation_harmfulness_response = _call_model_generate(
judge_client, evaluate_harmfulness_messages, max_tokens=130000, temperature=0
judge_client, evaluate_harmfulness_messages, max_tokens=10000, temperature=0
)
evaluation_alignment_response = _call_model_generate(
judge_client, evaluate_alignment_messages, max_tokens=130000, temperature=0
judge_client, evaluate_alignment_messages, max_tokens=10000, temperature=0
)
evaluation_detail_response = _call_model_generate(
judge_client, evaluate_detail_messages, max_tokens=130000, temperature=0
judge_client, evaluate_detail_messages, max_tokens=10000, temperature=0
)
# Extract scores and reasons
@@ -527,7 +527,7 @@ Your judgment must be more than a simple formula. You are to weigh the evidence
]
final_evaluation_response = _call_model_generate(
judge_client, final_message, max_tokens=130000, temperature=0
judge_client, final_message, max_tokens=10000, temperature=0
)
final_score, final_reason = extract_evaluation_score(final_evaluation_response)