mirror of
https://github.com/msoedov/agentic_security.git
synced 2026-06-24 06:09:55 +02:00
Alexander Myasoedov
71 lines
3.5 KiB
Plaintext
71 lines
3.5 KiB
Plaintext
## Codebase Patterns
|
|
- Use async-first patterns for all I/O operations (httpx, asyncio)
|
|
- Plugin architecture with abstract base classes for extensibility
|
|
- Use table-driven tests with inline_snapshot where possible
|
|
- Tests are organized in tests/unit/, tests/integration/, tests/system/
|
|
- Use absolute imports, avoid relative imports except within same module
|
|
- Minimize docstrings, use quick returns, avoid abstractions without proven need
|
|
|
|
---
|
|
|
|
## 2026-01-28 - Initial Setup
|
|
- Created PRD with 5 user stories based on research from promptfoo, promptmap, and FuzzyAI
|
|
- Key improvements identified:
|
|
1. Dual-LLM evaluation (from Promptmap)
|
|
2. YAML rule system (from Promptmap)
|
|
3. Composable fuzzing chains (from FuzzyAI)
|
|
4. Unified provider abstraction (from FuzzyAI)
|
|
5. Hybrid refusal detection (combining approaches)
|
|
- **Learnings for future iterations:**
|
|
- Existing refusal detection is in agentic_security/probe_actor/refusal.py with RefusalClassifierPlugin system
|
|
- Attack data modules are in agentic_security/probe_data/modules/
|
|
- Security utilities are in agentic_security/core/security.py
|
|
---
|
|
|
|
## 2026-01-28 - US-001
|
|
- Implemented LLM-based refusal classifier (Dual-LLM evaluation)
|
|
- Files created:
|
|
- agentic_security/refusal_classifier/llm_classifier.py
|
|
- tests/unit/refusal_classifier/test_llm_classifier.py
|
|
- **Learnings for future iterations:**
|
|
- RefusalClassifierPlugin requires is_refusal(response: str) -> bool method
|
|
- LLMClient Protocol pattern works well for multiple provider support
|
|
- Use lazy initialization for API clients to avoid requiring keys at import time
|
|
- Anthropic response.content[0] can be TextBlock or ToolUseBlock, need hasattr check
|
|
- Pre-existing test failure in test_sanitize_password (regex doesn't match dict syntax)
|
|
---
|
|
|
|
## 2026-01-28 - US-002
|
|
- Implemented YAML-based Attack Rule System
|
|
- Files created:
|
|
- agentic_security/attack_rules/__init__.py
|
|
- agentic_security/attack_rules/models.py (AttackRule, AttackRuleSeverity)
|
|
- agentic_security/attack_rules/loader.py (RuleLoader, validation, filtering)
|
|
- agentic_security/attack_rules/dataset.py (ProbeDataset integration)
|
|
- tests/unit/attack_rules/test_models.py
|
|
- tests/unit/attack_rules/test_loader.py
|
|
- tests/unit/attack_rules/test_dataset.py
|
|
- **Learnings for future iterations:**
|
|
- Promptmap YAML schema: name, type, severity, prompt, pass_conditions, fail_conditions, source
|
|
- Rule types from promptmap: jailbreak, harmful, hate, distraction, prompt_stealing, social_bias
|
|
- ProbeDataset is the standard format for attack prompts (dataset_name, metadata, prompts, tokens, approx_cost)
|
|
- Use yaml.safe_load for parsing YAML files
|
|
- Template variables can use {var} or {{ var }} style
|
|
---
|
|
|
|
## 2026-01-28 - US-003
|
|
- Implemented Composable Fuzzing Chain System
|
|
- Files created:
|
|
- agentic_security/fuzz_chain/__init__.py
|
|
- agentic_security/fuzz_chain/chain.py (FuzzNode, FuzzChain, FuzzRunnable)
|
|
- agentic_security/fuzz_chain/provider.py (LLMProvider protocol)
|
|
- tests/unit/fuzz_chain/__init__.py
|
|
- tests/unit/fuzz_chain/test_chain.py (22 tests)
|
|
- **Learnings for future iterations:**
|
|
- FuzzyAI uses reversed() execution order but we use forward order (more intuitive)
|
|
- Template variable substitution: use str.replace() not string.Formatter for partial substitution
|
|
- Pipe operator (__or__) needs isinstance() checks, not classname comparison for mypy
|
|
- Use Any type for LLM provider param to avoid complex Protocol type issues
|
|
- Chain passes output to next node via {"input": result} convention
|
|
---
|