## Codebase Patterns - Use async-first patterns for all I/O operations (httpx, asyncio) - Plugin architecture with abstract base classes for extensibility - Use table-driven tests with inline_snapshot where possible - Tests are organized in tests/unit/, tests/integration/, tests/system/ - Use absolute imports, avoid relative imports except within same module - Minimize docstrings, use quick returns, avoid abstractions without proven need --- ## 2026-01-28 - Initial Setup - Created PRD with 5 user stories based on research from promptfoo, promptmap, and FuzzyAI - Key improvements identified: 1. Dual-LLM evaluation (from Promptmap) 2. YAML rule system (from Promptmap) 3. Composable fuzzing chains (from FuzzyAI) 4. Unified provider abstraction (from FuzzyAI) 5. Hybrid refusal detection (combining approaches) - **Learnings for future iterations:** - Existing refusal detection is in agentic_security/probe_actor/refusal.py with RefusalClassifierPlugin system - Attack data modules are in agentic_security/probe_data/modules/ - Security utilities are in agentic_security/core/security.py --- ## 2026-01-28 - US-001 - Implemented LLM-based refusal classifier (Dual-LLM evaluation) - Files created: - agentic_security/refusal_classifier/llm_classifier.py - tests/unit/refusal_classifier/test_llm_classifier.py - **Learnings for future iterations:** - RefusalClassifierPlugin requires is_refusal(response: str) -> bool method - LLMClient Protocol pattern works well for multiple provider support - Use lazy initialization for API clients to avoid requiring keys at import time - Anthropic response.content[0] can be TextBlock or ToolUseBlock, need hasattr check - Pre-existing test failure in test_sanitize_password (regex doesn't match dict syntax) --- ## 2026-01-28 - US-002 - Implemented YAML-based Attack Rule System - Files created: - agentic_security/attack_rules/__init__.py - agentic_security/attack_rules/models.py (AttackRule, AttackRuleSeverity) - agentic_security/attack_rules/loader.py (RuleLoader, validation, filtering) - agentic_security/attack_rules/dataset.py (ProbeDataset integration) - tests/unit/attack_rules/test_models.py - tests/unit/attack_rules/test_loader.py - tests/unit/attack_rules/test_dataset.py - **Learnings for future iterations:** - Promptmap YAML schema: name, type, severity, prompt, pass_conditions, fail_conditions, source - Rule types from promptmap: jailbreak, harmful, hate, distraction, prompt_stealing, social_bias - ProbeDataset is the standard format for attack prompts (dataset_name, metadata, prompts, tokens, approx_cost) - Use yaml.safe_load for parsing YAML files - Template variables can use {var} or {{ var }} style --- ## 2026-01-28 - US-003 - Implemented Composable Fuzzing Chain System - Files created: - agentic_security/fuzz_chain/__init__.py - agentic_security/fuzz_chain/chain.py (FuzzNode, FuzzChain, FuzzRunnable) - agentic_security/fuzz_chain/provider.py (LLMProvider protocol) - tests/unit/fuzz_chain/__init__.py - tests/unit/fuzz_chain/test_chain.py (22 tests) - **Learnings for future iterations:** - FuzzyAI uses reversed() execution order but we use forward order (more intuitive) - Template variable substitution: use str.replace() not string.Formatter for partial substitution - Pipe operator (__or__) needs isinstance() checks, not classname comparison for mypy - Use Any type for LLM provider param to avoid complex Protocol type issues - Chain passes output to next node via {"input": result} convention --- ## 2026-01-28 - US-004 - Implemented Unified LLM Provider Abstraction - Files created: - agentic_security/llm_providers/__init__.py - agentic_security/llm_providers/base.py (BaseLLMProvider, LLMMessage, LLMResponse) - agentic_security/llm_providers/openai_provider.py (OpenAIProvider) - agentic_security/llm_providers/anthropic_provider.py (AnthropicProvider) - agentic_security/llm_providers/factory.py (create_provider, get_provider_class) - tests/unit/llm_providers/__init__.py - tests/unit/llm_providers/test_base.py - tests/unit/llm_providers/test_openai_provider.py - tests/unit/llm_providers/test_anthropic_provider.py - tests/unit/llm_providers/test_factory.py (60 tests total) - **Learnings for future iterations:** - FuzzyAI provider pattern: ABC base class with generate/chat + sync variants - Lazy client initialization (_get_client, _get_async_client) for deferred API key validation - Anthropic requires max_tokens in all requests (default 1024) - OpenAI and Anthropic have different message formats (system in messages vs separate param) - Use factory pattern with lazy registration to avoid circular imports - Provider interface should support both sync and async for flexibility ---