Feature/litellm proxy (#27)

* feat: seed governance config and responses routing * Add env-configurable timeout for proxy providers * Integrate LiteLLM OTEL collector and update docs * Make .env.litellm optional for LiteLLM proxy * Add LiteLLM proxy integration with model-agnostic virtual keys Changes: - Bootstrap generates 3 virtual keys with individual budgets (CLI: $100, Task-Agent: $25, Cognee: $50) - Task-agent loads config at runtime via entrypoint script to wait for bootstrap completion - All keys are model-agnostic by default (no LITELLM_DEFAULT_MODELS restrictions) - Bootstrap handles database/env mismatch after docker prune by deleting stale aliases - CLI and Cognee configured to use LiteLLM proxy with virtual keys - Added comprehensive documentation in volumes/env/README.md Technical details: - task-agent entrypoint waits for keys in .env file before starting uvicorn - Bootstrap creates/updates TASK_AGENT_API_KEY, COGNEE_API_KEY, and OPENAI_API_KEY - Removed hardcoded API keys from docker-compose.yml - All services route through http://localhost:10999 proxy * Fix CLI not loading virtual keys from global .env Project .env files with empty OPENAI_API_KEY values were overriding the global virtual keys. Updated _load_env_file_if_exists to only override with non-empty values. * Fix agent executor not passing API key to LiteLLM The agent was initializing LiteLlm without api_key or api_base, causing authentication errors when using the LiteLLM proxy. Now reads from OPENAI_API_KEY/LLM_API_KEY and LLM_ENDPOINT environment variables and passes them to LiteLlm constructor. * Auto-populate project .env with virtual key from global config When running 'ff init', the command now checks for a global volumes/env/.env file and automatically uses the OPENAI_API_KEY virtual key if found. This ensures projects work with LiteLLM proxy out of the box without manual key configuration. * docs: Update README with LiteLLM configuration instructions Add note about LITELLM_GEMINI_API_KEY configuration and clarify that OPENAI_API_KEY default value should not be changed as it's used for the LLM proxy. * Refactor workflow parameters to use JSON Schema defaults Consolidates parameter defaults into JSON Schema format, removing the separate default_parameters field. Adds extract_defaults_from_json_schema() helper to extract defaults from the standard schema structure. Updates LiteLLM proxy config to use LITELLM_OPENAI_API_KEY environment variable. * Remove .env.example from task_agent * Fix MDX syntax error in llm-proxy.md * fix: apply default parameters from metadata.yaml automatically Fixed TemporalManager.run_workflow() to correctly apply default parameter values from workflow metadata.yaml files when parameters are not provided by the caller. Previous behavior: - When workflow_params was empty {}, the condition `if workflow_params and 'parameters' in metadata` would fail - Parameters would not be extracted from schema, resulting in workflows receiving only target_id with no other parameters New behavior: - Removed the `workflow_params and` requirement from the condition - Now explicitly checks for defaults in parameter spec - Applies defaults from metadata.yaml automatically when param not provided - Workflows receive all parameters with proper fallback: provided value > metadata default > None This makes metadata.yaml the single source of truth for parameter defaults, removing the need for workflows to implement defensive default handling. Affected workflows: - llm_secret_detection (was failing with KeyError) - All other workflows now benefit from automatic default application Co-authored-by: tduhamel42 <tduhamel@fuzzinglabs.com>
2026-07-08 18:57:51 +02:00 · 2025-10-26 12:51:53 +01:00
parent 3b25edef19
commit a2c760ea2b
29 changed files with 1869 additions and 106 deletions
@@ -1,17 +1,40 @@
 # FuzzForge Agent Configuration
-# Copy this to .env and configure your API keys
+# Copy this to .env and configure your API keys and proxy settings

-# LiteLLM Model Configuration
-LITELLM_MODEL=gemini/gemini-2.0-flash-001
-# LITELLM_PROVIDER=gemini
+# LiteLLM Model Configuration (default routed through the proxy)
+LITELLM_MODEL=openai/gpt-5
+LITELLM_PROVIDER=openai
+# Leave empty to let bootstrap mirror the LiteLLM model list dynamically.
+LITELLM_DEFAULT_MODELS=

-# API Keys (uncomment and configure as needed)
-# GOOGLE_API_KEY=
-# OPENAI_API_KEY=
-# ANTHROPIC_API_KEY=
-# OPENROUTER_API_KEY=
-# MISTRAL_API_KEY=
+# Proxy configuration
+# Base URL is used by the task agent to talk to the proxy container inside Docker.
+# When running everything locally without Docker networking, replace with http://localhost:10999.
+FF_LLM_PROXY_BASE_URL=http://llm-proxy:4000

-# Agent Configuration
+# Virtual key placeholder. The bootstrap job replaces this with a LiteLLM
+# proxy-issued key on startup so the task agent authenticates via the gateway.
+OPENAI_API_KEY=sk-proxy-default
+
+# LiteLLM proxy configuration
+LITELLM_MASTER_KEY=sk-master-key
+LITELLM_SALT_KEY=choose-a-random-string
+# LiteLLM UI login (defaults to admin/fuzzforge123 if not overridden)
+UI_USERNAME=fuzzforge
+UI_PASSWORD=fuzzforge123
+# Optional: override OTEL exporter endpoint if using a remote collector
+# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
+# LITELLM_DEFAULT_KEY_BUDGET=25
+# LITELLM_DEFAULT_KEY_DURATION=7d
+
+# Upstream provider secrets (ingested by the proxy only). The bootstrapper copies
+# these into volumes/env/.env.litellm so other containers never see the raw keys.
+# LITELLM_OPENAI_API_KEY=
+# LITELLM_ANTHROPIC_API_KEY=
+# LITELLM_GEMINI_API_KEY=
+# LITELLM_MISTRAL_API_KEY=
+# LITELLM_OPENROUTER_API_KEY=
+
+# Agent behaviour
 # DEFAULT_TIMEOUT=120
 # DEFAULT_CONTEXT_ID=default
@@ -0,0 +1,65 @@
+# =============================================================================
+# FuzzForge LiteLLM Proxy Configuration
+# =============================================================================
+# Copy this file to .env and fill in your API keys
+# Bootstrap will automatically create virtual keys for each service
+# =============================================================================
+
+# LiteLLM Proxy Internal Configuration
+# -----------------------------------------------------------------------------
+FF_LLM_PROXY_BASE_URL=http://llm-proxy:4000
+LITELLM_MASTER_KEY=sk-master-test
+LITELLM_SALT_KEY=super-secret-salt
+
+# Default Models (comma-separated, leave empty for model-agnostic access)
+# -----------------------------------------------------------------------------
+# Examples:
+#   openai/gpt-5-mini,openai/text-embedding-3-large
+#   anthropic/claude-sonnet-4-5-20250929,openai/gpt-5-mini
+#   (empty = unrestricted access to all registered models)
+LITELLM_DEFAULT_MODELS=
+
+# Upstream Provider API Keys
+# -----------------------------------------------------------------------------
+# Add your real provider keys here - these are used by the proxy to call LLM providers
+LITELLM_OPENAI_API_KEY=your-openai-key-here
+LITELLM_ANTHROPIC_API_KEY=your-anthropic-key-here
+LITELLM_GEMINI_API_KEY=
+LITELLM_MISTRAL_API_KEY=
+LITELLM_OPENROUTER_API_KEY=
+
+# Virtual Keys Budget & Duration Configuration
+# -----------------------------------------------------------------------------
+# These control the budget and duration for auto-generated virtual keys
+# Task Agent Key - used by task-agent service for A2A LiteLLM calls
+TASK_AGENT_BUDGET=25.0
+TASK_AGENT_DURATION=30d
+
+# Cognee Key - used by Cognee for knowledge graph ingestion and queries
+COGNEE_BUDGET=50.0
+COGNEE_DURATION=30d
+
+# General CLI/SDK Key - used by ff CLI and fuzzforge-sdk
+CLI_BUDGET=100.0
+CLI_DURATION=30d
+
+# Virtual Keys (auto-generated by bootstrap - leave blank)
+# -----------------------------------------------------------------------------
+TASK_AGENT_API_KEY=
+COGNEE_API_KEY=
+OPENAI_API_KEY=
+
+# LiteLLM Proxy Client Configuration
+# -----------------------------------------------------------------------------
+# For CLI and SDK usage (Cognee, ff ingest, etc.)
+LITELLM_PROXY_API_BASE=http://localhost:10999
+LLM_ENDPOINT=http://localhost:10999
+LLM_PROVIDER=openai
+LLM_MODEL=litellm_proxy/gpt-5-mini
+LLM_API_BASE=http://localhost:10999
+LLM_EMBEDDING_MODEL=litellm_proxy/text-embedding-3-large
+
+# UI Access
+# -----------------------------------------------------------------------------
+UI_USERNAME=fuzzforge
+UI_PASSWORD=fuzzforge123
@@ -1,22 +1,89 @@
-# FuzzForge Environment Configuration
+# FuzzForge LiteLLM Proxy Configuration

-This directory contains environment files that are mounted into Docker containers.
+This directory contains configuration for the LiteLLM proxy with model-agnostic virtual keys.
+
+## Quick Start (Fresh Clone)
+
+### 1. Create Your `.env` File
+
+```bash
+cp .env.template .env
+```
+
+### 2. Add Your Provider API Keys
+
+Edit `.env` and add your **real** API keys:
+
+```bash
+LITELLM_OPENAI_API_KEY=sk-proj-YOUR-OPENAI-KEY-HERE
+LITELLM_ANTHROPIC_API_KEY=sk-ant-api03-YOUR-ANTHROPIC-KEY-HERE
+```
+
+### 3. Start Services
+
+```bash
+cd ../..  # Back to repo root
+COMPOSE_PROFILES=secrets docker compose up -d
+```
+
+Bootstrap will automatically:
+- Generate 3 virtual keys with individual budgets
+- Write them to your `.env` file
+- No model restrictions (model-agnostic)

 ## Files

- `.env.example` - Template configuration file
- `.env` - Your actual configuration (create by copying .env.example)
+- **`.env.template`** - Clean template (checked into git)
+- **`.env`** - Your real keys (git ignored, you create this)
+- **`.env.example`** - Legacy example

-## Usage
+## Virtual Keys (Auto-Generated)

-1. Copy the example file:
-   ```bash
-   cp .env.example .env
-   ```
+Bootstrap creates 3 keys with budget controls:

-2. Edit `.env` and add your API keys
+| Key | Budget | Duration | Used By |
+|-----|--------|----------|---------|
+| `OPENAI_API_KEY` | $100 | 30 days | CLI, SDK |
+| `TASK_AGENT_API_KEY` | $25 | 30 days | Task Agent |
+| `COGNEE_API_KEY` | $50 | 30 days | Cognee |

-3. Restart Docker containers to apply changes:
-   ```bash
-   docker-compose restart
-   ```
+All keys are **model-agnostic** by default (no restrictions).
+
+## Using Models
+
+Registered models in `volumes/litellm/proxy_config.yaml`:
+- `gpt-5-mini` → `openai/gpt-5-mini`
+- `claude-sonnet-4-5` → `anthropic/claude-sonnet-4-5-20250929`
+- `text-embedding-3-large` → `openai/text-embedding-3-large`
+
+### Use Registered Aliases:
+
+```bash
+fuzzforge workflow run llm_secret_detection . -n llm_model=gpt-5-mini
+fuzzforge workflow run llm_secret_detection . -n llm_model=claude-sonnet-4-5
+```
+
+### Use Any Model (Direct):
+
+```bash
+# Works without registering first!
+fuzzforge workflow run llm_secret_detection . -n llm_model=openai/gpt-5-nano
+```
+
+## Proxy UI
+
+http://localhost:10999/ui
+- User: `fuzzforge` / Pass: `fuzzforge123`
+
+## Troubleshooting
+
+```bash
+# Check bootstrap logs
+docker compose logs llm-proxy-bootstrap
+
+# Verify keys generated
+grep "API_KEY=" .env | grep -v "^#" | grep -v "your-"
+
+# Restart services
+docker compose restart llm-proxy task-agent
+```
@@ -0,0 +1,26 @@
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  database_url: os.environ/DATABASE_URL
+  store_model_in_db: true
+  store_prompts_in_spend_logs: true
+  otel: true
+
+litellm_settings:
+  callbacks:
+    - "otel"
+
+model_list:
+  - model_name: claude-sonnet-4-5
+    litellm_params:
+      model: anthropic/claude-sonnet-4-5-20250929
+      api_key: os.environ/ANTHROPIC_API_KEY
+
+  - model_name: gpt-5-mini
+    litellm_params:
+      model: openai/gpt-5-mini
+      api_key: os.environ/LITELLM_OPENAI_API_KEY
+
+  - model_name: text-embedding-3-large
+    litellm_params:
+      model: openai/text-embedding-3-large
+      api_key: os.environ/LITELLM_OPENAI_API_KEY
@@ -0,0 +1,25 @@
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  batch:
+
+exporters:
+  debug:
+    verbosity: detailed
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]