diff --git a/README.md b/README.md index 7ce42f97..64a86676 100644 --- a/README.md +++ b/README.md @@ -118,6 +118,7 @@ CyberStrikeAI connects planning, execution, human oversight, evidence, and repla - 🧰 **Security tools** include 100+ curated YAML recipes with custom extensions and role-scoped access. - 🔌 **MCP integration** supports HTTP, stdio, SSE, external federation, and dynamic tool discovery. +- ⏱️ **Resilient tool execution** runs blocking MCP/tool calls in workers with bounded agent waits, resumable `execution_id` polling, cancellation, per-server circuit breakers, concurrency limits, and unified output caps. - 🎯 **Agent Skills** follow the standard Skill layout and support progressive, on-demand loading. - 📚 **Knowledge base** combines query rewriting, vector retrieval, reranking, and result post-processing. - 🖼️ **Vision analysis** uses a separate vision model for screenshots, captchas, and UI while retaining text summaries only. @@ -127,7 +128,7 @@ CyberStrikeAI connects planning, execution, human oversight, evidence, and repla - 🧑‍⚖️ **Human in the loop** provides approval modes, tool allowlists, audit-agent review, and traceable decisions. - 🔐 **Platform RBAC** supports multiple users, system and custom roles, scoped permissions, ownership, and explicit assignments. - 🔒 **Security and audit** provide authenticated access, audit logs, SQLite persistence, and operational evidence retention. -- 📄 **Result governance** supports pagination, compression, archival, and search for large tool outputs. +- 📄 **Result governance** stores the same capped tool result seen by the agent, protects resume paths from oversized historical output, and adds UI safeguards for large detail views. See [Tool Execution Governance](docs/en-US/tool-execution-governance.md). ### Security operations diff --git a/README_CN.md b/README_CN.md index b126e079..aa0f01ad 100644 --- a/README_CN.md +++ b/README_CN.md @@ -117,6 +117,7 @@ CyberStrikeAI 将规划、执行、人工监督、证据与复盘连接在同一 - 🧰 **安全工具**:提供 100+ 精选 YAML 工具配方,支持自定义扩展和按角色控制。 - 🔌 **MCP 集成**:支持 HTTP、stdio、SSE、外部 MCP 联邦和动态工具发现。 +- ⏱️ **弹性工具执行**:阻塞型 MCP/工具调用交给 worker 执行,Agent 只有限等待;支持 `execution_id` 多轮等待、主动取消、单 server 熔断、并发限制和统一输出兜底。 - 🎯 **Agent Skills**:遵循标准 Skill 目录结构,支持渐进式按需加载。 - 📚 **知识库**:组合查询改写、向量检索、精排和结果后处理能力。 - 🖼️ **视觉分析**:使用独立视觉模型分析截图、验证码和 UI,对话中仅保留文字摘要。 @@ -126,7 +127,7 @@ CyberStrikeAI 将规划、执行、人工监督、证据与复盘连接在同一 - 🧑‍⚖️ **人机协同**:支持审批模式、工具白名单、审计 Agent 复核和决策追踪。 - 🔐 **平台 RBAC**:支持多用户、系统及自定义角色、权限 Scope、资源归属和显式授权。 - 🔒 **安全与审计**:提供登录保护、审计日志、SQLite 持久化和行动证据留存。 -- 📄 **结果治理**:支持大结果分页、压缩、归档和检索。 +- 📄 **结果治理**:数据库保存与 Agent 实际看到的同一份兜底后工具结果,恢复路径会再次防御历史超大输出,前端详情也有展示保护。详见[工具执行治理](docs/zh-CN/tool-execution-governance.md)。 ### 安全运营管理 diff --git a/docs/README.md b/docs/README.md index d6208022..38853195 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,6 +21,7 @@ CyberStrikeAI documentation is organized by user journey. Start with deployment, - [Skills 指南](zh-CN/skills-guide.md) - [Eino 多代理](zh-CN/MULTI_AGENT_EINO.md) - [工作流](zh-CN/workflow-graph.md) +- [工具执行治理](zh-CN/tool-execution-governance.md) - [人机协同最佳实践](zh-CN/hitl-best-practices.md) ### 功能指南 @@ -58,6 +59,7 @@ CyberStrikeAI documentation is organized by user journey. Start with deployment, - [Skills](en-US/skills-guide.md) - [Eino Multi-Agent](en-US/MULTI_AGENT_EINO.md) - [Workflows](en-US/workflow-graph.md) +- [Tool Execution Governance](en-US/tool-execution-governance.md) - [HITL Best Practices](en-US/hitl-best-practices.md) ### Feature guides diff --git a/docs/en-US/README.md b/docs/en-US/README.md index 2f53b7d1..6800de6c 100644 --- a/docs/en-US/README.md +++ b/docs/en-US/README.md @@ -13,7 +13,7 @@ - [Architecture](architecture.md) · [Security Model](security-model.md) · [RBAC](rbac.md) - [Agents and Roles](agent-and-role-guide.md) · [Skills](skills-guide.md) · [Eino Multi-Agent](MULTI_AGENT_EINO.md) -- [Workflows](workflow-graph.md) · [HITL Best Practices](hitl-best-practices.md) +- [Workflows](workflow-graph.md) · [Tool Execution Governance](tool-execution-governance.md) · [HITL Best Practices](hitl-best-practices.md) ## Feature guides diff --git a/docs/en-US/tool-execution-governance.md b/docs/en-US/tool-execution-governance.md new file mode 100644 index 00000000..f6ce8fd9 --- /dev/null +++ b/docs/en-US/tool-execution-governance.md @@ -0,0 +1,205 @@ +# Tool Execution Governance + +[Back to English documentation](README.md) + +This document describes how CyberStrikeAI governs long-running tools, blocking MCP calls, oversized outputs, cancellation, and context restore. The goal is to preserve standard Agent/Eino tool semantics while preventing tool hangs, context blowups, oversized database records, and unsafe resume behavior. + +## Goals + +- **Keep the agent runner responsive**: tools may run for a long time, but the current runner waits only for a bounded interval. +- **Allow long tasks to continue**: timeout returns an `execution_id`; later turns can call `wait_tool_execution`. +- **Support cancellation**: users and agents can cancel a running execution. +- **Keep DB and agent views identical**: the database stores the same canonical capped result returned to the agent. +- **Protect resume paths**: resume uses model-facing traces and caps historical oversized tool traces. +- **Isolate external MCP failures**: external MCP servers are protected by per-server concurrency limits, global concurrency limits, and circuit breakers. + +## Execution Model + +Tool calls still appear to Eino/Agent as standard tool invocations, but the blocking work runs in a worker: + +```text +Agent calls tool + -> ExecutionService creates execution + -> worker runs the real MCP/tool call + -> Agent bounded wait + -> completed: return tool result + -> still running: return execution_id, worker continues in background +``` + +This prevents MCP servers, `exec`, `sqlmap`, `nmap`, `nuclei`, and similar long-running tools from binding the current runner indefinitely. + +## Execution Statuses + +| Status | Meaning | +|---|---| +| `queued` | Execution exists and is waiting for a worker or concurrency slot | +| `running` | Worker is executing | +| `background_running` | UI display state: agent stopped waiting, background worker continues | +| `completed` | This tool call completed | +| `failed` | The tool actually failed | +| `cancelled` | User, agent, or session cleanup cancelled the execution | +| `hard_timeout` | The tool exceeded its hard timeout | +| `orphaned` | A persisted running execution no longer has a runtime worker | + +Important: when `wait_tool_execution` reaches `timeout_seconds` and the target execution is still running, the wait call itself is a completed observation, not a failed tool execution. + +## Control Tools + +| Tool | Purpose | +|---|---| +| `get_tool_execution` | Read current execution state | +| `wait_tool_execution` | Wait for a selected execution for a bounded interval | +| `cancel_tool_execution` | Cancel a selected execution | + +Typical flow: + +```text +1. Call a long-running tool such as exec/sqlmap/nmap +2. After tool_wait_timeout_seconds, receive execution_id +3. Agent can continue reasoning, use other tools, or call wait_tool_execution +4. If still incomplete, continue waiting or call cancel_tool_execution +``` + +## Cancellation and Session Cleanup + +- User stop cancels running tools for the current conversation. +- Normal session end cancels remaining running tools for the current conversation. +- Interrupt-and-continue style flows do not mass-cancel tools. +- Conversation-scoped cancellation avoids killing tools from other conversations. + +## External MCP Isolation + +External MCP servers can hang, disconnect, or return failures. CyberStrikeAI uses three protections: + +| Capability | Config | Description | +|---|---|---| +| Per-server concurrency | `external_mcp_max_concurrent_per_server` | Max simultaneous calls for one external MCP server | +| Global concurrency | `external_mcp_max_concurrent_total` | Max simultaneous external MCP calls across all servers | +| Circuit breaker | `external_mcp_circuit_failure_threshold` / `external_mcp_circuit_cooldown_seconds` | Temporarily fast-fails a server after repeated failures | + +Recommended defaults: + +```yaml +agent: + external_mcp_max_concurrent_per_server: 2 + external_mcp_max_concurrent_total: 16 + external_mcp_circuit_failure_threshold: 3 + external_mcp_circuit_cooldown_seconds: 60 +``` + +## Output Governance + +CyberStrikeAI uses `multi_agent.eino_middleware.reduction_max_length_for_trunc` as the unified tool result cap. The example configuration uses 50000 bytes. + +```yaml +multi_agent: + eino_middleware: + reduction_enable: true + reduction_max_length_for_trunc: 50000 +``` + +Coverage: + +| Channel | Behavior | +|---|---| +| Agent-facing tool result | Uses the canonical capped result | +| DB / monitor storage | Stores the same canonical result | +| `get_tool_execution` / `wait_tool_execution` | Reads the same canonical result | +| Eino `execute` / filesystem monitor records | Capped before completion is persisted | +| Non-streaming `exec` stdout/stderr | Source-side bounded buffer | +| Streaming `exec` stdout/stderr | Streamed UI output is also bounded | +| PTY execution path | Uses the same output cap | +| Frontend detail modal | Has an additional UI display cap | + +When the cap is reached, CyberStrikeAI keeps a prefix and appends a truncation marker. The marker is counted inside the configured budget, so a 50000-byte cap does not become `50000 + marker`. + +Example: + +```text + + +...[tool output limit reached: kept 50000 bytes; further output suppressed]... +``` + +The current strategy is “prefix + marker.” For stronger auditability, a future enhancement can use “head 25k + tail 25k + full-output artifact path.” + +## Database and Resume Context + +New results are written through this path: + +```text +tool completes + -> NormalizeToolResultForStorage + -> update in-memory execution + -> persist to DB + -> return to Agent +``` + +So, under normal operation, the DB stores exactly the result returned to the agent. + +Resume uses `LastAgentTraceInput`, which is the model-facing trace that actually reached ChatModel, not raw event accumulation. The restore path also caps historical tool content to prevent context blowups from: + +- pre-upgrade DB records that contain raw large output, +- manual imports or migrations, +- lowering the configured cap from a larger value, +- future bypasses that accidentally skip canonicalization. + +## Recommended Configuration + +For long-running security tasks: + +```yaml +agent: + max_iterations: 800 + tool_timeout_minutes: 60 + tool_wait_timeout_seconds: 30 + external_mcp_max_concurrent_per_server: 2 + external_mcp_max_concurrent_total: 16 + external_mcp_circuit_failure_threshold: 3 + external_mcp_circuit_cooldown_seconds: 60 + shell_no_output_timeout_seconds: 1200 + +multi_agent: + eino_middleware: + reduction_enable: true + reduction_max_length_for_trunc: 50000 +``` + +| Parameter | Recommended | Notes | +|---|---:|---| +| `max_iterations` | `300-1000` | Very large values weaken loop protection | +| `tool_timeout_minutes` | `60` | Hard timeout for tools such as sqlmap | +| `tool_wait_timeout_seconds` | `30-60` | Agent wait bound before returning `execution_id` | +| `shell_no_output_timeout_seconds` | `600-1200` | Kills silent hangs | +| `reduction_max_length_for_trunc` | `50000` | Unified tool result cap | + +Do not make `tool_wait_timeout_seconds` very large by default. Long tasks should continue in workers and be observed by `execution_id`, rather than blocking one turn for several minutes. + +## Testing + +Long-task test prompt: + +```text +Call exec to run sleep 120. If it is not done after 10 seconds, do not keep waiting; report execution_id, call wait_tool_execution for 5 seconds, then cancel_tool_execution if still incomplete and report final status. +``` + +Large-output test prompt: + +```text +Call exec to run: python3 - <<'PY' +print("A" * 200000) +PY +Then show the tool result length and whether it contains the truncation marker. +``` + +Expected behavior: + +- The initial long task returns an `execution_id` and status `running` or UI `background_running`. +- `wait_tool_execution` timing out while the target is still running is not displayed as a tool failure. +- Large output never exceeds `reduction_max_length_for_trunc`. +- DB, monitor details, and agent continuation use the same capped result. + +## Boundaries + +- CyberStrikeAI cannot control how a remote external MCP server collects output internally; it caps results after they enter CyberStrikeAI and protects calls with concurrency limits and circuit breakers. +- Full original output is not automatically written to an artifact path today. If complete audit recovery is required, add spill-to-file and include the artifact path in the bounded result. diff --git a/docs/zh-CN/README.md b/docs/zh-CN/README.md index 2b2b4dec..a0029dd6 100644 --- a/docs/zh-CN/README.md +++ b/docs/zh-CN/README.md @@ -13,7 +13,7 @@ - [架构说明](architecture.md) · [安全模型](security-model.md) · [RBAC](rbac.md) - [Agent 与角色](agent-and-role-guide.md) · [Skills](skills-guide.md) · [Eino 多代理](MULTI_AGENT_EINO.md) -- [工作流](workflow-graph.md) · [人机协同最佳实践](hitl-best-practices.md) +- [工作流](workflow-graph.md) · [工具执行治理](tool-execution-governance.md) · [人机协同最佳实践](hitl-best-practices.md) ## 功能指南 diff --git a/docs/zh-CN/tool-execution-governance.md b/docs/zh-CN/tool-execution-governance.md new file mode 100644 index 00000000..f07f9d59 --- /dev/null +++ b/docs/zh-CN/tool-execution-governance.md @@ -0,0 +1,207 @@ +# 工具执行治理 + +[返回中文文档](README.md) + +本文说明 CyberStrikeAI 对长时间工具、MCP 阻塞、大输出、取消和恢复上下文的治理策略。目标是让 Agent 保持标准工具语义,同时避免工具卡死、上下文爆炸、数据库膨胀或恢复时重新注入历史大输出。 + +## 设计目标 + +- **Agent 不被工具绑死**:工具调用可以很慢,但当前 runner 只等待有限时间。 +- **长任务可继续观察**:超时返回 `execution_id`,后续可用 `wait_tool_execution` 多轮等待。 +- **用户和 Agent 都能取消**:当前会话结束或用户停止任务时,会取消仍在运行的工具。 +- **数据库与 Agent 视图一致**:DB 保存的是 Agent 实际拿到的兜底后结果,不再保存另一份原始大输出。 +- **恢复不会撑爆上下文**:续跑使用 model-facing trace;历史异常大 tool trace 恢复时也会再次裁剪。 +- **外部 MCP 有隔离保护**:按 server 限并发、按全局限并发,并对连续失败的 server 熔断。 + +## 执行模型 + +普通工具调用仍然对 Eino/Agent 表现为一次标准 tool call,但底层执行分为两段: + +```text +Agent 调用工具 + -> ExecutionService 创建 execution + -> worker 执行真实 MCP/工具调用 + -> Agent bounded wait + -> 完成:返回工具结果 + -> 未完成:返回 execution_id,worker 继续后台运行 +``` + +这解决了 MCP server、`exec`、`sqlmap`、`nmap`、`nuclei` 等长任务阻塞当前 runner 的问题。 + +## 工具状态语义 + +| 状态 | 含义 | +|---|---| +| `queued` | execution 已创建,等待 worker 或并发槽位 | +| `running` | worker 正在执行 | +| `background_running` | 前端展示状态,表示本轮 Agent 已停止等待,但后台仍在跑 | +| `completed` | 本次 tool call 本身已完成 | +| `failed` | 工具真实失败 | +| `cancelled` | 用户、Agent 或会话清理主动取消 | +| `hard_timeout` | 超过硬超时,被系统终止 | +| `orphaned` | 重启/异常后发现 DB 中仍是 running,但运行时已无对应 worker | + +注意:`wait_tool_execution` 到达 `timeout_seconds` 时,如果目标 execution 仍在运行,**这次 wait 调用本身是完成的观察动作**,不是工具执行失败。返回体会说明目标仍为 `running`,前端不应显示为红色失败。 + +## 控制工具 + +| 工具 | 用途 | +|---|---| +| `get_tool_execution` | 读取 execution 当前状态 | +| `wait_tool_execution` | 等待指定 execution 一段时间 | +| `cancel_tool_execution` | 主动取消指定 execution | + +典型流程: + +```text +1. 调用 exec/sqlmap/nmap 等长任务 +2. 超过 tool_wait_timeout_seconds 后拿到 execution_id +3. Agent 可继续推理、改用其他工具,或调用 wait_tool_execution +4. 仍未完成时可继续等待,或调用 cancel_tool_execution +``` + +## 取消和会话清理 + +- 用户点击“停止任务”时,会取消当前会话仍在运行的工具。 +- 会话正常结束后,会批量取消当前会话仍 `running` 的工具。 +- “中断并继续”类流程不会做会话级批量取消,以免误杀后续需要等待的 worker。 +- 取消只针对当前 conversation 绑定的 execution,不会误杀其他会话的工具。 + +## 外部 MCP 隔离 + +外部 MCP 可能因为远端 server 卡住、断连或返回异常而拖慢 Agent。系统提供三层保护: + +| 能力 | 配置 | 说明 | +|---|---|---| +| 单 server 并发限制 | `external_mcp_max_concurrent_per_server` | 同一个外部 MCP server 同时运行的工具数 | +| 全局并发限制 | `external_mcp_max_concurrent_total` | 所有外部 MCP 工具总并发 | +| 熔断 | `external_mcp_circuit_failure_threshold` / `external_mcp_circuit_cooldown_seconds` | 单 server 连续失败后短期快速失败,避免反复打坏 server | + +推荐默认: + +```yaml +agent: + external_mcp_max_concurrent_per_server: 2 + external_mcp_max_concurrent_total: 16 + external_mcp_circuit_failure_threshold: 3 + external_mcp_circuit_cooldown_seconds: 60 +``` + +## 输出兜底 + +系统使用 `multi_agent.eino_middleware.reduction_max_length_for_trunc` 作为统一工具结果上限。当前示例配置为 50000 bytes。 + +```yaml +multi_agent: + eino_middleware: + reduction_enable: true + reduction_max_length_for_trunc: 50000 +``` + +兜底覆盖: + +| 渠道 | 行为 | +|---|---| +| Agent 实际拿到的工具结果 | 使用兜底后的 canonical result | +| DB/监控存储 | 保存同一份 canonical result | +| `get_tool_execution` / `wait_tool_execution` | 读取同一份 canonical result | +| Eino `execute` / filesystem 监控记录 | 完成记录前统一兜底 | +| 非流式 `exec` stdout/stderr | 源头 bounded buffer | +| 流式 `exec` stdout/stderr | 推送给前端的累计输出也受上限控制 | +| PTY 执行路径 | 同样受上限控制 | +| 前端详情弹窗 | 额外有 UI 展示截断保护 | + +触发上限后,系统会保留前段内容并追加截断 marker,marker 也计入总预算。因此当阈值为 50000 时,最终文本不会变成 `50000 + marker`。 + +示例: + +```text + + +...[tool output limit reached: kept 50000 bytes; further output suppressed]... +``` + +当前策略是“前段内容 + marker”。如果需要更强的审计体验,可以后续升级为“前 25k + 后 25k + 完整原文 artifact 路径”。 + +## DB 与恢复上下文 + +新执行结果的写入路径如下: + +```text +工具完成 + -> NormalizeToolResultForStorage + -> 写入内存 execution + -> 写入 DB + -> 返回给 Agent +``` + +因此正常情况下,DB 中保存的就是 Agent 拿到的结果。 + +续跑恢复时,系统使用 `LastAgentTraceInput` 中的 model-facing trace,也就是实际送入 ChatModel 的消息快照,而不是原始事件流累计。恢复入口还会对历史 tool 内容再次应用上限,防止以下情况撑爆上下文: + +- 升级前 DB 已经存过原始大输出。 +- 手工迁移或导入的数据绕过了当前写入路径。 +- 配置从更大阈值改成 50000。 +- 未来某条旁路写入漏掉 canonicalize。 + +## 关键配置建议 + +长任务场景推荐: + +```yaml +agent: + max_iterations: 800 + tool_timeout_minutes: 60 + tool_wait_timeout_seconds: 30 + external_mcp_max_concurrent_per_server: 2 + external_mcp_max_concurrent_total: 16 + external_mcp_circuit_failure_threshold: 3 + external_mcp_circuit_cooldown_seconds: 60 + shell_no_output_timeout_seconds: 1200 + +multi_agent: + eino_middleware: + reduction_enable: true + reduction_max_length_for_trunc: 50000 +``` + +参数说明: + +| 参数 | 建议 | 说明 | +|---|---:|---| +| `max_iterations` | `300-1000` | 太大等于放弃循环保护 | +| `tool_timeout_minutes` | `60` | 单次工具硬超时,适合 sqlmap 等长任务 | +| `tool_wait_timeout_seconds` | `30-60` | Agent 本轮等待上限,到时返回 `execution_id` | +| `shell_no_output_timeout_seconds` | `600-1200` | 连续无输出时终止,防止静默挂死 | +| `reduction_max_length_for_trunc` | `50000` | 工具结果统一上限 | + +不建议把 `tool_wait_timeout_seconds` 设置得很大。长任务应由 worker 后台跑,Agent 通过 `execution_id` 继续观察,而不是一轮等待数分钟。 + +## 测试建议 + +可以用以下对话测试长任务语义: + +```text +调用 exec 执行 sleep 120;如果超过 10 秒还没完成,不要一直等,告诉我 execution_id,然后调用 wait_tool_execution 等 5 秒;如果仍未完成,再调用 cancel_tool_execution,最后说明状态。 +``` + +可以用以下命令测试大输出兜底: + +```text +调用 exec 执行:python3 - <<'PY' +print("A" * 200000) +PY +然后展示工具结果长度和是否包含截断提示。 +``` + +预期: + +- 初始长任务会返回 `execution_id`,状态为 `running` 或前端展示 `background_running`。 +- `wait_tool_execution` 等待到上限但目标未完成时,本次 wait 调用不应显示为执行失败。 +- 大输出结果不会超过 `reduction_max_length_for_trunc`。 +- DB、监控详情、Agent 继续推理看到的是同一份兜底结果。 + +## 当前边界 + +- 外部 MCP 的远端 server 内部如何采集输出不由 CyberStrikeAI 控制;CyberStrikeAI 会在结果进入本系统后统一兜底、限并发和熔断。 +- 当前不会为被截断的完整原文自动生成 artifact 路径;如果需要完整审计链路,建议后续实现 spill-to-file,并在 bounded result 中返回文件路径。