mirror of
https://github.com/Ed1s0nZ/CyberStrikeAI.git
synced 2026-05-16 21:23:29 +02:00
Compare commits
24 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0b950f95db | |||
| d36984a1c1 | |||
| da2109a970 | |||
| 1866aa8089 | |||
| 5af06e539d | |||
| 7493e70686 | |||
| 81f7a601b7 | |||
| 27830d1399 | |||
| d9a0178f80 | |||
| 1dd8cc7f50 | |||
| 55045dd4e0 | |||
| 90508c9084 | |||
| 361480f2d1 | |||
| 538565117b | |||
| 1c8742b7b6 | |||
| 2fb6a1d1ef | |||
| 6e390acb3d | |||
| d6236e285d | |||
| ad8efffbb4 | |||
| 352d9b712c | |||
| acadbe19c6 | |||
| c265e66afb | |||
| 647bb4b5e4 | |||
| dd311f7a3b |
@@ -501,20 +501,6 @@ Compress the 5 MB nuclei report, summarize critical CVEs, and attach the artifac
|
||||
Build an attack chain for the latest engagement and export the node list with severity >= high.
|
||||
```
|
||||
|
||||
## Changelog
|
||||
|
||||
### Recent Highlights
|
||||
|
||||
- **2026-01-27** – OpenAPI documentation with interactive testing interface, supporting conversation management, message interaction, and result querying
|
||||
- **2026-01-15** – Skills system with 20+ predefined security testing skills
|
||||
- **2026-01-11** – Role-based testing with predefined security testing roles
|
||||
- **2026-01-08** – SSE transport mode support for external MCP servers
|
||||
- **2026-01-01** – Batch task management with queue-based execution
|
||||
- **2025-12-25** – Vulnerability management and conversation grouping features
|
||||
- **2025-12-20** – Knowledge base with vector search and hybrid retrieval
|
||||
|
||||
|
||||
|
||||
## 404Starlink
|
||||
|
||||
<img src="./images/404StarLinkLogo.png" width="30%">
|
||||
@@ -532,6 +518,22 @@ CyberStrikeAI has joined [404Starlink](https://github.com/knownsec/404StarLink)
|
||||

|
||||
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Disclaimer
|
||||
|
||||
**This tool is for educational and authorized testing purposes only!**
|
||||
|
||||
CyberStrikeAI is a professional security testing platform designed to assist security researchers, penetration testers, and IT professionals in conducting security assessments and vulnerability research **with explicit authorization**.
|
||||
|
||||
**By using this tool, you agree to:**
|
||||
- Use this tool only on systems where you have clear written authorization
|
||||
- Comply with all applicable laws, regulations, and ethical standards
|
||||
- Take full responsibility for any unauthorized use or misuse
|
||||
- Not use this tool for any illegal or malicious purposes
|
||||
|
||||
**The developers are not responsible for any misuse!** Please ensure your usage complies with local laws and regulations, and that you have obtained explicit authorization from the target system owner.
|
||||
|
||||
---
|
||||
|
||||
Need help or want to contribute? Open an issue or PR—community tooling additions are welcome!
|
||||
|
||||
+16
-13
@@ -500,19 +500,6 @@ CyberStrikeAI/
|
||||
构建最新一次测试的攻击链,只导出风险 >= 高的节点列表。
|
||||
```
|
||||
|
||||
## 更新日志
|
||||
|
||||
### 近期亮点
|
||||
|
||||
- **2026-01-27** – 新增 OpenAPI 文档,提供交互式测试界面,支持对话管理、消息交互和结果查询
|
||||
- **2026-01-15** – 新增 Skills 技能系统,内置 20+ 预设安全测试技能
|
||||
- **2026-01-11** – 新增角色化测试功能,支持预设安全测试角色
|
||||
- **2026-01-08** – 新增 SSE 传输模式支持,外部 MCP 联邦支持三种模式
|
||||
- **2026-01-01** – 新增批量任务管理功能,支持队列式任务执行
|
||||
- **2025-12-25** – 新增漏洞管理和对话分组功能
|
||||
- **2025-12-20** – 新增知识库功能,支持向量检索和混合搜索
|
||||
|
||||
|
||||
## 404星链计划
|
||||
<img src="./images/404StarLinkLogo.png" width="30%">
|
||||
|
||||
@@ -530,4 +517,20 @@ CyberStrikeAI 现已加入 [404星链计划](https://github.com/knownsec/404Star
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 免责声明
|
||||
|
||||
**本工具仅供教育和授权测试使用!**
|
||||
|
||||
CyberStrikeAI 是一个专业的安全测试平台,旨在帮助安全研究人员、渗透测试人员和IT专业人员在**获得明确授权**的情况下进行安全评估和漏洞研究。
|
||||
|
||||
**使用本工具即表示您同意:**
|
||||
- 仅在您拥有明确书面授权的系统上使用此工具
|
||||
- 遵守所有适用的法律法规和道德准则
|
||||
- 对任何未经授权的使用或滥用行为承担全部责任
|
||||
- 不会将本工具用于任何非法或恶意目的
|
||||
|
||||
**开发者不对任何滥用行为负责!** 请确保您的使用符合当地法律法规,并获得目标系统所有者的明确授权。
|
||||
|
||||
---
|
||||
|
||||
欢迎提交 Issue/PR 贡献新的工具模版或优化建议!
|
||||
|
||||
+17
-1
@@ -10,7 +10,7 @@
|
||||
# ============================================
|
||||
|
||||
# 前端显示的版本号(可选,不填则显示默认版本)
|
||||
version: "v1.3.13"
|
||||
version: "v1.3.18"
|
||||
|
||||
# 服务器配置
|
||||
server:
|
||||
@@ -116,6 +116,22 @@ knowledge:
|
||||
top_k: 5 # 检索返回的Top-K结果数量
|
||||
similarity_threshold: 0.7 # 相似度阈值(0-1),低于此值的结果将被过滤
|
||||
hybrid_weight: 0.7 # 混合检索权重(0-1),向量检索的权重,1.0表示纯向量检索,0.0表示纯关键词检索
|
||||
# ============================================
|
||||
# 索引配置(用于解决 API 限制问题)
|
||||
# ============================================
|
||||
indexing:
|
||||
# 分块配置
|
||||
chunk_size: 512 # 每个块的最大 token 数(默认 512),长文本会被分割成多个块
|
||||
chunk_overlap: 50 # 块之间的重叠 token 数(默认 50),保持上下文连贯性
|
||||
max_chunks_per_item: 0 # 单个知识项的最大块数量(0 表示不限制),防止单个文件消耗过多 API 配额
|
||||
# 速率限制配置(解决 429 错误)
|
||||
max_rpm: 0 # 每分钟最大请求数(默认 0 表示不限制),如 OpenAI 默认 200 RPM
|
||||
rate_limit_delay_ms: 300 # 请求间隔毫秒数(默认 300),用于避免 API 速率限制,设为 0 不限制
|
||||
# 建议值:200 次/分钟≈300ms, 100 次/分钟≈600ms
|
||||
|
||||
# 重试配置
|
||||
max_retries: 3 # 最大重试次数(默认 3),遇到速率限制或服务器错误时自动重试
|
||||
retry_delay_ms: 1000 # 重试间隔毫秒数(默认 1000),每次重试会递增延迟
|
||||
|
||||
# ============================================
|
||||
# 机器人配置(企业微信、钉钉、飞书)
|
||||
|
||||
@@ -1,18 +1,21 @@
|
||||
module cyberstrike-ai
|
||||
|
||||
go 1.23.0
|
||||
go 1.24.0
|
||||
|
||||
toolchain go1.24.4
|
||||
|
||||
require (
|
||||
github.com/creack/pty v1.1.24
|
||||
github.com/gin-gonic/gin v1.9.1
|
||||
github.com/google/uuid v1.5.0
|
||||
github.com/gorilla/websocket v1.5.0
|
||||
github.com/larksuite/oapi-sdk-go/v3 v3.4.22
|
||||
github.com/mattn/go-sqlite3 v1.14.18
|
||||
github.com/modelcontextprotocol/go-sdk v1.2.0
|
||||
github.com/open-dingtalk/dingtalk-stream-sdk-go v0.9.1
|
||||
github.com/pkoukk/tiktoken-go v0.1.8
|
||||
go.uber.org/zap v1.26.0
|
||||
golang.org/x/time v0.14.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
@@ -28,7 +31,6 @@ require (
|
||||
github.com/goccy/go-json v0.10.2 // indirect
|
||||
github.com/gogo/protobuf v1.3.2 // indirect
|
||||
github.com/google/jsonschema-go v0.3.0 // indirect
|
||||
github.com/gorilla/websocket v1.5.0 // indirect
|
||||
github.com/json-iterator/go v1.1.12 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.4 // indirect
|
||||
github.com/leodido/go-urn v1.2.4 // indirect
|
||||
|
||||
@@ -4,6 +4,8 @@ github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZX
|
||||
github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
|
||||
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams=
|
||||
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk=
|
||||
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
|
||||
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
@@ -127,6 +129,8 @@ golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=
|
||||
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
|
||||
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
|
||||
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
|
||||
|
||||
+10
-2
@@ -198,7 +198,7 @@ func New(cfg *config.Config, log *logger.Logger) (*App, error) {
|
||||
knowledgeRetriever = knowledge.NewRetriever(knowledgeDB, embedder, retrievalConfig, log.Logger)
|
||||
|
||||
// 创建索引器
|
||||
knowledgeIndexer = knowledge.NewIndexer(knowledgeDB, embedder, log.Logger)
|
||||
knowledgeIndexer = knowledge.NewIndexer(knowledgeDB, embedder, log.Logger, &cfg.Knowledge.Indexing)
|
||||
|
||||
// 注册知识检索工具到MCP服务器
|
||||
knowledge.RegisterKnowledgeTool(mcpServer, knowledgeRetriever, knowledgeManager, log.Logger)
|
||||
@@ -325,6 +325,7 @@ func New(cfg *config.Config, log *logger.Logger) (*App, error) {
|
||||
roleHandler.SetSkillsManager(skillsManager) // 设置Skills管理器到RoleHandler
|
||||
skillsHandler := handler.NewSkillsHandler(skillsManager, cfg, configPath, log.Logger)
|
||||
fofaHandler := handler.NewFofaHandler(cfg, log.Logger)
|
||||
terminalHandler := handler.NewTerminalHandler(log.Logger)
|
||||
if db != nil {
|
||||
skillsHandler.SetDB(db) // 设置数据库连接以便获取调用统计
|
||||
}
|
||||
@@ -431,6 +432,7 @@ func New(cfg *config.Config, log *logger.Logger) (*App, error) {
|
||||
roleHandler,
|
||||
skillsHandler,
|
||||
fofaHandler,
|
||||
terminalHandler,
|
||||
mcpServer,
|
||||
authManager,
|
||||
openAPIHandler,
|
||||
@@ -542,6 +544,7 @@ func setupRoutes(
|
||||
roleHandler *handler.RoleHandler,
|
||||
skillsHandler *handler.SkillsHandler,
|
||||
fofaHandler *handler.FofaHandler,
|
||||
terminalHandler *handler.TerminalHandler,
|
||||
mcpServer *mcp.Server,
|
||||
authManager *security.AuthManager,
|
||||
openAPIHandler *handler.OpenAPIHandler,
|
||||
@@ -628,6 +631,11 @@ func setupRoutes(
|
||||
protected.PUT("/config", configHandler.UpdateConfig)
|
||||
protected.POST("/config/apply", configHandler.ApplyConfig)
|
||||
|
||||
// 系统设置 - 终端(执行命令,提高运维效率)
|
||||
protected.POST("/terminal/run", terminalHandler.RunCommand)
|
||||
protected.POST("/terminal/run/stream", terminalHandler.RunCommandStream)
|
||||
protected.GET("/terminal/ws", terminalHandler.RunCommandWS)
|
||||
|
||||
// 外部MCP管理
|
||||
protected.GET("/external-mcp", externalMCPHandler.GetExternalMCPs)
|
||||
protected.GET("/external-mcp/stats", externalMCPHandler.GetExternalMCPStats)
|
||||
@@ -1094,7 +1102,7 @@ func initializeKnowledge(
|
||||
knowledgeRetriever := knowledge.NewRetriever(knowledgeDB, embedder, retrievalConfig, logger)
|
||||
|
||||
// 创建索引器
|
||||
knowledgeIndexer := knowledge.NewIndexer(knowledgeDB, embedder, logger)
|
||||
knowledgeIndexer := knowledge.NewIndexer(knowledgeDB, embedder, logger, &cfg.Knowledge.Indexing)
|
||||
|
||||
// 注册知识检索工具到MCP服务器
|
||||
knowledge.RegisterKnowledgeTool(mcpServer, knowledgeRetriever, knowledgeManager, logger)
|
||||
|
||||
@@ -595,6 +595,26 @@ type KnowledgeConfig struct {
|
||||
BasePath string `yaml:"base_path" json:"base_path"` // 知识库路径
|
||||
Embedding EmbeddingConfig `yaml:"embedding" json:"embedding"`
|
||||
Retrieval RetrievalConfig `yaml:"retrieval" json:"retrieval"`
|
||||
Indexing IndexingConfig `yaml:"indexing,omitempty" json:"indexing,omitempty"` // 索引构建配置
|
||||
}
|
||||
|
||||
// IndexingConfig 索引构建配置(用于控制知识库索引构建时的行为)
|
||||
type IndexingConfig struct {
|
||||
// 分块配置
|
||||
ChunkSize int `yaml:"chunk_size,omitempty" json:"chunk_size,omitempty"` // 每个块的最大 token 数(估算),默认 512
|
||||
ChunkOverlap int `yaml:"chunk_overlap,omitempty" json:"chunk_overlap,omitempty"` // 块之间的重叠 token 数,默认 50
|
||||
MaxChunksPerItem int `yaml:"max_chunks_per_item,omitempty" json:"max_chunks_per_item,omitempty"` // 单个知识项的最大块数量,0 表示不限制
|
||||
|
||||
// 速率限制配置(用于避免 API 速率限制)
|
||||
RateLimitDelayMs int `yaml:"rate_limit_delay_ms,omitempty" json:"rate_limit_delay_ms,omitempty"` // 请求间隔时间(毫秒),0 表示不使用固定延迟
|
||||
MaxRPM int `yaml:"max_rpm,omitempty" json:"max_rpm,omitempty"` // 每分钟最大请求数,0 表示不限制
|
||||
|
||||
// 重试配置(用于处理临时错误)
|
||||
MaxRetries int `yaml:"max_retries,omitempty" json:"max_retries,omitempty"` // 最大重试次数,默认 3
|
||||
RetryDelayMs int `yaml:"retry_delay_ms,omitempty" json:"retry_delay_ms,omitempty"` // 重试间隔(毫秒),默认 1000
|
||||
|
||||
// 批处理配置(用于批量嵌入,当前未使用,保留扩展)
|
||||
BatchSize int `yaml:"batch_size,omitempty" json:"batch_size,omitempty"` // 批量处理大小,0 表示逐个处理
|
||||
}
|
||||
|
||||
// EmbeddingConfig 嵌入配置
|
||||
|
||||
+169
-8
@@ -2,10 +2,14 @@ package handler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -108,11 +112,132 @@ func (h *AgentHandler) SetSkillsManager(manager *skills.Manager) {
|
||||
h.skillsManager = manager
|
||||
}
|
||||
|
||||
// ChatAttachment 聊天附件(用户上传的文件)
|
||||
type ChatAttachment struct {
|
||||
FileName string `json:"fileName"` // 文件名
|
||||
Content string `json:"content"` // 文本内容或 base64(由 MimeType 决定是否解码)
|
||||
MimeType string `json:"mimeType,omitempty"`
|
||||
}
|
||||
|
||||
// ChatRequest 聊天请求
|
||||
type ChatRequest struct {
|
||||
Message string `json:"message" binding:"required"`
|
||||
ConversationID string `json:"conversationId,omitempty"`
|
||||
Role string `json:"role,omitempty"` // 角色名称
|
||||
Message string `json:"message" binding:"required"`
|
||||
ConversationID string `json:"conversationId,omitempty"`
|
||||
Role string `json:"role,omitempty"` // 角色名称
|
||||
Attachments []ChatAttachment `json:"attachments,omitempty"`
|
||||
}
|
||||
|
||||
const (
|
||||
maxAttachments = 10
|
||||
chatUploadsDirName = "chat_uploads" // 对话附件保存的根目录(相对当前工作目录)
|
||||
)
|
||||
|
||||
// saveAttachmentsToDateAndConversationDir 将附件保存到 chat_uploads/YYYY-MM-DD/{conversationID}/,返回每个文件的保存路径(与 attachments 顺序一致)
|
||||
// conversationID 为空时使用 "_new" 作为目录名(新对话尚未有 ID)
|
||||
func saveAttachmentsToDateAndConversationDir(attachments []ChatAttachment, conversationID string, logger *zap.Logger) (savedPaths []string, err error) {
|
||||
if len(attachments) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
cwd, err := os.Getwd()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("获取当前工作目录失败: %w", err)
|
||||
}
|
||||
dateDir := filepath.Join(cwd, chatUploadsDirName, time.Now().Format("2006-01-02"))
|
||||
convDirName := strings.TrimSpace(conversationID)
|
||||
if convDirName == "" {
|
||||
convDirName = "_new"
|
||||
} else {
|
||||
convDirName = strings.ReplaceAll(convDirName, string(filepath.Separator), "_")
|
||||
}
|
||||
targetDir := filepath.Join(dateDir, convDirName)
|
||||
if err = os.MkdirAll(targetDir, 0755); err != nil {
|
||||
return nil, fmt.Errorf("创建上传目录失败: %w", err)
|
||||
}
|
||||
savedPaths = make([]string, 0, len(attachments))
|
||||
for i, a := range attachments {
|
||||
raw, decErr := attachmentContentToBytes(a)
|
||||
if decErr != nil {
|
||||
return nil, fmt.Errorf("附件 %s 解码失败: %w", a.FileName, decErr)
|
||||
}
|
||||
baseName := filepath.Base(a.FileName)
|
||||
if baseName == "" || baseName == "." {
|
||||
baseName = "file"
|
||||
}
|
||||
baseName = strings.ReplaceAll(baseName, string(filepath.Separator), "_")
|
||||
ext := filepath.Ext(baseName)
|
||||
nameNoExt := strings.TrimSuffix(baseName, ext)
|
||||
suffix := fmt.Sprintf("_%s_%s", time.Now().Format("150405"), shortRand(6))
|
||||
var unique string
|
||||
if ext != "" {
|
||||
unique = nameNoExt + suffix + ext
|
||||
} else {
|
||||
unique = baseName + suffix
|
||||
}
|
||||
fullPath := filepath.Join(targetDir, unique)
|
||||
if err = os.WriteFile(fullPath, raw, 0644); err != nil {
|
||||
return nil, fmt.Errorf("写入文件 %s 失败: %w", a.FileName, err)
|
||||
}
|
||||
absPath, _ := filepath.Abs(fullPath)
|
||||
savedPaths = append(savedPaths, absPath)
|
||||
if logger != nil {
|
||||
logger.Debug("对话附件已保存", zap.Int("index", i+1), zap.String("fileName", a.FileName), zap.String("path", absPath))
|
||||
}
|
||||
}
|
||||
return savedPaths, nil
|
||||
}
|
||||
|
||||
func shortRand(n int) string {
|
||||
const letters = "0123456789abcdef"
|
||||
b := make([]byte, n)
|
||||
_, _ = rand.Read(b)
|
||||
for i := range b {
|
||||
b[i] = letters[int(b[i])%len(letters)]
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func attachmentContentToBytes(a ChatAttachment) ([]byte, error) {
|
||||
content := a.Content
|
||||
if decoded, err := base64.StdEncoding.DecodeString(content); err == nil && len(decoded) > 0 {
|
||||
return decoded, nil
|
||||
}
|
||||
return []byte(content), nil
|
||||
}
|
||||
|
||||
// userMessageContentForStorage 返回要存入数据库的用户消息内容:有附件时在正文后追加附件名(及路径),刷新后仍能显示,继续对话时大模型也能从历史中拿到路径
|
||||
func userMessageContentForStorage(message string, attachments []ChatAttachment, savedPaths []string) string {
|
||||
if len(attachments) == 0 {
|
||||
return message
|
||||
}
|
||||
var b strings.Builder
|
||||
b.WriteString(message)
|
||||
for i, a := range attachments {
|
||||
b.WriteString("\n📎 ")
|
||||
b.WriteString(a.FileName)
|
||||
if i < len(savedPaths) && savedPaths[i] != "" {
|
||||
b.WriteString(": ")
|
||||
b.WriteString(savedPaths[i])
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// appendAttachmentsToMessage 仅将附件的保存路径追加到用户消息末尾,不再内联附件内容,避免上下文过长
|
||||
func appendAttachmentsToMessage(msg string, attachments []ChatAttachment, savedPaths []string) string {
|
||||
if len(attachments) == 0 {
|
||||
return msg
|
||||
}
|
||||
var b strings.Builder
|
||||
b.WriteString(msg)
|
||||
b.WriteString("\n\n[用户上传的文件已保存到以下路径(请按需读取文件内容,而不是依赖内联内容)]\n")
|
||||
for i, a := range attachments {
|
||||
if i < len(savedPaths) && savedPaths[i] != "" {
|
||||
b.WriteString(fmt.Sprintf("- %s: %s\n", a.FileName, savedPaths[i]))
|
||||
} else {
|
||||
b.WriteString(fmt.Sprintf("- %s: (路径未知,可能保存失败)\n", a.FileName))
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// ChatResponse 聊天响应
|
||||
@@ -181,6 +306,12 @@ func (h *AgentHandler) AgentLoop(c *gin.Context) {
|
||||
h.logger.Info("从ReAct数据恢复历史上下文", zap.Int("count", len(agentHistoryMessages)))
|
||||
}
|
||||
|
||||
// 校验附件数量(非流式)
|
||||
if len(req.Attachments) > maxAttachments {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("附件最多 %d 个", maxAttachments)})
|
||||
return
|
||||
}
|
||||
|
||||
// 应用角色用户提示词和工具配置
|
||||
finalMessage := req.Message
|
||||
var roleTools []string // 角色配置的工具列表
|
||||
@@ -206,9 +337,20 @@ func (h *AgentHandler) AgentLoop(c *gin.Context) {
|
||||
}
|
||||
}
|
||||
}
|
||||
var savedPaths []string
|
||||
if len(req.Attachments) > 0 {
|
||||
savedPaths, err = saveAttachmentsToDateAndConversationDir(req.Attachments, conversationID, h.logger)
|
||||
if err != nil {
|
||||
h.logger.Error("保存对话附件失败", zap.Error(err))
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": "保存上传文件失败: " + err.Error()})
|
||||
return
|
||||
}
|
||||
}
|
||||
finalMessage = appendAttachmentsToMessage(finalMessage, req.Attachments, savedPaths)
|
||||
|
||||
// 保存用户消息(保存原始消息,不包含角色提示词)
|
||||
_, err = h.db.AddMessage(conversationID, "user", req.Message, nil)
|
||||
// 保存用户消息:有附件时一并保存附件名与路径,刷新后显示、继续对话时大模型也能从历史中拿到路径
|
||||
userContent := userMessageContentForStorage(req.Message, req.Attachments, savedPaths)
|
||||
_, err = h.db.AddMessage(conversationID, "user", userContent, nil)
|
||||
if err != nil {
|
||||
h.logger.Error("保存用户消息失败", zap.Error(err))
|
||||
c.JSON(http.StatusInternalServerError, gin.H{"error": "保存用户消息失败: " + err.Error()})
|
||||
@@ -618,6 +760,12 @@ func (h *AgentHandler) AgentLoopStream(c *gin.Context) {
|
||||
h.logger.Info("从ReAct数据恢复历史上下文", zap.Int("count", len(agentHistoryMessages)))
|
||||
}
|
||||
|
||||
// 校验附件数量
|
||||
if len(req.Attachments) > maxAttachments {
|
||||
sendEvent("error", fmt.Sprintf("附件最多 %d 个", maxAttachments), nil)
|
||||
return
|
||||
}
|
||||
|
||||
// 应用角色用户提示词和工具配置
|
||||
finalMessage := req.Message
|
||||
var roleTools []string // 角色配置的工具列表
|
||||
@@ -645,10 +793,22 @@ func (h *AgentHandler) AgentLoopStream(c *gin.Context) {
|
||||
}
|
||||
}
|
||||
}
|
||||
var savedPaths []string
|
||||
if len(req.Attachments) > 0 {
|
||||
savedPaths, err = saveAttachmentsToDateAndConversationDir(req.Attachments, conversationID, h.logger)
|
||||
if err != nil {
|
||||
h.logger.Error("保存对话附件失败", zap.Error(err))
|
||||
sendEvent("error", "保存上传文件失败: "+err.Error(), nil)
|
||||
return
|
||||
}
|
||||
}
|
||||
// 仅将附件保存路径追加到 finalMessage,避免将文件内容内联到大模型上下文中
|
||||
finalMessage = appendAttachmentsToMessage(finalMessage, req.Attachments, savedPaths)
|
||||
// 如果roleTools为空,表示使用所有工具(默认角色或未配置工具的角色)
|
||||
|
||||
// 保存用户消息(保存原始消息,不包含角色提示词)
|
||||
_, err = h.db.AddMessage(conversationID, "user", req.Message, nil)
|
||||
// 保存用户消息:有附件时一并保存附件名与路径,刷新后显示、继续对话时大模型也能从历史中拿到路径
|
||||
userContent := userMessageContentForStorage(req.Message, req.Attachments, savedPaths)
|
||||
_, err = h.db.AddMessage(conversationID, "user", userContent, nil)
|
||||
if err != nil {
|
||||
h.logger.Error("保存用户消息失败", zap.Error(err))
|
||||
}
|
||||
@@ -1284,7 +1444,8 @@ func (h *AgentHandler) executeBatchQueue(queueID string) {
|
||||
// 执行任务(使用包含角色提示词的finalMessage和角色工具列表)
|
||||
h.logger.Info("执行批量任务", zap.String("queueId", queueID), zap.String("taskId", task.ID), zap.String("message", task.Message), zap.String("role", queue.Role), zap.String("conversationId", conversationID))
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
// 单个子任务超时时间:从30分钟调整为6小时,适配长时间渗透/扫描任务
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Hour)
|
||||
// 存储取消函数,以便在取消队列时能够取消当前任务
|
||||
h.batchTaskManager.SetTaskCancel(queueID, cancel)
|
||||
// 使用队列配置的角色工具列表(如果为空,表示使用所有工具)
|
||||
|
||||
@@ -1062,6 +1062,16 @@ func updateKnowledgeConfig(doc *yaml.Node, cfg config.KnowledgeConfig) {
|
||||
setIntInMap(retrievalNode, "top_k", cfg.Retrieval.TopK)
|
||||
setFloatInMap(retrievalNode, "similarity_threshold", cfg.Retrieval.SimilarityThreshold)
|
||||
setFloatInMap(retrievalNode, "hybrid_weight", cfg.Retrieval.HybridWeight)
|
||||
|
||||
// 更新索引配置
|
||||
indexingNode := ensureMap(knowledgeNode, "indexing")
|
||||
setIntInMap(indexingNode, "chunk_size", cfg.Indexing.ChunkSize)
|
||||
setIntInMap(indexingNode, "chunk_overlap", cfg.Indexing.ChunkOverlap)
|
||||
setIntInMap(indexingNode, "max_chunks_per_item", cfg.Indexing.MaxChunksPerItem)
|
||||
setIntInMap(indexingNode, "max_rpm", cfg.Indexing.MaxRPM)
|
||||
setIntInMap(indexingNode, "rate_limit_delay_ms", cfg.Indexing.RateLimitDelayMs)
|
||||
setIntInMap(indexingNode, "max_retries", cfg.Indexing.MaxRetries)
|
||||
setIntInMap(indexingNode, "retry_delay_ms", cfg.Indexing.RetryDelayMs)
|
||||
}
|
||||
|
||||
func updateRobotsConfig(doc *yaml.Node, cfg config.RobotsConfig) {
|
||||
|
||||
@@ -75,7 +75,7 @@ func (h *KnowledgeHandler) GetItems(c *gin.Context) {
|
||||
groupedByCategory[cat] = append(groupedByCategory[cat], item)
|
||||
}
|
||||
|
||||
// 转换为CategoryWithItems格式
|
||||
// 转换为 CategoryWithItems 格式
|
||||
categoriesWithItems := make([]*knowledge.CategoryWithItems, 0, len(groupedByCategory))
|
||||
for cat, catItems := range groupedByCategory {
|
||||
categoriesWithItems = append(categoriesWithItems, &knowledge.CategoryWithItems{
|
||||
@@ -107,7 +107,7 @@ func (h *KnowledgeHandler) GetItems(c *gin.Context) {
|
||||
categoryPageMode := c.Query("categoryPage") != "false" // 默认使用分类分页
|
||||
|
||||
// 分页参数
|
||||
limit := 50 // 默认每页50条(分类分页时为分类数,项分页时为项数)
|
||||
limit := 50 // 默认每页 50 条(分类分页时为分类数,项分页时为项数)
|
||||
offset := 0
|
||||
if limitStr := c.Query("limit"); limitStr != "" {
|
||||
if parsed, err := parseInt(limitStr); err == nil && parsed > 0 && parsed <= 500 {
|
||||
@@ -120,7 +120,7 @@ func (h *KnowledgeHandler) GetItems(c *gin.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// 如果指定了category参数,且使用分类分页模式,则只返回该分类
|
||||
// 如果指定了 category 参数,且使用分类分页模式,则只返回该分类
|
||||
if category != "" && categoryPageMode {
|
||||
// 单分类模式:返回该分类的所有知识项(不分页)
|
||||
items, total, err := h.manager.GetItemsSummary(category, 0, 0)
|
||||
@@ -150,9 +150,9 @@ func (h *KnowledgeHandler) GetItems(c *gin.Context) {
|
||||
|
||||
if categoryPageMode {
|
||||
// 按分类分页模式(默认)
|
||||
// limit表示每页分类数,推荐5-10个分类
|
||||
// limit 表示每页分类数,推荐 5-10 个分类
|
||||
if limit <= 0 || limit > 100 {
|
||||
limit = 10 // 默认每页10个分类
|
||||
limit = 10 // 默认每页 10 个分类
|
||||
}
|
||||
|
||||
categoriesWithItems, totalCategories, err := h.manager.GetCategoriesWithItems(limit, offset)
|
||||
@@ -172,7 +172,7 @@ func (h *KnowledgeHandler) GetItems(c *gin.Context) {
|
||||
}
|
||||
|
||||
// 按项分页模式(向后兼容)
|
||||
// 是否包含完整内容(默认false,只返回摘要)
|
||||
// 是否包含完整内容(默认 false,只返回摘要)
|
||||
includeContent := c.Query("includeContent") == "true"
|
||||
|
||||
if includeContent {
|
||||
@@ -358,7 +358,7 @@ func (h *KnowledgeHandler) ScanKnowledgeBase(c *gin.Context) {
|
||||
)
|
||||
}
|
||||
|
||||
// 如果连续失败2次,立即停止增量索引
|
||||
// 如果连续失败 2 次,立即停止增量索引
|
||||
if consecutiveFailures >= 2 {
|
||||
h.logger.Error("连续索引失败次数过多,立即停止增量索引",
|
||||
zap.Int("consecutiveFailures", consecutiveFailures),
|
||||
@@ -397,7 +397,7 @@ func (h *KnowledgeHandler) ScanKnowledgeBase(c *gin.Context) {
|
||||
func (h *KnowledgeHandler) GetRetrievalLogs(c *gin.Context) {
|
||||
conversationID := c.Query("conversationId")
|
||||
messageID := c.Query("messageId")
|
||||
limit := 50 // 默认50条
|
||||
limit := 50 // 默认 50 条
|
||||
|
||||
if limitStr := c.Query("limit"); limitStr != "" {
|
||||
if parsed, err := parseInt(limitStr); err == nil && parsed > 0 {
|
||||
@@ -441,18 +441,40 @@ func (h *KnowledgeHandler) GetIndexStatus(c *gin.Context) {
|
||||
if h.indexer != nil {
|
||||
lastError, lastErrorTime := h.indexer.GetLastError()
|
||||
if lastError != "" {
|
||||
// 如果错误是最近发生的(5分钟内),则返回错误信息
|
||||
// 如果错误是最近发生的(5 分钟内),则返回错误信息
|
||||
if time.Since(lastErrorTime) < 5*time.Minute {
|
||||
status["last_error"] = lastError
|
||||
status["last_error_time"] = lastErrorTime.Format(time.RFC3339)
|
||||
}
|
||||
}
|
||||
|
||||
// 获取重建索引状态
|
||||
isRebuilding, totalItems, current, failed, lastItemID, lastChunks, startTime := h.indexer.GetRebuildStatus()
|
||||
if isRebuilding {
|
||||
status["is_rebuilding"] = true
|
||||
status["rebuild_total"] = totalItems
|
||||
status["rebuild_current"] = current
|
||||
status["rebuild_failed"] = failed
|
||||
status["rebuild_start_time"] = startTime.Format(time.RFC3339)
|
||||
if lastItemID != "" {
|
||||
status["rebuild_last_item_id"] = lastItemID
|
||||
}
|
||||
if lastChunks > 0 {
|
||||
status["rebuild_last_chunks"] = lastChunks
|
||||
}
|
||||
// 重建中时,is_complete 为 false
|
||||
status["is_complete"] = false
|
||||
// 计算重建进度百分比
|
||||
if totalItems > 0 {
|
||||
status["progress_percent"] = float64(current) / float64(totalItems) * 100
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, status)
|
||||
}
|
||||
|
||||
// Search 搜索知识库(用于API调用,Agent内部使用Retriever)
|
||||
// Search 搜索知识库(用于 API 调用,Agent 内部使用 Retriever)
|
||||
func (h *KnowledgeHandler) Search(c *gin.Context) {
|
||||
var req knowledge.SearchRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
package handler
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
const (
|
||||
terminalMaxCommandLen = 4096
|
||||
terminalMaxOutputLen = 256 * 1024 // 256KB
|
||||
terminalTimeout = 120 * time.Second
|
||||
)
|
||||
|
||||
// TerminalHandler 处理系统设置中的终端命令执行
|
||||
type TerminalHandler struct {
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
// maskTerminalCommand 对可能包含敏感信息的终端命令做脱敏,避免在日志中直接记录密码等内容
|
||||
func maskTerminalCommand(cmd string) string {
|
||||
trimmed := strings.TrimSpace(cmd)
|
||||
lower := strings.ToLower(trimmed)
|
||||
if strings.Contains(lower, "sudo") || strings.Contains(lower, "password") {
|
||||
return "[masked sensitive terminal command]"
|
||||
}
|
||||
if len(trimmed) > 256 {
|
||||
return trimmed[:256] + "..."
|
||||
}
|
||||
return trimmed
|
||||
}
|
||||
|
||||
// NewTerminalHandler 创建终端处理器
|
||||
func NewTerminalHandler(logger *zap.Logger) *TerminalHandler {
|
||||
return &TerminalHandler{logger: logger}
|
||||
}
|
||||
|
||||
// RunCommandRequest 执行命令请求
|
||||
type RunCommandRequest struct {
|
||||
Command string `json:"command"`
|
||||
Shell string `json:"shell,omitempty"`
|
||||
Cwd string `json:"cwd,omitempty"`
|
||||
}
|
||||
|
||||
// RunCommandResponse 执行命令响应
|
||||
type RunCommandResponse struct {
|
||||
Stdout string `json:"stdout"`
|
||||
Stderr string `json:"stderr"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// RunCommand 执行终端命令(需登录)
|
||||
func (h *TerminalHandler) RunCommand(c *gin.Context) {
|
||||
var req RunCommandRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "请求体无效,需要 command 字段"})
|
||||
return
|
||||
}
|
||||
|
||||
cmdStr := strings.TrimSpace(req.Command)
|
||||
if cmdStr == "" {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "command 不能为空"})
|
||||
return
|
||||
}
|
||||
if len(cmdStr) > terminalMaxCommandLen {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "命令过长"})
|
||||
return
|
||||
}
|
||||
|
||||
shell := req.Shell
|
||||
if shell == "" {
|
||||
if runtime.GOOS == "windows" {
|
||||
shell = "cmd"
|
||||
} else {
|
||||
shell = "sh"
|
||||
}
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(c.Request.Context(), terminalTimeout)
|
||||
defer cancel()
|
||||
|
||||
var cmd *exec.Cmd
|
||||
if runtime.GOOS == "windows" {
|
||||
cmd = exec.CommandContext(ctx, "cmd", "/c", cmdStr)
|
||||
} else {
|
||||
cmd = exec.CommandContext(ctx, shell, "-c", cmdStr)
|
||||
// 无 TTY 时设置 COLUMNS/TERM,使 ping 等工具的 usage 排版与真实终端一致
|
||||
cmd.Env = append(os.Environ(), "COLUMNS=256", "LINES=40", "TERM=xterm-256color")
|
||||
}
|
||||
|
||||
if req.Cwd != "" {
|
||||
absCwd, err := filepath.Abs(req.Cwd)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "工作目录无效"})
|
||||
return
|
||||
}
|
||||
cur, _ := os.Getwd()
|
||||
curAbs, _ := filepath.Abs(cur)
|
||||
rel, err := filepath.Rel(curAbs, absCwd)
|
||||
if err != nil || strings.HasPrefix(rel, "..") || rel == ".." {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "工作目录必须在当前进程目录下"})
|
||||
return
|
||||
}
|
||||
cmd.Dir = absCwd
|
||||
}
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout = &stdout
|
||||
cmd.Stderr = &stderr
|
||||
|
||||
err := cmd.Run()
|
||||
stdoutBytes := stdout.Bytes()
|
||||
stderrBytes := stderr.Bytes()
|
||||
|
||||
// 限制输出长度,防止内存占用过大(复制后截断,避免修改原 buffer)
|
||||
truncSuffix := []byte("\n...(输出已截断)\n")
|
||||
if len(stdoutBytes) > terminalMaxOutputLen {
|
||||
tmp := make([]byte, terminalMaxOutputLen+len(truncSuffix))
|
||||
n := copy(tmp, stdoutBytes[:terminalMaxOutputLen])
|
||||
copy(tmp[n:], truncSuffix)
|
||||
stdoutBytes = tmp
|
||||
}
|
||||
if len(stderrBytes) > terminalMaxOutputLen {
|
||||
tmp := make([]byte, terminalMaxOutputLen+len(truncSuffix))
|
||||
n := copy(tmp, stderrBytes[:terminalMaxOutputLen])
|
||||
copy(tmp[n:], truncSuffix)
|
||||
stderrBytes = tmp
|
||||
}
|
||||
|
||||
exitCode := 0
|
||||
if err != nil {
|
||||
if exitErr, ok := err.(*exec.ExitError); ok {
|
||||
exitCode = exitErr.ExitCode()
|
||||
} else {
|
||||
exitCode = -1
|
||||
}
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
so := strings.ReplaceAll(string(stdoutBytes), "\r\n", "\n")
|
||||
so = strings.ReplaceAll(so, "\r", "\n")
|
||||
se := strings.ReplaceAll(string(stderrBytes), "\r\n", "\n")
|
||||
se = strings.ReplaceAll(se, "\r", "\n")
|
||||
resp := RunCommandResponse{
|
||||
Stdout: so,
|
||||
Stderr: se,
|
||||
ExitCode: -1,
|
||||
Error: "命令执行超时(" + terminalTimeout.String() + ")",
|
||||
}
|
||||
c.JSON(http.StatusOK, resp)
|
||||
return
|
||||
}
|
||||
h.logger.Debug("终端命令执行异常", zap.String("command", maskTerminalCommand(cmdStr)), zap.Error(err))
|
||||
}
|
||||
|
||||
// 统一为 \n,避免前端因 \r 出现错位/对角线排版
|
||||
stdoutStr := strings.ReplaceAll(string(stdoutBytes), "\r\n", "\n")
|
||||
stdoutStr = strings.ReplaceAll(stdoutStr, "\r", "\n")
|
||||
stderrStr := strings.ReplaceAll(string(stderrBytes), "\r\n", "\n")
|
||||
stderrStr = strings.ReplaceAll(stderrStr, "\r", "\n")
|
||||
|
||||
resp := RunCommandResponse{
|
||||
Stdout: stdoutStr,
|
||||
Stderr: stderrStr,
|
||||
ExitCode: exitCode,
|
||||
}
|
||||
if err != nil && exitCode != 0 {
|
||||
resp.Error = err.Error()
|
||||
}
|
||||
c.JSON(http.StatusOK, resp)
|
||||
}
|
||||
|
||||
// streamEvent SSE 事件
|
||||
type streamEvent struct {
|
||||
T string `json:"t"` // "out" | "err" | "exit"
|
||||
D string `json:"d,omitempty"`
|
||||
C int `json:"c"` // exit code(不用 omitempty,否则 0 不序列化导致前端显示 [exit undefined])
|
||||
}
|
||||
|
||||
// RunCommandStream 流式执行命令,输出实时推送到前端(SSE)
|
||||
func (h *TerminalHandler) RunCommandStream(c *gin.Context) {
|
||||
var req RunCommandRequest
|
||||
if err := c.ShouldBindJSON(&req); err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "请求体无效,需要 command 字段"})
|
||||
return
|
||||
}
|
||||
cmdStr := strings.TrimSpace(req.Command)
|
||||
if cmdStr == "" {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "command 不能为空"})
|
||||
return
|
||||
}
|
||||
if len(cmdStr) > terminalMaxCommandLen {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "命令过长"})
|
||||
return
|
||||
}
|
||||
shell := req.Shell
|
||||
if shell == "" {
|
||||
if runtime.GOOS == "windows" {
|
||||
shell = "cmd"
|
||||
} else {
|
||||
shell = "sh"
|
||||
}
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(c.Request.Context(), terminalTimeout)
|
||||
defer cancel()
|
||||
|
||||
var cmd *exec.Cmd
|
||||
if runtime.GOOS == "windows" {
|
||||
cmd = exec.CommandContext(ctx, "cmd", "/c", cmdStr)
|
||||
} else {
|
||||
cmd = exec.CommandContext(ctx, shell, "-c", cmdStr)
|
||||
cmd.Env = append(os.Environ(), "COLUMNS=256", "LINES=40", "TERM=xterm-256color")
|
||||
}
|
||||
if req.Cwd != "" {
|
||||
absCwd, err := filepath.Abs(req.Cwd)
|
||||
if err != nil {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "工作目录无效"})
|
||||
return
|
||||
}
|
||||
cur, _ := os.Getwd()
|
||||
curAbs, _ := filepath.Abs(cur)
|
||||
rel, err := filepath.Rel(curAbs, absCwd)
|
||||
if err != nil || strings.HasPrefix(rel, "..") || rel == ".." {
|
||||
c.JSON(http.StatusBadRequest, gin.H{"error": "工作目录必须在当前进程目录下"})
|
||||
return
|
||||
}
|
||||
cmd.Dir = absCwd
|
||||
}
|
||||
|
||||
c.Header("Content-Type", "text/event-stream")
|
||||
c.Header("Cache-Control", "no-cache")
|
||||
c.Header("Connection", "keep-alive")
|
||||
c.Header("X-Accel-Buffering", "no")
|
||||
c.Writer.WriteHeader(http.StatusOK)
|
||||
flusher, ok := c.Writer.(http.Flusher)
|
||||
if !ok {
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
|
||||
sendEvent := func(ev streamEvent) {
|
||||
body, _ := json.Marshal(ev)
|
||||
c.SSEvent("", string(body))
|
||||
flusher.Flush()
|
||||
}
|
||||
|
||||
runCommandStreamImpl(cmd, sendEvent, ctx)
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
//go:build !windows
|
||||
|
||||
package handler
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os/exec"
|
||||
"strings"
|
||||
|
||||
"github.com/creack/pty"
|
||||
)
|
||||
|
||||
const ptyCols = 256
|
||||
const ptyRows = 40
|
||||
|
||||
// runCommandStreamImpl 在 Unix 下用 PTY 执行,使 ping 等命令按终端宽度排版(isatty 为真)
|
||||
func runCommandStreamImpl(cmd *exec.Cmd, sendEvent func(streamEvent), ctx context.Context) {
|
||||
ptmx, err := pty.StartWithSize(cmd, &pty.Winsize{Cols: ptyCols, Rows: ptyRows})
|
||||
if err != nil {
|
||||
sendEvent(streamEvent{T: "exit", C: -1})
|
||||
return
|
||||
}
|
||||
defer ptmx.Close()
|
||||
|
||||
normalize := func(s string) string {
|
||||
s = strings.ReplaceAll(s, "\r\n", "\n")
|
||||
return strings.ReplaceAll(s, "\r", "\n")
|
||||
}
|
||||
sc := bufio.NewScanner(ptmx)
|
||||
for sc.Scan() {
|
||||
sendEvent(streamEvent{T: "out", D: normalize(sc.Text())})
|
||||
}
|
||||
exitCode := 0
|
||||
if err := cmd.Wait(); err != nil {
|
||||
if exitErr, ok := err.(*exec.ExitError); ok {
|
||||
exitCode = exitErr.ExitCode()
|
||||
} else {
|
||||
exitCode = -1
|
||||
}
|
||||
}
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
exitCode = -1
|
||||
}
|
||||
sendEvent(streamEvent{T: "exit", C: exitCode})
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
//go:build windows
|
||||
|
||||
package handler
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// runCommandStreamImpl 在 Windows 下用 stdout/stderr 管道执行
|
||||
func runCommandStreamImpl(cmd *exec.Cmd, sendEvent func(streamEvent), ctx context.Context) {
|
||||
stdoutPipe, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
sendEvent(streamEvent{T: "exit", C: -1})
|
||||
return
|
||||
}
|
||||
stderrPipe, err := cmd.StderrPipe()
|
||||
if err != nil {
|
||||
sendEvent(streamEvent{T: "exit", C: -1})
|
||||
return
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
sendEvent(streamEvent{T: "exit", C: -1})
|
||||
return
|
||||
}
|
||||
|
||||
normalize := func(s string) string {
|
||||
s = strings.ReplaceAll(s, "\r\n", "\n")
|
||||
return strings.ReplaceAll(s, "\r", "\n")
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(2)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
sc := bufio.NewScanner(stdoutPipe)
|
||||
for sc.Scan() {
|
||||
sendEvent(streamEvent{T: "out", D: normalize(sc.Text())})
|
||||
}
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
sc := bufio.NewScanner(stderrPipe)
|
||||
for sc.Scan() {
|
||||
sendEvent(streamEvent{T: "err", D: normalize(sc.Text())})
|
||||
}
|
||||
}()
|
||||
|
||||
wg.Wait()
|
||||
exitCode := 0
|
||||
if err := cmd.Wait(); err != nil {
|
||||
if exitErr, ok := err.(*exec.ExitError); ok {
|
||||
exitCode = exitErr.ExitCode()
|
||||
} else {
|
||||
exitCode = -1
|
||||
}
|
||||
}
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
exitCode = -1
|
||||
}
|
||||
sendEvent(streamEvent{T: "exit", C: exitCode})
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
//go:build !windows
|
||||
|
||||
package handler
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"time"
|
||||
|
||||
"github.com/creack/pty"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/gorilla/websocket"
|
||||
)
|
||||
|
||||
// wsUpgrader 仅用于系统设置中的终端 WebSocket,会复用已有的登录保护(JWT 中间件在上层路由组)
|
||||
var wsUpgrader = websocket.Upgrader{
|
||||
CheckOrigin: func(r *http.Request) bool {
|
||||
// 由于已在 Gin 路由层做了认证,这里放宽 Origin,方便在同一域名下通过 HTTPS/WSS 访问
|
||||
return true
|
||||
},
|
||||
}
|
||||
|
||||
// RunCommandWS 提供真正交互式 Shell:基于 WebSocket + PTY 的长会话
|
||||
// 前端建立 WebSocket 连接后,所有键盘输入都会透传到 Shell,Shell 的输出也会实时写回前端。
|
||||
func (h *TerminalHandler) RunCommandWS(c *gin.Context) {
|
||||
conn, err := wsUpgrader.Upgrade(c.Writer, c.Request, nil)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
// 启动交互式 Shell,这里优先使用 bash,找不到则退回 sh
|
||||
shell := "bash"
|
||||
if _, err := exec.LookPath(shell); err != nil {
|
||||
shell = "sh"
|
||||
}
|
||||
cmd := exec.Command(shell)
|
||||
cmd.Env = append(os.Environ(),
|
||||
"COLUMNS=256",
|
||||
"LINES=40",
|
||||
"TERM=xterm-256color",
|
||||
)
|
||||
|
||||
ptmx, err := pty.StartWithSize(cmd, &pty.Winsize{Cols: ptyCols, Rows: ptyRows})
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer ptmx.Close()
|
||||
|
||||
// Shell -> WebSocket:将 PTY 输出实时发给前端
|
||||
doneChan := make(chan struct{})
|
||||
go func() {
|
||||
buf := make([]byte, 4096)
|
||||
for {
|
||||
n, err := ptmx.Read(buf)
|
||||
if n > 0 {
|
||||
_ = conn.WriteMessage(websocket.BinaryMessage, buf[:n])
|
||||
}
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
close(doneChan)
|
||||
}()
|
||||
|
||||
// WebSocket -> Shell:将前端输入写入 PTY(包括 sudo 密码、Ctrl+C 等)
|
||||
conn.SetReadLimit(64 * 1024)
|
||||
_ = conn.SetReadDeadline(time.Now().Add(terminalTimeout))
|
||||
conn.SetPongHandler(func(string) error {
|
||||
_ = conn.SetReadDeadline(time.Now().Add(terminalTimeout))
|
||||
return nil
|
||||
})
|
||||
|
||||
for {
|
||||
msgType, data, err := conn.ReadMessage()
|
||||
if err != nil {
|
||||
_ = cmd.Process.Kill()
|
||||
break
|
||||
}
|
||||
if msgType != websocket.TextMessage && msgType != websocket.BinaryMessage {
|
||||
continue
|
||||
}
|
||||
if len(data) == 0 {
|
||||
continue
|
||||
}
|
||||
if _, err := ptmx.Write(data); err != nil {
|
||||
_ = cmd.Process.Kill()
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
<-doneChan
|
||||
}
|
||||
|
||||
+149
-31
@@ -6,39 +6,75 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"cyberstrike-ai/internal/config"
|
||||
"cyberstrike-ai/internal/openai"
|
||||
|
||||
"go.uber.org/zap"
|
||||
"golang.org/x/time/rate"
|
||||
)
|
||||
|
||||
// Embedder 文本嵌入器
|
||||
type Embedder struct {
|
||||
openAIClient *openai.Client
|
||||
config *config.KnowledgeConfig
|
||||
openAIConfig *config.OpenAIConfig // 用于获取API Key
|
||||
logger *zap.Logger
|
||||
openAIClient *openai.Client
|
||||
config *config.KnowledgeConfig
|
||||
openAIConfig *config.OpenAIConfig // 用于获取 API Key
|
||||
logger *zap.Logger
|
||||
rateLimiter *rate.Limiter // 速率限制器
|
||||
rateLimitDelay time.Duration // 请求间隔时间
|
||||
maxRetries int // 最大重试次数
|
||||
retryDelay time.Duration // 重试间隔
|
||||
mu sync.Mutex // 保护 rateLimiter
|
||||
}
|
||||
|
||||
// NewEmbedder 创建新的嵌入器
|
||||
func NewEmbedder(cfg *config.KnowledgeConfig, openAIConfig *config.OpenAIConfig, openAIClient *openai.Client, logger *zap.Logger) *Embedder {
|
||||
// 初始化速率限制器
|
||||
var rateLimiter *rate.Limiter
|
||||
var rateLimitDelay time.Duration
|
||||
|
||||
// 如果配置了 MaxRPM,根据 RPM 计算速率限制
|
||||
if cfg.Indexing.MaxRPM > 0 {
|
||||
rpm := cfg.Indexing.MaxRPM
|
||||
rateLimiter = rate.NewLimiter(rate.Every(time.Minute/time.Duration(rpm)), rpm)
|
||||
logger.Info("知识库索引速率限制已启用", zap.Int("maxRPM", rpm))
|
||||
} else if cfg.Indexing.RateLimitDelayMs > 0 {
|
||||
// 如果没有配置 MaxRPM 但配置了固定延迟,使用固定延迟模式
|
||||
rateLimitDelay = time.Duration(cfg.Indexing.RateLimitDelayMs) * time.Millisecond
|
||||
logger.Info("知识库索引固定延迟已启用", zap.Duration("delay", rateLimitDelay))
|
||||
}
|
||||
|
||||
// 重试配置
|
||||
maxRetries := 3
|
||||
retryDelay := 1000 * time.Millisecond
|
||||
if cfg.Indexing.MaxRetries > 0 {
|
||||
maxRetries = cfg.Indexing.MaxRetries
|
||||
}
|
||||
if cfg.Indexing.RetryDelayMs > 0 {
|
||||
retryDelay = time.Duration(cfg.Indexing.RetryDelayMs) * time.Millisecond
|
||||
}
|
||||
|
||||
return &Embedder{
|
||||
openAIClient: openAIClient,
|
||||
config: cfg,
|
||||
openAIConfig: openAIConfig,
|
||||
logger: logger,
|
||||
openAIClient: openAIClient,
|
||||
config: cfg,
|
||||
openAIConfig: openAIConfig,
|
||||
logger: logger,
|
||||
rateLimiter: rateLimiter,
|
||||
rateLimitDelay: rateLimitDelay,
|
||||
maxRetries: maxRetries,
|
||||
retryDelay: retryDelay,
|
||||
}
|
||||
}
|
||||
|
||||
// EmbeddingRequest OpenAI嵌入请求
|
||||
// EmbeddingRequest OpenAI 嵌入请求
|
||||
type EmbeddingRequest struct {
|
||||
Model string `json:"model"`
|
||||
Input []string `json:"input"`
|
||||
}
|
||||
|
||||
// EmbeddingResponse OpenAI嵌入响应
|
||||
// EmbeddingResponse OpenAI 嵌入响应
|
||||
type EmbeddingResponse struct {
|
||||
Data []EmbeddingData `json:"data"`
|
||||
Error *EmbeddingError `json:"error,omitempty"`
|
||||
@@ -56,12 +92,69 @@ type EmbeddingError struct {
|
||||
Type string `json:"type"`
|
||||
}
|
||||
|
||||
// EmbedText 对文本进行嵌入
|
||||
func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error) {
|
||||
if e.openAIClient == nil {
|
||||
return nil, fmt.Errorf("OpenAI客户端未初始化")
|
||||
// waitRateLimiter 等待速率限制器
|
||||
func (e *Embedder) waitRateLimiter() {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
|
||||
if e.rateLimiter != nil {
|
||||
// 等待令牌
|
||||
ctx := context.Background()
|
||||
if err := e.rateLimiter.Wait(ctx); err != nil {
|
||||
e.logger.Warn("速率限制器等待失败", zap.Error(err))
|
||||
}
|
||||
}
|
||||
|
||||
if e.rateLimitDelay > 0 {
|
||||
time.Sleep(e.rateLimitDelay)
|
||||
}
|
||||
}
|
||||
|
||||
// EmbedText 对文本进行嵌入(带重试和速率限制)
|
||||
func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error) {
|
||||
if e.openAIClient == nil {
|
||||
return nil, fmt.Errorf("OpenAI 客户端未初始化")
|
||||
}
|
||||
|
||||
var lastErr error
|
||||
for attempt := 0; attempt < e.maxRetries; attempt++ {
|
||||
// 速率限制
|
||||
if attempt > 0 {
|
||||
// 重试时等待更长时间
|
||||
waitTime := e.retryDelay * time.Duration(attempt)
|
||||
e.logger.Debug("重试前等待", zap.Int("attempt", attempt+1), zap.Duration("waitTime", waitTime))
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(waitTime):
|
||||
}
|
||||
} else {
|
||||
e.waitRateLimiter()
|
||||
}
|
||||
|
||||
result, err := e.doEmbedText(ctx, text)
|
||||
if err == nil {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
lastErr = err
|
||||
|
||||
// 检查是否是可重试的错误(429 速率限制、5xx 服务器错误、网络错误)
|
||||
if !e.isRetryableError(err) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
e.logger.Debug("嵌入请求失败,准备重试",
|
||||
zap.Int("attempt", attempt+1),
|
||||
zap.Int("maxRetries", e.maxRetries),
|
||||
zap.Error(err))
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("达到最大重试次数 (%d): %v", e.maxRetries, lastErr)
|
||||
}
|
||||
|
||||
// doEmbedText 执行实际的嵌入请求(内部方法)
|
||||
func (e *Embedder) doEmbedText(ctx context.Context, text string) ([]float32, error) {
|
||||
// 使用配置的嵌入模型
|
||||
model := e.config.Embedding.Model
|
||||
if model == "" {
|
||||
@@ -73,7 +166,7 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
Input: []string{text},
|
||||
}
|
||||
|
||||
// 清理baseURL:去除前后空格和尾部斜杠
|
||||
// 清理 baseURL:去除前后空格和尾部斜杠
|
||||
baseURL := strings.TrimSpace(e.config.Embedding.BaseURL)
|
||||
baseURL = strings.TrimSuffix(baseURL, "/")
|
||||
if baseURL == "" {
|
||||
@@ -83,24 +176,24 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
// 构建请求
|
||||
body, err := json.Marshal(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("序列化请求失败: %w", err)
|
||||
return nil, fmt.Errorf("序列化请求失败:%w", err)
|
||||
}
|
||||
|
||||
requestURL := baseURL + "/embeddings"
|
||||
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, requestURL, strings.NewReader(string(body)))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("创建请求失败: %w", err)
|
||||
return nil, fmt.Errorf("创建请求失败:%w", err)
|
||||
}
|
||||
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
|
||||
// 使用配置的API Key,如果没有则使用OpenAI配置的
|
||||
|
||||
// 使用配置的 API Key,如果没有则使用 OpenAI 配置的
|
||||
apiKey := strings.TrimSpace(e.config.Embedding.APIKey)
|
||||
if apiKey == "" && e.openAIConfig != nil {
|
||||
apiKey = e.openAIConfig.APIKey
|
||||
}
|
||||
if apiKey == "" {
|
||||
return nil, fmt.Errorf("API Key未配置")
|
||||
return nil, fmt.Errorf("API Key 未配置")
|
||||
}
|
||||
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
|
||||
@@ -110,7 +203,7 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
}
|
||||
resp, err := httpClient.Do(httpReq)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("发送请求失败: %w", err)
|
||||
return nil, fmt.Errorf("发送请求失败:%w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
@@ -132,7 +225,7 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
if len(requestBodyPreview) > 200 {
|
||||
requestBodyPreview = requestBodyPreview[:200] + "..."
|
||||
}
|
||||
e.logger.Debug("嵌入API请求",
|
||||
e.logger.Debug("嵌入 API 请求",
|
||||
zap.String("url", httpReq.URL.String()),
|
||||
zap.String("model", model),
|
||||
zap.String("requestBody", requestBodyPreview),
|
||||
@@ -148,12 +241,12 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
if len(bodyPreview) > 500 {
|
||||
bodyPreview = bodyPreview[:500] + "..."
|
||||
}
|
||||
return nil, fmt.Errorf("解析响应失败 (URL: %s, 状态码: %d, 响应长度: %d字节): %w\n请求体: %s\n响应内容预览: %s",
|
||||
return nil, fmt.Errorf("解析响应失败 (URL: %s, 状态码:%d, 响应长度:%d字节): %w\n请求体:%s\n响应内容预览:%s",
|
||||
requestURL, resp.StatusCode, len(bodyBytes), err, requestBodyPreview, bodyPreview)
|
||||
}
|
||||
|
||||
if embeddingResp.Error != nil {
|
||||
return nil, fmt.Errorf("OpenAI API错误 (状态码: %d): 类型=%s, 消息=%s",
|
||||
return nil, fmt.Errorf("OpenAI API 错误 (状态码:%d): 类型=%s, 消息=%s",
|
||||
resp.StatusCode, embeddingResp.Error.Type, embeddingResp.Error.Message)
|
||||
}
|
||||
|
||||
@@ -162,7 +255,7 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
if len(bodyPreview) > 500 {
|
||||
bodyPreview = bodyPreview[:500] + "..."
|
||||
}
|
||||
return nil, fmt.Errorf("HTTP请求失败 (URL: %s, 状态码: %d): 响应内容=%s", requestURL, resp.StatusCode, bodyPreview)
|
||||
return nil, fmt.Errorf("HTTP 请求失败 (URL: %s, 状态码:%d): 响应内容=%s", requestURL, resp.StatusCode, bodyPreview)
|
||||
}
|
||||
|
||||
if len(embeddingResp.Data) == 0 {
|
||||
@@ -170,11 +263,11 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
if len(bodyPreview) > 500 {
|
||||
bodyPreview = bodyPreview[:500] + "..."
|
||||
}
|
||||
return nil, fmt.Errorf("未收到嵌入数据 (状态码: %d, 响应长度: %d字节)\n响应内容: %s",
|
||||
return nil, fmt.Errorf("未收到嵌入数据 (状态码:%d, 响应长度:%d字节)\n响应内容:%s",
|
||||
resp.StatusCode, len(bodyBytes), bodyPreview)
|
||||
}
|
||||
|
||||
// 转换为float32
|
||||
// 转换为 float32
|
||||
embedding := make([]float32, len(embeddingResp.Data[0].Embedding))
|
||||
for i, v := range embeddingResp.Data[0].Embedding {
|
||||
embedding[i] = float32(v)
|
||||
@@ -183,23 +276,48 @@ func (e *Embedder) EmbedText(ctx context.Context, text string) ([]float32, error
|
||||
return embedding, nil
|
||||
}
|
||||
|
||||
// isRetryableError 判断是否是可重试的错误
|
||||
func (e *Embedder) isRetryableError(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
|
||||
errStr := err.Error()
|
||||
|
||||
// 429 速率限制错误
|
||||
if strings.Contains(errStr, "429") || strings.Contains(errStr, "rate limit") {
|
||||
return true
|
||||
}
|
||||
|
||||
// 5xx 服务器错误
|
||||
if strings.Contains(errStr, "500") || strings.Contains(errStr, "502") ||
|
||||
strings.Contains(errStr, "503") || strings.Contains(errStr, "504") {
|
||||
return true
|
||||
}
|
||||
|
||||
// 网络错误
|
||||
if strings.Contains(errStr, "timeout") || strings.Contains(errStr, "connection") ||
|
||||
strings.Contains(errStr, "network") || strings.Contains(errStr, "EOF") {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// EmbedTexts 批量嵌入文本
|
||||
func (e *Embedder) EmbedTexts(ctx context.Context, texts []string) ([][]float32, error) {
|
||||
if len(texts) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// OpenAI API支持批量,但为了简单起见,我们逐个处理
|
||||
// 实际可以使用批量API以提高效率
|
||||
embeddings := make([][]float32, len(texts))
|
||||
for i, text := range texts {
|
||||
embedding, err := e.EmbedText(ctx, text)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("嵌入文本[%d]失败: %w", i, err)
|
||||
return nil, fmt.Errorf("嵌入文本 [%d] 失败:%w", i, err)
|
||||
}
|
||||
embeddings[i] = embedding
|
||||
}
|
||||
|
||||
return embeddings, nil
|
||||
}
|
||||
|
||||
|
||||
+369
-97
@@ -10,56 +10,133 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"cyberstrike-ai/internal/config"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
// Indexer 索引器,负责将知识项分块并向量化
|
||||
type Indexer struct {
|
||||
db *sql.DB
|
||||
embedder *Embedder
|
||||
logger *zap.Logger
|
||||
chunkSize int // 每个块的最大token数(估算)
|
||||
overlap int // 块之间的重叠token数
|
||||
|
||||
db *sql.DB
|
||||
embedder *Embedder
|
||||
logger *zap.Logger
|
||||
chunkSize int // 每个块的最大 token 数(估算)
|
||||
overlap int // 块之间的重叠 token 数
|
||||
maxChunks int // 单个知识项的最大块数量(0 表示不限制)
|
||||
|
||||
// 错误跟踪
|
||||
mu sync.RWMutex
|
||||
lastError string // 最近一次错误信息
|
||||
mu sync.RWMutex
|
||||
lastError string // 最近一次错误信息
|
||||
lastErrorTime time.Time // 最近一次错误时间
|
||||
errorCount int // 连续错误计数
|
||||
errorCount int // 连续错误计数
|
||||
|
||||
// 重建索引状态跟踪
|
||||
rebuildMu sync.RWMutex
|
||||
isRebuilding bool // 是否正在重建索引
|
||||
rebuildTotalItems int // 重建总项数
|
||||
rebuildCurrent int // 当前已处理项数
|
||||
rebuildFailed int // 重建失败项数
|
||||
rebuildStartTime time.Time // 重建开始时间
|
||||
rebuildLastItemID string // 最近处理的项 ID
|
||||
rebuildLastChunks int // 最近处理的项的分块数
|
||||
}
|
||||
|
||||
// NewIndexer 创建新的索引器
|
||||
func NewIndexer(db *sql.DB, embedder *Embedder, logger *zap.Logger) *Indexer {
|
||||
func NewIndexer(db *sql.DB, embedder *Embedder, logger *zap.Logger, indexingCfg *config.IndexingConfig) *Indexer {
|
||||
chunkSize := 512
|
||||
overlap := 50
|
||||
maxChunks := 0
|
||||
if indexingCfg != nil {
|
||||
if indexingCfg.ChunkSize > 0 {
|
||||
chunkSize = indexingCfg.ChunkSize
|
||||
}
|
||||
if indexingCfg.ChunkOverlap >= 0 {
|
||||
overlap = indexingCfg.ChunkOverlap
|
||||
}
|
||||
if indexingCfg.MaxChunksPerItem > 0 {
|
||||
maxChunks = indexingCfg.MaxChunksPerItem
|
||||
}
|
||||
}
|
||||
return &Indexer{
|
||||
db: db,
|
||||
embedder: embedder,
|
||||
logger: logger,
|
||||
chunkSize: 512, // 默认512 tokens
|
||||
overlap: 50, // 默认50 tokens重叠
|
||||
chunkSize: chunkSize,
|
||||
overlap: overlap,
|
||||
maxChunks: maxChunks,
|
||||
}
|
||||
}
|
||||
|
||||
// ChunkText 将文本分块(支持重叠)
|
||||
// ChunkText 将文本分块(支持重叠,保留标题上下文)
|
||||
func (idx *Indexer) ChunkText(text string) []string {
|
||||
// 按Markdown标题分割
|
||||
chunks := idx.splitByMarkdownHeaders(text)
|
||||
// 按 Markdown 标题分割,获取带标题的块
|
||||
sections := idx.splitByMarkdownHeadersWithContent(text)
|
||||
|
||||
// 如果块太大,进一步分割
|
||||
// 处理每个块
|
||||
result := make([]string, 0)
|
||||
for _, chunk := range chunks {
|
||||
if idx.estimateTokens(chunk) <= idx.chunkSize {
|
||||
result = append(result, chunk)
|
||||
for _, section := range sections {
|
||||
// 构建父级标题路径(不包含最后一级标题,因为内容中已经包含)
|
||||
// 例如:["# A", "## B", "### C"] -> "[# A > ## B]"
|
||||
var parentHeaderPath string
|
||||
if len(section.HeaderPath) > 1 {
|
||||
parentHeaderPath = strings.Join(section.HeaderPath[:len(section.HeaderPath)-1], " > ")
|
||||
}
|
||||
|
||||
// 提取内容的第一行作为标题(如 "# Prompt Injection")
|
||||
firstLine, remainingContent := extractFirstLine(section.Content)
|
||||
|
||||
// 如果剩余内容为空或只有空白,说明这个块只有标题没有正文,跳过
|
||||
if strings.TrimSpace(remainingContent) == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// 如果块太大,进一步分割
|
||||
if idx.estimateTokens(section.Content) <= idx.chunkSize {
|
||||
// 块大小合适,添加父级标题前缀
|
||||
if parentHeaderPath != "" {
|
||||
result = append(result, fmt.Sprintf("[%s] %s", parentHeaderPath, section.Content))
|
||||
} else {
|
||||
result = append(result, section.Content)
|
||||
}
|
||||
} else {
|
||||
// 按段落分割
|
||||
subChunks := idx.splitByParagraphs(chunk)
|
||||
for _, subChunk := range subChunks {
|
||||
if idx.estimateTokens(subChunk) <= idx.chunkSize {
|
||||
result = append(result, subChunk)
|
||||
} else {
|
||||
// 按句子分割(支持重叠)
|
||||
chunksWithOverlap := idx.splitBySentencesWithOverlap(subChunk)
|
||||
result = append(result, chunksWithOverlap...)
|
||||
// 块太大,按子标题或段落分割,保持标题上下文
|
||||
// 首先尝试按子标题分割(保留子标题结构)
|
||||
subSections := idx.splitBySubHeaders(section.Content, firstLine, parentHeaderPath)
|
||||
if len(subSections) > 1 {
|
||||
// 成功按子标题分割,递归处理每个子块
|
||||
for _, sub := range subSections {
|
||||
if idx.estimateTokens(sub) <= idx.chunkSize {
|
||||
result = append(result, sub)
|
||||
} else {
|
||||
// 子块仍然太大,按段落分割(保留标题前缀)
|
||||
paragraphs := idx.splitByParagraphsWithHeader(sub, parentHeaderPath)
|
||||
for _, para := range paragraphs {
|
||||
if idx.estimateTokens(para) <= idx.chunkSize {
|
||||
result = append(result, para)
|
||||
} else {
|
||||
// 段落仍太大,按句子分割
|
||||
sentenceChunks := idx.splitBySentencesWithOverlap(para)
|
||||
for _, chunk := range sentenceChunks {
|
||||
result = append(result, chunk)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// 没有子标题,按段落分割(保留标题前缀)
|
||||
paragraphs := idx.splitByParagraphsWithHeader(section.Content, parentHeaderPath)
|
||||
for _, para := range paragraphs {
|
||||
if idx.estimateTokens(para) <= idx.chunkSize {
|
||||
result = append(result, para)
|
||||
} else {
|
||||
// 段落仍太大,按句子分割
|
||||
sentenceChunks := idx.splitBySentencesWithOverlap(para)
|
||||
for _, chunk := range sentenceChunks {
|
||||
result = append(result, chunk)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -68,43 +145,183 @@ func (idx *Indexer) ChunkText(text string) []string {
|
||||
return result
|
||||
}
|
||||
|
||||
// splitByMarkdownHeaders 按Markdown标题分割
|
||||
func (idx *Indexer) splitByMarkdownHeaders(text string) []string {
|
||||
// 匹配Markdown标题 (# ## ### 等)
|
||||
// extractFirstLine 提取第一行内容和剩余内容
|
||||
func extractFirstLine(content string) (firstLine, remaining string) {
|
||||
lines := strings.SplitN(content, "\n", 2)
|
||||
if len(lines) == 0 {
|
||||
return "", ""
|
||||
}
|
||||
if len(lines) == 1 {
|
||||
return lines[0], ""
|
||||
}
|
||||
return lines[0], lines[1]
|
||||
}
|
||||
|
||||
// splitBySubHeaders 尝试按子标题分割内容(用于处理大块内容)
|
||||
// headerPrefix 是父级标题路径,用于添加到每个子块
|
||||
func (idx *Indexer) splitBySubHeaders(content, headerPrefix, parentPath string) []string {
|
||||
// 匹配 Markdown 子标题(## 及以上)
|
||||
subHeaderRegex := regexp.MustCompile(`(?m)^#{2,6}\s+.+$`)
|
||||
matches := subHeaderRegex.FindAllStringIndex(content, -1)
|
||||
|
||||
if len(matches) == 0 {
|
||||
// 没有子标题,返回原始内容
|
||||
return []string{content}
|
||||
}
|
||||
|
||||
result := make([]string, 0, len(matches))
|
||||
for i, match := range matches {
|
||||
start := match[0]
|
||||
nextStart := len(content)
|
||||
if i+1 < len(matches) {
|
||||
nextStart = matches[i+1][0]
|
||||
}
|
||||
|
||||
subContent := strings.TrimSpace(content[start:nextStart])
|
||||
|
||||
// 添加父级路径前缀
|
||||
if parentPath != "" {
|
||||
result = append(result, fmt.Sprintf("[%s] %s", parentPath, subContent))
|
||||
} else {
|
||||
result = append(result, subContent)
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// splitByParagraphsWithHeader 按段落分割,每个段落添加标题前缀(用于保持上下文)
|
||||
func (idx *Indexer) splitByParagraphsWithHeader(content, parentPath string) []string {
|
||||
// 提取第一行作为标题
|
||||
firstLine, _ := extractFirstLine(content)
|
||||
|
||||
paragraphs := strings.Split(content, "\n\n")
|
||||
result := make([]string, 0)
|
||||
|
||||
for i, p := range paragraphs {
|
||||
trimmed := strings.TrimSpace(p)
|
||||
if trimmed == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// 过滤掉只有标题的段落(没有实际内容)
|
||||
if strings.TrimSpace(trimmed) == strings.TrimSpace(firstLine) {
|
||||
continue
|
||||
}
|
||||
|
||||
// 第一个段落已经包含标题,不需要重复添加
|
||||
if i == 0 && strings.Contains(trimmed, firstLine) {
|
||||
if parentPath != "" {
|
||||
result = append(result, fmt.Sprintf("[%s] %s", parentPath, trimmed))
|
||||
} else {
|
||||
result = append(result, trimmed)
|
||||
}
|
||||
} else {
|
||||
// 其他段落添加标题前缀以保持上下文
|
||||
if parentPath != "" {
|
||||
result = append(result, fmt.Sprintf("[%s] %s\n%s", parentPath, firstLine, trimmed))
|
||||
} else {
|
||||
result = append(result, fmt.Sprintf("%s\n%s", firstLine, trimmed))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// Section 表示一个带标题路径的文本块
|
||||
type Section struct {
|
||||
HeaderPath []string // 标题路径(如 ["# SQL 注入", "## 检测方法"])
|
||||
Content string // 块内容
|
||||
}
|
||||
|
||||
// splitByMarkdownHeadersWithContent 按 Markdown 标题分割,返回带标题路径的块
|
||||
// 每个块的内容包含自己的标题,用于向量化检索
|
||||
//
|
||||
// 例如,对于以下 Markdown:
|
||||
// # Prompt Injection
|
||||
// 引言内容
|
||||
// ## Summary
|
||||
// 目录内容
|
||||
//
|
||||
// 返回:
|
||||
// [{HeaderPath: ["# Prompt Injection"], Content: "# Prompt Injection\n引言内容"},
|
||||
// {HeaderPath: ["# Prompt Injection", "## Summary"], Content: "## Summary\n目录内容"}]
|
||||
func (idx *Indexer) splitByMarkdownHeadersWithContent(text string) []Section {
|
||||
// 匹配 Markdown 标题 (# ## ### 等)
|
||||
headerRegex := regexp.MustCompile(`(?m)^#{1,6}\s+.+$`)
|
||||
|
||||
// 找到所有标题位置
|
||||
matches := headerRegex.FindAllStringIndex(text, -1)
|
||||
if len(matches) == 0 {
|
||||
return []string{text}
|
||||
// 没有标题,返回整个文本
|
||||
return []Section{{HeaderPath: []string{}, Content: text}}
|
||||
}
|
||||
|
||||
chunks := make([]string, 0)
|
||||
lastPos := 0
|
||||
sections := make([]Section, 0, len(matches))
|
||||
currentHeaderPath := []string{}
|
||||
|
||||
for _, match := range matches {
|
||||
for i, match := range matches {
|
||||
start := match[0]
|
||||
if start > lastPos {
|
||||
chunks = append(chunks, strings.TrimSpace(text[lastPos:start]))
|
||||
}
|
||||
lastPos = start
|
||||
}
|
||||
end := match[1]
|
||||
nextStart := len(text)
|
||||
|
||||
// 添加最后一部分
|
||||
if lastPos < len(text) {
|
||||
chunks = append(chunks, strings.TrimSpace(text[lastPos:]))
|
||||
// 找到下一个标题的位置
|
||||
if i+1 < len(matches) {
|
||||
nextStart = matches[i+1][0]
|
||||
}
|
||||
|
||||
// 提取当前标题
|
||||
headerLine := strings.TrimSpace(text[start:end])
|
||||
|
||||
// 计算标题层级(# 的数量)
|
||||
level := 0
|
||||
for _, ch := range headerLine {
|
||||
if ch == '#' {
|
||||
level++
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// 更新标题路径:移除比当前层级深或等于的子标题,然后添加当前标题
|
||||
newPath := make([]string, 0, len(currentHeaderPath)+1)
|
||||
for _, h := range currentHeaderPath {
|
||||
hLevel := 0
|
||||
for _, ch := range h {
|
||||
if ch == '#' {
|
||||
hLevel++
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
if hLevel < level {
|
||||
newPath = append(newPath, h)
|
||||
}
|
||||
}
|
||||
newPath = append(newPath, headerLine)
|
||||
currentHeaderPath = newPath
|
||||
|
||||
// 提取当前标题到下一个标题之间的内容(包含当前标题)
|
||||
content := strings.TrimSpace(text[start:nextStart])
|
||||
|
||||
// 创建块,使用当前标题路径(包含当前标题)
|
||||
sections = append(sections, Section{
|
||||
HeaderPath: append([]string(nil), currentHeaderPath...),
|
||||
Content: content,
|
||||
})
|
||||
}
|
||||
|
||||
// 过滤空块
|
||||
result := make([]string, 0)
|
||||
for _, chunk := range chunks {
|
||||
if strings.TrimSpace(chunk) != "" {
|
||||
result = append(result, chunk)
|
||||
result := make([]Section, 0, len(sections))
|
||||
for _, section := range sections {
|
||||
if strings.TrimSpace(section.Content) != "" {
|
||||
result = append(result, section)
|
||||
}
|
||||
}
|
||||
|
||||
if len(result) == 0 {
|
||||
return []string{text}
|
||||
return []Section{{HeaderPath: []string{}, Content: text}}
|
||||
}
|
||||
|
||||
return result
|
||||
@@ -124,8 +341,12 @@ func (idx *Indexer) splitByParagraphs(text string) []string {
|
||||
|
||||
// splitBySentences 按句子分割(用于内部,不包含重叠逻辑)
|
||||
func (idx *Indexer) splitBySentences(text string) []string {
|
||||
// 简单的句子分割(按句号、问号、感叹号)
|
||||
sentenceRegex := regexp.MustCompile(`[.!?]+\s+`)
|
||||
// 简单的句子分割(按句号、问号、感叹号,支持中英文)
|
||||
// . ! ? = 英文标点
|
||||
// \u3002 = 。(中文句号)
|
||||
// \uFF01 = !(中文叹号)
|
||||
// \uFF1F = ?(中文问号)
|
||||
sentenceRegex := regexp.MustCompile(`[.!?\x{3002}\x{FF01}\x{FF1F}]+`)
|
||||
sentences := sentenceRegex.Split(text, -1)
|
||||
result := make([]string, 0)
|
||||
for _, s := range sentences {
|
||||
@@ -221,13 +442,13 @@ func (idx *Indexer) splitBySentencesSimple(text string) []string {
|
||||
return result
|
||||
}
|
||||
|
||||
// extractLastTokens 从文本末尾提取指定token数量的内容
|
||||
// extractLastTokens 从文本末尾提取指定 token 数量的内容
|
||||
func (idx *Indexer) extractLastTokens(text string, tokenCount int) string {
|
||||
if tokenCount <= 0 || text == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
// 估算字符数(1 token ≈ 4字符)
|
||||
// 估算字符数(1 token ≈ 4 字符)
|
||||
charCount := tokenCount * 4
|
||||
runes := []rune(text)
|
||||
|
||||
@@ -236,12 +457,11 @@ func (idx *Indexer) extractLastTokens(text string, tokenCount int) string {
|
||||
}
|
||||
|
||||
// 从末尾提取指定数量的字符
|
||||
// 尝试在句子边界处截断,避免截断句子中间
|
||||
startPos := len(runes) - charCount
|
||||
extracted := string(runes[startPos:])
|
||||
|
||||
// 尝试找到第一个句子边界(句号、问号、感叹号后的空格)
|
||||
sentenceBoundary := regexp.MustCompile(`[.!?]+\s+`)
|
||||
// 尝试找到第一个句子边界(支持中英文标点)
|
||||
sentenceBoundary := regexp.MustCompile(`[.!?\x{3002}\x{FF01}\x{FF1F}]+`)
|
||||
matches := sentenceBoundary.FindStringIndex(extracted)
|
||||
if len(matches) > 0 && matches[0] > 0 {
|
||||
// 在句子边界处截断,保留完整句子
|
||||
@@ -251,41 +471,51 @@ func (idx *Indexer) extractLastTokens(text string, tokenCount int) string {
|
||||
return strings.TrimSpace(extracted)
|
||||
}
|
||||
|
||||
// estimateTokens 估算token数(简单估算:1 token ≈ 4字符)
|
||||
// estimateTokens 估算 token 数(简单估算:1 token ≈ 4 字符)
|
||||
func (idx *Indexer) estimateTokens(text string) int {
|
||||
return len([]rune(text)) / 4
|
||||
}
|
||||
|
||||
// IndexItem 索引知识项(分块并向量化)
|
||||
func (idx *Indexer) IndexItem(ctx context.Context, itemID string) error {
|
||||
// 获取知识项(包含category和title,用于向量化)
|
||||
// 获取知识项(包含 category 和 title,用于向量化)
|
||||
var content, category, title string
|
||||
err := idx.db.QueryRow("SELECT content, category, title FROM knowledge_base_items WHERE id = ?", itemID).Scan(&content, &category, &title)
|
||||
if err != nil {
|
||||
return fmt.Errorf("获取知识项失败: %w", err)
|
||||
return fmt.Errorf("获取知识项失败:%w", err)
|
||||
}
|
||||
|
||||
// 删除旧的向量(在 RebuildIndex 中已经统一清空,这里保留是为了单独调用 IndexItem 时的兼容性)
|
||||
_, err = idx.db.Exec("DELETE FROM knowledge_embeddings WHERE item_id = ?", itemID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("删除旧向量失败: %w", err)
|
||||
return fmt.Errorf("删除旧向量失败:%w", err)
|
||||
}
|
||||
|
||||
// 分块
|
||||
chunks := idx.ChunkText(content)
|
||||
|
||||
// 应用最大块数限制
|
||||
if idx.maxChunks > 0 && len(chunks) > idx.maxChunks {
|
||||
idx.logger.Info("知识项块数量超过限制,已截断",
|
||||
zap.String("itemId", itemID),
|
||||
zap.Int("originalChunks", len(chunks)),
|
||||
zap.Int("maxChunks", idx.maxChunks))
|
||||
chunks = chunks[:idx.maxChunks]
|
||||
}
|
||||
|
||||
idx.logger.Info("知识项分块完成", zap.String("itemId", itemID), zap.Int("chunks", len(chunks)))
|
||||
|
||||
// 跟踪该知识项的错误
|
||||
itemErrorCount := 0
|
||||
var firstError error
|
||||
firstErrorChunkIndex := -1
|
||||
|
||||
// 向量化每个块(包含category和title信息,以便向量检索时能匹配到风险类型)
|
||||
|
||||
// 向量化每个块(包含 category 和 title 信息,以便向量检索时能匹配到风险类型)
|
||||
for i, chunk := range chunks {
|
||||
// 将category和title信息包含到向量化的文本中
|
||||
// 格式:"[风险类型: {category}] [标题: {title}]\n{chunk内容}"
|
||||
// 这样向量嵌入就会包含风险类型信息,即使SQL过滤失败,向量相似度也能帮助匹配
|
||||
textForEmbedding := fmt.Sprintf("[风险类型: %s] [标题: %s]\n%s", category, title, chunk)
|
||||
// 将 category 和 title 信息包含到向量化的文本中
|
||||
// 格式:"[风险类型:{category}] [标题:{title}]\n{chunk 内容}"
|
||||
// 这样向量嵌入就会包含风险类型信息,即使 SQL 过滤失败,向量相似度也能帮助匹配
|
||||
textForEmbedding := fmt.Sprintf("[风险类型:%s] [标题:%s]\n%s", category, title, chunk)
|
||||
|
||||
embedding, err := idx.embedder.EmbedText(ctx, textForEmbedding)
|
||||
if err != nil {
|
||||
@@ -305,17 +535,17 @@ func (idx *Indexer) IndexItem(ctx context.Context, itemID string) error {
|
||||
zap.String("chunkPreview", chunkPreview),
|
||||
zap.Error(err),
|
||||
)
|
||||
|
||||
|
||||
// 更新全局错误跟踪
|
||||
errorMsg := fmt.Sprintf("向量化失败 (知识项: %s): %v", itemID, err)
|
||||
errorMsg := fmt.Sprintf("向量化失败 (知识项:%s): %v", itemID, err)
|
||||
idx.mu.Lock()
|
||||
idx.lastError = errorMsg
|
||||
idx.lastErrorTime = time.Now()
|
||||
idx.mu.Unlock()
|
||||
}
|
||||
|
||||
// 如果连续失败2个块,立即停止处理该知识项(降低阈值,更快停止)
|
||||
// 这样可以避免继续浪费API调用,同时也能更快地检测到配置问题
|
||||
|
||||
// 如果连续失败 2 个块,立即停止处理该知识项(降低阈值,更快停止)
|
||||
// 这样可以避免继续浪费 API 调用,同时也能更快地检测到配置问题
|
||||
if itemErrorCount >= 2 {
|
||||
idx.logger.Error("知识项连续向量化失败,停止处理",
|
||||
zap.String("itemId", itemID),
|
||||
@@ -344,6 +574,13 @@ func (idx *Indexer) IndexItem(ctx context.Context, itemID string) error {
|
||||
}
|
||||
|
||||
idx.logger.Info("知识项索引完成", zap.String("itemId", itemID), zap.Int("chunks", len(chunks)))
|
||||
|
||||
// 更新重建状态中的最近处理信息
|
||||
idx.rebuildMu.Lock()
|
||||
idx.rebuildLastItemID = itemID
|
||||
idx.rebuildLastChunks = len(chunks)
|
||||
idx.rebuildMu.Unlock()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -352,23 +589,38 @@ func (idx *Indexer) HasIndex() (bool, error) {
|
||||
var count int
|
||||
err := idx.db.QueryRow("SELECT COUNT(*) FROM knowledge_embeddings").Scan(&count)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("检查索引失败: %w", err)
|
||||
return false, fmt.Errorf("检查索引失败:%w", err)
|
||||
}
|
||||
return count > 0, nil
|
||||
}
|
||||
|
||||
// RebuildIndex 重建所有索引
|
||||
func (idx *Indexer) RebuildIndex(ctx context.Context) error {
|
||||
// 设置重建状态
|
||||
idx.rebuildMu.Lock()
|
||||
idx.isRebuilding = true
|
||||
idx.rebuildTotalItems = 0
|
||||
idx.rebuildCurrent = 0
|
||||
idx.rebuildFailed = 0
|
||||
idx.rebuildStartTime = time.Now()
|
||||
idx.rebuildLastItemID = ""
|
||||
idx.rebuildLastChunks = 0
|
||||
idx.rebuildMu.Unlock()
|
||||
|
||||
// 重置错误跟踪
|
||||
idx.mu.Lock()
|
||||
idx.lastError = ""
|
||||
idx.lastErrorTime = time.Time{}
|
||||
idx.errorCount = 0
|
||||
idx.mu.Unlock()
|
||||
|
||||
|
||||
rows, err := idx.db.Query("SELECT id FROM knowledge_base_items")
|
||||
if err != nil {
|
||||
return fmt.Errorf("查询知识项失败: %w", err)
|
||||
// 重置重建状态
|
||||
idx.rebuildMu.Lock()
|
||||
idx.isRebuilding = false
|
||||
idx.rebuildMu.Unlock()
|
||||
return fmt.Errorf("查询知识项失败:%w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
@@ -376,34 +628,36 @@ func (idx *Indexer) RebuildIndex(ctx context.Context) error {
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
return fmt.Errorf("扫描知识项ID失败: %w", err)
|
||||
// 重置重建状态
|
||||
idx.rebuildMu.Lock()
|
||||
idx.isRebuilding = false
|
||||
idx.rebuildMu.Unlock()
|
||||
return fmt.Errorf("扫描知识项 ID 失败:%w", err)
|
||||
}
|
||||
itemIDs = append(itemIDs, id)
|
||||
}
|
||||
|
||||
idx.rebuildMu.Lock()
|
||||
idx.rebuildTotalItems = len(itemIDs)
|
||||
idx.rebuildMu.Unlock()
|
||||
|
||||
idx.logger.Info("开始重建索引", zap.Int("totalItems", len(itemIDs)))
|
||||
|
||||
// 在开始重建前,先清空所有旧的向量,确保进度从0开始
|
||||
// 这样 GetIndexStatus 可以准确反映重建进度
|
||||
_, err = idx.db.Exec("DELETE FROM knowledge_embeddings")
|
||||
if err != nil {
|
||||
idx.logger.Warn("清空旧索引失败", zap.Error(err))
|
||||
// 继续执行,即使清空失败也尝试重建
|
||||
} else {
|
||||
idx.logger.Info("已清空旧索引,开始重建")
|
||||
}
|
||||
// 注意:不再清空所有旧索引,而是按增量方式更新
|
||||
// 每个知识项在 IndexItem 中会先删除自己的旧向量,然后插入新向量
|
||||
// 这样配置更新后只重新索引变化的知识项,保留其他知识项的索引
|
||||
|
||||
failedCount := 0
|
||||
consecutiveFailures := 0
|
||||
maxConsecutiveFailures := 2 // 连续失败2次后立即停止(降低阈值,更快停止)
|
||||
maxConsecutiveFailures := 2 // 连续失败 2 次后立即停止(降低阈值,更快停止)
|
||||
firstFailureItemID := ""
|
||||
var firstFailureError error
|
||||
|
||||
|
||||
for i, itemID := range itemIDs {
|
||||
if err := idx.IndexItem(ctx, itemID); err != nil {
|
||||
failedCount++
|
||||
consecutiveFailures++
|
||||
|
||||
|
||||
// 只在第一个失败时记录详细日志
|
||||
if consecutiveFailures == 1 {
|
||||
firstFailureItemID = itemID
|
||||
@@ -414,15 +668,15 @@ func (idx *Indexer) RebuildIndex(ctx context.Context) error {
|
||||
zap.Error(err),
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
// 如果连续失败过多,可能是配置问题,立即停止索引
|
||||
if consecutiveFailures >= maxConsecutiveFailures {
|
||||
errorMsg := fmt.Sprintf("连续 %d 个知识项索引失败,可能存在配置问题(如嵌入模型配置错误、API密钥无效、余额不足等)。第一个失败项: %s, 错误: %v", consecutiveFailures, firstFailureItemID, firstFailureError)
|
||||
errorMsg := fmt.Sprintf("连续 %d 个知识项索引失败,可能存在配置问题(如嵌入模型配置错误、API 密钥无效、余额不足等)。第一个失败项:%s, 错误:%v", consecutiveFailures, firstFailureItemID, firstFailureError)
|
||||
idx.mu.Lock()
|
||||
idx.lastError = errorMsg
|
||||
idx.lastErrorTime = time.Now()
|
||||
idx.mu.Unlock()
|
||||
|
||||
|
||||
idx.logger.Error("连续索引失败次数过多,立即停止索引",
|
||||
zap.Int("consecutiveFailures", consecutiveFailures),
|
||||
zap.Int("totalItems", len(itemIDs)),
|
||||
@@ -430,17 +684,17 @@ func (idx *Indexer) RebuildIndex(ctx context.Context) error {
|
||||
zap.String("firstFailureItemId", firstFailureItemID),
|
||||
zap.Error(firstFailureError),
|
||||
)
|
||||
return fmt.Errorf("连续索引失败次数过多: %v", firstFailureError)
|
||||
return fmt.Errorf("连续索引失败次数过多:%v", firstFailureError)
|
||||
}
|
||||
|
||||
// 如果失败的知识项过多,记录警告但继续处理(降低阈值到30%)
|
||||
|
||||
// 如果失败的知识项过多,记录警告但继续处理(降低阈值到 30%)
|
||||
if failedCount > len(itemIDs)*3/10 && failedCount == len(itemIDs)*3/10+1 {
|
||||
errorMsg := fmt.Sprintf("索引失败的知识项过多 (%d/%d),可能存在配置问题。第一个失败项: %s, 错误: %v", failedCount, len(itemIDs), firstFailureItemID, firstFailureError)
|
||||
errorMsg := fmt.Sprintf("索引失败的知识项过多 (%d/%d),可能存在配置问题。第一个失败项:%s, 错误:%v", failedCount, len(itemIDs), firstFailureItemID, firstFailureError)
|
||||
idx.mu.Lock()
|
||||
idx.lastError = errorMsg
|
||||
idx.lastErrorTime = time.Now()
|
||||
idx.mu.Unlock()
|
||||
|
||||
|
||||
idx.logger.Error("索引失败的知识项过多,可能存在配置问题",
|
||||
zap.Int("failedCount", failedCount),
|
||||
zap.Int("totalItems", len(itemIDs)),
|
||||
@@ -450,20 +704,31 @@ func (idx *Indexer) RebuildIndex(ctx context.Context) error {
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// 成功时重置连续失败计数和第一个失败信息
|
||||
if consecutiveFailures > 0 {
|
||||
consecutiveFailures = 0
|
||||
firstFailureItemID = ""
|
||||
firstFailureError = nil
|
||||
}
|
||||
|
||||
// 减少进度日志频率(每10个或每10%记录一次)
|
||||
|
||||
// 更新重建进度
|
||||
idx.rebuildMu.Lock()
|
||||
idx.rebuildCurrent = i + 1
|
||||
idx.rebuildFailed = failedCount
|
||||
idx.rebuildMu.Unlock()
|
||||
|
||||
// 减少进度日志频率(每 10 个或每 10% 记录一次)
|
||||
if (i+1)%10 == 0 || (len(itemIDs) > 0 && (i+1)*100/len(itemIDs)%10 == 0 && (i+1)*100/len(itemIDs) > 0) {
|
||||
idx.logger.Info("索引进度", zap.Int("current", i+1), zap.Int("total", len(itemIDs)), zap.Int("failed", failedCount))
|
||||
}
|
||||
}
|
||||
|
||||
// 重置重建状态
|
||||
idx.rebuildMu.Lock()
|
||||
idx.isRebuilding = false
|
||||
idx.rebuildMu.Unlock()
|
||||
|
||||
idx.logger.Info("索引重建完成", zap.Int("totalItems", len(itemIDs)), zap.Int("failedCount", failedCount))
|
||||
return nil
|
||||
}
|
||||
@@ -474,3 +739,10 @@ func (idx *Indexer) GetLastError() (string, time.Time) {
|
||||
defer idx.mu.RUnlock()
|
||||
return idx.lastError, idx.lastErrorTime
|
||||
}
|
||||
|
||||
// GetRebuildStatus 获取重建索引状态
|
||||
func (idx *Indexer) GetRebuildStatus() (isRebuilding bool, totalItems int, current int, failed int, lastItemID string, lastChunks int, startTime time.Time) {
|
||||
idx.rebuildMu.RLock()
|
||||
defer idx.rebuildMu.RUnlock()
|
||||
return idx.isRebuilding, idx.rebuildTotalItems, idx.rebuildCurrent, idx.rebuildFailed, idx.rebuildLastItemID, idx.rebuildLastChunks, idx.rebuildStartTime
|
||||
}
|
||||
|
||||
@@ -55,6 +55,14 @@ func New(level, output string) *Logger {
|
||||
}
|
||||
|
||||
func (l *Logger) Fatal(msg string, fields ...interface{}) {
|
||||
l.Logger.Fatal(msg, zap.Any("fields", fields))
|
||||
zapFields := make([]zap.Field, 0, len(fields))
|
||||
for _, f := range fields {
|
||||
switch v := f.(type) {
|
||||
case error:
|
||||
zapFields = append(zapFields, zap.Error(v))
|
||||
default:
|
||||
zapFields = append(zapFields, zap.Any("field", v))
|
||||
}
|
||||
}
|
||||
l.Logger.Fatal(msg, zapFields...)
|
||||
}
|
||||
|
||||
|
||||
@@ -1582,12 +1582,106 @@ header {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
.chat-input-container > .chat-input-with-files {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
min-width: 0;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.chat-input-container > .chat-input-field {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.chat-file-list {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 6px;
|
||||
align-items: center;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.chat-file-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
padding: 4px 10px;
|
||||
background: rgba(0, 102, 255, 0.08);
|
||||
border: 1px solid rgba(0, 102, 255, 0.2);
|
||||
border-radius: 8px;
|
||||
font-size: 12px;
|
||||
color: var(--text-primary);
|
||||
max-width: 200px;
|
||||
}
|
||||
|
||||
.chat-file-chip-name {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.chat-file-chip-remove {
|
||||
flex-shrink: 0;
|
||||
width: 18px;
|
||||
height: 18px;
|
||||
padding: 0;
|
||||
border: none;
|
||||
background: rgba(0, 0, 0, 0.08);
|
||||
border-radius: 50%;
|
||||
cursor: pointer;
|
||||
color: var(--text-muted);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
transition: background 0.2s, color 0.2s;
|
||||
}
|
||||
|
||||
.chat-file-chip-remove:hover {
|
||||
background: rgba(0, 0, 0, 0.12);
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.chat-file-input-hidden {
|
||||
position: absolute;
|
||||
width: 0;
|
||||
height: 0;
|
||||
opacity: 0;
|
||||
overflow: hidden;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
.chat-upload-btn {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
width: 40px;
|
||||
height: 40px;
|
||||
padding: 0;
|
||||
border: 1px solid rgba(0, 0, 0, 0.1);
|
||||
border-radius: 12px;
|
||||
background: #ffffff;
|
||||
color: var(--text-muted);
|
||||
cursor: pointer;
|
||||
transition: border-color 0.2s, color 0.2s, background 0.2s;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.chat-upload-btn:hover {
|
||||
border-color: var(--accent-color);
|
||||
color: var(--accent-color);
|
||||
background: rgba(0, 102, 255, 0.04);
|
||||
}
|
||||
|
||||
.chat-input-container.drag-over {
|
||||
background: rgba(0, 102, 255, 0.06);
|
||||
border-radius: 12px;
|
||||
outline: 2px dashed rgba(0, 102, 255, 0.35);
|
||||
outline-offset: -2px;
|
||||
}
|
||||
|
||||
.chat-input-field {
|
||||
flex: 1;
|
||||
position: relative;
|
||||
@@ -3038,6 +3132,164 @@ header {
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
/* 系统设置 - 终端 */
|
||||
.terminal-wrapper {
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 6px;
|
||||
overflow: hidden;
|
||||
background: #0d1117;
|
||||
}
|
||||
|
||||
.terminal-tabs {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 0;
|
||||
padding: 6px 10px 0;
|
||||
background: #161b22;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
min-height: 36px;
|
||||
}
|
||||
|
||||
.terminal-tab {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
padding: 5px 8px 5px 12px;
|
||||
font-size: 0.8125rem;
|
||||
color: #8b949e;
|
||||
background: transparent;
|
||||
border: none;
|
||||
border-radius: 4px 4px 0 0;
|
||||
cursor: pointer;
|
||||
margin-right: 2px;
|
||||
}
|
||||
|
||||
.terminal-tab-label {
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.terminal-tab-close {
|
||||
padding: 0;
|
||||
width: 18px;
|
||||
height: 18px;
|
||||
font-size: 1.1rem;
|
||||
line-height: 1;
|
||||
color: #8b949e;
|
||||
background: transparent;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
cursor: pointer;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.terminal-tab-close:hover {
|
||||
color: #ff7b72;
|
||||
background: rgba(255, 123, 114, 0.15);
|
||||
}
|
||||
|
||||
.terminal-tab:hover {
|
||||
color: #e6edf3;
|
||||
background: rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
|
||||
.terminal-tab.active {
|
||||
color: #e6edf3;
|
||||
background: #0d1117;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.terminal-tab-new {
|
||||
margin-left: 4px;
|
||||
width: 28px;
|
||||
height: 28px;
|
||||
font-size: 1.125rem;
|
||||
line-height: 1;
|
||||
color: #8b949e;
|
||||
background: transparent;
|
||||
border: 1px solid transparent;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.terminal-tab-new:hover {
|
||||
color: #58a6ff;
|
||||
background: rgba(88, 166, 255, 0.1);
|
||||
border-color: rgba(88, 166, 255, 0.3);
|
||||
}
|
||||
|
||||
.terminal-toolbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 10px 16px;
|
||||
background: #161b22;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.terminal-toolbar-title {
|
||||
font-size: 0.8125rem;
|
||||
font-weight: 600;
|
||||
color: #8b949e;
|
||||
letter-spacing: 0.02em;
|
||||
}
|
||||
|
||||
.terminal-btn {
|
||||
padding: 6px 12px;
|
||||
font-size: 0.8125rem;
|
||||
color: var(--text-secondary);
|
||||
background: transparent;
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
transition: color 0.15s, background 0.15s, border-color 0.15s;
|
||||
}
|
||||
|
||||
.terminal-btn:hover {
|
||||
color: var(--text-primary);
|
||||
background: rgba(255, 255, 255, 0.06);
|
||||
border-color: #8b949e;
|
||||
}
|
||||
|
||||
.terminal-panes {
|
||||
position: relative;
|
||||
min-height: 400px;
|
||||
}
|
||||
|
||||
.terminal-pane {
|
||||
display: none;
|
||||
min-height: 400px;
|
||||
}
|
||||
|
||||
.terminal-pane.active {
|
||||
display: block;
|
||||
}
|
||||
|
||||
.terminal-container {
|
||||
min-height: 400px;
|
||||
padding: 8px;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
.terminal-container .xterm {
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.terminal-container .xterm-viewport {
|
||||
border-radius: 0;
|
||||
}
|
||||
|
||||
.terminal-error {
|
||||
color: #ff7b72;
|
||||
padding: 16px;
|
||||
font-size: 0.875rem;
|
||||
}
|
||||
|
||||
.settings-section {
|
||||
margin-bottom: 32px;
|
||||
}
|
||||
|
||||
+165
-15
@@ -22,6 +22,12 @@ const DRAFT_STORAGE_KEY = 'cyberstrike-chat-draft';
|
||||
let draftSaveTimer = null;
|
||||
const DRAFT_SAVE_DELAY = 500; // 500ms防抖延迟
|
||||
|
||||
// 对话文件上传相关(后端会拼接路径与内容发给大模型,前端不再重复发文件列表)
|
||||
const MAX_CHAT_FILES = 10;
|
||||
const CHAT_FILE_DEFAULT_PROMPT = '请根据上传的文件内容进行分析。';
|
||||
/** @type {{ fileName: string, content: string, mimeType: string }[]} */
|
||||
let chatAttachments = [];
|
||||
|
||||
// 保存输入框草稿到localStorage(防抖版本)
|
||||
function saveChatDraftDebounced(content) {
|
||||
// 清除之前的定时器
|
||||
@@ -107,14 +113,22 @@ function adjustTextareaHeight(textarea) {
|
||||
// 发送消息
|
||||
async function sendMessage() {
|
||||
const input = document.getElementById('chat-input');
|
||||
const message = input.value.trim();
|
||||
|
||||
if (!message) {
|
||||
let message = input.value.trim();
|
||||
const hasAttachments = chatAttachments && chatAttachments.length > 0;
|
||||
|
||||
if (!message && !hasAttachments) {
|
||||
return;
|
||||
}
|
||||
|
||||
// 显示用户消息
|
||||
addMessage('user', message);
|
||||
// 有附件且用户未输入时,发一句简短默认提示即可(后端会拼接路径和文件内容给大模型)
|
||||
if (hasAttachments && !message) {
|
||||
message = CHAT_FILE_DEFAULT_PROMPT;
|
||||
}
|
||||
|
||||
// 显示用户消息(含附件名,便于用户确认)
|
||||
const displayMessage = hasAttachments
|
||||
? message + '\n' + chatAttachments.map(a => '📎 ' + a.fileName).join('\n')
|
||||
: message;
|
||||
addMessage('user', displayMessage);
|
||||
|
||||
// 清除防抖定时器,防止在清空输入框后重新保存草稿
|
||||
if (draftSaveTimer) {
|
||||
@@ -135,7 +149,24 @@ async function sendMessage() {
|
||||
input.value = '';
|
||||
// 强制重置输入框高度为初始高度(40px)
|
||||
input.style.height = '40px';
|
||||
|
||||
|
||||
// 构建请求体(含附件)
|
||||
const body = {
|
||||
message: message,
|
||||
conversationId: currentConversationId,
|
||||
role: typeof getCurrentRole === 'function' ? getCurrentRole() : ''
|
||||
};
|
||||
if (hasAttachments) {
|
||||
body.attachments = chatAttachments.map(a => ({
|
||||
fileName: a.fileName,
|
||||
content: a.content,
|
||||
mimeType: a.mimeType || ''
|
||||
}));
|
||||
}
|
||||
// 发送后清空附件列表
|
||||
chatAttachments = [];
|
||||
renderChatFileChips();
|
||||
|
||||
// 创建进度消息容器(使用详细的进度展示)
|
||||
const progressId = addProgressMessage();
|
||||
const progressElement = document.getElementById(progressId);
|
||||
@@ -145,19 +176,12 @@ async function sendMessage() {
|
||||
let mcpExecutionIds = [];
|
||||
|
||||
try {
|
||||
// 获取当前选中的角色(从 roles.js 的函数获取)
|
||||
const roleName = typeof getCurrentRole === 'function' ? getCurrentRole() : '';
|
||||
|
||||
const response = await apiFetch('/api/agent-loop/stream', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({
|
||||
message: message,
|
||||
conversationId: currentConversationId,
|
||||
role: roleName || undefined
|
||||
}),
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
@@ -222,6 +246,130 @@ async function sendMessage() {
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- 对话文件上传 ----------
|
||||
function renderChatFileChips() {
|
||||
const list = document.getElementById('chat-file-list');
|
||||
if (!list) return;
|
||||
list.innerHTML = '';
|
||||
if (!chatAttachments.length) return;
|
||||
chatAttachments.forEach((a, i) => {
|
||||
const chip = document.createElement('div');
|
||||
chip.className = 'chat-file-chip';
|
||||
chip.setAttribute('role', 'listitem');
|
||||
const name = document.createElement('span');
|
||||
name.className = 'chat-file-chip-name';
|
||||
name.title = a.fileName;
|
||||
name.textContent = a.fileName;
|
||||
const remove = document.createElement('button');
|
||||
remove.type = 'button';
|
||||
remove.className = 'chat-file-chip-remove';
|
||||
remove.title = '移除';
|
||||
remove.innerHTML = '×';
|
||||
remove.setAttribute('aria-label', '移除 ' + a.fileName);
|
||||
remove.addEventListener('click', () => removeChatAttachment(i));
|
||||
chip.appendChild(name);
|
||||
chip.appendChild(remove);
|
||||
list.appendChild(chip);
|
||||
});
|
||||
}
|
||||
|
||||
function removeChatAttachment(index) {
|
||||
chatAttachments.splice(index, 1);
|
||||
renderChatFileChips();
|
||||
}
|
||||
|
||||
// 有附件且输入框为空时,填入一句默认提示(可编辑);后端会单独拼接路径与内容给大模型
|
||||
function appendChatFilePrompt() {
|
||||
const input = document.getElementById('chat-input');
|
||||
if (!input || !chatAttachments.length) return;
|
||||
if (!input.value.trim()) {
|
||||
input.value = CHAT_FILE_DEFAULT_PROMPT;
|
||||
adjustTextareaHeight(input);
|
||||
}
|
||||
}
|
||||
|
||||
function readFileAsAttachment(file) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const mimeType = file.type || '';
|
||||
const isTextLike = /^text\//i.test(mimeType) || /^(application\/(json|xml|javascript)|image\/svg\+xml)/i.test(mimeType);
|
||||
const reader = new FileReader();
|
||||
reader.onload = () => {
|
||||
let content = reader.result;
|
||||
if (typeof content === 'string' && content.startsWith('data:')) {
|
||||
content = content.replace(/^data:[^;]+;base64,/, '');
|
||||
}
|
||||
resolve({ fileName: file.name, content: content, mimeType: mimeType });
|
||||
};
|
||||
reader.onerror = () => reject(reader.error);
|
||||
if (isTextLike) {
|
||||
reader.readAsText(file, 'UTF-8');
|
||||
} else {
|
||||
reader.readAsDataURL(file);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function addFilesToChat(files) {
|
||||
if (!files || !files.length) return;
|
||||
const next = Array.from(files);
|
||||
if (chatAttachments.length + next.length > MAX_CHAT_FILES) {
|
||||
alert('最多同时上传 ' + MAX_CHAT_FILES + ' 个文件,当前已选 ' + chatAttachments.length + ' 个。');
|
||||
return;
|
||||
}
|
||||
const addOne = (file) => {
|
||||
return readFileAsAttachment(file).then((a) => {
|
||||
chatAttachments.push(a);
|
||||
renderChatFileChips();
|
||||
appendChatFilePrompt();
|
||||
}).catch(() => {
|
||||
alert('读取文件失败:' + file.name);
|
||||
});
|
||||
};
|
||||
let p = Promise.resolve();
|
||||
next.forEach((file) => { p = p.then(() => addOne(file)); });
|
||||
p.then(() => {});
|
||||
}
|
||||
|
||||
function setupChatFileUpload() {
|
||||
const inputEl = document.getElementById('chat-file-input');
|
||||
const container = document.getElementById('chat-input-container') || document.querySelector('.chat-input-container');
|
||||
if (!inputEl || !container) return;
|
||||
|
||||
inputEl.addEventListener('change', function () {
|
||||
const files = this.files;
|
||||
if (files && files.length) {
|
||||
addFilesToChat(files);
|
||||
}
|
||||
this.value = '';
|
||||
});
|
||||
|
||||
container.addEventListener('dragover', function (e) {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
this.classList.add('drag-over');
|
||||
});
|
||||
container.addEventListener('dragleave', function (e) {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
if (!this.contains(e.relatedTarget)) {
|
||||
this.classList.remove('drag-over');
|
||||
}
|
||||
});
|
||||
container.addEventListener('drop', function (e) {
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
this.classList.remove('drag-over');
|
||||
const files = e.dataTransfer && e.dataTransfer.files;
|
||||
if (files && files.length) addFilesToChat(files);
|
||||
});
|
||||
}
|
||||
|
||||
// 确保 chat-input-container 有 id(若模板未写)
|
||||
function ensureChatInputContainerId() {
|
||||
const c = document.querySelector('.chat-input-container');
|
||||
if (c && !c.id) c.id = 'chat-input-container';
|
||||
}
|
||||
|
||||
function setupMentionSupport() {
|
||||
mentionSuggestionsEl = document.getElementById('mention-suggestions');
|
||||
if (mentionSuggestionsEl) {
|
||||
@@ -799,6 +947,8 @@ function initializeChatUI() {
|
||||
}
|
||||
activeTaskInterval = setInterval(() => loadActiveTasks(), ACTIVE_TASK_REFRESH_INTERVAL);
|
||||
setupMentionSupport();
|
||||
ensureChatInputContainerId();
|
||||
setupChatFileUpload();
|
||||
}
|
||||
|
||||
// 消息计数器,确保ID唯一
|
||||
|
||||
@@ -459,6 +459,9 @@ async function updateIndexProgress() {
|
||||
const isComplete = status.is_complete || false;
|
||||
const lastError = status.last_error || '';
|
||||
|
||||
// 检查是否正在重建索引(优先使用重建状态)
|
||||
const isRebuilding = status.is_rebuilding || false;
|
||||
|
||||
if (totalItems === 0) {
|
||||
// 没有知识项,隐藏进度条
|
||||
progressContainer.style.display = 'none';
|
||||
@@ -524,6 +527,45 @@ async function updateIndexProgress() {
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
// 优先处理重建状态
|
||||
if (isRebuilding) {
|
||||
const rebuildTotal = status.rebuild_total || totalItems;
|
||||
const rebuildCurrent = status.rebuild_current || 0;
|
||||
const rebuildFailed = status.rebuild_failed || 0;
|
||||
const rebuildLastItemID = status.rebuild_last_item_id || '';
|
||||
const rebuildLastChunks = status.rebuild_last_chunks || 0;
|
||||
const rebuildStartTime = status.rebuild_start_time || '';
|
||||
|
||||
// 计算进度百分比(使用重建进度)
|
||||
let rebuildProgress = progressPercent;
|
||||
if (rebuildTotal > 0) {
|
||||
rebuildProgress = (rebuildCurrent / rebuildTotal) * 100;
|
||||
}
|
||||
|
||||
progressContainer.innerHTML = `
|
||||
<div class="knowledge-index-progress">
|
||||
<div class="progress-header">
|
||||
<span class="progress-icon">🔨</span>
|
||||
<span class="progress-text">正在重建索引:${rebuildCurrent}/${rebuildTotal} (${rebuildProgress.toFixed(1)}%) - 失败:${rebuildFailed}</span>
|
||||
</div>
|
||||
<div class="progress-bar-container">
|
||||
<div class="progress-bar" style="width: ${rebuildProgress}%"></div>
|
||||
</div>
|
||||
<div class="progress-hint">
|
||||
${rebuildLastItemID ? `正在处理:${escapeHtml(rebuildLastItemID.substring(0, 36))}... (${rebuildLastChunks} chunks)` : '正在处理...'}
|
||||
${rebuildStartTime ? `<br>开始时间:${new Date(rebuildStartTime).toLocaleString()}` : ''}
|
||||
</div>
|
||||
</div>
|
||||
`;
|
||||
|
||||
// 重建中时继续轮询
|
||||
if (!indexProgressInterval) {
|
||||
indexProgressInterval = setInterval(updateIndexProgress, 2000);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (isComplete) {
|
||||
progressContainer.innerHTML = `
|
||||
<div class="knowledge-index-progress-complete">
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -46,6 +46,9 @@ function switchSettingsSection(section) {
|
||||
if (activeContent) {
|
||||
activeContent.classList.add('active');
|
||||
}
|
||||
if (section === 'terminal' && typeof initTerminal === 'function') {
|
||||
setTimeout(initTerminal, 0);
|
||||
}
|
||||
}
|
||||
|
||||
// 打开设置
|
||||
@@ -169,6 +172,43 @@ async function loadConfig(loadTools = true) {
|
||||
// 允许0.0值,只有undefined/null时才使用默认值
|
||||
retrievalWeightInput.value = (hybridWeight !== undefined && hybridWeight !== null) ? hybridWeight : 0.7;
|
||||
}
|
||||
|
||||
// 索引配置
|
||||
const indexing = knowledge.indexing || {};
|
||||
const chunkSizeInput = document.getElementById('knowledge-indexing-chunk-size');
|
||||
if (chunkSizeInput) {
|
||||
chunkSizeInput.value = indexing.chunk_size || 512;
|
||||
}
|
||||
|
||||
const chunkOverlapInput = document.getElementById('knowledge-indexing-chunk-overlap');
|
||||
if (chunkOverlapInput) {
|
||||
chunkOverlapInput.value = indexing.chunk_overlap ?? 50;
|
||||
}
|
||||
|
||||
const maxChunksPerItemInput = document.getElementById('knowledge-indexing-max-chunks-per-item');
|
||||
if (maxChunksPerItemInput) {
|
||||
maxChunksPerItemInput.value = indexing.max_chunks_per_item ?? 0;
|
||||
}
|
||||
|
||||
const maxRpmInput = document.getElementById('knowledge-indexing-max-rpm');
|
||||
if (maxRpmInput) {
|
||||
maxRpmInput.value = indexing.max_rpm ?? 0;
|
||||
}
|
||||
|
||||
const rateLimitDelayInput = document.getElementById('knowledge-indexing-rate-limit-delay-ms');
|
||||
if (rateLimitDelayInput) {
|
||||
rateLimitDelayInput.value = indexing.rate_limit_delay_ms ?? 300;
|
||||
}
|
||||
|
||||
const maxRetriesInput = document.getElementById('knowledge-indexing-max-retries');
|
||||
if (maxRetriesInput) {
|
||||
maxRetriesInput.value = indexing.max_retries ?? 3;
|
||||
}
|
||||
|
||||
const retryDelayInput = document.getElementById('knowledge-indexing-retry-delay-ms');
|
||||
if (retryDelayInput) {
|
||||
retryDelayInput.value = indexing.retry_delay_ms ?? 1000;
|
||||
}
|
||||
}
|
||||
|
||||
// 填充机器人配置
|
||||
@@ -725,6 +765,15 @@ async function applySettings() {
|
||||
const val = parseFloat(document.getElementById('knowledge-retrieval-hybrid-weight')?.value);
|
||||
return isNaN(val) ? 0.7 : val; // 允许0.0值,只有NaN时才使用默认值
|
||||
})()
|
||||
},
|
||||
indexing: {
|
||||
chunk_size: parseInt(document.getElementById("knowledge-indexing-chunk-size")?.value) || 512,
|
||||
chunk_overlap: parseInt(document.getElementById("knowledge-indexing-chunk-overlap")?.value) ?? 50,
|
||||
max_chunks_per_item: parseInt(document.getElementById("knowledge-indexing-max-chunks-per-item")?.value) ?? 0,
|
||||
max_rpm: parseInt(document.getElementById("knowledge-indexing-max-rpm")?.value) ?? 0,
|
||||
rate_limit_delay_ms: parseInt(document.getElementById("knowledge-indexing-rate-limit-delay-ms")?.value) ?? 300,
|
||||
max_retries: parseInt(document.getElementById("knowledge-indexing-max-retries")?.value) ?? 3,
|
||||
retry_delay_ms: parseInt(document.getElementById("knowledge-indexing-retry-delay-ms")?.value) ?? 1000
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,419 @@
|
||||
/**
|
||||
* 系统设置 - 终端:多标签、流式输出、命令历史、Ctrl+L 清屏、长时间可取消
|
||||
*/
|
||||
(function () {
|
||||
var getContext = HTMLCanvasElement.prototype.getContext;
|
||||
HTMLCanvasElement.prototype.getContext = function (type, attrs) {
|
||||
if (type === '2d') {
|
||||
attrs = (attrs && typeof attrs === 'object') ? Object.assign({ willReadFrequently: true }, attrs) : { willReadFrequently: true };
|
||||
return getContext.call(this, type, attrs);
|
||||
}
|
||||
return getContext.apply(this, arguments);
|
||||
};
|
||||
|
||||
var terminals = [];
|
||||
var currentTabId = 1;
|
||||
var inited = false;
|
||||
var tabIdCounter = 1;
|
||||
var PROMPT = ''; // 真实 Shell 自己输出提示符,这里不再自定义
|
||||
var HISTORY_MAX = 100;
|
||||
var CANCEL_AFTER_MS = 125000;
|
||||
|
||||
function getCurrent() {
|
||||
for (var i = 0; i < terminals.length; i++) {
|
||||
if (terminals[i].id === currentTabId) return terminals[i];
|
||||
}
|
||||
return terminals[0] || null;
|
||||
}
|
||||
|
||||
var WELCOME_LINE = 'CyberStrikeAI 终端 - 真实 Shell 会话,直接输入命令;Ctrl+L 清屏\r\n';
|
||||
|
||||
function writePrompt(tab) {
|
||||
// 提示符交由后端 Shell 自行输出,这里仅保留占位函数,避免旧代码报错
|
||||
}
|
||||
|
||||
function redrawTabDisplay(t) {
|
||||
if (!t || !t.term) return;
|
||||
t.term.clear();
|
||||
t.term.write(WELCOME_LINE);
|
||||
}
|
||||
|
||||
function writeln(tabOrS, s) {
|
||||
var t, text;
|
||||
if (arguments.length === 1) { text = tabOrS; t = getCurrent(); } else { t = tabOrS; text = s; }
|
||||
if (!t || !t.term) return;
|
||||
if (text) t.term.writeln(text);
|
||||
else t.term.writeln('');
|
||||
}
|
||||
|
||||
function writeOutput(tab, text, isError) {
|
||||
var t = tab || getCurrent();
|
||||
if (!t || !t.term || !text) return;
|
||||
var s = String(text).replace(/\r\n/g, '\n').replace(/\r/g, '\n');
|
||||
var lines = s.split('\n');
|
||||
var prefix = isError ? '\x1b[31m' : '';
|
||||
var suffix = isError ? '\x1b[0m' : '';
|
||||
t.term.write(prefix);
|
||||
for (var i = 0; i < lines.length; i++) {
|
||||
var line = lines[i].replace(/\r/g, '');
|
||||
t.term.writeln(line);
|
||||
}
|
||||
t.term.write(suffix);
|
||||
}
|
||||
|
||||
// 从本地存储中获取当前登录 token(与 auth.js 使用的结构保持一致)
|
||||
function getStoredAuthToken() {
|
||||
try {
|
||||
var raw = localStorage.getItem('cyberstrike-auth');
|
||||
if (!raw) return null;
|
||||
var o = JSON.parse(raw);
|
||||
if (o && o.token) return o.token;
|
||||
} catch (e) {}
|
||||
return null;
|
||||
}
|
||||
|
||||
// WebSocket 地址构造(兼容 http/https,并通过 query 传递 token 以通过后端鉴权)
|
||||
function buildTerminalWSURL() {
|
||||
var proto = (window.location.protocol === 'https:') ? 'wss://' : 'ws://';
|
||||
var url = proto + window.location.host + '/api/terminal/ws';
|
||||
var token = getStoredAuthToken();
|
||||
if (token) {
|
||||
url += '?token=' + encodeURIComponent(token);
|
||||
}
|
||||
return url;
|
||||
}
|
||||
|
||||
function ensureTerminalWS(tab) {
|
||||
if (tab.ws && (tab.ws.readyState === WebSocket.OPEN || tab.ws.readyState === WebSocket.CONNECTING)) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
var ws = new WebSocket(buildTerminalWSURL());
|
||||
tab.ws = ws;
|
||||
tab.running = true;
|
||||
|
||||
ws.onopen = function () {
|
||||
if (tab.term) {
|
||||
tab.term.focus();
|
||||
}
|
||||
};
|
||||
|
||||
ws.onmessage = function (ev) {
|
||||
if (!tab.term) return;
|
||||
// 处理二进制消息和文本消息
|
||||
if (ev.data instanceof ArrayBuffer) {
|
||||
var decoder = new TextDecoder('utf-8');
|
||||
tab.term.write(decoder.decode(ev.data));
|
||||
} else if (ev.data instanceof Blob) {
|
||||
// Blob 类型,需要异步读取
|
||||
var reader = new FileReader();
|
||||
reader.onload = function () {
|
||||
var decoder = new TextDecoder('utf-8');
|
||||
tab.term.write(decoder.decode(reader.result));
|
||||
};
|
||||
reader.readAsArrayBuffer(ev.data);
|
||||
} else {
|
||||
// 字符串类型
|
||||
tab.term.write(ev.data);
|
||||
}
|
||||
};
|
||||
|
||||
ws.onclose = function () {
|
||||
tab.running = false;
|
||||
if (tab.term) {
|
||||
tab.term.writeln('\r\n\x1b[2m[会话已关闭]\x1b[0m');
|
||||
}
|
||||
};
|
||||
|
||||
ws.onerror = function () {
|
||||
tab.running = false;
|
||||
if (tab.term) {
|
||||
tab.term.writeln('\r\n\x1b[31m[终端连接出错]\x1b[0m');
|
||||
}
|
||||
};
|
||||
} catch (e) {
|
||||
if (tab.term) {
|
||||
tab.term.writeln('\r\n\x1b[31m[无法连接终端服务: ' + String(e) + ']\x1b[0m');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function createTerminalInContainer(container, tab) {
|
||||
if (typeof Terminal === 'undefined') return null;
|
||||
if (!tab.history) tab.history = [];
|
||||
if (tab.historyIndex === undefined) tab.historyIndex = -1;
|
||||
if (tab.cursorIndex === undefined) tab.cursorIndex = 0;
|
||||
|
||||
var term = new Terminal({
|
||||
cursorBlink: true,
|
||||
cursorStyle: 'bar',
|
||||
fontSize: 13,
|
||||
fontFamily: 'Menlo, Monaco, "Courier New", monospace',
|
||||
lineHeight: 1.2,
|
||||
scrollback: 1000,
|
||||
theme: {
|
||||
background: '#0d1117',
|
||||
foreground: '#e6edf3',
|
||||
cursor: '#58a6ff',
|
||||
cursorAccent: '#0d1117',
|
||||
selection: 'rgba(88, 166, 255, 0.3)',
|
||||
black: '#484f58',
|
||||
red: '#ff7b72',
|
||||
green: '#3fb950',
|
||||
yellow: '#d29922',
|
||||
blue: '#58a6ff',
|
||||
magenta: '#bc8cff',
|
||||
cyan: '#39c5cf',
|
||||
white: '#e6edf3',
|
||||
brightBlack: '#6e7681',
|
||||
brightRed: '#ffa198',
|
||||
brightGreen: '#56d364',
|
||||
brightYellow: '#e3b341',
|
||||
brightBlue: '#79c0ff',
|
||||
brightMagenta: '#d2a8ff',
|
||||
brightCyan: '#56d4dd',
|
||||
brightWhite: '#f0f6fc'
|
||||
}
|
||||
});
|
||||
var fitAddon = null;
|
||||
if (typeof FitAddon !== 'undefined') {
|
||||
var FitCtor = (FitAddon.FitAddon || FitAddon);
|
||||
fitAddon = new FitCtor();
|
||||
term.loadAddon(fitAddon);
|
||||
}
|
||||
term.open(container);
|
||||
term.write(WELCOME_LINE);
|
||||
container.addEventListener('click', function () {
|
||||
switchTerminalTab(tab.id);
|
||||
if (term) term.focus();
|
||||
});
|
||||
container.setAttribute('tabindex', '0');
|
||||
container.title = '点击此处后输入命令';
|
||||
|
||||
function sendToWS(data) {
|
||||
ensureTerminalWS(tab);
|
||||
if (tab.ws && tab.ws.readyState === WebSocket.OPEN) {
|
||||
try {
|
||||
tab.ws.send(data);
|
||||
} catch (e) {}
|
||||
}
|
||||
}
|
||||
|
||||
term.onData(function (data) {
|
||||
// Ctrl+L:本地清屏,同时把 ^L 也发给后端
|
||||
if (data === '\x0c') {
|
||||
term.clear();
|
||||
sendToWS(data);
|
||||
return;
|
||||
}
|
||||
sendToWS(data);
|
||||
});
|
||||
|
||||
tab.term = term;
|
||||
tab.fitAddon = fitAddon;
|
||||
return term;
|
||||
}
|
||||
|
||||
function switchTerminalTab(id) {
|
||||
var prevId = currentTabId;
|
||||
currentTabId = id;
|
||||
document.querySelectorAll('.terminal-tab').forEach(function (el) {
|
||||
el.classList.toggle('active', parseInt(el.getAttribute('data-tab-id'), 10) === id);
|
||||
});
|
||||
document.querySelectorAll('.terminal-pane').forEach(function (el) {
|
||||
var paneId = el.getAttribute('id');
|
||||
var match = paneId && paneId.match(/terminal-pane-(\d+)/);
|
||||
var paneTabId = match ? parseInt(match[1], 10) : 0;
|
||||
el.classList.toggle('active', paneTabId === id);
|
||||
});
|
||||
var t = getCurrent();
|
||||
if (t && t.term) {
|
||||
if (prevId !== id) {
|
||||
requestAnimationFrame(function () {
|
||||
if (currentTabId === id && t.term) t.term.focus();
|
||||
});
|
||||
} else {
|
||||
t.term.focus();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function addTerminalTab() {
|
||||
if (typeof Terminal === 'undefined') return;
|
||||
tabIdCounter += 1;
|
||||
var id = tabIdCounter;
|
||||
var paneId = 'terminal-pane-' + id;
|
||||
var containerId = 'terminal-container-' + id;
|
||||
var tabsEl = document.querySelector('.terminal-tabs');
|
||||
var panesEl = document.querySelector('.terminal-panes');
|
||||
if (!tabsEl || !panesEl) return;
|
||||
|
||||
var tabDiv = document.createElement('div');
|
||||
tabDiv.className = 'terminal-tab';
|
||||
tabDiv.setAttribute('data-tab-id', String(id));
|
||||
var label = document.createElement('span');
|
||||
label.className = 'terminal-tab-label';
|
||||
label.textContent = '终端 ' + id;
|
||||
label.onclick = function () { switchTerminalTab(id); };
|
||||
var closeBtn = document.createElement('button');
|
||||
closeBtn.type = 'button';
|
||||
closeBtn.className = 'terminal-tab-close';
|
||||
closeBtn.title = '关闭';
|
||||
closeBtn.textContent = '×';
|
||||
closeBtn.onclick = function (e) { e.stopPropagation(); removeTerminalTab(id); };
|
||||
tabDiv.appendChild(label);
|
||||
tabDiv.appendChild(closeBtn);
|
||||
var plusBtn = tabsEl.querySelector('.terminal-tab-new');
|
||||
tabsEl.insertBefore(tabDiv, plusBtn);
|
||||
|
||||
var paneDiv = document.createElement('div');
|
||||
paneDiv.id = paneId;
|
||||
paneDiv.className = 'terminal-pane';
|
||||
var containerDiv = document.createElement('div');
|
||||
containerDiv.id = containerId;
|
||||
containerDiv.className = 'terminal-container';
|
||||
paneDiv.appendChild(containerDiv);
|
||||
panesEl.appendChild(paneDiv);
|
||||
|
||||
var tab = { id: id, paneId: paneId, containerId: containerId, lineBuffer: '', cursorIndex: 0, running: false, term: null, fitAddon: null, history: [], historyIndex: -1 };
|
||||
terminals.push(tab);
|
||||
createTerminalInContainer(containerDiv, tab);
|
||||
switchTerminalTab(id);
|
||||
updateTerminalTabCloseVisibility();
|
||||
setTimeout(function () {
|
||||
try { if (tab.fitAddon) tab.fitAddon.fit(); if (tab.term) tab.term.focus(); } catch (e) {}
|
||||
}, 50);
|
||||
}
|
||||
|
||||
function updateTerminalTabCloseVisibility() {
|
||||
var tabsEl = document.querySelector('.terminal-tabs');
|
||||
if (!tabsEl) return;
|
||||
var tabDivs = tabsEl.querySelectorAll('.terminal-tab');
|
||||
var showClose = terminals.length > 1;
|
||||
for (var i = 0; i < tabDivs.length; i++) {
|
||||
var btn = tabDivs[i].querySelector('.terminal-tab-close');
|
||||
if (btn) btn.style.display = showClose ? '' : 'none';
|
||||
}
|
||||
}
|
||||
|
||||
function removeTerminalTab(id) {
|
||||
if (terminals.length <= 1) return;
|
||||
var idx = -1;
|
||||
for (var i = 0; i < terminals.length; i++) { if (terminals[i].id === id) { idx = i; break; } }
|
||||
if (idx < 0) return;
|
||||
|
||||
var deletingCurrent = (currentTabId === id);
|
||||
var switchToIndex = deletingCurrent ? (idx > 0 ? idx - 1 : 0) : -1;
|
||||
|
||||
var tab = terminals[idx];
|
||||
if (tab.term && tab.term.dispose) tab.term.dispose();
|
||||
tab.term = null;
|
||||
tab.fitAddon = null;
|
||||
terminals.splice(idx, 1);
|
||||
|
||||
var tabDiv = document.querySelector('.terminal-tab[data-tab-id="' + id + '"]');
|
||||
var paneDiv = document.getElementById('terminal-pane-' + id);
|
||||
if (tabDiv && tabDiv.parentNode) tabDiv.parentNode.removeChild(tabDiv);
|
||||
if (paneDiv && paneDiv.parentNode) paneDiv.parentNode.removeChild(paneDiv);
|
||||
|
||||
var curIdxBeforeRenumber = -1;
|
||||
if (!deletingCurrent) {
|
||||
for (var i = 0; i < terminals.length; i++) {
|
||||
if (terminals[i].id === currentTabId) { curIdxBeforeRenumber = i; break; }
|
||||
}
|
||||
}
|
||||
|
||||
for (var i = 0; i < terminals.length; i++) {
|
||||
var t = terminals[i];
|
||||
t.id = i + 1;
|
||||
t.paneId = 'terminal-pane-' + (i + 1);
|
||||
t.containerId = 'terminal-container-' + (i + 1);
|
||||
}
|
||||
tabIdCounter = terminals.length;
|
||||
if (curIdxBeforeRenumber >= 0) currentTabId = terminals[curIdxBeforeRenumber].id;
|
||||
|
||||
var tabsEl = document.querySelector('.terminal-tabs');
|
||||
var panesEl = document.querySelector('.terminal-panes');
|
||||
if (tabsEl) {
|
||||
var tabDivs = tabsEl.querySelectorAll('.terminal-tab');
|
||||
for (var i = 0; i < tabDivs.length; i++) {
|
||||
var t = terminals[i];
|
||||
tabDivs[i].setAttribute('data-tab-id', String(t.id));
|
||||
var lbl = tabDivs[i].querySelector('.terminal-tab-label');
|
||||
if (lbl) lbl.textContent = '终端 ' + t.id;
|
||||
if (lbl) lbl.onclick = (function (tid) { return function () { switchTerminalTab(tid); }; })(t.id);
|
||||
var cb = tabDivs[i].querySelector('.terminal-tab-close');
|
||||
if (cb) cb.onclick = (function (tid) { return function (e) { e.stopPropagation(); removeTerminalTab(tid); }; })(t.id);
|
||||
}
|
||||
}
|
||||
if (panesEl) {
|
||||
var paneDivs = panesEl.querySelectorAll('.terminal-pane');
|
||||
for (var i = 0; i < paneDivs.length; i++) {
|
||||
var t = terminals[i];
|
||||
paneDivs[i].id = t.paneId;
|
||||
var cont = paneDivs[i].querySelector('.terminal-container');
|
||||
if (cont) cont.id = t.containerId;
|
||||
}
|
||||
}
|
||||
|
||||
updateTerminalTabCloseVisibility();
|
||||
|
||||
if (deletingCurrent && terminals.length > 0) {
|
||||
currentTabId = terminals[switchToIndex].id;
|
||||
switchTerminalTab(currentTabId);
|
||||
}
|
||||
}
|
||||
|
||||
function initTerminal() {
|
||||
var pane1 = document.getElementById('terminal-pane-1');
|
||||
var container1 = document.getElementById('terminal-container-1');
|
||||
if (!pane1 || !container1) return;
|
||||
if (inited) {
|
||||
var t = getCurrent();
|
||||
if (t && t.term) t.term.focus();
|
||||
terminals.forEach(function (tab) { try { if (tab.fitAddon) tab.fitAddon.fit(); } catch (e) {} });
|
||||
return;
|
||||
}
|
||||
inited = true;
|
||||
|
||||
if (typeof Terminal === 'undefined') {
|
||||
container1.innerHTML = '<p class="terminal-error">未加载 xterm.js,请刷新页面或检查网络。</p>';
|
||||
return;
|
||||
}
|
||||
|
||||
currentTabId = 1;
|
||||
var tab = { id: 1, paneId: 'terminal-pane-1', containerId: 'terminal-container-1', lineBuffer: '', cursorIndex: 0, running: false, term: null, fitAddon: null, history: [], historyIndex: -1 };
|
||||
terminals.push(tab);
|
||||
createTerminalInContainer(container1, tab);
|
||||
|
||||
updateTerminalTabCloseVisibility();
|
||||
|
||||
setTimeout(function () {
|
||||
try { if (tab.fitAddon) tab.fitAddon.fit(); if (tab.term) tab.term.focus(); } catch (e) {}
|
||||
}, 100);
|
||||
|
||||
var resizeTimer;
|
||||
window.addEventListener('resize', function () {
|
||||
clearTimeout(resizeTimer);
|
||||
resizeTimer = setTimeout(function () {
|
||||
terminals.forEach(function (t) { try { if (t.fitAddon) t.fitAddon.fit(); } catch (e) {} });
|
||||
}, 150);
|
||||
});
|
||||
}
|
||||
|
||||
function terminalClear() {
|
||||
var t = getCurrent();
|
||||
if (!t || !t.term) return;
|
||||
t.term.clear();
|
||||
t.lineBuffer = '';
|
||||
if (t.cursorIndex !== undefined) t.cursorIndex = 0;
|
||||
writePrompt(t);
|
||||
t.term.focus();
|
||||
}
|
||||
|
||||
window.initTerminal = initTerminal;
|
||||
window.terminalClear = terminalClear;
|
||||
window.switchTerminalTab = switchTerminalTab;
|
||||
window.addTerminalTab = addTerminalTab;
|
||||
window.removeTerminalTab = removeTerminalTab;
|
||||
})();
|
||||
@@ -7,6 +7,7 @@
|
||||
<link rel="icon" type="image/png" href="/static/logo.png">
|
||||
<link rel="shortcut icon" type="image/png" href="/static/favicon.ico">
|
||||
<link rel="stylesheet" href="/static/css/style.css">
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/xterm@4.19.0/css/xterm.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="login-overlay" class="login-overlay" style="display: none;">
|
||||
@@ -499,7 +500,7 @@
|
||||
</div>
|
||||
<div id="active-tasks-bar" class="active-tasks-bar"></div>
|
||||
<div id="chat-messages" class="chat-messages"></div>
|
||||
<div class="chat-input-container">
|
||||
<div id="chat-input-container" class="chat-input-container">
|
||||
<div class="role-selector-wrapper">
|
||||
<button id="role-selector-btn" class="role-selector-btn" onclick="toggleRoleSelectionPanel()" title="选择角色">
|
||||
<span id="role-selector-icon" class="role-selector-icon">🔵</span>
|
||||
@@ -521,10 +522,19 @@
|
||||
<div id="role-selection-list" class="role-selection-list-main"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="chat-input-field">
|
||||
<textarea id="chat-input" placeholder="输入测试目标或命令... (输入 @ 选择工具 | Shift+Enter 换行,Enter 发送)" rows="1"></textarea>
|
||||
<div id="mention-suggestions" class="mention-suggestions" role="listbox" aria-label="工具提及候选"></div>
|
||||
<div class="chat-input-with-files">
|
||||
<div id="chat-file-list" class="chat-file-list" aria-label="已选文件列表"></div>
|
||||
<div class="chat-input-field">
|
||||
<textarea id="chat-input" placeholder="输入测试目标或命令... (输入 @ 选择工具 | Shift+Enter 换行,Enter 发送)" rows="1"></textarea>
|
||||
<div id="mention-suggestions" class="mention-suggestions" role="listbox" aria-label="工具提及候选"></div>
|
||||
</div>
|
||||
</div>
|
||||
<input type="file" id="chat-file-input" class="chat-file-input-hidden" multiple accept="*" title="选择文件">
|
||||
<button type="button" class="chat-upload-btn" onclick="document.getElementById('chat-file-input').click()" title="上传文件(可多选或拖拽到此处)">
|
||||
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4M17 8l-5-5-5 5M12 3v12" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
</svg>
|
||||
</button>
|
||||
<button class="send-btn" onclick="sendMessage()">
|
||||
<span>发送</span>
|
||||
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
@@ -1065,6 +1075,9 @@
|
||||
<div class="settings-nav-item" data-section="robots" onclick="switchSettingsSection('robots')">
|
||||
<span>机器人设置</span>
|
||||
</div>
|
||||
<div class="settings-nav-item" data-section="terminal" onclick="switchSettingsSection('terminal')">
|
||||
<span>终端</span>
|
||||
</div>
|
||||
<div class="settings-nav-item" data-section="security" onclick="switchSettingsSection('security')">
|
||||
<span>安全设置</span>
|
||||
</div>
|
||||
@@ -1190,7 +1203,44 @@
|
||||
<small class="form-hint">向量检索的权重(0-1),1.0表示纯向量检索,0.0表示纯关键词检索</small>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="settings-subsection-header">
|
||||
<h5>索引配置</h5>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="knowledge-indexing-chunk-size">分块大小(Chunk Size)</label>
|
||||
<input type="number" id="knowledge-indexing-chunk-size" min="128" max="4096" placeholder="512" />
|
||||
<small class="form-hint">每个块的最大 token 数(默认 512),长文本会被分割成多个块</small>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="knowledge-indexing-chunk-overlap">分块重叠(Chunk Overlap)</label>
|
||||
<input type="number" id="knowledge-indexing-chunk-overlap" min="0" max="512" placeholder="50" />
|
||||
<small class="form-hint">块之间的重叠 token 数(默认 50),保持上下文连贯性</small>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="knowledge-indexing-max-chunks-per-item">单个知识项最大块数</label>
|
||||
<input type="number" id="knowledge-indexing-max-chunks-per-item" min="0" max="1000" placeholder="0" />
|
||||
<small class="form-hint">单个知识项的最大块数量(0 表示不限制),防止单个文件消耗过多 API 配额</small>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="knowledge-indexing-max-rpm">每分钟最大请求数(Max RPM)</label>
|
||||
<input type="number" id="knowledge-indexing-max-rpm" min="0" max="1000" placeholder="0" />
|
||||
<small class="form-hint">每分钟最大请求数(默认 0 表示不限制),如 OpenAI 默认 200 RPM</small>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="knowledge-indexing-rate-limit-delay-ms">请求间隔延迟(毫秒)</label>
|
||||
<input type="number" id="knowledge-indexing-rate-limit-delay-ms" min="0" max="10000" placeholder="300" />
|
||||
<small class="form-hint">请求间隔毫秒数(默认 300),用于避免 API 速率限制,设为 0 不限制</small>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="knowledge-indexing-max-retries">最大重试次数</label>
|
||||
<input type="number" id="knowledge-indexing-max-retries" min="0" max="10" placeholder="3" />
|
||||
<small class="form-hint">最大重试次数(默认 3),遇到速率限制或服务器错误时自动重试</small>
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="knowledge-indexing-retry-delay-ms">重试间隔(毫秒)</label>
|
||||
<input type="number" id="knowledge-indexing-retry-delay-ms" min="0" max="10000" placeholder="1000" />
|
||||
<small class="form-hint">重试间隔毫秒数(默认 1000),每次重试会递增延迟</small>
|
||||
</div> </div>
|
||||
|
||||
<div class="settings-actions">
|
||||
<button class="btn-primary" onclick="applySettings()">应用配置</button>
|
||||
@@ -1311,6 +1361,25 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 终端 -->
|
||||
<div id="settings-section-terminal" class="settings-section-content">
|
||||
<div class="settings-section-header">
|
||||
<h3>终端</h3>
|
||||
<p class="settings-description">在服务器上执行命令,便于运维与调试。命令在服务端执行,请勿执行敏感或破坏性操作。</p>
|
||||
</div>
|
||||
<div class="terminal-wrapper">
|
||||
<div class="terminal-tabs">
|
||||
<div class="terminal-tab active" data-tab-id="1"><span class="terminal-tab-label" onclick="switchTerminalTab(1)">终端 1</span><button type="button" class="terminal-tab-close" onclick="removeTerminalTab(1); event.stopPropagation();" title="关闭">×</button></div>
|
||||
<button type="button" class="terminal-tab-new" onclick="addTerminalTab()" title="新终端">+</button>
|
||||
</div>
|
||||
<div class="terminal-panes">
|
||||
<div id="terminal-pane-1" class="terminal-pane active">
|
||||
<div id="terminal-container-1" class="terminal-container"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 安全设置 -->
|
||||
<div id="settings-section-security" class="settings-section-content">
|
||||
<div class="settings-section-header">
|
||||
@@ -2135,6 +2204,9 @@ version: 1.0.0<br>
|
||||
<script src="/static/js/monitor.js"></script>
|
||||
<script src="/static/js/chat.js"></script>
|
||||
<script src="/static/js/settings.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/xterm@4.19.0/lib/xterm.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/xterm-addon-fit@0.5.0/lib/xterm-addon-fit.js"></script>
|
||||
<script src="/static/js/terminal.js"></script>
|
||||
<script src="/static/js/knowledge.js"></script>
|
||||
<script src="/static/js/skills.js"></script>
|
||||
<script src="/static/js/vulnerability.js?v=4"></script>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user