mirror of
https://github.com/Ed1s0nZ/CyberStrikeAI.git
synced 2026-10-18 13:41:42 +02:00
259 lines
14 KiB
Markdown
259 lines
14 KiB
Markdown
# Developer Guide
|
|
|
|
[中文](../zh-CN/developer-guide.md)
|
|
|
|
This guide is for contributors extending CyberStrikeAI. The project is a Go single-service application with a static frontend, SQLite persistence, Agent/MCP orchestration, and optional high-risk security subsystems.
|
|
|
|
## Project Layout
|
|
|
|
```text
|
|
cmd/server/ service entrypoint
|
|
internal/app/ app wiring, routes, MCP tool registration
|
|
internal/handler/ HTTP handlers
|
|
internal/database/ SQLite access
|
|
internal/security/ auth, rate limits, shell execution
|
|
internal/mcp/ MCP server and external MCP manager
|
|
internal/multiagent/ Eino single-agent, multi-agent, middleware
|
|
internal/workflow/ graph orchestration runtime
|
|
internal/knowledge/ indexing and retrieval
|
|
internal/c2/ built-in C2
|
|
internal/project/ project fact blackboard
|
|
web/static/ frontend JS/CSS/assets
|
|
web/templates/ HTML templates
|
|
tools/ YAML command tools
|
|
roles/ role YAML
|
|
agents/ multi-agent Markdown definitions
|
|
skills/ Agent Skills
|
|
docs/ documentation
|
|
```
|
|
|
|
## Development Startup
|
|
|
|
```bash
|
|
go run ./cmd/server --config config.yaml
|
|
```
|
|
|
|
The frontend is static. Most JS/CSS/template changes only require a browser refresh.
|
|
|
|
## One-Click Update: Letting an Installation Update Its Own Source
|
|
|
|
On a deployed machine the routine is "my tree moved forward by a few dozen commits -> bring this
|
|
installation up -> rebuild -> swap the binary -> restart". This used to be only `upgrade.sh`, which
|
|
pointed at one fixed upstream repository, so for anyone running a fork "upgrade" meant "overwrite
|
|
yourself with somebody else's code". The capability now lives in the platform (`internal/update`) and
|
|
the console page, the REST API and the CLI share one implementation, so the three surfaces cannot
|
|
disagree about why an update was refused.
|
|
|
|
### Three entry points
|
|
|
|
| entry | how it is used |
|
|
|---|---|
|
|
| console | System settings -> One-click update (`#system-update`, the old deep link still works). Opening this section reads the local tree only and does not go online; **Check** is what performs the fetch; **Update** starts a job the page polls; the "exit after updating" tick is pre-ticked when a supervisor is detected (launchd/systemd startup markers), and while the process is down the page watches for its return and reloads itself (sessions live in memory, so a fresh login follows); a rollback button sits below |
|
|
| REST | `GET /api/system/update` (state on disk, no network), `POST /api/system/update/check` (fetch, then report the gap), `POST /api/system/update/apply` (`202` with `job_id`, polled through `GET /api/system/update/job`), `POST /api/system/update/rollback` |
|
|
| CLI | `./cyberstrike-ai -check-update`, `./cyberstrike-ai -update`, `./cyberstrike-ai -update-rollback`. The install root is the directory holding `--config`, or the current directory when `--config` was not given |
|
|
|
|
Permissions are `update:read` (GET) and `update:apply` (the four mutating endpoints). `update:apply`
|
|
additionally requires a global (`all` scope) session: one machine has one source tree, and an
|
|
`assigned`/`own` session must not be able to move the code everybody else is running.
|
|
|
|
### Where the code comes from
|
|
|
|
From **the update source repository**: one address, the official repository
|
|
`https://github.com/AIPentest/CyberStrikeAI.git` by default, and the update follows that repository's
|
|
**own default branch** (nothing about branches is configured, and "main" is not hardcoded), so an
|
|
installation with no configuration at all still has exactly one answer to "where does an update come
|
|
from". `upgrade.sh` is now a thin shell over the same implementation: when this directory is a git
|
|
work tree it just calls `./cyberstrike-ai -update`.
|
|
|
|
### Changing the source (the update section of config.yaml)
|
|
|
|
The update source is a single field: a repository address. Empty means the official repository; to
|
|
follow your own fork, a mirror (when GitHub is hard to reach), or somebody else's repository, this is
|
|
the one place to change:
|
|
|
|
```yaml
|
|
update:
|
|
repo: https://github.com/AIPentest/CyberStrikeAI.git # one repository address; empty/absent = the official repository
|
|
```
|
|
|
|
The console's "Update source" block edits and saves this address (saving it empty goes back to the
|
|
official repository). Saving validates the address against an allowlist: https/http/ssh/git/file plus
|
|
local absolute paths - git's `ext::` transport executes commands and is rejected outright.
|
|
|
|
### Connecting a non-git installation (the tarball kind)
|
|
|
|
An installation that started as an unpacked Release archive has no `.git`; it can be connected from
|
|
the same page (the default official repository works without any configuration): the preview fetches in a throwaway repository and first lists
|
|
**the local files the target would replace** and **the operator content that will be kept**; on
|
|
confirm the directory becomes a git work tree (`git init`, origin added, the target branch checked
|
|
out), every replaced file is kept under `.update-backup/<unique-backup>/overwritten/`, operator content
|
|
is stashed and restored as usual, and the binary is rebuilt. From then on it is a normal
|
|
installation with one-click update and rollback - but note there is no git history before the
|
|
connection, so the connection itself has no earlier commit to roll back to; the first real update
|
|
writes the first rollback point.
|
|
|
|
### The version number follows the code
|
|
|
|
The repository's `config.example.yaml` carries the release version this code was cut from
|
|
(upstream bumps it in the release commit), while your `config.yaml` is operator data that no
|
|
update ever touches. So after a successful merge the update writes the new code's version back
|
|
into `config.yaml`'s `version` field (that one line only, temp file plus atomic rename; the job
|
|
log records the old and the new number). After a restart the header badge and the static
|
|
assets' `?v=` both match the code again - which is also why the frontend cache invalidates
|
|
itself on upgrade. A rollback puts the number back too (unless somebody hand-edited it in the
|
|
meantime), and connecting a tarball install syncs it as well.
|
|
|
|
### The four refusals
|
|
|
|
| situation | reason | meaning and what to do |
|
|
|---|---|---|
|
|
| product **source** modified locally (a path outside the protected list) | `local_source_edits` | An update is a download, not a conflict decision. The file names are listed; commit or restore them first |
|
|
| branch **diverged** from its remote (ahead and behind at once) | `diverged` | That is a merge decision; the counts are reported so you can resolve it by hand |
|
|
| no Go toolchain on this box | `no_toolchain` | **The source still updates**; only the binary is not swapped. Install `go` and press update again to finish |
|
|
| this directory is not a git work tree | `not_a_repo` (`installed: false` in the status) | A tarball installation has no remote to fast-forward; update it through the release package |
|
|
|
|
`apply` is asynchronous, so it always answers `202` first and the refusals above arrive in the polled
|
|
`job.failure` (`reason` / `message` / `items`). `rollback` is synchronous and maps refusals to status
|
|
codes (`local_source_edits`, `diverged`, `no_state`, `moved_since_update`, `no_binary` -> 409, anything
|
|
else -> 400). Other failures (`merge_failed`, `build_failed`, `swap_failed`) carry git's or the
|
|
compiler's own output; a failed build leaves the previous binary in place while the source has already
|
|
moved and the state file is written, so it stays rollback-able.
|
|
|
|
Check, update, adoption, rollback and restart share an operating-system file lock in the install
|
|
directory (`.update-lock`). Wait for the other operation when `update_busy` is reported. The system
|
|
releases the lock when its process exits; do not delete the lock file to bypass it.
|
|
|
|
### How operator content survives
|
|
|
|
`roles/ skills/ tools/ agents/ knowledge_base/ data/ log/ venv/ config.yaml .env` belong to the
|
|
**operator** (`update.Protected` is the judge). Before the merge, only those of those paths the
|
|
update actually writes to are copied into `.update-backup/<unique-backup>/` and removed from the tree -
|
|
your extra directories that upstream never touched are not disturbed - and they are put back
|
|
afterwards, which is what "update the code, keep my work" means. The result names every file in
|
|
`keptContent` instead of quietly dropping it: where both you and upstream changed the same file, the
|
|
version that lands is yours, and the result says so.
|
|
|
|
### Rollback
|
|
|
|
`./cyberstrike-ai -update-rollback`, or the button on the page: `git reset --hard` back to the commit
|
|
the update came from, the binary kept as `cyberstrike-ai.prev` is put back when that update swapped
|
|
one, and `.update-state.json` is removed. An update whose build failed never swapped a binary, so its
|
|
rollback moves the source only - the binary on disk already belongs to the commit it returns to. It
|
|
refuses when there is no update record (`no_state`), a swap is owed but no `cyberstrike-ai.prev` is
|
|
left (`no_binary`), a recorded commit is not in this repository (`bad_state`), the binary on disk is
|
|
not the one the record pairs with that commit (`binary_changed`), **HEAD has moved since that update**
|
|
(`moved_since_update` - a rollback undoes the update, not the work done after it), or source is
|
|
modified locally (`local_source_edits`).
|
|
|
|
Rollback writes `.update-rollback-pending.json` before changing source or binaries. Run rollback
|
|
again after an interruption to finish recovery; the console reports `rollbackPending` and pauses new
|
|
updates. Keep the recovery record, backup directory and `.prev` file. If recorded binary hashes do not
|
|
match the files on disk, recovery refuses to overwrite them.
|
|
|
|
### The two restart cases
|
|
|
|
The process only stands down (graceful `Shutdown`, then exit 0) when the request explicitly says
|
|
`restart: true` **and** a restart hook was wired at startup. Whether it actually comes back is up to a
|
|
supervisor (systemd, `run.sh`), and the page says exactly that rather than promising a boot that may
|
|
not happen. Asking for a restart with no hook wired is a `400`, instead of stopping the service and
|
|
claiming it restarted.
|
|
|
|
**A pending binary is a durable state, and it can be activated later.** The status endpoint compares
|
|
the binary identity captured at startup (size + nanosecond mtime) with what is on disk now: after an
|
|
update, a rollback or a CLI run swapped the file without a restart, `needsRestart` is true,
|
|
`binaryBuiltAt` says when it was built, and the console grows a persistent banner with **Restart now**
|
|
(`POST /api/system/update/restart`; `409 nothing_pending` when there is nothing to activate, `409`
|
|
while a job runs, `400` without a hook). The `supervised` field comes from environment markers
|
|
(launchd's `XPC_SERVICE_NAME`, systemd's `INVOCATION_ID`/`JOURNAL_STREAM`) and only decides the tick's
|
|
default.
|
|
|
|
**And the page recovers by itself.** While the process is down the console becomes a recovery view
|
|
that probes the status endpoint every 2 seconds: a 200 means the old process is still answering (keep
|
|
waiting), a dead connection means it is on its way out, and a 401 from a process that no longer knows
|
|
the session is the proof the restart happened - the page then `location.replace`s itself back to
|
|
`#system-update`. Leaving the console stops the watching instead of yanking the user back.
|
|
|
|
## Adding a Business Module
|
|
|
|
Do not add only a handler. A complete module usually needs:
|
|
|
|
1. Data model and SQLite migration.
|
|
2. Handler: parameters, errors, pagination/filtering.
|
|
3. Audit: management actions.
|
|
4. Monitor: long-running execution state.
|
|
5. MCP: whether Agents should call it.
|
|
6. HITL: approval boundary for MCP tools.
|
|
7. OpenAPI: update `/api/openapi/spec`.
|
|
8. Frontend: i18n, states, empty/error UI.
|
|
9. Tests: DB, handler, edge cases.
|
|
10. Docs: config, usage, troubleshooting, safety impact.
|
|
|
|
Missing one of these usually becomes a later usability or safety bug.
|
|
|
|
## Error Response Design
|
|
|
|
Prefer stable JSON:
|
|
|
|
```json
|
|
{
|
|
"error": "machine_readable_code",
|
|
"message": "human-readable explanation"
|
|
}
|
|
```
|
|
|
|
Frontend needs stable fields, users need actionable messages, and logs need detailed internal errors.
|
|
|
|
## Long-Running Tasks
|
|
|
|
For scanning, indexing, batch tasks, C2, or external operations, answer:
|
|
|
|
- Can it be cancelled?
|
|
- Can progress be queried?
|
|
- Can it be retried?
|
|
- Where is the result stored?
|
|
- Does state survive page refresh?
|
|
- Does it block the HTTP request?
|
|
|
|
If not, use task tables, event streams, or monitoring.
|
|
|
|
## Extending Tools
|
|
|
|
Prefer `tools/*.yaml` for command tools. Use Go built-in tools when the tool needs internal state or structured integration.
|
|
|
|
Built-in tools should define clear input schemas, handle timeouts and errors, and respect HITL for risky actions.
|
|
|
|
## Frontend Changes
|
|
|
|
Use existing helpers such as `apiFetch`, modal utilities, notifications, and i18n. Update both `web/static/i18n/zh-CN.json` and `web/static/i18n/en-US.json` for new visible text.
|
|
|
|
Avoid putting secrets or provider keys in frontend code.
|
|
|
|
## Test Priority
|
|
|
|
High-value tests:
|
|
|
|
- config hot-apply;
|
|
- HITL branches;
|
|
- shell timeout/no-output;
|
|
- external MCP recovery;
|
|
- KB indexing and post-processing;
|
|
- WebShell OS/encoding detection;
|
|
- SQLite migration compatibility.
|
|
|
|
## Source Anchors
|
|
|
|
- App wiring: `internal/app/app.go`
|
|
- Config apply: `internal/handler/config.go`
|
|
- OpenAPI: `internal/handler/openapi.go`
|
|
- Tool executor: `internal/security/executor.go`
|
|
- Skill package: `internal/skillpackage/`
|
|
- One-click update: `internal/update/` (status, apply, rollback), HTTP surface in
|
|
`internal/handler/update.go`, CLI switches in `cmd/server/update_cli.go`, thin shell in
|
|
`upgrade.sh`
|
|
|
|
### Recovery evidence and MCP activation edits
|
|
|
|
Self-update recovery validates JSON field names/types, duplicate keys, commit/hash relationships and the actual binary files before resuming filesystem changes. The plain-text legacy build marker remains supported, but any accompanying State is validated too. An ambiguous, incomplete or unsupported record is refused with the files and recovery evidence preserved. Do not delete a pending record just to bypass the refusal: first reconcile the source commit, live binary and previous binary/backup.
|
|
|
|
Editing an existing external MCP without activation fields preserves its current enabled/stopped state. Explicit `disabled` or `external_mcp_enable` fields change activation; `disabled: true` takes precedence. A new server defaults to enabled. Start/Stop also persist the corresponding disabled state, so editing or restarting the application does not silently enable a stopped server.
|