Files
anonymous_github/deploy/logging/README.md
T

362 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Operations logging
This is an opt-in stack for one Docker host. It does not deploy itself, change
production, or send notifications. The application writes structured JSON to
stdout. Vector tails tagged Docker json-file logs and host Nginx access JSON,
rebuilds each record from an allowlist, then spools sanitized records to disk.
VictoriaLogs stores searchable compressed records. Grafana provides investigation
panels. Prometheus monitors the logging components and vmalert evaluates LogsQL
alerts. Alertmanager groups firing and resolved state with an empty receiver.
## Deploy after approval
Review the application and infrastructure changes together. This PR is stacked
on `fix/remaining-production-findings`; do not deploy it without that base.
1. Reserve at least 20 GiB of dedicated logging storage and 2 GiB of RAM, outside
the application and MongoDB budgets. The configured memory ceilings total
1,856 MiB. Check real free memory before enabling this stack. Docker volumes
survive container replacement, but not deletion of the volumes or host loss.
2. Create a strong Grafana password in an external secret file. Give UID 472 read
access and restrict the parent directory to operators. Do not place the file
in the repository. Export `GRAFANA_PASSWORD_FILE` as its absolute path and
`GRAFANA_ADMIN_USER` as a non-default username. No password value belongs in
a shell command, Compose file, issue, or PR.
3. Set `APP_RELEASE` to the deployed 7 to 40 character Git SHA. Append the logging
overrides to the deployment's existing `COMPOSE_FILE`; do not replace existing
overrides. In particular, GitHub App deployments need
`docker-compose.github-app.yml` to retain the read-only signing-key mount in
both the API and streamers. For a GitHub App deployment with admin logging,
the complete list is
`COMPOSE_FILE=docker-compose.yml:docker-compose.github-app.yml:deploy/logging/app.compose.yml:deploy/logging/admin.compose.yml`.
Preserve any other site-specific overrides too. Run `scripts/deploy.sh`, which
checks that each replacement service can sign with the configured key before
stopping live services. Never print the key or expanded Compose environment.
This applies explicit service tags and nonblocking 10 MiB × 3 local buffers to
the API, all four streamers, MongoDB, Redis and the backup process. BullMQ
sandbox workers log through the API container. Existing containers need
recreation to change their Docker log configuration.
4. Include `../nginx/anonymous-logging.conf` in the host Nginx `http` context and
configure the relevant server with
`access_log /var/log/nginx/anonymous.access.json anonymous_json;`.
Test with `nginx -t`, then reload through the existing deployment workflow.
Keep Nginx `real_ip` and proxy trust unchanged. Do not enable Mongo profiling,
query tracing, Redis MONITOR, or request-body logging.
5. Install `nginx.logrotate` through the host configuration workflow and run it
hourly with its own state file. Keep logs root/adm-readable only. This retains
about six hourly files, with a 10 MiB rotation trigger checked hourly. A burst
can exceed 10 MiB between checks; this is not a hard file quota. Use a dedicated
filesystem quota if hard local containment is needed. Use rename/reopen, not
copytruncate. Vector excludes compressed history and reads new files from end.
6. Set `NGINX_LOG_GID` to the numeric group owning the access log
(`stat -c %g /var/log/nginx/anonymous.access.json`; normally the host `adm`
group). Vector receives this supplementary group while retaining no Linux
capabilities and read-only mounts. Keep logs `0640` and ensure logrotate
creates replacements with the same group, as in `nginx.logrotate`; do not
make access logs world-readable. Recreate Vector after changing this setting.
Set `DOCKER_LOG_DIR` if Docker's data root differs from `/var/lib/docker/containers`,
and `NGINX_LOG_DIR` if needed. The collector mounts these directories read-only;
it has no Docker socket, host PID namespace or ability to restart applications.
Start with `docker compose -f deploy/logging/compose.yml up -d`.
7. Tunnel Grafana: `ssh -L 3300:127.0.0.1:3300 your-logging-host`. Open
`http://127.0.0.1:3300`, sign in, and open **Anonymous GitHub operations**.
Grafana is the only published port, bound to loopback. Ingest, query,
Prometheus and Alertmanager are private container-network services. Do not
publish their ports or put them behind the public application proxy.
8. Verify a synthetic request appears across API/streamer using its response
`X-Request-ID`, inspect all four streamer heartbeats and three queue heartbeats,
and inspect `ALERTS` in Grafana. Check Docker restart counts and all scrape
targets. Observe memory and ingestion drops before increasing traffic.
Images are version-and-digest pinned in `compose.yml`. Grafana installs the
signed VictoriaLogs datasource version 0.34.0 on startup; initial installation
needs outbound HTTPS. Automatic plugin upgrades are disabled. For an offline
host, preinstall that exact signed plugin in a staged image and validate it before
removing the plugin-download network. Never enable unsigned plugins.
## Privacy and correlation
`src/core/log-schema.ts` contains reviewed literal module, event and error codes.
Unknown messages become `application_log`; arbitrary text is not retained.
`log-record.ts` permits numeric measurements, fixed diagnostic values and UUIDs.
URLs, paths, repository IDs, owner IDs, cookies, headers, request/response bodies,
OAuth query parameters and stacks are omitted. Extend the application schema and
collector allowlist together when a new diagnostic is needed. Do not turn unknown
strings into labels. Raw Node fatal stderr and dependency output can still reach
short local Docker buffers; the collector never forwards their raw text.
Requests get server-created IDs at the API boundary. Existing API-to-streamer
header propagation is unchanged, including its validation. IDs are diagnostics,
not authentication or proof of origin. New download, removal and cache jobs carry the
request UUID and a random job UUID. Sandbox execution adds attempt and worker
service context. Older jobs get a new opaque job UUID
when executed. Their historical enqueue request cannot be reconstructed.
Only `service,severity` identify VictoriaLogs streams. Request, job, instance,
release and edge IDs are searchable fields, not stream or metric labels.
Nginx edge IDs link to application IDs; Cloudflare ray IDs are accepted only in a
fixed format. MongoDB contributes numeric event ID and severity, never `msg`,
`attr`, command or query text. Redis and backup output contributes presence only.
Malformed tagged records become a metadata event. Oversized lines are discarded
with collector error counters. There is no sampling of accepted records.
Central records omit IP addresses entirely, including unsalted IP hashes.
Restricted host Nginx buffers preserve IP evidence for a short incident window.
Do not attach those files to issues or PRs. Inspect them locally as an authorized
operator. Extending IP retention or exporting IPs needs a separate reviewed change.
## Budgets and failure behavior
VictoriaLogs has seven-day retention, a 10 GiB retention target and a 5 GiB free
space reserve. It deletes old daily partitions, so the target is not a hard quota:
current-day data, merges and indexes can exceed it. Put `/storage` on a dedicated
filesystem or apply a host filesystem quota for hard isolation. Reserve 20 GiB
for this stack and alert at 8 GiB used or under 8 GiB free. Keep application data
on a different filesystem for protection from a logging flood. At the free-space
reserve, ingest stops and Vector queues sanitized records.
Vector's persistent disk buffer is 256 MiB, followed by a 1,000-event memory
overflow buffer with `drop_newest`. The memory stage makes full-buffer losses
observable in Vector 0.61.0; its disk-only drop path omits fullness-loss counters.
The overflow stage is volatile and can deliver records out of order. Disk files and
checkpoints survive collector restarts. Allocation and metadata add overhead;
budget 512 MiB on disk. Outages cause retries with one in-flight request. Once
the buffer fills, new records are lost; applications do not wait. Vector syncs
the buffer periodically, so sudden power loss can lose the latest records.
Collector reads are not a transaction with Docker rotation: downtime beyond the
local buffers loses events. Delivery can duplicate records after a retry. There
is no exactly-once or host-failure durability claim.
The application drops stdout writes above 256 KiB pending, counts losses in its
heartbeat and handles a broken pipe without crashing. A broken stdout switches to bounded
stderr output so later worker heartbeats can still report drops. If both pipes
fail, that process cannot report its own telemetry. Docker's nonblocking
1 MiB per-container buffer can also drop independently, without an application
counter. Compare edge traffic against request completions during overload.
Each application container retains at most roughly 30 MiB of Docker files,
excluding an oversized single record and filesystem overhead. Logging-stack
containers use 5 MiB × 2 files each. Prometheus retains seven days or 512 MB of
blocks; WAL/head space needs additional headroom. Alertmanager/Grafana volumes
also need space. No compression ratio is promised for production data.
Admin error pages keep their existing Redis ring and counters by default. Their
payloads now contain safe structured diagnostics, so private URLs and stacks are
no longer available there. The mirror has no offline queue and a bounded command
queue. Once operators use Grafana, `LOG_REDIS_MIRROR=off` removes per-warning
Redis writes and ring duplication. This deliberately stops new admin error
history; existing history remains until normal expiry/overwrite or an approved
cleanup. Leave it on if that admin feature is required. Request/performance
metrics and queue admin functions are unchanged.
## Investigation
Dashboard panels separate first-byte delay, transfer time and named stages;
show client cancellations separately; compare 5xx and latency by release; and
show queue waiting age, job failures, memory and process starts. Paste a response
UUID into Request UUID, or a job UUID into Job UUID. Use the edge panel to find
the application ID for an edge request, then paste that ID into Request UUID.
Useful LogsQL queries, with the Grafana time picker limiting scope:
```text
requestId:="REQUEST_UUID"
jobId:="JOB_UUID"
event:process_fatal_error OR event:process_started
errorCode:github_rate_limit_exceeded OR errorCode:github_app_reconnect_required
service:nginx status:429 | stats by (route, limit_req_status) count() as requests
service:api outcome:completed | stats by (release) quantile(0.95, firstByteMs) as p95_ms
module:github-app-webhook | stats by (phase, upstreamStatus) count() as failures
```
The alert rules distinguish completed server failures from cancellations and
expected 4xx. 5xx require both a minimum count and a ratio sustained for five
minutes. Rate-limited traffic stays visible without paging. Queue age ignores
intentional delayed jobs. Telemetry alerts expect one API, four streamers and
three queues; update the streamer threshold during planned scaling. Memory
pressure uses cgroup usage including workers/page cache, with process RSS retained
for diagnosis. Missing telemetry is necessary because dead processes cannot emit
a final error. A logging-host outage also defeats same-host alerts; independent
external monitoring must be configured separately. No receiver is enabled here.
## Separate logging host and rollback
For host-failure durability, run VictoriaLogs, Grafana, Prometheus, vmalert and
Alertmanager on a separate host with encrypted backups of their persistent volumes.
Keep Vector and its local spool on the application host. Change the Vector sink
endpoint to a TLS-authenticated private reverse proxy or a restricted VPN address;
permit only collector ingress, never public ingestion. Use certificate files or
Docker secrets for credentials, not committed values. Scrape Vector over that
same private network. This repository's same-host Compose does not claim to set
up that network, certificates, backup policy or authentication proxy for you.
Test the remote arrangement with synthetic records and a network outage before
cutover. Backups must preserve the same access restrictions as live logs.
Rollback the application image and remove `app.compose.yml` from the deployment
command, then recreate application containers using the prior logging settings.
Stop the logging stack with `docker compose -f deploy/logging/compose.yml down`
without `-v`; that preserves logs, spool, dashboards and alert state. Restore the
prior Nginx logging configuration through the host workflow and test before
reloading. Keep any retained data private until its approved deletion date.
Re-enable `LOG_REDIS_MIRROR=on` if it was disabled. None of these steps requires
changing MongoDB data or application queues. Do not delete volumes as rollback.
## Local validation and references
Run `npm ci`, `npx tsc --noEmit`, and
`LOG_REDIS_MIRROR=off npx mocha test/structured-logging.test.js test/request-monitoring.test.js`.
Run `python3 scripts/test-logging.py --stress` with Docker Compose and PyYAML installed.
It creates a random project and synthetic files, opens temporary loopback test
ports, and removes only its own containers/volumes. It tests actual ingestion,
privacy, correlation, query syntax, rotation, restart, backend outage and recovery,
retention rejection and fixture storage size. It never reads production logs or
uses the application `.env`. The optional `--stress` writes 500,000 synthetic records while the backend is
stopped, checks the buffer bound and collector liveness, and waits for the real
CollectorDrops alert to fire. Allow 1 GiB temporary disk and up to 15 minutes with the 0.5 CPU collector limit.
These fixtures do not establish sustained production throughput.
Local results on 2026-11-14: application build, TypeScript and lint passed. The full
suite had 1,656 passing, 441 skipped and five failures also reproduced on base
`b456ab8`: one missing `cache_miss` translation, three activity-date mocks missing
`hint()`, and one existing Vue split-preview expectation. A focused logging and
job-reconnection run passed 34 tests. Skipped database tests were not executed.
The synthetic 10,000-record batch contained 2,070,000 raw JSON bytes; the entire
VictoriaLogs fixture store occupied about 357–358 KB of data and indexes,
including earlier records. This is a fixture measurement, not a production ratio.
The backend-outage stress run submitted 500,000 records, observed 136,532,570
buffered bytes without collector restart/OOM, and verified that `CollectorDrops`
reached firing after its sustained two-minute threshold. The overflow stage
reported its losses. A separate real query verified terminal worker drop counts
in the dashboard.
Official references checked 2026-11-14:
- [VictoriaLogs retention and resource flags](https://docs.victoriametrics.com/victorialogs/)
- [Vector ingestion setup](https://docs.victoriametrics.com/victorialogs/data-ingestion/vector/)
- [Vector buffering guarantees and loss behavior](https://vector.dev/docs/architecture/buffering-model/)
- [VictoriaLogs metrics](https://docs.victoriametrics.com/victorialogs/metrics/)
- [vmalert LogsQL integration](https://docs.victoriametrics.com/victorialogs/vmalert/)
- [Grafana plugin provisioning](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/)
- [Pinned Vector buffer sender accounting](https://github.com/vectordotdev/vector/blob/v0.61.0/lib/vector-buffers/src/topology/channel/sender.rs)
## Admin Errors integration
The existing Errors page has a central investigation section and a separate
Redis history section. The application starts and serves normally without
VictoriaLogs. Only authenticated admins can invoke the lookup endpoints; ordinary
request handlers never query logging services. With `ADMIN_LOG_URL` unset, lookup
is disabled and Redis remains available by default. `LOG_REDIS_MIRROR=off` is
shown as collection off, even when old Redis rows remain readable.
To enable lookup after approval, attach only the API to the logging stack's
private network with the optional `admin.compose.yml` override. Set
`LOGGING_NETWORK` to that existing network name, for example `logging_private`
when the logging Compose project is named `logging`. Do not publish query or
collector ports. The override sets `ADMIN_LOG_URL=http://victorialogs:9428` and
`ADMIN_LOG_VECTOR_URL=http://vector:9598`. Both are operator-controlled origins,
without paths, query strings or embedded credentials; no browser can choose an
endpoint. A separate logging host must use a restricted private network or VPN.
An authenticated proxy must provide its own operator-managed access mechanism;
this client deliberately does not forward user cookies or authorization headers.
Optionally set `ADMIN_LOG_GRAFANA_URL` to the private dashboard URL, for example
`https://grafana.internal/d/anonymous-operations`. It is an outbound operator link,
not a replacement for the in-page timeline. The application removes configured
query strings before adding bounded time and opaque correlation variables.
Never put a secret in any of these URL settings.
Admins can select 15 minutes, one hour, six hours or 24 hours, a fixed service,
and an exact release SHA. The server fetches at most 1,001 recent warning/error
records, including Nginx edge responses with status 500–599 even when their
recorded severity is info, and shows at most 1,000. Successful edge responses and
429 rate limits are not server failures; their records remain available through
correlated timelines. The unknown release filter matches both missing fields and
explicit `unknown` values for compatibility with existing collected data. Groups count those returned records, with safe
fingerprints based on service/module/event/error code/severity/status, independent
of request/job IDs and release. First/last occurrence applies only to that bounded
snapshot. A limit or malformed record produces partial coverage, not a complete
count. These counts are never added to Redis counters or mirrored rows.
Click a request or job ID for at most 200 earliest matching events across services
in the selected window. Job lookups can additionally follow up to four associated
request UUIDs. IDs are diagnostics, not authorization. The server accepts only
fixed templates, UUIDs and enumerated filters, rejects unknown parameters,
rejects backend redirects, caps each response at 2 MiB, times out the entire
lookup after four seconds, and allows two concurrent investigations per API
process. No arbitrary LogsQL is accepted. Backend records and legacy Redis rows
are projected through the safe application schema before reaching the browser;
URLs, request bodies, private identities and IPs are omitted.
Health is separate from error counts. Heartbeats use an independent five-minute
bounded scan and a 90-second freshness threshold, expecting one API, four
streamers and three queues. Missing data, stale telemetry, truncation, disabled
configuration, invalid configuration, a busy reader and an unreachable backend
have distinct states. Collector counters exclude intentional transform filtering
but include full-buffer drops, and are explicitly unknown if collector metrics
are disabled/unreachable. Application drops are per-process cumulative values
since startup, not window deltas; collector counters reset on collector restart.
A successful empty error query alone never means telemetry is healthy. Central
queries run only on page entry or explicit Apply / refresh; the UI marks a
snapshot older than 90 seconds stale without polling the backend. Redis refresh
is separate. Clear and CSV export actions apply only to Redis history.
Rollback this integration by removing `admin.compose.yml` from the application
Compose invocation and unsetting the three `ADMIN_LOG_*` variables, then recreate
only the API through the existing workflow. This does not delete central data or
change collection. Keep the infrastructure running independently if desired.
Verify disabled lookup and Redis availability locally before applying changes.
Local verification for the admin integration:
```sh
npm run build
LOG_REDIS_MIRROR=off npx mocha test/admin-log-investigation.test.js
LOG_REDIS_MIRROR=off node scripts/test-admin-logs.js
```
The last command starts a temporary loopback-only VictoriaLogs container from the
pinned image, runs synthetic fixtures through the pinned Vector collector program,
and verifies grouping/privacy/correlation and historical release/edge compatibility,
stops the backend and verifies unavailable/disabled responses, then removes its
container. It does not read `.env`, production data or send notifications. UI
regressions are in `test/vue-ui.test.js`; rebuild the UI before running them.
The Redis admin reader admits at most four concurrent history/statistics operations,
including the overview's error counters. Excess history requests receive a busy
response; overview error counters retain their existing fallback. Its 256-command queue fits four complete 51-command statistics
transactions with headroom. Concurrent first requests share a connection attempt;
no offline command backlog or automatic reconnect loop is enabled. Admission
slots remain held until Redis work finishes, even if the HTTP client disconnects.
Run `node scripts/test-admin-redis.js` to exercise the actual admin routes and
node-redis client against a disposable loopback-only Redis container. It checks
cold concurrent statistics, recent/grouped-history reads, overload rejection and
subsequent transaction integrity, unavailable fallback, and restart recovery.
The fixture uses only synthetic counters and removes its container on exit.
## Safe exception diagnostics
New application errors retain the normalized route template, HTTP method/status,
a finite error type/code, and up to eight recognized application frames (1 KiB maximum per stack) for the
exception and the deepest inspected cause (at most three levels, cycles stopped). Raw messages,
Upstream URLs, query strings, external frames and absolute filesystem roots are
not stored. Repository/file parameters are excluded except for the explicitly
scoped anonymous-request reproduction context below. Unknown error types use `UnknownError`; unknown
function names become `<anonymous>`. Source locations are build-relative for
compiled code and source-relative in development; line numbers refer to that
release, not an inferred source map.
`scripts/generate-log-diagnostics.js` derives the finite project file/function
allowlist from checked-in TypeScript and writes both application and collector
validation. Regenerate it when adding diagnostic code locations. The admin API
revalidates every field for both Redis and central logs. Deploy matching app and
collector versions to preserve new fields; older collectors deliberately drop
unknown fields. Existing discarded exception details cannot be recovered.
### Admin reproduction context
Application requests to the supported anonymous repository endpoints retain their public anonymous repository ID and encoded request pathname, up to 2 KiB, under the existing admin log access controls. This is an explicit exception to template-only diagnostics: repository-relative filenames are retained for reproduction. Query strings, headers, bodies and upstream owner/repository URLs are still excluded. Filenames are operator-visible data under this policy; no generic filter can guarantee that a user has not put sensitive text in a filename. Capture includes successful, failed and interrupted requests; metrics still use only normalized routes. Supported paths are repository metadata, file contents, listings/counts/search, ZIP, options, refresh, coauthors and extend under `/api/repo/:repoId`; other endpoints and unsafe, oversized, malformed or traversal paths omit reproduction context. Listing capture is limited to the actual `/files`, `/files/counts` and `/files/search` endpoints; folder/search query values are not retained. Action paths refresh, extend and coauthors require POST; GET/HEAD source metadata lookups using those repository names are excluded. Missing or unsupported methods omit reproduction context. Source lookups such as `/api/repo/:owner/files/readme` are excluded. Historical records cannot be backfilled. The UI copies the exact captured pathname without a host or query.
For the focused restricted-file/rotation regression, run
`LOGGING_ACCESS_ONLY=1 python3 scripts/test-logging.py`. This uses synthetic
`0640` logs in an isolated Docker project and preserves capability restrictions.