diff --git a/deploy/logging/README.md b/deploy/logging/README.md new file mode 100644 index 0000000..e2dff54 --- /dev/null +++ b/deploy/logging/README.md @@ -0,0 +1,208 @@ +# Operations logging + +This is an opt-in stack for one Docker host. It does not deploy itself, change +production, or send notifications. The application writes structured JSON to +stdout. Vector tails tagged Docker json-file logs and host Nginx access JSON, +rebuilds each record from an allowlist, then spools sanitized records to disk. +VictoriaLogs stores searchable compressed records. Grafana provides investigation +panels. Prometheus monitors the logging components and vmalert evaluates LogsQL +alerts. Alertmanager groups firing and resolved state with an empty receiver. + +## Deploy after approval + +Review the application and infrastructure changes together. This PR is stacked +on `fix/remaining-production-findings`; do not deploy it without that base. + +1. Reserve at least 20 GiB of dedicated logging storage and 2 GiB of RAM, outside + the application and MongoDB budgets. The configured memory ceilings total + 1,856 MiB. Check real free memory before enabling this stack. Docker volumes + survive container replacement, but not deletion of the volumes or host loss. +2. Create a strong Grafana password in an external secret file. Give UID 472 read + access and restrict the parent directory to operators. Do not place the file + in the repository. Export `GRAFANA_PASSWORD_FILE` as its absolute path and + `GRAFANA_ADMIN_USER` as a non-default username. No password value belongs in + a shell command, Compose file, issue, or PR. +3. Set `APP_RELEASE` to the deployed 7 to 40 character Git SHA. Build the app through + the existing deployment workflow. Add the override when recreating services: + `docker compose -f docker-compose.yml -f deploy/logging/app.compose.yml up -d`. + This applies explicit service tags and nonblocking 10 MiB × 3 local buffers to + the API, all four streamers, MongoDB, Redis and the backup process. BullMQ + sandbox workers log through the API container. Existing containers need + recreation to change their Docker log configuration. +4. Include `../nginx/anonymous-logging.conf` in the host Nginx `http` context and + configure the relevant server with + `access_log /var/log/nginx/anonymous.access.json anonymous_json;`. + Test with `nginx -t`, then reload through the existing deployment workflow. + Keep Nginx `real_ip` and proxy trust unchanged. Do not enable Mongo profiling, + query tracing, Redis MONITOR, or request-body logging. +5. Install `nginx.logrotate` through the host configuration workflow and run it + hourly with its own state file. Keep logs root/adm-readable only. This retains + about six hourly files, with a 10 MiB rotation trigger checked hourly. A burst + can exceed 10 MiB between checks; this is not a hard file quota. Use a dedicated + filesystem quota if hard local containment is needed. Use rename/reopen, not + copytruncate. Vector excludes compressed history and reads new files from end. +6. Set `DOCKER_LOG_DIR` if Docker's data root differs from `/var/lib/docker/containers`, + and `NGINX_LOG_DIR` if needed. The collector mounts these directories read-only; + it has no Docker socket, host PID namespace or ability to restart applications. + Start with `docker compose -f deploy/logging/compose.yml up -d`. +7. Tunnel Grafana: `ssh -L 3300:127.0.0.1:3300 your-logging-host`. Open + `http://127.0.0.1:3300`, sign in, and open **Anonymous GitHub operations**. + Grafana is the only published port, bound to loopback. Ingest, query, + Prometheus and Alertmanager are private container-network services. Do not + publish their ports or put them behind the public application proxy. +8. Verify a synthetic request appears across API/streamer using its response + `X-Request-ID`, inspect all four streamer heartbeats and three queue heartbeats, + and inspect `ALERTS` in Grafana. Check Docker restart counts and all scrape + targets. Observe memory and ingestion drops before increasing traffic. + +Images are version-and-digest pinned in `compose.yml`. Grafana installs the +signed VictoriaLogs datasource version 0.34.0 on startup; initial installation +needs outbound HTTPS. Automatic plugin upgrades are disabled. For an offline +host, preinstall that exact signed plugin in a staged image and validate it before +removing the plugin-download network. Never enable unsigned plugins. + +## Privacy and correlation + +`src/core/log-schema.ts` contains reviewed literal module, event and error codes. +Unknown messages become `application_log`; arbitrary text is not retained. +`log-record.ts` permits numeric measurements, fixed diagnostic values and UUIDs. +URLs, paths, repository IDs, owner IDs, cookies, headers, request/response bodies, +OAuth query parameters and stacks are omitted. Extend the application schema and +collector allowlist together when a new diagnostic is needed. Do not turn unknown +strings into labels. Raw Node fatal stderr and dependency output can still reach +short local Docker buffers; the collector never forwards their raw text. + +Requests get server-created IDs at the API boundary. Existing API-to-streamer +header propagation is unchanged, including its validation. IDs are diagnostics, +not authentication or proof of origin. New download, removal and cache jobs carry the +request UUID and a random job UUID. Sandbox execution adds attempt and worker +service context. Older jobs get a new opaque job UUID +when executed. Their historical enqueue request cannot be reconstructed. + +Only `service,severity` identify VictoriaLogs streams. Request, job, instance, +release and edge IDs are searchable fields, not stream or metric labels. +Nginx edge IDs link to application IDs; Cloudflare ray IDs are accepted only in a +fixed format. MongoDB contributes numeric event ID and severity, never `msg`, +`attr`, command or query text. Redis and backup output contributes presence only. +Malformed tagged records become a metadata event. Oversized lines are discarded +with collector error counters. There is no sampling of accepted records. + +Central records omit IP addresses entirely, including unsalted IP hashes. +Restricted host Nginx buffers preserve IP evidence for a short incident window. +Do not attach those files to issues or PRs. Inspect them locally as an authorized +operator. Extending IP retention or exporting IPs needs a separate reviewed change. + +## Budgets and failure behavior + +VictoriaLogs has seven-day retention, a 10 GiB retention target and a 5 GiB free +space reserve. It deletes old daily partitions, so the target is not a hard quota: +current-day data, merges and indexes can exceed it. Put `/storage` on a dedicated +filesystem or apply a host filesystem quota for hard isolation. Reserve 20 GiB +for this stack and alert at 8 GiB used or under 8 GiB free. Keep application data +on a different filesystem for protection from a logging flood. At the free-space +reserve, ingest stops and Vector queues sanitized records. + +Vector's persistent disk buffer is 256 MiB, with `drop_newest`. Its files and +checkpoints survive collector restarts. Allocation and metadata add overhead; +budget 512 MiB on disk. Outages cause retries with one in-flight request. Once +the buffer fills, new records are lost; applications do not wait. Vector syncs +the buffer periodically, so sudden power loss can lose the latest records. +Collector reads are not a transaction with Docker rotation: downtime beyond the +local buffers loses events. Delivery can duplicate records after a retry. There +is no exactly-once or host-failure durability claim. + +The application drops stdout writes above 256 KiB pending, counts losses in its +heartbeat and handles a broken pipe without crashing. A broken stdout switches to bounded +stderr output so later worker heartbeats can still report drops. If both pipes +fail, that process cannot report its own telemetry. Docker's nonblocking +1 MiB per-container buffer can also drop independently, without an application +counter. Compare edge traffic against request completions during overload. +Each application container retains at most roughly 30 MiB of Docker files, +excluding an oversized single record and filesystem overhead. Logging-stack +containers use 5 MiB × 2 files each. Prometheus retains seven days or 512 MB of +blocks; WAL/head space needs additional headroom. Alertmanager/Grafana volumes +also need space. No compression ratio is promised for production data. + +Admin error pages keep their existing Redis ring and counters by default. Their +payloads now contain safe structured diagnostics, so private URLs and stacks are +no longer available there. The mirror has no offline queue and a bounded command +queue. Once operators use Grafana, `LOG_REDIS_MIRROR=off` removes per-warning +Redis writes and ring duplication. This deliberately stops new admin error +history; existing history remains until normal expiry/overwrite or an approved +cleanup. Leave it on if that admin feature is required. Request/performance +metrics and queue admin functions are unchanged. + +## Investigation + +Dashboard panels separate first-byte delay, transfer time and named stages; +show client cancellations separately; compare 5xx and latency by release; and +show queue waiting age, job failures, memory and process starts. Paste a response +UUID into Request UUID, or a job UUID into Job UUID. Use the edge panel to find +the application ID for an edge request, then paste that ID into Request UUID. + +Useful LogsQL queries, with the Grafana time picker limiting scope: + +```text +requestId:="REQUEST_UUID" +jobId:="JOB_UUID" +event:process_fatal_error OR event:process_started +errorCode:github_rate_limit_exceeded OR errorCode:github_app_reconnect_required +service:nginx status:429 | stats by (route, limit_req_status) count() as requests +service:api outcome:completed | stats by (release) quantile(0.95, firstByteMs) as p95_ms +module:github-app-webhook | stats by (phase, upstreamStatus) count() as failures +``` + +The alert rules distinguish completed server failures from cancellations and +expected 4xx. 5xx require both a minimum count and a ratio sustained for five +minutes. Rate-limited traffic stays visible without paging. Queue age ignores +intentional delayed jobs. Telemetry alerts expect one API, four streamers and +three queues; update the streamer threshold during planned scaling. Memory +pressure uses cgroup usage including workers/page cache, with process RSS retained +for diagnosis. Missing telemetry is necessary because dead processes cannot emit +a final error. A logging-host outage also defeats same-host alerts; independent +external monitoring must be configured separately. No receiver is enabled here. + +## Separate logging host and rollback + +For host-failure durability, run VictoriaLogs, Grafana, Prometheus, vmalert and +Alertmanager on a separate host with encrypted backups of their persistent volumes. +Keep Vector and its local spool on the application host. Change the Vector sink +endpoint to a TLS-authenticated private reverse proxy or a restricted VPN address; +permit only collector ingress, never public ingestion. Use certificate files or +Docker secrets for credentials, not committed values. Scrape Vector over that +same private network. This repository's same-host Compose does not claim to set +up that network, certificates, backup policy or authentication proxy for you. +Test the remote arrangement with synthetic records and a network outage before +cutover. Backups must preserve the same access restrictions as live logs. + +Rollback the application image and remove `app.compose.yml` from the deployment +command, then recreate application containers using the prior logging settings. +Stop the logging stack with `docker compose -f deploy/logging/compose.yml down` +without `-v`; that preserves logs, spool, dashboards and alert state. Restore the +prior Nginx logging configuration through the host workflow and test before +reloading. Keep any retained data private until its approved deletion date. +Re-enable `LOG_REDIS_MIRROR=on` if it was disabled. None of these steps requires +changing MongoDB data or application queues. Do not delete volumes as rollback. + +## Local validation and references + +Run `npm ci`, `npx tsc --noEmit`, and +`LOG_REDIS_MIRROR=off npx mocha test/structured-logging.test.js test/request-monitoring.test.js`. +Run `python3 scripts/test-logging.py --stress` with Docker Compose and PyYAML installed. +It creates a random project and synthetic files, opens temporary loopback test +ports, and removes only its own containers/volumes. It tests actual ingestion, +privacy, correlation, query syntax, rotation, restart, backend outage and recovery, +retention rejection and fixture storage size. It never reads production logs or +uses the application `.env`. The optional `--stress` writes 500,000 synthetic records while the backend is +stopped, checks the buffer bound and collector liveness, and waits for the real +CollectorDrops alert to fire. Allow 1 GiB temporary disk and several minutes. +These fixtures do not establish sustained production throughput. + +Official references checked 2026-11-14: + +- [VictoriaLogs retention and resource flags](https://docs.victoriametrics.com/victorialogs/) +- [Vector ingestion setup](https://docs.victoriametrics.com/victorialogs/data-ingestion/vector/) +- [Vector buffering guarantees and loss behavior](https://vector.dev/docs/architecture/buffering-model/) +- [VictoriaLogs metrics](https://docs.victoriametrics.com/victorialogs/metrics/) +- [vmalert LogsQL integration](https://docs.victoriametrics.com/victorialogs/vmalert/) +- [Grafana plugin provisioning](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/) diff --git a/deploy/logging/alertmanager.yml b/deploy/logging/alertmanager.yml new file mode 100644 index 0000000..7621f6e --- /dev/null +++ b/deploy/logging/alertmanager.yml @@ -0,0 +1,10 @@ +route: + receiver: local-only + group_by: [alertname, service, queue] + group_wait: 1m + group_interval: 5m + repeat_interval: 4h +receivers: + - name: local-only +# Deliberately no email, webhook, chat or pager receiver. Firing/resolved state +# remains available in the private UI. Add a receiver only after approval. diff --git a/deploy/logging/app.compose.yml b/deploy/logging/app.compose.yml new file mode 100644 index 0000000..ef5e157 --- /dev/null +++ b/deploy/logging/app.compose.yml @@ -0,0 +1,35 @@ +# Add after docker-compose.yml. Recreating containers is a deployment action. +x-buffer: &buffer + driver: json-file + options: &options + max-size: '10m' + max-file: '3' + mode: non-blocking + max-buffer-size: '1m' +services: + anonymous_github: + logging: + <<: *buffer + options: {<<: *options, tag: api} + environment: + APP_RELEASE: ${APP_RELEASE:-unknown} + LOG_REDIS_MIRROR: ${LOG_REDIS_MIRROR:-on} + streamer: + logging: + <<: *buffer + options: {<<: *options, tag: streamer} + environment: + APP_RELEASE: ${APP_RELEASE:-unknown} + LOG_REDIS_MIRROR: ${LOG_REDIS_MIRROR:-on} + mongodb: + logging: + <<: *buffer + options: {<<: *options, tag: mongodb} + redis: + logging: + <<: *buffer + options: {<<: *options, tag: redis} + mongodb-backup: + logging: + <<: *buffer + options: {<<: *options, tag: mongodb-backup} diff --git a/deploy/logging/compose.yml b/deploy/logging/compose.yml new file mode 100644 index 0000000..e1bee46 --- /dev/null +++ b/deploy/logging/compose.yml @@ -0,0 +1,94 @@ +name: anonymous-observability +x-common: &common + restart: unless-stopped + logging: + driver: json-file + options: {max-size: 5m, max-file: '2', mode: non-blocking, max-buffer-size: 1m} + security_opt: [no-new-privileges:true] + cap_drop: [ALL] +services: + victorialogs: + <<: *common + image: victoriametrics/victoria-logs:v1.54.0@sha256:ee4e820cac3721f5f5d91641b950c22439593adc4224f9655e5333d5a84c1dfe + command: + - -storageDataPath=/storage + - -retentionPeriod=7d + - -retention.maxDiskSpaceUsageBytes=10GiB + - -storage.minFreeDiskSpaceBytes=5GiB + - -memory.allowedBytes=512MiB + volumes: [logs:/storage] + mem_limit: 768m + cpus: 1.0 + networks: [private] + vector: + <<: *common + image: timberio/vector:0.61.0-debian@sha256:f6c23b5a7e16fb2838d33619ef7acf4bad2289a7cf22bc90efc2371f520c4ea2 + command: [--config, /etc/vector/vector.yaml] + # File access only: no Docker API socket and no host PID namespace. + user: '0:0' + volumes: + - ./vector.yaml:/etc/vector/vector.yaml:ro + - ./sanitize.vrl:/etc/vector/sanitize.vrl:ro + - ${DOCKER_LOG_DIR:-/var/lib/docker/containers}:/host/docker:ro + - ${NGINX_LOG_DIR:-/var/log/nginx}:/host/nginx:ro + - spool:/var/lib/vector + environment: + VECTOR_LOG: warn + mem_limit: 256m + cpus: 0.5 + networks: [private] + grafana: + <<: *common + image: grafana/grafana:13.3.1@sha256:0615ec4fbd4072004b7d86b46ded63013a74b7d6eb1119273dd89e8fa25c13cf + ports: ['127.0.0.1:${GRAFANA_PORT:-3300}:3000'] + environment: + GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?Set a non-default admin username} + GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_password + GF_USERS_ALLOW_SIGN_UP: 'false' + GF_AUTH_ANONYMOUS_ENABLED: 'false' + GF_ANALYTICS_REPORTING_ENABLED: 'false' + GF_ANALYTICS_CHECK_FOR_UPDATES: 'false' + GF_PLUGINS_PREINSTALL_AUTO_UPDATE: 'false' + GF_PLUGINS_DISABLE_PLUGINS: grafana-lokiexplore-app,grafana-metricsdrilldown-app,grafana-exploretraces-app,grafana-pyroscope-app,grafana-pathfinder-app + GF_PLUGINS_PREINSTALL_SYNC: victoriametrics-logs-datasource@0.34.0 + secrets: [grafana_password] + volumes: + - grafana:/var/lib/grafana + - ./grafana/provisioning:/etc/grafana/provisioning:ro + - ./grafana/dashboards:/var/lib/grafana/dashboards:ro + mem_limit: 384m + networks: [private, plugin_download] + prometheus: + <<: *common + image: prom/prometheus:v3.16.0@sha256:e04987641b57704f408e31f6373ba4138ecf27cb4a165d24aabb6c38fddd0f06 + command: [--config.file=/etc/prometheus/prometheus.yml, --storage.tsdb.path=/prometheus, --storage.tsdb.retention.time=7d, --storage.tsdb.retention.size=512MB, --web.enable-remote-write-receiver] + volumes: [metrics:/prometheus, './prometheus.yml:/etc/prometheus/prometheus.yml:ro', './metrics-alerts.yml:/etc/prometheus/alerts.yml:ro'] + mem_limit: 256m + networks: [private] + vmalert: + <<: *common + image: victoriametrics/vmalert:v1.156.0@sha256:4fea7d1a888d4db4ff18804434ef0f3ca8eb6482e289c50d32001c2db218287c + command: [-rule=/etc/alerts.yml, -datasource.url=http://victorialogs:9428, -notifier.url=http://alertmanager:9093, -remoteWrite.url=http://prometheus:9090, -remoteRead.url=http://prometheus:9090] + volumes: ['./log-alerts.yml:/etc/alerts.yml:ro'] + mem_limit: 128m + networks: [private] + alertmanager: + <<: *common + image: prom/alertmanager:v0.35.0@sha256:e638797febd034838eca8b98cdbd46f951e9bf7f178777aa8669167406a9946c + command: [--config.file=/etc/alertmanager.yml, --storage.path=/alertmanager, --cluster.listen-address=] + volumes: [alerts:/alertmanager, './alertmanager.yml:/etc/alertmanager.yml:ro'] + mem_limit: 64m + networks: [private] +secrets: + grafana_password: + file: ${GRAFANA_PASSWORD_FILE:?Set path to an existing secret file} +networks: + private: + internal: true + plugin_download: {} +volumes: + logs: {} + spool: {} + grafana: {} + metrics: {} + alerts: {} diff --git a/deploy/logging/grafana/dashboards/operations.json b/deploy/logging/grafana/dashboards/operations.json new file mode 100644 index 0000000..19b7057 --- /dev/null +++ b/deploy/logging/grafana/dashboards/operations.json @@ -0,0 +1,467 @@ +{ + "uid": "anonymous-operations", + "title": "Anonymous GitHub operations", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": { + "from": "now-1h", + "to": "now" + }, + "tags": [ + "operations" + ], + "templating": { + "list": [ + { + "name": "request_id", + "label": "Request UUID", + "type": "textbox", + "current": { + "text": "", + "value": "" + } + }, + { + "name": "job_id", + "label": "Job UUID", + "type": "textbox", + "current": { + "text": "", + "value": "" + } + }, + { + "name": "edge_id", + "label": "Edge request ID", + "type": "textbox", + "current": { + "text": "", + "value": "" + } + } + ] + }, + "panels": [ + { + "id": 1, + "title": "Failures and cancellations", + "type": "logs", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 0, + "y": 0, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "severity:error OR event:request_cancelled", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 2, + "title": "First byte versus transfer time", + "type": "table", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 12, + "y": 0, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "event:request* | stats by (service, route) quantile(0.95, firstByteMs) as first_byte_p95_ms, quantile(0.95, responseMs) as transfer_p95_ms", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 3, + "title": "Request stages", + "type": "table", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 0, + "y": 8, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "event:request* | stats by (service) avg(stage_upstream_ms) as upstream_ms, avg(stage_worker_wait_ms) as worker_wait_ms, avg(stage_anonymize_ms) as anonymize_ms", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 4, + "title": "Traffic and rate limits", + "type": "table", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 12, + "y": 8, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "service:nginx | stats by (status, limit_req_status) count() as requests", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 5, + "title": "Authentication versus upstream failures", + "type": "table", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 0, + "y": 16, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "errorCode:* OR status:401 OR status:403 OR status:>=500 | stats by (service, errorCode, status) count() as events", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 6, + "title": "Release comparison", + "type": "table", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 12, + "y": 16, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "service:api outcome:completed | stats by (release) count() as requests, count() if (status:>=500) as server_errors, quantile(0.95, firstByteMs) as first_byte_p95_ms", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 7, + "title": "Crashes and restarts", + "type": "logs", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 0, + "y": 24, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "event:process_fatal_error OR event:process_started", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 8, + "title": "Queue stalls and failures", + "type": "logs", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 12, + "y": 24, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "event:job_stalled OR event:job_failed OR event:queue_heartbeat", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 9, + "title": "Memory and lost application logs", + "type": "table", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 0, + "y": 32, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "event:process_heartbeat | stats by (service, instance) max(rss) as rss, max(memoryRatio) as memory_ratio, max(droppedLogs) as dropped_logs", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 10, + "title": "Request correlation", + "type": "logs", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 12, + "y": 32, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "requestId:=\"$request_id\"", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 11, + "title": "Job correlation", + "type": "logs", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 0, + "y": 40, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "jobId:=\"$job_id\"", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 12, + "title": "Edge to application correlation", + "type": "logs", + "datasource": { + "type": "victoriametrics-logs-datasource", + "uid": "anonymous-logs" + }, + "gridPos": { + "x": 12, + "y": 40, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "edge_request_id:=\"$edge_id\" OR requestId:=\"$request_id\"", + "queryType": "instant", + "editorMode": "code", + "maxLines": 200 + } + ], + "options": { + "showTime": true, + "sortOrder": "Descending" + } + }, + { + "id": 20, + "title": "Storage bytes", + "type": "timeseries", + "datasource": { + "type": "prometheus", + "uid": "logging-health" + }, + "gridPos": { + "x": 0, + "y": 48, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "sum(vl_data_size_bytes)" + } + ] + }, + { + "id": 21, + "title": "Free disk bytes", + "type": "timeseries", + "datasource": { + "type": "prometheus", + "uid": "logging-health" + }, + "gridPos": { + "x": 12, + "y": 48, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "min(vl_free_disk_space_bytes)" + } + ] + }, + { + "id": 22, + "title": "Collector discarded events", + "type": "timeseries", + "datasource": { + "type": "prometheus", + "uid": "logging-health" + }, + "gridPos": { + "x": 0, + "y": 56, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "(sum(rate(vector_buffer_discarded_events_total[5m])) or vector(0)) + (sum(rate(vector_component_discarded_events_total{intentional!=\"true\"}[5m])) or vector(0))" + } + ] + }, + { + "id": 23, + "title": "Firing alerts", + "type": "timeseries", + "datasource": { + "type": "prometheus", + "uid": "logging-health" + }, + "gridPos": { + "x": 12, + "y": 56, + "w": 12, + "h": 8 + }, + "targets": [ + { + "refId": "A", + "expr": "ALERTS{alertstate=\"firing\"}" + } + ] + } + ] +} diff --git a/deploy/logging/grafana/provisioning/dashboards/logs.yml b/deploy/logging/grafana/provisioning/dashboards/logs.yml new file mode 100644 index 0000000..eb4d396 --- /dev/null +++ b/deploy/logging/grafana/provisioning/dashboards/logs.yml @@ -0,0 +1,6 @@ +apiVersion: 1 +providers: + - name: Anonymous GitHub operations + type: file + allowUiUpdates: false + options: {path: /var/lib/grafana/dashboards} diff --git a/deploy/logging/grafana/provisioning/datasources/logs.yml b/deploy/logging/grafana/provisioning/datasources/logs.yml new file mode 100644 index 0000000..ac83f59 --- /dev/null +++ b/deploy/logging/grafana/provisioning/datasources/logs.yml @@ -0,0 +1,22 @@ +apiVersion: 1 +datasources: + - name: Logs + uid: anonymous-logs + type: victoriametrics-logs-datasource + access: proxy + url: http://victorialogs:9428 + isDefault: true + editable: false + - name: Logging health + uid: logging-health + type: prometheus + access: proxy + url: http://prometheus:9090 + editable: false + - name: Alert state + uid: logging-alerts + type: alertmanager + access: proxy + url: http://alertmanager:9093 + jsonData: {implementation: prometheus} + editable: false diff --git a/deploy/logging/log-alerts.yml b/deploy/logging/log-alerts.yml new file mode 100644 index 0000000..a06f814 --- /dev/null +++ b/deploy/logging/log-alerts.yml @@ -0,0 +1,55 @@ +groups: + - name: application-logs + type: vlogs + interval: 30s + rules: + - alert: SustainedServerErrors + expr: '_time:5m service:api outcome:completed | stats count() as total, count() if (status:>=500) as errors | filter errors:>=10 | math errors / total as ratio | filter ratio:>0.05 | fields ratio' + for: 5m + labels: {severity: warning} + annotations: {summary: 'API 5xx exceed 5 percent and 10 requests per five minutes'} + - alert: ApiTelemetryMissing + expr: '_time:3m service:api event:process_heartbeat | stats count() as heartbeats | filter heartbeats:0' + for: 3m + labels: {severity: warning} + annotations: {summary: 'API heartbeat is missing'} + - alert: StreamerTelemetryMissing + expr: '_time:3m service:streamer event:process_heartbeat | stats count_uniq(instance) as instances | filter instances:<4' + for: 3m + labels: {severity: warning} + annotations: {summary: 'Fewer than four streamer instances report heartbeats; adjust for planned scaling'} + - alert: ApplicationRestartBurst + expr: '_time:10m event:process_started | stats by (service) count() as restarts | filter restarts:>4' + for: 2m + labels: {severity: warning} + annotations: {summary: 'Repeated process starts in ten minutes'} + - alert: ApplicationMemoryPressure + expr: '_time:3m event:process_heartbeat | stats by (service) max(memoryRatio) as ratio | filter ratio:>0.85' + for: 5m + labels: {severity: warning} + annotations: {summary: 'Container usage exceeds 85 percent of its cgroup memory limit'} + - alert: QueueStalled + expr: '_time:5m event:job_stalled | stats by (queue) count() as stalls | filter stalls:>0' + for: 2m + labels: {severity: warning} + annotations: {summary: 'BullMQ reports stalled jobs'} + - alert: QueueWaitingTooLong + expr: '_time:3m event:queue_heartbeat | stats by (queue) max(oldestWaitingMs) as age | filter age:>600000' + for: 5m + labels: {severity: warning} + annotations: {summary: 'A runnable job has waited over ten minutes'} + - alert: QueueTelemetryMissing + expr: '_time:3m event:queue_heartbeat | stats count_uniq(queue) as queues | filter queues:<3' + for: 3m + labels: {severity: warning} + annotations: {summary: 'Queue telemetry is missing'} + - alert: WebhookFailures + expr: '_time:5m route:"/github/app/webhook" status:>=500 outcome:completed | stats count() as errors | filter errors:>=3' + for: 3m + labels: {severity: warning} + annotations: {summary: 'Repeated webhook server failures; signature rejection is excluded'} + - alert: ApplicationLogDrops + expr: '_time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service) max(droppedLogs) as drops | filter drops:>0' + for: 2m + labels: {severity: warning} + annotations: {summary: 'An application process dropped stdout logs since startup'} diff --git a/deploy/logging/metrics-alerts.yml b/deploy/logging/metrics-alerts.yml new file mode 100644 index 0000000..df90025 --- /dev/null +++ b/deploy/logging/metrics-alerts.yml @@ -0,0 +1,33 @@ +groups: + - name: logging-infrastructure + rules: + - alert: LoggingTelemetryMissing + expr: up == 0 + for: 3m + labels: {severity: warning} + annotations: {summary: 'A logging component cannot be scraped'} + - alert: CollectorDrops + expr: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0 + for: 2m + labels: {severity: warning} + annotations: {summary: 'Collector dropped events; inspect buffer and transform counters'} + - alert: CollectorErrors + expr: sum(increase(vector_component_errors_total[5m])) > 0 + for: 3m + labels: {severity: warning} + annotations: {summary: 'Collector errors persist'} + - alert: LogStoragePressure + expr: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3 + for: 5m + labels: {severity: warning} + annotations: {summary: 'Log storage is near the retention budget or free-space reserve'} + - alert: LogIngestionRejected + expr: sum(increase(vl_rows_dropped_total[5m])) > 0 + for: 2m + labels: {severity: warning} + annotations: {summary: 'VictoriaLogs rejected records'} + - alert: LogAlertEvaluationFailed + expr: sum(increase(vmalert_execution_errors_total[5m])) > 0 + for: 3m + labels: {severity: warning} + annotations: {summary: 'Log alert queries are failing'} diff --git a/deploy/logging/nginx.logrotate b/deploy/logging/nginx.logrotate new file mode 100644 index 0000000..ae8812e --- /dev/null +++ b/deploy/logging/nginx.logrotate @@ -0,0 +1,15 @@ +# Run this configuration hourly, using a dedicated logrotate state file. +/var/log/nginx/anonymous.access.json { + hourly + rotate 6 + maxsize 10M + missingok + notifempty + compress + delaycompress + create 0640 www-data adm + sharedscripts + postrotate + /usr/sbin/nginx -s reopen + endscript +} diff --git a/deploy/logging/prometheus.yml b/deploy/logging/prometheus.yml new file mode 100644 index 0000000..458aab9 --- /dev/null +++ b/deploy/logging/prometheus.yml @@ -0,0 +1,21 @@ +global: + scrape_interval: 30s + evaluation_interval: 30s +rule_files: [/etc/prometheus/alerts.yml] +alerting: + alertmanagers: + - static_configs: + - targets: [alertmanager:9093] +scrape_configs: + - job_name: victorialogs + static_configs: + - targets: [victorialogs:9428] + - job_name: collector + static_configs: + - targets: [vector:9598] + - job_name: log-alerts + static_configs: + - targets: [vmalert:8880] + - job_name: prometheus + static_configs: + - targets: [localhost:9090] diff --git a/deploy/logging/sanitize.vrl b/deploy/logging/sanitize.vrl new file mode 100644 index 0000000..ecaf197 --- /dev/null +++ b/deploy/logging/sanitize.vrl @@ -0,0 +1,61 @@ +# Rebuild the event from scratch BEFORE the disk buffer. Never retain message, +# Docker metadata, Mongo attr/command, request path, host, UA, cookie or headers. +raw, err = parse_json(string!(.message)) +if err != null { raw = {} } +service = "nginx" +record = raw +if starts_with(string!(.file), "/host/docker/") { + service = string(raw.attrs.tag) ?? "unknown" + if !includes(["api", "streamer", "mongodb", "redis", "mongodb-backup"], service) { abort } + record, err = parse_json(string(raw.log) ?? "") + if err != null { record = {} } +} +out = {"timestamp": now(), "service": service, "severity": "info", "event": "service_output"} +# Use capture time for unsupported formats. Invalid/old producer timestamps +# cannot defeat retention or crash the collector. +event_time = raw.time +if exists(record.time) { event_time = record.time } +if exists(record.timestamp) { event_time = record.timestamp } +parsed, err = parse_timestamp(string(event_time) ?? "", format: "%+") +if err == null { out.timestamp = parsed } +if service == "mongodb" { + out.event = "mongodb_event" + if is_integer(record.id) { out.mongoEventId = record.id } + if includes(["W", "E", "F"], record.s) { out.severity = "warn" } + if includes(["E", "F"], record.s) { out.severity = "error" } +} else if service == "nginx" { + out.event = "edge_request" + record.requestId = record.app_request_id + record.firstByteMs = (to_float(record.upstream_header_time) ?? 0.0) * 1000.0 + record.ms = (to_float(record.request_time) ?? 0.0) * 1000.0 + if is_string(record.edge_request_id) && match(string!(record.edge_request_id), r'^[a-f0-9]{32}$') { out.edge_request_id = record.edge_request_id } + if is_string(record.cf_ray) && match(string!(record.cf_ray), r'^[a-f0-9]{16}-[A-Z]{3}$') { out.cf_ray = record.cf_ray } + if includes(["PASSED", "DELAYED", "REJECTED", "DELAYED_DRY_RUN", "REJECTED_DRY_RUN"], record.limit_req_status) { out.limit_req_status = record.limit_req_status } + # Deliberately omit IPs by default. Original edge JSON has short, restricted + # host retention for incident response; no plain or unsalted hashed IP archive. +} else if includes(["api", "streamer"], service) { + if record.service == "worker" { out.service = "worker" } + if includes(["debug", "info", "warn", "error"], record.severity) { out.severity = record.severity } + if includes(["account_removed", "application_log", "billing_maintenance_failed", "billing_reconciliation_deferred", "billing_requires_attention", "billing_worker_failed", "blob_api_failed_falling_back_to_raw_url", "branch_not_found", "cache_removal_failed", "cache_removed", "checking_repository_status_and_unused_repositories", "cleared_lock_for_active_job", "conference_expire_failed", "content_expiration_failed", "daily_stats_snapshot_failed", "daily_stats_snapshot_stored", "download_completed", "download_start", "downloaded", "downloadfile_failed", "downloading_file", "downloading_gist", "downloading_pull_request", "downloading_via_raw_url_lfs", "duplicate_stripe_customer_requires_review", "ensuretodaysnapshot_failed", "error_log_redis_connect_failed", "expired_repository_cleanup_failed", "failed_removal_recovery_failed", "failed_to_persist_error_status", "failed_to_record_repository_removal_error", "gettoken", "github_primary_rate_limit_hit", "github_secondary_rate_limit_hit", "grant_revocation_failed", "initial_repository_status_check_failed", "job_completed", "job_failed", "job_stalled", "job_started", "lastusedat_update_failed", "login_token_failed", "login_token_req_login_failed", "logout_after_account_removal_failed", "logout_failed", "lookup_failed", "markerrorifinflight_failed", "markerrorifremoving_failed", "metadata_cleanup_complete", "metadata_cleanup_deferred", "owner_access_alert_reset_unavailable", "owner_access_email_accepted", "owner_access_email_audit_unavailable", "owner_access_email_claim_cleanup_unavailable", "owner_access_email_deferred_recovery_unavailable", "owner_access_email_delivery_failed", "owner_access_email_preparation_cleanup_unavailable", "owner_access_email_recovery_recorded", "owner_access_email_recovery_unavailable", "owner_access_email_result_reconciliation_required", "owner_access_email_unavailable", "pr_comments_404_continuing_without_them", "process_fatal_error", "process_heartbeat", "process_started", "progress", "queue_heartbeat", "queued_for_download", "rate_limit_gate_active_delaying_job_before_work", "rate_limit_gate_set", "rate_limited_delaying_job", "recover_failed", "recovered_expired_repository_cleanup", "recovered_file_from_truncated_tree", "recovered_file_invalidation_failed", "recovered_stuck_repo", "recoverstuckpreparing_failed", "recoverstuckremoving_failed", "redis_client_error", "redis_rate_limit_gate_written", "rejected_upstream_download_destination", "removal_failed", "removal_recovery_failed", "removecache_save_failed", "removed_cache_for_unused_repository", "removing_cache", "removing_repository", "repo_claimed", "repository_cache_removal_failed", "repository_expiration_failed", "repository_expired", "repository_removal_failed", "repository_removed", "request_cancelled", "request_completed", "request_slow", "requeued_failed_removal", "requeued_interrupted_removal", "reset", "retired_tree_cleanup_deferred", "retired_tree_cleanup_failed", "running_daily_stats_snapshot", "s3_body_stream_error", "send_failed", "server_started", "source_stream_failed_cache_write_discarded", "staged_tree_cleanup_failed", "staged_tree_completion_deferred", "streamer_fetch_failed", "streamer_started", "streamer_startup_failed", "streamer_zip_fetch_failed", "temporary_storage_cleanup_deferred", "token_refresh_failed_backing_off", "up_to_date", "update_queued", "verify_failed", "waiting_for_rate_limit_gate", "webhook_log", "webhook_processing_failed", "write_failed", "zip_stream_failed"], record.event) { out.event = record.event } + if is_string(record.release) && match(string!(record.release), r'^[a-f0-9]{7,40}$') { out.release = record.release } + if is_string(record.instance) && match(string!(record.instance), r'^[a-f0-9-]{36}$') { out.instance = record.instance } + if includes(["ABORT_ERR", "ECONNREFUSED", "ECONNRESET", "ENOTFOUND", "EPIPE", "ETIMEDOUT", "billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_app_reconnect_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_token_invalid", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_signature", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "token_refresh_failed", "quota_exceeded", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_found", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "upstream_timeout", "user_not_found", "username_not_defined", "webhook_processing_failed"], record.errorCode) { out.errorCode = record.errorCode } + if includes(["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"], record.module) { out.module = record.module } + if includes(["protocol", "credentials", "port", "host"], record.reason) { out.reason = record.reason } + if includes(["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"], record.phase) { out.phase = record.phase } + if includes(["github_app_authorization", "installation", "installation_repositories"], record.webhookEvent) { out.webhookEvent = record.webhookEvent } + if includes(["github.com", "api.github.com", "codeload.github.com", "raw.githubusercontent.com", "media.githubusercontent.com", "other"], record.upstreamHost) { out.upstreamHost = record.upstreamHost } + if includes(["download", "remove", "cache"], record.queue) { out.queue = record.queue } + if includes(["completed", "interrupted"], record.outcome) { out.outcome = record.outcome } +} +for_each(["requestId", "jobId", "deliveryId"]) -> |_index, key| { + value = get(record, [key]) ?? null + if is_string(value) && match(string!(value), r'^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$') { out = set!(out, [key], value) } +} +for_each(["status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus", "stage_repository_ms", "stage_credential_resolution_ms", "stage_upstream_ms", "stage_cache_lookup_ms", "stage_cache_fill_ms", "stage_worker_wait_ms", "stage_anonymize_ms", "stage_streamer_headers_ms"]) -> |_index, key| { + value = get(record, [key]) ?? null + if is_integer(value) || is_float(value) { out = set!(out, [key], value) } +} +if includes(["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "OTHER"], record.method) { out.method = record.method } +if includes(["/", "other", "/dashboard", "/anonymize", "/healthcheck", "/api/healthcheck", "/api", "/api/download", "/api/stat", "/api/stat/history", "/api/options", "/api/message", "/api/user", "/api/user/quota", "/api/user/dashboard", "/api/repo", "/api/anonymize-preview", "/api/admin/overview", "/api/admin/performance", "/api/admin/errors", "/api/admin/errors/stats", "/api/admin/queues", "/api/admin/queues/metrics", "/github/app/webhook", "/api/repo/:repoId", "/api/repo/:repoId/file/:path", "/api/repo/:repoId/files", "/api/repo/:repoId/files/counts", "/api/repo/:repoId/files/search", "/api/repo/:repoId/zip", "/api/repo/:repoId/options", "/api/repo/:repoId/refresh", "/api/repo/:repoId/coauthors", "/api/repo/:repoId/extend", "/api/repo/:owner/:repo/readme", "/api/repo/:owner/:repo/branches", "/w/:repoId/:path", "/r/:repoId/:path", "/api/pr/:id/:path", "/api/gist/:id/:path", "/assets/:path", "/github/:action", "/api/admin/:action", "/api/user/:action", "/api/repo/:id/file/:path", "/api/repo/:id/files/counts", "/api/repo/:id/files/search", "/api/repo/:id/files", "/api/repo/:id/zip", "/api/repo/:id", "/viewer/:id/:path"], record.route) { out.route = record.route } +. = out diff --git a/deploy/logging/vector.yaml b/deploy/logging/vector.yaml new file mode 100644 index 0000000..89f20b8 --- /dev/null +++ b/deploy/logging/vector.yaml @@ -0,0 +1,49 @@ +data_dir: /var/lib/vector +sources: + docker: + type: file + include: [/host/docker/*/*-json.log] + read_from: end + max_line_bytes: 65536 + max_read_bytes: 65536 + rotate_wait_secs: 60 + fingerprint: {strategy: device_and_inode} + nginx: + type: file + include: [/host/nginx/anonymous.access.json*] + exclude: ['*.gz'] + read_from: end + max_line_bytes: 65536 + fingerprint: {strategy: device_and_inode} + internal: + type: internal_metrics + scrape_interval_secs: 15 +transforms: + safe: + type: remap + inputs: [docker, nginx] + file: /etc/vector/sanitize.vrl + drop_on_error: true + drop_on_abort: true +sinks: + logs: + type: elasticsearch + inputs: [safe] + endpoints: [http://victorialogs:9428/insert/elasticsearch/] + api_version: v8 + compression: gzip + healthcheck: {enabled: false} + query: + _msg_field: event + _time_field: timestamp + _stream_fields: service,severity + batch: {max_bytes: 1048576, timeout_secs: 1} + request: {timeout_secs: 5, concurrency: 1} + buffer: + type: disk + max_size: 268435488 + when_full: drop_newest + metrics: + type: prometheus_exporter + inputs: [internal] + address: 0.0.0.0:9598 diff --git a/docs/performance-monitoring.md b/docs/performance-monitoring.md index 7e828d2..4f78415 100644 --- a/docs/performance-monitoring.md +++ b/docs/performance-monitoring.md @@ -156,3 +156,7 @@ The constrained-heap regression fixture can be run without network or MongoDB: See [Shared blob cache](shared-cache.md) for configuration, storage and quota accounting, admin metrics, production observations, rollout and benchmark commands. + +For opt-in persistent structured logs, cross-service request/job investigation, +and private Grafana dashboards, see [operations logging](../deploy/logging/README.md). +The existing Redis performance/admin endpoints remain available during migration. diff --git a/scripts/check-production-logs.js b/scripts/check-production-logs.js index feead60..a0dbb7e 100644 --- a/scripts/check-production-logs.js +++ b/scripts/check-production-logs.js @@ -9,17 +9,26 @@ function observeLine(summary, line) { // Older releases rely on Node's stderr. New releases also emit a redacted // process_fatal_error event before Node terminates. if (/ECONNRESET|socket hang up/i.test(line)) summary.connectionResets++; - const match = /^\S+ (INFO|WARN|ERROR) \[([^\]]+)\] .*? (\{.*\})$/.exec(line); - if (!match) { - if (/Unhandled ['"]error['"] event|unhandled(?:promise)?rejection|FATAL ERROR|heap out of memory/i.test(line)) summary.fatalErrors++; - return; + let data, level, scope; + try { + const record = JSON.parse(line); + if (record && typeof record === "object" && typeof record.severity === "string") { + data = record; level = record.severity.toUpperCase(); scope = record.module; + data.code = record.errorCode || record.event; + } + } catch { /* Support pre-migration mixed text/JSON too. */ } + if (!data) { + const match = /^\S+ (INFO|WARN|ERROR) \[([^\]]+)\] .*? (\{.*\})$/.exec(line); + if (!match) { + if (/Unhandled ['"]error['"] event|unhandled(?:promise)?rejection|FATAL ERROR|heap out of memory/i.test(line)) summary.fatalErrors++; + return; + } + try { data = JSON.parse(match[3]); } catch { return; } + [, level, scope] = match; } - let data; - try { data = JSON.parse(match[3]); } catch { return; } - const [, level, scope] = match; if (level === "ERROR") summary.errors++; if (level === "WARN") summary.warnings++; - if (data.code === "process_fatal_error" || /startup failed/i.test(line)) summary.fatalErrors++; + if (data.code === "process_fatal_error" || /startup[ _]failed/i.test(line)) summary.fatalErrors++; if (data.code === "token_refresh_failed") summary.tokenRefreshFailures++; if (scope === "anonymized-file" && data.code === "file_not_found") summary.missingFiles++; if (scope === "requests" && typeof data.ms === "number") { diff --git a/scripts/test-logging.py b/scripts/test-logging.py new file mode 100755 index 0000000..ae64af6 --- /dev/null +++ b/scripts/test-logging.py @@ -0,0 +1,236 @@ +#!/usr/bin/env python3 +"""Isolated local Docker integration test. Never reads production logs or .env. +Requires Docker Compose, Python 3 and PyYAML. Removes only its own project. +""" +import json +import base64 +import os +from pathlib import Path +import secrets +import sys +import subprocess +import tempfile +import time +import urllib.parse +import urllib.request +import uuid +import yaml + +ROOT = Path(__file__).resolve().parents[1] +CONFIG = ROOT / 'deploy/logging' +PROJECT = 'logging-test-' + secrets.token_hex(4) + +def run(args, **kw): + return subprocess.run(args, check=True, text=True, capture_output=True, **kw).stdout.strip() + +def wait_for(fn, timeout=60): + end = time.monotonic() + timeout + last = None + while time.monotonic() < end: + try: + result = fn() + if result: + return result + except Exception as e: + last = e + time.sleep(1) + raise AssertionError('Timed out: ' + str(last)) + +with tempfile.TemporaryDirectory(prefix=PROJECT) as temp: + temp = Path(temp) + docker = temp / 'docker' + nginx = temp / 'nginx' + docker.mkdir(); nginx.mkdir() + (docker / 'fixture').mkdir() + log = docker / 'fixture/fixture-json.log' + log.touch() + edge = nginx / 'anonymous.access.json' + edge.touch() + password = temp / 'password' + password.write_text(secrets.token_urlsafe(32)) + password.chmod(0o644) # temp directory is 0700; Grafana uid must read the bind mount + env = {**os.environ, 'GRAFANA_ADMIN_USER': 'local-test', 'GRAFANA_PASSWORD_FILE': str(password), + 'DOCKER_LOG_DIR': str(docker), 'NGINX_LOG_DIR': str(nginx), 'GRAFANA_PORT': '0'} + override = temp / 'override.yml' + override.write_text(yaml.safe_dump({'services': { + 'victorialogs': {'ports': ['127.0.0.1::9428']}, + 'prometheus': {'ports': ['127.0.0.1::9090']}, + 'vector': {'ports': ['127.0.0.1::9598']}, + 'vmalert': {'ports': ['127.0.0.1::8880']}, + }, 'networks': {'private': {'internal': False}}})) + cmd = ['docker', 'compose', '-p', PROJECT, '-f', str(CONFIG / 'compose.yml'), '-f', str(override)] + def compose(*args): + return run(cmd + list(args), env=env) + def url(service, port): + return 'http://' + compose('port', service, str(port)) + def get(base, path, params=None): + address = base + path + ('?' + urllib.parse.urlencode(params) if params else '') + return urllib.request.urlopen(address, timeout=10).read().decode() + def query(expression): + return [json.loads(line) for line in get(vl, '/select/logsql/query', {'query': expression, 'limit': 50000}).splitlines() if line] + def append(service, record): + with log.open('a') as out: + out.write(json.dumps({'log': json.dumps(record) + '\n', 'attrs': {'tag': service}, 'stream': 'stdout'}) + '\n') + def count(): + return int(query('* | stats count() as n')[0]['n']) + try: + compose('up', '-d') + vl = url('victorialogs', 9428) + prom = url('prometheus', 9090) + vector = url('vector', 9598) + alerts = url('vmalert', 8880) + grafana = url('grafana', 3000) + wait_for(lambda: get(vl, '/health') is not None) + wait_for(lambda: get(vector, '/metrics')) + wait_for(lambda: get(grafana, '/api/health')) + wait_for(lambda: 'application-logs' in get(alerts, '/api/v1/rules')) + time.sleep(2) # file watcher must discover empty files before append + request_id, job_id = str(uuid.uuid4()), str(uuid.uuid4()) + sentinel = 'PRIVATE_SENTINEL_owner_repository_cookie_oauth_state' + for service in ['api', 'streamer']: + append(service, {'event': 'request_completed', 'severity': 'info', 'requestId': request_id, + 'route': '/api/repo/:repoId/file/:path', 'status': 200, 'outcome': 'completed', + 'firstByteMs': 40, 'responseMs': 600, 'ms': 640, 'stage_upstream_ms': 20, + 'message': sentinel, 'url': 'https://example.invalid/' + sentinel, 'cookie': sentinel, + 'authorization': sentinel, 'requestBody': {'secret': sentinel}, 'release': 'abcdef1'}) + append('api', {'service': 'worker', 'event': 'job_failed', 'severity': 'error', 'requestId': request_id, + 'jobId': job_id, 'attempt': 2, 'queue': 'download', 'errorCode': 'ETIMEDOUT', 'raw': sentinel}) + append('mongodb', {'id': 51803, 's': 'E', 'msg': sentinel, 'attr': {'command': sentinel}}) + append('redis', {'message': sentinel}) + append('mongodb-backup', {'message': sentinel}) + append('unrelated-container', {'message': sentinel}) + # Malformed and malicious field values are represented without their raw content. + append('api', {'event': sentinel, 'route': '/' + sentinel, 'requestId': sentinel, 'severity': sentinel, 'module': sentinel}) + with log.open('a') as out: + out.write(json.dumps({'log': sentinel, 'attrs': {'tag': 'api'}}) + '\n') + with edge.open('a') as out: + out.write(json.dumps({'status': 429, 'request_time': 0.1, 'upstream_header_time': '0.05', + 'app_request_id': request_id, 'edge_request_id': 'f' * 32, 'route': '/api/repo/:id', + 'limit_req_status': 'REJECTED', 'client_ip': '192.0.2.1', 'user_agent': sentinel, + 'host': sentinel, 'request': sentinel}) + '\n') + wait_for(lambda: count() >= 9) + rows = query('*') + stored = json.dumps(rows) + assert sentinel not in stored and '192.0.2.1' not in stored + correlated = query('requestId:="' + request_id + '"') + assert {r['service'] for r in correlated} == {'api', 'streamer', 'worker', 'nginx'} + assert len(query('jobId:="' + job_id + '"')) == 1 + assert all('requestId' not in r['_stream'] and 'instance' not in r['_stream'] for r in rows) + print('PASS ingestion, privacy allowlist, malformed input, API/streamer/job/edge correlation') + # Query every shipped dashboard and alert expression against the real server. + dashboard = json.loads((CONFIG / 'grafana/dashboards/operations.json').read_text()) + for panel in dashboard['panels']: + for target in panel['targets']: + expr = target['expr'].replace('$request_id', request_id).replace('$job_id', job_id).replace('$edge_id', 'f' * 32) + if panel['datasource']['uid'] == 'anonymous-logs': + query(expr) + else: + get(prom, '/api/v1/query', {'query': expr}) + for group in yaml.safe_load((CONFIG / 'log-alerts.yml').read_text())['groups']: + for rule in group['rules']: + get(vl, '/select/logsql/stats_query', {'query': rule['expr']}) + auth = base64.b64encode(('local-test:' + password.read_text()).encode()).decode() + def grafana_request(path, payload=None): + request = urllib.request.Request(grafana + path, + data=json.dumps(payload).encode() if payload is not None else None, + headers={'Authorization': 'Basic ' + auth, 'Content-Type': 'application/json'}) + return json.loads(urllib.request.urlopen(request, timeout=20).read()) + assert grafana_request('/api/dashboards/uid/anonymous-operations')['dashboard']['uid'] == 'anonymous-operations' + assert grafana_request('/api/datasources/uid/anonymous-logs/health')['status'] == 'OK' + result = grafana_request('/api/ds/query', {'from': str(int((time.time() - 3600) * 1000)), + 'to': str(int(time.time() * 1000)), 'queries': [{'refId': 'A', + 'datasource': {'type': 'victoriametrics-logs-datasource', 'uid': 'anonymous-logs'}, + 'expr': 'requestId:="' + request_id + '"', 'queryType': 'instant', 'maxLines': 100}]}) + assert not result['results']['A'].get('error'), result['results']['A'].get('error') + assert result['results']['A']['frames'] + print('PASS dashboard and alert queries, provisioned Grafana datasource query') + # Rename rotation, then collector restart, preserving the spool/checkpoint volume. + before = count() + log.rename(log.with_suffix('.log.1')) + log.touch() + append('api', {'event': 'process_started', 'severity': 'info'}) + wait_for(lambda: count() > before) + compose('restart', 'vector') + vector = url('vector', 9598) + wait_for(lambda: get(vector, '/metrics')) + before = count() + append('api', {'event': 'process_heartbeat', 'severity': 'info'}) + wait_for(lambda: count() > before) + print('PASS rename rotation and collector checkpoint restart') + # Backend failure: sanitized disk spool remains bounded; source writer stays independent. + before = count() + compose('stop', 'victorialogs') + started = time.monotonic() + for _ in range(2000): + append('api', {'event': 'request_completed', 'severity': 'info', 'requestId': str(uuid.uuid4()), 'status': 200}) + assert time.monotonic() - started < 10 + wait_for(lambda: 'vector_buffer_size_events' in get(vector, '/metrics')) + time.sleep(3) + compose('restart', 'vector') + compose('start', 'victorialogs') + vl = url('victorialogs', 9428) + wait_for(lambda: count() >= before + 2000, 120) + print('PASS unavailable backend, spool persistence, recovery of 2000 records') + # Reject data older than retention rather than resurrecting it. + old = json.dumps({'timestamp': '2000-01-01T00:00:00Z', 'event': 'expired_fixture'}).encode() + req = urllib.request.Request(vl + '/insert/jsonline?_time_field=timestamp', data=old, headers={'Content-Type': 'application/stream+json'}) + urllib.request.urlopen(req, timeout=10).read() + time.sleep(2) + assert not query('event:expired_fixture') + assert 'reason="too_small_timestamp"' in get(vl, '/metrics') + print('PASS retention rejection') + # Compression observation, synthetic workload only, including random trace IDs. + raw_bytes = 0 + for _ in range(10000): + record = {'event': 'request_completed', 'severity': 'info', 'requestId': str(uuid.uuid4()), + 'status': 200, 'ms': 50, 'firstByteMs': 20, 'responseMs': 30, 'route': '/api/repo/:repoId/file/:path'} + raw_bytes += len(json.dumps(record).encode()) + 1 + append('api', record) + wait_for(lambda: count() >= before + 12000, 120) + compose('restart', 'victorialogs') + vl = url('victorialogs', 9428) + wait_for(lambda: get(vl, '/health') is not None) + metrics_text = get(vl, '/metrics') + size = sum(float(line.split()[-1]) for line in metrics_text.splitlines() if line.startswith('vl_data_size_bytes{')) + print(f'MEASURED fixture JSON bytes={raw_bytes}; total store data+index bytes={int(size)}; store also contains earlier fixtures; no production ratio claimed') + # Check fixed limits and no accidental public ingress. + config = json.loads(compose('config', '--format', 'json')) + for service, spec in config['services'].items(): + assert int(spec['mem_limit']) > 0, service + for port in spec.get('ports', []): + assert port['host_ip'] == '127.0.0.1', service + print('PASS configured memory/rotation/disk bounds and loopback-only test ports') + if '--stress' in sys.argv: + compose('stop', 'victorialogs') + vector = url('vector', 9598) + numeric_fields = ['status', 'httpStatus', 'ms', 'firstByteMs', 'responseMs', 'bytes', 'bytesPerSecond', + 'attempt', 'rss', 'memoryLimitBytes', 'memoryRatio', 'uptime', 'activeRequests', 'waiting', + 'active', 'delayed', 'failed', 'oldestWaitingMs', 'droppedLogs', 'loopP95Ms', 'upstreamStatus', + 'stage_repository_ms', 'stage_credential_resolution_ms', 'stage_upstream_ms', 'stage_cache_lookup_ms', + 'stage_cache_fill_ms', 'stage_worker_wait_ms', 'stage_anonymize_ms', 'stage_streamer_headers_ms'] + stress = {field: 123456789 for field in numeric_fields} + stress.update(event='request_completed', severity='info', requestId=request_id) + line = json.dumps({'log': json.dumps(stress), 'attrs': {'tag': 'api'}}) + '\n' + with log.open('a') as output: + for _ in range(500): + output.write(line * 1000) + def metric_value(name): + lines = get(vector, '/metrics').splitlines() + return sum(float(line.split()[1]) for line in lines if line.startswith(name + '{') and ('buffer_id="logs"' in line if name.startswith('vector_buffer_') else 'component_id="logs"' in line)) + wait_for(lambda: metric_value('vector_buffer_discarded_events_total') > 0 or metric_value('vector_component_discarded_events_total') > 0, 240) + buffered = metric_value('vector_buffer_size_bytes') + assert buffered <= 268435488 + 1048576, buffered + inspection = json.loads(run(['docker', 'inspect', compose('ps', '-q', 'vector')]))[0] + assert inspection['RestartCount'] == 0 and not inspection['State']['OOMKilled'] + print(f'PASS full-spool drop-newest stress: 500000 events; buffered={int(buffered)} bytes; collector stayed alive') + # Wait through the production two-minute 'for', rather than merely + # checking PromQL syntax or that a counter exists. + wait_for(lambda: json.loads(get(prom, '/api/v1/query', {'query': + 'ALERTS{alertname="CollectorDrops",alertstate="firing"}'}))['data']['result'], 210) + print('PASS sustained CollectorDrops alert fires on full-spool loss') + except Exception: + # Only synthetic fixture logs exist in this project. No production reads. + print(compose('logs', '--tail', '30', 'vector', 'victorialogs', 'vmalert', 'grafana')) + raise + finally: + compose('down', '-v', '--remove-orphans') diff --git a/src/core/Repository.ts b/src/core/Repository.ts index f61bfef..2b48dda 100644 --- a/src/core/Repository.ts +++ b/src/core/Repository.ts @@ -1,3 +1,4 @@ +import { jobTrace } from "./log-context"; import { REPOSITORY_ID_COLLATION } from "./repository-id"; import { reserveRepository } from "./billing/quotas"; import { hasAutomaticUpdates } from "./billing/expiration"; @@ -855,7 +856,7 @@ export default class Repository { const current = await AnonymizedRepositoryModel.exists({ _id: this.model._id, ...this.refreshFilter() }); if (!current) throw new AnonymousError("invalid_status", { httpStatus: 409 }); } - await downloadQueue.add(this.repoId, { repoId: this.repoId }, { + await downloadQueue.add(this.repoId, { repoId: this.repoId, ...jobTrace() }, { jobId: `repo-${this.repoId}`, attempts: 3, }); diff --git a/src/core/log-context.ts b/src/core/log-context.ts new file mode 100644 index 0000000..02d464f --- /dev/null +++ b/src/core/log-context.ts @@ -0,0 +1,11 @@ +import { AsyncLocalStorage } from "async_hooks"; +import { randomUUID } from "crypto"; + +export interface LogContext { requestId?: string; jobId?: string; attempt?: number; service?: string } +export const logContext = new AsyncLocalStorage(); +export const validTraceId = (value: unknown): value is string => typeof value === "string" && + /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(value); +export function jobTrace() { + const requestId = logContext.getStore()?.requestId; + return { logJobId: randomUUID(), ...(requestId ? { requestId } : {}) }; +} diff --git a/src/core/log-memory.ts b/src/core/log-memory.ts new file mode 100644 index 0000000..56a0637 --- /dev/null +++ b/src/core/log-memory.ts @@ -0,0 +1,15 @@ +import { readFile } from "fs/promises"; +/** Container usage includes sandbox workers and page cache; process RSS does not. */ +export async function logMemory() { + const rss = process.memoryUsage().rss; + for (const [usage, limit] of [ + ["/sys/fs/cgroup/memory.current", "/sys/fs/cgroup/memory.max"], + ["/sys/fs/cgroup/memory/memory.usage_in_bytes", "/sys/fs/cgroup/memory/memory.limit_in_bytes"], + ]) { + const [used, max] = await Promise.all([usage, limit].map(path => readFile(path, "utf8").then(Number).catch(() => NaN))); + if (Number.isFinite(used) && max > 0 && max < Number.MAX_SAFE_INTEGER) { + return { rss, memoryLimitBytes: max, memoryRatio: used / max }; + } + } + return { rss }; +} diff --git a/src/core/log-record.ts b/src/core/log-record.ts new file mode 100644 index 0000000..6f22efe --- /dev/null +++ b/src/core/log-record.ts @@ -0,0 +1,59 @@ +import { LOG_MODULES, ERROR_CODES, LOG_EVENTS } from "./log-schema"; +import { randomUUID } from "crypto"; +import { logContext, validTraceId } from "./log-context"; + +// No arbitrary strings: upstream errors, paths and even innocuous-looking messages +// can contain a private repository name. Add diagnostics here deliberately. +const instance = randomUUID(); +const events: Record = { + ...LOG_EVENTS, + request: "request_completed", "slow request": "request_slow", "request interrupted": "request_cancelled", + "process fatal error": "process_fatal_error", "process started": "process_started", + "process heartbeat": "process_heartbeat", "job started": "job_started", "job completed": "job_completed", + "job failed": "job_failed", "job stalled": "job_stalled", "queue heartbeat": "queue_heartbeat", +}; +const errorCodes = new Set([ + ...ERROR_CODES, + "token_refresh_failed", "quota_exceeded", "github_rate_limit_exceeded", "github_app_reconnect_required", "github_token_invalid", + "repository_expired", "repository_not_ready", "repository_not_found", "repository_job_cancelled", + "ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND", "EPIPE", "ABORT_ERR", + "upstream_error", "upstream_timeout", "invalid_signature", "webhook_processing_failed", +]); +const numbers = ["status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", + "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", + "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus"]; +const stages = ["repository", "credential_resolution", "upstream", "cache_lookup", "cache_fill", "worker_wait", "anonymize", "streamer_headers"]; +export function logRecord(level: string, module: string, args: unknown[]) { + const context = logContext.getStore() || {}; + const details = Object.assign({}, ...args.filter(a => a && typeof a === "object" && !Array.isArray(a))); + const first = typeof args[0] === "string" ? args[0] : ""; + const event = events[first] || (module.includes("webhook") ? "webhook_log" : "application_log"); + const record: Record = { timestamp: new Date().toISOString(), severity: level, + service: context.service || (["api", "streamer", "worker"].includes(details.service) ? details.service : undefined) || (process.env.SERVICE_NAME?.toLowerCase() === "streamer" ? "streamer" : "api"), + instance, release: /^[a-f0-9]{7,40}$/.test(process.env.APP_RELEASE || "") ? process.env.APP_RELEASE : "unknown", + event, message: event, module: LOG_MODULES.includes(module) ? module : "application" }; + for (const key of ["requestId", "jobId"] as const) { + const value = details[key] || context[key]; + if (validTraceId(value)) record[key] = value; + } + if (Number.isSafeInteger(context.attempt)) record.attempt = context.attempt; + for (const key of numbers) if (typeof details[key] === "number" && Number.isFinite(details[key])) record[key] = details[key]; + const code = [details.code, details.message].find(value => errorCodes.has(value)); + if (code) record.errorCode = code; + if (["protocol", "credentials", "port", "host"].includes(details.reason)) record.reason = details.reason; + if (["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"].includes(details.phase)) record.phase = details.phase; + if (["github_app_authorization", "installation", "installation_repositories"].includes(details.event)) record.webhookEvent = details.event; + if (validTraceId(details.deliveryId)) record.deliveryId = details.deliveryId; + if (["github.com", "api.github.com", "codeload.github.com", "raw.githubusercontent.com", "media.githubusercontent.com"].includes(details.hostname)) record.upstreamHost = details.hostname; + else if (details.hostname) record.upstreamHost = "other"; + if (["completed", "interrupted"].includes(details.outcome)) record.outcome = details.outcome; + if (["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "OTHER"].includes(details.method)) record.method = details.method; + // Route labels are generated by request-monitoring, never from URLs in arbitrary errors. + if (module === "requests" && typeof details.route === "string") record.route = details.route; + if (["download", "remove", "cache"].includes(details.queue)) record.queue = details.queue; + if (details.stages && typeof details.stages === "object") for (const key of stages) { + const value = details.stages[key]; + if (typeof value === "number" && Number.isFinite(value)) record[`stage_${key}_ms`] = value; + } + return record; +} diff --git a/src/core/log-schema.ts b/src/core/log-schema.ts new file mode 100644 index 0000000..ecdf3b2 --- /dev/null +++ b/src/core/log-schema.ts @@ -0,0 +1,121 @@ +// Reviewed literal diagnostics. Never extend this list from runtime data. +export const LOG_MODULES = ["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"]; +export const ERROR_CODES = ["billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "user_not_found", "username_not_defined"]; +export const LOG_EVENTS: Record = { + "PR comments 404, continuing without them": "pr_comments_404_continuing_without_them", + "S3 body stream error": "s3_body_stream_error", + "account removed": "account_removed", + "billing maintenance failed": "billing_maintenance_failed", + "billing reconciliation deferred": "billing_reconciliation_deferred", + "billing requires attention": "billing_requires_attention", + "billing worker failed": "billing_worker_failed", + "blob API failed, falling back to raw URL": "blob_api_failed_falling_back_to_raw_url", + "branch not found": "branch_not_found", + "cache removal failed": "cache_removal_failed", + "cache removed": "cache_removed", + "checking repository status and unused repositories": "checking_repository_status_and_unused_repositories", + "cleared lock for active job": "cleared_lock_for_active_job", + "conference expire failed": "conference_expire_failed", + "content expiration failed": "content_expiration_failed", + "daily stats snapshot failed": "daily_stats_snapshot_failed", + "daily stats snapshot stored": "daily_stats_snapshot_stored", + "download completed": "download_completed", + "download start": "download_start", + "downloadFile failed": "downloadfile_failed", + "downloaded": "downloaded", + "downloading file": "downloading_file", + "downloading gist": "downloading_gist", + "downloading pull request": "downloading_pull_request", + "downloading via raw URL (LFS)": "downloading_via_raw_url_lfs", + "duplicate Stripe customer requires review": "duplicate_stripe_customer_requires_review", + "ensureTodaySnapshot failed": "ensuretodaysnapshot_failed", + "error log redis connect failed": "error_log_redis_connect_failed", + "expired repository cleanup failed": "expired_repository_cleanup_failed", + "failed removal recovery failed": "failed_removal_recovery_failed", + "failed to persist ERROR status": "failed_to_persist_error_status", + "failed to record repository removal error": "failed_to_record_repository_removal_error", + "getToken": "gettoken", + "github primary rate limit hit": "github_primary_rate_limit_hit", + "github secondary rate limit hit": "github_secondary_rate_limit_hit", + "grant revocation failed": "grant_revocation_failed", + "initial repository status check failed": "initial_repository_status_check_failed", + "job completed": "job_completed", + "job stalled": "job_stalled", + "job started": "job_started", + "lastUsedAt update failed": "lastusedat_update_failed", + "login-token failed": "login_token_failed", + "login-token req.login failed": "login_token_req_login_failed", + "logout after account removal failed": "logout_after_account_removal_failed", + "logout failed": "logout_failed", + "lookup failed": "lookup_failed", + "markErrorIfInFlight failed": "markerrorifinflight_failed", + "markErrorIfRemoving failed": "markerrorifremoving_failed", + "metadata cleanup complete": "metadata_cleanup_complete", + "metadata cleanup deferred": "metadata_cleanup_deferred", + "owner access alert reset unavailable": "owner_access_alert_reset_unavailable", + "owner access email accepted": "owner_access_email_accepted", + "owner access email audit unavailable": "owner_access_email_audit_unavailable", + "owner access email claim cleanup unavailable": "owner_access_email_claim_cleanup_unavailable", + "owner access email deferred recovery unavailable": "owner_access_email_deferred_recovery_unavailable", + "owner access email delivery failed": "owner_access_email_delivery_failed", + "owner access email preparation cleanup unavailable": "owner_access_email_preparation_cleanup_unavailable", + "owner access email recovery recorded": "owner_access_email_recovery_recorded", + "owner access email recovery unavailable": "owner_access_email_recovery_unavailable", + "owner access email result reconciliation required": "owner_access_email_result_reconciliation_required", + "owner access email unavailable": "owner_access_email_unavailable", + "process fatal error": "process_fatal_error", + "process heartbeat": "process_heartbeat", + "process started": "process_started", + "progress": "progress", + "queue heartbeat": "queue_heartbeat", + "queued for download": "queued_for_download", + "rate limit gate active, delaying job before work": "rate_limit_gate_active_delaying_job_before_work", + "rate limit gate set": "rate_limit_gate_set", + "rate-limited, delaying job": "rate_limited_delaying_job", + "recover failed": "recover_failed", + "recoverStuckPreparing failed": "recoverstuckpreparing_failed", + "recoverStuckRemoving failed": "recoverstuckremoving_failed", + "recovered expired repository cleanup": "recovered_expired_repository_cleanup", + "recovered file from truncated tree": "recovered_file_from_truncated_tree", + "recovered file invalidation failed": "recovered_file_invalidation_failed", + "recovered stuck repo": "recovered_stuck_repo", + "redis client error": "redis_client_error", + "redis rate limit gate written": "redis_rate_limit_gate_written", + "rejected upstream download destination": "rejected_upstream_download_destination", + "removal failed": "removal_failed", + "removal recovery failed": "removal_recovery_failed", + "removeCache save failed": "removecache_save_failed", + "removed cache for unused repository": "removed_cache_for_unused_repository", + "removing cache": "removing_cache", + "removing repository": "removing_repository", + "repo claimed": "repo_claimed", + "repository cache removal failed": "repository_cache_removal_failed", + "repository expiration failed": "repository_expiration_failed", + "repository expired": "repository_expired", + "repository removal failed": "repository_removal_failed", + "repository removed": "repository_removed", + "requeued failed removal": "requeued_failed_removal", + "requeued interrupted removal": "requeued_interrupted_removal", + "reset": "reset", + "retired tree cleanup deferred": "retired_tree_cleanup_deferred", + "retired tree cleanup failed": "retired_tree_cleanup_failed", + "running daily stats snapshot": "running_daily_stats_snapshot", + "send failed": "send_failed", + "server started": "server_started", + "source stream failed; cache write discarded": "source_stream_failed_cache_write_discarded", + "staged tree cleanup failed": "staged_tree_cleanup_failed", + "staged tree completion deferred": "staged_tree_completion_deferred", + "streamer fetch failed": "streamer_fetch_failed", + "streamer started": "streamer_started", + "streamer startup failed": "streamer_startup_failed", + "streamer zip fetch failed": "streamer_zip_fetch_failed", + "temporary storage cleanup deferred": "temporary_storage_cleanup_deferred", + "token refresh failed; backing off": "token_refresh_failed_backing_off", + "up to date": "up_to_date", + "update queued": "update_queued", + "verify failed": "verify_failed", + "waiting for rate limit gate": "waiting_for_rate_limit_gate", + "webhook processing failed": "webhook_processing_failed", + "write failed": "write_failed", + "zip stream failed": "zip_stream_failed" +}; diff --git a/src/core/logger.ts b/src/core/logger.ts index b3be112..32bd902 100644 --- a/src/core/logger.ts +++ b/src/core/logger.ts @@ -1,3 +1,4 @@ +import { logRecord } from "./log-record"; import { redactSecrets } from "./redact-secrets"; import { createClient, RedisClientType } from "redis"; import config from "../config"; @@ -40,16 +41,6 @@ function resolveThreshold(): number { const threshold = resolveThreshold(); -function formatArg(a: unknown): string { - if (typeof a === "string") return a; - if (a instanceof Error) return JSON.stringify(serializeError(a)); - try { - return JSON.stringify(a); - } catch { - return String(a); - } -} - let redisClient: RedisClientType | null = null; let redisDisabled = false; @@ -58,6 +49,8 @@ function getRedis(): RedisClientType | null { if (redisClient) return redisClient; try { redisClient = createClient({ + disableOfflineQueue: true, + commandsQueueMaxLength: 128, socket: { host: config.REDIS_HOSTNAME, port: config.REDIS_PORT, @@ -92,6 +85,13 @@ function getRedis(): RedisClientType | null { // Redis is back. Read by /admin/errors/stats so the admin page surfaces // "you're losing logs" instead of silently rendering an empty table. let droppedInProcess = 0; +let droppedOutput = 0; +let outputBroken = false; +let errorOutputBroken = false; +// A closed log reader must not become an uncaught EPIPE on the serving process. +process.stdout.on("error", () => { outputBroken = true; droppedOutput++; }); +process.stderr.on("error", () => { errorOutputBroken = true; droppedOutput++; }); +export function getDroppedOutput() { return droppedOutput; } export function getInProcessDropped(): number { return droppedInProcess; } @@ -192,8 +192,9 @@ function persistError(entry: { message: string; raw: unknown[]; }) { + if (process.env.LOG_REDIS_MIRROR === "off") return; const client = getRedis(); - if (!client || !client.isOpen) { + if (!client || !client.isReady) { droppedInProcess++; return; } @@ -204,12 +205,12 @@ function persistError(entry: { ) as Record | undefined; const bucket = bucketFor(detail, entry.level); const code = - (detail && typeof detail.message === "string" - ? (detail.message as string) - : "") || (detail && typeof detail.code === "string" ? (detail.code as string) : "") || + (detail && typeof detail.message === "string" + ? (detail.message as string) + : "") || "_"; // Routine client misuse (401/403/404 and the allowlisted "expected" 4xx // codes below) drowns out real errors in the recent-entries ring. Keep the @@ -254,27 +255,25 @@ function emit(level: Level, module: string, args: unknown[]) { if (LEVEL_ORDER[level] < threshold) return; args = args.map(arg => redactSecrets(arg instanceof Error ? serializeError(arg) : arg)); const ts = new Date().toISOString(); - const formatted = args.map(formatArg); - const line = `${ts} ${level.toUpperCase()} [${module}] ${formatted.join(" ")}`; - const sink = - level === "error" - ? console.error - : level === "warn" - ? console.warn - : level === "debug" - ? console.debug - : console.log; - sink(line); + const record = logRecord(level, module, args); + // stdout is a pipe in Docker. Respect backpressure without awaiting it or + // building an unbounded userland queue. The next heartbeat reports losses. + const output = outputBroken ? process.stderr : process.stdout; + if (!(outputBroken && errorOutputBroken) && output.writableLength < 256 * 1024) { + try { output.write(JSON.stringify(record) + "\n"); } + catch { + if (output === process.stdout) outputBroken = true; + else errorOutputBroken = true; + droppedOutput++; + } + } else { droppedOutput++; } if (level === "error" || level === "warn") { persistError({ ts, level, module, - message: typeof args[0] === "string" ? args[0] : "", - raw: args.map((a) => { - if (a instanceof Error) return serializeError(a); - return a; - }), + message: String(record.event), + raw: [{ ...record, code: record.errorCode || record.event }], }); } } diff --git a/src/core/process-monitoring.ts b/src/core/process-monitoring.ts index e33e2da..a68657f 100644 --- a/src/core/process-monitoring.ts +++ b/src/core/process-monitoring.ts @@ -1,4 +1,5 @@ -import { createLogger, serializeError } from "./logger"; +import { logMemory } from "./log-memory"; +import { createLogger, serializeError, getDroppedOutput } from "./logger"; /** Observe fatal errors without overriding Node's default exit behavior. */ export function installFatalErrorLogging(service: "api" | "streamer") { @@ -6,6 +7,12 @@ export function installFatalErrorLogging(service: "api" | "streamer") { const report = (error: Error, origin: string) => logger.error("process fatal error", { ...serializeError(error), code: "process_fatal_error", service, origin, }); + logger.info("process started", { service }); + const heartbeat = setInterval(() => { + void logMemory().then(memory => logger.info("process heartbeat", { + service, uptime: process.uptime(), ...memory, droppedLogs: getDroppedOutput() })); + }, 30000); + heartbeat.unref(); process.on("uncaughtExceptionMonitor", report); - return () => process.removeListener("uncaughtExceptionMonitor", report); + return () => { clearInterval(heartbeat); process.removeListener("uncaughtExceptionMonitor", report); }; } diff --git a/src/core/request-monitoring.ts b/src/core/request-monitoring.ts index f4ca19b..1647619 100644 --- a/src/core/request-monitoring.ts +++ b/src/core/request-monitoring.ts @@ -1,3 +1,4 @@ +import { logContext } from "./log-context"; import { AsyncLocalStorage } from "async_hooks"; import { randomUUID } from "crypto"; import { performance } from "perf_hooks"; @@ -164,6 +165,6 @@ export function monitorRequests(service: Service, metrics = requestMetrics, else if (!route.includes("healthcheck")) log.info("request", detail); }; res.once("finish", done); res.once("close", done); - context.run(trace, next); + context.run(trace, () => logContext.run({ requestId: id, service }, next)); }; } diff --git a/src/queue/index.ts b/src/queue/index.ts index 738ca21..ac73668 100644 --- a/src/queue/index.ts +++ b/src/queue/index.ts @@ -1,3 +1,4 @@ +import { jobTrace } from "../core/log-context"; import { isQuotaFailure } from "../core/repository-failure-status"; import UserModel from "../core/model/users/users.model"; import { JobsOptions, Queue, Worker } from "bullmq"; @@ -16,6 +17,8 @@ const logger = createLogger("queue"); // ERR_BUFFER_OUT_OF_BOUNDS on long term lists / large nested fields. export interface RepoJobData { repoId: string; + requestId?: string; + logJobId?: string; } const IN_FLIGHT_STATUSES: RepositoryStatus[] = [ @@ -139,7 +142,7 @@ export async function addRemovalJob( if (LIVE_JOB_STATES.has(state)) return false; await existing.remove().catch(() => undefined); } - await queue.add(repoId, { repoId }, { ...REMOVAL_JOB_OPTIONS, jobId }); + await queue.add(repoId, { repoId, ...jobTrace() }, { ...REMOVAL_JOB_OPTIONS, jobId }); return true; } @@ -302,6 +305,22 @@ export function startWorker() { autorun: true, } ); + for (const [queue, worker] of [["cache", cacheWorker], ["remove", removeWorker], ["download", downloadWorker]] as const) { + worker.on("stalled", () => logger.error("job stalled", { queue })); + } + let sampling = false; + const timer = setInterval(async () => { + if (sampling) return; + sampling = true; + try { + for (const [queue, client] of [["cache", cacheQueue], ["remove", removeQueue], ["download", downloadQueue]] as const) { + const counts = await client.getJobCounts("waiting", "active", "delayed", "failed"); + const oldest = (await client.getJobs(["waiting"], 0, 0, true))[0]; + logger.info("queue heartbeat", { queue, ...counts, oldestWaitingMs: oldest ? Date.now() - oldest.timestamp : 0 }); + } + } catch { /* queue availability is reported through missing heartbeat alerts */ } + finally { sampling = false; } + }, 30000); timer.unref(); if (!downloadWorker.isRunning()) downloadWorker.run(); downloadWorker.on("active", async (job) => { diff --git a/src/queue/log-job.ts b/src/queue/log-job.ts new file mode 100644 index 0000000..7fbb0d6 --- /dev/null +++ b/src/queue/log-job.ts @@ -0,0 +1,28 @@ +import { logMemory } from "../core/log-memory"; +import { randomUUID } from "crypto"; +import { SandboxedJob } from "bullmq"; +import { logContext, validTraceId } from "../core/log-context"; +import { createLogger, getDroppedOutput } from "../core/logger"; +import type { RepoJobData } from "./index"; +const logger = createLogger("queue"); +export function loggedJob(queue: "download" | "remove" | "cache", work: (job: SandboxedJob) => Promise) { + return (job: SandboxedJob) => logContext.run({ + service: "worker", jobId: validTraceId(job.data.logJobId) ? job.data.logJobId : randomUUID(), + requestId: validTraceId(job.data.requestId) ? job.data.requestId : undefined, + attempt: job.attemptsMade + 1, + }, async () => { + logger.info("job started", { queue }); + const started = Date.now(); + const heartbeat = setInterval(() => { + void logMemory().then(memory => logger.info("process heartbeat", { queue, ...memory, uptime: process.uptime(), droppedLogs: getDroppedOutput() })); + }, 30000); heartbeat.unref(); + try { + await work(job); + logger.info("job completed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() }); + } catch (error) { + const expected = error instanceof Error && ["DelayedError", "RateLimitDelayError"].includes(error.name); + logger[expected ? "info" : "error"]("job failed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() }); + throw error; + } finally { clearInterval(heartbeat); } + }); +} diff --git a/src/queue/processes/downloadRepository.ts b/src/queue/processes/downloadRepository.ts index 49ac4a9..73d3488 100644 --- a/src/queue/processes/downloadRepository.ts +++ b/src/queue/processes/downloadRepository.ts @@ -1,3 +1,4 @@ +import { loggedJob } from "../log-job"; import { githubQuotaKey } from "../../core/github-token-context"; import { isQuotaFailure } from "../../core/repository-failure-status"; import { SandboxedJob, UnrecoverableError } from "bullmq"; @@ -13,7 +14,7 @@ import { DelayedError } from "bullmq"; const logger = createLogger("queue:download"); -export default async function (job: SandboxedJob) { +export default loggedJob("download", async function (job: SandboxedJob) { const { connect, getRepository, @@ -170,4 +171,4 @@ export default async function (job: SandboxedJob) { } finally { clearInterval(statusInterval); } -} +}); diff --git a/src/queue/processes/removeCache.ts b/src/queue/processes/removeCache.ts index d2ac1ef..c893207 100644 --- a/src/queue/processes/removeCache.ts +++ b/src/queue/processes/removeCache.ts @@ -1,3 +1,4 @@ +import { loggedJob } from "../log-job"; import { SandboxedJob } from "bullmq"; import { getRepository as getRepositoryImport } from "../../server/database"; import { RepoJobData } from "../index"; @@ -42,4 +43,4 @@ export function createRemoveCacheProcessor(database?: Database) { ) => processRemoveCache(job, database); } -export default createRemoveCacheProcessor(); +export default loggedJob("cache", createRemoveCacheProcessor()); diff --git a/src/queue/processes/removeRepository.ts b/src/queue/processes/removeRepository.ts index 38ae7f5..7c6f46d 100644 --- a/src/queue/processes/removeRepository.ts +++ b/src/queue/processes/removeRepository.ts @@ -1,3 +1,4 @@ +import { loggedJob } from "../log-job"; import { SandboxedJob } from "bullmq"; import { getRepository as getRepositoryImport } from "../../server/database"; import { RepositoryStatus } from "../../core/types"; @@ -61,4 +62,4 @@ export function createRemoveRepositoryProcessor(database?: Database) { ) => processRemoveRepository(job, database); } -export default createRemoveRepositoryProcessor(); +export default loggedJob("remove", createRemoveRepositoryProcessor()); diff --git a/src/server/routes/admin.ts b/src/server/routes/admin.ts index 23367a6..5c2176f 100644 --- a/src/server/routes/admin.ts +++ b/src/server/routes/admin.ts @@ -1,3 +1,4 @@ +import { jobTrace } from "../../core/log-context"; import { cacheMetricsReport } from "../../core/cache-observability"; import { sharedBlobCache, resultBlobCache } from "../../core/blob-cache"; import { AsyncCache } from "../../core/async-cache"; @@ -983,7 +984,7 @@ router.delete( const repo = await getRepo(req, res, { nocheck: true }); if (!repo) return; try { - await cacheQueue.add(repo.repoId, { repoId: repo.repoId }, { jobId: `repo-${repo.repoId}` }); + await cacheQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, { jobId: `repo-${repo.repoId}` }); return res.json({ status: repo.status }); } catch (error) { handleError(error, res, req); diff --git a/src/server/routes/repository-private.ts b/src/server/routes/repository-private.ts index 82b092f..42a23cc 100644 --- a/src/server/routes/repository-private.ts +++ b/src/server/routes/repository-private.ts @@ -1,3 +1,4 @@ +import { jobTrace } from "../../core/log-context"; import Repository from "../../core/Repository"; import { sourceStorageSize } from "../../core/source/Source"; import { maximumExpiration, neverExpirationDeadline, validateBillingExpiration } from "../../core/billing/expiration"; @@ -42,7 +43,7 @@ router.use(ensureAuthenticated); async function enqueueSavedSnapshot(model: IAnonymizedRepositoryDocument) { try { - await downloadQueue.add(model.repoId, { repoId: model.repoId }, { + await downloadQueue.add(model.repoId, { repoId: model.repoId, ...jobTrace() }, { jobId: `repo-${model.repoId}`, attempts: 3, }); } catch (error) { @@ -878,7 +879,7 @@ router.post("/", async (req, res) => { } res.send({ status: repo.status }); - downloadQueue.add(repo.repoId, { repoId: repo.repoId }, { + downloadQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, { jobId: `repo-${repo.repoId}`, attempts: 3, }); diff --git a/src/server/routes/repository-public.ts b/src/server/routes/repository-public.ts index 381ab52..04f350e 100644 --- a/src/server/routes/repository-public.ts +++ b/src/server/routes/repository-public.ts @@ -1,3 +1,4 @@ +import { jobTrace } from "../../core/log-context"; import { hasVisibleSubmodules } from "../../core/file-submodules"; import { FILE_LISTING_LIMIT } from "../../core/file-tree"; import { folderCounts as scopedFolderCounts } from "../../core/folder-counts"; @@ -342,7 +343,7 @@ router.get( repo.statusActivityDate < fiveMinuteAgo ) { await repo.updateStatus(RepositoryStatus.PREPARING); - await downloadQueue.add(repo.repoId, { repoId: repo.repoId }, { + await downloadQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, { jobId: `repo-${repo.repoId}`, attempts: 3, }); diff --git a/test/backend-reliability.test.js b/test/backend-reliability.test.js index b943bb4..4b7e82e 100644 --- a/test/backend-reliability.test.js +++ b/test/backend-reliability.test.js @@ -125,7 +125,8 @@ describe("removal recovery", function () { AnonymizedRepositoryModel.find = () => ({ lean: async () => [] }); await queueModule.recoverStuckRemoving(); expect(repo.status).to.equal("removing"); - expect(added[1]).to.deep.equal({ repoId: repo.repoId }); + expect(added[1]).to.deep.include({ repoId: repo.repoId }); + expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/); }); } @@ -164,7 +165,8 @@ describe("removal recovery", function () { let added; queueModule.removeQueue.add = async (...args) => { added = args; }; await queueModule.recoverStuckRemoving(); - expect(added[1]).to.deep.equal({ repoId: repo.repoId }); + expect(added[1]).to.deep.include({ repoId: repo.repoId }); + expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/); }); }); @@ -392,7 +394,8 @@ describe("removal workers", function () { expect(await addRemovalJob("repo-1", queue)).to.equal(true); expect(removed).to.equal(true); expect(added[0]).to.equal("repo-1"); - expect(added[1]).to.deep.equal({ repoId: "repo-1" }); + expect(added[1]).to.deep.include({ repoId: "repo-1" }); + expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/); expect(added[2].jobId).to.equal("repo-repo-1"); expect(added[2].attempts).to.equal(3); expect(added[2].removeOnFail).to.deep.equal({ count: 1000 }); diff --git a/test/credential-webhook-diagnostics.test.js b/test/credential-webhook-diagnostics.test.js index 3fe6bab..f2883fa 100644 --- a/test/credential-webhook-diagnostics.test.js +++ b/test/credential-webhook-diagnostics.test.js @@ -19,15 +19,17 @@ describe("credential and webhook failure diagnostics", () => { afterEach(() => restores.splice(0).reverse().forEach(restore => restore())); it("logs generic upstream 401 as warning while retaining server errors", () => { const warnings = [], errors = []; - stub(console, "warn", (...args) => warnings.push(args)); - stub(console, "error", (...args) => errors.push(args)); + stub(process.stdout, "write", line => { + const record = JSON.parse(line); + (record.severity === "warn" ? warnings : errors).push(line); return true; + }); handleError(Object.assign(new Error("denied"), { status: 401 }), response()); handleError(Object.assign(new Error("failed"), { response: { statusCode: 503 } }), response()); expect(warnings).to.have.length(1); expect(errors).to.have.length(1); expect(errors.flat().join(" ")).to.include('"httpStatus":503'); }); it("logs got HTTP401 as warning with its actual status", () => { - const warnings = []; stub(console, "warn", (...args) => warnings.push(args)); + const warnings = []; stub(process.stdout, "write", line => { warnings.push(line); return true; }); const failure = Object.assign(new Error("Unauthorized"), { response: { statusCode: 401 } }); Object.setPrototypeOf(failure, require("got").HTTPError.prototype); const res = response(); handleError(failure, res); @@ -51,16 +53,16 @@ describe("credential and webhook failure diagnostics", () => { it("logs signed webhook persistence failures without secrets and still returns 503", async () => { stub(config, "GITHUB_APP_ENABLED", true); stub(config, "GITHUB_APP_ID", "123"); stub(config, "GITHUB_APP_WEBHOOK_SECRET", "test-secret"); - const messages = []; stub(console, "error", (...args) => messages.push(args)); + const messages = []; stub(process.stdout, "write", line => { messages.push(line); return true; }); stub(Installation, "updateOne", async () => { throw new Error("secret-token https://private/?token=secret"); }); const body = Buffer.from(JSON.stringify({ action: "added", installation: { id: 4, app_id: 123 }, private: "private-body" })); - const req = { body, headers: { "x-github-event": "installation_repositories", "x-github-delivery": "abcd-1234", + const req = { body, headers: { "x-github-event": "installation_repositories", "x-github-delivery": "abcdef12-1234-1234-1234-abcdef123456", "x-hub-signature-256": "sha256=" + createHmac("sha256", "test-secret").update(body).digest("hex") } }; const handler = githubAppWebhook.stack.find(layer => layer.route).route.stack.at(-1).handle; const res = response(); await handler(req, res); expect(res.code).to.equal(503); expect(res.body.error).to.equal("webhook_processing_failed"); const text = JSON.stringify(messages); - expect(text).to.include("persist_installation_block").and.include("abcd-1234"); + expect(text).to.include("persist_installation_block").and.include("abcdef12-1234-1234-1234-abcdef123456"); expect(text).not.to.include("secret-token"); expect(text).not.to.include("private-body"); expect(text).not.to.include("https://"); expect(text).not.to.include(req.headers["x-hub-signature-256"]); }); diff --git a/test/download-progress-fences.test.js b/test/download-progress-fences.test.js index e5250ef..d2a44b4 100644 --- a/test/download-progress-fences.test.js +++ b/test/download-progress-fences.test.js @@ -27,7 +27,7 @@ describe("download progress publication fences", function () { it("keeps a captured publication fence valid when a queue progress tick runs", async function () { const { repo, stored } = setup(); let tick; - stub(global, "setInterval", callback => { tick = callback; return 1; }); + stub(global, "setInterval", callback => { tick = callback; return { unref() {} }; }); stub(global, "clearInterval", () => {}); stub(db, "connect", async () => {}); stub(db, "getRepository", async () => repo); diff --git a/test/folder-file-security.test.js b/test/folder-file-security.test.js index e6dee48..562f316 100644 --- a/test/folder-file-security.test.js +++ b/test/folder-file-security.test.js @@ -52,7 +52,7 @@ describe("folder file endpoint disclosure", function () { }); it("blocks a real HTTP redirect to a GitHub tree page before fetching its HTML", async () => { const warnings = []; - stub(console, "warn", (...args) => warnings.push(args.join(" "))); + stub(process.stdout, "write", line => { warnings.push(line); return true; }); const server = await listen(http.createServer((_req, res) => { res.writeHead(302, { location: "https://github.com/example/private-repo/tree/main/folder?token=secret-query" }); res.end(); })); const original = got.stream; stub(got, "stream", (_url, options) => original(`http://127.0.0.1:${server.address().port}/raw`, options)); diff --git a/test/repository-refresh.test.js b/test/repository-refresh.test.js index 3e63db9..8d54153 100644 --- a/test/repository-refresh.test.js +++ b/test/repository-refresh.test.js @@ -84,7 +84,8 @@ describe("repository refresh and restoration", () => { expect(response.status).to.equal("preparing"); expect(repo.model.source.commit).to.equal(commit); expect(repo.repoId).to.equal("restore-me"); - expect(added[1]).to.deep.equal({ repoId: "restore-me" }); + expect(added[1]).to.deep.include({ repoId: "restore-me" }); + expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/); }); } diff --git a/test/storage-error-origin.test.js b/test/storage-error-origin.test.js index e93d860..f9e8f90 100644 --- a/test/storage-error-origin.test.js +++ b/test/storage-error-origin.test.js @@ -9,17 +9,16 @@ require("../src/core/storage"); const FileSystem = require("../src/core/storage/FileSystem").default; describe("cache write error origins", function () { - let directory, previousFolder, messages, previousWarn, previousError; + let directory, previousFolder, messages, previousWrite; beforeEach(function () { directory = fs.mkdtempSync(path.join(os.tmpdir(), "cache-errors-")); previousFolder = config.FOLDER; config.FOLDER = directory; messages = []; - previousWarn = console.warn; previousError = console.error; - console.warn = (...args) => messages.push(["warn", args.join(" ")]); - console.error = (...args) => messages.push(["error", args.join(" ")]); + previousWrite = process.stdout.write; + process.stdout.write = line => { const record = JSON.parse(line); messages.push([record.severity, record.event]); return true; }; }); afterEach(function () { - console.warn = previousWarn; console.error = previousError; + process.stdout.write = previousWrite; config.FOLDER = previousFolder; fs.rmSync(directory, { recursive: true, force: true }); }); @@ -34,8 +33,8 @@ describe("cache write error origins", function () { let caught; try { await storage.write("repo", "file.txt", input); } catch (error) { caught = error; } expect(caught).to.equal(failure); - expect(messages.some(([level, text]) => level === "warn" && text.includes("source stream failed"))).to.equal(true); - expect(messages.some(([level, text]) => level === "error" && text.includes("write failed"))).to.equal(false); + expect(messages.some(([level, text]) => level === "warn" && text.includes("source_stream_failed"))).to.equal(true); + expect(messages.some(([level, text]) => level === "error" && text.includes("write_failed"))).to.equal(false); expect(fs.readFileSync(path.join(directory, "repo/original/file.txt"), "utf8")).to.equal("previous"); expect(fs.readdirSync(path.join(directory, "repo/original"))).to.deep.equal(["file.txt"]); }); @@ -45,7 +44,7 @@ describe("cache write error origins", function () { let caught; try { await storage.write("repo", "file.txt", "new content"); } catch (error) { caught = error; } expect(caught).to.be.instanceOf(Error); - expect(messages.some(([level, text]) => level === "error" && text.includes("write failed"))).to.equal(true); + expect(messages.some(([level, text]) => level === "error" && text.includes("write_failed"))).to.equal(true); expect(messages.some(([level]) => level === "warn")).to.equal(false); expect(fs.readdirSync(path.join(directory, "repo/original"))).to.deep.equal(["file.txt"]); }); diff --git a/test/structured-logging.test.js b/test/structured-logging.test.js new file mode 100644 index 0000000..ae457e7 --- /dev/null +++ b/test/structured-logging.test.js @@ -0,0 +1,104 @@ +const { setTimeout } = require("node:timers"); +const { expect } = require('chai'); +const { randomUUID } = require('node:crypto'); +const { spawn, spawnSync } = require('node:child_process'); +require('ts-node/register/transpile-only'); +const { logRecord } = require('../src/core/log-record'); +const { logContext, jobTrace } = require('../src/core/log-context'); +const { loggedJob } = require('../src/queue/log-job'); + +describe('structured logging privacy and context', () => { + it('keeps the production summary compatible with JSON and legacy records', () => { + const { logSummary, observeLine } = require('../scripts/check-production-logs'); + const summary = logSummary(); + observeLine(summary, JSON.stringify(logRecord('info', 'requests', ['request', {ms: 1, status: 503, outcome: 'completed'}]))); + observeLine(summary, JSON.stringify(logRecord('error', 'process', ['process fatal error']))); + observeLine(summary, '2026-01-01T00:00:00Z WARN [requests] request {"ms":2,"status":429,"outcome":"completed"}'); + expect(summary).to.include({completed: 2, completed5xx: 1, fatalErrors: 1, errors: 1, warnings: 1, rateLimited: 1}); + }); + it('omits arbitrary text, private identities, request bodies and nested secrets', () => { + const secret = 'private-owner/private-repository?state=secret'; + const record = logRecord('error', 'upstream', [secret, { + message: secret, url: secret, headers: { authorization: secret }, repoId: secret, + requestBody: secret, status: 502, code: 'ETIMEDOUT', stages: { upstream: 42, [secret]: 1 }, + }]); + expect(JSON.stringify(record)).not.to.include(secret); + expect(record).to.include({ status: 502, errorCode: 'ETIMEDOUT', stage_upstream_ms: 42 }); + expect(record).to.have.keys('timestamp', 'severity', 'service', 'instance', 'release', 'event', 'message', 'module', 'status', 'errorCode', 'stage_upstream_ms'); + }); + it('keeps concurrent request contexts separate and propagates an opaque job trace', async () => { + const ids = [randomUUID(), randomUUID()]; + const rows = await Promise.all(ids.map((requestId, i) => logContext.run({ requestId }, async () => { + await new Promise(resolve => setTimeout(resolve, 10 - i)); + const trace = jobTrace(); + expect(trace.requestId).to.equal(requestId); + expect(trace.logJobId).not.to.equal(requestId); + return logRecord('info', 'fixture', ['request']); + }))); + expect(rows.map(row => row.requestId)).to.deep.equal(ids); + expect(logRecord('info', 'fixture', ['request'])).not.to.have.property('requestId'); + }); + it('sets sandbox job context and preserves success/failure semantics', async () => { + const requestId = randomUUID(), jobId = randomUUID(); + const processor = loggedJob('download', async () => { + expect(logContext.getStore()).to.deep.equal({ service: 'worker', requestId, jobId, attempt: 3 }); + throw new Error('fixture failure'); + }); + try { await processor({ data: { repoId: 'private-id', requestId, logJobId: jobId }, attemptsMade: 2 }); } + catch (error) { expect(error.message).to.equal('fixture failure'); return; } + throw new Error('job failure was swallowed'); + }); + it('rejects injected correlation IDs and non-finite measurements', () => { + const row = logRecord('warn', 'fixture', ['request', { requestId: 'bad\nsecret', jobId: 'private', ms: Infinity, status: NaN }]); + expect(row).not.to.have.any.keys('requestId', 'jobId', 'ms', 'status'); + }); + it('survives a reader closing stdout', async function () { + this.timeout(10000); + const child = spawn(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', ` + const { createLogger, getDroppedOutput } = require('./src/core/logger'); + const logger = createLogger('process'); + process.on('message', () => { + logger.info('process heartbeat'); + setTimeout(() => { logger.info('process heartbeat'); process.send(getDroppedOutput()); process.disconnect(); }, 50); + }); + process.send('ready'); + `], { env: { ...process.env, LOG_REDIS_MIRROR: 'off' }, stdio: ['ignore', 'pipe', 'pipe', 'ipc'] }); + let drops, fallback = ""; + child.stderr.on("data", chunk => { fallback += chunk; }); + child.on('message', message => { + if (message === 'ready') { child.stdout.destroy(); child.send('write'); } + else drops = message; + }); + const code = await new Promise((resolve, reject) => { child.on('exit', resolve); child.on('error', reject); }); + expect(code).to.equal(0); + expect(drops).to.be.greaterThan(0); + expect(fallback).to.include('"event":"process_heartbeat"'); + }); + it('preserves admin error codes and expected-error ring suppression', () => { + const child = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', ` + const redis = require('redis'); + let pushes = 0; const fields = []; + const tx = {lPush() {pushes++; return tx;}, lTrim() {return tx;}, + hIncrBy(key, field) { fields.push(field); return tx; }, expire() {return tx;}, exec() {return Promise.resolve([]);} }; + redis.createClient = () => ({isReady:true, isOpen:true, on() {}, connect() {return Promise.resolve();}, multi() {return tx;}}); + require('./src/core/logger').createLogger('route').warn('repository_not_ready', {message:'repository_not_ready', httpStatus:409}); + if (pushes !== 0 || !fields.includes('cb:warn:repository_not_ready')) process.exitCode = 1; + `], {env: {...process.env, LOG_REDIS_MIRROR: 'on'}, encoding: 'utf8', timeout: 10000}); + expect(child.status, child.stderr).to.equal(0); + }); + it('writes JSON without Redis and bounds a blocked stdout pipe', () => { + const child = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', ` + const { createLogger, getDroppedOutput } = require('./src/core/logger'); + const logger = createLogger('fixture'); + logger.info('request', {status: 200, requestBody: 'secret-fixture'}); + Object.defineProperty(process.stdout, 'writableLength', {value: 300000}); + for (let i = 0; i < 10000; i++) logger.warn('request', {status: 429}); + if (getDroppedOutput() !== 10000) process.exitCode = 1; + `], { cwd: process.cwd(), env: { ...process.env, LOG_REDIS_MIRROR: 'off' }, encoding: 'utf8', timeout: 10000 }); + expect(child.status, child.stderr).to.equal(0); + const lines = child.stdout.trim().split('\n'); + expect(lines).to.have.length(1); + expect(JSON.parse(lines[0])).to.include({ event: 'request_completed', status: 200 }); + expect(child.stdout).not.to.include('secret-fixture'); + }); +});