feat: add private structured logging and investigation dashboards

This commit is contained in:
Codex committed 2026-11-14 13:40:40 +00:00
1 parent b456ab8739
commit 7e2a6e9d75
39 files changed
+1772 -71

No files matched your search

+208
View File
@@ -0,0 +1,208 @@
# Operations logging
This is an opt-in stack for one Docker host. It does not deploy itself, change
production, or send notifications. The application writes structured JSON to
stdout. Vector tails tagged Docker json-file logs and host Nginx access JSON,
rebuilds each record from an allowlist, then spools sanitized records to disk.
VictoriaLogs stores searchable compressed records. Grafana provides investigation
panels. Prometheus monitors the logging components and vmalert evaluates LogsQL
alerts. Alertmanager groups firing and resolved state with an empty receiver.
## Deploy after approval
Review the application and infrastructure changes together. This PR is stacked
on `fix/remaining-production-findings`; do not deploy it without that base.
1. Reserve at least 20 GiB of dedicated logging storage and 2 GiB of RAM, outside
the application and MongoDB budgets. The configured memory ceilings total
1,856 MiB. Check real free memory before enabling this stack. Docker volumes
survive container replacement, but not deletion of the volumes or host loss.
2. Create a strong Grafana password in an external secret file. Give UID 472 read
access and restrict the parent directory to operators. Do not place the file
in the repository. Export `GRAFANA_PASSWORD_FILE` as its absolute path and
`GRAFANA_ADMIN_USER` as a non-default username. No password value belongs in
a shell command, Compose file, issue, or PR.
3. Set `APP_RELEASE` to the deployed 7 to 40 character Git SHA. Build the app through
the existing deployment workflow. Add the override when recreating services:
`docker compose -f docker-compose.yml -f deploy/logging/app.compose.yml up -d`.
This applies explicit service tags and nonblocking 10 MiB × 3 local buffers to
the API, all four streamers, MongoDB, Redis and the backup process. BullMQ
sandbox workers log through the API container. Existing containers need
recreation to change their Docker log configuration.
4. Include `../nginx/anonymous-logging.conf` in the host Nginx `http` context and
configure the relevant server with
`access_log /var/log/nginx/anonymous.access.json anonymous_json;`.
Test with `nginx -t`, then reload through the existing deployment workflow.
Keep Nginx `real_ip` and proxy trust unchanged. Do not enable Mongo profiling,
query tracing, Redis MONITOR, or request-body logging.
5. Install `nginx.logrotate` through the host configuration workflow and run it
hourly with its own state file. Keep logs root/adm-readable only. This retains
about six hourly files, with a 10 MiB rotation trigger checked hourly. A burst
can exceed 10 MiB between checks; this is not a hard file quota. Use a dedicated
filesystem quota if hard local containment is needed. Use rename/reopen, not
copytruncate. Vector excludes compressed history and reads new files from end.
6. Set `DOCKER_LOG_DIR` if Docker's data root differs from `/var/lib/docker/containers`,
and `NGINX_LOG_DIR` if needed. The collector mounts these directories read-only;
it has no Docker socket, host PID namespace or ability to restart applications.
Start with `docker compose -f deploy/logging/compose.yml up -d`.
7. Tunnel Grafana: `ssh -L 3300:127.0.0.1:3300 your-logging-host`. Open
`http://127.0.0.1:3300`, sign in, and open **Anonymous GitHub operations**.
Grafana is the only published port, bound to loopback. Ingest, query,
Prometheus and Alertmanager are private container-network services. Do not
publish their ports or put them behind the public application proxy.
8. Verify a synthetic request appears across API/streamer using its response
`X-Request-ID`, inspect all four streamer heartbeats and three queue heartbeats,
and inspect `ALERTS` in Grafana. Check Docker restart counts and all scrape
targets. Observe memory and ingestion drops before increasing traffic.
Images are version-and-digest pinned in `compose.yml`. Grafana installs the
signed VictoriaLogs datasource version 0.34.0 on startup; initial installation
needs outbound HTTPS. Automatic plugin upgrades are disabled. For an offline
host, preinstall that exact signed plugin in a staged image and validate it before
removing the plugin-download network. Never enable unsigned plugins.
## Privacy and correlation
`src/core/log-schema.ts` contains reviewed literal module, event and error codes.
Unknown messages become `application_log`; arbitrary text is not retained.
`log-record.ts` permits numeric measurements, fixed diagnostic values and UUIDs.
URLs, paths, repository IDs, owner IDs, cookies, headers, request/response bodies,
OAuth query parameters and stacks are omitted. Extend the application schema and
collector allowlist together when a new diagnostic is needed. Do not turn unknown
strings into labels. Raw Node fatal stderr and dependency output can still reach
short local Docker buffers; the collector never forwards their raw text.
Requests get server-created IDs at the API boundary. Existing API-to-streamer
header propagation is unchanged, including its validation. IDs are diagnostics,
not authentication or proof of origin. New download, removal and cache jobs carry the
request UUID and a random job UUID. Sandbox execution adds attempt and worker
service context. Older jobs get a new opaque job UUID
when executed. Their historical enqueue request cannot be reconstructed.
Only `service,severity` identify VictoriaLogs streams. Request, job, instance,
release and edge IDs are searchable fields, not stream or metric labels.
Nginx edge IDs link to application IDs; Cloudflare ray IDs are accepted only in a
fixed format. MongoDB contributes numeric event ID and severity, never `msg`,
`attr`, command or query text. Redis and backup output contributes presence only.
Malformed tagged records become a metadata event. Oversized lines are discarded
with collector error counters. There is no sampling of accepted records.
Central records omit IP addresses entirely, including unsalted IP hashes.
Restricted host Nginx buffers preserve IP evidence for a short incident window.
Do not attach those files to issues or PRs. Inspect them locally as an authorized
operator. Extending IP retention or exporting IPs needs a separate reviewed change.
## Budgets and failure behavior
VictoriaLogs has seven-day retention, a 10 GiB retention target and a 5 GiB free
space reserve. It deletes old daily partitions, so the target is not a hard quota:
current-day data, merges and indexes can exceed it. Put `/storage` on a dedicated
filesystem or apply a host filesystem quota for hard isolation. Reserve 20 GiB
for this stack and alert at 8 GiB used or under 8 GiB free. Keep application data
on a different filesystem for protection from a logging flood. At the free-space
reserve, ingest stops and Vector queues sanitized records.
Vector's persistent disk buffer is 256 MiB, with `drop_newest`. Its files and
checkpoints survive collector restarts. Allocation and metadata add overhead;
budget 512 MiB on disk. Outages cause retries with one in-flight request. Once
the buffer fills, new records are lost; applications do not wait. Vector syncs
the buffer periodically, so sudden power loss can lose the latest records.
Collector reads are not a transaction with Docker rotation: downtime beyond the
local buffers loses events. Delivery can duplicate records after a retry. There
is no exactly-once or host-failure durability claim.
The application drops stdout writes above 256 KiB pending, counts losses in its
heartbeat and handles a broken pipe without crashing. A broken stdout switches to bounded
stderr output so later worker heartbeats can still report drops. If both pipes
fail, that process cannot report its own telemetry. Docker's nonblocking
1 MiB per-container buffer can also drop independently, without an application
counter. Compare edge traffic against request completions during overload.
Each application container retains at most roughly 30 MiB of Docker files,
excluding an oversized single record and filesystem overhead. Logging-stack
containers use 5 MiB × 2 files each. Prometheus retains seven days or 512 MB of
blocks; WAL/head space needs additional headroom. Alertmanager/Grafana volumes
also need space. No compression ratio is promised for production data.
Admin error pages keep their existing Redis ring and counters by default. Their
payloads now contain safe structured diagnostics, so private URLs and stacks are
no longer available there. The mirror has no offline queue and a bounded command
queue. Once operators use Grafana, `LOG_REDIS_MIRROR=off` removes per-warning
Redis writes and ring duplication. This deliberately stops new admin error
history; existing history remains until normal expiry/overwrite or an approved
cleanup. Leave it on if that admin feature is required. Request/performance
metrics and queue admin functions are unchanged.
## Investigation
Dashboard panels separate first-byte delay, transfer time and named stages;
show client cancellations separately; compare 5xx and latency by release; and
show queue waiting age, job failures, memory and process starts. Paste a response
UUID into Request UUID, or a job UUID into Job UUID. Use the edge panel to find
the application ID for an edge request, then paste that ID into Request UUID.
Useful LogsQL queries, with the Grafana time picker limiting scope:
```text
requestId:="REQUEST_UUID"
jobId:="JOB_UUID"
event:process_fatal_error OR event:process_started
errorCode:github_rate_limit_exceeded OR errorCode:github_app_reconnect_required
service:nginx status:429 | stats by (route, limit_req_status) count() as requests
service:api outcome:completed | stats by (release) quantile(0.95, firstByteMs) as p95_ms
module:github-app-webhook | stats by (phase, upstreamStatus) count() as failures
```
The alert rules distinguish completed server failures from cancellations and
expected 4xx. 5xx require both a minimum count and a ratio sustained for five
minutes. Rate-limited traffic stays visible without paging. Queue age ignores
intentional delayed jobs. Telemetry alerts expect one API, four streamers and
three queues; update the streamer threshold during planned scaling. Memory
pressure uses cgroup usage including workers/page cache, with process RSS retained
for diagnosis. Missing telemetry is necessary because dead processes cannot emit
a final error. A logging-host outage also defeats same-host alerts; independent
external monitoring must be configured separately. No receiver is enabled here.
## Separate logging host and rollback
For host-failure durability, run VictoriaLogs, Grafana, Prometheus, vmalert and
Alertmanager on a separate host with encrypted backups of their persistent volumes.
Keep Vector and its local spool on the application host. Change the Vector sink
endpoint to a TLS-authenticated private reverse proxy or a restricted VPN address;
permit only collector ingress, never public ingestion. Use certificate files or
Docker secrets for credentials, not committed values. Scrape Vector over that
same private network. This repository's same-host Compose does not claim to set
up that network, certificates, backup policy or authentication proxy for you.
Test the remote arrangement with synthetic records and a network outage before
cutover. Backups must preserve the same access restrictions as live logs.
Rollback the application image and remove `app.compose.yml` from the deployment
command, then recreate application containers using the prior logging settings.
Stop the logging stack with `docker compose -f deploy/logging/compose.yml down`
without `-v`; that preserves logs, spool, dashboards and alert state. Restore the
prior Nginx logging configuration through the host workflow and test before
reloading. Keep any retained data private until its approved deletion date.
Re-enable `LOG_REDIS_MIRROR=on` if it was disabled. None of these steps requires
changing MongoDB data or application queues. Do not delete volumes as rollback.
## Local validation and references
Run `npm ci`, `npx tsc --noEmit`, and
`LOG_REDIS_MIRROR=off npx mocha test/structured-logging.test.js test/request-monitoring.test.js`.
Run `python3 scripts/test-logging.py --stress` with Docker Compose and PyYAML installed.
It creates a random project and synthetic files, opens temporary loopback test
ports, and removes only its own containers/volumes. It tests actual ingestion,
privacy, correlation, query syntax, rotation, restart, backend outage and recovery,
retention rejection and fixture storage size. It never reads production logs or
uses the application `.env`. The optional `--stress` writes 500,000 synthetic records while the backend is
stopped, checks the buffer bound and collector liveness, and waits for the real
CollectorDrops alert to fire. Allow 1 GiB temporary disk and several minutes.
These fixtures do not establish sustained production throughput.
Official references checked 2026-11-14:
- [VictoriaLogs retention and resource flags](https://docs.victoriametrics.com/victorialogs/)
- [Vector ingestion setup](https://docs.victoriametrics.com/victorialogs/data-ingestion/vector/)
- [Vector buffering guarantees and loss behavior](https://vector.dev/docs/architecture/buffering-model/)
- [VictoriaLogs metrics](https://docs.victoriametrics.com/victorialogs/metrics/)
- [vmalert LogsQL integration](https://docs.victoriametrics.com/victorialogs/vmalert/)
- [Grafana plugin provisioning](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/)
+10
View File
@@ -0,0 +1,10 @@
route:
receiver: local-only
group_by: [alertname, service, queue]
group_wait: 1m
group_interval: 5m
repeat_interval: 4h
receivers:
- name: local-only
# Deliberately no email, webhook, chat or pager receiver. Firing/resolved state
# remains available in the private UI. Add a receiver only after approval.
+35
View File
@@ -0,0 +1,35 @@
# Add after docker-compose.yml. Recreating containers is a deployment action.
x-buffer: &buffer
driver: json-file
options: &options
max-size: '10m'
max-file: '3'
mode: non-blocking
max-buffer-size: '1m'
services:
anonymous_github:
logging:
<<: *buffer
options: {<<: *options, tag: api}
environment:
APP_RELEASE: ${APP_RELEASE:-unknown}
LOG_REDIS_MIRROR: ${LOG_REDIS_MIRROR:-on}
streamer:
logging:
<<: *buffer
options: {<<: *options, tag: streamer}
environment:
APP_RELEASE: ${APP_RELEASE:-unknown}
LOG_REDIS_MIRROR: ${LOG_REDIS_MIRROR:-on}
mongodb:
logging:
<<: *buffer
options: {<<: *options, tag: mongodb}
redis:
logging:
<<: *buffer
options: {<<: *options, tag: redis}
mongodb-backup:
logging:
<<: *buffer
options: {<<: *options, tag: mongodb-backup}
+94
View File
@@ -0,0 +1,94 @@
name: anonymous-observability
x-common: &common
restart: unless-stopped
logging:
driver: json-file
options: {max-size: 5m, max-file: '2', mode: non-blocking, max-buffer-size: 1m}
security_opt: [no-new-privileges:true]
cap_drop: [ALL]
services:
victorialogs:
<<: *common
image: victoriametrics/victoria-logs:v1.54.0@sha256:ee4e820cac3721f5f5d91641b950c22439593adc4224f9655e5333d5a84c1dfe
command:
- -storageDataPath=/storage
- -retentionPeriod=7d
- -retention.maxDiskSpaceUsageBytes=10GiB
- -storage.minFreeDiskSpaceBytes=5GiB
- -memory.allowedBytes=512MiB
volumes: [logs:/storage]
mem_limit: 768m
cpus: 1.0
networks: [private]
vector:
<<: *common
image: timberio/vector:0.61.0-debian@sha256:f6c23b5a7e16fb2838d33619ef7acf4bad2289a7cf22bc90efc2371f520c4ea2
command: [--config, /etc/vector/vector.yaml]
# File access only: no Docker API socket and no host PID namespace.
user: '0:0'
volumes:
- ./vector.yaml:/etc/vector/vector.yaml:ro
- ./sanitize.vrl:/etc/vector/sanitize.vrl:ro
- ${DOCKER_LOG_DIR:-/var/lib/docker/containers}:/host/docker:ro
- ${NGINX_LOG_DIR:-/var/log/nginx}:/host/nginx:ro
- spool:/var/lib/vector
environment:
VECTOR_LOG: warn
mem_limit: 256m
cpus: 0.5
networks: [private]
grafana:
<<: *common
image: grafana/grafana:13.3.1@sha256:0615ec4fbd4072004b7d86b46ded63013a74b7d6eb1119273dd89e8fa25c13cf
ports: ['127.0.0.1:${GRAFANA_PORT:-3300}:3000']
environment:
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?Set a non-default admin username}
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_password
GF_USERS_ALLOW_SIGN_UP: 'false'
GF_AUTH_ANONYMOUS_ENABLED: 'false'
GF_ANALYTICS_REPORTING_ENABLED: 'false'
GF_ANALYTICS_CHECK_FOR_UPDATES: 'false'
GF_PLUGINS_PREINSTALL_AUTO_UPDATE: 'false'
GF_PLUGINS_DISABLE_PLUGINS: grafana-lokiexplore-app,grafana-metricsdrilldown-app,grafana-exploretraces-app,grafana-pyroscope-app,grafana-pathfinder-app
GF_PLUGINS_PREINSTALL_SYNC: victoriametrics-logs-datasource@0.34.0
secrets: [grafana_password]
volumes:
- grafana:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
mem_limit: 384m
networks: [private, plugin_download]
prometheus:
<<: *common
image: prom/prometheus:v3.16.0@sha256:e04987641b57704f408e31f6373ba4138ecf27cb4a165d24aabb6c38fddd0f06
command: [--config.file=/etc/prometheus/prometheus.yml, --storage.tsdb.path=/prometheus, --storage.tsdb.retention.time=7d, --storage.tsdb.retention.size=512MB, --web.enable-remote-write-receiver]
volumes: [metrics:/prometheus, './prometheus.yml:/etc/prometheus/prometheus.yml:ro', './metrics-alerts.yml:/etc/prometheus/alerts.yml:ro']
mem_limit: 256m
networks: [private]
vmalert:
<<: *common
image: victoriametrics/vmalert:v1.156.0@sha256:4fea7d1a888d4db4ff18804434ef0f3ca8eb6482e289c50d32001c2db218287c
command: [-rule=/etc/alerts.yml, -datasource.url=http://victorialogs:9428, -notifier.url=http://alertmanager:9093, -remoteWrite.url=http://prometheus:9090, -remoteRead.url=http://prometheus:9090]
volumes: ['./log-alerts.yml:/etc/alerts.yml:ro']
mem_limit: 128m
networks: [private]
alertmanager:
<<: *common
image: prom/alertmanager:v0.35.0@sha256:e638797febd034838eca8b98cdbd46f951e9bf7f178777aa8669167406a9946c
command: [--config.file=/etc/alertmanager.yml, --storage.path=/alertmanager, --cluster.listen-address=]
volumes: [alerts:/alertmanager, './alertmanager.yml:/etc/alertmanager.yml:ro']
mem_limit: 64m
networks: [private]
secrets:
grafana_password:
file: ${GRAFANA_PASSWORD_FILE:?Set path to an existing secret file}
networks:
private:
internal: true
plugin_download: {}
volumes:
logs: {}
spool: {}
grafana: {}
metrics: {}
alerts: {}
@@ -0,0 +1,467 @@
{
"uid": "anonymous-operations",
"title": "Anonymous GitHub operations",
"schemaVersion": 39,
"version": 1,
"refresh": "30s",
"time": {
"from": "now-1h",
"to": "now"
},
"tags": [
"operations"
],
"templating": {
"list": [
{
"name": "request_id",
"label": "Request UUID",
"type": "textbox",
"current": {
"text": "",
"value": ""
}
},
{
"name": "job_id",
"label": "Job UUID",
"type": "textbox",
"current": {
"text": "",
"value": ""
}
},
{
"name": "edge_id",
"label": "Edge request ID",
"type": "textbox",
"current": {
"text": "",
"value": ""
}
}
]
},
"panels": [
{
"id": 1,
"title": "Failures and cancellations",
"type": "logs",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 0,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "severity:error OR event:request_cancelled",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 2,
"title": "First byte versus transfer time",
"type": "table",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 12,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "event:request* | stats by (service, route) quantile(0.95, firstByteMs) as first_byte_p95_ms, quantile(0.95, responseMs) as transfer_p95_ms",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 3,
"title": "Request stages",
"type": "table",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 0,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "event:request* | stats by (service) avg(stage_upstream_ms) as upstream_ms, avg(stage_worker_wait_ms) as worker_wait_ms, avg(stage_anonymize_ms) as anonymize_ms",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 4,
"title": "Traffic and rate limits",
"type": "table",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 12,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "service:nginx | stats by (status, limit_req_status) count() as requests",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 5,
"title": "Authentication versus upstream failures",
"type": "table",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 0,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "errorCode:* OR status:401 OR status:403 OR status:>=500 | stats by (service, errorCode, status) count() as events",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 6,
"title": "Release comparison",
"type": "table",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 12,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "service:api outcome:completed | stats by (release) count() as requests, count() if (status:>=500) as server_errors, quantile(0.95, firstByteMs) as first_byte_p95_ms",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 7,
"title": "Crashes and restarts",
"type": "logs",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 0,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "event:process_fatal_error OR event:process_started",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 8,
"title": "Queue stalls and failures",
"type": "logs",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 12,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "event:job_stalled OR event:job_failed OR event:queue_heartbeat",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 9,
"title": "Memory and lost application logs",
"type": "table",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 0,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "event:process_heartbeat | stats by (service, instance) max(rss) as rss, max(memoryRatio) as memory_ratio, max(droppedLogs) as dropped_logs",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 10,
"title": "Request correlation",
"type": "logs",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 12,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "requestId:=\"$request_id\"",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 11,
"title": "Job correlation",
"type": "logs",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 0,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "jobId:=\"$job_id\"",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 12,
"title": "Edge to application correlation",
"type": "logs",
"datasource": {
"type": "victoriametrics-logs-datasource",
"uid": "anonymous-logs"
},
"gridPos": {
"x": 12,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "edge_request_id:=\"$edge_id\" OR requestId:=\"$request_id\"",
"queryType": "instant",
"editorMode": "code",
"maxLines": 200
}
],
"options": {
"showTime": true,
"sortOrder": "Descending"
}
},
{
"id": 20,
"title": "Storage bytes",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "logging-health"
},
"gridPos": {
"x": 0,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(vl_data_size_bytes)"
}
]
},
{
"id": 21,
"title": "Free disk bytes",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "logging-health"
},
"gridPos": {
"x": 12,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "min(vl_free_disk_space_bytes)"
}
]
},
{
"id": 22,
"title": "Collector discarded events",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "logging-health"
},
"gridPos": {
"x": 0,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "(sum(rate(vector_buffer_discarded_events_total[5m])) or vector(0)) + (sum(rate(vector_component_discarded_events_total{intentional!=\"true\"}[5m])) or vector(0))"
}
]
},
{
"id": 23,
"title": "Firing alerts",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "logging-health"
},
"gridPos": {
"x": 12,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "ALERTS{alertstate=\"firing\"}"
}
]
}
]
}
@@ -0,0 +1,6 @@
apiVersion: 1
providers:
- name: Anonymous GitHub operations
type: file
allowUiUpdates: false
options: {path: /var/lib/grafana/dashboards}
@@ -0,0 +1,22 @@
apiVersion: 1
datasources:
- name: Logs
uid: anonymous-logs
type: victoriametrics-logs-datasource
access: proxy
url: http://victorialogs:9428
isDefault: true
editable: false
- name: Logging health
uid: logging-health
type: prometheus
access: proxy
url: http://prometheus:9090
editable: false
- name: Alert state
uid: logging-alerts
type: alertmanager
access: proxy
url: http://alertmanager:9093
jsonData: {implementation: prometheus}
editable: false
+55
View File
@@ -0,0 +1,55 @@
groups:
- name: application-logs
type: vlogs
interval: 30s
rules:
- alert: SustainedServerErrors
expr: '_time:5m service:api outcome:completed | stats count() as total, count() if (status:>=500) as errors | filter errors:>=10 | math errors / total as ratio | filter ratio:>0.05 | fields ratio'
for: 5m
labels: {severity: warning}
annotations: {summary: 'API 5xx exceed 5 percent and 10 requests per five minutes'}
- alert: ApiTelemetryMissing
expr: '_time:3m service:api event:process_heartbeat | stats count() as heartbeats | filter heartbeats:0'
for: 3m
labels: {severity: warning}
annotations: {summary: 'API heartbeat is missing'}
- alert: StreamerTelemetryMissing
expr: '_time:3m service:streamer event:process_heartbeat | stats count_uniq(instance) as instances | filter instances:<4'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Fewer than four streamer instances report heartbeats; adjust for planned scaling'}
- alert: ApplicationRestartBurst
expr: '_time:10m event:process_started | stats by (service) count() as restarts | filter restarts:>4'
for: 2m
labels: {severity: warning}
annotations: {summary: 'Repeated process starts in ten minutes'}
- alert: ApplicationMemoryPressure
expr: '_time:3m event:process_heartbeat | stats by (service) max(memoryRatio) as ratio | filter ratio:>0.85'
for: 5m
labels: {severity: warning}
annotations: {summary: 'Container usage exceeds 85 percent of its cgroup memory limit'}
- alert: QueueStalled
expr: '_time:5m event:job_stalled | stats by (queue) count() as stalls | filter stalls:>0'
for: 2m
labels: {severity: warning}
annotations: {summary: 'BullMQ reports stalled jobs'}
- alert: QueueWaitingTooLong
expr: '_time:3m event:queue_heartbeat | stats by (queue) max(oldestWaitingMs) as age | filter age:>600000'
for: 5m
labels: {severity: warning}
annotations: {summary: 'A runnable job has waited over ten minutes'}
- alert: QueueTelemetryMissing
expr: '_time:3m event:queue_heartbeat | stats count_uniq(queue) as queues | filter queues:<3'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Queue telemetry is missing'}
- alert: WebhookFailures
expr: '_time:5m route:"/github/app/webhook" status:>=500 outcome:completed | stats count() as errors | filter errors:>=3'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Repeated webhook server failures; signature rejection is excluded'}
- alert: ApplicationLogDrops
expr: '_time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service) max(droppedLogs) as drops | filter drops:>0'
for: 2m
labels: {severity: warning}
annotations: {summary: 'An application process dropped stdout logs since startup'}
+33
View File
@@ -0,0 +1,33 @@
groups:
- name: logging-infrastructure
rules:
- alert: LoggingTelemetryMissing
expr: up == 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'A logging component cannot be scraped'}
- alert: CollectorDrops
expr: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
for: 2m
labels: {severity: warning}
annotations: {summary: 'Collector dropped events; inspect buffer and transform counters'}
- alert: CollectorErrors
expr: sum(increase(vector_component_errors_total[5m])) > 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'Collector errors persist'}
- alert: LogStoragePressure
expr: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
for: 5m
labels: {severity: warning}
annotations: {summary: 'Log storage is near the retention budget or free-space reserve'}
- alert: LogIngestionRejected
expr: sum(increase(vl_rows_dropped_total[5m])) > 0
for: 2m
labels: {severity: warning}
annotations: {summary: 'VictoriaLogs rejected records'}
- alert: LogAlertEvaluationFailed
expr: sum(increase(vmalert_execution_errors_total[5m])) > 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'Log alert queries are failing'}
+15
View File
@@ -0,0 +1,15 @@
# Run this configuration hourly, using a dedicated logrotate state file.
/var/log/nginx/anonymous.access.json {
hourly
rotate 6
maxsize 10M
missingok
notifempty
compress
delaycompress
create 0640 www-data adm
sharedscripts
postrotate
/usr/sbin/nginx -s reopen
endscript
}
+21
View File
@@ -0,0 +1,21 @@
global:
scrape_interval: 30s
evaluation_interval: 30s
rule_files: [/etc/prometheus/alerts.yml]
alerting:
alertmanagers:
- static_configs:
- targets: [alertmanager:9093]
scrape_configs:
- job_name: victorialogs
static_configs:
- targets: [victorialogs:9428]
- job_name: collector
static_configs:
- targets: [vector:9598]
- job_name: log-alerts
static_configs:
- targets: [vmalert:8880]
- job_name: prometheus
static_configs:
- targets: [localhost:9090]
+61
View File
@@ -0,0 +1,61 @@
# Rebuild the event from scratch BEFORE the disk buffer. Never retain message,
# Docker metadata, Mongo attr/command, request path, host, UA, cookie or headers.
raw, err = parse_json(string!(.message))
if err != null { raw = {} }
service = "nginx"
record = raw
if starts_with(string!(.file), "/host/docker/") {
service = string(raw.attrs.tag) ?? "unknown"
if !includes(["api", "streamer", "mongodb", "redis", "mongodb-backup"], service) { abort }
record, err = parse_json(string(raw.log) ?? "")
if err != null { record = {} }
}
out = {"timestamp": now(), "service": service, "severity": "info", "event": "service_output"}
# Use capture time for unsupported formats. Invalid/old producer timestamps
# cannot defeat retention or crash the collector.
event_time = raw.time
if exists(record.time) { event_time = record.time }
if exists(record.timestamp) { event_time = record.timestamp }
parsed, err = parse_timestamp(string(event_time) ?? "", format: "%+")
if err == null { out.timestamp = parsed }
if service == "mongodb" {
out.event = "mongodb_event"
if is_integer(record.id) { out.mongoEventId = record.id }
if includes(["W", "E", "F"], record.s) { out.severity = "warn" }
if includes(["E", "F"], record.s) { out.severity = "error" }
} else if service == "nginx" {
out.event = "edge_request"
record.requestId = record.app_request_id
record.firstByteMs = (to_float(record.upstream_header_time) ?? 0.0) * 1000.0
record.ms = (to_float(record.request_time) ?? 0.0) * 1000.0
if is_string(record.edge_request_id) && match(string!(record.edge_request_id), r'^[a-f0-9]{32}$') { out.edge_request_id = record.edge_request_id }
if is_string(record.cf_ray) && match(string!(record.cf_ray), r'^[a-f0-9]{16}-[A-Z]{3}$') { out.cf_ray = record.cf_ray }
if includes(["PASSED", "DELAYED", "REJECTED", "DELAYED_DRY_RUN", "REJECTED_DRY_RUN"], record.limit_req_status) { out.limit_req_status = record.limit_req_status }
# Deliberately omit IPs by default. Original edge JSON has short, restricted
# host retention for incident response; no plain or unsalted hashed IP archive.
} else if includes(["api", "streamer"], service) {
if record.service == "worker" { out.service = "worker" }
if includes(["debug", "info", "warn", "error"], record.severity) { out.severity = record.severity }
if includes(["account_removed", "application_log", "billing_maintenance_failed", "billing_reconciliation_deferred", "billing_requires_attention", "billing_worker_failed", "blob_api_failed_falling_back_to_raw_url", "branch_not_found", "cache_removal_failed", "cache_removed", "checking_repository_status_and_unused_repositories", "cleared_lock_for_active_job", "conference_expire_failed", "content_expiration_failed", "daily_stats_snapshot_failed", "daily_stats_snapshot_stored", "download_completed", "download_start", "downloaded", "downloadfile_failed", "downloading_file", "downloading_gist", "downloading_pull_request", "downloading_via_raw_url_lfs", "duplicate_stripe_customer_requires_review", "ensuretodaysnapshot_failed", "error_log_redis_connect_failed", "expired_repository_cleanup_failed", "failed_removal_recovery_failed", "failed_to_persist_error_status", "failed_to_record_repository_removal_error", "gettoken", "github_primary_rate_limit_hit", "github_secondary_rate_limit_hit", "grant_revocation_failed", "initial_repository_status_check_failed", "job_completed", "job_failed", "job_stalled", "job_started", "lastusedat_update_failed", "login_token_failed", "login_token_req_login_failed", "logout_after_account_removal_failed", "logout_failed", "lookup_failed", "markerrorifinflight_failed", "markerrorifremoving_failed", "metadata_cleanup_complete", "metadata_cleanup_deferred", "owner_access_alert_reset_unavailable", "owner_access_email_accepted", "owner_access_email_audit_unavailable", "owner_access_email_claim_cleanup_unavailable", "owner_access_email_deferred_recovery_unavailable", "owner_access_email_delivery_failed", "owner_access_email_preparation_cleanup_unavailable", "owner_access_email_recovery_recorded", "owner_access_email_recovery_unavailable", "owner_access_email_result_reconciliation_required", "owner_access_email_unavailable", "pr_comments_404_continuing_without_them", "process_fatal_error", "process_heartbeat", "process_started", "progress", "queue_heartbeat", "queued_for_download", "rate_limit_gate_active_delaying_job_before_work", "rate_limit_gate_set", "rate_limited_delaying_job", "recover_failed", "recovered_expired_repository_cleanup", "recovered_file_from_truncated_tree", "recovered_file_invalidation_failed", "recovered_stuck_repo", "recoverstuckpreparing_failed", "recoverstuckremoving_failed", "redis_client_error", "redis_rate_limit_gate_written", "rejected_upstream_download_destination", "removal_failed", "removal_recovery_failed", "removecache_save_failed", "removed_cache_for_unused_repository", "removing_cache", "removing_repository", "repo_claimed", "repository_cache_removal_failed", "repository_expiration_failed", "repository_expired", "repository_removal_failed", "repository_removed", "request_cancelled", "request_completed", "request_slow", "requeued_failed_removal", "requeued_interrupted_removal", "reset", "retired_tree_cleanup_deferred", "retired_tree_cleanup_failed", "running_daily_stats_snapshot", "s3_body_stream_error", "send_failed", "server_started", "source_stream_failed_cache_write_discarded", "staged_tree_cleanup_failed", "staged_tree_completion_deferred", "streamer_fetch_failed", "streamer_started", "streamer_startup_failed", "streamer_zip_fetch_failed", "temporary_storage_cleanup_deferred", "token_refresh_failed_backing_off", "up_to_date", "update_queued", "verify_failed", "waiting_for_rate_limit_gate", "webhook_log", "webhook_processing_failed", "write_failed", "zip_stream_failed"], record.event) { out.event = record.event }
if is_string(record.release) && match(string!(record.release), r'^[a-f0-9]{7,40}$') { out.release = record.release }
if is_string(record.instance) && match(string!(record.instance), r'^[a-f0-9-]{36}$') { out.instance = record.instance }
if includes(["ABORT_ERR", "ECONNREFUSED", "ECONNRESET", "ENOTFOUND", "EPIPE", "ETIMEDOUT", "billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_app_reconnect_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_token_invalid", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_signature", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "token_refresh_failed", "quota_exceeded", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_found", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "upstream_timeout", "user_not_found", "username_not_defined", "webhook_processing_failed"], record.errorCode) { out.errorCode = record.errorCode }
if includes(["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"], record.module) { out.module = record.module }
if includes(["protocol", "credentials", "port", "host"], record.reason) { out.reason = record.reason }
if includes(["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"], record.phase) { out.phase = record.phase }
if includes(["github_app_authorization", "installation", "installation_repositories"], record.webhookEvent) { out.webhookEvent = record.webhookEvent }
if includes(["github.com", "api.github.com", "codeload.github.com", "raw.githubusercontent.com", "media.githubusercontent.com", "other"], record.upstreamHost) { out.upstreamHost = record.upstreamHost }
if includes(["download", "remove", "cache"], record.queue) { out.queue = record.queue }
if includes(["completed", "interrupted"], record.outcome) { out.outcome = record.outcome }
}
for_each(["requestId", "jobId", "deliveryId"]) -> |_index, key| {
value = get(record, [key]) ?? null
if is_string(value) && match(string!(value), r'^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$') { out = set!(out, [key], value) }
}
for_each(["status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus", "stage_repository_ms", "stage_credential_resolution_ms", "stage_upstream_ms", "stage_cache_lookup_ms", "stage_cache_fill_ms", "stage_worker_wait_ms", "stage_anonymize_ms", "stage_streamer_headers_ms"]) -> |_index, key| {
value = get(record, [key]) ?? null
if is_integer(value) || is_float(value) { out = set!(out, [key], value) }
}
if includes(["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "OTHER"], record.method) { out.method = record.method }
if includes(["/", "other", "/dashboard", "/anonymize", "/healthcheck", "/api/healthcheck", "/api", "/api/download", "/api/stat", "/api/stat/history", "/api/options", "/api/message", "/api/user", "/api/user/quota", "/api/user/dashboard", "/api/repo", "/api/anonymize-preview", "/api/admin/overview", "/api/admin/performance", "/api/admin/errors", "/api/admin/errors/stats", "/api/admin/queues", "/api/admin/queues/metrics", "/github/app/webhook", "/api/repo/:repoId", "/api/repo/:repoId/file/:path", "/api/repo/:repoId/files", "/api/repo/:repoId/files/counts", "/api/repo/:repoId/files/search", "/api/repo/:repoId/zip", "/api/repo/:repoId/options", "/api/repo/:repoId/refresh", "/api/repo/:repoId/coauthors", "/api/repo/:repoId/extend", "/api/repo/:owner/:repo/readme", "/api/repo/:owner/:repo/branches", "/w/:repoId/:path", "/r/:repoId/:path", "/api/pr/:id/:path", "/api/gist/:id/:path", "/assets/:path", "/github/:action", "/api/admin/:action", "/api/user/:action", "/api/repo/:id/file/:path", "/api/repo/:id/files/counts", "/api/repo/:id/files/search", "/api/repo/:id/files", "/api/repo/:id/zip", "/api/repo/:id", "/viewer/:id/:path"], record.route) { out.route = record.route }
. = out
+49
View File
@@ -0,0 +1,49 @@
data_dir: /var/lib/vector
sources:
docker:
type: file
include: [/host/docker/*/*-json.log]
read_from: end
max_line_bytes: 65536
max_read_bytes: 65536
rotate_wait_secs: 60
fingerprint: {strategy: device_and_inode}
nginx:
type: file
include: [/host/nginx/anonymous.access.json*]
exclude: ['*.gz']
read_from: end
max_line_bytes: 65536
fingerprint: {strategy: device_and_inode}
internal:
type: internal_metrics
scrape_interval_secs: 15
transforms:
safe:
type: remap
inputs: [docker, nginx]
file: /etc/vector/sanitize.vrl
drop_on_error: true
drop_on_abort: true
sinks:
logs:
type: elasticsearch
inputs: [safe]
endpoints: [http://victorialogs:9428/insert/elasticsearch/]
api_version: v8
compression: gzip
healthcheck: {enabled: false}
query:
_msg_field: event
_time_field: timestamp
_stream_fields: service,severity
batch: {max_bytes: 1048576, timeout_secs: 1}
request: {timeout_secs: 5, concurrency: 1}
buffer:
type: disk
max_size: 268435488
when_full: drop_newest
metrics:
type: prometheus_exporter
inputs: [internal]
address: 0.0.0.0:9598
+4
View File
@@ -156,3 +156,7 @@ The constrained-heap regression fixture can be run without network or MongoDB:
See [Shared blob cache](shared-cache.md) for configuration, storage and quota
accounting, admin metrics, production observations, rollout and benchmark commands.
For opt-in persistent structured logs, cross-service request/job investigation,
and private Grafana dashboards, see [operations logging](../deploy/logging/README.md).
The existing Redis performance/admin endpoints remain available during migration.
+17 -8
View File
@@ -9,17 +9,26 @@ function observeLine(summary, line) {
// Older releases rely on Node's stderr. New releases also emit a redacted
// process_fatal_error event before Node terminates.
if (/ECONNRESET|socket hang up/i.test(line)) summary.connectionResets++;
const match = /^\S+ (INFO|WARN|ERROR) \[([^\]]+)\] .*? (\{.*\})$/.exec(line);
if (!match) {
if (/Unhandled ['"]error['"] event|unhandled(?:promise)?rejection|FATAL ERROR|heap out of memory/i.test(line)) summary.fatalErrors++;
return;
let data, level, scope;
try {
const record = JSON.parse(line);
if (record && typeof record === "object" && typeof record.severity === "string") {
data = record; level = record.severity.toUpperCase(); scope = record.module;
data.code = record.errorCode || record.event;
}
} catch { /* Support pre-migration mixed text/JSON too. */ }
if (!data) {
const match = /^\S+ (INFO|WARN|ERROR) \[([^\]]+)\] .*? (\{.*\})$/.exec(line);
if (!match) {
if (/Unhandled ['"]error['"] event|unhandled(?:promise)?rejection|FATAL ERROR|heap out of memory/i.test(line)) summary.fatalErrors++;
return;
}
try { data = JSON.parse(match[3]); } catch { return; }
[, level, scope] = match;
}
let data;
try { data = JSON.parse(match[3]); } catch { return; }
const [, level, scope] = match;
if (level === "ERROR") summary.errors++;
if (level === "WARN") summary.warnings++;
if (data.code === "process_fatal_error" || /startup failed/i.test(line)) summary.fatalErrors++;
if (data.code === "process_fatal_error" || /startup[ _]failed/i.test(line)) summary.fatalErrors++;
if (data.code === "token_refresh_failed") summary.tokenRefreshFailures++;
if (scope === "anonymized-file" && data.code === "file_not_found") summary.missingFiles++;
if (scope === "requests" && typeof data.ms === "number") {
+236
View File
@@ -0,0 +1,236 @@
#!/usr/bin/env python3
"""Isolated local Docker integration test. Never reads production logs or .env.
Requires Docker Compose, Python 3 and PyYAML. Removes only its own project.
"""
import json
import base64
import os
from pathlib import Path
import secrets
import sys
import subprocess
import tempfile
import time
import urllib.parse
import urllib.request
import uuid
import yaml
ROOT = Path(__file__).resolve().parents[1]
CONFIG = ROOT / 'deploy/logging'
PROJECT = 'logging-test-' + secrets.token_hex(4)
def run(args, **kw):
return subprocess.run(args, check=True, text=True, capture_output=True, **kw).stdout.strip()
def wait_for(fn, timeout=60):
end = time.monotonic() + timeout
last = None
while time.monotonic() < end:
try:
result = fn()
if result:
return result
except Exception as e:
last = e
time.sleep(1)
raise AssertionError('Timed out: ' + str(last))
with tempfile.TemporaryDirectory(prefix=PROJECT) as temp:
temp = Path(temp)
docker = temp / 'docker'
nginx = temp / 'nginx'
docker.mkdir(); nginx.mkdir()
(docker / 'fixture').mkdir()
log = docker / 'fixture/fixture-json.log'
log.touch()
edge = nginx / 'anonymous.access.json'
edge.touch()
password = temp / 'password'
password.write_text(secrets.token_urlsafe(32))
password.chmod(0o644) # temp directory is 0700; Grafana uid must read the bind mount
env = {**os.environ, 'GRAFANA_ADMIN_USER': 'local-test', 'GRAFANA_PASSWORD_FILE': str(password),
'DOCKER_LOG_DIR': str(docker), 'NGINX_LOG_DIR': str(nginx), 'GRAFANA_PORT': '0'}
override = temp / 'override.yml'
override.write_text(yaml.safe_dump({'services': {
'victorialogs': {'ports': ['127.0.0.1::9428']},
'prometheus': {'ports': ['127.0.0.1::9090']},
'vector': {'ports': ['127.0.0.1::9598']},
'vmalert': {'ports': ['127.0.0.1::8880']},
}, 'networks': {'private': {'internal': False}}}))
cmd = ['docker', 'compose', '-p', PROJECT, '-f', str(CONFIG / 'compose.yml'), '-f', str(override)]
def compose(*args):
return run(cmd + list(args), env=env)
def url(service, port):
return 'http://' + compose('port', service, str(port))
def get(base, path, params=None):
address = base + path + ('?' + urllib.parse.urlencode(params) if params else '')
return urllib.request.urlopen(address, timeout=10).read().decode()
def query(expression):
return [json.loads(line) for line in get(vl, '/select/logsql/query', {'query': expression, 'limit': 50000}).splitlines() if line]
def append(service, record):
with log.open('a') as out:
out.write(json.dumps({'log': json.dumps(record) + '\n', 'attrs': {'tag': service}, 'stream': 'stdout'}) + '\n')
def count():
return int(query('* | stats count() as n')[0]['n'])
try:
compose('up', '-d')
vl = url('victorialogs', 9428)
prom = url('prometheus', 9090)
vector = url('vector', 9598)
alerts = url('vmalert', 8880)
grafana = url('grafana', 3000)
wait_for(lambda: get(vl, '/health') is not None)
wait_for(lambda: get(vector, '/metrics'))
wait_for(lambda: get(grafana, '/api/health'))
wait_for(lambda: 'application-logs' in get(alerts, '/api/v1/rules'))
time.sleep(2) # file watcher must discover empty files before append
request_id, job_id = str(uuid.uuid4()), str(uuid.uuid4())
sentinel = 'PRIVATE_SENTINEL_owner_repository_cookie_oauth_state'
for service in ['api', 'streamer']:
append(service, {'event': 'request_completed', 'severity': 'info', 'requestId': request_id,
'route': '/api/repo/:repoId/file/:path', 'status': 200, 'outcome': 'completed',
'firstByteMs': 40, 'responseMs': 600, 'ms': 640, 'stage_upstream_ms': 20,
'message': sentinel, 'url': 'https://example.invalid/' + sentinel, 'cookie': sentinel,
'authorization': sentinel, 'requestBody': {'secret': sentinel}, 'release': 'abcdef1'})
append('api', {'service': 'worker', 'event': 'job_failed', 'severity': 'error', 'requestId': request_id,
'jobId': job_id, 'attempt': 2, 'queue': 'download', 'errorCode': 'ETIMEDOUT', 'raw': sentinel})
append('mongodb', {'id': 51803, 's': 'E', 'msg': sentinel, 'attr': {'command': sentinel}})
append('redis', {'message': sentinel})
append('mongodb-backup', {'message': sentinel})
append('unrelated-container', {'message': sentinel})
# Malformed and malicious field values are represented without their raw content.
append('api', {'event': sentinel, 'route': '/' + sentinel, 'requestId': sentinel, 'severity': sentinel, 'module': sentinel})
with log.open('a') as out:
out.write(json.dumps({'log': sentinel, 'attrs': {'tag': 'api'}}) + '\n')
with edge.open('a') as out:
out.write(json.dumps({'status': 429, 'request_time': 0.1, 'upstream_header_time': '0.05',
'app_request_id': request_id, 'edge_request_id': 'f' * 32, 'route': '/api/repo/:id',
'limit_req_status': 'REJECTED', 'client_ip': '192.0.2.1', 'user_agent': sentinel,
'host': sentinel, 'request': sentinel}) + '\n')
wait_for(lambda: count() >= 9)
rows = query('*')
stored = json.dumps(rows)
assert sentinel not in stored and '192.0.2.1' not in stored
correlated = query('requestId:="' + request_id + '"')
assert {r['service'] for r in correlated} == {'api', 'streamer', 'worker', 'nginx'}
assert len(query('jobId:="' + job_id + '"')) == 1
assert all('requestId' not in r['_stream'] and 'instance' not in r['_stream'] for r in rows)
print('PASS ingestion, privacy allowlist, malformed input, API/streamer/job/edge correlation')
# Query every shipped dashboard and alert expression against the real server.
dashboard = json.loads((CONFIG / 'grafana/dashboards/operations.json').read_text())
for panel in dashboard['panels']:
for target in panel['targets']:
expr = target['expr'].replace('$request_id', request_id).replace('$job_id', job_id).replace('$edge_id', 'f' * 32)
if panel['datasource']['uid'] == 'anonymous-logs':
query(expr)
else:
get(prom, '/api/v1/query', {'query': expr})
for group in yaml.safe_load((CONFIG / 'log-alerts.yml').read_text())['groups']:
for rule in group['rules']:
get(vl, '/select/logsql/stats_query', {'query': rule['expr']})
auth = base64.b64encode(('local-test:' + password.read_text()).encode()).decode()
def grafana_request(path, payload=None):
request = urllib.request.Request(grafana + path,
data=json.dumps(payload).encode() if payload is not None else None,
headers={'Authorization': 'Basic ' + auth, 'Content-Type': 'application/json'})
return json.loads(urllib.request.urlopen(request, timeout=20).read())
assert grafana_request('/api/dashboards/uid/anonymous-operations')['dashboard']['uid'] == 'anonymous-operations'
assert grafana_request('/api/datasources/uid/anonymous-logs/health')['status'] == 'OK'
result = grafana_request('/api/ds/query', {'from': str(int((time.time() - 3600) * 1000)),
'to': str(int(time.time() * 1000)), 'queries': [{'refId': 'A',
'datasource': {'type': 'victoriametrics-logs-datasource', 'uid': 'anonymous-logs'},
'expr': 'requestId:="' + request_id + '"', 'queryType': 'instant', 'maxLines': 100}]})
assert not result['results']['A'].get('error'), result['results']['A'].get('error')
assert result['results']['A']['frames']
print('PASS dashboard and alert queries, provisioned Grafana datasource query')
# Rename rotation, then collector restart, preserving the spool/checkpoint volume.
before = count()
log.rename(log.with_suffix('.log.1'))
log.touch()
append('api', {'event': 'process_started', 'severity': 'info'})
wait_for(lambda: count() > before)
compose('restart', 'vector')
vector = url('vector', 9598)
wait_for(lambda: get(vector, '/metrics'))
before = count()
append('api', {'event': 'process_heartbeat', 'severity': 'info'})
wait_for(lambda: count() > before)
print('PASS rename rotation and collector checkpoint restart')
# Backend failure: sanitized disk spool remains bounded; source writer stays independent.
before = count()
compose('stop', 'victorialogs')
started = time.monotonic()
for _ in range(2000):
append('api', {'event': 'request_completed', 'severity': 'info', 'requestId': str(uuid.uuid4()), 'status': 200})
assert time.monotonic() - started < 10
wait_for(lambda: 'vector_buffer_size_events' in get(vector, '/metrics'))
time.sleep(3)
compose('restart', 'vector')
compose('start', 'victorialogs')
vl = url('victorialogs', 9428)
wait_for(lambda: count() >= before + 2000, 120)
print('PASS unavailable backend, spool persistence, recovery of 2000 records')
# Reject data older than retention rather than resurrecting it.
old = json.dumps({'timestamp': '2000-01-01T00:00:00Z', 'event': 'expired_fixture'}).encode()
req = urllib.request.Request(vl + '/insert/jsonline?_time_field=timestamp', data=old, headers={'Content-Type': 'application/stream+json'})
urllib.request.urlopen(req, timeout=10).read()
time.sleep(2)
assert not query('event:expired_fixture')
assert 'reason="too_small_timestamp"' in get(vl, '/metrics')
print('PASS retention rejection')
# Compression observation, synthetic workload only, including random trace IDs.
raw_bytes = 0
for _ in range(10000):
record = {'event': 'request_completed', 'severity': 'info', 'requestId': str(uuid.uuid4()),
'status': 200, 'ms': 50, 'firstByteMs': 20, 'responseMs': 30, 'route': '/api/repo/:repoId/file/:path'}
raw_bytes += len(json.dumps(record).encode()) + 1
append('api', record)
wait_for(lambda: count() >= before + 12000, 120)
compose('restart', 'victorialogs')
vl = url('victorialogs', 9428)
wait_for(lambda: get(vl, '/health') is not None)
metrics_text = get(vl, '/metrics')
size = sum(float(line.split()[-1]) for line in metrics_text.splitlines() if line.startswith('vl_data_size_bytes{'))
print(f'MEASURED fixture JSON bytes={raw_bytes}; total store data+index bytes={int(size)}; store also contains earlier fixtures; no production ratio claimed')
# Check fixed limits and no accidental public ingress.
config = json.loads(compose('config', '--format', 'json'))
for service, spec in config['services'].items():
assert int(spec['mem_limit']) > 0, service
for port in spec.get('ports', []):
assert port['host_ip'] == '127.0.0.1', service
print('PASS configured memory/rotation/disk bounds and loopback-only test ports')
if '--stress' in sys.argv:
compose('stop', 'victorialogs')
vector = url('vector', 9598)
numeric_fields = ['status', 'httpStatus', 'ms', 'firstByteMs', 'responseMs', 'bytes', 'bytesPerSecond',
'attempt', 'rss', 'memoryLimitBytes', 'memoryRatio', 'uptime', 'activeRequests', 'waiting',
'active', 'delayed', 'failed', 'oldestWaitingMs', 'droppedLogs', 'loopP95Ms', 'upstreamStatus',
'stage_repository_ms', 'stage_credential_resolution_ms', 'stage_upstream_ms', 'stage_cache_lookup_ms',
'stage_cache_fill_ms', 'stage_worker_wait_ms', 'stage_anonymize_ms', 'stage_streamer_headers_ms']
stress = {field: 123456789 for field in numeric_fields}
stress.update(event='request_completed', severity='info', requestId=request_id)
line = json.dumps({'log': json.dumps(stress), 'attrs': {'tag': 'api'}}) + '\n'
with log.open('a') as output:
for _ in range(500):
output.write(line * 1000)
def metric_value(name):
lines = get(vector, '/metrics').splitlines()
return sum(float(line.split()[1]) for line in lines if line.startswith(name + '{') and ('buffer_id="logs"' in line if name.startswith('vector_buffer_') else 'component_id="logs"' in line))
wait_for(lambda: metric_value('vector_buffer_discarded_events_total') > 0 or metric_value('vector_component_discarded_events_total') > 0, 240)
buffered = metric_value('vector_buffer_size_bytes')
assert buffered <= 268435488 + 1048576, buffered
inspection = json.loads(run(['docker', 'inspect', compose('ps', '-q', 'vector')]))[0]
assert inspection['RestartCount'] == 0 and not inspection['State']['OOMKilled']
print(f'PASS full-spool drop-newest stress: 500000 events; buffered={int(buffered)} bytes; collector stayed alive')
# Wait through the production two-minute 'for', rather than merely
# checking PromQL syntax or that a counter exists.
wait_for(lambda: json.loads(get(prom, '/api/v1/query', {'query':
'ALERTS{alertname="CollectorDrops",alertstate="firing"}'}))['data']['result'], 210)
print('PASS sustained CollectorDrops alert fires on full-spool loss')
except Exception:
# Only synthetic fixture logs exist in this project. No production reads.
print(compose('logs', '--tail', '30', 'vector', 'victorialogs', 'vmalert', 'grafana'))
raise
finally:
compose('down', '-v', '--remove-orphans')
+2 -1
View File
@@ -1,3 +1,4 @@
import { jobTrace } from "./log-context";
import { REPOSITORY_ID_COLLATION } from "./repository-id";
import { reserveRepository } from "./billing/quotas";
import { hasAutomaticUpdates } from "./billing/expiration";
@@ -855,7 +856,7 @@ export default class Repository {
const current = await AnonymizedRepositoryModel.exists({ _id: this.model._id, ...this.refreshFilter() });
if (!current) throw new AnonymousError("invalid_status", { httpStatus: 409 });
}
await downloadQueue.add(this.repoId, { repoId: this.repoId }, {
await downloadQueue.add(this.repoId, { repoId: this.repoId, ...jobTrace() }, {
jobId: `repo-${this.repoId}`,
attempts: 3,
});
+11
View File
@@ -0,0 +1,11 @@
import { AsyncLocalStorage } from "async_hooks";
import { randomUUID } from "crypto";
export interface LogContext { requestId?: string; jobId?: string; attempt?: number; service?: string }
export const logContext = new AsyncLocalStorage<LogContext>();
export const validTraceId = (value: unknown): value is string => typeof value === "string" &&
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(value);
export function jobTrace() {
const requestId = logContext.getStore()?.requestId;
return { logJobId: randomUUID(), ...(requestId ? { requestId } : {}) };
}
+15
View File
@@ -0,0 +1,15 @@
import { readFile } from "fs/promises";
/** Container usage includes sandbox workers and page cache; process RSS does not. */
export async function logMemory() {
const rss = process.memoryUsage().rss;
for (const [usage, limit] of [
["/sys/fs/cgroup/memory.current", "/sys/fs/cgroup/memory.max"],
["/sys/fs/cgroup/memory/memory.usage_in_bytes", "/sys/fs/cgroup/memory/memory.limit_in_bytes"],
]) {
const [used, max] = await Promise.all([usage, limit].map(path => readFile(path, "utf8").then(Number).catch(() => NaN)));
if (Number.isFinite(used) && max > 0 && max < Number.MAX_SAFE_INTEGER) {
return { rss, memoryLimitBytes: max, memoryRatio: used / max };
}
}
return { rss };
}
+59
View File
@@ -0,0 +1,59 @@
import { LOG_MODULES, ERROR_CODES, LOG_EVENTS } from "./log-schema";
import { randomUUID } from "crypto";
import { logContext, validTraceId } from "./log-context";
// No arbitrary strings: upstream errors, paths and even innocuous-looking messages
// can contain a private repository name. Add diagnostics here deliberately.
const instance = randomUUID();
const events: Record<string, string> = {
...LOG_EVENTS,
request: "request_completed", "slow request": "request_slow", "request interrupted": "request_cancelled",
"process fatal error": "process_fatal_error", "process started": "process_started",
"process heartbeat": "process_heartbeat", "job started": "job_started", "job completed": "job_completed",
"job failed": "job_failed", "job stalled": "job_stalled", "queue heartbeat": "queue_heartbeat",
};
const errorCodes = new Set([
...ERROR_CODES,
"token_refresh_failed", "quota_exceeded", "github_rate_limit_exceeded", "github_app_reconnect_required", "github_token_invalid",
"repository_expired", "repository_not_ready", "repository_not_found", "repository_job_cancelled",
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND", "EPIPE", "ABORT_ERR",
"upstream_error", "upstream_timeout", "invalid_signature", "webhook_processing_failed",
]);
const numbers = ["status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond",
"attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed",
"oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus"];
const stages = ["repository", "credential_resolution", "upstream", "cache_lookup", "cache_fill", "worker_wait", "anonymize", "streamer_headers"];
export function logRecord(level: string, module: string, args: unknown[]) {
const context = logContext.getStore() || {};
const details = Object.assign({}, ...args.filter(a => a && typeof a === "object" && !Array.isArray(a)));
const first = typeof args[0] === "string" ? args[0] : "";
const event = events[first] || (module.includes("webhook") ? "webhook_log" : "application_log");
const record: Record<string, unknown> = { timestamp: new Date().toISOString(), severity: level,
service: context.service || (["api", "streamer", "worker"].includes(details.service) ? details.service : undefined) || (process.env.SERVICE_NAME?.toLowerCase() === "streamer" ? "streamer" : "api"),
instance, release: /^[a-f0-9]{7,40}$/.test(process.env.APP_RELEASE || "") ? process.env.APP_RELEASE : "unknown",
event, message: event, module: LOG_MODULES.includes(module) ? module : "application" };
for (const key of ["requestId", "jobId"] as const) {
const value = details[key] || context[key];
if (validTraceId(value)) record[key] = value;
}
if (Number.isSafeInteger(context.attempt)) record.attempt = context.attempt;
for (const key of numbers) if (typeof details[key] === "number" && Number.isFinite(details[key])) record[key] = details[key];
const code = [details.code, details.message].find(value => errorCodes.has(value));
if (code) record.errorCode = code;
if (["protocol", "credentials", "port", "host"].includes(details.reason)) record.reason = details.reason;
if (["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"].includes(details.phase)) record.phase = details.phase;
if (["github_app_authorization", "installation", "installation_repositories"].includes(details.event)) record.webhookEvent = details.event;
if (validTraceId(details.deliveryId)) record.deliveryId = details.deliveryId;
if (["github.com", "api.github.com", "codeload.github.com", "raw.githubusercontent.com", "media.githubusercontent.com"].includes(details.hostname)) record.upstreamHost = details.hostname;
else if (details.hostname) record.upstreamHost = "other";
if (["completed", "interrupted"].includes(details.outcome)) record.outcome = details.outcome;
if (["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "OTHER"].includes(details.method)) record.method = details.method;
// Route labels are generated by request-monitoring, never from URLs in arbitrary errors.
if (module === "requests" && typeof details.route === "string") record.route = details.route;
if (["download", "remove", "cache"].includes(details.queue)) record.queue = details.queue;
if (details.stages && typeof details.stages === "object") for (const key of stages) {
const value = details.stages[key];
if (typeof value === "number" && Number.isFinite(value)) record[`stage_${key}_ms`] = value;
}
return record;
}
+121
View File
@@ -0,0 +1,121 @@
// Reviewed literal diagnostics. Never extend this list from runtime data.
export const LOG_MODULES = ["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"];
export const ERROR_CODES = ["billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "user_not_found", "username_not_defined"];
export const LOG_EVENTS: Record<string, string> = {
"PR comments 404, continuing without them": "pr_comments_404_continuing_without_them",
"S3 body stream error": "s3_body_stream_error",
"account removed": "account_removed",
"billing maintenance failed": "billing_maintenance_failed",
"billing reconciliation deferred": "billing_reconciliation_deferred",
"billing requires attention": "billing_requires_attention",
"billing worker failed": "billing_worker_failed",
"blob API failed, falling back to raw URL": "blob_api_failed_falling_back_to_raw_url",
"branch not found": "branch_not_found",
"cache removal failed": "cache_removal_failed",
"cache removed": "cache_removed",
"checking repository status and unused repositories": "checking_repository_status_and_unused_repositories",
"cleared lock for active job": "cleared_lock_for_active_job",
"conference expire failed": "conference_expire_failed",
"content expiration failed": "content_expiration_failed",
"daily stats snapshot failed": "daily_stats_snapshot_failed",
"daily stats snapshot stored": "daily_stats_snapshot_stored",
"download completed": "download_completed",
"download start": "download_start",
"downloadFile failed": "downloadfile_failed",
"downloaded": "downloaded",
"downloading file": "downloading_file",
"downloading gist": "downloading_gist",
"downloading pull request": "downloading_pull_request",
"downloading via raw URL (LFS)": "downloading_via_raw_url_lfs",
"duplicate Stripe customer requires review": "duplicate_stripe_customer_requires_review",
"ensureTodaySnapshot failed": "ensuretodaysnapshot_failed",
"error log redis connect failed": "error_log_redis_connect_failed",
"expired repository cleanup failed": "expired_repository_cleanup_failed",
"failed removal recovery failed": "failed_removal_recovery_failed",
"failed to persist ERROR status": "failed_to_persist_error_status",
"failed to record repository removal error": "failed_to_record_repository_removal_error",
"getToken": "gettoken",
"github primary rate limit hit": "github_primary_rate_limit_hit",
"github secondary rate limit hit": "github_secondary_rate_limit_hit",
"grant revocation failed": "grant_revocation_failed",
"initial repository status check failed": "initial_repository_status_check_failed",
"job completed": "job_completed",
"job stalled": "job_stalled",
"job started": "job_started",
"lastUsedAt update failed": "lastusedat_update_failed",
"login-token failed": "login_token_failed",
"login-token req.login failed": "login_token_req_login_failed",
"logout after account removal failed": "logout_after_account_removal_failed",
"logout failed": "logout_failed",
"lookup failed": "lookup_failed",
"markErrorIfInFlight failed": "markerrorifinflight_failed",
"markErrorIfRemoving failed": "markerrorifremoving_failed",
"metadata cleanup complete": "metadata_cleanup_complete",
"metadata cleanup deferred": "metadata_cleanup_deferred",
"owner access alert reset unavailable": "owner_access_alert_reset_unavailable",
"owner access email accepted": "owner_access_email_accepted",
"owner access email audit unavailable": "owner_access_email_audit_unavailable",
"owner access email claim cleanup unavailable": "owner_access_email_claim_cleanup_unavailable",
"owner access email deferred recovery unavailable": "owner_access_email_deferred_recovery_unavailable",
"owner access email delivery failed": "owner_access_email_delivery_failed",
"owner access email preparation cleanup unavailable": "owner_access_email_preparation_cleanup_unavailable",
"owner access email recovery recorded": "owner_access_email_recovery_recorded",
"owner access email recovery unavailable": "owner_access_email_recovery_unavailable",
"owner access email result reconciliation required": "owner_access_email_result_reconciliation_required",
"owner access email unavailable": "owner_access_email_unavailable",
"process fatal error": "process_fatal_error",
"process heartbeat": "process_heartbeat",
"process started": "process_started",
"progress": "progress",
"queue heartbeat": "queue_heartbeat",
"queued for download": "queued_for_download",
"rate limit gate active, delaying job before work": "rate_limit_gate_active_delaying_job_before_work",
"rate limit gate set": "rate_limit_gate_set",
"rate-limited, delaying job": "rate_limited_delaying_job",
"recover failed": "recover_failed",
"recoverStuckPreparing failed": "recoverstuckpreparing_failed",
"recoverStuckRemoving failed": "recoverstuckremoving_failed",
"recovered expired repository cleanup": "recovered_expired_repository_cleanup",
"recovered file from truncated tree": "recovered_file_from_truncated_tree",
"recovered file invalidation failed": "recovered_file_invalidation_failed",
"recovered stuck repo": "recovered_stuck_repo",
"redis client error": "redis_client_error",
"redis rate limit gate written": "redis_rate_limit_gate_written",
"rejected upstream download destination": "rejected_upstream_download_destination",
"removal failed": "removal_failed",
"removal recovery failed": "removal_recovery_failed",
"removeCache save failed": "removecache_save_failed",
"removed cache for unused repository": "removed_cache_for_unused_repository",
"removing cache": "removing_cache",
"removing repository": "removing_repository",
"repo claimed": "repo_claimed",
"repository cache removal failed": "repository_cache_removal_failed",
"repository expiration failed": "repository_expiration_failed",
"repository expired": "repository_expired",
"repository removal failed": "repository_removal_failed",
"repository removed": "repository_removed",
"requeued failed removal": "requeued_failed_removal",
"requeued interrupted removal": "requeued_interrupted_removal",
"reset": "reset",
"retired tree cleanup deferred": "retired_tree_cleanup_deferred",
"retired tree cleanup failed": "retired_tree_cleanup_failed",
"running daily stats snapshot": "running_daily_stats_snapshot",
"send failed": "send_failed",
"server started": "server_started",
"source stream failed; cache write discarded": "source_stream_failed_cache_write_discarded",
"staged tree cleanup failed": "staged_tree_cleanup_failed",
"staged tree completion deferred": "staged_tree_completion_deferred",
"streamer fetch failed": "streamer_fetch_failed",
"streamer started": "streamer_started",
"streamer startup failed": "streamer_startup_failed",
"streamer zip fetch failed": "streamer_zip_fetch_failed",
"temporary storage cleanup deferred": "temporary_storage_cleanup_deferred",
"token refresh failed; backing off": "token_refresh_failed_backing_off",
"up to date": "up_to_date",
"update queued": "update_queued",
"verify failed": "verify_failed",
"waiting for rate limit gate": "waiting_for_rate_limit_gate",
"webhook processing failed": "webhook_processing_failed",
"write failed": "write_failed",
"zip stream failed": "zip_stream_failed"
};
+29 -30
View File
@@ -1,3 +1,4 @@
import { logRecord } from "./log-record";
import { redactSecrets } from "./redact-secrets";
import { createClient, RedisClientType } from "redis";
import config from "../config";
@@ -40,16 +41,6 @@ function resolveThreshold(): number {
const threshold = resolveThreshold();
function formatArg(a: unknown): string {
if (typeof a === "string") return a;
if (a instanceof Error) return JSON.stringify(serializeError(a));
try {
return JSON.stringify(a);
} catch {
return String(a);
}
}
let redisClient: RedisClientType | null = null;
let redisDisabled = false;
@@ -58,6 +49,8 @@ function getRedis(): RedisClientType | null {
if (redisClient) return redisClient;
try {
redisClient = createClient({
disableOfflineQueue: true,
commandsQueueMaxLength: 128,
socket: {
host: config.REDIS_HOSTNAME,
port: config.REDIS_PORT,
@@ -92,6 +85,13 @@ function getRedis(): RedisClientType | null {
// Redis is back. Read by /admin/errors/stats so the admin page surfaces
// "you're losing logs" instead of silently rendering an empty table.
let droppedInProcess = 0;
let droppedOutput = 0;
let outputBroken = false;
let errorOutputBroken = false;
// A closed log reader must not become an uncaught EPIPE on the serving process.
process.stdout.on("error", () => { outputBroken = true; droppedOutput++; });
process.stderr.on("error", () => { errorOutputBroken = true; droppedOutput++; });
export function getDroppedOutput() { return droppedOutput; }
export function getInProcessDropped(): number {
return droppedInProcess;
}
@@ -192,8 +192,9 @@ function persistError(entry: {
message: string;
raw: unknown[];
}) {
if (process.env.LOG_REDIS_MIRROR === "off") return;
const client = getRedis();
if (!client || !client.isOpen) {
if (!client || !client.isReady) {
droppedInProcess++;
return;
}
@@ -204,12 +205,12 @@ function persistError(entry: {
) as Record<string, unknown> | undefined;
const bucket = bucketFor(detail, entry.level);
const code =
(detail && typeof detail.message === "string"
? (detail.message as string)
: "") ||
(detail && typeof detail.code === "string"
? (detail.code as string)
: "") ||
(detail && typeof detail.message === "string"
? (detail.message as string)
: "") ||
"_";
// Routine client misuse (401/403/404 and the allowlisted "expected" 4xx
// codes below) drowns out real errors in the recent-entries ring. Keep the
@@ -254,27 +255,25 @@ function emit(level: Level, module: string, args: unknown[]) {
if (LEVEL_ORDER[level] < threshold) return;
args = args.map(arg => redactSecrets(arg instanceof Error ? serializeError(arg) : arg));
const ts = new Date().toISOString();
const formatted = args.map(formatArg);
const line = `${ts} ${level.toUpperCase()} [${module}] ${formatted.join(" ")}`;
const sink =
level === "error"
? console.error
: level === "warn"
? console.warn
: level === "debug"
? console.debug
: console.log;
sink(line);
const record = logRecord(level, module, args);
// stdout is a pipe in Docker. Respect backpressure without awaiting it or
// building an unbounded userland queue. The next heartbeat reports losses.
const output = outputBroken ? process.stderr : process.stdout;
if (!(outputBroken && errorOutputBroken) && output.writableLength < 256 * 1024) {
try { output.write(JSON.stringify(record) + "\n"); }
catch {
if (output === process.stdout) outputBroken = true;
else errorOutputBroken = true;
droppedOutput++;
}
} else { droppedOutput++; }
if (level === "error" || level === "warn") {
persistError({
ts,
level,
module,
message: typeof args[0] === "string" ? args[0] : "",
raw: args.map((a) => {
if (a instanceof Error) return serializeError(a);
return a;
}),
message: String(record.event),
raw: [{ ...record, code: record.errorCode || record.event }],
});
}
}
+9 -2
View File
@@ -1,4 +1,5 @@
import { createLogger, serializeError } from "./logger";
import { logMemory } from "./log-memory";
import { createLogger, serializeError, getDroppedOutput } from "./logger";
/** Observe fatal errors without overriding Node's default exit behavior. */
export function installFatalErrorLogging(service: "api" | "streamer") {
@@ -6,6 +7,12 @@ export function installFatalErrorLogging(service: "api" | "streamer") {
const report = (error: Error, origin: string) => logger.error("process fatal error", {
...serializeError(error), code: "process_fatal_error", service, origin,
});
logger.info("process started", { service });
const heartbeat = setInterval(() => {
void logMemory().then(memory => logger.info("process heartbeat", {
service, uptime: process.uptime(), ...memory, droppedLogs: getDroppedOutput() }));
}, 30000);
heartbeat.unref();
process.on("uncaughtExceptionMonitor", report);
return () => process.removeListener("uncaughtExceptionMonitor", report);
return () => { clearInterval(heartbeat); process.removeListener("uncaughtExceptionMonitor", report); };
}
+2 -1
View File
@@ -1,3 +1,4 @@
import { logContext } from "./log-context";
import { AsyncLocalStorage } from "async_hooks";
import { randomUUID } from "crypto";
import { performance } from "perf_hooks";
@@ -164,6 +165,6 @@ export function monitorRequests(service: Service, metrics = requestMetrics,
else if (!route.includes("healthcheck")) log.info("request", detail);
};
res.once("finish", done); res.once("close", done);
context.run(trace, next);
context.run(trace, () => logContext.run({ requestId: id, service }, next));
};
}
+20 -1
View File
@@ -1,3 +1,4 @@
import { jobTrace } from "../core/log-context";
import { isQuotaFailure } from "../core/repository-failure-status";
import UserModel from "../core/model/users/users.model";
import { JobsOptions, Queue, Worker } from "bullmq";
@@ -16,6 +17,8 @@ const logger = createLogger("queue");
// ERR_BUFFER_OUT_OF_BOUNDS on long term lists / large nested fields.
export interface RepoJobData {
repoId: string;
requestId?: string;
logJobId?: string;
}
const IN_FLIGHT_STATUSES: RepositoryStatus[] = [
@@ -139,7 +142,7 @@ export async function addRemovalJob(
if (LIVE_JOB_STATES.has(state)) return false;
await existing.remove().catch(() => undefined);
}
await queue.add(repoId, { repoId }, { ...REMOVAL_JOB_OPTIONS, jobId });
await queue.add(repoId, { repoId, ...jobTrace() }, { ...REMOVAL_JOB_OPTIONS, jobId });
return true;
}
@@ -302,6 +305,22 @@ export function startWorker() {
autorun: true,
}
);
for (const [queue, worker] of [["cache", cacheWorker], ["remove", removeWorker], ["download", downloadWorker]] as const) {
worker.on("stalled", () => logger.error("job stalled", { queue }));
}
let sampling = false;
const timer = setInterval(async () => {
if (sampling) return;
sampling = true;
try {
for (const [queue, client] of [["cache", cacheQueue], ["remove", removeQueue], ["download", downloadQueue]] as const) {
const counts = await client.getJobCounts("waiting", "active", "delayed", "failed");
const oldest = (await client.getJobs(["waiting"], 0, 0, true))[0];
logger.info("queue heartbeat", { queue, ...counts, oldestWaitingMs: oldest ? Date.now() - oldest.timestamp : 0 });
}
} catch { /* queue availability is reported through missing heartbeat alerts */ }
finally { sampling = false; }
}, 30000); timer.unref();
if (!downloadWorker.isRunning()) downloadWorker.run();
downloadWorker.on("active", async (job) => {
+28
View File
@@ -0,0 +1,28 @@
import { logMemory } from "../core/log-memory";
import { randomUUID } from "crypto";
import { SandboxedJob } from "bullmq";
import { logContext, validTraceId } from "../core/log-context";
import { createLogger, getDroppedOutput } from "../core/logger";
import type { RepoJobData } from "./index";
const logger = createLogger("queue");
export function loggedJob(queue: "download" | "remove" | "cache", work: (job: SandboxedJob<RepoJobData, void>) => Promise<void>) {
return (job: SandboxedJob<RepoJobData, void>) => logContext.run({
service: "worker", jobId: validTraceId(job.data.logJobId) ? job.data.logJobId : randomUUID(),
requestId: validTraceId(job.data.requestId) ? job.data.requestId : undefined,
attempt: job.attemptsMade + 1,
}, async () => {
logger.info("job started", { queue });
const started = Date.now();
const heartbeat = setInterval(() => {
void logMemory().then(memory => logger.info("process heartbeat", { queue, ...memory, uptime: process.uptime(), droppedLogs: getDroppedOutput() }));
}, 30000); heartbeat.unref();
try {
await work(job);
logger.info("job completed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() });
} catch (error) {
const expected = error instanceof Error && ["DelayedError", "RateLimitDelayError"].includes(error.name);
logger[expected ? "info" : "error"]("job failed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() });
throw error;
} finally { clearInterval(heartbeat); }
});
}
+3 -2
View File
@@ -1,3 +1,4 @@
import { loggedJob } from "../log-job";
import { githubQuotaKey } from "../../core/github-token-context";
import { isQuotaFailure } from "../../core/repository-failure-status";
import { SandboxedJob, UnrecoverableError } from "bullmq";
@@ -13,7 +14,7 @@ import { DelayedError } from "bullmq";
const logger = createLogger("queue:download");
export default async function (job: SandboxedJob<RepoJobData, void>) {
export default loggedJob("download", async function (job: SandboxedJob<RepoJobData, void>) {
const {
connect,
getRepository,
@@ -170,4 +171,4 @@ export default async function (job: SandboxedJob<RepoJobData, void>) {
} finally {
clearInterval(statusInterval);
}
}
});
+2 -1
View File
@@ -1,3 +1,4 @@
import { loggedJob } from "../log-job";
import { SandboxedJob } from "bullmq";
import { getRepository as getRepositoryImport } from "../../server/database";
import { RepoJobData } from "../index";
@@ -42,4 +43,4 @@ export function createRemoveCacheProcessor(database?: Database) {
) => processRemoveCache(job, database);
}
export default createRemoveCacheProcessor();
export default loggedJob("cache", createRemoveCacheProcessor());
+2 -1
View File
@@ -1,3 +1,4 @@
import { loggedJob } from "../log-job";
import { SandboxedJob } from "bullmq";
import { getRepository as getRepositoryImport } from "../../server/database";
import { RepositoryStatus } from "../../core/types";
@@ -61,4 +62,4 @@ export function createRemoveRepositoryProcessor(database?: Database) {
) => processRemoveRepository(job, database);
}
export default createRemoveRepositoryProcessor();
export default loggedJob("remove", createRemoveRepositoryProcessor());
+2 -1
View File
@@ -1,3 +1,4 @@
import { jobTrace } from "../../core/log-context";
import { cacheMetricsReport } from "../../core/cache-observability";
import { sharedBlobCache, resultBlobCache } from "../../core/blob-cache";
import { AsyncCache } from "../../core/async-cache";
@@ -983,7 +984,7 @@ router.delete(
const repo = await getRepo(req, res, { nocheck: true });
if (!repo) return;
try {
await cacheQueue.add(repo.repoId, { repoId: repo.repoId }, { jobId: `repo-${repo.repoId}` });
await cacheQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, { jobId: `repo-${repo.repoId}` });
return res.json({ status: repo.status });
} catch (error) {
handleError(error, res, req);
+3 -2
View File
@@ -1,3 +1,4 @@
import { jobTrace } from "../../core/log-context";
import Repository from "../../core/Repository";
import { sourceStorageSize } from "../../core/source/Source";
import { maximumExpiration, neverExpirationDeadline, validateBillingExpiration } from "../../core/billing/expiration";
@@ -42,7 +43,7 @@ router.use(ensureAuthenticated);
async function enqueueSavedSnapshot(model: IAnonymizedRepositoryDocument) {
try {
await downloadQueue.add(model.repoId, { repoId: model.repoId }, {
await downloadQueue.add(model.repoId, { repoId: model.repoId, ...jobTrace() }, {
jobId: `repo-${model.repoId}`, attempts: 3,
});
} catch (error) {
@@ -878,7 +879,7 @@ router.post("/", async (req, res) => {
}
res.send({ status: repo.status });
downloadQueue.add(repo.repoId, { repoId: repo.repoId }, {
downloadQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, {
jobId: `repo-${repo.repoId}`,
attempts: 3,
});
+2 -1
View File
@@ -1,3 +1,4 @@
import { jobTrace } from "../../core/log-context";
import { hasVisibleSubmodules } from "../../core/file-submodules";
import { FILE_LISTING_LIMIT } from "../../core/file-tree";
import { folderCounts as scopedFolderCounts } from "../../core/folder-counts";
@@ -342,7 +343,7 @@ router.get(
repo.statusActivityDate < fiveMinuteAgo
) {
await repo.updateStatus(RepositoryStatus.PREPARING);
await downloadQueue.add(repo.repoId, { repoId: repo.repoId }, {
await downloadQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, {
jobId: `repo-${repo.repoId}`,
attempts: 3,
});
+6 -3
View File
@@ -125,7 +125,8 @@ describe("removal recovery", function () {
AnonymizedRepositoryModel.find = () => ({ lean: async () => [] });
await queueModule.recoverStuckRemoving();
expect(repo.status).to.equal("removing");
expect(added[1]).to.deep.equal({ repoId: repo.repoId });
expect(added[1]).to.deep.include({ repoId: repo.repoId });
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
});
}
@@ -164,7 +165,8 @@ describe("removal recovery", function () {
let added;
queueModule.removeQueue.add = async (...args) => { added = args; };
await queueModule.recoverStuckRemoving();
expect(added[1]).to.deep.equal({ repoId: repo.repoId });
expect(added[1]).to.deep.include({ repoId: repo.repoId });
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
});
});
@@ -392,7 +394,8 @@ describe("removal workers", function () {
expect(await addRemovalJob("repo-1", queue)).to.equal(true);
expect(removed).to.equal(true);
expect(added[0]).to.equal("repo-1");
expect(added[1]).to.deep.equal({ repoId: "repo-1" });
expect(added[1]).to.deep.include({ repoId: "repo-1" });
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
expect(added[2].jobId).to.equal("repo-repo-1");
expect(added[2].attempts).to.equal(3);
expect(added[2].removeOnFail).to.deep.equal({ count: 1000 });
+8 -6
View File
@@ -19,15 +19,17 @@ describe("credential and webhook failure diagnostics", () => {
afterEach(() => restores.splice(0).reverse().forEach(restore => restore()));
it("logs generic upstream 401 as warning while retaining server errors", () => {
const warnings = [], errors = [];
stub(console, "warn", (...args) => warnings.push(args));
stub(console, "error", (...args) => errors.push(args));
stub(process.stdout, "write", line => {
const record = JSON.parse(line);
(record.severity === "warn" ? warnings : errors).push(line); return true;
});
handleError(Object.assign(new Error("denied"), { status: 401 }), response());
handleError(Object.assign(new Error("failed"), { response: { statusCode: 503 } }), response());
expect(warnings).to.have.length(1); expect(errors).to.have.length(1);
expect(errors.flat().join(" ")).to.include('"httpStatus":503');
});
it("logs got HTTP401 as warning with its actual status", () => {
const warnings = []; stub(console, "warn", (...args) => warnings.push(args));
const warnings = []; stub(process.stdout, "write", line => { warnings.push(line); return true; });
const failure = Object.assign(new Error("Unauthorized"), { response: { statusCode: 401 } });
Object.setPrototypeOf(failure, require("got").HTTPError.prototype);
const res = response(); handleError(failure, res);
@@ -51,16 +53,16 @@ describe("credential and webhook failure diagnostics", () => {
it("logs signed webhook persistence failures without secrets and still returns 503", async () => {
stub(config, "GITHUB_APP_ENABLED", true); stub(config, "GITHUB_APP_ID", "123");
stub(config, "GITHUB_APP_WEBHOOK_SECRET", "test-secret");
const messages = []; stub(console, "error", (...args) => messages.push(args));
const messages = []; stub(process.stdout, "write", line => { messages.push(line); return true; });
stub(Installation, "updateOne", async () => { throw new Error("secret-token https://private/?token=secret"); });
const body = Buffer.from(JSON.stringify({ action: "added", installation: { id: 4, app_id: 123 }, private: "private-body" }));
const req = { body, headers: { "x-github-event": "installation_repositories", "x-github-delivery": "abcd-1234",
const req = { body, headers: { "x-github-event": "installation_repositories", "x-github-delivery": "abcdef12-1234-1234-1234-abcdef123456",
"x-hub-signature-256": "sha256=" + createHmac("sha256", "test-secret").update(body).digest("hex") } };
const handler = githubAppWebhook.stack.find(layer => layer.route).route.stack.at(-1).handle;
const res = response(); await handler(req, res);
expect(res.code).to.equal(503); expect(res.body.error).to.equal("webhook_processing_failed");
const text = JSON.stringify(messages);
expect(text).to.include("persist_installation_block").and.include("abcd-1234");
expect(text).to.include("persist_installation_block").and.include("abcdef12-1234-1234-1234-abcdef123456");
expect(text).not.to.include("secret-token"); expect(text).not.to.include("private-body"); expect(text).not.to.include("https://");
expect(text).not.to.include(req.headers["x-hub-signature-256"]);
});
+1 -1
View File
@@ -27,7 +27,7 @@ describe("download progress publication fences", function () {
it("keeps a captured publication fence valid when a queue progress tick runs", async function () {
const { repo, stored } = setup(); let tick;
stub(global, "setInterval", callback => { tick = callback; return 1; });
stub(global, "setInterval", callback => { tick = callback; return { unref() {} }; });
stub(global, "clearInterval", () => {});
stub(db, "connect", async () => {});
stub(db, "getRepository", async () => repo);
+1 -1
View File
@@ -52,7 +52,7 @@ describe("folder file endpoint disclosure", function () {
});
it("blocks a real HTTP redirect to a GitHub tree page before fetching its HTML", async () => {
const warnings = [];
stub(console, "warn", (...args) => warnings.push(args.join(" ")));
stub(process.stdout, "write", line => { warnings.push(line); return true; });
const server = await listen(http.createServer((_req, res) => { res.writeHead(302, { location: "https://github.com/example/private-repo/tree/main/folder?token=secret-query" }); res.end(); }));
const original = got.stream;
stub(got, "stream", (_url, options) => original(`http://127.0.0.1:${server.address().port}/raw`, options));
+2 -1
View File
@@ -84,7 +84,8 @@ describe("repository refresh and restoration", () => {
expect(response.status).to.equal("preparing");
expect(repo.model.source.commit).to.equal(commit);
expect(repo.repoId).to.equal("restore-me");
expect(added[1]).to.deep.equal({ repoId: "restore-me" });
expect(added[1]).to.deep.include({ repoId: "restore-me" });
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
});
}
+7 -8
View File
@@ -9,17 +9,16 @@ require("../src/core/storage");
const FileSystem = require("../src/core/storage/FileSystem").default;
describe("cache write error origins", function () {
let directory, previousFolder, messages, previousWarn, previousError;
let directory, previousFolder, messages, previousWrite;
beforeEach(function () {
directory = fs.mkdtempSync(path.join(os.tmpdir(), "cache-errors-"));
previousFolder = config.FOLDER; config.FOLDER = directory;
messages = [];
previousWarn = console.warn; previousError = console.error;
console.warn = (...args) => messages.push(["warn", args.join(" ")]);
console.error = (...args) => messages.push(["error", args.join(" ")]);
previousWrite = process.stdout.write;
process.stdout.write = line => { const record = JSON.parse(line); messages.push([record.severity, record.event]); return true; };
});
afterEach(function () {
console.warn = previousWarn; console.error = previousError;
process.stdout.write = previousWrite;
config.FOLDER = previousFolder;
fs.rmSync(directory, { recursive: true, force: true });
});
@@ -34,8 +33,8 @@ describe("cache write error origins", function () {
let caught;
try { await storage.write("repo", "file.txt", input); } catch (error) { caught = error; }
expect(caught).to.equal(failure);
expect(messages.some(([level, text]) => level === "warn" && text.includes("source stream failed"))).to.equal(true);
expect(messages.some(([level, text]) => level === "error" && text.includes("write failed"))).to.equal(false);
expect(messages.some(([level, text]) => level === "warn" && text.includes("source_stream_failed"))).to.equal(true);
expect(messages.some(([level, text]) => level === "error" && text.includes("write_failed"))).to.equal(false);
expect(fs.readFileSync(path.join(directory, "repo/original/file.txt"), "utf8")).to.equal("previous");
expect(fs.readdirSync(path.join(directory, "repo/original"))).to.deep.equal(["file.txt"]);
});
@@ -45,7 +44,7 @@ describe("cache write error origins", function () {
let caught;
try { await storage.write("repo", "file.txt", "new content"); } catch (error) { caught = error; }
expect(caught).to.be.instanceOf(Error);
expect(messages.some(([level, text]) => level === "error" && text.includes("write failed"))).to.equal(true);
expect(messages.some(([level, text]) => level === "error" && text.includes("write_failed"))).to.equal(true);
expect(messages.some(([level]) => level === "warn")).to.equal(false);
expect(fs.readdirSync(path.join(directory, "repo/original"))).to.deep.equal(["file.txt"]);
});
+104
View File
@@ -0,0 +1,104 @@
const { setTimeout } = require("node:timers");
const { expect } = require('chai');
const { randomUUID } = require('node:crypto');
const { spawn, spawnSync } = require('node:child_process');
require('ts-node/register/transpile-only');
const { logRecord } = require('../src/core/log-record');
const { logContext, jobTrace } = require('../src/core/log-context');
const { loggedJob } = require('../src/queue/log-job');
describe('structured logging privacy and context', () => {
it('keeps the production summary compatible with JSON and legacy records', () => {
const { logSummary, observeLine } = require('../scripts/check-production-logs');
const summary = logSummary();
observeLine(summary, JSON.stringify(logRecord('info', 'requests', ['request', {ms: 1, status: 503, outcome: 'completed'}])));
observeLine(summary, JSON.stringify(logRecord('error', 'process', ['process fatal error'])));
observeLine(summary, '2026-01-01T00:00:00Z WARN [requests] request {"ms":2,"status":429,"outcome":"completed"}');
expect(summary).to.include({completed: 2, completed5xx: 1, fatalErrors: 1, errors: 1, warnings: 1, rateLimited: 1});
});
it('omits arbitrary text, private identities, request bodies and nested secrets', () => {
const secret = 'private-owner/private-repository?state=secret';
const record = logRecord('error', 'upstream', [secret, {
message: secret, url: secret, headers: { authorization: secret }, repoId: secret,
requestBody: secret, status: 502, code: 'ETIMEDOUT', stages: { upstream: 42, [secret]: 1 },
}]);
expect(JSON.stringify(record)).not.to.include(secret);
expect(record).to.include({ status: 502, errorCode: 'ETIMEDOUT', stage_upstream_ms: 42 });
expect(record).to.have.keys('timestamp', 'severity', 'service', 'instance', 'release', 'event', 'message', 'module', 'status', 'errorCode', 'stage_upstream_ms');
});
it('keeps concurrent request contexts separate and propagates an opaque job trace', async () => {
const ids = [randomUUID(), randomUUID()];
const rows = await Promise.all(ids.map((requestId, i) => logContext.run({ requestId }, async () => {
await new Promise(resolve => setTimeout(resolve, 10 - i));
const trace = jobTrace();
expect(trace.requestId).to.equal(requestId);
expect(trace.logJobId).not.to.equal(requestId);
return logRecord('info', 'fixture', ['request']);
})));
expect(rows.map(row => row.requestId)).to.deep.equal(ids);
expect(logRecord('info', 'fixture', ['request'])).not.to.have.property('requestId');
});
it('sets sandbox job context and preserves success/failure semantics', async () => {
const requestId = randomUUID(), jobId = randomUUID();
const processor = loggedJob('download', async () => {
expect(logContext.getStore()).to.deep.equal({ service: 'worker', requestId, jobId, attempt: 3 });
throw new Error('fixture failure');
});
try { await processor({ data: { repoId: 'private-id', requestId, logJobId: jobId }, attemptsMade: 2 }); }
catch (error) { expect(error.message).to.equal('fixture failure'); return; }
throw new Error('job failure was swallowed');
});
it('rejects injected correlation IDs and non-finite measurements', () => {
const row = logRecord('warn', 'fixture', ['request', { requestId: 'bad\nsecret', jobId: 'private', ms: Infinity, status: NaN }]);
expect(row).not.to.have.any.keys('requestId', 'jobId', 'ms', 'status');
});
it('survives a reader closing stdout', async function () {
this.timeout(10000);
const child = spawn(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', `
const { createLogger, getDroppedOutput } = require('./src/core/logger');
const logger = createLogger('process');
process.on('message', () => {
logger.info('process heartbeat');
setTimeout(() => { logger.info('process heartbeat'); process.send(getDroppedOutput()); process.disconnect(); }, 50);
});
process.send('ready');
`], { env: { ...process.env, LOG_REDIS_MIRROR: 'off' }, stdio: ['ignore', 'pipe', 'pipe', 'ipc'] });
let drops, fallback = "";
child.stderr.on("data", chunk => { fallback += chunk; });
child.on('message', message => {
if (message === 'ready') { child.stdout.destroy(); child.send('write'); }
else drops = message;
});
const code = await new Promise((resolve, reject) => { child.on('exit', resolve); child.on('error', reject); });
expect(code).to.equal(0);
expect(drops).to.be.greaterThan(0);
expect(fallback).to.include('"event":"process_heartbeat"');
});
it('preserves admin error codes and expected-error ring suppression', () => {
const child = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', `
const redis = require('redis');
let pushes = 0; const fields = [];
const tx = {lPush() {pushes++; return tx;}, lTrim() {return tx;},
hIncrBy(key, field) { fields.push(field); return tx; }, expire() {return tx;}, exec() {return Promise.resolve([]);} };
redis.createClient = () => ({isReady:true, isOpen:true, on() {}, connect() {return Promise.resolve();}, multi() {return tx;}});
require('./src/core/logger').createLogger('route').warn('repository_not_ready', {message:'repository_not_ready', httpStatus:409});
if (pushes !== 0 || !fields.includes('cb:warn:repository_not_ready')) process.exitCode = 1;
`], {env: {...process.env, LOG_REDIS_MIRROR: 'on'}, encoding: 'utf8', timeout: 10000});
expect(child.status, child.stderr).to.equal(0);
});
it('writes JSON without Redis and bounds a blocked stdout pipe', () => {
const child = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', `
const { createLogger, getDroppedOutput } = require('./src/core/logger');
const logger = createLogger('fixture');
logger.info('request', {status: 200, requestBody: 'secret-fixture'});
Object.defineProperty(process.stdout, 'writableLength', {value: 300000});
for (let i = 0; i < 10000; i++) logger.warn('request', {status: 429});
if (getDroppedOutput() !== 10000) process.exitCode = 1;
`], { cwd: process.cwd(), env: { ...process.env, LOG_REDIS_MIRROR: 'off' }, encoding: 'utf8', timeout: 10000 });
expect(child.status, child.stderr).to.equal(0);
const lines = child.stdout.trim().split('\n');
expect(lines).to.have.length(1);
expect(JSON.parse(lines[0])).to.include({ event: 'request_completed', status: 200 });
expect(child.stdout).not.to.include('secret-fixture');
});
});