mirror of
https://github.com/tdurieux/anonymous_github.git
synced 2026-11-17 17:44:09 +01:00
feat: add private structured logging and investigation dashboards
This commit is contained in:
1 parent
b456ab8739
commit
7e2a6e9d75
39 files changed
+1772
-71
No files matched your search
@@ -0,0 +1,208 @@
|
||||
# Operations logging
|
||||
|
||||
This is an opt-in stack for one Docker host. It does not deploy itself, change
|
||||
production, or send notifications. The application writes structured JSON to
|
||||
stdout. Vector tails tagged Docker json-file logs and host Nginx access JSON,
|
||||
rebuilds each record from an allowlist, then spools sanitized records to disk.
|
||||
VictoriaLogs stores searchable compressed records. Grafana provides investigation
|
||||
panels. Prometheus monitors the logging components and vmalert evaluates LogsQL
|
||||
alerts. Alertmanager groups firing and resolved state with an empty receiver.
|
||||
|
||||
## Deploy after approval
|
||||
|
||||
Review the application and infrastructure changes together. This PR is stacked
|
||||
on `fix/remaining-production-findings`; do not deploy it without that base.
|
||||
|
||||
1. Reserve at least 20 GiB of dedicated logging storage and 2 GiB of RAM, outside
|
||||
the application and MongoDB budgets. The configured memory ceilings total
|
||||
1,856 MiB. Check real free memory before enabling this stack. Docker volumes
|
||||
survive container replacement, but not deletion of the volumes or host loss.
|
||||
2. Create a strong Grafana password in an external secret file. Give UID 472 read
|
||||
access and restrict the parent directory to operators. Do not place the file
|
||||
in the repository. Export `GRAFANA_PASSWORD_FILE` as its absolute path and
|
||||
`GRAFANA_ADMIN_USER` as a non-default username. No password value belongs in
|
||||
a shell command, Compose file, issue, or PR.
|
||||
3. Set `APP_RELEASE` to the deployed 7 to 40 character Git SHA. Build the app through
|
||||
the existing deployment workflow. Add the override when recreating services:
|
||||
`docker compose -f docker-compose.yml -f deploy/logging/app.compose.yml up -d`.
|
||||
This applies explicit service tags and nonblocking 10 MiB × 3 local buffers to
|
||||
the API, all four streamers, MongoDB, Redis and the backup process. BullMQ
|
||||
sandbox workers log through the API container. Existing containers need
|
||||
recreation to change their Docker log configuration.
|
||||
4. Include `../nginx/anonymous-logging.conf` in the host Nginx `http` context and
|
||||
configure the relevant server with
|
||||
`access_log /var/log/nginx/anonymous.access.json anonymous_json;`.
|
||||
Test with `nginx -t`, then reload through the existing deployment workflow.
|
||||
Keep Nginx `real_ip` and proxy trust unchanged. Do not enable Mongo profiling,
|
||||
query tracing, Redis MONITOR, or request-body logging.
|
||||
5. Install `nginx.logrotate` through the host configuration workflow and run it
|
||||
hourly with its own state file. Keep logs root/adm-readable only. This retains
|
||||
about six hourly files, with a 10 MiB rotation trigger checked hourly. A burst
|
||||
can exceed 10 MiB between checks; this is not a hard file quota. Use a dedicated
|
||||
filesystem quota if hard local containment is needed. Use rename/reopen, not
|
||||
copytruncate. Vector excludes compressed history and reads new files from end.
|
||||
6. Set `DOCKER_LOG_DIR` if Docker's data root differs from `/var/lib/docker/containers`,
|
||||
and `NGINX_LOG_DIR` if needed. The collector mounts these directories read-only;
|
||||
it has no Docker socket, host PID namespace or ability to restart applications.
|
||||
Start with `docker compose -f deploy/logging/compose.yml up -d`.
|
||||
7. Tunnel Grafana: `ssh -L 3300:127.0.0.1:3300 your-logging-host`. Open
|
||||
`http://127.0.0.1:3300`, sign in, and open **Anonymous GitHub operations**.
|
||||
Grafana is the only published port, bound to loopback. Ingest, query,
|
||||
Prometheus and Alertmanager are private container-network services. Do not
|
||||
publish their ports or put them behind the public application proxy.
|
||||
8. Verify a synthetic request appears across API/streamer using its response
|
||||
`X-Request-ID`, inspect all four streamer heartbeats and three queue heartbeats,
|
||||
and inspect `ALERTS` in Grafana. Check Docker restart counts and all scrape
|
||||
targets. Observe memory and ingestion drops before increasing traffic.
|
||||
|
||||
Images are version-and-digest pinned in `compose.yml`. Grafana installs the
|
||||
signed VictoriaLogs datasource version 0.34.0 on startup; initial installation
|
||||
needs outbound HTTPS. Automatic plugin upgrades are disabled. For an offline
|
||||
host, preinstall that exact signed plugin in a staged image and validate it before
|
||||
removing the plugin-download network. Never enable unsigned plugins.
|
||||
|
||||
## Privacy and correlation
|
||||
|
||||
`src/core/log-schema.ts` contains reviewed literal module, event and error codes.
|
||||
Unknown messages become `application_log`; arbitrary text is not retained.
|
||||
`log-record.ts` permits numeric measurements, fixed diagnostic values and UUIDs.
|
||||
URLs, paths, repository IDs, owner IDs, cookies, headers, request/response bodies,
|
||||
OAuth query parameters and stacks are omitted. Extend the application schema and
|
||||
collector allowlist together when a new diagnostic is needed. Do not turn unknown
|
||||
strings into labels. Raw Node fatal stderr and dependency output can still reach
|
||||
short local Docker buffers; the collector never forwards their raw text.
|
||||
|
||||
Requests get server-created IDs at the API boundary. Existing API-to-streamer
|
||||
header propagation is unchanged, including its validation. IDs are diagnostics,
|
||||
not authentication or proof of origin. New download, removal and cache jobs carry the
|
||||
request UUID and a random job UUID. Sandbox execution adds attempt and worker
|
||||
service context. Older jobs get a new opaque job UUID
|
||||
when executed. Their historical enqueue request cannot be reconstructed.
|
||||
|
||||
Only `service,severity` identify VictoriaLogs streams. Request, job, instance,
|
||||
release and edge IDs are searchable fields, not stream or metric labels.
|
||||
Nginx edge IDs link to application IDs; Cloudflare ray IDs are accepted only in a
|
||||
fixed format. MongoDB contributes numeric event ID and severity, never `msg`,
|
||||
`attr`, command or query text. Redis and backup output contributes presence only.
|
||||
Malformed tagged records become a metadata event. Oversized lines are discarded
|
||||
with collector error counters. There is no sampling of accepted records.
|
||||
|
||||
Central records omit IP addresses entirely, including unsalted IP hashes.
|
||||
Restricted host Nginx buffers preserve IP evidence for a short incident window.
|
||||
Do not attach those files to issues or PRs. Inspect them locally as an authorized
|
||||
operator. Extending IP retention or exporting IPs needs a separate reviewed change.
|
||||
|
||||
## Budgets and failure behavior
|
||||
|
||||
VictoriaLogs has seven-day retention, a 10 GiB retention target and a 5 GiB free
|
||||
space reserve. It deletes old daily partitions, so the target is not a hard quota:
|
||||
current-day data, merges and indexes can exceed it. Put `/storage` on a dedicated
|
||||
filesystem or apply a host filesystem quota for hard isolation. Reserve 20 GiB
|
||||
for this stack and alert at 8 GiB used or under 8 GiB free. Keep application data
|
||||
on a different filesystem for protection from a logging flood. At the free-space
|
||||
reserve, ingest stops and Vector queues sanitized records.
|
||||
|
||||
Vector's persistent disk buffer is 256 MiB, with `drop_newest`. Its files and
|
||||
checkpoints survive collector restarts. Allocation and metadata add overhead;
|
||||
budget 512 MiB on disk. Outages cause retries with one in-flight request. Once
|
||||
the buffer fills, new records are lost; applications do not wait. Vector syncs
|
||||
the buffer periodically, so sudden power loss can lose the latest records.
|
||||
Collector reads are not a transaction with Docker rotation: downtime beyond the
|
||||
local buffers loses events. Delivery can duplicate records after a retry. There
|
||||
is no exactly-once or host-failure durability claim.
|
||||
|
||||
The application drops stdout writes above 256 KiB pending, counts losses in its
|
||||
heartbeat and handles a broken pipe without crashing. A broken stdout switches to bounded
|
||||
stderr output so later worker heartbeats can still report drops. If both pipes
|
||||
fail, that process cannot report its own telemetry. Docker's nonblocking
|
||||
1 MiB per-container buffer can also drop independently, without an application
|
||||
counter. Compare edge traffic against request completions during overload.
|
||||
Each application container retains at most roughly 30 MiB of Docker files,
|
||||
excluding an oversized single record and filesystem overhead. Logging-stack
|
||||
containers use 5 MiB × 2 files each. Prometheus retains seven days or 512 MB of
|
||||
blocks; WAL/head space needs additional headroom. Alertmanager/Grafana volumes
|
||||
also need space. No compression ratio is promised for production data.
|
||||
|
||||
Admin error pages keep their existing Redis ring and counters by default. Their
|
||||
payloads now contain safe structured diagnostics, so private URLs and stacks are
|
||||
no longer available there. The mirror has no offline queue and a bounded command
|
||||
queue. Once operators use Grafana, `LOG_REDIS_MIRROR=off` removes per-warning
|
||||
Redis writes and ring duplication. This deliberately stops new admin error
|
||||
history; existing history remains until normal expiry/overwrite or an approved
|
||||
cleanup. Leave it on if that admin feature is required. Request/performance
|
||||
metrics and queue admin functions are unchanged.
|
||||
|
||||
## Investigation
|
||||
|
||||
Dashboard panels separate first-byte delay, transfer time and named stages;
|
||||
show client cancellations separately; compare 5xx and latency by release; and
|
||||
show queue waiting age, job failures, memory and process starts. Paste a response
|
||||
UUID into Request UUID, or a job UUID into Job UUID. Use the edge panel to find
|
||||
the application ID for an edge request, then paste that ID into Request UUID.
|
||||
|
||||
Useful LogsQL queries, with the Grafana time picker limiting scope:
|
||||
|
||||
```text
|
||||
requestId:="REQUEST_UUID"
|
||||
jobId:="JOB_UUID"
|
||||
event:process_fatal_error OR event:process_started
|
||||
errorCode:github_rate_limit_exceeded OR errorCode:github_app_reconnect_required
|
||||
service:nginx status:429 | stats by (route, limit_req_status) count() as requests
|
||||
service:api outcome:completed | stats by (release) quantile(0.95, firstByteMs) as p95_ms
|
||||
module:github-app-webhook | stats by (phase, upstreamStatus) count() as failures
|
||||
```
|
||||
|
||||
The alert rules distinguish completed server failures from cancellations and
|
||||
expected 4xx. 5xx require both a minimum count and a ratio sustained for five
|
||||
minutes. Rate-limited traffic stays visible without paging. Queue age ignores
|
||||
intentional delayed jobs. Telemetry alerts expect one API, four streamers and
|
||||
three queues; update the streamer threshold during planned scaling. Memory
|
||||
pressure uses cgroup usage including workers/page cache, with process RSS retained
|
||||
for diagnosis. Missing telemetry is necessary because dead processes cannot emit
|
||||
a final error. A logging-host outage also defeats same-host alerts; independent
|
||||
external monitoring must be configured separately. No receiver is enabled here.
|
||||
|
||||
## Separate logging host and rollback
|
||||
|
||||
For host-failure durability, run VictoriaLogs, Grafana, Prometheus, vmalert and
|
||||
Alertmanager on a separate host with encrypted backups of their persistent volumes.
|
||||
Keep Vector and its local spool on the application host. Change the Vector sink
|
||||
endpoint to a TLS-authenticated private reverse proxy or a restricted VPN address;
|
||||
permit only collector ingress, never public ingestion. Use certificate files or
|
||||
Docker secrets for credentials, not committed values. Scrape Vector over that
|
||||
same private network. This repository's same-host Compose does not claim to set
|
||||
up that network, certificates, backup policy or authentication proxy for you.
|
||||
Test the remote arrangement with synthetic records and a network outage before
|
||||
cutover. Backups must preserve the same access restrictions as live logs.
|
||||
|
||||
Rollback the application image and remove `app.compose.yml` from the deployment
|
||||
command, then recreate application containers using the prior logging settings.
|
||||
Stop the logging stack with `docker compose -f deploy/logging/compose.yml down`
|
||||
without `-v`; that preserves logs, spool, dashboards and alert state. Restore the
|
||||
prior Nginx logging configuration through the host workflow and test before
|
||||
reloading. Keep any retained data private until its approved deletion date.
|
||||
Re-enable `LOG_REDIS_MIRROR=on` if it was disabled. None of these steps requires
|
||||
changing MongoDB data or application queues. Do not delete volumes as rollback.
|
||||
|
||||
## Local validation and references
|
||||
|
||||
Run `npm ci`, `npx tsc --noEmit`, and
|
||||
`LOG_REDIS_MIRROR=off npx mocha test/structured-logging.test.js test/request-monitoring.test.js`.
|
||||
Run `python3 scripts/test-logging.py --stress` with Docker Compose and PyYAML installed.
|
||||
It creates a random project and synthetic files, opens temporary loopback test
|
||||
ports, and removes only its own containers/volumes. It tests actual ingestion,
|
||||
privacy, correlation, query syntax, rotation, restart, backend outage and recovery,
|
||||
retention rejection and fixture storage size. It never reads production logs or
|
||||
uses the application `.env`. The optional `--stress` writes 500,000 synthetic records while the backend is
|
||||
stopped, checks the buffer bound and collector liveness, and waits for the real
|
||||
CollectorDrops alert to fire. Allow 1 GiB temporary disk and several minutes.
|
||||
These fixtures do not establish sustained production throughput.
|
||||
|
||||
Official references checked 2026-11-14:
|
||||
|
||||
- [VictoriaLogs retention and resource flags](https://docs.victoriametrics.com/victorialogs/)
|
||||
- [Vector ingestion setup](https://docs.victoriametrics.com/victorialogs/data-ingestion/vector/)
|
||||
- [Vector buffering guarantees and loss behavior](https://vector.dev/docs/architecture/buffering-model/)
|
||||
- [VictoriaLogs metrics](https://docs.victoriametrics.com/victorialogs/metrics/)
|
||||
- [vmalert LogsQL integration](https://docs.victoriametrics.com/victorialogs/vmalert/)
|
||||
- [Grafana plugin provisioning](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/)
|
||||
@@ -0,0 +1,10 @@
|
||||
route:
|
||||
receiver: local-only
|
||||
group_by: [alertname, service, queue]
|
||||
group_wait: 1m
|
||||
group_interval: 5m
|
||||
repeat_interval: 4h
|
||||
receivers:
|
||||
- name: local-only
|
||||
# Deliberately no email, webhook, chat or pager receiver. Firing/resolved state
|
||||
# remains available in the private UI. Add a receiver only after approval.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Add after docker-compose.yml. Recreating containers is a deployment action.
|
||||
x-buffer: &buffer
|
||||
driver: json-file
|
||||
options: &options
|
||||
max-size: '10m'
|
||||
max-file: '3'
|
||||
mode: non-blocking
|
||||
max-buffer-size: '1m'
|
||||
services:
|
||||
anonymous_github:
|
||||
logging:
|
||||
<<: *buffer
|
||||
options: {<<: *options, tag: api}
|
||||
environment:
|
||||
APP_RELEASE: ${APP_RELEASE:-unknown}
|
||||
LOG_REDIS_MIRROR: ${LOG_REDIS_MIRROR:-on}
|
||||
streamer:
|
||||
logging:
|
||||
<<: *buffer
|
||||
options: {<<: *options, tag: streamer}
|
||||
environment:
|
||||
APP_RELEASE: ${APP_RELEASE:-unknown}
|
||||
LOG_REDIS_MIRROR: ${LOG_REDIS_MIRROR:-on}
|
||||
mongodb:
|
||||
logging:
|
||||
<<: *buffer
|
||||
options: {<<: *options, tag: mongodb}
|
||||
redis:
|
||||
logging:
|
||||
<<: *buffer
|
||||
options: {<<: *options, tag: redis}
|
||||
mongodb-backup:
|
||||
logging:
|
||||
<<: *buffer
|
||||
options: {<<: *options, tag: mongodb-backup}
|
||||
@@ -0,0 +1,94 @@
|
||||
name: anonymous-observability
|
||||
x-common: &common
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: json-file
|
||||
options: {max-size: 5m, max-file: '2', mode: non-blocking, max-buffer-size: 1m}
|
||||
security_opt: [no-new-privileges:true]
|
||||
cap_drop: [ALL]
|
||||
services:
|
||||
victorialogs:
|
||||
<<: *common
|
||||
image: victoriametrics/victoria-logs:v1.54.0@sha256:ee4e820cac3721f5f5d91641b950c22439593adc4224f9655e5333d5a84c1dfe
|
||||
command:
|
||||
- -storageDataPath=/storage
|
||||
- -retentionPeriod=7d
|
||||
- -retention.maxDiskSpaceUsageBytes=10GiB
|
||||
- -storage.minFreeDiskSpaceBytes=5GiB
|
||||
- -memory.allowedBytes=512MiB
|
||||
volumes: [logs:/storage]
|
||||
mem_limit: 768m
|
||||
cpus: 1.0
|
||||
networks: [private]
|
||||
vector:
|
||||
<<: *common
|
||||
image: timberio/vector:0.61.0-debian@sha256:f6c23b5a7e16fb2838d33619ef7acf4bad2289a7cf22bc90efc2371f520c4ea2
|
||||
command: [--config, /etc/vector/vector.yaml]
|
||||
# File access only: no Docker API socket and no host PID namespace.
|
||||
user: '0:0'
|
||||
volumes:
|
||||
- ./vector.yaml:/etc/vector/vector.yaml:ro
|
||||
- ./sanitize.vrl:/etc/vector/sanitize.vrl:ro
|
||||
- ${DOCKER_LOG_DIR:-/var/lib/docker/containers}:/host/docker:ro
|
||||
- ${NGINX_LOG_DIR:-/var/log/nginx}:/host/nginx:ro
|
||||
- spool:/var/lib/vector
|
||||
environment:
|
||||
VECTOR_LOG: warn
|
||||
mem_limit: 256m
|
||||
cpus: 0.5
|
||||
networks: [private]
|
||||
grafana:
|
||||
<<: *common
|
||||
image: grafana/grafana:13.3.1@sha256:0615ec4fbd4072004b7d86b46ded63013a74b7d6eb1119273dd89e8fa25c13cf
|
||||
ports: ['127.0.0.1:${GRAFANA_PORT:-3300}:3000']
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?Set a non-default admin username}
|
||||
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_password
|
||||
GF_USERS_ALLOW_SIGN_UP: 'false'
|
||||
GF_AUTH_ANONYMOUS_ENABLED: 'false'
|
||||
GF_ANALYTICS_REPORTING_ENABLED: 'false'
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: 'false'
|
||||
GF_PLUGINS_PREINSTALL_AUTO_UPDATE: 'false'
|
||||
GF_PLUGINS_DISABLE_PLUGINS: grafana-lokiexplore-app,grafana-metricsdrilldown-app,grafana-exploretraces-app,grafana-pyroscope-app,grafana-pathfinder-app
|
||||
GF_PLUGINS_PREINSTALL_SYNC: victoriametrics-logs-datasource@0.34.0
|
||||
secrets: [grafana_password]
|
||||
volumes:
|
||||
- grafana:/var/lib/grafana
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
mem_limit: 384m
|
||||
networks: [private, plugin_download]
|
||||
prometheus:
|
||||
<<: *common
|
||||
image: prom/prometheus:v3.16.0@sha256:e04987641b57704f408e31f6373ba4138ecf27cb4a165d24aabb6c38fddd0f06
|
||||
command: [--config.file=/etc/prometheus/prometheus.yml, --storage.tsdb.path=/prometheus, --storage.tsdb.retention.time=7d, --storage.tsdb.retention.size=512MB, --web.enable-remote-write-receiver]
|
||||
volumes: [metrics:/prometheus, './prometheus.yml:/etc/prometheus/prometheus.yml:ro', './metrics-alerts.yml:/etc/prometheus/alerts.yml:ro']
|
||||
mem_limit: 256m
|
||||
networks: [private]
|
||||
vmalert:
|
||||
<<: *common
|
||||
image: victoriametrics/vmalert:v1.156.0@sha256:4fea7d1a888d4db4ff18804434ef0f3ca8eb6482e289c50d32001c2db218287c
|
||||
command: [-rule=/etc/alerts.yml, -datasource.url=http://victorialogs:9428, -notifier.url=http://alertmanager:9093, -remoteWrite.url=http://prometheus:9090, -remoteRead.url=http://prometheus:9090]
|
||||
volumes: ['./log-alerts.yml:/etc/alerts.yml:ro']
|
||||
mem_limit: 128m
|
||||
networks: [private]
|
||||
alertmanager:
|
||||
<<: *common
|
||||
image: prom/alertmanager:v0.35.0@sha256:e638797febd034838eca8b98cdbd46f951e9bf7f178777aa8669167406a9946c
|
||||
command: [--config.file=/etc/alertmanager.yml, --storage.path=/alertmanager, --cluster.listen-address=]
|
||||
volumes: [alerts:/alertmanager, './alertmanager.yml:/etc/alertmanager.yml:ro']
|
||||
mem_limit: 64m
|
||||
networks: [private]
|
||||
secrets:
|
||||
grafana_password:
|
||||
file: ${GRAFANA_PASSWORD_FILE:?Set path to an existing secret file}
|
||||
networks:
|
||||
private:
|
||||
internal: true
|
||||
plugin_download: {}
|
||||
volumes:
|
||||
logs: {}
|
||||
spool: {}
|
||||
grafana: {}
|
||||
metrics: {}
|
||||
alerts: {}
|
||||
@@ -0,0 +1,467 @@
|
||||
{
|
||||
"uid": "anonymous-operations",
|
||||
"title": "Anonymous GitHub operations",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "30s",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"tags": [
|
||||
"operations"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "request_id",
|
||||
"label": "Request UUID",
|
||||
"type": "textbox",
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "job_id",
|
||||
"label": "Job UUID",
|
||||
"type": "textbox",
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "edge_id",
|
||||
"label": "Edge request ID",
|
||||
"type": "textbox",
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Failures and cancellations",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "severity:error OR event:request_cancelled",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "First byte versus transfer time",
|
||||
"type": "table",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "event:request* | stats by (service, route) quantile(0.95, firstByteMs) as first_byte_p95_ms, quantile(0.95, responseMs) as transfer_p95_ms",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "Request stages",
|
||||
"type": "table",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "event:request* | stats by (service) avg(stage_upstream_ms) as upstream_ms, avg(stage_worker_wait_ms) as worker_wait_ms, avg(stage_anonymize_ms) as anonymize_ms",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Traffic and rate limits",
|
||||
"type": "table",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "service:nginx | stats by (status, limit_req_status) count() as requests",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Authentication versus upstream failures",
|
||||
"type": "table",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "errorCode:* OR status:401 OR status:403 OR status:>=500 | stats by (service, errorCode, status) count() as events",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"title": "Release comparison",
|
||||
"type": "table",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "service:api outcome:completed | stats by (release) count() as requests, count() if (status:>=500) as server_errors, quantile(0.95, firstByteMs) as first_byte_p95_ms",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"title": "Crashes and restarts",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "event:process_fatal_error OR event:process_started",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"title": "Queue stalls and failures",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "event:job_stalled OR event:job_failed OR event:queue_heartbeat",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"title": "Memory and lost application logs",
|
||||
"type": "table",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "event:process_heartbeat | stats by (service, instance) max(rss) as rss, max(memoryRatio) as memory_ratio, max(droppedLogs) as dropped_logs",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"title": "Request correlation",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "requestId:=\"$request_id\"",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"title": "Job correlation",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "jobId:=\"$job_id\"",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"title": "Edge to application correlation",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "victoriametrics-logs-datasource",
|
||||
"uid": "anonymous-logs"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "edge_request_id:=\"$edge_id\" OR requestId:=\"$request_id\"",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 20,
|
||||
"title": "Storage bytes",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "logging-health"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(vl_data_size_bytes)"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 21,
|
||||
"title": "Free disk bytes",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "logging-health"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "min(vl_free_disk_space_bytes)"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
"title": "Collector discarded events",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "logging-health"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(vector_buffer_discarded_events_total[5m])) or vector(0)) + (sum(rate(vector_component_discarded_events_total{intentional!=\"true\"}[5m])) or vector(0))"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 23,
|
||||
"title": "Firing alerts",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "logging-health"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "ALERTS{alertstate=\"firing\"}"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: Anonymous GitHub operations
|
||||
type: file
|
||||
allowUiUpdates: false
|
||||
options: {path: /var/lib/grafana/dashboards}
|
||||
@@ -0,0 +1,22 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Logs
|
||||
uid: anonymous-logs
|
||||
type: victoriametrics-logs-datasource
|
||||
access: proxy
|
||||
url: http://victorialogs:9428
|
||||
isDefault: true
|
||||
editable: false
|
||||
- name: Logging health
|
||||
uid: logging-health
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
editable: false
|
||||
- name: Alert state
|
||||
uid: logging-alerts
|
||||
type: alertmanager
|
||||
access: proxy
|
||||
url: http://alertmanager:9093
|
||||
jsonData: {implementation: prometheus}
|
||||
editable: false
|
||||
@@ -0,0 +1,55 @@
|
||||
groups:
|
||||
- name: application-logs
|
||||
type: vlogs
|
||||
interval: 30s
|
||||
rules:
|
||||
- alert: SustainedServerErrors
|
||||
expr: '_time:5m service:api outcome:completed | stats count() as total, count() if (status:>=500) as errors | filter errors:>=10 | math errors / total as ratio | filter ratio:>0.05 | fields ratio'
|
||||
for: 5m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'API 5xx exceed 5 percent and 10 requests per five minutes'}
|
||||
- alert: ApiTelemetryMissing
|
||||
expr: '_time:3m service:api event:process_heartbeat | stats count() as heartbeats | filter heartbeats:0'
|
||||
for: 3m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'API heartbeat is missing'}
|
||||
- alert: StreamerTelemetryMissing
|
||||
expr: '_time:3m service:streamer event:process_heartbeat | stats count_uniq(instance) as instances | filter instances:<4'
|
||||
for: 3m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Fewer than four streamer instances report heartbeats; adjust for planned scaling'}
|
||||
- alert: ApplicationRestartBurst
|
||||
expr: '_time:10m event:process_started | stats by (service) count() as restarts | filter restarts:>4'
|
||||
for: 2m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Repeated process starts in ten minutes'}
|
||||
- alert: ApplicationMemoryPressure
|
||||
expr: '_time:3m event:process_heartbeat | stats by (service) max(memoryRatio) as ratio | filter ratio:>0.85'
|
||||
for: 5m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Container usage exceeds 85 percent of its cgroup memory limit'}
|
||||
- alert: QueueStalled
|
||||
expr: '_time:5m event:job_stalled | stats by (queue) count() as stalls | filter stalls:>0'
|
||||
for: 2m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'BullMQ reports stalled jobs'}
|
||||
- alert: QueueWaitingTooLong
|
||||
expr: '_time:3m event:queue_heartbeat | stats by (queue) max(oldestWaitingMs) as age | filter age:>600000'
|
||||
for: 5m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'A runnable job has waited over ten minutes'}
|
||||
- alert: QueueTelemetryMissing
|
||||
expr: '_time:3m event:queue_heartbeat | stats count_uniq(queue) as queues | filter queues:<3'
|
||||
for: 3m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Queue telemetry is missing'}
|
||||
- alert: WebhookFailures
|
||||
expr: '_time:5m route:"/github/app/webhook" status:>=500 outcome:completed | stats count() as errors | filter errors:>=3'
|
||||
for: 3m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Repeated webhook server failures; signature rejection is excluded'}
|
||||
- alert: ApplicationLogDrops
|
||||
expr: '_time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service) max(droppedLogs) as drops | filter drops:>0'
|
||||
for: 2m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'An application process dropped stdout logs since startup'}
|
||||
@@ -0,0 +1,33 @@
|
||||
groups:
|
||||
- name: logging-infrastructure
|
||||
rules:
|
||||
- alert: LoggingTelemetryMissing
|
||||
expr: up == 0
|
||||
for: 3m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'A logging component cannot be scraped'}
|
||||
- alert: CollectorDrops
|
||||
expr: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
|
||||
for: 2m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Collector dropped events; inspect buffer and transform counters'}
|
||||
- alert: CollectorErrors
|
||||
expr: sum(increase(vector_component_errors_total[5m])) > 0
|
||||
for: 3m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Collector errors persist'}
|
||||
- alert: LogStoragePressure
|
||||
expr: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
|
||||
for: 5m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Log storage is near the retention budget or free-space reserve'}
|
||||
- alert: LogIngestionRejected
|
||||
expr: sum(increase(vl_rows_dropped_total[5m])) > 0
|
||||
for: 2m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'VictoriaLogs rejected records'}
|
||||
- alert: LogAlertEvaluationFailed
|
||||
expr: sum(increase(vmalert_execution_errors_total[5m])) > 0
|
||||
for: 3m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Log alert queries are failing'}
|
||||
@@ -0,0 +1,15 @@
|
||||
# Run this configuration hourly, using a dedicated logrotate state file.
|
||||
/var/log/nginx/anonymous.access.json {
|
||||
hourly
|
||||
rotate 6
|
||||
maxsize 10M
|
||||
missingok
|
||||
notifempty
|
||||
compress
|
||||
delaycompress
|
||||
create 0640 www-data adm
|
||||
sharedscripts
|
||||
postrotate
|
||||
/usr/sbin/nginx -s reopen
|
||||
endscript
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
global:
|
||||
scrape_interval: 30s
|
||||
evaluation_interval: 30s
|
||||
rule_files: [/etc/prometheus/alerts.yml]
|
||||
alerting:
|
||||
alertmanagers:
|
||||
- static_configs:
|
||||
- targets: [alertmanager:9093]
|
||||
scrape_configs:
|
||||
- job_name: victorialogs
|
||||
static_configs:
|
||||
- targets: [victorialogs:9428]
|
||||
- job_name: collector
|
||||
static_configs:
|
||||
- targets: [vector:9598]
|
||||
- job_name: log-alerts
|
||||
static_configs:
|
||||
- targets: [vmalert:8880]
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: [localhost:9090]
|
||||
@@ -0,0 +1,61 @@
|
||||
# Rebuild the event from scratch BEFORE the disk buffer. Never retain message,
|
||||
# Docker metadata, Mongo attr/command, request path, host, UA, cookie or headers.
|
||||
raw, err = parse_json(string!(.message))
|
||||
if err != null { raw = {} }
|
||||
service = "nginx"
|
||||
record = raw
|
||||
if starts_with(string!(.file), "/host/docker/") {
|
||||
service = string(raw.attrs.tag) ?? "unknown"
|
||||
if !includes(["api", "streamer", "mongodb", "redis", "mongodb-backup"], service) { abort }
|
||||
record, err = parse_json(string(raw.log) ?? "")
|
||||
if err != null { record = {} }
|
||||
}
|
||||
out = {"timestamp": now(), "service": service, "severity": "info", "event": "service_output"}
|
||||
# Use capture time for unsupported formats. Invalid/old producer timestamps
|
||||
# cannot defeat retention or crash the collector.
|
||||
event_time = raw.time
|
||||
if exists(record.time) { event_time = record.time }
|
||||
if exists(record.timestamp) { event_time = record.timestamp }
|
||||
parsed, err = parse_timestamp(string(event_time) ?? "", format: "%+")
|
||||
if err == null { out.timestamp = parsed }
|
||||
if service == "mongodb" {
|
||||
out.event = "mongodb_event"
|
||||
if is_integer(record.id) { out.mongoEventId = record.id }
|
||||
if includes(["W", "E", "F"], record.s) { out.severity = "warn" }
|
||||
if includes(["E", "F"], record.s) { out.severity = "error" }
|
||||
} else if service == "nginx" {
|
||||
out.event = "edge_request"
|
||||
record.requestId = record.app_request_id
|
||||
record.firstByteMs = (to_float(record.upstream_header_time) ?? 0.0) * 1000.0
|
||||
record.ms = (to_float(record.request_time) ?? 0.0) * 1000.0
|
||||
if is_string(record.edge_request_id) && match(string!(record.edge_request_id), r'^[a-f0-9]{32}$') { out.edge_request_id = record.edge_request_id }
|
||||
if is_string(record.cf_ray) && match(string!(record.cf_ray), r'^[a-f0-9]{16}-[A-Z]{3}$') { out.cf_ray = record.cf_ray }
|
||||
if includes(["PASSED", "DELAYED", "REJECTED", "DELAYED_DRY_RUN", "REJECTED_DRY_RUN"], record.limit_req_status) { out.limit_req_status = record.limit_req_status }
|
||||
# Deliberately omit IPs by default. Original edge JSON has short, restricted
|
||||
# host retention for incident response; no plain or unsalted hashed IP archive.
|
||||
} else if includes(["api", "streamer"], service) {
|
||||
if record.service == "worker" { out.service = "worker" }
|
||||
if includes(["debug", "info", "warn", "error"], record.severity) { out.severity = record.severity }
|
||||
if includes(["account_removed", "application_log", "billing_maintenance_failed", "billing_reconciliation_deferred", "billing_requires_attention", "billing_worker_failed", "blob_api_failed_falling_back_to_raw_url", "branch_not_found", "cache_removal_failed", "cache_removed", "checking_repository_status_and_unused_repositories", "cleared_lock_for_active_job", "conference_expire_failed", "content_expiration_failed", "daily_stats_snapshot_failed", "daily_stats_snapshot_stored", "download_completed", "download_start", "downloaded", "downloadfile_failed", "downloading_file", "downloading_gist", "downloading_pull_request", "downloading_via_raw_url_lfs", "duplicate_stripe_customer_requires_review", "ensuretodaysnapshot_failed", "error_log_redis_connect_failed", "expired_repository_cleanup_failed", "failed_removal_recovery_failed", "failed_to_persist_error_status", "failed_to_record_repository_removal_error", "gettoken", "github_primary_rate_limit_hit", "github_secondary_rate_limit_hit", "grant_revocation_failed", "initial_repository_status_check_failed", "job_completed", "job_failed", "job_stalled", "job_started", "lastusedat_update_failed", "login_token_failed", "login_token_req_login_failed", "logout_after_account_removal_failed", "logout_failed", "lookup_failed", "markerrorifinflight_failed", "markerrorifremoving_failed", "metadata_cleanup_complete", "metadata_cleanup_deferred", "owner_access_alert_reset_unavailable", "owner_access_email_accepted", "owner_access_email_audit_unavailable", "owner_access_email_claim_cleanup_unavailable", "owner_access_email_deferred_recovery_unavailable", "owner_access_email_delivery_failed", "owner_access_email_preparation_cleanup_unavailable", "owner_access_email_recovery_recorded", "owner_access_email_recovery_unavailable", "owner_access_email_result_reconciliation_required", "owner_access_email_unavailable", "pr_comments_404_continuing_without_them", "process_fatal_error", "process_heartbeat", "process_started", "progress", "queue_heartbeat", "queued_for_download", "rate_limit_gate_active_delaying_job_before_work", "rate_limit_gate_set", "rate_limited_delaying_job", "recover_failed", "recovered_expired_repository_cleanup", "recovered_file_from_truncated_tree", "recovered_file_invalidation_failed", "recovered_stuck_repo", "recoverstuckpreparing_failed", "recoverstuckremoving_failed", "redis_client_error", "redis_rate_limit_gate_written", "rejected_upstream_download_destination", "removal_failed", "removal_recovery_failed", "removecache_save_failed", "removed_cache_for_unused_repository", "removing_cache", "removing_repository", "repo_claimed", "repository_cache_removal_failed", "repository_expiration_failed", "repository_expired", "repository_removal_failed", "repository_removed", "request_cancelled", "request_completed", "request_slow", "requeued_failed_removal", "requeued_interrupted_removal", "reset", "retired_tree_cleanup_deferred", "retired_tree_cleanup_failed", "running_daily_stats_snapshot", "s3_body_stream_error", "send_failed", "server_started", "source_stream_failed_cache_write_discarded", "staged_tree_cleanup_failed", "staged_tree_completion_deferred", "streamer_fetch_failed", "streamer_started", "streamer_startup_failed", "streamer_zip_fetch_failed", "temporary_storage_cleanup_deferred", "token_refresh_failed_backing_off", "up_to_date", "update_queued", "verify_failed", "waiting_for_rate_limit_gate", "webhook_log", "webhook_processing_failed", "write_failed", "zip_stream_failed"], record.event) { out.event = record.event }
|
||||
if is_string(record.release) && match(string!(record.release), r'^[a-f0-9]{7,40}$') { out.release = record.release }
|
||||
if is_string(record.instance) && match(string!(record.instance), r'^[a-f0-9-]{36}$') { out.instance = record.instance }
|
||||
if includes(["ABORT_ERR", "ECONNREFUSED", "ECONNRESET", "ENOTFOUND", "EPIPE", "ETIMEDOUT", "billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_app_reconnect_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_token_invalid", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_signature", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "token_refresh_failed", "quota_exceeded", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_found", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "upstream_timeout", "user_not_found", "username_not_defined", "webhook_processing_failed"], record.errorCode) { out.errorCode = record.errorCode }
|
||||
if includes(["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"], record.module) { out.module = record.module }
|
||||
if includes(["protocol", "credentials", "port", "host"], record.reason) { out.reason = record.reason }
|
||||
if includes(["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"], record.phase) { out.phase = record.phase }
|
||||
if includes(["github_app_authorization", "installation", "installation_repositories"], record.webhookEvent) { out.webhookEvent = record.webhookEvent }
|
||||
if includes(["github.com", "api.github.com", "codeload.github.com", "raw.githubusercontent.com", "media.githubusercontent.com", "other"], record.upstreamHost) { out.upstreamHost = record.upstreamHost }
|
||||
if includes(["download", "remove", "cache"], record.queue) { out.queue = record.queue }
|
||||
if includes(["completed", "interrupted"], record.outcome) { out.outcome = record.outcome }
|
||||
}
|
||||
for_each(["requestId", "jobId", "deliveryId"]) -> |_index, key| {
|
||||
value = get(record, [key]) ?? null
|
||||
if is_string(value) && match(string!(value), r'^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$') { out = set!(out, [key], value) }
|
||||
}
|
||||
for_each(["status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus", "stage_repository_ms", "stage_credential_resolution_ms", "stage_upstream_ms", "stage_cache_lookup_ms", "stage_cache_fill_ms", "stage_worker_wait_ms", "stage_anonymize_ms", "stage_streamer_headers_ms"]) -> |_index, key| {
|
||||
value = get(record, [key]) ?? null
|
||||
if is_integer(value) || is_float(value) { out = set!(out, [key], value) }
|
||||
}
|
||||
if includes(["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "OTHER"], record.method) { out.method = record.method }
|
||||
if includes(["/", "other", "/dashboard", "/anonymize", "/healthcheck", "/api/healthcheck", "/api", "/api/download", "/api/stat", "/api/stat/history", "/api/options", "/api/message", "/api/user", "/api/user/quota", "/api/user/dashboard", "/api/repo", "/api/anonymize-preview", "/api/admin/overview", "/api/admin/performance", "/api/admin/errors", "/api/admin/errors/stats", "/api/admin/queues", "/api/admin/queues/metrics", "/github/app/webhook", "/api/repo/:repoId", "/api/repo/:repoId/file/:path", "/api/repo/:repoId/files", "/api/repo/:repoId/files/counts", "/api/repo/:repoId/files/search", "/api/repo/:repoId/zip", "/api/repo/:repoId/options", "/api/repo/:repoId/refresh", "/api/repo/:repoId/coauthors", "/api/repo/:repoId/extend", "/api/repo/:owner/:repo/readme", "/api/repo/:owner/:repo/branches", "/w/:repoId/:path", "/r/:repoId/:path", "/api/pr/:id/:path", "/api/gist/:id/:path", "/assets/:path", "/github/:action", "/api/admin/:action", "/api/user/:action", "/api/repo/:id/file/:path", "/api/repo/:id/files/counts", "/api/repo/:id/files/search", "/api/repo/:id/files", "/api/repo/:id/zip", "/api/repo/:id", "/viewer/:id/:path"], record.route) { out.route = record.route }
|
||||
. = out
|
||||
@@ -0,0 +1,49 @@
|
||||
data_dir: /var/lib/vector
|
||||
sources:
|
||||
docker:
|
||||
type: file
|
||||
include: [/host/docker/*/*-json.log]
|
||||
read_from: end
|
||||
max_line_bytes: 65536
|
||||
max_read_bytes: 65536
|
||||
rotate_wait_secs: 60
|
||||
fingerprint: {strategy: device_and_inode}
|
||||
nginx:
|
||||
type: file
|
||||
include: [/host/nginx/anonymous.access.json*]
|
||||
exclude: ['*.gz']
|
||||
read_from: end
|
||||
max_line_bytes: 65536
|
||||
fingerprint: {strategy: device_and_inode}
|
||||
internal:
|
||||
type: internal_metrics
|
||||
scrape_interval_secs: 15
|
||||
transforms:
|
||||
safe:
|
||||
type: remap
|
||||
inputs: [docker, nginx]
|
||||
file: /etc/vector/sanitize.vrl
|
||||
drop_on_error: true
|
||||
drop_on_abort: true
|
||||
sinks:
|
||||
logs:
|
||||
type: elasticsearch
|
||||
inputs: [safe]
|
||||
endpoints: [http://victorialogs:9428/insert/elasticsearch/]
|
||||
api_version: v8
|
||||
compression: gzip
|
||||
healthcheck: {enabled: false}
|
||||
query:
|
||||
_msg_field: event
|
||||
_time_field: timestamp
|
||||
_stream_fields: service,severity
|
||||
batch: {max_bytes: 1048576, timeout_secs: 1}
|
||||
request: {timeout_secs: 5, concurrency: 1}
|
||||
buffer:
|
||||
type: disk
|
||||
max_size: 268435488
|
||||
when_full: drop_newest
|
||||
metrics:
|
||||
type: prometheus_exporter
|
||||
inputs: [internal]
|
||||
address: 0.0.0.0:9598
|
||||
@@ -156,3 +156,7 @@ The constrained-heap regression fixture can be run without network or MongoDB:
|
||||
|
||||
See [Shared blob cache](shared-cache.md) for configuration, storage and quota
|
||||
accounting, admin metrics, production observations, rollout and benchmark commands.
|
||||
|
||||
For opt-in persistent structured logs, cross-service request/job investigation,
|
||||
and private Grafana dashboards, see [operations logging](../deploy/logging/README.md).
|
||||
The existing Redis performance/admin endpoints remain available during migration.
|
||||
@@ -9,17 +9,26 @@ function observeLine(summary, line) {
|
||||
// Older releases rely on Node's stderr. New releases also emit a redacted
|
||||
// process_fatal_error event before Node terminates.
|
||||
if (/ECONNRESET|socket hang up/i.test(line)) summary.connectionResets++;
|
||||
const match = /^\S+ (INFO|WARN|ERROR) \[([^\]]+)\] .*? (\{.*\})$/.exec(line);
|
||||
if (!match) {
|
||||
if (/Unhandled ['"]error['"] event|unhandled(?:promise)?rejection|FATAL ERROR|heap out of memory/i.test(line)) summary.fatalErrors++;
|
||||
return;
|
||||
let data, level, scope;
|
||||
try {
|
||||
const record = JSON.parse(line);
|
||||
if (record && typeof record === "object" && typeof record.severity === "string") {
|
||||
data = record; level = record.severity.toUpperCase(); scope = record.module;
|
||||
data.code = record.errorCode || record.event;
|
||||
}
|
||||
} catch { /* Support pre-migration mixed text/JSON too. */ }
|
||||
if (!data) {
|
||||
const match = /^\S+ (INFO|WARN|ERROR) \[([^\]]+)\] .*? (\{.*\})$/.exec(line);
|
||||
if (!match) {
|
||||
if (/Unhandled ['"]error['"] event|unhandled(?:promise)?rejection|FATAL ERROR|heap out of memory/i.test(line)) summary.fatalErrors++;
|
||||
return;
|
||||
}
|
||||
try { data = JSON.parse(match[3]); } catch { return; }
|
||||
[, level, scope] = match;
|
||||
}
|
||||
let data;
|
||||
try { data = JSON.parse(match[3]); } catch { return; }
|
||||
const [, level, scope] = match;
|
||||
if (level === "ERROR") summary.errors++;
|
||||
if (level === "WARN") summary.warnings++;
|
||||
if (data.code === "process_fatal_error" || /startup failed/i.test(line)) summary.fatalErrors++;
|
||||
if (data.code === "process_fatal_error" || /startup[ _]failed/i.test(line)) summary.fatalErrors++;
|
||||
if (data.code === "token_refresh_failed") summary.tokenRefreshFailures++;
|
||||
if (scope === "anonymized-file" && data.code === "file_not_found") summary.missingFiles++;
|
||||
if (scope === "requests" && typeof data.ms === "number") {
|
||||
|
||||
Executable
+236
@@ -0,0 +1,236 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Isolated local Docker integration test. Never reads production logs or .env.
|
||||
Requires Docker Compose, Python 3 and PyYAML. Removes only its own project.
|
||||
"""
|
||||
import json
|
||||
import base64
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import sys
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import uuid
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
CONFIG = ROOT / 'deploy/logging'
|
||||
PROJECT = 'logging-test-' + secrets.token_hex(4)
|
||||
|
||||
def run(args, **kw):
|
||||
return subprocess.run(args, check=True, text=True, capture_output=True, **kw).stdout.strip()
|
||||
|
||||
def wait_for(fn, timeout=60):
|
||||
end = time.monotonic() + timeout
|
||||
last = None
|
||||
while time.monotonic() < end:
|
||||
try:
|
||||
result = fn()
|
||||
if result:
|
||||
return result
|
||||
except Exception as e:
|
||||
last = e
|
||||
time.sleep(1)
|
||||
raise AssertionError('Timed out: ' + str(last))
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix=PROJECT) as temp:
|
||||
temp = Path(temp)
|
||||
docker = temp / 'docker'
|
||||
nginx = temp / 'nginx'
|
||||
docker.mkdir(); nginx.mkdir()
|
||||
(docker / 'fixture').mkdir()
|
||||
log = docker / 'fixture/fixture-json.log'
|
||||
log.touch()
|
||||
edge = nginx / 'anonymous.access.json'
|
||||
edge.touch()
|
||||
password = temp / 'password'
|
||||
password.write_text(secrets.token_urlsafe(32))
|
||||
password.chmod(0o644) # temp directory is 0700; Grafana uid must read the bind mount
|
||||
env = {**os.environ, 'GRAFANA_ADMIN_USER': 'local-test', 'GRAFANA_PASSWORD_FILE': str(password),
|
||||
'DOCKER_LOG_DIR': str(docker), 'NGINX_LOG_DIR': str(nginx), 'GRAFANA_PORT': '0'}
|
||||
override = temp / 'override.yml'
|
||||
override.write_text(yaml.safe_dump({'services': {
|
||||
'victorialogs': {'ports': ['127.0.0.1::9428']},
|
||||
'prometheus': {'ports': ['127.0.0.1::9090']},
|
||||
'vector': {'ports': ['127.0.0.1::9598']},
|
||||
'vmalert': {'ports': ['127.0.0.1::8880']},
|
||||
}, 'networks': {'private': {'internal': False}}}))
|
||||
cmd = ['docker', 'compose', '-p', PROJECT, '-f', str(CONFIG / 'compose.yml'), '-f', str(override)]
|
||||
def compose(*args):
|
||||
return run(cmd + list(args), env=env)
|
||||
def url(service, port):
|
||||
return 'http://' + compose('port', service, str(port))
|
||||
def get(base, path, params=None):
|
||||
address = base + path + ('?' + urllib.parse.urlencode(params) if params else '')
|
||||
return urllib.request.urlopen(address, timeout=10).read().decode()
|
||||
def query(expression):
|
||||
return [json.loads(line) for line in get(vl, '/select/logsql/query', {'query': expression, 'limit': 50000}).splitlines() if line]
|
||||
def append(service, record):
|
||||
with log.open('a') as out:
|
||||
out.write(json.dumps({'log': json.dumps(record) + '\n', 'attrs': {'tag': service}, 'stream': 'stdout'}) + '\n')
|
||||
def count():
|
||||
return int(query('* | stats count() as n')[0]['n'])
|
||||
try:
|
||||
compose('up', '-d')
|
||||
vl = url('victorialogs', 9428)
|
||||
prom = url('prometheus', 9090)
|
||||
vector = url('vector', 9598)
|
||||
alerts = url('vmalert', 8880)
|
||||
grafana = url('grafana', 3000)
|
||||
wait_for(lambda: get(vl, '/health') is not None)
|
||||
wait_for(lambda: get(vector, '/metrics'))
|
||||
wait_for(lambda: get(grafana, '/api/health'))
|
||||
wait_for(lambda: 'application-logs' in get(alerts, '/api/v1/rules'))
|
||||
time.sleep(2) # file watcher must discover empty files before append
|
||||
request_id, job_id = str(uuid.uuid4()), str(uuid.uuid4())
|
||||
sentinel = 'PRIVATE_SENTINEL_owner_repository_cookie_oauth_state'
|
||||
for service in ['api', 'streamer']:
|
||||
append(service, {'event': 'request_completed', 'severity': 'info', 'requestId': request_id,
|
||||
'route': '/api/repo/:repoId/file/:path', 'status': 200, 'outcome': 'completed',
|
||||
'firstByteMs': 40, 'responseMs': 600, 'ms': 640, 'stage_upstream_ms': 20,
|
||||
'message': sentinel, 'url': 'https://example.invalid/' + sentinel, 'cookie': sentinel,
|
||||
'authorization': sentinel, 'requestBody': {'secret': sentinel}, 'release': 'abcdef1'})
|
||||
append('api', {'service': 'worker', 'event': 'job_failed', 'severity': 'error', 'requestId': request_id,
|
||||
'jobId': job_id, 'attempt': 2, 'queue': 'download', 'errorCode': 'ETIMEDOUT', 'raw': sentinel})
|
||||
append('mongodb', {'id': 51803, 's': 'E', 'msg': sentinel, 'attr': {'command': sentinel}})
|
||||
append('redis', {'message': sentinel})
|
||||
append('mongodb-backup', {'message': sentinel})
|
||||
append('unrelated-container', {'message': sentinel})
|
||||
# Malformed and malicious field values are represented without their raw content.
|
||||
append('api', {'event': sentinel, 'route': '/' + sentinel, 'requestId': sentinel, 'severity': sentinel, 'module': sentinel})
|
||||
with log.open('a') as out:
|
||||
out.write(json.dumps({'log': sentinel, 'attrs': {'tag': 'api'}}) + '\n')
|
||||
with edge.open('a') as out:
|
||||
out.write(json.dumps({'status': 429, 'request_time': 0.1, 'upstream_header_time': '0.05',
|
||||
'app_request_id': request_id, 'edge_request_id': 'f' * 32, 'route': '/api/repo/:id',
|
||||
'limit_req_status': 'REJECTED', 'client_ip': '192.0.2.1', 'user_agent': sentinel,
|
||||
'host': sentinel, 'request': sentinel}) + '\n')
|
||||
wait_for(lambda: count() >= 9)
|
||||
rows = query('*')
|
||||
stored = json.dumps(rows)
|
||||
assert sentinel not in stored and '192.0.2.1' not in stored
|
||||
correlated = query('requestId:="' + request_id + '"')
|
||||
assert {r['service'] for r in correlated} == {'api', 'streamer', 'worker', 'nginx'}
|
||||
assert len(query('jobId:="' + job_id + '"')) == 1
|
||||
assert all('requestId' not in r['_stream'] and 'instance' not in r['_stream'] for r in rows)
|
||||
print('PASS ingestion, privacy allowlist, malformed input, API/streamer/job/edge correlation')
|
||||
# Query every shipped dashboard and alert expression against the real server.
|
||||
dashboard = json.loads((CONFIG / 'grafana/dashboards/operations.json').read_text())
|
||||
for panel in dashboard['panels']:
|
||||
for target in panel['targets']:
|
||||
expr = target['expr'].replace('$request_id', request_id).replace('$job_id', job_id).replace('$edge_id', 'f' * 32)
|
||||
if panel['datasource']['uid'] == 'anonymous-logs':
|
||||
query(expr)
|
||||
else:
|
||||
get(prom, '/api/v1/query', {'query': expr})
|
||||
for group in yaml.safe_load((CONFIG / 'log-alerts.yml').read_text())['groups']:
|
||||
for rule in group['rules']:
|
||||
get(vl, '/select/logsql/stats_query', {'query': rule['expr']})
|
||||
auth = base64.b64encode(('local-test:' + password.read_text()).encode()).decode()
|
||||
def grafana_request(path, payload=None):
|
||||
request = urllib.request.Request(grafana + path,
|
||||
data=json.dumps(payload).encode() if payload is not None else None,
|
||||
headers={'Authorization': 'Basic ' + auth, 'Content-Type': 'application/json'})
|
||||
return json.loads(urllib.request.urlopen(request, timeout=20).read())
|
||||
assert grafana_request('/api/dashboards/uid/anonymous-operations')['dashboard']['uid'] == 'anonymous-operations'
|
||||
assert grafana_request('/api/datasources/uid/anonymous-logs/health')['status'] == 'OK'
|
||||
result = grafana_request('/api/ds/query', {'from': str(int((time.time() - 3600) * 1000)),
|
||||
'to': str(int(time.time() * 1000)), 'queries': [{'refId': 'A',
|
||||
'datasource': {'type': 'victoriametrics-logs-datasource', 'uid': 'anonymous-logs'},
|
||||
'expr': 'requestId:="' + request_id + '"', 'queryType': 'instant', 'maxLines': 100}]})
|
||||
assert not result['results']['A'].get('error'), result['results']['A'].get('error')
|
||||
assert result['results']['A']['frames']
|
||||
print('PASS dashboard and alert queries, provisioned Grafana datasource query')
|
||||
# Rename rotation, then collector restart, preserving the spool/checkpoint volume.
|
||||
before = count()
|
||||
log.rename(log.with_suffix('.log.1'))
|
||||
log.touch()
|
||||
append('api', {'event': 'process_started', 'severity': 'info'})
|
||||
wait_for(lambda: count() > before)
|
||||
compose('restart', 'vector')
|
||||
vector = url('vector', 9598)
|
||||
wait_for(lambda: get(vector, '/metrics'))
|
||||
before = count()
|
||||
append('api', {'event': 'process_heartbeat', 'severity': 'info'})
|
||||
wait_for(lambda: count() > before)
|
||||
print('PASS rename rotation and collector checkpoint restart')
|
||||
# Backend failure: sanitized disk spool remains bounded; source writer stays independent.
|
||||
before = count()
|
||||
compose('stop', 'victorialogs')
|
||||
started = time.monotonic()
|
||||
for _ in range(2000):
|
||||
append('api', {'event': 'request_completed', 'severity': 'info', 'requestId': str(uuid.uuid4()), 'status': 200})
|
||||
assert time.monotonic() - started < 10
|
||||
wait_for(lambda: 'vector_buffer_size_events' in get(vector, '/metrics'))
|
||||
time.sleep(3)
|
||||
compose('restart', 'vector')
|
||||
compose('start', 'victorialogs')
|
||||
vl = url('victorialogs', 9428)
|
||||
wait_for(lambda: count() >= before + 2000, 120)
|
||||
print('PASS unavailable backend, spool persistence, recovery of 2000 records')
|
||||
# Reject data older than retention rather than resurrecting it.
|
||||
old = json.dumps({'timestamp': '2000-01-01T00:00:00Z', 'event': 'expired_fixture'}).encode()
|
||||
req = urllib.request.Request(vl + '/insert/jsonline?_time_field=timestamp', data=old, headers={'Content-Type': 'application/stream+json'})
|
||||
urllib.request.urlopen(req, timeout=10).read()
|
||||
time.sleep(2)
|
||||
assert not query('event:expired_fixture')
|
||||
assert 'reason="too_small_timestamp"' in get(vl, '/metrics')
|
||||
print('PASS retention rejection')
|
||||
# Compression observation, synthetic workload only, including random trace IDs.
|
||||
raw_bytes = 0
|
||||
for _ in range(10000):
|
||||
record = {'event': 'request_completed', 'severity': 'info', 'requestId': str(uuid.uuid4()),
|
||||
'status': 200, 'ms': 50, 'firstByteMs': 20, 'responseMs': 30, 'route': '/api/repo/:repoId/file/:path'}
|
||||
raw_bytes += len(json.dumps(record).encode()) + 1
|
||||
append('api', record)
|
||||
wait_for(lambda: count() >= before + 12000, 120)
|
||||
compose('restart', 'victorialogs')
|
||||
vl = url('victorialogs', 9428)
|
||||
wait_for(lambda: get(vl, '/health') is not None)
|
||||
metrics_text = get(vl, '/metrics')
|
||||
size = sum(float(line.split()[-1]) for line in metrics_text.splitlines() if line.startswith('vl_data_size_bytes{'))
|
||||
print(f'MEASURED fixture JSON bytes={raw_bytes}; total store data+index bytes={int(size)}; store also contains earlier fixtures; no production ratio claimed')
|
||||
# Check fixed limits and no accidental public ingress.
|
||||
config = json.loads(compose('config', '--format', 'json'))
|
||||
for service, spec in config['services'].items():
|
||||
assert int(spec['mem_limit']) > 0, service
|
||||
for port in spec.get('ports', []):
|
||||
assert port['host_ip'] == '127.0.0.1', service
|
||||
print('PASS configured memory/rotation/disk bounds and loopback-only test ports')
|
||||
if '--stress' in sys.argv:
|
||||
compose('stop', 'victorialogs')
|
||||
vector = url('vector', 9598)
|
||||
numeric_fields = ['status', 'httpStatus', 'ms', 'firstByteMs', 'responseMs', 'bytes', 'bytesPerSecond',
|
||||
'attempt', 'rss', 'memoryLimitBytes', 'memoryRatio', 'uptime', 'activeRequests', 'waiting',
|
||||
'active', 'delayed', 'failed', 'oldestWaitingMs', 'droppedLogs', 'loopP95Ms', 'upstreamStatus',
|
||||
'stage_repository_ms', 'stage_credential_resolution_ms', 'stage_upstream_ms', 'stage_cache_lookup_ms',
|
||||
'stage_cache_fill_ms', 'stage_worker_wait_ms', 'stage_anonymize_ms', 'stage_streamer_headers_ms']
|
||||
stress = {field: 123456789 for field in numeric_fields}
|
||||
stress.update(event='request_completed', severity='info', requestId=request_id)
|
||||
line = json.dumps({'log': json.dumps(stress), 'attrs': {'tag': 'api'}}) + '\n'
|
||||
with log.open('a') as output:
|
||||
for _ in range(500):
|
||||
output.write(line * 1000)
|
||||
def metric_value(name):
|
||||
lines = get(vector, '/metrics').splitlines()
|
||||
return sum(float(line.split()[1]) for line in lines if line.startswith(name + '{') and ('buffer_id="logs"' in line if name.startswith('vector_buffer_') else 'component_id="logs"' in line))
|
||||
wait_for(lambda: metric_value('vector_buffer_discarded_events_total') > 0 or metric_value('vector_component_discarded_events_total') > 0, 240)
|
||||
buffered = metric_value('vector_buffer_size_bytes')
|
||||
assert buffered <= 268435488 + 1048576, buffered
|
||||
inspection = json.loads(run(['docker', 'inspect', compose('ps', '-q', 'vector')]))[0]
|
||||
assert inspection['RestartCount'] == 0 and not inspection['State']['OOMKilled']
|
||||
print(f'PASS full-spool drop-newest stress: 500000 events; buffered={int(buffered)} bytes; collector stayed alive')
|
||||
# Wait through the production two-minute 'for', rather than merely
|
||||
# checking PromQL syntax or that a counter exists.
|
||||
wait_for(lambda: json.loads(get(prom, '/api/v1/query', {'query':
|
||||
'ALERTS{alertname="CollectorDrops",alertstate="firing"}'}))['data']['result'], 210)
|
||||
print('PASS sustained CollectorDrops alert fires on full-spool loss')
|
||||
except Exception:
|
||||
# Only synthetic fixture logs exist in this project. No production reads.
|
||||
print(compose('logs', '--tail', '30', 'vector', 'victorialogs', 'vmalert', 'grafana'))
|
||||
raise
|
||||
finally:
|
||||
compose('down', '-v', '--remove-orphans')
|
||||
@@ -1,3 +1,4 @@
|
||||
import { jobTrace } from "./log-context";
|
||||
import { REPOSITORY_ID_COLLATION } from "./repository-id";
|
||||
import { reserveRepository } from "./billing/quotas";
|
||||
import { hasAutomaticUpdates } from "./billing/expiration";
|
||||
@@ -855,7 +856,7 @@ export default class Repository {
|
||||
const current = await AnonymizedRepositoryModel.exists({ _id: this.model._id, ...this.refreshFilter() });
|
||||
if (!current) throw new AnonymousError("invalid_status", { httpStatus: 409 });
|
||||
}
|
||||
await downloadQueue.add(this.repoId, { repoId: this.repoId }, {
|
||||
await downloadQueue.add(this.repoId, { repoId: this.repoId, ...jobTrace() }, {
|
||||
jobId: `repo-${this.repoId}`,
|
||||
attempts: 3,
|
||||
});
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
import { AsyncLocalStorage } from "async_hooks";
|
||||
import { randomUUID } from "crypto";
|
||||
|
||||
export interface LogContext { requestId?: string; jobId?: string; attempt?: number; service?: string }
|
||||
export const logContext = new AsyncLocalStorage<LogContext>();
|
||||
export const validTraceId = (value: unknown): value is string => typeof value === "string" &&
|
||||
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(value);
|
||||
export function jobTrace() {
|
||||
const requestId = logContext.getStore()?.requestId;
|
||||
return { logJobId: randomUUID(), ...(requestId ? { requestId } : {}) };
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { readFile } from "fs/promises";
|
||||
/** Container usage includes sandbox workers and page cache; process RSS does not. */
|
||||
export async function logMemory() {
|
||||
const rss = process.memoryUsage().rss;
|
||||
for (const [usage, limit] of [
|
||||
["/sys/fs/cgroup/memory.current", "/sys/fs/cgroup/memory.max"],
|
||||
["/sys/fs/cgroup/memory/memory.usage_in_bytes", "/sys/fs/cgroup/memory/memory.limit_in_bytes"],
|
||||
]) {
|
||||
const [used, max] = await Promise.all([usage, limit].map(path => readFile(path, "utf8").then(Number).catch(() => NaN)));
|
||||
if (Number.isFinite(used) && max > 0 && max < Number.MAX_SAFE_INTEGER) {
|
||||
return { rss, memoryLimitBytes: max, memoryRatio: used / max };
|
||||
}
|
||||
}
|
||||
return { rss };
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
import { LOG_MODULES, ERROR_CODES, LOG_EVENTS } from "./log-schema";
|
||||
import { randomUUID } from "crypto";
|
||||
import { logContext, validTraceId } from "./log-context";
|
||||
|
||||
// No arbitrary strings: upstream errors, paths and even innocuous-looking messages
|
||||
// can contain a private repository name. Add diagnostics here deliberately.
|
||||
const instance = randomUUID();
|
||||
const events: Record<string, string> = {
|
||||
...LOG_EVENTS,
|
||||
request: "request_completed", "slow request": "request_slow", "request interrupted": "request_cancelled",
|
||||
"process fatal error": "process_fatal_error", "process started": "process_started",
|
||||
"process heartbeat": "process_heartbeat", "job started": "job_started", "job completed": "job_completed",
|
||||
"job failed": "job_failed", "job stalled": "job_stalled", "queue heartbeat": "queue_heartbeat",
|
||||
};
|
||||
const errorCodes = new Set([
|
||||
...ERROR_CODES,
|
||||
"token_refresh_failed", "quota_exceeded", "github_rate_limit_exceeded", "github_app_reconnect_required", "github_token_invalid",
|
||||
"repository_expired", "repository_not_ready", "repository_not_found", "repository_job_cancelled",
|
||||
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND", "EPIPE", "ABORT_ERR",
|
||||
"upstream_error", "upstream_timeout", "invalid_signature", "webhook_processing_failed",
|
||||
]);
|
||||
const numbers = ["status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond",
|
||||
"attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed",
|
||||
"oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus"];
|
||||
const stages = ["repository", "credential_resolution", "upstream", "cache_lookup", "cache_fill", "worker_wait", "anonymize", "streamer_headers"];
|
||||
export function logRecord(level: string, module: string, args: unknown[]) {
|
||||
const context = logContext.getStore() || {};
|
||||
const details = Object.assign({}, ...args.filter(a => a && typeof a === "object" && !Array.isArray(a)));
|
||||
const first = typeof args[0] === "string" ? args[0] : "";
|
||||
const event = events[first] || (module.includes("webhook") ? "webhook_log" : "application_log");
|
||||
const record: Record<string, unknown> = { timestamp: new Date().toISOString(), severity: level,
|
||||
service: context.service || (["api", "streamer", "worker"].includes(details.service) ? details.service : undefined) || (process.env.SERVICE_NAME?.toLowerCase() === "streamer" ? "streamer" : "api"),
|
||||
instance, release: /^[a-f0-9]{7,40}$/.test(process.env.APP_RELEASE || "") ? process.env.APP_RELEASE : "unknown",
|
||||
event, message: event, module: LOG_MODULES.includes(module) ? module : "application" };
|
||||
for (const key of ["requestId", "jobId"] as const) {
|
||||
const value = details[key] || context[key];
|
||||
if (validTraceId(value)) record[key] = value;
|
||||
}
|
||||
if (Number.isSafeInteger(context.attempt)) record.attempt = context.attempt;
|
||||
for (const key of numbers) if (typeof details[key] === "number" && Number.isFinite(details[key])) record[key] = details[key];
|
||||
const code = [details.code, details.message].find(value => errorCodes.has(value));
|
||||
if (code) record.errorCode = code;
|
||||
if (["protocol", "credentials", "port", "host"].includes(details.reason)) record.reason = details.reason;
|
||||
if (["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"].includes(details.phase)) record.phase = details.phase;
|
||||
if (["github_app_authorization", "installation", "installation_repositories"].includes(details.event)) record.webhookEvent = details.event;
|
||||
if (validTraceId(details.deliveryId)) record.deliveryId = details.deliveryId;
|
||||
if (["github.com", "api.github.com", "codeload.github.com", "raw.githubusercontent.com", "media.githubusercontent.com"].includes(details.hostname)) record.upstreamHost = details.hostname;
|
||||
else if (details.hostname) record.upstreamHost = "other";
|
||||
if (["completed", "interrupted"].includes(details.outcome)) record.outcome = details.outcome;
|
||||
if (["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS", "OTHER"].includes(details.method)) record.method = details.method;
|
||||
// Route labels are generated by request-monitoring, never from URLs in arbitrary errors.
|
||||
if (module === "requests" && typeof details.route === "string") record.route = details.route;
|
||||
if (["download", "remove", "cache"].includes(details.queue)) record.queue = details.queue;
|
||||
if (details.stages && typeof details.stages === "object") for (const key of stages) {
|
||||
const value = details.stages[key];
|
||||
if (typeof value === "number" && Number.isFinite(value)) record[`stage_${key}_ms`] = value;
|
||||
}
|
||||
return record;
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
// Reviewed literal diagnostics. Never extend this list from runtime data.
|
||||
export const LOG_MODULES = ["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"];
|
||||
export const ERROR_CODES = ["billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "user_not_found", "username_not_defined"];
|
||||
export const LOG_EVENTS: Record<string, string> = {
|
||||
"PR comments 404, continuing without them": "pr_comments_404_continuing_without_them",
|
||||
"S3 body stream error": "s3_body_stream_error",
|
||||
"account removed": "account_removed",
|
||||
"billing maintenance failed": "billing_maintenance_failed",
|
||||
"billing reconciliation deferred": "billing_reconciliation_deferred",
|
||||
"billing requires attention": "billing_requires_attention",
|
||||
"billing worker failed": "billing_worker_failed",
|
||||
"blob API failed, falling back to raw URL": "blob_api_failed_falling_back_to_raw_url",
|
||||
"branch not found": "branch_not_found",
|
||||
"cache removal failed": "cache_removal_failed",
|
||||
"cache removed": "cache_removed",
|
||||
"checking repository status and unused repositories": "checking_repository_status_and_unused_repositories",
|
||||
"cleared lock for active job": "cleared_lock_for_active_job",
|
||||
"conference expire failed": "conference_expire_failed",
|
||||
"content expiration failed": "content_expiration_failed",
|
||||
"daily stats snapshot failed": "daily_stats_snapshot_failed",
|
||||
"daily stats snapshot stored": "daily_stats_snapshot_stored",
|
||||
"download completed": "download_completed",
|
||||
"download start": "download_start",
|
||||
"downloadFile failed": "downloadfile_failed",
|
||||
"downloaded": "downloaded",
|
||||
"downloading file": "downloading_file",
|
||||
"downloading gist": "downloading_gist",
|
||||
"downloading pull request": "downloading_pull_request",
|
||||
"downloading via raw URL (LFS)": "downloading_via_raw_url_lfs",
|
||||
"duplicate Stripe customer requires review": "duplicate_stripe_customer_requires_review",
|
||||
"ensureTodaySnapshot failed": "ensuretodaysnapshot_failed",
|
||||
"error log redis connect failed": "error_log_redis_connect_failed",
|
||||
"expired repository cleanup failed": "expired_repository_cleanup_failed",
|
||||
"failed removal recovery failed": "failed_removal_recovery_failed",
|
||||
"failed to persist ERROR status": "failed_to_persist_error_status",
|
||||
"failed to record repository removal error": "failed_to_record_repository_removal_error",
|
||||
"getToken": "gettoken",
|
||||
"github primary rate limit hit": "github_primary_rate_limit_hit",
|
||||
"github secondary rate limit hit": "github_secondary_rate_limit_hit",
|
||||
"grant revocation failed": "grant_revocation_failed",
|
||||
"initial repository status check failed": "initial_repository_status_check_failed",
|
||||
"job completed": "job_completed",
|
||||
"job stalled": "job_stalled",
|
||||
"job started": "job_started",
|
||||
"lastUsedAt update failed": "lastusedat_update_failed",
|
||||
"login-token failed": "login_token_failed",
|
||||
"login-token req.login failed": "login_token_req_login_failed",
|
||||
"logout after account removal failed": "logout_after_account_removal_failed",
|
||||
"logout failed": "logout_failed",
|
||||
"lookup failed": "lookup_failed",
|
||||
"markErrorIfInFlight failed": "markerrorifinflight_failed",
|
||||
"markErrorIfRemoving failed": "markerrorifremoving_failed",
|
||||
"metadata cleanup complete": "metadata_cleanup_complete",
|
||||
"metadata cleanup deferred": "metadata_cleanup_deferred",
|
||||
"owner access alert reset unavailable": "owner_access_alert_reset_unavailable",
|
||||
"owner access email accepted": "owner_access_email_accepted",
|
||||
"owner access email audit unavailable": "owner_access_email_audit_unavailable",
|
||||
"owner access email claim cleanup unavailable": "owner_access_email_claim_cleanup_unavailable",
|
||||
"owner access email deferred recovery unavailable": "owner_access_email_deferred_recovery_unavailable",
|
||||
"owner access email delivery failed": "owner_access_email_delivery_failed",
|
||||
"owner access email preparation cleanup unavailable": "owner_access_email_preparation_cleanup_unavailable",
|
||||
"owner access email recovery recorded": "owner_access_email_recovery_recorded",
|
||||
"owner access email recovery unavailable": "owner_access_email_recovery_unavailable",
|
||||
"owner access email result reconciliation required": "owner_access_email_result_reconciliation_required",
|
||||
"owner access email unavailable": "owner_access_email_unavailable",
|
||||
"process fatal error": "process_fatal_error",
|
||||
"process heartbeat": "process_heartbeat",
|
||||
"process started": "process_started",
|
||||
"progress": "progress",
|
||||
"queue heartbeat": "queue_heartbeat",
|
||||
"queued for download": "queued_for_download",
|
||||
"rate limit gate active, delaying job before work": "rate_limit_gate_active_delaying_job_before_work",
|
||||
"rate limit gate set": "rate_limit_gate_set",
|
||||
"rate-limited, delaying job": "rate_limited_delaying_job",
|
||||
"recover failed": "recover_failed",
|
||||
"recoverStuckPreparing failed": "recoverstuckpreparing_failed",
|
||||
"recoverStuckRemoving failed": "recoverstuckremoving_failed",
|
||||
"recovered expired repository cleanup": "recovered_expired_repository_cleanup",
|
||||
"recovered file from truncated tree": "recovered_file_from_truncated_tree",
|
||||
"recovered file invalidation failed": "recovered_file_invalidation_failed",
|
||||
"recovered stuck repo": "recovered_stuck_repo",
|
||||
"redis client error": "redis_client_error",
|
||||
"redis rate limit gate written": "redis_rate_limit_gate_written",
|
||||
"rejected upstream download destination": "rejected_upstream_download_destination",
|
||||
"removal failed": "removal_failed",
|
||||
"removal recovery failed": "removal_recovery_failed",
|
||||
"removeCache save failed": "removecache_save_failed",
|
||||
"removed cache for unused repository": "removed_cache_for_unused_repository",
|
||||
"removing cache": "removing_cache",
|
||||
"removing repository": "removing_repository",
|
||||
"repo claimed": "repo_claimed",
|
||||
"repository cache removal failed": "repository_cache_removal_failed",
|
||||
"repository expiration failed": "repository_expiration_failed",
|
||||
"repository expired": "repository_expired",
|
||||
"repository removal failed": "repository_removal_failed",
|
||||
"repository removed": "repository_removed",
|
||||
"requeued failed removal": "requeued_failed_removal",
|
||||
"requeued interrupted removal": "requeued_interrupted_removal",
|
||||
"reset": "reset",
|
||||
"retired tree cleanup deferred": "retired_tree_cleanup_deferred",
|
||||
"retired tree cleanup failed": "retired_tree_cleanup_failed",
|
||||
"running daily stats snapshot": "running_daily_stats_snapshot",
|
||||
"send failed": "send_failed",
|
||||
"server started": "server_started",
|
||||
"source stream failed; cache write discarded": "source_stream_failed_cache_write_discarded",
|
||||
"staged tree cleanup failed": "staged_tree_cleanup_failed",
|
||||
"staged tree completion deferred": "staged_tree_completion_deferred",
|
||||
"streamer fetch failed": "streamer_fetch_failed",
|
||||
"streamer started": "streamer_started",
|
||||
"streamer startup failed": "streamer_startup_failed",
|
||||
"streamer zip fetch failed": "streamer_zip_fetch_failed",
|
||||
"temporary storage cleanup deferred": "temporary_storage_cleanup_deferred",
|
||||
"token refresh failed; backing off": "token_refresh_failed_backing_off",
|
||||
"up to date": "up_to_date",
|
||||
"update queued": "update_queued",
|
||||
"verify failed": "verify_failed",
|
||||
"waiting for rate limit gate": "waiting_for_rate_limit_gate",
|
||||
"webhook processing failed": "webhook_processing_failed",
|
||||
"write failed": "write_failed",
|
||||
"zip stream failed": "zip_stream_failed"
|
||||
};
|
||||
+29
-30
@@ -1,3 +1,4 @@
|
||||
import { logRecord } from "./log-record";
|
||||
import { redactSecrets } from "./redact-secrets";
|
||||
import { createClient, RedisClientType } from "redis";
|
||||
import config from "../config";
|
||||
@@ -40,16 +41,6 @@ function resolveThreshold(): number {
|
||||
|
||||
const threshold = resolveThreshold();
|
||||
|
||||
function formatArg(a: unknown): string {
|
||||
if (typeof a === "string") return a;
|
||||
if (a instanceof Error) return JSON.stringify(serializeError(a));
|
||||
try {
|
||||
return JSON.stringify(a);
|
||||
} catch {
|
||||
return String(a);
|
||||
}
|
||||
}
|
||||
|
||||
let redisClient: RedisClientType | null = null;
|
||||
let redisDisabled = false;
|
||||
|
||||
@@ -58,6 +49,8 @@ function getRedis(): RedisClientType | null {
|
||||
if (redisClient) return redisClient;
|
||||
try {
|
||||
redisClient = createClient({
|
||||
disableOfflineQueue: true,
|
||||
commandsQueueMaxLength: 128,
|
||||
socket: {
|
||||
host: config.REDIS_HOSTNAME,
|
||||
port: config.REDIS_PORT,
|
||||
@@ -92,6 +85,13 @@ function getRedis(): RedisClientType | null {
|
||||
// Redis is back. Read by /admin/errors/stats so the admin page surfaces
|
||||
// "you're losing logs" instead of silently rendering an empty table.
|
||||
let droppedInProcess = 0;
|
||||
let droppedOutput = 0;
|
||||
let outputBroken = false;
|
||||
let errorOutputBroken = false;
|
||||
// A closed log reader must not become an uncaught EPIPE on the serving process.
|
||||
process.stdout.on("error", () => { outputBroken = true; droppedOutput++; });
|
||||
process.stderr.on("error", () => { errorOutputBroken = true; droppedOutput++; });
|
||||
export function getDroppedOutput() { return droppedOutput; }
|
||||
export function getInProcessDropped(): number {
|
||||
return droppedInProcess;
|
||||
}
|
||||
@@ -192,8 +192,9 @@ function persistError(entry: {
|
||||
message: string;
|
||||
raw: unknown[];
|
||||
}) {
|
||||
if (process.env.LOG_REDIS_MIRROR === "off") return;
|
||||
const client = getRedis();
|
||||
if (!client || !client.isOpen) {
|
||||
if (!client || !client.isReady) {
|
||||
droppedInProcess++;
|
||||
return;
|
||||
}
|
||||
@@ -204,12 +205,12 @@ function persistError(entry: {
|
||||
) as Record<string, unknown> | undefined;
|
||||
const bucket = bucketFor(detail, entry.level);
|
||||
const code =
|
||||
(detail && typeof detail.message === "string"
|
||||
? (detail.message as string)
|
||||
: "") ||
|
||||
(detail && typeof detail.code === "string"
|
||||
? (detail.code as string)
|
||||
: "") ||
|
||||
(detail && typeof detail.message === "string"
|
||||
? (detail.message as string)
|
||||
: "") ||
|
||||
"_";
|
||||
// Routine client misuse (401/403/404 and the allowlisted "expected" 4xx
|
||||
// codes below) drowns out real errors in the recent-entries ring. Keep the
|
||||
@@ -254,27 +255,25 @@ function emit(level: Level, module: string, args: unknown[]) {
|
||||
if (LEVEL_ORDER[level] < threshold) return;
|
||||
args = args.map(arg => redactSecrets(arg instanceof Error ? serializeError(arg) : arg));
|
||||
const ts = new Date().toISOString();
|
||||
const formatted = args.map(formatArg);
|
||||
const line = `${ts} ${level.toUpperCase()} [${module}] ${formatted.join(" ")}`;
|
||||
const sink =
|
||||
level === "error"
|
||||
? console.error
|
||||
: level === "warn"
|
||||
? console.warn
|
||||
: level === "debug"
|
||||
? console.debug
|
||||
: console.log;
|
||||
sink(line);
|
||||
const record = logRecord(level, module, args);
|
||||
// stdout is a pipe in Docker. Respect backpressure without awaiting it or
|
||||
// building an unbounded userland queue. The next heartbeat reports losses.
|
||||
const output = outputBroken ? process.stderr : process.stdout;
|
||||
if (!(outputBroken && errorOutputBroken) && output.writableLength < 256 * 1024) {
|
||||
try { output.write(JSON.stringify(record) + "\n"); }
|
||||
catch {
|
||||
if (output === process.stdout) outputBroken = true;
|
||||
else errorOutputBroken = true;
|
||||
droppedOutput++;
|
||||
}
|
||||
} else { droppedOutput++; }
|
||||
if (level === "error" || level === "warn") {
|
||||
persistError({
|
||||
ts,
|
||||
level,
|
||||
module,
|
||||
message: typeof args[0] === "string" ? args[0] : "",
|
||||
raw: args.map((a) => {
|
||||
if (a instanceof Error) return serializeError(a);
|
||||
return a;
|
||||
}),
|
||||
message: String(record.event),
|
||||
raw: [{ ...record, code: record.errorCode || record.event }],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { createLogger, serializeError } from "./logger";
|
||||
import { logMemory } from "./log-memory";
|
||||
import { createLogger, serializeError, getDroppedOutput } from "./logger";
|
||||
|
||||
/** Observe fatal errors without overriding Node's default exit behavior. */
|
||||
export function installFatalErrorLogging(service: "api" | "streamer") {
|
||||
@@ -6,6 +7,12 @@ export function installFatalErrorLogging(service: "api" | "streamer") {
|
||||
const report = (error: Error, origin: string) => logger.error("process fatal error", {
|
||||
...serializeError(error), code: "process_fatal_error", service, origin,
|
||||
});
|
||||
logger.info("process started", { service });
|
||||
const heartbeat = setInterval(() => {
|
||||
void logMemory().then(memory => logger.info("process heartbeat", {
|
||||
service, uptime: process.uptime(), ...memory, droppedLogs: getDroppedOutput() }));
|
||||
}, 30000);
|
||||
heartbeat.unref();
|
||||
process.on("uncaughtExceptionMonitor", report);
|
||||
return () => process.removeListener("uncaughtExceptionMonitor", report);
|
||||
return () => { clearInterval(heartbeat); process.removeListener("uncaughtExceptionMonitor", report); };
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { logContext } from "./log-context";
|
||||
import { AsyncLocalStorage } from "async_hooks";
|
||||
import { randomUUID } from "crypto";
|
||||
import { performance } from "perf_hooks";
|
||||
@@ -164,6 +165,6 @@ export function monitorRequests(service: Service, metrics = requestMetrics,
|
||||
else if (!route.includes("healthcheck")) log.info("request", detail);
|
||||
};
|
||||
res.once("finish", done); res.once("close", done);
|
||||
context.run(trace, next);
|
||||
context.run(trace, () => logContext.run({ requestId: id, service }, next));
|
||||
};
|
||||
}
|
||||
+20
-1
@@ -1,3 +1,4 @@
|
||||
import { jobTrace } from "../core/log-context";
|
||||
import { isQuotaFailure } from "../core/repository-failure-status";
|
||||
import UserModel from "../core/model/users/users.model";
|
||||
import { JobsOptions, Queue, Worker } from "bullmq";
|
||||
@@ -16,6 +17,8 @@ const logger = createLogger("queue");
|
||||
// ERR_BUFFER_OUT_OF_BOUNDS on long term lists / large nested fields.
|
||||
export interface RepoJobData {
|
||||
repoId: string;
|
||||
requestId?: string;
|
||||
logJobId?: string;
|
||||
}
|
||||
|
||||
const IN_FLIGHT_STATUSES: RepositoryStatus[] = [
|
||||
@@ -139,7 +142,7 @@ export async function addRemovalJob(
|
||||
if (LIVE_JOB_STATES.has(state)) return false;
|
||||
await existing.remove().catch(() => undefined);
|
||||
}
|
||||
await queue.add(repoId, { repoId }, { ...REMOVAL_JOB_OPTIONS, jobId });
|
||||
await queue.add(repoId, { repoId, ...jobTrace() }, { ...REMOVAL_JOB_OPTIONS, jobId });
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -302,6 +305,22 @@ export function startWorker() {
|
||||
autorun: true,
|
||||
}
|
||||
);
|
||||
for (const [queue, worker] of [["cache", cacheWorker], ["remove", removeWorker], ["download", downloadWorker]] as const) {
|
||||
worker.on("stalled", () => logger.error("job stalled", { queue }));
|
||||
}
|
||||
let sampling = false;
|
||||
const timer = setInterval(async () => {
|
||||
if (sampling) return;
|
||||
sampling = true;
|
||||
try {
|
||||
for (const [queue, client] of [["cache", cacheQueue], ["remove", removeQueue], ["download", downloadQueue]] as const) {
|
||||
const counts = await client.getJobCounts("waiting", "active", "delayed", "failed");
|
||||
const oldest = (await client.getJobs(["waiting"], 0, 0, true))[0];
|
||||
logger.info("queue heartbeat", { queue, ...counts, oldestWaitingMs: oldest ? Date.now() - oldest.timestamp : 0 });
|
||||
}
|
||||
} catch { /* queue availability is reported through missing heartbeat alerts */ }
|
||||
finally { sampling = false; }
|
||||
}, 30000); timer.unref();
|
||||
if (!downloadWorker.isRunning()) downloadWorker.run();
|
||||
|
||||
downloadWorker.on("active", async (job) => {
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
import { logMemory } from "../core/log-memory";
|
||||
import { randomUUID } from "crypto";
|
||||
import { SandboxedJob } from "bullmq";
|
||||
import { logContext, validTraceId } from "../core/log-context";
|
||||
import { createLogger, getDroppedOutput } from "../core/logger";
|
||||
import type { RepoJobData } from "./index";
|
||||
const logger = createLogger("queue");
|
||||
export function loggedJob(queue: "download" | "remove" | "cache", work: (job: SandboxedJob<RepoJobData, void>) => Promise<void>) {
|
||||
return (job: SandboxedJob<RepoJobData, void>) => logContext.run({
|
||||
service: "worker", jobId: validTraceId(job.data.logJobId) ? job.data.logJobId : randomUUID(),
|
||||
requestId: validTraceId(job.data.requestId) ? job.data.requestId : undefined,
|
||||
attempt: job.attemptsMade + 1,
|
||||
}, async () => {
|
||||
logger.info("job started", { queue });
|
||||
const started = Date.now();
|
||||
const heartbeat = setInterval(() => {
|
||||
void logMemory().then(memory => logger.info("process heartbeat", { queue, ...memory, uptime: process.uptime(), droppedLogs: getDroppedOutput() }));
|
||||
}, 30000); heartbeat.unref();
|
||||
try {
|
||||
await work(job);
|
||||
logger.info("job completed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() });
|
||||
} catch (error) {
|
||||
const expected = error instanceof Error && ["DelayedError", "RateLimitDelayError"].includes(error.name);
|
||||
logger[expected ? "info" : "error"]("job failed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() });
|
||||
throw error;
|
||||
} finally { clearInterval(heartbeat); }
|
||||
});
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { loggedJob } from "../log-job";
|
||||
import { githubQuotaKey } from "../../core/github-token-context";
|
||||
import { isQuotaFailure } from "../../core/repository-failure-status";
|
||||
import { SandboxedJob, UnrecoverableError } from "bullmq";
|
||||
@@ -13,7 +14,7 @@ import { DelayedError } from "bullmq";
|
||||
|
||||
const logger = createLogger("queue:download");
|
||||
|
||||
export default async function (job: SandboxedJob<RepoJobData, void>) {
|
||||
export default loggedJob("download", async function (job: SandboxedJob<RepoJobData, void>) {
|
||||
const {
|
||||
connect,
|
||||
getRepository,
|
||||
@@ -170,4 +171,4 @@ export default async function (job: SandboxedJob<RepoJobData, void>) {
|
||||
} finally {
|
||||
clearInterval(statusInterval);
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { loggedJob } from "../log-job";
|
||||
import { SandboxedJob } from "bullmq";
|
||||
import { getRepository as getRepositoryImport } from "../../server/database";
|
||||
import { RepoJobData } from "../index";
|
||||
@@ -42,4 +43,4 @@ export function createRemoveCacheProcessor(database?: Database) {
|
||||
) => processRemoveCache(job, database);
|
||||
}
|
||||
|
||||
export default createRemoveCacheProcessor();
|
||||
export default loggedJob("cache", createRemoveCacheProcessor());
|
||||
@@ -1,3 +1,4 @@
|
||||
import { loggedJob } from "../log-job";
|
||||
import { SandboxedJob } from "bullmq";
|
||||
import { getRepository as getRepositoryImport } from "../../server/database";
|
||||
import { RepositoryStatus } from "../../core/types";
|
||||
@@ -61,4 +62,4 @@ export function createRemoveRepositoryProcessor(database?: Database) {
|
||||
) => processRemoveRepository(job, database);
|
||||
}
|
||||
|
||||
export default createRemoveRepositoryProcessor();
|
||||
export default loggedJob("remove", createRemoveRepositoryProcessor());
|
||||
@@ -1,3 +1,4 @@
|
||||
import { jobTrace } from "../../core/log-context";
|
||||
import { cacheMetricsReport } from "../../core/cache-observability";
|
||||
import { sharedBlobCache, resultBlobCache } from "../../core/blob-cache";
|
||||
import { AsyncCache } from "../../core/async-cache";
|
||||
@@ -983,7 +984,7 @@ router.delete(
|
||||
const repo = await getRepo(req, res, { nocheck: true });
|
||||
if (!repo) return;
|
||||
try {
|
||||
await cacheQueue.add(repo.repoId, { repoId: repo.repoId }, { jobId: `repo-${repo.repoId}` });
|
||||
await cacheQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, { jobId: `repo-${repo.repoId}` });
|
||||
return res.json({ status: repo.status });
|
||||
} catch (error) {
|
||||
handleError(error, res, req);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { jobTrace } from "../../core/log-context";
|
||||
import Repository from "../../core/Repository";
|
||||
import { sourceStorageSize } from "../../core/source/Source";
|
||||
import { maximumExpiration, neverExpirationDeadline, validateBillingExpiration } from "../../core/billing/expiration";
|
||||
@@ -42,7 +43,7 @@ router.use(ensureAuthenticated);
|
||||
|
||||
async function enqueueSavedSnapshot(model: IAnonymizedRepositoryDocument) {
|
||||
try {
|
||||
await downloadQueue.add(model.repoId, { repoId: model.repoId }, {
|
||||
await downloadQueue.add(model.repoId, { repoId: model.repoId, ...jobTrace() }, {
|
||||
jobId: `repo-${model.repoId}`, attempts: 3,
|
||||
});
|
||||
} catch (error) {
|
||||
@@ -878,7 +879,7 @@ router.post("/", async (req, res) => {
|
||||
}
|
||||
|
||||
res.send({ status: repo.status });
|
||||
downloadQueue.add(repo.repoId, { repoId: repo.repoId }, {
|
||||
downloadQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, {
|
||||
jobId: `repo-${repo.repoId}`,
|
||||
attempts: 3,
|
||||
});
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { jobTrace } from "../../core/log-context";
|
||||
import { hasVisibleSubmodules } from "../../core/file-submodules";
|
||||
import { FILE_LISTING_LIMIT } from "../../core/file-tree";
|
||||
import { folderCounts as scopedFolderCounts } from "../../core/folder-counts";
|
||||
@@ -342,7 +343,7 @@ router.get(
|
||||
repo.statusActivityDate < fiveMinuteAgo
|
||||
) {
|
||||
await repo.updateStatus(RepositoryStatus.PREPARING);
|
||||
await downloadQueue.add(repo.repoId, { repoId: repo.repoId }, {
|
||||
await downloadQueue.add(repo.repoId, { repoId: repo.repoId, ...jobTrace() }, {
|
||||
jobId: `repo-${repo.repoId}`,
|
||||
attempts: 3,
|
||||
});
|
||||
|
||||
@@ -125,7 +125,8 @@ describe("removal recovery", function () {
|
||||
AnonymizedRepositoryModel.find = () => ({ lean: async () => [] });
|
||||
await queueModule.recoverStuckRemoving();
|
||||
expect(repo.status).to.equal("removing");
|
||||
expect(added[1]).to.deep.equal({ repoId: repo.repoId });
|
||||
expect(added[1]).to.deep.include({ repoId: repo.repoId });
|
||||
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -164,7 +165,8 @@ describe("removal recovery", function () {
|
||||
let added;
|
||||
queueModule.removeQueue.add = async (...args) => { added = args; };
|
||||
await queueModule.recoverStuckRemoving();
|
||||
expect(added[1]).to.deep.equal({ repoId: repo.repoId });
|
||||
expect(added[1]).to.deep.include({ repoId: repo.repoId });
|
||||
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -392,7 +394,8 @@ describe("removal workers", function () {
|
||||
expect(await addRemovalJob("repo-1", queue)).to.equal(true);
|
||||
expect(removed).to.equal(true);
|
||||
expect(added[0]).to.equal("repo-1");
|
||||
expect(added[1]).to.deep.equal({ repoId: "repo-1" });
|
||||
expect(added[1]).to.deep.include({ repoId: "repo-1" });
|
||||
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
|
||||
expect(added[2].jobId).to.equal("repo-repo-1");
|
||||
expect(added[2].attempts).to.equal(3);
|
||||
expect(added[2].removeOnFail).to.deep.equal({ count: 1000 });
|
||||
|
||||
@@ -19,15 +19,17 @@ describe("credential and webhook failure diagnostics", () => {
|
||||
afterEach(() => restores.splice(0).reverse().forEach(restore => restore()));
|
||||
it("logs generic upstream 401 as warning while retaining server errors", () => {
|
||||
const warnings = [], errors = [];
|
||||
stub(console, "warn", (...args) => warnings.push(args));
|
||||
stub(console, "error", (...args) => errors.push(args));
|
||||
stub(process.stdout, "write", line => {
|
||||
const record = JSON.parse(line);
|
||||
(record.severity === "warn" ? warnings : errors).push(line); return true;
|
||||
});
|
||||
handleError(Object.assign(new Error("denied"), { status: 401 }), response());
|
||||
handleError(Object.assign(new Error("failed"), { response: { statusCode: 503 } }), response());
|
||||
expect(warnings).to.have.length(1); expect(errors).to.have.length(1);
|
||||
expect(errors.flat().join(" ")).to.include('"httpStatus":503');
|
||||
});
|
||||
it("logs got HTTP401 as warning with its actual status", () => {
|
||||
const warnings = []; stub(console, "warn", (...args) => warnings.push(args));
|
||||
const warnings = []; stub(process.stdout, "write", line => { warnings.push(line); return true; });
|
||||
const failure = Object.assign(new Error("Unauthorized"), { response: { statusCode: 401 } });
|
||||
Object.setPrototypeOf(failure, require("got").HTTPError.prototype);
|
||||
const res = response(); handleError(failure, res);
|
||||
@@ -51,16 +53,16 @@ describe("credential and webhook failure diagnostics", () => {
|
||||
it("logs signed webhook persistence failures without secrets and still returns 503", async () => {
|
||||
stub(config, "GITHUB_APP_ENABLED", true); stub(config, "GITHUB_APP_ID", "123");
|
||||
stub(config, "GITHUB_APP_WEBHOOK_SECRET", "test-secret");
|
||||
const messages = []; stub(console, "error", (...args) => messages.push(args));
|
||||
const messages = []; stub(process.stdout, "write", line => { messages.push(line); return true; });
|
||||
stub(Installation, "updateOne", async () => { throw new Error("secret-token https://private/?token=secret"); });
|
||||
const body = Buffer.from(JSON.stringify({ action: "added", installation: { id: 4, app_id: 123 }, private: "private-body" }));
|
||||
const req = { body, headers: { "x-github-event": "installation_repositories", "x-github-delivery": "abcd-1234",
|
||||
const req = { body, headers: { "x-github-event": "installation_repositories", "x-github-delivery": "abcdef12-1234-1234-1234-abcdef123456",
|
||||
"x-hub-signature-256": "sha256=" + createHmac("sha256", "test-secret").update(body).digest("hex") } };
|
||||
const handler = githubAppWebhook.stack.find(layer => layer.route).route.stack.at(-1).handle;
|
||||
const res = response(); await handler(req, res);
|
||||
expect(res.code).to.equal(503); expect(res.body.error).to.equal("webhook_processing_failed");
|
||||
const text = JSON.stringify(messages);
|
||||
expect(text).to.include("persist_installation_block").and.include("abcd-1234");
|
||||
expect(text).to.include("persist_installation_block").and.include("abcdef12-1234-1234-1234-abcdef123456");
|
||||
expect(text).not.to.include("secret-token"); expect(text).not.to.include("private-body"); expect(text).not.to.include("https://");
|
||||
expect(text).not.to.include(req.headers["x-hub-signature-256"]);
|
||||
});
|
||||
|
||||
@@ -27,7 +27,7 @@ describe("download progress publication fences", function () {
|
||||
|
||||
it("keeps a captured publication fence valid when a queue progress tick runs", async function () {
|
||||
const { repo, stored } = setup(); let tick;
|
||||
stub(global, "setInterval", callback => { tick = callback; return 1; });
|
||||
stub(global, "setInterval", callback => { tick = callback; return { unref() {} }; });
|
||||
stub(global, "clearInterval", () => {});
|
||||
stub(db, "connect", async () => {});
|
||||
stub(db, "getRepository", async () => repo);
|
||||
|
||||
@@ -52,7 +52,7 @@ describe("folder file endpoint disclosure", function () {
|
||||
});
|
||||
it("blocks a real HTTP redirect to a GitHub tree page before fetching its HTML", async () => {
|
||||
const warnings = [];
|
||||
stub(console, "warn", (...args) => warnings.push(args.join(" ")));
|
||||
stub(process.stdout, "write", line => { warnings.push(line); return true; });
|
||||
const server = await listen(http.createServer((_req, res) => { res.writeHead(302, { location: "https://github.com/example/private-repo/tree/main/folder?token=secret-query" }); res.end(); }));
|
||||
const original = got.stream;
|
||||
stub(got, "stream", (_url, options) => original(`http://127.0.0.1:${server.address().port}/raw`, options));
|
||||
|
||||
@@ -84,7 +84,8 @@ describe("repository refresh and restoration", () => {
|
||||
expect(response.status).to.equal("preparing");
|
||||
expect(repo.model.source.commit).to.equal(commit);
|
||||
expect(repo.repoId).to.equal("restore-me");
|
||||
expect(added[1]).to.deep.equal({ repoId: "restore-me" });
|
||||
expect(added[1]).to.deep.include({ repoId: "restore-me" });
|
||||
expect(added[1].logJobId).to.match(/^[a-f0-9-]{36}$/);
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -9,17 +9,16 @@ require("../src/core/storage");
|
||||
const FileSystem = require("../src/core/storage/FileSystem").default;
|
||||
|
||||
describe("cache write error origins", function () {
|
||||
let directory, previousFolder, messages, previousWarn, previousError;
|
||||
let directory, previousFolder, messages, previousWrite;
|
||||
beforeEach(function () {
|
||||
directory = fs.mkdtempSync(path.join(os.tmpdir(), "cache-errors-"));
|
||||
previousFolder = config.FOLDER; config.FOLDER = directory;
|
||||
messages = [];
|
||||
previousWarn = console.warn; previousError = console.error;
|
||||
console.warn = (...args) => messages.push(["warn", args.join(" ")]);
|
||||
console.error = (...args) => messages.push(["error", args.join(" ")]);
|
||||
previousWrite = process.stdout.write;
|
||||
process.stdout.write = line => { const record = JSON.parse(line); messages.push([record.severity, record.event]); return true; };
|
||||
});
|
||||
afterEach(function () {
|
||||
console.warn = previousWarn; console.error = previousError;
|
||||
process.stdout.write = previousWrite;
|
||||
config.FOLDER = previousFolder;
|
||||
fs.rmSync(directory, { recursive: true, force: true });
|
||||
});
|
||||
@@ -34,8 +33,8 @@ describe("cache write error origins", function () {
|
||||
let caught;
|
||||
try { await storage.write("repo", "file.txt", input); } catch (error) { caught = error; }
|
||||
expect(caught).to.equal(failure);
|
||||
expect(messages.some(([level, text]) => level === "warn" && text.includes("source stream failed"))).to.equal(true);
|
||||
expect(messages.some(([level, text]) => level === "error" && text.includes("write failed"))).to.equal(false);
|
||||
expect(messages.some(([level, text]) => level === "warn" && text.includes("source_stream_failed"))).to.equal(true);
|
||||
expect(messages.some(([level, text]) => level === "error" && text.includes("write_failed"))).to.equal(false);
|
||||
expect(fs.readFileSync(path.join(directory, "repo/original/file.txt"), "utf8")).to.equal("previous");
|
||||
expect(fs.readdirSync(path.join(directory, "repo/original"))).to.deep.equal(["file.txt"]);
|
||||
});
|
||||
@@ -45,7 +44,7 @@ describe("cache write error origins", function () {
|
||||
let caught;
|
||||
try { await storage.write("repo", "file.txt", "new content"); } catch (error) { caught = error; }
|
||||
expect(caught).to.be.instanceOf(Error);
|
||||
expect(messages.some(([level, text]) => level === "error" && text.includes("write failed"))).to.equal(true);
|
||||
expect(messages.some(([level, text]) => level === "error" && text.includes("write_failed"))).to.equal(true);
|
||||
expect(messages.some(([level]) => level === "warn")).to.equal(false);
|
||||
expect(fs.readdirSync(path.join(directory, "repo/original"))).to.deep.equal(["file.txt"]);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
const { setTimeout } = require("node:timers");
|
||||
const { expect } = require('chai');
|
||||
const { randomUUID } = require('node:crypto');
|
||||
const { spawn, spawnSync } = require('node:child_process');
|
||||
require('ts-node/register/transpile-only');
|
||||
const { logRecord } = require('../src/core/log-record');
|
||||
const { logContext, jobTrace } = require('../src/core/log-context');
|
||||
const { loggedJob } = require('../src/queue/log-job');
|
||||
|
||||
describe('structured logging privacy and context', () => {
|
||||
it('keeps the production summary compatible with JSON and legacy records', () => {
|
||||
const { logSummary, observeLine } = require('../scripts/check-production-logs');
|
||||
const summary = logSummary();
|
||||
observeLine(summary, JSON.stringify(logRecord('info', 'requests', ['request', {ms: 1, status: 503, outcome: 'completed'}])));
|
||||
observeLine(summary, JSON.stringify(logRecord('error', 'process', ['process fatal error'])));
|
||||
observeLine(summary, '2026-01-01T00:00:00Z WARN [requests] request {"ms":2,"status":429,"outcome":"completed"}');
|
||||
expect(summary).to.include({completed: 2, completed5xx: 1, fatalErrors: 1, errors: 1, warnings: 1, rateLimited: 1});
|
||||
});
|
||||
it('omits arbitrary text, private identities, request bodies and nested secrets', () => {
|
||||
const secret = 'private-owner/private-repository?state=secret';
|
||||
const record = logRecord('error', 'upstream', [secret, {
|
||||
message: secret, url: secret, headers: { authorization: secret }, repoId: secret,
|
||||
requestBody: secret, status: 502, code: 'ETIMEDOUT', stages: { upstream: 42, [secret]: 1 },
|
||||
}]);
|
||||
expect(JSON.stringify(record)).not.to.include(secret);
|
||||
expect(record).to.include({ status: 502, errorCode: 'ETIMEDOUT', stage_upstream_ms: 42 });
|
||||
expect(record).to.have.keys('timestamp', 'severity', 'service', 'instance', 'release', 'event', 'message', 'module', 'status', 'errorCode', 'stage_upstream_ms');
|
||||
});
|
||||
it('keeps concurrent request contexts separate and propagates an opaque job trace', async () => {
|
||||
const ids = [randomUUID(), randomUUID()];
|
||||
const rows = await Promise.all(ids.map((requestId, i) => logContext.run({ requestId }, async () => {
|
||||
await new Promise(resolve => setTimeout(resolve, 10 - i));
|
||||
const trace = jobTrace();
|
||||
expect(trace.requestId).to.equal(requestId);
|
||||
expect(trace.logJobId).not.to.equal(requestId);
|
||||
return logRecord('info', 'fixture', ['request']);
|
||||
})));
|
||||
expect(rows.map(row => row.requestId)).to.deep.equal(ids);
|
||||
expect(logRecord('info', 'fixture', ['request'])).not.to.have.property('requestId');
|
||||
});
|
||||
it('sets sandbox job context and preserves success/failure semantics', async () => {
|
||||
const requestId = randomUUID(), jobId = randomUUID();
|
||||
const processor = loggedJob('download', async () => {
|
||||
expect(logContext.getStore()).to.deep.equal({ service: 'worker', requestId, jobId, attempt: 3 });
|
||||
throw new Error('fixture failure');
|
||||
});
|
||||
try { await processor({ data: { repoId: 'private-id', requestId, logJobId: jobId }, attemptsMade: 2 }); }
|
||||
catch (error) { expect(error.message).to.equal('fixture failure'); return; }
|
||||
throw new Error('job failure was swallowed');
|
||||
});
|
||||
it('rejects injected correlation IDs and non-finite measurements', () => {
|
||||
const row = logRecord('warn', 'fixture', ['request', { requestId: 'bad\nsecret', jobId: 'private', ms: Infinity, status: NaN }]);
|
||||
expect(row).not.to.have.any.keys('requestId', 'jobId', 'ms', 'status');
|
||||
});
|
||||
it('survives a reader closing stdout', async function () {
|
||||
this.timeout(10000);
|
||||
const child = spawn(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', `
|
||||
const { createLogger, getDroppedOutput } = require('./src/core/logger');
|
||||
const logger = createLogger('process');
|
||||
process.on('message', () => {
|
||||
logger.info('process heartbeat');
|
||||
setTimeout(() => { logger.info('process heartbeat'); process.send(getDroppedOutput()); process.disconnect(); }, 50);
|
||||
});
|
||||
process.send('ready');
|
||||
`], { env: { ...process.env, LOG_REDIS_MIRROR: 'off' }, stdio: ['ignore', 'pipe', 'pipe', 'ipc'] });
|
||||
let drops, fallback = "";
|
||||
child.stderr.on("data", chunk => { fallback += chunk; });
|
||||
child.on('message', message => {
|
||||
if (message === 'ready') { child.stdout.destroy(); child.send('write'); }
|
||||
else drops = message;
|
||||
});
|
||||
const code = await new Promise((resolve, reject) => { child.on('exit', resolve); child.on('error', reject); });
|
||||
expect(code).to.equal(0);
|
||||
expect(drops).to.be.greaterThan(0);
|
||||
expect(fallback).to.include('"event":"process_heartbeat"');
|
||||
});
|
||||
it('preserves admin error codes and expected-error ring suppression', () => {
|
||||
const child = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', `
|
||||
const redis = require('redis');
|
||||
let pushes = 0; const fields = [];
|
||||
const tx = {lPush() {pushes++; return tx;}, lTrim() {return tx;},
|
||||
hIncrBy(key, field) { fields.push(field); return tx; }, expire() {return tx;}, exec() {return Promise.resolve([]);} };
|
||||
redis.createClient = () => ({isReady:true, isOpen:true, on() {}, connect() {return Promise.resolve();}, multi() {return tx;}});
|
||||
require('./src/core/logger').createLogger('route').warn('repository_not_ready', {message:'repository_not_ready', httpStatus:409});
|
||||
if (pushes !== 0 || !fields.includes('cb:warn:repository_not_ready')) process.exitCode = 1;
|
||||
`], {env: {...process.env, LOG_REDIS_MIRROR: 'on'}, encoding: 'utf8', timeout: 10000});
|
||||
expect(child.status, child.stderr).to.equal(0);
|
||||
});
|
||||
it('writes JSON without Redis and bounds a blocked stdout pipe', () => {
|
||||
const child = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', `
|
||||
const { createLogger, getDroppedOutput } = require('./src/core/logger');
|
||||
const logger = createLogger('fixture');
|
||||
logger.info('request', {status: 200, requestBody: 'secret-fixture'});
|
||||
Object.defineProperty(process.stdout, 'writableLength', {value: 300000});
|
||||
for (let i = 0; i < 10000; i++) logger.warn('request', {status: 429});
|
||||
if (getDroppedOutput() !== 10000) process.exitCode = 1;
|
||||
`], { cwd: process.cwd(), env: { ...process.env, LOG_REDIS_MIRROR: 'off' }, encoding: 'utf8', timeout: 10000 });
|
||||
expect(child.status, child.stderr).to.equal(0);
|
||||
const lines = child.stdout.trim().split('\n');
|
||||
expect(lines).to.have.length(1);
|
||||
expect(JSON.parse(lines[0])).to.include({ event: 'request_completed', status: 200 });
|
||||
expect(child.stdout).not.to.include('secret-fixture');
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user