Merge pull request #972 from tdurieux/feat/operational-alerts-incidents

feat: add operational incident evidence and recovering alerts
This commit is contained in:
Thomas Durieux authored and GitHub committed 2026-11-15 16:28:02 +01:00
commit 128a7366b7
19 files changed
+616 -127

No files matched your search

+79
View File
@@ -412,3 +412,82 @@ zero cost. Unattributed time is visible; it is not evidence of a stall.
The admin API and Redis projection preserve these allowlisted fields. Collector
and application versions must be updated together. No ordinary request depends
on the collector, Grafana or VictoriaLogs being available.
## Alert investigation
Log rules carry a fixed investigation query, an action description and a relative
Grafana link. The links target `/d/anonymous-investigation` from dashboard PR #971,
with anchored service/event filters and a one-hour context window. Resolve relative
links against the configured private Grafana origin, not a public host. Admin
links preserve bounded epoch time limits and add anchored service/release filters;
configure `ADMIN_LOG_GRAFANA_URL` to that investigation dashboard to use them.
Metrics rules link to the operations dashboard and include the exact PromQL query.
No notification receiver is added or enabled.
`ApplicationLogDrops` uses `droppedLogsInterval`, the losses since the last stream-acknowledged
process report, so it recovers after losses stop. `droppedLogs` remains cumulative
for historical diagnosis. If emitting the report itself drops a record, the
watermark stays unchanged and the next report includes those losses. Concurrent
reports before acknowledgement can repeat a delta; it is an alert signal, not an
exact window loss total. A successful write does not prove collector ingestion. A dead
process or broken pair of stdout/stderr pipes still needs missing-heartbeat alerts.
Older applications without interval fields cannot drive the new interval rule;
roll out matching application and alert versions together. Collector/storage
loss counters retain their rate-based alerts and reset semantics.
Run `python3 scripts/test-operational-alerts.py` for isolated, pinned rule tests.
Real VictoriaLogs/vmalert tests accelerate only lookback/hold/evaluation durations,
keeping predicates and numeric thresholds unchanged. They prove firing and recovery
for 5xx, queue stalls/wait age, missing process/queue telemetry, memory, restart,
webhook and application loss alerts. The notifier is explicitly blackholed.
Promtool uses simulated time for every infrastructure rule, including disk reserve,
collector drops/errors, rejected rows and evaluation errors. These tests do not
prove production notification delivery, which remains disabled.
## Deployment evidence
`scripts/deploy.sh` emits finite JSON events for start, restart-window start and
terminal outcome. `deploymentId` correlates the run; `deploymentOperation=rollback`
is an explicit operator label via `DEPLOY_OPERATION=rollback`, never an inferred or
automatically executed rollback. `deploymentOutcome` is completed, failed or
unchanged. `APP_RELEASE` supplies a validated commit SHA; absent/invalid values
remain unknown. Elapsed and restart durations use second-resolution wall-clock
measurements expressed in milliseconds. The restart window begins before stopping
or removing old instances and ends at the terminal event. It is an orchestration
window, not measured user-visible downtime. SIGKILL/host loss can omit the terminal
event; a started event alone does not imply success. Failed API health checks now
produce a nonzero deployment result instead of a successful exit.
JSON always goes to the operator's stdout. Central collection is opt-in: an
operator may pre-create `/var/log/nginx/anonymous.deploy.json` owned by the deploy
operator, group `adm`, mode `0640`, then set `DEPLOY_EVENT_FILE` to that file when
running the deployment script. It must be an existing regular non-symlink file;
failure to append is reported locally and never blocks deployment. Vector reads
it through the existing read-only Nginx directory mount and supplemental group.
The supplied hourly rotation stanza retains six archives at a 1 MiB threshold.
Copytruncate preserves operator ownership but has a small concurrent-write loss
window. Do not claim this optional host file is enabled until configured and
verified. Configuration and VRL files remain 0644; no capability or secret mount
changes are made. Use the complete application Compose override list and mounted
GitHub App key described by the deployment runbook. This PR does not deploy.
## Incident counts and release comparison
Central groups retain first/last occurrence within the bounded selected snapshot.
`count` is records, `affectedRequests` is distinct captured request UUIDs within a
group, and `uncorrelatedRecords` counts records without a request UUID. Do not add
request counts across service groups or Redis mirrors. Fingerprints include safe
error code/type and the first recognized file/function, ignoring line/column shifts
between builds. Different functions remain distinct; unavailable stacks cannot
establish a precise source location.
When an exact release is selected, one extra bounded query compares with other
known releases in the preceding 24 hours. `known` means that fingerprint was
observed there; `not_seen_in_baseline` means it was absent from a nonempty complete
baseline, a candidate regression rather than proof. Missing, empty or truncated
baselines yield `unknown` for unseen signatures. The API returns comparison bounds
and coverage explicitly. Each query remains capped at 1,001 records/2 MiB and the
whole investigation shares its existing four-second deadline and admission limit.
Historical repositories with blank status messages are not reconstructed and are
not treated as new service failures. Finite lifecycle codes such as
`preparation_interrupted` and `commit_not_found` are retained for future evidence.
+114 -54
View File
@@ -1,55 +1,115 @@
groups:
- name: application-logs
type: vlogs
interval: 30s
rules:
- alert: SustainedServerErrors
expr: '_time:5m service:api outcome:completed | stats count() as total, count() if (status:>=500) as errors | filter errors:>=10 | math errors / total as ratio | filter ratio:>0.05 | fields ratio'
for: 5m
labels: {severity: warning}
annotations: {summary: 'API 5xx exceed 5 percent and 10 requests per five minutes'}
- alert: ApiTelemetryMissing
expr: '_time:3m service:api event:process_heartbeat | stats count() as heartbeats | filter heartbeats:0'
for: 3m
labels: {severity: warning}
annotations: {summary: 'API heartbeat is missing'}
- alert: StreamerTelemetryMissing
expr: '_time:3m service:streamer event:process_heartbeat | stats count_uniq(instance) as instances | filter instances:<4'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Fewer than four streamer instances report heartbeats; adjust for planned scaling'}
- alert: ApplicationRestartBurst
expr: '_time:10m event:process_started | stats by (service) count() as restarts | filter restarts:>4'
for: 2m
labels: {severity: warning}
annotations: {summary: 'Repeated process starts in ten minutes'}
- alert: ApplicationMemoryPressure
expr: '_time:3m event:process_heartbeat | stats by (service) max(memoryRatio) as ratio | filter ratio:>0.85'
for: 5m
labels: {severity: warning}
annotations: {summary: 'Container usage exceeds 85 percent of its cgroup memory limit'}
- alert: QueueStalled
expr: '_time:5m event:job_stalled | stats by (queue) count() as stalls | filter stalls:>0'
for: 2m
labels: {severity: warning}
annotations: {summary: 'BullMQ reports stalled jobs'}
- alert: QueueWaitingTooLong
expr: '_time:3m event:queue_heartbeat | stats by (queue) max(oldestWaitingMs) as age | filter age:>600000'
for: 5m
labels: {severity: warning}
annotations: {summary: 'A runnable job has waited over ten minutes'}
- alert: QueueTelemetryMissing
expr: '_time:3m event:queue_heartbeat | stats count_uniq(queue) as queues | filter queues:<3'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Queue telemetry is missing'}
- alert: WebhookFailures
expr: '_time:5m route:"/github/app/webhook" status:>=500 outcome:completed | stats count() as errors | filter errors:>=3'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Repeated webhook server failures; signature rejection is excluded'}
- alert: ApplicationLogDrops
expr: '_time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service) max(droppedLogs) as drops | filter drops:>0'
for: 2m
labels: {severity: warning}
annotations: {summary: 'An application process dropped stdout logs since startup'}
- name: application-logs
type: vlogs
interval: 30s
rules:
- alert: SustainedServerErrors
expr: _time:5m service:api outcome:completed | stats count() as total, count() if (status:>=500) as errors | filter errors:>=10 | math errors / total as ratio | filter ratio:>0.05 | fields ratio
for: 5m
labels:
severity: warning
annotations:
summary: API 5xx exceed 5 percent and 10 requests per five minutes
description: Compare completed API 5xx by release and route; follow a request UUID to streamer and edge evidence.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Eapi%24&var-event=.%2A
investigation_query: _time:5m service:api outcome:completed
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApiTelemetryMissing
expr: _time:3m service:api event:process_heartbeat | stats count() as heartbeats | filter heartbeats:0
for: 3m
labels:
severity: warning
annotations:
summary: API heartbeat is missing
description: Check API container health and collector ingestion. Missing logs alone do not establish an API outage.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Eapi%24&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m service:api event:process_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: StreamerTelemetryMissing
expr: _time:3m service:streamer event:process_heartbeat | stats count_uniq(instance) as instances | filter instances:<4
for: 3m
labels:
severity: warning
annotations:
summary: Fewer than four streamer instances report heartbeats; adjust for planned scaling
description: Check expected replica count and recent deployment window, then container health and collector ingestion.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Estreamer%24&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m service:streamer event:process_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApplicationRestartBurst
expr: _time:10m event:process_started | stats by (service) count() as restarts | filter restarts:>4
for: 2m
labels:
severity: warning
annotations:
summary: Repeated process starts in ten minutes
description: Compare process starts with deployment events; inspect fatal errors and container exit reasons.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Eprocess_started%24
investigation_query: _time:10m event:process_started
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApplicationMemoryPressure
expr: _time:3m event:process_heartbeat | stats by (service) max(memoryRatio) as ratio | filter ratio:>0.85
for: 5m
labels:
severity: warning
annotations:
summary: Container usage exceeds 85 percent of its cgroup memory limit
description: Inspect cgroup usage and release, including cache and worker memory; RSS alone excludes page cache.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m event:process_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: QueueStalled
expr: _time:5m event:job_stalled | stats by (queue) count() as stalls | filter stalls:>0
for: 2m
labels:
severity: warning
annotations:
summary: BullMQ reports stalled jobs
description: Follow job UUID and attempt; check worker exits and Redis lock renewal before retrying.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Ejob_stalled%24
investigation_query: _time:5m event:job_stalled
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: QueueWaitingTooLong
expr: _time:3m event:queue_heartbeat | stats by (queue) max(oldestWaitingMs) as age | filter age:>600000
for: 5m
labels:
severity: warning
annotations:
summary: A runnable job has waited over ten minutes
description: Compare runnable waiting age with worker heartbeats, concurrency and failures; delayed jobs are excluded.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Equeue_heartbeat%24
investigation_query: _time:3m event:queue_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: QueueTelemetryMissing
expr: _time:3m event:queue_heartbeat | stats count_uniq(queue) as queues | filter queues:<3
for: 3m
labels:
severity: warning
annotations:
summary: Queue telemetry is missing
description: Check API queue sampler and Redis connectivity; missing telemetry is not proof that queues are empty.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Equeue_heartbeat%24
investigation_query: _time:3m event:queue_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: WebhookFailures
expr: _time:5m route:"/github/app/webhook" status:>=500 outcome:completed | stats count() as errors | filter errors:>=3
for: 3m
labels:
severity: warning
annotations:
summary: Repeated webhook server failures; signature rejection is excluded
description: Inspect webhook route 5xx and safe error code; signature rejection is excluded.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Eapi%24&var-event=.%2A
investigation_query: _time:5m route:"/github/app/webhook" status:>=500 outcome:completed
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApplicationLogDrops
expr: _time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service) sum(droppedLogsInterval) as drops | filter drops:>0
for: 2m
labels:
severity: warning
annotations:
summary: Application log output dropped records in the last three minutes
description: Inspect interval loss and stdout backpressure; cumulative droppedLogs is historical, not a current rate.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed)
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
+62 -32
View File
@@ -1,33 +1,63 @@
groups:
- name: logging-infrastructure
rules:
- alert: LoggingTelemetryMissing
expr: up == 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'A logging component cannot be scraped'}
- alert: CollectorDrops
expr: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
for: 2m
labels: {severity: warning}
annotations: {summary: 'Collector dropped events; inspect buffer and transform counters'}
- alert: CollectorErrors
expr: sum(increase(vector_component_errors_total[5m])) > 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'Collector errors persist'}
- alert: LogStoragePressure
expr: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
for: 5m
labels: {severity: warning}
annotations: {summary: 'Log storage is near the retention budget or free-space reserve'}
- alert: LogIngestionRejected
expr: sum(increase(vl_rows_dropped_total[5m])) > 0
for: 2m
labels: {severity: warning}
annotations: {summary: 'VictoriaLogs rejected records'}
- alert: LogAlertEvaluationFailed
expr: sum(increase(vmalert_execution_errors_total[5m])) > 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'Log alert queries are failing'}
- name: logging-infrastructure
rules:
- alert: LoggingTelemetryMissing
expr: up == 0
for: 3m
labels:
severity: warning
annotations:
summary: A logging component cannot be scraped
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: up == 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: CollectorDrops
expr: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
for: 2m
labels:
severity: warning
annotations:
summary: Collector dropped events; inspect buffer and transform counters
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: CollectorErrors
expr: sum(increase(vector_component_errors_total[5m])) > 0
for: 3m
labels:
severity: warning
annotations:
summary: Collector errors persist
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vector_component_errors_total[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: LogStoragePressure
expr: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
for: 5m
labels:
severity: warning
annotations:
summary: Log storage is near the retention budget or free-space reserve
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: LogIngestionRejected
expr: sum(increase(vl_rows_dropped_total[5m])) > 0
for: 2m
labels:
severity: warning
annotations:
summary: VictoriaLogs rejected records
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vl_rows_dropped_total[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: LogAlertEvaluationFailed
expr: sum(increase(vmalert_execution_errors_total[5m])) > 0
for: 3m
labels:
severity: warning
annotations:
summary: Log alert queries are failing
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vmalert_execution_errors_total[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
+13
View File
@@ -13,3 +13,16 @@
/usr/sbin/nginx -s reopen
endscript
}
# Optional deployment event file. Pre-create it for the deploy operator with
# group adm and mode 0640; copytruncate preserves that explicit ownership.
/var/log/nginx/anonymous.deploy.json {
hourly
rotate 6
maxsize 1M
missingok
notifempty
compress
delaycompress
copytruncate
}
+9 -5
View File
@@ -10,6 +10,7 @@ if starts_with(string!(.file), "/host/docker/") {
record, err = parse_json(string(raw.log) ?? "")
if err != null { record = {} }
}
if starts_with(string!(.file), "/host/nginx/anonymous.deploy.json") { service = "api" }
out = {"timestamp": now(), "service": service, "severity": "info", "event": "service_output"}
# Use capture time for unsupported formats. Invalid/old producer timestamps
# cannot defeat retention or crash the collector.
@@ -36,10 +37,10 @@ if service == "mongodb" {
} else if includes(["api", "streamer"], service) {
if record.service == "worker" { out.service = "worker" }
if includes(["debug", "info", "warn", "error"], record.severity) { out.severity = record.severity }
if includes(["account_removed", "application_log", "billing_maintenance_failed", "billing_reconciliation_deferred", "billing_requires_attention", "billing_worker_failed", "blob_api_failed_falling_back_to_raw_url", "branch_not_found", "cache_removal_failed", "cache_removed", "checking_repository_status_and_unused_repositories", "cleared_lock_for_active_job", "conference_expire_failed", "content_expiration_failed", "daily_stats_snapshot_failed", "daily_stats_snapshot_stored", "download_completed", "download_start", "downloaded", "downloadfile_failed", "downloading_file", "downloading_gist", "downloading_pull_request", "downloading_via_raw_url_lfs", "duplicate_stripe_customer_requires_review", "ensuretodaysnapshot_failed", "error_log_redis_connect_failed", "expired_repository_cleanup_failed", "failed_removal_recovery_failed", "failed_to_persist_error_status", "failed_to_record_repository_removal_error", "gettoken", "github_primary_rate_limit_hit", "github_secondary_rate_limit_hit", "grant_revocation_failed", "initial_repository_status_check_failed", "job_completed", "job_failed", "job_stalled", "job_started", "lastusedat_update_failed", "login_token_failed", "login_token_req_login_failed", "logout_after_account_removal_failed", "logout_failed", "lookup_failed", "markerrorifinflight_failed", "markerrorifremoving_failed", "metadata_cleanup_complete", "metadata_cleanup_deferred", "owner_access_alert_reset_unavailable", "owner_access_email_accepted", "owner_access_email_audit_unavailable", "owner_access_email_claim_cleanup_unavailable", "owner_access_email_deferred_recovery_unavailable", "owner_access_email_delivery_failed", "owner_access_email_preparation_cleanup_unavailable", "owner_access_email_recovery_recorded", "owner_access_email_recovery_unavailable", "owner_access_email_result_reconciliation_required", "owner_access_email_unavailable", "pr_comments_404_continuing_without_them", "process_fatal_error", "process_heartbeat", "process_started", "progress", "queue_heartbeat", "queued_for_download", "rate_limit_gate_active_delaying_job_before_work", "rate_limit_gate_set", "rate_limited_delaying_job", "recover_failed", "recovered_expired_repository_cleanup", "recovered_file_from_truncated_tree", "recovered_file_invalidation_failed", "recovered_stuck_repo", "recoverstuckpreparing_failed", "recoverstuckremoving_failed", "redis_client_error", "redis_rate_limit_gate_written", "rejected_upstream_download_destination", "removal_failed", "removal_recovery_failed", "removecache_save_failed", "removed_cache_for_unused_repository", "removing_cache", "removing_repository", "repo_claimed", "repository_cache_removal_failed", "repository_expiration_failed", "repository_expired", "repository_removal_failed", "repository_removed", "request_cancelled", "request_completed", "request_slow", "requeued_failed_removal", "requeued_interrupted_removal", "reset", "retired_tree_cleanup_deferred", "retired_tree_cleanup_failed", "running_daily_stats_snapshot", "s3_body_stream_error", "send_failed", "server_started", "source_stream_failed_cache_write_discarded", "staged_tree_cleanup_failed", "staged_tree_completion_deferred", "streamer_fetch_failed", "streamer_started", "streamer_startup_failed", "streamer_zip_fetch_failed", "temporary_storage_cleanup_deferred", "token_refresh_failed_backing_off", "up_to_date", "update_queued", "verify_failed", "waiting_for_rate_limit_gate", "webhook_log", "webhook_processing_failed", "write_failed", "zip_stream_failed"], record.event) { out.event = record.event }
if includes(["deployment", "account_removed", "application_log", "billing_maintenance_failed", "billing_reconciliation_deferred", "billing_requires_attention", "billing_worker_failed", "blob_api_failed_falling_back_to_raw_url", "branch_not_found", "cache_removal_failed", "cache_removed", "checking_repository_status_and_unused_repositories", "cleared_lock_for_active_job", "conference_expire_failed", "content_expiration_failed", "daily_stats_snapshot_failed", "daily_stats_snapshot_stored", "download_completed", "download_start", "downloaded", "downloadfile_failed", "downloading_file", "downloading_gist", "downloading_pull_request", "downloading_via_raw_url_lfs", "duplicate_stripe_customer_requires_review", "ensuretodaysnapshot_failed", "error_log_redis_connect_failed", "expired_repository_cleanup_failed", "failed_removal_recovery_failed", "failed_to_persist_error_status", "failed_to_record_repository_removal_error", "gettoken", "github_primary_rate_limit_hit", "github_secondary_rate_limit_hit", "grant_revocation_failed", "initial_repository_status_check_failed", "job_completed", "job_failed", "job_stalled", "job_started", "lastusedat_update_failed", "login_token_failed", "login_token_req_login_failed", "logout_after_account_removal_failed", "logout_failed", "lookup_failed", "markerrorifinflight_failed", "markerrorifremoving_failed", "metadata_cleanup_complete", "metadata_cleanup_deferred", "owner_access_alert_reset_unavailable", "owner_access_email_accepted", "owner_access_email_audit_unavailable", "owner_access_email_claim_cleanup_unavailable", "owner_access_email_deferred_recovery_unavailable", "owner_access_email_delivery_failed", "owner_access_email_preparation_cleanup_unavailable", "owner_access_email_recovery_recorded", "owner_access_email_recovery_unavailable", "owner_access_email_result_reconciliation_required", "owner_access_email_unavailable", "pr_comments_404_continuing_without_them", "process_fatal_error", "process_heartbeat", "process_started", "progress", "queue_heartbeat", "queued_for_download", "rate_limit_gate_active_delaying_job_before_work", "rate_limit_gate_set", "rate_limited_delaying_job", "recover_failed", "recovered_expired_repository_cleanup", "recovered_file_from_truncated_tree", "recovered_file_invalidation_failed", "recovered_stuck_repo", "recoverstuckpreparing_failed", "recoverstuckremoving_failed", "redis_client_error", "redis_rate_limit_gate_written", "rejected_upstream_download_destination", "removal_failed", "removal_recovery_failed", "removecache_save_failed", "removed_cache_for_unused_repository", "removing_cache", "removing_repository", "repo_claimed", "repository_cache_removal_failed", "repository_expiration_failed", "repository_expired", "repository_removal_failed", "repository_removed", "request_cancelled", "request_completed", "request_slow", "requeued_failed_removal", "requeued_interrupted_removal", "reset", "retired_tree_cleanup_deferred", "retired_tree_cleanup_failed", "running_daily_stats_snapshot", "s3_body_stream_error", "send_failed", "server_started", "source_stream_failed_cache_write_discarded", "staged_tree_cleanup_failed", "staged_tree_completion_deferred", "streamer_fetch_failed", "streamer_started", "streamer_startup_failed", "streamer_zip_fetch_failed", "temporary_storage_cleanup_deferred", "token_refresh_failed_backing_off", "up_to_date", "update_queued", "verify_failed", "waiting_for_rate_limit_gate", "webhook_log", "webhook_processing_failed", "write_failed", "zip_stream_failed"], record.event) { out.event = record.event }
if is_string(record.release) && match(string!(record.release), r'^[a-f0-9]{7,40}$') { out.release = record.release }
if is_string(record.instance) && match(string!(record.instance), r'^[a-f0-9-]{36}$') { out.instance = record.instance }
if includes(["ABORT_ERR", "ECONNREFUSED", "ECONNRESET", "ENOTFOUND", "EPIPE", "ETIMEDOUT", "billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_app_reconnect_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_token_invalid", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_signature", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "token_refresh_failed", "quota_exceeded", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_found", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "upstream_timeout", "user_not_found", "username_not_defined", "webhook_processing_failed"], record.errorCode) { out.errorCode = record.errorCode }
if includes(["repo_renamed", "preparation_interrupted", "commit_not_found", "zip_not_available", "ABORT_ERR", "ECONNREFUSED", "ECONNRESET", "ENOTFOUND", "EPIPE", "ETIMEDOUT", "billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_app_reconnect_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_token_invalid", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_signature", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "token_refresh_failed", "quota_exceeded", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_found", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "upstream_timeout", "user_not_found", "username_not_defined", "webhook_processing_failed"], record.errorCode) { out.errorCode = record.errorCode }
if includes(["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"], record.module) { out.module = record.module }
if includes(["protocol", "credentials", "port", "host"], record.reason) { out.reason = record.reason }
if includes(["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"], record.phase) { out.phase = record.phase }
@@ -48,14 +49,17 @@ if service == "mongodb" {
if includes(["download", "remove", "cache"], record.queue) { out.queue = record.queue }
if includes(["unknown", "api", "streamer", "timeout", "upstream"], record.cancellationOrigin) { out.cancellationOrigin = record.cancellationOrigin }
if includes(["response_close", "request_aborted", "upstream_error", "timeout", "local_abort"], record.cancellationEvidence) { out.cancellationEvidence = record.cancellationEvidence }
if includes(["deploy", "rollback"], record.deploymentOperation) { out.deploymentOperation = record.deploymentOperation }
if includes(["start", "restart", "finish"], record.deploymentPhase) { out.deploymentPhase = record.deploymentPhase }
if includes(["started", "completed", "failed", "unchanged"], record.deploymentOutcome) { out.deploymentOutcome = record.deploymentOutcome }
if includes(["public", "internal"], record.endpointKind) { out.endpointKind = record.endpointKind }
if includes(["completed", "interrupted"], record.outcome) { out.outcome = record.outcome }
}
for_each(["requestId", "jobId", "deliveryId", "hopId", "parentHopId"]) -> |_index, key| {
for_each(["requestId", "jobId", "deliveryId", "hopId", "parentHopId", "deploymentId"]) -> |_index, key| {
value = get(record, [key]) ?? null
if is_string(value) && match(string!(value), r'^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$') { out = set!(out, [key], value) }
}
for_each(["stage_transfer_ms", "stage_transfer_count", "stage_transfer_open", "stage_database_ms", "stage_database_count", "stage_database_open", "expectedBytes", "measuredMs", "unattributedMs", "overlapMs", "stage_repository_count", "stage_repository_open", "stage_credential_resolution_count", "stage_credential_resolution_open", "stage_upstream_count", "stage_upstream_open", "stage_cache_lookup_count", "stage_cache_lookup_open", "stage_cache_fill_count", "stage_cache_fill_open", "stage_worker_wait_count", "stage_worker_wait_open", "stage_anonymize_count", "stage_anonymize_open", "stage_streamer_headers_count", "stage_streamer_headers_open", "status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus", "stage_repository_ms", "stage_credential_resolution_ms", "stage_upstream_ms", "stage_cache_lookup_ms", "stage_cache_fill_ms", "stage_worker_wait_ms", "stage_anonymize_ms", "stage_streamer_headers_ms"]) -> |_index, key| {
for_each(["restartMs", "exitCode", "droppedLogsInterval", "stage_transfer_ms", "stage_transfer_count", "stage_transfer_open", "stage_database_ms", "stage_database_count", "stage_database_open", "expectedBytes", "measuredMs", "unattributedMs", "overlapMs", "stage_repository_count", "stage_repository_open", "stage_credential_resolution_count", "stage_credential_resolution_open", "stage_upstream_count", "stage_upstream_open", "stage_cache_lookup_count", "stage_cache_lookup_open", "stage_cache_fill_count", "stage_cache_fill_open", "stage_worker_wait_count", "stage_worker_wait_open", "stage_anonymize_count", "stage_anonymize_open", "stage_streamer_headers_count", "stage_streamer_headers_open", "status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus", "stage_repository_ms", "stage_credential_resolution_ms", "stage_upstream_ms", "stage_cache_lookup_ms", "stage_cache_fill_ms", "stage_worker_wait_ms", "stage_anonymize_ms", "stage_streamer_headers_ms"]) -> |_index, key| {
value = get(record, [key]) ?? null
if is_integer(value) || is_float(value) { out = set!(out, [key], value) }
}
@@ -75,7 +79,7 @@ if includes(["/", "other", "/dashboard", "/anonymize", "/healthcheck", "/api/hea
}
}
# BEGIN GENERATED DIAGNOSTICS
source_functions = {"src/cli/index.ts":["<anonymous>","generateRandomFileName","main"],"build/cli/index.js":["<anonymous>","generateRandomFileName","main"],"src/config.ts":["<anonymous>"],"build/config.js":["<anonymous>"],"src/core/AnonymizedFile.ts":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"build/core/AnonymizedFile.js":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"src/core/AnonymousError.ts":["<anonymous>","detail","toString","url"],"build/core/AnonymousError.js":["<anonymous>","detail","toString","url"],"src/core/Conference.ts":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"build/core/Conference.js":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"src/core/Gist.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/Gist.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/GitHubUtils.ts":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"build/core/GitHubUtils.js":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"src/core/PullRequest.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/PullRequest.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/Repository.ts":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"build/core/Repository.js":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"src/core/User.ts":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"build/core/User.js":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"src/core/anonymization-pool.ts":["<anonymous>","anonymizeOnWorker","cancel","exited","failed","finish","getAnonymizationPoolStats","message","pump","release","releaseSpool","reserveSpool"],"build/core/anonymization-pool.js":["Line truncated
source_functions = {"src/cli/index.ts":["<anonymous>","generateRandomFileName","main"],"build/cli/index.js":["<anonymous>","generateRandomFileName","main"],"src/config.ts":["<anonymous>"],"build/config.js":["<anonymous>"],"src/core/AnonymizedFile.ts":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"build/core/AnonymizedFile.js":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"src/core/AnonymousError.ts":["<anonymous>","detail","toString","url"],"build/core/AnonymousError.js":["<anonymous>","detail","toString","url"],"src/core/Conference.ts":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"build/core/Conference.js":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"src/core/Gist.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/Gist.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/GitHubUtils.ts":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"build/core/GitHubUtils.js":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"src/core/PullRequest.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/PullRequest.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/Repository.ts":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"build/core/Repository.js":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"src/core/User.ts":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"build/core/User.js":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"src/core/anonymization-pool.ts":["<anonymous>","anonymizeOnWorker","cancel","exited","failed","finish","getAnonymizationPoolStats","message","pump","release","releaseSpool","reserveSpool"],"build/core/anonymization-pool.js":["Line truncated
for_each(["errorType", "causeType"]) -> |_index, key| {
value = get(record, [key]) ?? null
if includes(["Error","TypeError","RangeError","ReferenceError","SyntaxError","URIError","EvalError","AggregateError","AnonymousError","HTTPError","RequestError","MongoServerError","MongoNetworkError","MongoServerSelectionError","ValidationError","TimeoutError","AbortError","UnknownError"], value) { out = set!(out, [key], value) }
+8 -1
View File
@@ -15,13 +15,20 @@ sources:
read_from: end
max_line_bytes: 65536
fingerprint: {strategy: device_and_inode}
deployments:
type: file
include: [/host/nginx/anonymous.deploy.json*]
exclude: ['*.gz']
read_from: end
max_line_bytes: 4096
fingerprint: {strategy: device_and_inode}
internal:
type: internal_metrics
scrape_interval_secs: 15
transforms:
safe:
type: remap
inputs: [docker, nginx]
inputs: [docker, nginx, deployments]
file: /etc/vector/sanitize.vrl
drop_on_error: true
drop_on_abort: true