Merge pull request #972 from tdurieux/feat/operational-alerts-incidents

feat: add operational incident evidence and recovering alerts
This commit is contained in:
Thomas Durieux authored and GitHub committed 2026-11-15 16:28:02 +01:00
commit 128a7366b7
19 files changed
+616 -127

No files matched your search

+79
View File
@@ -412,3 +412,82 @@ zero cost. Unattributed time is visible; it is not evidence of a stall.
The admin API and Redis projection preserve these allowlisted fields. Collector
and application versions must be updated together. No ordinary request depends
on the collector, Grafana or VictoriaLogs being available.
## Alert investigation
Log rules carry a fixed investigation query, an action description and a relative
Grafana link. The links target `/d/anonymous-investigation` from dashboard PR #971,
with anchored service/event filters and a one-hour context window. Resolve relative
links against the configured private Grafana origin, not a public host. Admin
links preserve bounded epoch time limits and add anchored service/release filters;
configure `ADMIN_LOG_GRAFANA_URL` to that investigation dashboard to use them.
Metrics rules link to the operations dashboard and include the exact PromQL query.
No notification receiver is added or enabled.
`ApplicationLogDrops` uses `droppedLogsInterval`, the losses since the last stream-acknowledged
process report, so it recovers after losses stop. `droppedLogs` remains cumulative
for historical diagnosis. If emitting the report itself drops a record, the
watermark stays unchanged and the next report includes those losses. Concurrent
reports before acknowledgement can repeat a delta; it is an alert signal, not an
exact window loss total. A successful write does not prove collector ingestion. A dead
process or broken pair of stdout/stderr pipes still needs missing-heartbeat alerts.
Older applications without interval fields cannot drive the new interval rule;
roll out matching application and alert versions together. Collector/storage
loss counters retain their rate-based alerts and reset semantics.
Run `python3 scripts/test-operational-alerts.py` for isolated, pinned rule tests.
Real VictoriaLogs/vmalert tests accelerate only lookback/hold/evaluation durations,
keeping predicates and numeric thresholds unchanged. They prove firing and recovery
for 5xx, queue stalls/wait age, missing process/queue telemetry, memory, restart,
webhook and application loss alerts. The notifier is explicitly blackholed.
Promtool uses simulated time for every infrastructure rule, including disk reserve,
collector drops/errors, rejected rows and evaluation errors. These tests do not
prove production notification delivery, which remains disabled.
## Deployment evidence
`scripts/deploy.sh` emits finite JSON events for start, restart-window start and
terminal outcome. `deploymentId` correlates the run; `deploymentOperation=rollback`
is an explicit operator label via `DEPLOY_OPERATION=rollback`, never an inferred or
automatically executed rollback. `deploymentOutcome` is completed, failed or
unchanged. `APP_RELEASE` supplies a validated commit SHA; absent/invalid values
remain unknown. Elapsed and restart durations use second-resolution wall-clock
measurements expressed in milliseconds. The restart window begins before stopping
or removing old instances and ends at the terminal event. It is an orchestration
window, not measured user-visible downtime. SIGKILL/host loss can omit the terminal
event; a started event alone does not imply success. Failed API health checks now
produce a nonzero deployment result instead of a successful exit.
JSON always goes to the operator's stdout. Central collection is opt-in: an
operator may pre-create `/var/log/nginx/anonymous.deploy.json` owned by the deploy
operator, group `adm`, mode `0640`, then set `DEPLOY_EVENT_FILE` to that file when
running the deployment script. It must be an existing regular non-symlink file;
failure to append is reported locally and never blocks deployment. Vector reads
it through the existing read-only Nginx directory mount and supplemental group.
The supplied hourly rotation stanza retains six archives at a 1 MiB threshold.
Copytruncate preserves operator ownership but has a small concurrent-write loss
window. Do not claim this optional host file is enabled until configured and
verified. Configuration and VRL files remain 0644; no capability or secret mount
changes are made. Use the complete application Compose override list and mounted
GitHub App key described by the deployment runbook. This PR does not deploy.
## Incident counts and release comparison
Central groups retain first/last occurrence within the bounded selected snapshot.
`count` is records, `affectedRequests` is distinct captured request UUIDs within a
group, and `uncorrelatedRecords` counts records without a request UUID. Do not add
request counts across service groups or Redis mirrors. Fingerprints include safe
error code/type and the first recognized file/function, ignoring line/column shifts
between builds. Different functions remain distinct; unavailable stacks cannot
establish a precise source location.
When an exact release is selected, one extra bounded query compares with other
known releases in the preceding 24 hours. `known` means that fingerprint was
observed there; `not_seen_in_baseline` means it was absent from a nonempty complete
baseline, a candidate regression rather than proof. Missing, empty or truncated
baselines yield `unknown` for unseen signatures. The API returns comparison bounds
and coverage explicitly. Each query remains capped at 1,001 records/2 MiB and the
whole investigation shares its existing four-second deadline and admission limit.
Historical repositories with blank status messages are not reconstructed and are
not treated as new service failures. Finite lifecycle codes such as
`preparation_interrupted` and `commit_not_found` are retained for future evidence.
+114 -54
View File
@@ -1,55 +1,115 @@
groups:
- name: application-logs
type: vlogs
interval: 30s
rules:
- alert: SustainedServerErrors
expr: '_time:5m service:api outcome:completed | stats count() as total, count() if (status:>=500) as errors | filter errors:>=10 | math errors / total as ratio | filter ratio:>0.05 | fields ratio'
for: 5m
labels: {severity: warning}
annotations: {summary: 'API 5xx exceed 5 percent and 10 requests per five minutes'}
- alert: ApiTelemetryMissing
expr: '_time:3m service:api event:process_heartbeat | stats count() as heartbeats | filter heartbeats:0'
for: 3m
labels: {severity: warning}
annotations: {summary: 'API heartbeat is missing'}
- alert: StreamerTelemetryMissing
expr: '_time:3m service:streamer event:process_heartbeat | stats count_uniq(instance) as instances | filter instances:<4'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Fewer than four streamer instances report heartbeats; adjust for planned scaling'}
- alert: ApplicationRestartBurst
expr: '_time:10m event:process_started | stats by (service) count() as restarts | filter restarts:>4'
for: 2m
labels: {severity: warning}
annotations: {summary: 'Repeated process starts in ten minutes'}
- alert: ApplicationMemoryPressure
expr: '_time:3m event:process_heartbeat | stats by (service) max(memoryRatio) as ratio | filter ratio:>0.85'
for: 5m
labels: {severity: warning}
annotations: {summary: 'Container usage exceeds 85 percent of its cgroup memory limit'}
- alert: QueueStalled
expr: '_time:5m event:job_stalled | stats by (queue) count() as stalls | filter stalls:>0'
for: 2m
labels: {severity: warning}
annotations: {summary: 'BullMQ reports stalled jobs'}
- alert: QueueWaitingTooLong
expr: '_time:3m event:queue_heartbeat | stats by (queue) max(oldestWaitingMs) as age | filter age:>600000'
for: 5m
labels: {severity: warning}
annotations: {summary: 'A runnable job has waited over ten minutes'}
- alert: QueueTelemetryMissing
expr: '_time:3m event:queue_heartbeat | stats count_uniq(queue) as queues | filter queues:<3'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Queue telemetry is missing'}
- alert: WebhookFailures
expr: '_time:5m route:"/github/app/webhook" status:>=500 outcome:completed | stats count() as errors | filter errors:>=3'
for: 3m
labels: {severity: warning}
annotations: {summary: 'Repeated webhook server failures; signature rejection is excluded'}
- alert: ApplicationLogDrops
expr: '_time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service) max(droppedLogs) as drops | filter drops:>0'
for: 2m
labels: {severity: warning}
annotations: {summary: 'An application process dropped stdout logs since startup'}
- name: application-logs
type: vlogs
interval: 30s
rules:
- alert: SustainedServerErrors
expr: _time:5m service:api outcome:completed | stats count() as total, count() if (status:>=500) as errors | filter errors:>=10 | math errors / total as ratio | filter ratio:>0.05 | fields ratio
for: 5m
labels:
severity: warning
annotations:
summary: API 5xx exceed 5 percent and 10 requests per five minutes
description: Compare completed API 5xx by release and route; follow a request UUID to streamer and edge evidence.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Eapi%24&var-event=.%2A
investigation_query: _time:5m service:api outcome:completed
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApiTelemetryMissing
expr: _time:3m service:api event:process_heartbeat | stats count() as heartbeats | filter heartbeats:0
for: 3m
labels:
severity: warning
annotations:
summary: API heartbeat is missing
description: Check API container health and collector ingestion. Missing logs alone do not establish an API outage.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Eapi%24&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m service:api event:process_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: StreamerTelemetryMissing
expr: _time:3m service:streamer event:process_heartbeat | stats count_uniq(instance) as instances | filter instances:<4
for: 3m
labels:
severity: warning
annotations:
summary: Fewer than four streamer instances report heartbeats; adjust for planned scaling
description: Check expected replica count and recent deployment window, then container health and collector ingestion.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Estreamer%24&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m service:streamer event:process_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApplicationRestartBurst
expr: _time:10m event:process_started | stats by (service) count() as restarts | filter restarts:>4
for: 2m
labels:
severity: warning
annotations:
summary: Repeated process starts in ten minutes
description: Compare process starts with deployment events; inspect fatal errors and container exit reasons.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Eprocess_started%24
investigation_query: _time:10m event:process_started
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApplicationMemoryPressure
expr: _time:3m event:process_heartbeat | stats by (service) max(memoryRatio) as ratio | filter ratio:>0.85
for: 5m
labels:
severity: warning
annotations:
summary: Container usage exceeds 85 percent of its cgroup memory limit
description: Inspect cgroup usage and release, including cache and worker memory; RSS alone excludes page cache.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m event:process_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: QueueStalled
expr: _time:5m event:job_stalled | stats by (queue) count() as stalls | filter stalls:>0
for: 2m
labels:
severity: warning
annotations:
summary: BullMQ reports stalled jobs
description: Follow job UUID and attempt; check worker exits and Redis lock renewal before retrying.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Ejob_stalled%24
investigation_query: _time:5m event:job_stalled
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: QueueWaitingTooLong
expr: _time:3m event:queue_heartbeat | stats by (queue) max(oldestWaitingMs) as age | filter age:>600000
for: 5m
labels:
severity: warning
annotations:
summary: A runnable job has waited over ten minutes
description: Compare runnable waiting age with worker heartbeats, concurrency and failures; delayed jobs are excluded.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Equeue_heartbeat%24
investigation_query: _time:3m event:queue_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: QueueTelemetryMissing
expr: _time:3m event:queue_heartbeat | stats count_uniq(queue) as queues | filter queues:<3
for: 3m
labels:
severity: warning
annotations:
summary: Queue telemetry is missing
description: Check API queue sampler and Redis connectivity; missing telemetry is not proof that queues are empty.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Equeue_heartbeat%24
investigation_query: _time:3m event:queue_heartbeat
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: WebhookFailures
expr: _time:5m route:"/github/app/webhook" status:>=500 outcome:completed | stats count() as errors | filter errors:>=3
for: 3m
labels:
severity: warning
annotations:
summary: Repeated webhook server failures; signature rejection is excluded
description: Inspect webhook route 5xx and safe error code; signature rejection is excluded.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=%5Eapi%24&var-event=.%2A
investigation_query: _time:5m route:"/github/app/webhook" status:>=500 outcome:completed
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: ApplicationLogDrops
expr: _time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service) sum(droppedLogsInterval) as drops | filter drops:>0
for: 2m
labels:
severity: warning
annotations:
summary: Application log output dropped records in the last three minutes
description: Inspect interval loss and stdout backpressure; cumulative droppedLogs is historical, not a current rate.
investigation_url: /d/anonymous-investigation?from=now-1h&to=now&var-service=.%2A&var-event=%5Eprocess_heartbeat%24
investigation_query: _time:3m (event:process_heartbeat OR event:job_completed OR event:job_failed)
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
+62 -32
View File
@@ -1,33 +1,63 @@
groups:
- name: logging-infrastructure
rules:
- alert: LoggingTelemetryMissing
expr: up == 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'A logging component cannot be scraped'}
- alert: CollectorDrops
expr: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
for: 2m
labels: {severity: warning}
annotations: {summary: 'Collector dropped events; inspect buffer and transform counters'}
- alert: CollectorErrors
expr: sum(increase(vector_component_errors_total[5m])) > 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'Collector errors persist'}
- alert: LogStoragePressure
expr: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
for: 5m
labels: {severity: warning}
annotations: {summary: 'Log storage is near the retention budget or free-space reserve'}
- alert: LogIngestionRejected
expr: sum(increase(vl_rows_dropped_total[5m])) > 0
for: 2m
labels: {severity: warning}
annotations: {summary: 'VictoriaLogs rejected records'}
- alert: LogAlertEvaluationFailed
expr: sum(increase(vmalert_execution_errors_total[5m])) > 0
for: 3m
labels: {severity: warning}
annotations: {summary: 'Log alert queries are failing'}
- name: logging-infrastructure
rules:
- alert: LoggingTelemetryMissing
expr: up == 0
for: 3m
labels:
severity: warning
annotations:
summary: A logging component cannot be scraped
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: up == 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: CollectorDrops
expr: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
for: 2m
labels:
severity: warning
annotations:
summary: Collector dropped events; inspect buffer and transform counters
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vector_buffer_discarded_events_total[5m])) > 0 or sum(increase(vector_component_discarded_events_total{intentional!="true"}[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: CollectorErrors
expr: sum(increase(vector_component_errors_total[5m])) > 0
for: 3m
labels:
severity: warning
annotations:
summary: Collector errors persist
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vector_component_errors_total[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: LogStoragePressure
expr: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
for: 5m
labels:
severity: warning
annotations:
summary: Log storage is near the retention budget or free-space reserve
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(vl_data_size_bytes) > 8 * 1024^3 or min(vl_free_disk_space_bytes) < 8 * 1024^3
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: LogIngestionRejected
expr: sum(increase(vl_rows_dropped_total[5m])) > 0
for: 2m
labels:
severity: warning
annotations:
summary: VictoriaLogs rejected records
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vl_rows_dropped_total[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
- alert: LogAlertEvaluationFailed
expr: sum(increase(vmalert_execution_errors_total[5m])) > 0
for: 3m
labels:
severity: warning
annotations:
summary: Log alert queries are failing
investigation_url: /d/anonymous-operations?from=now-1h&to=now
investigation_query: sum(increase(vmalert_execution_errors_total[5m])) > 0
runbook_url: https://github.com/tdurieux/anonymous_github/blob/main/deploy/logging/README.md#alert-investigation
+13
View File
@@ -13,3 +13,16 @@
/usr/sbin/nginx -s reopen
endscript
}
# Optional deployment event file. Pre-create it for the deploy operator with
# group adm and mode 0640; copytruncate preserves that explicit ownership.
/var/log/nginx/anonymous.deploy.json {
hourly
rotate 6
maxsize 1M
missingok
notifempty
compress
delaycompress
copytruncate
}
+9 -5
View File
@@ -10,6 +10,7 @@ if starts_with(string!(.file), "/host/docker/") {
record, err = parse_json(string(raw.log) ?? "")
if err != null { record = {} }
}
if starts_with(string!(.file), "/host/nginx/anonymous.deploy.json") { service = "api" }
out = {"timestamp": now(), "service": service, "severity": "info", "event": "service_output"}
# Use capture time for unsupported formats. Invalid/old producer timestamps
# cannot defeat retention or crash the collector.
@@ -36,10 +37,10 @@ if service == "mongodb" {
} else if includes(["api", "streamer"], service) {
if record.service == "worker" { out.service = "worker" }
if includes(["debug", "info", "warn", "error"], record.severity) { out.severity = record.severity }
if includes(["account_removed", "application_log", "billing_maintenance_failed", "billing_reconciliation_deferred", "billing_requires_attention", "billing_worker_failed", "blob_api_failed_falling_back_to_raw_url", "branch_not_found", "cache_removal_failed", "cache_removed", "checking_repository_status_and_unused_repositories", "cleared_lock_for_active_job", "conference_expire_failed", "content_expiration_failed", "daily_stats_snapshot_failed", "daily_stats_snapshot_stored", "download_completed", "download_start", "downloaded", "downloadfile_failed", "downloading_file", "downloading_gist", "downloading_pull_request", "downloading_via_raw_url_lfs", "duplicate_stripe_customer_requires_review", "ensuretodaysnapshot_failed", "error_log_redis_connect_failed", "expired_repository_cleanup_failed", "failed_removal_recovery_failed", "failed_to_persist_error_status", "failed_to_record_repository_removal_error", "gettoken", "github_primary_rate_limit_hit", "github_secondary_rate_limit_hit", "grant_revocation_failed", "initial_repository_status_check_failed", "job_completed", "job_failed", "job_stalled", "job_started", "lastusedat_update_failed", "login_token_failed", "login_token_req_login_failed", "logout_after_account_removal_failed", "logout_failed", "lookup_failed", "markerrorifinflight_failed", "markerrorifremoving_failed", "metadata_cleanup_complete", "metadata_cleanup_deferred", "owner_access_alert_reset_unavailable", "owner_access_email_accepted", "owner_access_email_audit_unavailable", "owner_access_email_claim_cleanup_unavailable", "owner_access_email_deferred_recovery_unavailable", "owner_access_email_delivery_failed", "owner_access_email_preparation_cleanup_unavailable", "owner_access_email_recovery_recorded", "owner_access_email_recovery_unavailable", "owner_access_email_result_reconciliation_required", "owner_access_email_unavailable", "pr_comments_404_continuing_without_them", "process_fatal_error", "process_heartbeat", "process_started", "progress", "queue_heartbeat", "queued_for_download", "rate_limit_gate_active_delaying_job_before_work", "rate_limit_gate_set", "rate_limited_delaying_job", "recover_failed", "recovered_expired_repository_cleanup", "recovered_file_from_truncated_tree", "recovered_file_invalidation_failed", "recovered_stuck_repo", "recoverstuckpreparing_failed", "recoverstuckremoving_failed", "redis_client_error", "redis_rate_limit_gate_written", "rejected_upstream_download_destination", "removal_failed", "removal_recovery_failed", "removecache_save_failed", "removed_cache_for_unused_repository", "removing_cache", "removing_repository", "repo_claimed", "repository_cache_removal_failed", "repository_expiration_failed", "repository_expired", "repository_removal_failed", "repository_removed", "request_cancelled", "request_completed", "request_slow", "requeued_failed_removal", "requeued_interrupted_removal", "reset", "retired_tree_cleanup_deferred", "retired_tree_cleanup_failed", "running_daily_stats_snapshot", "s3_body_stream_error", "send_failed", "server_started", "source_stream_failed_cache_write_discarded", "staged_tree_cleanup_failed", "staged_tree_completion_deferred", "streamer_fetch_failed", "streamer_started", "streamer_startup_failed", "streamer_zip_fetch_failed", "temporary_storage_cleanup_deferred", "token_refresh_failed_backing_off", "up_to_date", "update_queued", "verify_failed", "waiting_for_rate_limit_gate", "webhook_log", "webhook_processing_failed", "write_failed", "zip_stream_failed"], record.event) { out.event = record.event }
if includes(["deployment", "account_removed", "application_log", "billing_maintenance_failed", "billing_reconciliation_deferred", "billing_requires_attention", "billing_worker_failed", "blob_api_failed_falling_back_to_raw_url", "branch_not_found", "cache_removal_failed", "cache_removed", "checking_repository_status_and_unused_repositories", "cleared_lock_for_active_job", "conference_expire_failed", "content_expiration_failed", "daily_stats_snapshot_failed", "daily_stats_snapshot_stored", "download_completed", "download_start", "downloaded", "downloadfile_failed", "downloading_file", "downloading_gist", "downloading_pull_request", "downloading_via_raw_url_lfs", "duplicate_stripe_customer_requires_review", "ensuretodaysnapshot_failed", "error_log_redis_connect_failed", "expired_repository_cleanup_failed", "failed_removal_recovery_failed", "failed_to_persist_error_status", "failed_to_record_repository_removal_error", "gettoken", "github_primary_rate_limit_hit", "github_secondary_rate_limit_hit", "grant_revocation_failed", "initial_repository_status_check_failed", "job_completed", "job_failed", "job_stalled", "job_started", "lastusedat_update_failed", "login_token_failed", "login_token_req_login_failed", "logout_after_account_removal_failed", "logout_failed", "lookup_failed", "markerrorifinflight_failed", "markerrorifremoving_failed", "metadata_cleanup_complete", "metadata_cleanup_deferred", "owner_access_alert_reset_unavailable", "owner_access_email_accepted", "owner_access_email_audit_unavailable", "owner_access_email_claim_cleanup_unavailable", "owner_access_email_deferred_recovery_unavailable", "owner_access_email_delivery_failed", "owner_access_email_preparation_cleanup_unavailable", "owner_access_email_recovery_recorded", "owner_access_email_recovery_unavailable", "owner_access_email_result_reconciliation_required", "owner_access_email_unavailable", "pr_comments_404_continuing_without_them", "process_fatal_error", "process_heartbeat", "process_started", "progress", "queue_heartbeat", "queued_for_download", "rate_limit_gate_active_delaying_job_before_work", "rate_limit_gate_set", "rate_limited_delaying_job", "recover_failed", "recovered_expired_repository_cleanup", "recovered_file_from_truncated_tree", "recovered_file_invalidation_failed", "recovered_stuck_repo", "recoverstuckpreparing_failed", "recoverstuckremoving_failed", "redis_client_error", "redis_rate_limit_gate_written", "rejected_upstream_download_destination", "removal_failed", "removal_recovery_failed", "removecache_save_failed", "removed_cache_for_unused_repository", "removing_cache", "removing_repository", "repo_claimed", "repository_cache_removal_failed", "repository_expiration_failed", "repository_expired", "repository_removal_failed", "repository_removed", "request_cancelled", "request_completed", "request_slow", "requeued_failed_removal", "requeued_interrupted_removal", "reset", "retired_tree_cleanup_deferred", "retired_tree_cleanup_failed", "running_daily_stats_snapshot", "s3_body_stream_error", "send_failed", "server_started", "source_stream_failed_cache_write_discarded", "staged_tree_cleanup_failed", "staged_tree_completion_deferred", "streamer_fetch_failed", "streamer_started", "streamer_startup_failed", "streamer_zip_fetch_failed", "temporary_storage_cleanup_deferred", "token_refresh_failed_backing_off", "up_to_date", "update_queued", "verify_failed", "waiting_for_rate_limit_gate", "webhook_log", "webhook_processing_failed", "write_failed", "zip_stream_failed"], record.event) { out.event = record.event }
if is_string(record.release) && match(string!(record.release), r'^[a-f0-9]{7,40}$') { out.release = record.release }
if is_string(record.instance) && match(string!(record.instance), r'^[a-f0-9-]{36}$') { out.instance = record.instance }
if includes(["ABORT_ERR", "ECONNREFUSED", "ECONNRESET", "ENOTFOUND", "EPIPE", "ETIMEDOUT", "billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_app_reconnect_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_token_invalid", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_signature", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "token_refresh_failed", "quota_exceeded", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_found", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "upstream_timeout", "user_not_found", "username_not_defined", "webhook_processing_failed"], record.errorCode) { out.errorCode = record.errorCode }
if includes(["repo_renamed", "preparation_interrupted", "commit_not_found", "zip_not_available", "ABORT_ERR", "ECONNREFUSED", "ECONNRESET", "ENOTFOUND", "EPIPE", "ETIMEDOUT", "billing_address_missing", "billing_city_missing", "billing_country_missing", "billing_email_missing", "billing_missing", "billing_name_missing", "billing_zip_missing", "branch_not_found", "branch_not_specified", "cache_busy", "cache_miss", "cannot_coauthor_self", "commit_not_specified", "conf_end_date_invalid", "conf_end_date_missing", "conf_id_missing", "conf_id_used", "conf_name_missing", "conf_not_activated", "conf_not_found", "conf_start_date_invalid", "conf_start_date_missing", "conference_not_found", "connection_changed", "download_not_enabled", "expiration_limit_reached", "file_listing_too_large", "file_not_accessible", "file_not_found", "file_not_supported", "file_too_big", "folder_not_supported", "gistId_already_used", "gistId_not_specified", "gist_expired", "gist_not_found", "gist_not_ready", "github_app_access_required", "github_app_reconnect_required", "github_identity_mismatch", "github_rate_limit_exceeded", "github_token_invalid", "github_user_not_found", "invalid_commit_format", "invalid_cursor", "invalid_file_path", "invalid_gistId", "invalid_path", "invalid_plan", "invalid_preserve_files_format", "invalid_project_name", "invalid_pullRequestId", "invalid_repo", "invalid_repoId", "invalid_request", "invalid_settings", "invalid_signature", "invalid_sort", "invalid_status", "invalid_terms_format", "invalid_token_id", "invalid_token_options", "is_removed", "not_authorized", "not_connected", "options_not_provided", "page_not_activated", "page_not_supported_on_different_branch", "path_not_defined", "path_not_specified", "pullRequestId_already_used", "pullRequestId_is_not_a_number", "pullRequestId_not_specified", "pull_request_expired", "pull_request_not_found", "pull_request_not_ready", "token_refresh_failed", "quota_exceeded", "rate_limited", "readme_not_available", "repoId_already_used", "repoId_not_defined", "repoUrl_not_defined", "repo_access_limited", "repo_empty", "repo_not_accessible", "repo_not_found", "repo_saml_enforcement", "repository_archived", "repository_changed", "repository_expired", "repository_job_cancelled", "repository_not_found", "repository_not_ready", "repository_not_specified", "repository_tree_incomplete", "repository_tree_too_large", "s3_config_not_provided", "source_not_provided", "storage_delete_failed", "storage_write_size_mismatch", "submodule_not_available", "terms_not_specified", "token_expired", "token_limit_reached", "unable_to_connect_user", "unable_to_download", "upstream_error", "upstream_timeout", "user_not_found", "username_not_defined", "webhook_processing_failed"], record.errorCode) { out.errorCode = record.errorCode }
if includes(["admin", "anonymized-file", "auth", "billing", "dailyStats", "file-tree", "fs", "gh-stream", "gist", "github", "github-app-webhook", "owner-notifications", "process", "pull-request", "queue", "queue:cache", "queue:download", "queue:remove", "repository", "repository-public", "requests", "route", "route:repo", "s3", "schedule", "server", "streamer", "temporary-storage", "token-auth", "user", "zip-stream"], record.module) { out.module = record.module }
if includes(["protocol", "credentials", "port", "host"], record.reason) { out.reason = record.reason }
if includes(["parse", "find_grant_owners", "reconcile_grant", "persist_installation_block", "reconcile_installation"], record.phase) { out.phase = record.phase }
@@ -48,14 +49,17 @@ if service == "mongodb" {
if includes(["download", "remove", "cache"], record.queue) { out.queue = record.queue }
if includes(["unknown", "api", "streamer", "timeout", "upstream"], record.cancellationOrigin) { out.cancellationOrigin = record.cancellationOrigin }
if includes(["response_close", "request_aborted", "upstream_error", "timeout", "local_abort"], record.cancellationEvidence) { out.cancellationEvidence = record.cancellationEvidence }
if includes(["deploy", "rollback"], record.deploymentOperation) { out.deploymentOperation = record.deploymentOperation }
if includes(["start", "restart", "finish"], record.deploymentPhase) { out.deploymentPhase = record.deploymentPhase }
if includes(["started", "completed", "failed", "unchanged"], record.deploymentOutcome) { out.deploymentOutcome = record.deploymentOutcome }
if includes(["public", "internal"], record.endpointKind) { out.endpointKind = record.endpointKind }
if includes(["completed", "interrupted"], record.outcome) { out.outcome = record.outcome }
}
for_each(["requestId", "jobId", "deliveryId", "hopId", "parentHopId"]) -> |_index, key| {
for_each(["requestId", "jobId", "deliveryId", "hopId", "parentHopId", "deploymentId"]) -> |_index, key| {
value = get(record, [key]) ?? null
if is_string(value) && match(string!(value), r'^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$') { out = set!(out, [key], value) }
}
for_each(["stage_transfer_ms", "stage_transfer_count", "stage_transfer_open", "stage_database_ms", "stage_database_count", "stage_database_open", "expectedBytes", "measuredMs", "unattributedMs", "overlapMs", "stage_repository_count", "stage_repository_open", "stage_credential_resolution_count", "stage_credential_resolution_open", "stage_upstream_count", "stage_upstream_open", "stage_cache_lookup_count", "stage_cache_lookup_open", "stage_cache_fill_count", "stage_cache_fill_open", "stage_worker_wait_count", "stage_worker_wait_open", "stage_anonymize_count", "stage_anonymize_open", "stage_streamer_headers_count", "stage_streamer_headers_open", "status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus", "stage_repository_ms", "stage_credential_resolution_ms", "stage_upstream_ms", "stage_cache_lookup_ms", "stage_cache_fill_ms", "stage_worker_wait_ms", "stage_anonymize_ms", "stage_streamer_headers_ms"]) -> |_index, key| {
for_each(["restartMs", "exitCode", "droppedLogsInterval", "stage_transfer_ms", "stage_transfer_count", "stage_transfer_open", "stage_database_ms", "stage_database_count", "stage_database_open", "expectedBytes", "measuredMs", "unattributedMs", "overlapMs", "stage_repository_count", "stage_repository_open", "stage_credential_resolution_count", "stage_credential_resolution_open", "stage_upstream_count", "stage_upstream_open", "stage_cache_lookup_count", "stage_cache_lookup_open", "stage_cache_fill_count", "stage_cache_fill_open", "stage_worker_wait_count", "stage_worker_wait_open", "stage_anonymize_count", "stage_anonymize_open", "stage_streamer_headers_count", "stage_streamer_headers_open", "status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond", "attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed", "oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus", "stage_repository_ms", "stage_credential_resolution_ms", "stage_upstream_ms", "stage_cache_lookup_ms", "stage_cache_fill_ms", "stage_worker_wait_ms", "stage_anonymize_ms", "stage_streamer_headers_ms"]) -> |_index, key| {
value = get(record, [key]) ?? null
if is_integer(value) || is_float(value) { out = set!(out, [key], value) }
}
@@ -75,7 +79,7 @@ if includes(["/", "other", "/dashboard", "/anonymize", "/healthcheck", "/api/hea
}
}
# BEGIN GENERATED DIAGNOSTICS
source_functions = {"src/cli/index.ts":["<anonymous>","generateRandomFileName","main"],"build/cli/index.js":["<anonymous>","generateRandomFileName","main"],"src/config.ts":["<anonymous>"],"build/config.js":["<anonymous>"],"src/core/AnonymizedFile.ts":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"build/core/AnonymizedFile.js":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"src/core/AnonymousError.ts":["<anonymous>","detail","toString","url"],"build/core/AnonymousError.js":["<anonymous>","detail","toString","url"],"src/core/Conference.ts":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"build/core/Conference.js":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"src/core/Gist.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/Gist.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/GitHubUtils.ts":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"build/core/GitHubUtils.js":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"src/core/PullRequest.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/PullRequest.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/Repository.ts":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"build/core/Repository.js":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"src/core/User.ts":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"build/core/User.js":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"src/core/anonymization-pool.ts":["<anonymous>","anonymizeOnWorker","cancel","exited","failed","finish","getAnonymizationPoolStats","message","pump","release","releaseSpool","reserveSpool"],"build/core/anonymization-pool.js":["Line truncated
source_functions = {"src/cli/index.ts":["<anonymous>","generateRandomFileName","main"],"build/cli/index.js":["<anonymous>","generateRandomFileName","main"],"src/config.ts":["<anonymous>"],"build/config.js":["<anonymous>"],"src/core/AnonymizedFile.ts":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"build/core/AnonymizedFile.js":["<anonymous>","anonymizedContent","assertContentFile","cancel","content","defaultMaskCandidateRegex","escapeRegex","extension","forwardHeaders","getFileInfo","handleStreamError","isFileSupported","isImage","originalPath","probe","recoverTruncatedFile","send","sha","size","streamerErrorToAnonymous","visibleFile"],"src/core/AnonymousError.ts":["<anonymous>","detail","toString","url"],"build/core/AnonymousError.js":["<anonymous>","detail","toString","url"],"src/core/Conference.ts":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"build/core/Conference.js":["<anonymous>","expire","isExpired","remove","repositories","toJSON","updateStatus"],"src/core/Gist.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/Gist.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","downloadWithAccess","expire","getAccess","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/GitHubUtils.ts":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"build/core/GitHubUtils.js":["<anonymous>","abortableGateLookup","cancelled","checkToken","ensureRedisGateClient","getRedisGateResetAt","getToken","getTokenGateResetAt","octokit","rateLimitDetail","resolveRepositoryToken","run","setRedisGate","setRedisGateFromWorker","setTokenGate","waitForTokenGate"],"src/core/PullRequest.ts":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"build/core/PullRequest.js":["<anonymous>","anonymize","check","conference","content","countView","download","downloadContent","expire","getToken","markExpired","remove","resetSate","toJSON","updateIfNeeded","updateStatus"],"src/core/Repository.ts":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"build/core/Repository.js":["<anonymous>","anonymize","anonymizeTreeRecursive","anonymizedFiles","assertNotArchived","buildPathIndex","check","checkRefreshWrite","cleanupRetiredFileTrees","cleanupStagedFileTree","clearRows","completeRecoveredFileMetadata","computeSize","conference","countView","current","ensureFileTree","expire","fileSnapshotFilter","files","findAnonymizedPath","generateAnonymizeTransformer","getToken","hasEmptyTree","indexPaths","insertBatch","invalidateFileMetadata","isReady","markCachePresent","markExpired","pathKey","readFiles","refresh","refreshFilter","refreshIfNeeded","remove","removeCache","resetSate","searchFiles","snapshotValue","submoduleMounts","toJSON","updateIfNeeded","updateProgressMessage","updateRetiringPathIndex","updateStatus","zip"],"src/core/User.ts":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"build/core/User.js":["<anonymous>","getAccessToken","getGists","getGitHubRepositories","getPullRequests","getRepositories","markListExpired","quotaRepositories","repositoryMembership","toJSON"],"src/core/anonymization-pool.ts":["<anonymous>","anonymizeOnWorker","cancel","exited","failed","finish","getAnonymizationPoolStats","message","pump","release","releaseSpool","reserveSpool"],"build/core/anonymization-pool.js":["Line truncated
for_each(["errorType", "causeType"]) -> |_index, key| {
value = get(record, [key]) ?? null
if includes(["Error","TypeError","RangeError","ReferenceError","SyntaxError","URIError","EvalError","AggregateError","AnonymousError","HTTPError","RequestError","MongoServerError","MongoNetworkError","MongoServerSelectionError","ValidationError","TimeoutError","AbortError","UnknownError"], value) { out = set!(out, [key], value) }
+8 -1
View File
@@ -15,13 +15,20 @@ sources:
read_from: end
max_line_bytes: 65536
fingerprint: {strategy: device_and_inode}
deployments:
type: file
include: [/host/nginx/anonymous.deploy.json*]
exclude: ['*.gz']
read_from: end
max_line_bytes: 4096
fingerprint: {strategy: device_and_inode}
internal:
type: internal_metrics
scrape_interval_secs: 15
transforms:
safe:
type: remap
inputs: [docker, nginx]
inputs: [docker, nginx, deployments]
file: /etc/vector/sanitize.vrl
drop_on_error: true
drop_on_abort: true
+50 -1
View File
@@ -18,6 +18,46 @@ COORDINATED_RESTART="${COORDINATED_RESTART:-0}"
log() { printf '==> %s\n' "$*"; }
warn() { printf '!! %s\n' "$*" >&2; }
# Local, finite deployment evidence. Collection is opt-in via a pre-created file.
# Logging failure must never change deployment behavior or expose shell commands.
deployment_started=$(date +%s)
restart_started=0
deployment_id=$(cat /proc/sys/kernel/random/uuid 2>/dev/null || true)
[[ $deployment_id =~ ^[a-f0-9-]{36}$ ]] || deployment_id=""
deployment_release=${APP_RELEASE:-unknown}
[[ $deployment_release =~ ^[a-f0-9]{7,40}$ ]] || deployment_release=unknown
deployment_operation=${DEPLOY_OPERATION:-deploy}
[[ $deployment_operation == rollback ]] || deployment_operation=deploy
deployment_outcome=completed
deployment_event() {
local phase="$1" outcome="$2" code="$3" now elapsed restart_ms record
now=$(date +%s); elapsed=$(( (now - deployment_started) * 1000 )); restart_ms=0
if (( restart_started > 0 )); then restart_ms=$(( (now - restart_started) * 1000 )); fi
printf -v record '{"timestamp":"%s","service":"api","severity":"%s","event":"deployment","deploymentId":"%s","release":"%s","deploymentOperation":"%s","deploymentPhase":"%s","deploymentOutcome":"%s","ms":%d,"restartMs":%d,"exitCode":%d}' \
"$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$([[ $code == 0 ]] && echo info || echo error)" "$deployment_id" "$deployment_release" "$deployment_operation" "$phase" "$outcome" "$elapsed" "$restart_ms" "$code"
printf '%s\n' "$record" || true
if [[ -n ${DEPLOY_EVENT_FILE:-} ]]; then
if [[ -f $DEPLOY_EVENT_FILE && ! -L $DEPLOY_EVENT_FILE ]]; then
(printf '%s\n' "$record" >> "$DEPLOY_EVENT_FILE") 2>/dev/null || warn "deployment event file unavailable"
else warn "deployment event file must already exist and must not be a symlink"; fi
fi
return 0
}
restart_event() {
if (( restart_started == 0 )); then restart_started=$(date +%s); deployment_event restart started 0; fi
}
deployment_finished() {
local code=$?
if (( code != 0 )); then deployment_outcome=failed; fi
deployment_event finish "$deployment_outcome" "$code"
exit "$code"
}
trap deployment_finished EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM
deployment_event start started 0
image_id() { docker image inspect --format '{{.Id}}' "$1" 2>/dev/null || true; }
wait_healthy() {
@@ -67,6 +107,7 @@ if [[ $legacy_alerts == 1 ]]; then
[[ $protocol == recipient-v2 ]] || { warn "replacement image lacks recipient alert migration support"; exit 1; }
TIMEOUT="${MIGRATION_HEALTH_TIMEOUT:-900}"
log "Pausing legacy application writers before recipient alert migration"
restart_event
docker compose stop -t 30 anonymous_github streamer
for app in "${apps[@]}"; do
running="$(docker inspect -f '{{.State.Running}}' "$app")"
@@ -96,6 +137,7 @@ for app in "${apps[@]}"; do
done
if [[ $COORDINATED_RESTART == 1 ]]; then
log "Coordinated application restart for shared cache compatibility"
restart_event
docker compose stop -t 30 anonymous_github streamer
for app in "${apps[@]}"; do
[[ $(docker inspect -f '{{.State.Running}}' "$app") == false ]] || { warn "cache writer $app is still running"; exit 1; }
@@ -116,13 +158,18 @@ fi
mapfile -t olds < <(list_streamers)
if [[ ${#olds[@]} -eq 0 ]]; then
log "Cold start — bringing the whole stack up"
restart_event
docker compose up -d --remove-orphans
api="$(docker compose ps -q anonymous_github)"
[[ -n $api ]] || { warn "API container missing after cold start"; exit 1; }
wait_healthy "$api"
docker compose ps
exit 0
fi
# 3. No-op if image unchanged
if [[ -n $before && $before == "$after" && $FORCE != 1 ]]; then
deployment_outcome=unchanged
log "Image unchanged — nothing to deploy (FORCE=1 to override)"
docker compose ps
exit 0
@@ -152,6 +199,7 @@ if [[ $fail == 1 ]]; then
fi
# Drop olds and reconcile scale.
restart_event
docker rm -f "${olds[@]}" >/dev/null
docker compose up -d --no-deps --no-recreate --scale "streamer=${REPLICAS}" streamer >/dev/null
@@ -160,6 +208,7 @@ docker compose up -d --no-deps --no-recreate --scale "streamer=${REPLICAS}" stre
log "Recreating API"
docker compose up -d --no-deps --remove-orphans anonymous_github
api="$(docker compose ps -q anonymous_github | head -1)"
[[ -n $api ]] && wait_healthy "$api" || true
[[ -n $api ]] || { warn "API container missing after restart"; exit 1; }
wait_healthy "$api"
docker compose ps
+17 -2
View File
@@ -23,10 +23,11 @@ const sleep = ms => new Promise(resolve=>setTimeout(resolve,ms));
const requestId=randomUUID(),jobId=randomUUID();
const records=['api','streamer','worker'].map((service,i)=>({timestamp:new Date(Date.now()-10000+i).toISOString(),event:i===2?'job_failed':'request_completed',severity:'error',service,module:'queue',status:502,errorCode:'upstream_error',release:'abcdef1',requestId,...(i===2?{jobId}:{}),url:'private-owner/private-repository?token=secret',body:'<script>secret</script>'}));
records.push({...records[0],requestId:randomUUID(),timestamp:new Date(Date.now()-5000).toISOString()});
records.push({...records[0], release:'abcdef0', timestamp:new Date(Date.now()-2*60*60*1000).toISOString()});
const ingest = await fetch(process.env.ADMIN_LOG_URL+'/insert/jsonline?_time_field=timestamp', {method:'POST',body:records.map(JSON.stringify).join('\n'),headers:{'Content-Type':'application/stream+json'}});
assert(ingest.ok); await sleep(2000);
const errors=await investigateLogs(logOptions({service:'api',release:'abcdef1'}));
assert.equal(errors.state,'stale'); assert.equal(errors.groups.length,1); assert.equal(errors.groups[0].count,2);
assert.equal(errors.state,'stale'); assert.equal(errors.groups.length,1); assert.equal(errors.groups[0].count,2); assert.equal(errors.groups[0].affectedRequests,2); assert.equal(errors.groups[0].regression,'known'); assert.equal(errors.baseline.partial,false);
assert(!JSON.stringify(errors).includes('private-owner')); assert(!JSON.stringify(errors).includes('secret'));
const timeline=await investigateLogs(logOptions({kind:'jobId',id:jobId},true),true);
assert.equal(timeline.state,'ready'); assert.deepEqual([...new Set(timeline.events.map(row=>row.service))].sort(),['api','streamer','worker']);
@@ -91,7 +92,7 @@ const sleep = ms => new Promise(resolve=>setTimeout(resolve,ms));
const redis=safeRedisLog({ts:diagnostic.timestamp,level:'error',module:'route',raw:[diagnostic]});
for(const row of [traced.events[0],redis.raw[0]]) {assert.equal(row.requestPath,diagnostic.requestPath);assert.equal(row.repoId,'demo-123');assert.equal(row.route,diagnostic.route);assert.equal(row.method,'GET');assert.equal(row.errorStack,projected.errorStack);assert.equal(row.causeType,'RangeError');assert(!JSON.stringify(row).includes(secret));}
const paths=['/api/repo/demo-123/refresh','/api/repo/demo-123/extend','/api/repo/demo-123/coauthors','/api/repo/demo-123','/api/repo/demo-123/zip','/api/repo/demo-123/files','/api/repo/demo-123/files/counts','/api/repo/demo-123/options','/api/repo/demo-123/file/README.md'];
const pathRecords=paths.map((requestPath,i)=>logContext.run({requestId:randomUUID(),requestPath,method:/\/(refresh|extend|coauthors)$/.test(requestPath)?'POST':'GET'},()=>logRecord(i%2?'warn':'info','requests',['request',{status:200,outcome:i%2?'interrupted':'completed'}])));
const pathRecords=paths.map((requestPath,i)=>logContext.run({requestId:randomUUID(),requestPath,method:/\/(refresh|extend|coauthors)$/.test(requestPath)?'POST':'GET'},()=>logRecord(i%2?'warn':'info','requests',['request',{status:200,outcome:i%2?'interrupted':'completed',bytes:7,expectedBytes:100,cancellationOrigin:'unknown',cancellationEvidence:'response_close',hopId:randomUUID(),endpointKind:'public',measuredMs:50,unattributedMs:20,overlapMs:10,stages:{database:10,transfer:40},stageCounts:{database:2},stageOpen:{database:0}}])));
const pathSources=pathRecords.map(record=>({...rawSource,message:JSON.stringify({attrs:{tag:'api'},log:JSON.stringify(record)})}));
const pathOutput=execFileSync('docker',['run','--rm','-i','--network','none','--memory','256m',vectorImage,'vrl','-i','/dev/stdin',fs.readFileSync('deploy/logging/sanitize.vrl','utf8')+'\nencode_json(.)'],{encoding:'utf8',input:pathSources.map(JSON.stringify).join('\n')+'\n'}).trim().split('\n').map(line=>JSON.parse(JSON.parse(line)));
assert.deepEqual(pathOutput.map(x=>x.requestPath),paths);
@@ -101,9 +102,23 @@ const sleep = ms => new Promise(resolve=>setTimeout(resolve,ms));
const result=await investigateLogs(logOptions({kind:'requestId',id:pathRecords[i].requestId},true),true);
assert.equal(result.events[0].requestPath,paths[i]);
assert.equal(result.events[0].repoId,'demo-123');
assert.equal(result.events[0].bytes,7);assert.equal(result.events[0].expectedBytes,100);
assert.equal(result.events[0].cancellationOrigin,'unknown');assert.equal(result.events[0].cancellationEvidence,'response_close');
assert.equal(result.events[0].hopId,pathRecords[i].hopId);assert.equal(result.events[0].endpointKind,'public');
assert.equal(result.events[0].stage_database_ms,10);assert.equal(result.events[0].stage_database_count,2);
assert.equal(result.events[0].measuredMs,50);assert.equal(result.events[0].unattributedMs,20);assert.equal(result.events[0].overlapMs,10);
}
console.log('PASS successful and interrupted anonymous endpoint paths through collector and central API');
console.log('PASS synthetic exception logger → pinned collector → VictoriaLogs/Redis safe API with normalized route and bounded frames');
const deploymentId=randomUUID();
const deployment={timestamp:new Date().toISOString(),event:'deployment',severity:'error',release:'abcdef3',deploymentId,
deploymentPhase:'finish',deploymentOperation:'rollback',deploymentOutcome:'failed',restartMs:3000,ms:5000,exitCode:1,message:secret,url:secret};
const deployOutput=execFileSync('docker',['run','--rm','-i','--network','none','--memory','256m',vectorImage,'vrl','-i','/dev/stdin',fs.readFileSync('deploy/logging/sanitize.vrl','utf8')+'\nencode_json(.)'],{encoding:'utf8',input:JSON.stringify({file:'/host/nginx/anonymous.deploy.json',message:JSON.stringify(deployment)})+'\n'}).trim().split('\n').map(line=>JSON.parse(JSON.parse(line)))[0];
assert.equal(deployOutput.event,'deployment');assert.equal(deployOutput.deploymentId,deploymentId);assert(!JSON.stringify(deployOutput).includes(secret));
assert((await fetch(process.env.ADMIN_LOG_URL+'/insert/jsonline?_time_field=timestamp',{method:'POST',body:JSON.stringify(deployOutput),headers:{'Content-Type':'application/stream+json'}})).ok);await sleep(2000);
const deployView=await investigateLogs(logOptions({release:'abcdef3'}));
assert.equal(deployView.groups[0].sample.deploymentId,deploymentId);assert.equal(deployView.groups[0].sample.deploymentOutcome,'failed');assert.equal(deployView.groups[0].sample.restartMs,3000);
console.log('PASS deployment evidence, request timings/bytes and release baseline through pinned collector and central API');
docker('stop',name);
const unavailable=await investigateLogs(logOptions({})); assert.equal(unavailable.state,'unreachable');
delete process.env.ADMIN_LOG_URL;
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""Pinned, isolated alert firing/recovery. No production access or notifications.
Log tests accelerate windows/for/interval only; expressions and numeric thresholds
are otherwise shipped rules. PromQL tests use promtool's simulated clock.
"""
import json, pathlib, re, subprocess, tempfile, time, urllib.request, urllib.parse, uuid
import yaml
ROOT = pathlib.Path(__file__).resolve().parents[1]
CONFIG = ROOT / 'deploy/logging'
images = yaml.safe_load((CONFIG/'compose.yml').read_text())['services']
def run(*args):
result=subprocess.run(args, capture_output=True, text=True)
if result.returncode:raise RuntimeError(result.stdout+'\n'+result.stderr)
return result.stdout.strip()
def request(url, data=None):
return json.loads(urllib.request.urlopen(urllib.request.Request(url, data=data,
headers={'Content-Type':'application/stream+json'} if data else {}), timeout=5).read())
def wait(fn, seconds=30):
end=time.monotonic()+seconds; last=None
while time.monotonic()<end:
try:
result=fn()
if result:return result
except Exception as error:last=error
time.sleep(.25)
raise AssertionError('condition did not become true: '+str(last))
with tempfile.TemporaryDirectory(prefix='operational-alerts-') as tmp:
tmp=pathlib.Path(tmp); tmp.chmod(0o755); project='operational-alerts-'+uuid.uuid4().hex[:10]
rules=yaml.safe_load((CONFIG/'log-alerts.yml').read_text())
for group in rules['groups']:
group['interval']='1s'
for rule in group['rules']:
rule['expr']=re.sub(r'_time:\d+m', '_time:5s', rule['expr']);rule['for']='2s'
(tmp/'rules.yml').write_text(yaml.safe_dump(rules))
compose={'services':{
'logs':{'image':images['victorialogs']['image'],'mem_limit':'512m','command':['-storageDataPath=/storage','-storage.minFreeDiskSpaceBytes=1MB','-memory.allowedBytes=256MiB'], 'ports':['127.0.0.1::9428']},
'alerts':{'image':images['vmalert']['image'],'command':['-rule=/rules.yml','-datasource.url=http://logs:9428','-notifier.blackhole','-rule.evalDelay=0s'], 'volumes':[str(tmp/'rules.yml')+':/rules.yml:ro'],'ports':['127.0.0.1::8880']},
},'networks':{'default':{'internal':False}}}
(tmp/'compose.yml').write_text(yaml.safe_dump(compose))
cmd=['docker','compose','-p',project,'-f',str(tmp/'compose.yml')]
try:
run(*cmd,'up','-d')
logs='http://'+run(*cmd,'port','logs','9428');alerts='http://'+run(*cmd,'port','alerts','8880')
def states():
groups=request(alerts+'/api/v1/rules')['data']['groups']
return {r['name']:r['state'] for g in groups for r in g['rules']}
wait(states)
def emit(rows):
data=('\n'.join(json.dumps({**r,"message":r.get("event", "fixture")}) for r in rows)+'\n').encode()
urllib.request.urlopen(urllib.request.Request(logs+'/insert/jsonline?_msg_field=message',data=data,headers={'Content-Type':'application/stream+json'}),timeout=5).read()
def healthy():
rows=[{'service':'api','instance':'api','event':'process_heartbeat','memoryRatio':.2,'droppedLogsInterval':0}]
rows += [{'service':'streamer','instance':str(i),'event':'process_heartbeat','memoryRatio':.2,'droppedLogsInterval':0} for i in range(4)]
rows += [{'service':'api','queue':q,'event':'queue_heartbeat','oldestWaitingMs':0} for q in ['download','remove','cache']]
return rows
# Threshold violations, then missing telemetry, then healthy recovery.
target={'SustainedServerErrors','ApplicationRestartBurst','ApplicationMemoryPressure','QueueStalled','QueueWaitingTooLong','WebhookFailures','ApplicationLogDrops'}
def failures():
rows=healthy()
rows += [{'service':'api','event':'request_completed','outcome':'completed','status':503,'route':'/github/app/webhook'} for _ in range(12)]
rows += [{'service':'api','event':'process_started'} for _ in range(5)]
rows += [{'service':'api','event':'process_heartbeat','instance':'api','memoryRatio':.95,'droppedLogsInterval':2}]
rows += [{'service':'api','event':'job_stalled','queue':'download'},{'service':'api','event':'queue_heartbeat','queue':'download','oldestWaitingMs':700000}]
return rows
seen=set()
for _ in range(16):
emit(failures());time.sleep(.5);seen.update(n for n,s in states().items() if s=='firing')
assert target <= seen,(target-seen,states())
print('PASS real vmalert firing for 5xx, restarts, memory, stalls, waiting age, webhook and interval drops',flush=True)
missing={'ApiTelemetryMissing','StreamerTelemetryMissing','QueueTelemetryMissing'}
wait(lambda: missing <= {n for n,s in states().items() if s=='firing'},20)
print('PASS real missing-heartbeat alerts fire',flush=True)
for _ in range(16):emit(healthy());time.sleep(.5)
assert all(s=='inactive' for s in states().values()),states()
print('PASS all log alerts recover; no external notifier configured',flush=True)
except Exception:
print(run(*cmd,'logs','--tail','25'),flush=True);raise
finally:run(*cmd,'down','-v','--remove-orphans')
metric_rules=yaml.safe_load((CONFIG/'metrics-alerts.yml').read_text())
(tmp/'metrics.yml').write_text(yaml.safe_dump(metric_rules))
series=[
{'series':'up{job="collector",instance="vector:9598"}','values':'0+0x6 1+0x14'},
{'series':'vector_buffer_discarded_events_total','values':'0+1x6 6+0x14'},
{'series':'vector_component_discarded_events_total{intentional="false"}','values':'0+1x6 6+0x14'},
{'series':'vector_component_errors_total','values':'0+1x6 6+0x14'},
{'series':'vl_data_size_bytes','values':str(9*1024**3)+'+0x6 1000+0x14'},
{'series':'vl_free_disk_space_bytes','values':str(10*1024**3)+'+0x21'},
{'series':'vl_rows_dropped_total','values':'0+1x6 6+0x14'},
{'series':'vmalert_execution_errors_total','values':'0+1x6 6+0x14'},
]
checks=[]
for group in metric_rules['groups']:
for rule in group['rules']:
labels=dict(rule['labels'])
if rule['alert']=='LoggingTelemetryMissing':labels.update(job='collector',instance='vector:9598')
checks += [{'eval_time':'6m','alertname':rule['alert'],'exp_alerts':[{'exp_labels':labels,'exp_annotations':rule['annotations']}]},
{'eval_time':'15m','alertname':rule['alert'],'exp_alerts':[]},
{'eval_time':'1m','alertname':rule['alert'],'exp_alerts':[]}]
tests={'rule_files':['/fixtures/metrics.yml'],'evaluation_interval':'1m','tests':[{'interval':'1m','input_series':series,'alert_rule_test':checks}]}
(tmp/'tests.yml').write_text(yaml.safe_dump(tests))
print(run('docker','run','--rm','--network','none','--entrypoint','/bin/promtool','-v',str(tmp)+':/fixtures:ro',images['prometheus']['image'],'test','rules','/fixtures/tests.yml'),flush=True)
print('PASS pinned Prometheus firing, sustained thresholds and recovery for every infrastructure alert',flush=True)
+6 -2
View File
@@ -1079,6 +1079,7 @@ export const LOG_SOURCE_FUNCTIONS: Record<string, readonly string[]> = {
"getRedis",
"hourKey",
"persistError",
"reportOutputDrops",
"resolveThreshold",
"serializeError",
"trimErrorLike",
@@ -1096,6 +1097,7 @@ export const LOG_SOURCE_FUNCTIONS: Record<string, readonly string[]> = {
"getRedis",
"hourKey",
"persistError",
"reportOutputDrops",
"resolveThreshold",
"serializeError",
"trimErrorLike",
@@ -2205,7 +2207,7 @@ export const LOG_SOURCE_FUNCTIONS: Record<string, readonly string[]> = {
"collector",
"endpoint",
"grafana",
"group",
"incidentGroups",
"investigateLogs",
"logOptions",
"range",
@@ -2213,6 +2215,7 @@ export const LOG_SOURCE_FUNCTIONS: Record<string, readonly string[]> = {
"rows",
"safeRedisLog",
"safeRow",
"signature",
"timestampKey"
],
"build/server/admin-log-investigation.js": [
@@ -2220,7 +2223,7 @@ export const LOG_SOURCE_FUNCTIONS: Record<string, readonly string[]> = {
"collector",
"endpoint",
"grafana",
"group",
"incidentGroups",
"investigateLogs",
"logOptions",
"range",
@@ -2228,6 +2231,7 @@ export const LOG_SOURCE_FUNCTIONS: Record<string, readonly string[]> = {
"rows",
"safeRedisLog",
"safeRow",
"signature",
"timestampKey"
],
"src/server/dailyStatsSnapshot.ts": [
+7 -7
View File
@@ -10,7 +10,7 @@ import { logContext, validTraceId } from "./log-context";
const instance = randomUUID();
const allowed = (values: readonly string[], value: unknown): value is string => typeof value === "string" && values.includes(value);
const events: Record<string, string> = {
...LOG_EVENTS,
...LOG_EVENTS, deployment: "deployment",
request: "request_completed", "slow request": "request_slow", "request interrupted": "request_cancelled",
"process fatal error": "process_fatal_error", "process started": "process_started",
"process heartbeat": "process_heartbeat", "job started": "job_started", "job completed": "job_completed",
@@ -18,19 +18,19 @@ const events: Record<string, string> = {
};
const errorCodes = new Set([
...ERROR_CODES,
"token_refresh_failed", "quota_exceeded", "github_rate_limit_exceeded", "github_app_reconnect_required", "github_token_invalid",
"repo_renamed", "preparation_interrupted", "commit_not_found", "zip_not_available", "token_refresh_failed", "quota_exceeded", "github_rate_limit_exceeded", "github_app_reconnect_required", "github_token_invalid",
"repository_expired", "repository_not_ready", "repository_not_found", "repository_job_cancelled",
"ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND", "EPIPE", "ABORT_ERR",
"upstream_error", "upstream_timeout", "invalid_signature", "webhook_processing_failed",
]);
export const LOG_NUMBERS = ["expectedBytes", "measuredMs", "unattributedMs", "overlapMs","status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond",
export const LOG_NUMBERS = ["restartMs", "exitCode","droppedLogsInterval","expectedBytes", "measuredMs", "unattributedMs", "overlapMs","status", "httpStatus", "ms", "firstByteMs", "responseMs", "bytes", "bytesPerSecond",
"attempt", "rss", "memoryLimitBytes", "memoryRatio", "uptime", "activeRequests", "waiting", "active", "delayed", "failed",
"oldestWaitingMs", "droppedLogs", "loopP95Ms", "upstreamStatus"];
export const LOG_STAGES = ["transfer", "database","repository", "credential_resolution", "upstream", "cache_lookup", "cache_fill", "worker_wait", "anonymize", "streamer_headers"];
export function logRecord(level: string, module: string, args: unknown[]) {
const context = logContext.getStore() || {};
const details: Record<string, unknown> = {};
const keys = [...LOG_NUMBERS, "requestId", "jobId", "hopId", "parentHopId", "cancellationOrigin", "cancellationEvidence", "endpointKind", "stageCounts", "stageOpen", "code", "message", "outcome", "method", "route", "queue",
const keys = [...LOG_NUMBERS, "requestId", "jobId", "hopId", "parentHopId", "deploymentId", "deploymentOperation", "deploymentPhase", "deploymentOutcome", "cancellationOrigin", "cancellationEvidence", "endpointKind", "stageCounts", "stageOpen", "code", "message", "outcome", "method", "route", "queue",
"errorType", "errorStack", "causeType", "causeStack", "requestPath", "stages", "reason", "phase", "event", "deliveryId", "hostname", "service"];
// Do not traverse arbitrary objects or invoke error getters while logging.
for (const arg of args.slice(0, 8)) if (arg && typeof arg === "object") {
@@ -50,8 +50,8 @@ export function logRecord(level: string, module: string, args: unknown[]) {
service: context.service || (allowed(["api", "streamer", "worker"], details.service) ? details.service : undefined) || (process.env.SERVICE_NAME?.toLowerCase() === "streamer" ? "streamer" : "api"),
instance, release: /^[a-f0-9]{7,40}$/.test(process.env.APP_RELEASE || "") ? process.env.APP_RELEASE : "unknown",
event, message: event, module: LOG_MODULES.includes(module) ? module : "application" };
for (const key of ["requestId", "jobId", "hopId", "parentHopId"] as const) {
const value = details[key] || context[key];
for (const key of ["requestId", "jobId", "hopId", "parentHopId", "deploymentId"] as const) {
const value = details[key] || (key === "deploymentId" ? undefined : context[key]);
if (validTraceId(value)) record[key] = value;
}
if (Number.isSafeInteger(context.attempt)) record.attempt = context.attempt;
@@ -64,7 +64,7 @@ export function logRecord(level: string, module: string, args: unknown[]) {
if (validTraceId(details.deliveryId)) record.deliveryId = details.deliveryId;
if (allowed(["github.com", "api.github.com", "codeload.github.com", "raw.githubusercontent.com", "media.githubusercontent.com"], details.hostname)) record.upstreamHost = details.hostname;
else if (details.hostname) record.upstreamHost = "other";
for (const [key, values] of Object.entries({ cancellationOrigin: ["unknown", "api", "streamer", "timeout", "upstream"],
for (const [key, values] of Object.entries({ deploymentOperation: ["deploy", "rollback"], deploymentPhase: ["start", "restart", "finish"], deploymentOutcome: ["started", "completed", "failed", "unchanged"], cancellationOrigin: ["unknown", "api", "streamer", "timeout", "upstream"],
cancellationEvidence: ["response_close", "request_aborted", "upstream_error", "timeout", "local_abort"], endpointKind: ["public", "internal"] })) {
if (allowed(values, details[key])) record[key] = details[key];
}
+20 -3
View File
@@ -90,9 +90,19 @@ let droppedOutput = 0;
let outputBroken = false;
let errorOutputBroken = false;
// A closed log reader must not become an uncaught EPIPE on the serving process.
process.stdout.on("error", () => { outputBroken = true; droppedOutput++; });
process.stderr.on("error", () => { errorOutputBroken = true; droppedOutput++; });
process.stdout.on("error", () => { outputBroken = true; });
process.stderr.on("error", () => { errorOutputBroken = true; });
export function getDroppedOutput() { return droppedOutput; }
let reportedDrops = 0;
let dropReportWatermark: number | undefined;
/** Advance only after the output stream acknowledges the report's write. */
export function reportOutputDrops(report: (fields: { droppedLogs: number; droppedLogsInterval: number }) => void) {
const before = droppedOutput;
const previous = dropReportWatermark;
dropReportWatermark = before;
try { report({ droppedLogs: before, droppedLogsInterval: Math.max(0, before - reportedDrops) }); }
finally { dropReportWatermark = previous; }
}
export function getInProcessDropped(): number {
return droppedInProcess;
}
@@ -275,7 +285,14 @@ function emit(level: Level, module: string, args: unknown[]) {
// building an unbounded userland queue. The next heartbeat reports losses.
const output = outputBroken ? process.stderr : process.stdout;
if (!(outputBroken && errorOutputBroken) && output.writableLength < 256 * 1024) {
try { output.write(JSON.stringify(record) + "\n"); }
const watermark = dropReportWatermark;
try { output.write(JSON.stringify(record) + "\n", error => {
if (error) {
if (output === process.stdout) outputBroken = true;
else errorOutputBroken = true;
droppedOutput++;
} else if (watermark !== undefined) reportedDrops = Math.max(reportedDrops, watermark);
}); }
catch {
if (output === process.stdout) outputBroken = true;
else errorOutputBroken = true;
+3 -3
View File
@@ -1,5 +1,5 @@
import { logMemory } from "./log-memory";
import { createLogger, serializeError, getDroppedOutput } from "./logger";
import { createLogger, serializeError, reportOutputDrops } from "./logger";
/** Observe fatal errors without overriding Node's default exit behavior. */
export function installFatalErrorLogging(service: "api" | "streamer") {
@@ -9,8 +9,8 @@ export function installFatalErrorLogging(service: "api" | "streamer") {
});
logger.info("process started", { service });
const heartbeat = setInterval(() => {
void logMemory().then(memory => logger.info("process heartbeat", {
service, uptime: process.uptime(), ...memory, droppedLogs: getDroppedOutput() }));
void logMemory().then(memory => reportOutputDrops(drops => logger.info("process heartbeat", {
service, uptime: process.uptime(), ...memory, ...drops })));
}, 30000);
heartbeat.unref();
process.on("uncaughtExceptionMonitor", report);
+4 -4
View File
@@ -2,7 +2,7 @@ import { logMemory } from "../core/log-memory";
import { randomUUID } from "crypto";
import { SandboxedJob } from "bullmq";
import { logContext, validTraceId } from "../core/log-context";
import { createLogger, getDroppedOutput } from "../core/logger";
import { createLogger, reportOutputDrops } from "../core/logger";
import type { RepoJobData } from "./index";
const logger = createLogger("queue");
export function loggedJob(queue: "download" | "remove" | "cache", work: (job: SandboxedJob<RepoJobData, void>) => Promise<void>) {
@@ -14,14 +14,14 @@ export function loggedJob(queue: "download" | "remove" | "cache", work: (job: Sa
logger.info("job started", { queue });
const started = Date.now();
const heartbeat = setInterval(() => {
void logMemory().then(memory => logger.info("process heartbeat", { queue, ...memory, uptime: process.uptime(), droppedLogs: getDroppedOutput() }));
void logMemory().then(memory => reportOutputDrops(drops => logger.info("process heartbeat", { queue, ...memory, uptime: process.uptime(), ...drops })));
}, 30000); heartbeat.unref();
try {
await work(job);
logger.info("job completed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() });
reportOutputDrops(drops => logger.info("job completed", { queue, ms: Date.now() - started, ...drops }));
} catch (error) {
const expected = error instanceof Error && ["DelayedError", "RateLimitDelayError"].includes(error.name);
logger[expected ? "info" : "error"]("job failed", { queue, ms: Date.now() - started, droppedLogs: getDroppedOutput() });
reportOutputDrops(drops => logger[expected ? "info" : "error"]("job failed", { queue, ms: Date.now() - started, ...drops }));
throw error;
} finally { clearInterval(heartbeat); }
});
+28 -10
View File
@@ -4,7 +4,7 @@ import { logContext, validTraceId } from "../core/log-context";
import { LOG_EVENTS } from "../core/log-schema";
const services = ["api", "streamer", "worker", "nginx", "mongodb", "redis", "backup"];
const events = new Set([...Object.values(LOG_EVENTS), "application_log", "webhook_log", "request_completed", "request_slow", "request_cancelled", "process_fatal_error", "process_started", "process_heartbeat", "job_started", "job_completed", "job_failed", "job_stalled", "queue_heartbeat", "edge_request", "infrastructure_log"]);
const events = new Set([...Object.values(LOG_EVENTS), "deployment", "application_log", "webhook_log", "request_completed", "request_slow", "request_cancelled", "process_fatal_error", "process_started", "process_heartbeat", "job_started", "job_completed", "job_failed", "job_stalled", "queue_heartbeat", "edge_request", "infrastructure_log"]);
const windows: Record<string, number> = { "15m": 15, "1h": 60, "6h": 360, "24h": 1440 };
let active = 0;
export class InvalidLogQuery extends Error {}
@@ -59,7 +59,7 @@ function safeRow(input: Record<string, unknown>): Row | null {
// Revalidate finite route and diagnostic allowlists; never pass raw strings.
const projected = logContext.run({}, () => logRecord(["error", "warn", "info", "debug"].includes(String(input.severity)) ? String(input.severity) : "info", String(input.module || "application"), ["", data]));
const result: Row = { timestamp: /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,9})?Z$/.test(time) ? time : new Date(time).toISOString(), severity: String(projected.severity), service: services.includes(String(input.service)) ? String(input.service) : "unknown", event: events.has(String(input.event)) ? String(input.event) : "application_log", module: String(projected.module), release: /^(?:[a-f0-9]{7,40}|unknown)$/.test(String(input.release)) ? String(input.release) : "unknown" };
for (const key of [...numeric, "outcome", "endpointKind", "cancellationOrigin", "cancellationEvidence", "hopId", "parentHopId", "errorCode", "requestId", "jobId", "queue", "route", "method", "errorType", "errorStack", "causeType", "causeStack", "repoId", "requestPath"]) if (typeof projected[key] === "string" || typeof projected[key] === "number") result[key] = projected[key] as string | number;
for (const key of [...numeric, "deploymentId", "deploymentOperation", "deploymentPhase", "deploymentOutcome", "outcome", "endpointKind", "cancellationOrigin", "cancellationEvidence", "hopId", "parentHopId", "errorCode", "requestId", "jobId", "queue", "route", "method", "errorType", "errorStack", "causeType", "causeStack", "repoId", "requestPath"]) if (typeof projected[key] === "string" || typeof projected[key] === "number") result[key] = projected[key] as string | number;
if (validTraceId(input.instance)) result.instance = input.instance;
return result;
}
@@ -88,6 +88,8 @@ function grafana(o: Options) {
if (!["http:", "https:"].includes(url.protocol) || url.username || url.password) return null;
url.search = ""; url.hash = "";
url.searchParams.set("from", String(Date.parse(o.start))); url.searchParams.set("to", String(Date.parse(o.end)));
if (o.service) url.searchParams.set("var-service", `^${o.service}$`);
if (o.release) url.searchParams.set("var-release", `^${o.release}$`);
if (o.id) url.searchParams.set(o.kind === "jobId" ? "var-job_id" : "var-request_id", o.id);
return url.toString();
} catch { return null; }
@@ -97,16 +99,28 @@ function grafana(o: Options) {
function timestampKey(value: string | number) {
return String(value).replace(/(?:\.(\d+))?Z$/, (_match, fraction = "") => `.${fraction.padEnd(9, "0")}Z`);
}
function group(records: Row[]) {
const groups = new Map<string, { fingerprint: string; count: number; first: string; last: string; sample: Row }>();
export function incidentGroups(records: Row[], baseline?: { records: Row[]; partial: boolean }) {
const signature = (row: Row) => JSON.stringify([row.service, row.module, row.event, row.errorCode || "", row.severity,
row.status || row.httpStatus || "", row.errorType || "", String(row.errorStack || "").split("\n")[0].replace(/:\d+:\d+\)$/, ")")]);
const known = new Set((baseline?.records || []).map(signature));
const groups = new Map<string, { fingerprint: string; count: number; first: string; last: string; sample: Row;
requests: Set<string>; uncorrelatedRecords: number; releases: Set<string>; regression: string }>();
for (const row of records) {
const key = JSON.stringify([row.service, row.module, row.event, row.errorCode || "", row.severity, row.status || row.httpStatus || "", row.errorType || "", String(row.errorStack || "").split("\n")[0]]);
const key = signature(row);
const fingerprint = createHash("sha256").update(key).digest("hex").slice(0, 24);
const existing = groups.get(fingerprint);
let existing = groups.get(fingerprint);
if (existing) { existing.count++; existing.first = timestampKey(existing.first) < timestampKey(row.timestamp) ? existing.first : String(row.timestamp); existing.last = timestampKey(existing.last) > timestampKey(row.timestamp) ? existing.last : String(row.timestamp); }
else groups.set(fingerprint, { fingerprint, count: 1, first: String(row.timestamp), last: String(row.timestamp), sample: row });
else {
existing = { fingerprint, count: 1, first: String(row.timestamp), last: String(row.timestamp), sample: row,
requests: new Set(), releases: new Set(), uncorrelatedRecords: 0,
regression: known.has(key) ? "known" : baseline && !baseline.partial && baseline.records.length ? "not_seen_in_baseline" : "unknown" };
groups.set(fingerprint, existing);
}
if (validTraceId(row.requestId)) existing.requests.add(row.requestId); else existing.uncorrelatedRecords++;
if (typeof row.release === "string") existing.releases.add(row.release);
}
return [...groups.values()].sort((a, b) => timestampKey(b.last).localeCompare(timestampKey(a.last)));
return [...groups.values()].map(({ requests, releases, ...value }) => ({ ...value, affectedRequests: requests.size, releases: [...releases] }))
.sort((a, b) => timestampKey(b.last).localeCompare(timestampKey(a.last)));
}
async function collector(signal: AbortSignal) {
try {
@@ -152,10 +166,13 @@ export async function investigateLogs(options: Options, correlation = false) {
// Historical collector output omits unknown releases and keeps edge 5xx at info.
const release = options.release === "unknown" ? ' (release:unknown OR release:"")' : options.release ? ` release:${options.release}` : "";
const filter = `${range(options)} (severity:error OR severity:warn OR (service:nginx event:edge_request status:>=500 status:<600))${options.service ? ` service:${options.service}` : ""}${release}`;
const [errors, heartbeat, drops] = await Promise.all([
const baselineStart = new Date(Date.parse(options.start) - 24 * 60 * 60 * 1000).toISOString();
const compareRelease = options.release && options.release !== "unknown";
const [errors, heartbeat, drops, baseline] = await Promise.all([
rows(base, `${filter} | sort by (_time) desc`, 1001, controller.signal),
rows(base, '_time:5m (event:process_heartbeat OR event:queue_heartbeat OR event:job_completed OR event:job_failed) | sort by (_time) desc', 501, controller.signal),
collector(controller.signal),
compareRelease ? rows(base, `_time:[${baselineStart}, ${options.start}) (severity:error OR severity:warn OR (service:nginx event:edge_request status:>=500 status:<600))${options.service ? ` service:${options.service}` : ""} NOT release:${options.release} NOT release:unknown NOT release:"" | sort by (_time) desc`, 1001, controller.signal).catch(() => ({ records: [], partial: true })) : Promise.resolve(undefined),
]);
const latest = new Map<string, Row>();
for (const row of heartbeat.records) {
@@ -167,7 +184,8 @@ export async function investigateLogs(options: Options, correlation = false) {
const queues = new Set(recent.filter(row => row.event === "queue_heartbeat").map(row => row.queue).filter(Boolean)).size;
const missing = [...(!recent.some(row => row.service === "api" && row.event === "process_heartbeat") ? ["API"] : []), ...(streamers < 4 ? ["streamers"] : []), ...(queues < 3 ? ["queues"] : [])];
const records = errors.records.slice(0, 1000);
return { ...common, state: errors.partial || heartbeat.partial ? "partial" : missing.length ? "stale" : records.length ? "ready" : "empty", groups: group(records), captured: records.length, limit: 1000, telemetry: { missing, streamers, queues, partial: heartbeat.partial, instances: [...latest.values()].map(row => ({ service: row.service, instance: row.instance, last: row.timestamp, dropped: row.droppedLogs ?? null })), collector: drops } };
return { ...common, state: errors.partial || heartbeat.partial ? "partial" : missing.length ? "stale" : records.length ? "ready" : "empty", groups: incidentGroups(records, baseline), baseline: baseline ? { start: baselineStart, end: options.start, captured: baseline.records.length,
partial: baseline.partial, comparison: "other_releases_previous_24h" } : null, captured: records.length, limit: 1000, telemetry: { missing, streamers, queues, partial: heartbeat.partial, instances: [...latest.values()].map(row => ({ service: row.service, instance: row.instance, last: row.timestamp, dropped: row.droppedLogs ?? null })), collector: drops } };
} catch { return { ...common, state: "unreachable", groups: [], events: [] }; }
finally { controller.abort(); clearTimeout(timer); active--; }
}
+16
View File
@@ -40,6 +40,22 @@ describe('bounded admin log investigation', function () {
expect(calls[0][1].body.get('limit')).to.equal('1001');
expect(calls[0][1].body.get('query')).to.include('service:worker release:abcdef1');
});
for (const failure of ['oversized', 'unavailable']) it(`keeps current incidents when the optional baseline is ${failure}`, async () => {
process.env.ADMIN_LOG_URL = 'http://logging.test';
global.fetch = async (url, options) => {
const query=options.body.get('query');
if (query.includes('NOT release:')) {
if (failure === 'unavailable') throw new Error('backend unavailable');
return new Response((JSON.stringify(event({release:'abcdef2'}))+'\n').repeat(10000));
}
return new Response(query.includes('severity:error') ? JSON.stringify(event({})) : '');
};
const result=await investigateLogs(logOptions({release:'abcdef1'}));
expect(result.state).equal('stale');
expect(result.groups).length(1);
expect(result.groups[0]).include({count:1,regression:'unknown'});
expect(result.baseline).include({partial:true,captured:0});
});
it('sanitizes legacy Redis history and does not invent correlation IDs', () => {
const row = logContext.run({requestId:id}, () => safeRedisLog({ts:new Date().toISOString(),level:'error',module:'route',message:'secret',raw:[{message:'invalid_request',url:'private',stack:'secret',cookies:'secret',status:400}]}));
expect(row.message).to.equal('invalid_request'); expect(JSON.stringify(row)).not.to.match(/private|secret|requestId/);
+25 -3
View File
@@ -25,10 +25,10 @@ if (args[0] === "inspect") {
if (args.some(a => a.includes("Config.Env"))) finish(state.sharedCache ? "SHARED_CACHE_ENABLED=true\\n" : "");
if (args.some(a => a.includes("Labels"))) finish(state.oldProtocol || "<no value>");
if (args.includes("{{.State.Running}}")) finish(state.stopped && !state.refuseStop ? "false" : "true");
finish("healthy");
finish(state.unhealthy ? "unhealthy" : "healthy");
}
if (args[0] === "compose") {
if (args[1] === "build") { if (state.buildFails) finish("", 1); state.built = true; finish(); }
if (args[1] === "build") { if (state.signal) { process.kill(process.ppid, state.signal); finish(); } if (state.buildFails) finish("", 1); state.built = true; finish(); }
if (args[1] === "stop") { state.stopped = true; finish(); }
if (args[1] === "run" && args.at(-1).endsWith("check-github-app-key.js")) { finish("", args.includes(state.preflightFails) ? 1 : 0); }
if (args[1] === "run") { if (state.reconcileFails) finish("", 1); state.reconciled = true; finish(); }
@@ -45,12 +45,34 @@ finish("unexpected docker call", 2);
const result = spawnSync("bash", [path.join(directory, "scripts/deploy.sh")], {
encoding: "utf8", timeout: 8000,
env: { ...process.env, PATH: directory + path.delimiter + process.env.PATH,
COORDINATED_RESTART: String(options.coordinated ?? 0), FORCE: String(options.force ?? 1), STREAMER_REPLICAS: "4", TASK_DOCKER_STATE: statePath, TASK_DOCKER_LOG: logPath },
APP_RELEASE: "abcdef1", DEPLOY_OPERATION: options.operation || "deploy", HEALTH_TIMEOUT: options.unhealthy ? "1" : "90", MIGRATION_HEALTH_TIMEOUT: options.unhealthy ? "1" : "900", COORDINATED_RESTART: String(options.coordinated ?? 0), FORCE: String(options.force ?? 1), STREAMER_REPLICAS: "4", TASK_DOCKER_STATE: statePath, TASK_DOCKER_LOG: logPath },
});
return { ...result, calls: fs.readFileSync(logPath, "utf8").trim().split("\n").map(line => JSON.parse(line)) };
} finally { fs.rmSync(directory, { recursive: true, force: true }); }
}
it("records finite deployment start, restart and rollback outcome", () => {
const result = deploy({operation:"rollback"});
expect(result.status, result.stderr).equal(0);
const events = result.stdout.split("\n").filter(line=>line.startsWith("{")).map(JSON.parse);
expect(events.map(row=>row.deploymentPhase)).deep.equal(["start","restart","finish"]);
expect(events.at(-1)).include({deploymentOperation:"rollback",deploymentOutcome:"completed",release:"abcdef1",exitCode:0});
expect(new Set(events.map(row=>row.deploymentId)).size).equal(1);
expect(events.at(-1).restartMs).at.least(0);
});
for (const [signal, code] of [["SIGHUP",129],["SIGINT",130],["SIGTERM",143]]) it(`records ${signal} termination as failed`, () => {
const result=deploy({signal,operation:"rollback"});
expect(result.status,result.stderr).equal(code);
const events=result.stdout.split("\n").filter(line=>line.startsWith("{")).map(JSON.parse);
expect(events.at(-1)).include({deploymentPhase:"finish",deploymentOutcome:"failed",exitCode:code});
});
it("records failure when API or streamers fail health checks", () => {
const result=deploy({unhealthy:true});
expect(result.status).not.equal(0);
const events=result.stdout.split("\n").filter(line=>line.startsWith("{")).map(JSON.parse);
expect(events.at(-1)).include({deploymentPhase:"finish",deploymentOutcome:"failed"});
expect(events.at(-1).exitCode).not.equal(0);
});
for (const preflightFails of ["anonymous_github", "streamer"]) it(`keeps working containers when ${preflightFails} key preflight fails`, () => {
const result = deploy({ preflightFails });
expect(result.status).not.to.equal(0);
+20
View File
@@ -0,0 +1,20 @@
const {expect}=require('chai');
const {randomUUID}=require('node:crypto');
require('ts-node/register/transpile-only');
const {incidentGroups}=require('../src/server/admin-log-investigation');
describe('bounded incident evidence',()=>{
const base={timestamp:'2026-11-15T10:00:00Z',service:'api',module:'route',event:'application_log',severity:'error',errorCode:'upstream_error',errorType:'Error',httpStatus:502,release:'abcdef1',errorStack:'at printError (src/server/routes/route-utils.ts:200:7)'};
it('counts affected request UUIDs once without inventing identities for historical records',()=>{
const id=randomUUID();const groups=incidentGroups([base,{...base,requestId:id},{...base,requestId:id},{...base,requestId:randomUUID(),timestamp:'2026-11-15T11:00:00Z'}]);
expect(groups).length(1);expect(groups[0]).include({count:4,affectedRequests:2,uncorrelatedRecords:1,first:base.timestamp,last:'2026-11-15T11:00:00Z',regression:'unknown'});
});
it('compares stable stack functions across line shifts, retaining unknown for missing/partial baselines',()=>{
const shifted={...base,errorStack:'at printError (src/server/routes/route-utils.ts:220:9)',release:'abcdef2'};
expect(incidentGroups([shifted],{records:[base],partial:true})[0].regression).equal('known');
expect(incidentGroups([shifted],{records:[],partial:false})[0].regression).equal('unknown');
const different={...base,errorCode:'preparation_interrupted'};
expect(incidentGroups([different],{records:[base],partial:true})[0].regression).equal('unknown');
expect(incidentGroups([different],{records:[base],partial:false})[0].regression).equal('not_seen_in_baseline');
expect(incidentGroups([base,different])).length(2);
});
});
+33
View File
@@ -111,6 +111,39 @@ describe('structured logging privacy and context', function () {
`],{env:{...process.env,LOG_REDIS_MIRROR:'on'},encoding:'utf8',timeout:10000});
expect(child.status,child.stderr).equal(0);
});
it('reports interval drops once and retries a dropped report without resetting history', () => {
const child=spawnSync(process.execPath,['-r','ts-node/register/transpile-only','-e',`
const assert=require('node:assert/strict');
const {createLogger,reportOutputDrops}=require('./src/core/logger');
const logger=createLogger('process');let blocked=true;
Object.defineProperty(process.stdout,'writableLength',{get:()=>blocked?300000:0});
logger.info('request');logger.info('request');
reportOutputDrops(fields=>logger.info('process heartbeat',fields));
blocked=false;
reportOutputDrops(fields=>{assert.equal(fields.droppedLogsInterval,3);logger.info('process heartbeat',fields);});
setImmediate(()=>reportOutputDrops(fields=>{assert.equal(fields.droppedLogsInterval,0);assert.equal(fields.droppedLogs,3);logger.info('process heartbeat',fields);}));
`],{env:{...process.env,LOG_REDIS_MIRROR:'off'},encoding:'utf8',timeout:10000});
expect(child.status,child.stderr).equal(0);
const rows=child.stdout.trim().split('\n').map(JSON.parse);
expect(rows.map(r=>r.droppedLogsInterval)).deep.equal([3,0]);
});
it('retains earlier losses when a report write fails asynchronously', () => {
const child=spawnSync(process.execPath,['-r','ts-node/register/transpile-only','-e',`
const assert=require('node:assert/strict');
const {createLogger,reportOutputDrops}=require('./src/core/logger');
const logger=createLogger('process');let blocked=true;
Object.defineProperty(process.stdout,'writableLength',{get:()=>blocked?300000:0});
logger.info('request');logger.info('request');blocked=false;
process.stdout.write=(line,callback)=>{setImmediate(()=>callback(new Error('EPIPE')));return true;};
reportOutputDrops(fields=>logger.info('process heartbeat',fields));
setImmediate(()=>{
reportOutputDrops(fields=>{assert.equal(fields.droppedLogsInterval,3);logger.info('process heartbeat',fields);});
setImmediate(()=>reportOutputDrops(fields=>{assert.equal(fields.droppedLogsInterval,0);logger.info('process heartbeat',fields);}));
});
`],{env:{...process.env,LOG_REDIS_MIRROR:'off'},encoding:'utf8',timeout:10000});
expect(child.status,child.stderr).equal(0);
expect(child.stderr.trim().split('\n').filter(line=>line.startsWith('{')).map(JSON.parse).map(r=>r.droppedLogsInterval)).deep.equal([3,0]);
});
it('writes JSON without Redis and bounds a blocked stdout pipe', () => {
const child = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', `
const { createLogger, getDroppedOutput } = require('./src/core/logger');