mirror of
https://github.com/tdurieux/anonymous_github.git
synced 2026-11-17 17:44:09 +01:00
fix: make buffer losses observable and preserve admin error codes
This commit is contained in:
1 parent
7e2a6e9d75
commit
f084f5cbfc
8 files changed
+90
-31
No files matched your search
@@ -102,7 +102,10 @@ for this stack and alert at 8 GiB used or under 8 GiB free. Keep application dat
|
||||
on a different filesystem for protection from a logging flood. At the free-space
|
||||
reserve, ingest stops and Vector queues sanitized records.
|
||||
|
||||
Vector's persistent disk buffer is 256 MiB, with `drop_newest`. Its files and
|
||||
Vector's persistent disk buffer is 256 MiB, followed by a 1,000-event memory
|
||||
overflow buffer with `drop_newest`. The memory stage makes full-buffer losses
|
||||
observable in Vector 0.61.0; its disk-only drop path omits fullness-loss counters.
|
||||
The overflow stage is volatile and can deliver records out of order. Disk files and
|
||||
checkpoints survive collector restarts. Allocation and metadata add overhead;
|
||||
budget 512 MiB on disk. Outages cause retries with one in-flight request. Once
|
||||
the buffer fills, new records are lost; applications do not wait. Vector syncs
|
||||
@@ -195,9 +198,23 @@ privacy, correlation, query syntax, rotation, restart, backend outage and recove
|
||||
retention rejection and fixture storage size. It never reads production logs or
|
||||
uses the application `.env`. The optional `--stress` writes 500,000 synthetic records while the backend is
|
||||
stopped, checks the buffer bound and collector liveness, and waits for the real
|
||||
CollectorDrops alert to fire. Allow 1 GiB temporary disk and several minutes.
|
||||
CollectorDrops alert to fire. Allow 1 GiB temporary disk and up to 15 minutes with the 0.5 CPU collector limit.
|
||||
These fixtures do not establish sustained production throughput.
|
||||
|
||||
Local results on 2026-11-14: application build, TypeScript and lint passed. The full
|
||||
suite had 1,656 passing, 441 skipped and five failures also reproduced on base
|
||||
`b456ab8`: one missing `cache_miss` translation, three activity-date mocks missing
|
||||
`hint()`, and one existing Vue split-preview expectation. A focused logging and
|
||||
job-reconnection run passed 34 tests. Skipped database tests were not executed.
|
||||
The synthetic 10,000-record batch contained 2,070,000 raw JSON bytes; the entire
|
||||
VictoriaLogs fixture store occupied about 357–358 KB of data and indexes,
|
||||
including earlier records. This is a fixture measurement, not a production ratio.
|
||||
The backend-outage stress run submitted 500,000 records, observed 136,532,570
|
||||
buffered bytes without collector restart/OOM, and verified that `CollectorDrops`
|
||||
reached firing after its sustained two-minute threshold. The overflow stage
|
||||
reported its losses. A separate real query verified terminal worker drop counts
|
||||
in the dashboard.
|
||||
|
||||
Official references checked 2026-11-14:
|
||||
|
||||
- [VictoriaLogs retention and resource flags](https://docs.victoriametrics.com/victorialogs/)
|
||||
@@ -206,3 +223,5 @@ Official references checked 2026-11-14:
|
||||
- [VictoriaLogs metrics](https://docs.victoriametrics.com/victorialogs/metrics/)
|
||||
- [vmalert LogsQL integration](https://docs.victoriametrics.com/victorialogs/vmalert/)
|
||||
- [Grafana plugin provisioning](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/)
|
||||
|
||||
- [Pinned Vector buffer sender accounting](https://github.com/vectordotdev/vector/blob/v0.61.0/lib/vector-buffers/src/topology/channel/sender.rs)
|
||||
@@ -284,7 +284,7 @@
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "event:process_heartbeat | stats by (service, instance) max(rss) as rss, max(memoryRatio) as memory_ratio, max(droppedLogs) as dropped_logs",
|
||||
"expr": "(event:process_heartbeat OR event:job_completed OR event:job_failed) | stats by (service, instance) max(rss) as rss, max(memoryRatio) as memory_ratio, max(droppedLogs) as dropped_logs",
|
||||
"queryType": "instant",
|
||||
"editorMode": "code",
|
||||
"maxLines": 200
|
||||
|
||||
@@ -39,10 +39,15 @@ sinks:
|
||||
_stream_fields: service,severity
|
||||
batch: {max_bytes: 1048576, timeout_secs: 1}
|
||||
request: {timeout_secs: 5, concurrency: 1}
|
||||
# Disk fullness drops lack accounting in Vector 0.61.0. Overflow into a
|
||||
# bounded memory stage whose drop_newest losses have observable counters.
|
||||
buffer:
|
||||
type: disk
|
||||
max_size: 268435488
|
||||
when_full: drop_newest
|
||||
- type: disk
|
||||
max_size: 268435488
|
||||
when_full: overflow
|
||||
- type: memory
|
||||
max_events: 1000
|
||||
when_full: drop_newest
|
||||
metrics:
|
||||
type: prometheus_exporter
|
||||
inputs: [internal]
|
||||
|
||||
Reference in new issue
Block a user