anonymous_github/docs/performance-monitoring.md

Request performance monitoring

The admin overview shows API and streamer request latency, interrupted responses, first-byte latency, work stages and process resources. It uses the existing Redis service. /api/admin/performance?minutes=15 returns the same data after the existing admin authentication and authorization checks. The other supported window is 60 minutes. Responses are private and never HTTP-cached.

Request measurement starts before body parsing, authentication and rate limiting. Each response is counted once on completion or premature close. A response that sent HTTP 200 and then disconnected is reported as interrupted. Completion means Node handed the response to the operating system; it does not prove client receipt. First-byte timing measures the first response headers, including explicit header flushes. Byte counts measure encoded response body writes and exclude HTTP headers.

Latency and diagnosis

The latency histograms use millisecond boundaries of 10, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000 and 60000, plus an overflow bucket. Displayed p95 and p99 values are bucket upper bounds, not exact percentiles. No samples or a percentile above 60 seconds produces null; the UI distinguishes these cases using count. Downloads include body transfer time, so compare their first-byte and total latency before treating a long transfer as slow processing.

Slow requests produce structured warning logs above one second for ordinary API calls and above ten seconds for file, website and streamer transfers. Interrupted requests also produce warnings. These thresholds identify requests for diagnosis; they do not automatically page anyone. Healthcheck timings are counted separately and successful healthchecks do not produce access logs.

Logs contain a generated request ID, a fixed route label, method, status, terminal outcome, duration, first-byte time, body bytes and work stages. API calls forward the request ID to streamers. Repository IDs, source names, file paths, bodies and query values are absent from these new request records and metric labels.

Stages include repository lookup, authorization, GitHub requests, source-cache lookup and fill, worker admission, worker anonymization and streamer response headers. Source and transformed-cache hits and misses are counted separately. Source download timing includes waiting for response headers and failures before streaming begins. Stages can overlap and must not be summed to reconstruct total latency. An archive upstream span also includes downstream backpressure; it is not pure network time. Work outside HTTP requests, such as background download jobs, stays in the existing queue monitoring rather than these request histograms.

Resources, retention and overhead

Each process batches at most 512 metric groups every 15 seconds. Minute counters expire after two hours and survive application rollouts. Read windows are restricted to 15 or 60 minutes, with a ten-second cache and at most 32 live instances. Redis commands have a two-second monitoring deadline and offline command queuing is disabled. Monitoring never awaits Redis from an HTTP request handler, except the admin report itself. Failures discard a bounded batch and show unavailable reports or dropped-batch counters instead of silently implying healthy coverage.

Samples include process RSS, heap, external buffers, CPU usage, cgroup memory limit, event-loop p95 and maximum delay, active requests, PID-owned sockets and CLOSE_WAIT states, and worker activity, queue and byte reservations. Samples expire after 90 seconds. The UI highlights samples older than 45 seconds, dropped metrics, CLOSE_WAIT, event-loop p95 above 100 ms and RSS above 80 percent of its limit. CPU usage is relative to one CPU and can exceed 100 percent across multiple cores. RSS is process memory, not Docker's accounting of the entire cgroup.

A bounded runtime history keeps 240 API samples and 960 streamer samples, enough for roughly one hour at one API and four streamers. The response includes this history and the UI shows each live process's peak observed RSS in the selected window. Additional replicas or rollout overlap shorten the history. Retired processes disappear from the live table while retained request counters and resource history remain available. Missing samples do not establish health.

Application Docker logs rotate at 25 MB with four files per container. The admin overview's disk check runs asynchronously and caches its result for 30 seconds. These settings avoid making monitoring itself block the Node event loop or fill the host disk with unbounded access logs.

Performance changes

ZIP HEAD requests check repository access and download permission, then return headers without requesting a token, counting a view or generating an archive. File and website HEAD requests validate metadata and content gates before ending without content download or transformation. File validators accept weak and multiple If-None-Match values using Express freshness handling.

Website responses use private, revalidated ETags covering content SHA, commit, tree generation, cache revision, file path, rendering version and current options. Validation follows repository lifecycle, page settings and content checks. Matching validators avoid fetching and rendering content again; changed terms or content invalidate them. Raw Markdown served after a rendering failure has a separate validator, so later requests retry rendering when the transient failure clears.