anonymous_github/docs/performance-monitoring.md

# Request performance monitoring

The admin overview shows API and streamer request latency, interrupted responses,
first-byte latency, work stages and process resources. It uses the existing Redis
service. `/api/admin/performance?minutes=15` returns the same data after the existing
admin authentication and authorization checks. The other supported window is 60
minutes. Responses are private and never HTTP-cached.

Request measurement starts before body parsing, authentication and rate limiting.
Each response is counted once on completion or premature close. A response that
sent HTTP 200 and then disconnected is reported as interrupted. Completion means
Node handed the response to the operating system; it does not prove client receipt.
First-byte timing measures the first response headers, including explicit header
flushes. Byte counts measure encoded response body writes and exclude HTTP headers.

## Latency and diagnosis

The latency histograms use millisecond boundaries of 10, 50, 100, 250, 500, 1000,
2500, 5000, 10000, 30000 and 60000, plus an overflow bucket. Displayed p95 and p99
values are bucket upper bounds, not exact percentiles. No samples or a percentile
above 60 seconds produces `null`; the UI distinguishes these cases using count.
Downloads include body transfer time, so compare their first-byte and total latency
before treating a long transfer as slow processing.

Slow requests produce structured warning logs above one second for ordinary API
calls and above ten seconds for file, website and streamer transfers. Interrupted
requests also produce warnings. These thresholds identify requests for diagnosis;
they do not automatically page anyone. Healthcheck timings are counted separately
and successful healthchecks do not produce access logs.

Logs contain a generated request ID, a fixed route label, method, status, terminal
outcome, duration, first-byte time, body bytes and work stages. API calls forward
the request ID to streamers. Repository IDs, source names, file paths, bodies and
query values are absent from these new request records and metric labels.

Stages include repository lookup, authorization, GitHub requests, source-cache
lookup and fill, worker admission, worker anonymization and streamer response
headers. Source and transformed-cache hits and misses are counted separately.
Source download timing includes waiting for response headers and failures before
streaming begins.
Stages can overlap and must not be summed to reconstruct total latency. An archive
upstream span also includes downstream backpressure; it is not pure network time.
Work outside HTTP requests, such as background download jobs, stays in the existing
queue monitoring rather than these request histograms.

## Resources, retention and overhead

Each process batches at most 512 metric groups every 15 seconds. Minute counters
expire after two hours and survive application rollouts. Read windows are restricted
to 15 or 60 minutes, with a ten-second cache and at most 32 live instances. Redis
commands have a two-second monitoring deadline and offline command queuing is
disabled. Monitoring never awaits Redis from an HTTP request handler, except the
admin report itself. Failures discard a bounded batch and show unavailable reports
or dropped-batch counters instead of silently implying healthy coverage.

Samples include process RSS, heap, external buffers, CPU usage, cgroup memory
limit, event-loop p95 and maximum delay, active requests, PID-owned sockets and
CLOSE_WAIT states, and worker activity, queue and byte reservations. Samples expire
after 90 seconds. The UI highlights samples older than 45 seconds, dropped metrics,
CLOSE_WAIT, event-loop p95 above 100 ms and RSS above 80 percent of its limit.
CPU usage is relative to one CPU and can exceed 100 percent across multiple cores.
RSS is process memory, not Docker's accounting of the entire cgroup.

A bounded runtime history keeps 240 API samples and 960 streamer samples, enough
for roughly one hour at one API and four streamers. The response includes this
history and the UI shows each live process's peak observed RSS in the selected
window. Additional replicas or rollout overlap shorten the history. Retired
processes disappear from the live table while retained request counters and
resource history remain available. Missing samples do not establish health.

Application Docker logs rotate at 25 MB with four files per container. The admin
overview's disk check runs asynchronously and caches its result for 30 seconds.
These settings avoid making monitoring itself block the Node event loop or fill
the host disk with unbounded access logs.

## Performance changes

ZIP HEAD requests check repository access and download permission, then return
headers without requesting a token, counting a view or generating an archive.
File and website HEAD requests validate metadata and content gates before ending
without content download or transformation. File validators accept weak and
multiple If-None-Match values using Express freshness handling.

Website responses use private, revalidated ETags covering content SHA, commit,
tree generation, cache revision, file path, rendering version and current options.
Validation follows repository lifecycle, page settings and content checks. Matching
validators avoid fetching and rendering content again; changed terms or content
invalidate them.
Raw Markdown served after a rendering failure has a separate validator, so later
requests retry rendering when the transient failure clears.

## Production failures and restarts

Run `node scripts/check-production-logs.js 15` from a machine with Node and the
Docker CLI to summarize recent application logs and container states. For a
remote Docker host, use
`DOCKER_HOST=ssh://ubuntu@host node scripts/check-production-logs.js 15`.
The optional
window is 1–1440 minutes. The report includes completed 5xx responses, interrupted
requests, token-refresh failures, missing-file warnings, native unhandled stream
errors, and Docker restart counts. It prints counts and container metadata;
request URLs, repository IDs, file paths, credentials, and Docker environment
variables are excluded. Restart counts cover the container's lifetime, while log
counts cover the selected window. Compare restart counts with the last deployment
baseline even when every container is currently healthy.

The API and streamers log `process_fatal_error` through Node's
`uncaughtExceptionMonitor`. This records fatal errors before the default process
exit; it does not resume execution or swallow uncaught exceptions. Native stderr
may describe the same crash as well, so fatal log indicators are not a count of
unique process exits. Use Docker restart counts to confirm exits.

Legacy OAuth reset requests are coalesced by owner, credential, and OAuth app.
A failed 401/404 reset backs off for 30 minutes; transient failures back off for
one minute. A replaced credential or changed OAuth configuration gets a new key.
GitHub App and personal access tokens do not use the OAuth reset endpoint. Refresh
failure leaves the credential unchanged and continues through normal token
validation and fallback.

Confirmed GitHub content 404s back off for 30 seconds in a bounded process-local
cache. The key includes repository, path, commit, blob SHA, cache generation,
revision, and credential. Permission failures and transport errors are not cached.
This reduces repeated upstream misses without changing the file tree or treating
an unavailable file as a successful cache fill. Persistent 404s still require
checking the repository's source snapshot. Application rate limits remain in
force; 429 replies now include `rate_limited`, `resetAt`, and `Retry-After` so
clients can wait for the actual store reset.