feat(cli): keep a log journal on disk with file headers and an upload limit

ctrld keeps a second log file, ctrld-journal.log, next to ctrld.log. The
journal holds all warnings and errors and each event with the field
journal=true. It stays on disk after a start of the service and after a
self-upgrade. Each log file starts with a Log header line. Each log send
upload starts with the same line, made at send time, and has a limit of
about 16 MB.

docs/config.md

@@ -121,10 +121,28 @@ Logging level you wish to enable.
 ### log_path
 Relative or absolute path of the log file. 
 
+Point `log_path` at a file that no other program rotates. ctrld renames `<log_path>.1` to `<log_path>.N` and uploads these files. A lower `log_max_backups` leaves the older numbered `log_path` files in place, but ctrld removes the extra numbered files of its internal logs.
+
 - Type: string
 - Required: no
 - Default: ""
 
+### log_max_size_mb
+Maximum size of the debug log file in MB. When the file reaches this size, ctrld rotates it. This limit also applies to the `log_path` file.
+
+- Type: integer
+- Required: no
+- Valid values: 1 to 1024
+- Default: 10 on desktop platforms, 5 on routers
+
+### log_max_backups
+Number of rotated debug log files that ctrld keeps. If the value is 0, ctrld keeps no rotated file. This number also applies to the `log_path` file.
+
+- Type: integer
+- Required: no
+- Valid values: 0 to 64
+- Default: 4 on desktop platforms, 1 on routers
+
 ### cache_enable
 When `cache_enable = true`, all resolved DNS query responses will be cached for duration of the upstream record TTLs.
 

docs/internal-domains.md

@@ -153,7 +153,10 @@ removed domain leaves no rule and no `internal_*` upstream behind.
 Generated `internal_*` upstreams carry the organization's private resolver
 addresses, so anything naming one stays at debug, and the level above it carries
 only a classification (`timeout`, `refused`, `unreachable`, `source_unavailable`,
-`error`). That covers every consumer, not just query handling:
+`error`). The per-query error sampler applies to the classification line as
+well. After 5 lines in one minute for the same upstream and class, the next
+classification lines go to debug, and one `Per-query errors sampled` line gives
+the count. That covers every consumer, not just query handling:
 
 - the bootstrap-IP line in `setupUpstream`;
 - the transport error from a failed query, which names the endpoint;

docs/network-recovery-diagnostics.md

@@ -1,9 +1,12 @@
 # Network-recovery diagnostics
 
-Use continuous `ctrld log tail` capture during a reproduction. Internal logs rotate by size.
-A later `log send` can contain recent debug events and older warnings.
-Match timestamps and IDs. The last retained error does not necessarily describe the current network.
-Issue #604 tracks upload-time device and version headers separately.
+The retained stream is the journal, the file `ctrld-journal.log` in the ctrld home directory.
+It keeps every warning and every error, plus every event marked `journal=true`.
+The journal is on disk, so it keeps these events across a restart and a self-upgrade.
+Its budget is 2 MB with 2 backup files on a desktop, and 1 MB with 1 backup file on a router.
+Every `ctrld log send` upload holds the whole journal, so a reproduction does not need a continuous `ctrld log tail` capture.
+Match timestamps and IDs. The last retained error does not describe the current network in every case.
+Each upload starts with a header line that names the build, the intercept mode, and the network at send time.
 
 ## Transitions and recovery
 
@@ -16,10 +19,10 @@ For minor callbacks and late major callbacks in that epoch, ctrld reads current
 This read does not change host DNS. A failed read defers source writes, but reconciliation continues.
 
 `Removed stale resolver source` is a warning only when ctrld removes an invalid address.
-The warning gives the reason and survives debug rotation.
+The warning gives the reason. Every warning enters the journal, so it stays on disk after a debug rotation and after a restart.
 `Recovery begin` and `Recovery end` connect the transition to a `recovery_generation`, reason, outcome, and elapsed time.
 Only the first upstream failure in each recovery produces a bounded summary.
-If a recovery encounters a failure, its final outcome also enters the warning stream.
+If a recovery encounters a failure, its final outcome also enters the journal.
 Normal successful recovery stays at debug level.
 
 These summaries exclude upstream URLs and queried names.
@@ -43,9 +46,9 @@ The cancellation event alone does not prove that interface DNS remains removed.
 
 A changed unsuccessful condition produces a warning. The condition consists of the stage, error code, outcome, and target.
 Equal results stay at debug level. The next changed condition includes the preceding repeat count.
-The first restored receipt also produces a warning-stream event. This event prevents an older failure from appearing current.
+The first restored receipt also produces a warning, which enters the journal. This event prevents an older failure from appearing current.
 Normal receipts and shutdown cancellation stay at debug level.
-The warning buffer is finite. These events do not guarantee indefinite retention.
+The journal budget is finite and the journal rotates by size. It does not keep these events without a limit.
 
 The outcomes have these meanings:
 

docs/runtime-internal-logging.md

@@ -1,46 +1,94 @@
-# Runtime Internal Logging
-
-When no logging is configured (i.e., `log_path` is not set), ctrld automatically enables an internal logging system. This system stores logs in memory to provide troubleshooting information when problems occur.
+# Runtime internal logging
 
 ## Purpose
 
-The runtime internal logging system is designed primarily for **ctrld developers**, not end users. It captures detailed diagnostic information that can be useful for troubleshooting issues when they arise, especially in production environments where explicit logging may not be configured.
+ctrld writes its own log when the configuration sets no log file. The log gives Control D support and ctrld developers the evidence they need after a problem. Two of the three streams are files on disk. They keep their content across a restart and a self-upgrade.
 
-## When It's Enabled
+## When it is on
 
-Internal logging is automatically enabled when:
+Internal logging is on when these three conditions are true at the same time:
 
-- ctrld is running in Control D mode (i.e., `--cd` flag is provided)
-- No log file is configured (i.e., `log_path` is empty or not set)
+- ctrld runs in Control D mode, with the `--cd` flag
+- The configuration sets no `log_path`
+- Silent mode is off
 
-If a log file is explicitly configured via `log_path`, internal logging will **not** be enabled, as the configured log file serves the logging purpose.
+When `log_path` is set, ctrld writes that file and starts no internal stream. The journal needs internal logging, so `log_path` mode has no journal.
 
-## How It Works
+## Streams and files
 
-The internal logging system:
+ctrld writes three streams. The debug file and the journal file live in the ctrld home directory, next to each other.
 
-- Stores logs in **in-memory buffers** (not written to disk)
-- Captures logs at **debug level** for normal operations and **warn level** for warnings
-- Maintains separate buffers for normal logs and warning logs
-- Automatically manages buffer size to prevent unbounded memory growth
-- Preserves initialization logs even when buffers overflow
+| Stream | Level filter | File | Desktop budget | Router budget | Survives a restart |
+|---|---|---|---|---|---|
+| Console | Notice and above. The `-v` flag adds the lower levels | none | none | none | No |
+| Debug | Every level | `ctrld.log` | 10 MB, 4 backup files | 5 MB, 1 backup file | Yes |
+| Journal | Warning and above, plus every event with `journal=true` | `ctrld-journal.log` | 2 MB, 2 backup files | 1 MB, 1 backup file | Yes |
 
-## Configuration
+The `-v` flag changes the level of every stream, the files included. Without the flag, ctrld sets the level to debug and holds the console at notice. With `-v` the level is info, so no debug line reaches the debug file. With `-vv` the level is debug again, and the console shows every line.
 
-**Important**: The `log_level` configuration option does **not** affect the internal logging system. Internal logging always operates at debug level for normal logs and warn level for warnings, regardless of the `log_level` setting in the configuration file.
+A backup file carries a number suffix. On a desktop the debug backup files are `ctrld.log.1` to `ctrld.log.4`, and `ctrld.log.1` holds the newest rotated content. The journal backup files are `ctrld-journal.log.1` and `ctrld-journal.log.2`.
 
-The `log_level` setting only affects:
-- Console output (when running interactively)
-- File-based logging (when `log_path` is configured)
+Rotation is by size. A write that crosses the size limit rotates the file first. ctrld deletes the file beyond the backup count.
 
-## Accessing Internal Logs
+Each stream also keeps a memory buffer: 5 MB for the debug stream and 1 MB for the journal. These buffers serve `ctrld log tail` and the fallback path when a file cannot open. The files hold the history.
 
-Internal logs can be accessed through the control server API endpoints. This functionality is intended for developers and support personnel who need to diagnose issues.
+A restart deletes the memory buffers and keeps the files. The new process appends a header line, which marks the restart point inside each file.
+
+## Header and rotation events
+
+Every debug file and every journal file starts with one JSON line, the header. ctrld writes the header when it opens a file at start and after every rotation. On an existing file at start, ctrld appends the header.
+
+The message of the line is `Log header`. The line carries these fields:
+
+- Build: `version` and `commit`
+- Host: `os`, `arch`, `pid`, and `start_time`
+- Mode: `intercept_mode`, `listeners`, `upstream_count`, and `upstream_types`
+- Identity: `resolver_uid`
+- Files: `log_files`
+- Network: `network` with `default_route_interface`, `have_v4`, `have_v6`, and `interfaces`
+
+The header holds no provisioning token, no upstream endpoint URL, and no query name. `upstream_types` holds the type words, for example `doh` or `os`. `listeners` holds the listen addresses.
+
+After each rotation ctrld logs one `Log rotated` event at info level, with the fields `file`, `bytes_written`, `first_event_at`, `last_event_at`, and `backups`. This event is a journal event, so it lands in the debug stream and in the journal.
+
+## Sampling of per-query errors
+
+A broken network produces one error line per query. ctrld bounds this flood for each error class and each upstream. The first 5 lines in a 60 s window stay at error level. Later lines in the same window drop to debug level, so the debug stream stays complete. A run with `-v` holds every stream at info, and the debug stream then loses these lines.
+
+When a window closes, ctrld logs one `Per-query errors sampled` summary at error level. The summary carries `class`, `upstream`, `count`, `suppressed`, and `window_s`. ctrld logs the summary only when the window suppressed at least one line.
+
+An upstream logs one `Upstream state changed` event per transition. The down event is at warn level and carries `state` `down`, `failure_count`, and `reason`. The up event is at info level and carries `state` `up`, `down_for_ms`, and `failure_count`. Both are journal events.
+
+## Configuration keys
+
+The `log_level` key does not change the internal streams. The debug file keeps every level, and the journal keeps the warnings, the errors, and the journal events. The `log_level` key changes the console output and the `log_path` file. The `-v` flag changes every stream, as the section "Streams and files" explains.
+
+Two keys in the `[service]` section size the debug stream:
+
+- The key `log_max_size_mb` sets the size limit of `ctrld.log` in MB. The default is 10 on a desktop and 5 on a router.
+- The key `log_max_backups` sets the number of backup files. The default is 4 on a desktop and 1 on a router. The value 0 keeps no backup file.
+
+Both keys also apply to the `log_path` file, which rotates the same way. The journal budget is fixed per platform.
+
+## Uploads
+
+`ctrld log send` uploads the log to Control D. The body holds these parts, in this order:
+
+- One header line rendered at send time, with the field `trigger` set to `send`
+- The newest 10 MB of the debug files, oldest file first, cut at a line boundary
+- The marker `=== LOG_END ===`
+- Every journal file, oldest first
+
+The `network` object of the send-time header comes from a fresh read of the interfaces and the default route. `ctrld log send --full` sends every debug file in place of the newest 10 MB. The reported size equals the number of bytes uploaded.
+
+`ctrld log view` prints the same composition on the terminal and takes the same `--full` flag. ctrld accepts one upload per minute. The body is plain text without gzip.
+
+In `log_path` mode the body holds the send-time header, then the `log_path` file and its backup files, oldest first. This body has no marker and no journal.
+
+`ctrld log tail` follows the live debug stream and prints each new line, journal events included.
 
 ## Notes
 
-- Internal logging is **not** a replacement for proper log file configuration in production environments
-- For production deployments, it is recommended to configure `log_path` to enable persistent file-based logging
-- Internal logs are stored in memory and will be lost if the process terminates unexpectedly
-- The internal logging system is automatically disabled when explicit logging is configured
-
+- Internal logging is not a replacement for a configured `log_path` file in production.
+- The journal is small on purpose. It keeps the events that describe a problem, not the events of each query.
+- Support reads the upload without access to the host. The header line names the build, the mode, and the network, so support does not have to ask for them.