mirror of
https://github.com/Control-D-Inc/ctrld.git
synced 2026-10-12 01:58:46 +02:00
feat(cli): port the log journal and network journal events to master
This commit is contained in:
Anthony Wong authored and Cuong Manh Le committed
1 parent
488065f6e8
commit
37b835c791
109 files changed
+17258
-905
No files matched your search
@@ -117,10 +117,28 @@ Logging level you wish to enable.
|
||||
### log_path
|
||||
Relative or absolute path of the log file.
|
||||
|
||||
Point `log_path` at a file that no other program rotates. ctrld renames `<log_path>.1` to `<log_path>.N` and uploads these files. A lower `log_max_backups` leaves the older numbered `log_path` files in place, but ctrld removes the extra numbered files of its internal logs.
|
||||
|
||||
- Type: string
|
||||
- Required: no
|
||||
- Default: ""
|
||||
|
||||
### log_max_size_mb
|
||||
Maximum size of the debug log file in MB. When the file reaches this size, ctrld rotates it. This limit also applies to the `log_path` file.
|
||||
|
||||
- Type: integer
|
||||
- Required: no
|
||||
- Valid values: 1 to 1024
|
||||
- Default: 10
|
||||
|
||||
### log_max_backups
|
||||
Number of rotated debug log files that ctrld keeps. If the value is 0, ctrld keeps no rotated file. This number also applies to the `log_path` file.
|
||||
|
||||
- Type: integer
|
||||
- Required: no
|
||||
- Valid values: 0 to 64
|
||||
- Default: 4
|
||||
|
||||
### cache_enable
|
||||
When `cache_enable = true`, all resolved DNS query responses will be cached for duration of the upstream record TTLs.
|
||||
|
||||
|
||||
@@ -235,6 +235,10 @@ DNS intercept: pf anchor "com.controld.ctrld" active with 3 rules
|
||||
DNS intercept: pf redirect active — all outbound DNS (port 53) redirected to 127.0.0.1:53
|
||||
```
|
||||
|
||||
DNS intercept mode on macOS also runs the wake detector. That detector reports a `Host woke` journal event with the source `detector` and the length of the sleep.
|
||||
Outside DNS intercept mode the netmon delta is the only wake source, and it reports no length of its own.
|
||||
The events of the journal are listed in [Network-recovery diagnostics](network-recovery-diagnostics.md#journal-events).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Windows
|
||||
@@ -306,12 +310,14 @@ VPN apps commonly add rules like `pass out quick on ipsec0 inet all` that match
|
||||
|
||||
### 3. Dynamic Tunnel Interface Detection
|
||||
|
||||
The network change monitor (`validInterfacesMap()`) only tracks physical hardware ports (en0, bridge0, etc.) — it doesn't see tunnel interfaces (utun*, ipsec*, etc.) created by VPN software. When a VPN connects and creates a new interface (e.g., utun420 for WireGuard), ctrld detects this through a separate tunnel interface change check and rebuilds the pf anchor to include explicit intercept rules for the new interface. This runs on every network change event, even if no physical interface changed.
|
||||
The network change monitor (`validInterfacesMap()`) only tracks physical hardware ports (en0, bridge0, etc.) — it doesn't see tunnel interfaces (utun*, ipsec*, etc.) created by VPN software. When a VPN connects and creates a new interface (e.g., utun420 for WireGuard), ctrld detects this through a separate tunnel interface change check and rebuilds the pf anchor to include explicit intercept rules for the new interface. This runs on every network change event that is not noise, even if no physical interface changed. A delta that touches AirDrop or virtual adapters alone is noise, and ctrld skips the handler for it, so no tunnel discovery runs.
|
||||
|
||||
### 4. pf Watchdog + Network Change Hooks
|
||||
|
||||
A background watchdog (30s interval) plus immediate checks on network change events detect when another program replaces the entire pf ruleset (e.g., Windscribe's `pfctl -f /etc/pf.conf`). When detected, ctrld rebuilds its anchor with up-to-date tunnel interface rules and re-injects the anchor reference at the top of the ruleset. A 2-second delayed re-check catches race conditions where the other program clears rules slightly after the network event.
|
||||
|
||||
A noise delta starts no immediate check and no delayed re-check. The watchdog keeps its own 30 s cadence, so it still restores a missing anchor during a delta storm.
|
||||
|
||||
### 4a. Active Interception Probe (pf Translation State Corruption)
|
||||
|
||||
Programs like Parallels Desktop reload `/etc/pf.conf` when creating/destroying virtual network interfaces (bridge100, vmenet0). This can corrupt pf's internal translation engine — rdr rules survive in text form but stop evaluating, causing DNS interception to silently fail while the watchdog reports "intact."
|
||||
@@ -639,6 +645,8 @@ Network change detected (netmon callback)
|
||||
└─ (same as 2s — catches slower VPN teardowns)
|
||||
```
|
||||
|
||||
A noise delta skips this whole chain. ctrld runs no pf read, no tunnel discovery, no OS resolver read, and no VPN DNS refresh for it. The noise class holds the AirDrop interfaces (`awdl*`, `llw*`) and the virtual adapters. The pf watchdog keeps its own 30 s cadence, so it still tests the rules during a delta storm.
|
||||
|
||||
### VPN Connect Sequence
|
||||
|
||||
1. VPN creates tunnel interface (e.g., utun420)
|
||||
|
||||
@@ -153,7 +153,10 @@ removed domain leaves no rule and no `internal_*` upstream behind.
|
||||
Generated `internal_*` upstreams carry the organization's private resolver
|
||||
addresses, so anything naming one stays at debug, and the level above it carries
|
||||
only a classification (`timeout`, `refused`, `unreachable`, `source_unavailable`,
|
||||
`error`). That covers every consumer, not just query handling:
|
||||
`error`). The per-query error sampler applies to the classification line as
|
||||
well. After 5 lines in one minute for the same upstream and class, the next
|
||||
classification lines go to debug, and one `Per-query errors sampled` line gives
|
||||
the count. That covers every consumer, not just query handling:
|
||||
|
||||
- the bootstrap-IP line in `setupUpstream`;
|
||||
- the transport error from a failed query, which names the endpoint;
|
||||
|
||||
@@ -1,14 +1,148 @@
|
||||
# Network-recovery diagnostics
|
||||
|
||||
Use continuous `ctrld log tail` capture during a reproduction. Internal logs rotate by size.
|
||||
A later `log send` can contain recent debug events and older warnings.
|
||||
Match timestamps and IDs. The last retained error does not necessarily describe the current network.
|
||||
Issue #604 tracks upload-time device and version headers separately.
|
||||
The retained stream is the journal, the file `ctrld-journal.log` in the ctrld home directory.
|
||||
It keeps every warning and every error, plus every event marked `journal=true`.
|
||||
The journal is on disk, so it keeps these events across a restart and a self-upgrade.
|
||||
Its budget is 2 MB with 2 backup files.
|
||||
Every `ctrld log send` upload holds the whole journal, so a reproduction does not need a continuous `ctrld log tail` capture.
|
||||
Match timestamps and IDs. The last retained error does not describe the current network in every case.
|
||||
Each upload starts with a header line that names the build, the intercept mode, and the network at send time.
|
||||
|
||||
## Journal events
|
||||
|
||||
These events describe the host network. Each one carries `journal=true`, so it stays on disk after a rotation and after a restart.
|
||||
netmon reports every network change as a delta, which holds the state before the change and the state after it.
|
||||
The table gives the trigger, the level, and the fields of each event.
|
||||
|
||||
| Event | Trigger | Level | Fields |
|
||||
|---|---|---|---|
|
||||
| `Network snapshot` | Start, an accepted transition, a recovery begin, and a recovery end. Two limits hold some of them back | info | `trigger` (`start`, `transition`, `recovery_begin`, `recovery_end`), `network` |
|
||||
| `Network interface changed` | One event per changed interface of a delta that is not noise. A mixed delta gives an event for the interfaces outside the noise class only | info | `transition_id`, `interface`, `action`, `class`, `hardware_port`, `service`, `ips_before`, `ips_after`, `flags`, `mtu`, `is_default_route` |
|
||||
| `Network transition` | The end of every network callback that is not noise | info, debug for the outcomes `ignored` and `snapshot_superseded` | `transition_id`, `outcome`, `is_major_change`, `changed`, `interface`, `changed_interfaces`, `time_jumped`, `default_route_before`, `default_route_after`, `have_v4_before`, `have_v4_after`, `have_v6_before`, `have_v6_after`, `source_ipv4_before`, `source_ipv4_after`, `source_ipv6_before`, `source_ipv6_after`, `source_snapshot_available` |
|
||||
| `Network delta noise` | Each noise delta, and one journal line per 10 minutes of a storm | debug, info for the journal line | `interfaces`, `count`, `first_at`, `last_at` |
|
||||
| `Host woke` | One event per wake: a time jump in a netmon delta, or the macOS wake detector, which runs in DNS intercept mode only | info | `gap_known`, `gap_ms` (only when `gap_known` is true), `source` (`netmon`, `detector`), `default_route`, `interfaces_up` |
|
||||
| `OS resolver set changed` | A nameserver read that differs from the read before | info | `source` (`scutil`, `dhcp`, `resolv.conf`, `unknown`), `before`, `after`, `default_route`, `reason` (`start`, `recovery`, `delayed_recheck`, `wake_probe`, `vpn_settle`, `unspecified`) |
|
||||
| `Reinitialized OS resolver with nameservers: …` | ctrld read the nameservers of the system again and built the OS resolver again | info | none, the nameservers are part of the message |
|
||||
| `DNS configuration changed` | A `scutil --dns` poll that finds a changed resolver, one event per resolver (macOS) | info | `nameservers`, `if_index`, `interface`, `scoped`, `action` (`added`, `changed`, `removed`), `flags`, `search_domain_count`, `order`, `reachable` |
|
||||
| `Recovery begin` | The start of every recovery | info | `transition_id`, `recovery_generation`, `recovery_reason`, `intercept` |
|
||||
| `Recovery end` | The end of every recovery | info, warn for a canceled recovery and for a recovery with a failure | `transition_id`, `recovery_generation`, `recovery_reason`, `outcome`, `had_failure`, `duration_ms`, `recovered_upstream`, `dhcp_servers`, `dhcp_server_count`, `intercept_target_action`, `bypass_active` |
|
||||
| `Upstream … recovered …` | An upstream that was down answers again | info | `down_for_ms` |
|
||||
| `Recovery canceled with no successor …` | A recovery stops and no other recovery follows it | info | none |
|
||||
| `DNS intercept recovery: enabling DHCP bypass …` | ctrld sends the queries of a recovery to the OS resolver | info | none |
|
||||
| `DNS intercept recovery complete: disabling DHCP bypass …` | The recovery ends and the normal query path returns | info | none |
|
||||
| `DNS intercept recovery: found DHCP nameservers: …` | ctrld read the DHCP nameservers of the service | info | `dhcp_servers` |
|
||||
| `DNS intercept recovery: no DHCP nameservers found` | The service gave no DHCP nameserver | warn | none |
|
||||
| `intercept DNS target: service … set …` | ctrld wrote the loopback target into a network service | warn | `service`, `target`, `reason` |
|
||||
| `intercept DNS target: removed …` | ctrld took the loopback target out of the service | info | `service`, `target`, `reason` |
|
||||
| `intercept DNS target: … DNS changed externally …` | Another program changed the DNS of the service, so the target of ctrld is gone (macOS) | info | `service`, `target`, `reason` (`external_change`) |
|
||||
| `dns64: discovered NAT64 prefix …` | A NAT64 prefix that differs from the result before | info | none |
|
||||
| `dns64: no NAT64 prefix present …` | A no-prefix result that differs from the result before | info | none |
|
||||
| `DNS intercept: interface appeared/disappeared …` | An interface appeared or disappeared, and the probe monitor starts | info | `interface`, `class`, `action` |
|
||||
| `PF anchor list changed` | A missing anchor, a restored anchor, a stabilization start, and a stabilization end (macOS) | warn for the reason `missing`, info for the rest | `reason`, `anchors`, `anchors_known`, `pf_enabled`, `pf_since` |
|
||||
| `Tunnel interface changed` | The set of tunnel interfaces changes (macOS). A retry of the same change writes no second event | info | `added`, `removed`, `owner` |
|
||||
| `DNS intercept: VPN connecting …` | A new tunnel interface starts the stabilization mode (macOS) | info | none |
|
||||
| `DNS intercept: pf stable for …` | The pf ruleset stops changing and stabilization ends (macOS) | info | none |
|
||||
| `Query health` | A change of the health class that holds for two evaluations, and a heartbeat every 15 minutes | info | `class`, `window_s`, `queries`, `failed_queries`, `cache_hits`, `failures_by_class`, `upstreams_down`, `bypass_active`, `cache_hit_ratio` |
|
||||
|
||||
No event in this catalog holds a query name, an upstream URL, or a token.
|
||||
The journal does hold network addresses. It holds the LAN and VPN resolver addresses, the gateways, the addresses of the host, and the DHCP servers of a service.
|
||||
It holds no search suffix. `DNS configuration changed` counts the suffixes in `search_domain_count` and logs the count alone.
|
||||
The field `owner` of `Tunnel interface changed` names the VPN product that owns the tunnel. On Windows the field `hardware_port` names the VPN product of an adapter in the same way.
|
||||
The Wi-Fi network name is never logged. On macOS 14 and later, only a process with the Location Services permission can read it, and a daemon cannot request that permission.
|
||||
|
||||
### The network object
|
||||
|
||||
The `network` object describes the host network at one moment.
|
||||
The `Network snapshot` event and the header line of every log file render the same object, with these fields:
|
||||
|
||||
- Routes: `default_route_v4`, `default_route_v6`, `gateway_v4`, `gateway_v6`
|
||||
- Families: `have_v4`, `have_v6`
|
||||
- Interfaces: `interfaces`, an array of `name`, `class`, `up`, `ips`, `hardware_port`, `service`, and the count `interfaces_omitted`
|
||||
- Resolvers: `resolvers`, the host and the port of each OS nameserver
|
||||
- Source addresses: `source_ipv4`, `source_ipv6`
|
||||
- Intercept state: `intercept_target`, `dns_less_target`, `bypass_active`, `recovery_running`, `pf_stabilizing`
|
||||
- Link: `link_type`, `tethered`, `clat_present`, `nat64_prefix`
|
||||
|
||||
The field `default_route_interface` is gone. Read `default_route_v4` and `default_route_v6` in its place.
|
||||
The two routes and the two gateways come from the route table of macOS, one route per address family. A tunnel that owns the default route has no next hop, so its route names the tunnel and its gateway stays empty.
|
||||
Windows and Linux read no route table in this release. They fill `default_route_v4` from the network monitor and leave `default_route_v6`, `gateway_v4`, and `gateway_v6` empty.
|
||||
|
||||
`link_type` is one of `wifi`, `ethernet`, `usb_tether`, `tunnel`, and `unknown`.
|
||||
macOS reads it from the name of the hardware port. Linux reads `wifi`, `ethernet`, `usb_tether`, and `tunnel` from the kernel.
|
||||
Windows reads the description of the adapter in place of a port name. The words Ethernet, LAN, and Thunderbolt win over the tunnel class on every platform, so a VPN adapter with a description such as `SSL VPN Virtual Ethernet Adapter` reads as `ethernet`.
|
||||
Windows gives `tunnel` to the other VPN adapters. The other links are `unknown`.
|
||||
The field `service` names a macOS network service. It stays empty on Windows and on Linux.
|
||||
`tethered` is true on every platform when the interface of the default route holds an address of the range 172.20.10.0/28.
|
||||
It is also true when the default gateway is in that range, and when the hardware port of the default route is `iPhone USB`.
|
||||
|
||||
The list `interfaces` holds the interfaces that are up or hold an address, and it stops at 32 entries.
|
||||
`interfaces_omitted` counts the interfaces that the list leaves out, the ones over the limit included.
|
||||
A container host holds hundreds of interfaces, and a list of all of them fills the journal.
|
||||
|
||||
ctrld does not write a snapshot for every trigger. A snapshot equal to the snapshot written last reaches no journal.
|
||||
Every other snapshot waits 60 s after the snapshot written last. A new value of `default_route_v4`, `default_route_v6`, or `resolvers` passes that wait.
|
||||
The trigger `start` always writes.
|
||||
|
||||
### Noise class
|
||||
|
||||
A noise delta changes interfaces of the noise class only, and it does not move the default route.
|
||||
The noise class holds `awdl*` and `llw*` on macOS.
|
||||
On Linux it holds the names `docker*`, `veth*`, `virbr*`, and `br-*`. A device of the tun or tap type is a tunnel, not a noise interface.
|
||||
On Windows it holds the virtual adapters without a connector. A Windows adapter whose description names TAP, Wintun, WireGuard, or VPN is a tunnel, not a noise interface.
|
||||
On Linux and on Windows, an interface that holds a global address on either side of the delta is never noise.
|
||||
A delta is also not noise when it flips `have_v4` or `have_v6`, or when it carries a time jump.
|
||||
A noise delta runs no reconcile work.
|
||||
ctrld starts no `scutil` process, reads no PF rule, discovers no tunnel, refreshes no VPN DNS, and schedules no delayed recheck for it.
|
||||
A delta that touches a hardware port and a noise interface in the same callback is not noise.
|
||||
The PF watchdog keeps its own 30 s timer, so it still tests the PF rules during a delta storm.
|
||||
|
||||
Each noise delta logs one debug line. The first noise delta of a storm also writes a journal line with `count` 1.
|
||||
While the storm lasts, one journal line of the same name arrives every 10 minutes, with the count of that window.
|
||||
AirDrop in the Finder is the common source of such a storm on macOS.
|
||||
|
||||
### DNS configuration poll (macOS)
|
||||
|
||||
ctrld polls `scutil --dns` on macOS only. The first poll runs at start.
|
||||
The poll runs every 15 s for 10 minutes after a network change, a wake, or the end of a recovery.
|
||||
Each of these three events starts the fast cadence at once, even while the poll waits.
|
||||
At rest the poll runs every 5 minutes. Each changed resolver of a poll gives one `DNS configuration changed` event.
|
||||
|
||||
### Change-only lines
|
||||
|
||||
These lines log at debug level on change only. Each one carries `repeats`, the number of equal results before it:
|
||||
|
||||
- `Got system nameservers`
|
||||
- `Final available nameservers`
|
||||
- `pf anchor intact`
|
||||
- `discovered active tunnel interfaces`
|
||||
- `VPN DNS refresh completed`
|
||||
|
||||
### PF anchor list (macOS)
|
||||
|
||||
A `PF anchor list changed` event reaches the journal when its reason, its anchor list, or `pf_enabled` differs from the event before it.
|
||||
The field `pf_since` is no part of that test, because the uptime of pf grows between two reads of one state.
|
||||
A failed read of the anchor names gives `anchors_known` false and no anchor list.
|
||||
|
||||
### Host woke
|
||||
|
||||
ctrld writes one `Host woke` event per wake. Two sources that report the same wake inside 30 s give one event.
|
||||
The macOS wake detector runs in DNS intercept mode only. Outside that mode, and on Windows and Linux, the netmon delta is the only source.
|
||||
The netmon source knows no gap, so its report waits 8 s for the detector. A detector report inside that hold gives the event, with `gap_known` true and the length of the sleep in `gap_ms`. Without a detector report, the netmon report goes out with `gap_known` false and no `gap_ms`. Every later report of the same wake is dropped.
|
||||
|
||||
### Query health
|
||||
|
||||
ctrld grades the query path over a window of 15 minutes and reads the window every minute.
|
||||
Each failed client query counts one failure, whatever number of upstreams that query tried.
|
||||
`failures_by_class` keeps one count per failure class. The class `internal_domain` holds the failures of an Internal Domain resolver.
|
||||
The grade leaves the class `internal_domain` out, and `upstreams_down` counts no Internal Domain resolver.
|
||||
A laptop away from the company network cannot reach these resolvers. Without this rule the grade stays degraded for the whole day.
|
||||
A change of the class reports after it holds for two reads of the window.
|
||||
|
||||
## Transitions and recovery
|
||||
|
||||
`Network transition` debug events include a `transition_id`, affected interface, default route, and stored source addresses before and after the callback.
|
||||
The outcome identifies an accepted, ignored, superseded, or unusable change.
|
||||
`Network transition` events include a `transition_id`, affected interface, default route, and stored source addresses before and after the callback.
|
||||
The outcome identifies an accepted, ignored, superseded, snapshot_superseded, or unusable change.
|
||||
A callback whose state read a newer snapshot gives the outcome `snapshot_superseded` at debug level with `transition_id` 0, and it gives no interface event.
|
||||
Newer accepted changes supersede pending recovery. Ignored changes do not cancel accepted recovery.
|
||||
Source commits validate addresses against the current network state.
|
||||
Netmon does not order minor callbacks within one major cache epoch.
|
||||
@@ -16,11 +150,12 @@ For minor callbacks and late major callbacks in that epoch, ctrld reads current
|
||||
This read does not change host DNS. A failed read defers source writes, but reconciliation continues.
|
||||
|
||||
`Removed stale resolver source` is a warning only when ctrld removes an invalid address.
|
||||
The warning gives the reason and survives debug rotation.
|
||||
The warning gives the reason. Every warning enters the journal, so it stays on disk after a debug rotation and after a restart.
|
||||
`Recovery begin` and `Recovery end` connect the transition to a `recovery_generation`, reason, outcome, and elapsed time.
|
||||
Only the first upstream failure in each recovery produces a bounded summary.
|
||||
If a recovery encounters a failure, its final outcome also enters the warning stream.
|
||||
Normal successful recovery stays at debug level.
|
||||
Every recovery outcome enters the journal.
|
||||
A canceled recovery and a recovery with a failure log at warn level.
|
||||
For the outcome `superseded`, `intercept_target_action` is `unchanged` and `bypass_active` is false. The recovery that follows owns both of them. The outcome `failed` means that the preparation of the recovery reported an error, and the event is at warn level.
|
||||
|
||||
These summaries exclude upstream URLs and queried names.
|
||||
Each PF probe records the recovery generation at its start.
|
||||
@@ -45,9 +180,9 @@ Note: On the 1.x (`v1.0`) line, the previous message was `Recovery canceled; DNS
|
||||
|
||||
A changed unsuccessful condition produces a warning. The condition consists of the stage, error code, outcome, and target.
|
||||
Equal results stay at debug level. The next changed condition includes the preceding repeat count.
|
||||
The first restored receipt also produces a warning-stream event. This event prevents an older failure from appearing current.
|
||||
The first restored receipt also produces a warning, which enters the journal. This event prevents an older failure from appearing current.
|
||||
Normal receipts and shutdown cancellation stay at debug level.
|
||||
The warning buffer is finite. These events do not guarantee indefinite retention.
|
||||
The journal budget is finite and the journal rotates by size. It does not keep these events without a limit.
|
||||
|
||||
The outcomes have these meanings:
|
||||
|
||||
|
||||
@@ -249,6 +249,13 @@ The monitor probes with exponential backoff (0, 0.5, 1, 2, 4s) to win the race
|
||||
against async pf reloads. Only one monitor runs at a time (singleton). The
|
||||
watchdog also runs the probe every 30s as a safety net.
|
||||
|
||||
A network delta that touches AirDrop or virtual adapters alone is noise. The
|
||||
handler returns at once for it, so no probe monitor starts and no pf read runs.
|
||||
A delta that moves the default route, that flips the v4 or the v6 family, or
|
||||
that carries a time jump is not noise, whatever its interfaces are.
|
||||
The watchdog keeps its own 30 s cadence, so it still tests the rules during a
|
||||
delta storm.
|
||||
|
||||
The full pf reload is VPN-safe: it reassembles from `pfctl -sr` + `pfctl -sn`
|
||||
(the current running state), preserving all existing anchors and rules.
|
||||
|
||||
|
||||
@@ -1,46 +1,107 @@
|
||||
# Runtime Internal Logging
|
||||
|
||||
When no logging is configured (i.e., `log_path` is not set), ctrld automatically enables an internal logging system. This system stores logs in memory to provide troubleshooting information when problems occur.
|
||||
# Runtime internal logging
|
||||
|
||||
## Purpose
|
||||
|
||||
The runtime internal logging system is designed primarily for **ctrld developers**, not end users. It captures detailed diagnostic information that can be useful for troubleshooting issues when they arise, especially in production environments where explicit logging may not be configured.
|
||||
ctrld writes its own log when the configuration sets no log file. The log gives Control D support and ctrld developers the evidence they need after a problem. Two of the three streams are files on disk. They keep their content across a restart and a self-upgrade.
|
||||
|
||||
## When It's Enabled
|
||||
## When it is on
|
||||
|
||||
Internal logging is automatically enabled when:
|
||||
Internal logging is on when these three conditions are true at the same time:
|
||||
|
||||
- ctrld is running in Control D mode (i.e., `--cd` flag is provided)
|
||||
- No log file is configured (i.e., `log_path` is empty or not set)
|
||||
- ctrld runs in Control D mode, with the `--cd` flag
|
||||
- The configuration sets no `log_path`
|
||||
- Silent mode is off
|
||||
|
||||
If a log file is explicitly configured via `log_path`, internal logging will **not** be enabled, as the configured log file serves the logging purpose.
|
||||
When `log_path` is set, ctrld writes that file and starts no internal stream. The journal needs internal logging, so `log_path` mode has no journal.
|
||||
|
||||
## How It Works
|
||||
## Streams and files
|
||||
|
||||
The internal logging system:
|
||||
ctrld writes three streams. The debug file and the journal file live in the ctrld home directory, next to each other.
|
||||
|
||||
- Stores logs in **in-memory buffers** (not written to disk)
|
||||
- Captures logs at **debug level** for normal operations and **warn level** for warnings
|
||||
- Maintains separate buffers for normal logs and warning logs
|
||||
- Automatically manages buffer size to prevent unbounded memory growth
|
||||
- Preserves initialization logs even when buffers overflow
|
||||
| Stream | Level filter | File | Budget | Survives a restart |
|
||||
|---|---|---|---|---|
|
||||
| Console | Notice and above. The `-v` flag adds the lower levels | none | none | No |
|
||||
| Debug | Every level | `ctrld.log` | 10 MB, 4 backup files | Yes |
|
||||
| Journal | Warning and above, notice included, plus every event with `journal=true` | `ctrld-journal.log` | 2 MB, 2 backup files | Yes |
|
||||
|
||||
## Configuration
|
||||
The `-v` flag changes the console level and the level of the `log_path` file. Without the flag, the console shows notice and above. With `-v` it shows info and above. With `-vv` it shows every line. The debug file and the journal keep their filters in each case.
|
||||
|
||||
**Important**: The `log_level` configuration option does **not** affect the internal logging system. Internal logging always operates at debug level for normal logs and warn level for warnings, regardless of the `log_level` setting in the configuration file.
|
||||
A backup file carries a number suffix. The debug backup files are `ctrld.log.1` to `ctrld.log.4`, and `ctrld.log.1` holds the newest rotated content. The journal backup files are `ctrld-journal.log.1` and `ctrld-journal.log.2`.
|
||||
|
||||
The `log_level` setting only affects:
|
||||
- Console output (when running interactively)
|
||||
- File-based logging (when `log_path` is configured)
|
||||
Rotation is by size. A write that crosses the size limit rotates the file first. ctrld deletes the file beyond the backup count.
|
||||
|
||||
## Accessing Internal Logs
|
||||
Each stream also keeps a memory buffer: 5 MB for the debug stream and 1 MB for the journal. These buffers serve `ctrld log tail` and the fallback path when a file cannot open. The files hold the history.
|
||||
|
||||
Internal logs can be accessed through the control server API endpoints. This functionality is intended for developers and support personnel who need to diagnose issues.
|
||||
A restart deletes the memory buffers and keeps the files. The new process appends a header line, which marks the restart point inside each file.
|
||||
|
||||
## Header and rotation events
|
||||
|
||||
Every debug file and every journal file starts with one JSON line, the header. ctrld writes the header when it opens a file at start and after every rotation. On an existing file at start, ctrld appends the header. The journal holds JSON lines. The other lines of the debug file have the text format of the console.
|
||||
|
||||
The message of the line is `Log header`. The line carries these fields:
|
||||
|
||||
- Build: `version` and `commit`
|
||||
- Host: `os`, `arch`, `pid`, and `start_time`
|
||||
- Mode: `intercept_mode`, `listeners`, `upstream_count`, and `upstream_types`
|
||||
- Identity: `resolver_uid`
|
||||
- Files: `log_files`
|
||||
- Network: `network`, the network snapshot of the host
|
||||
|
||||
The header holds no provisioning token, no upstream endpoint URL, and no query name. `upstream_types` holds the type words, for example `doh` or `os`. `listeners` holds the listen addresses.
|
||||
|
||||
The `network` object holds the routes, the gateways, the interfaces, the resolvers, the link type, and the intercept state. It has no `default_route_interface` field. The runbook [Network-recovery diagnostics](network-recovery-diagnostics.md) lists every field of the object.
|
||||
|
||||
After each rotation ctrld logs one `Log rotated` event at info level, with the fields `file`, `bytes_written`, `first_event_at`, `last_event_at`, and `backups`. This event is a journal event, so it lands in the debug stream and in the journal.
|
||||
|
||||
## Network state events
|
||||
|
||||
The journal holds the events that describe the host network. `Network snapshot` renders the `network` object at start, at each accepted transition, and at each begin and end of a recovery.
|
||||
A snapshot equal to the snapshot written last reaches no journal, and one snapshot per 60 s is the limit. A new default route and a new resolver set pass that limit. `Network interface changed`, `Network transition`, `Host woke`, `OS resolver set changed`, and `DNS configuration changed` name each change of the network. `Recovery begin`, `Recovery end`, `PF anchor list changed`, and `Tunnel interface changed` name the repair work.
|
||||
|
||||
AirDrop and virtual adapters produce a storm of network callbacks. ctrld puts these interfaces in a noise class. A delta that touches noise interfaces alone skips the handler. The first delta of a storm writes one `Network delta noise` journal line, and one more line covers each 10 minutes of the storm.
|
||||
|
||||
`Query health` grades the query path over a window of 15 minutes. It logs at each change between the classes `healthy`, `degraded`, and `failing`, and once every 15 minutes as a heartbeat. A class change reports after it holds for two reads of the window.
|
||||
|
||||
The runbook [Network-recovery diagnostics](network-recovery-diagnostics.md) lists every event with its trigger, its level, and its fields.
|
||||
|
||||
## Sampling of per-query errors
|
||||
|
||||
A broken network produces one error line per query. ctrld bounds this flood for each error class and each upstream. The first 5 lines in a 60 s window stay at error level. Later lines in the same window drop to debug level, so the debug stream stays complete. The console and the `log_path` file show these lines only at debug level. The debug file keeps them.
|
||||
|
||||
When a window closes, ctrld logs one `Per-query errors sampled` summary at error level. The summary carries `class`, `upstream`, `count`, `suppressed`, and `window_s`. ctrld logs the summary only when the window suppressed at least one line.
|
||||
|
||||
An upstream logs one `Upstream state changed` event per transition. The down event is at warn level and carries `state` `down`, `failure_count`, and `reason`. The up event is at info level and carries `state` `up`, `down_for_ms`, and `failure_count`. Both are journal events.
|
||||
|
||||
## Configuration keys
|
||||
|
||||
The `log_level` key does not change the internal streams. The debug file keeps every level, and the journal keeps the warnings, the notices, the errors, and the journal events. A notice shares the level value of a warning. The `log_level` key changes the `log_path` file only. The `-v` flag changes the console level and the level of the `log_path` file, as the section "Streams and files" explains.
|
||||
|
||||
Two keys in the `[service]` section size the debug stream:
|
||||
|
||||
- The key `log_max_size_mb` sets the size limit of `ctrld.log` in MB. The default is 10.
|
||||
- The key `log_max_backups` sets the number of backup files. The default is 4. The value 0 keeps no backup file.
|
||||
|
||||
Both keys also apply to the `log_path` file, which rotates the same way. The journal budget is fixed.
|
||||
|
||||
## Uploads
|
||||
|
||||
`ctrld log send` uploads the log to Control D. The body holds these parts, in this order:
|
||||
|
||||
- One header line rendered at send time, with the field `trigger` set to `send`
|
||||
- The newest 10 MB of the debug files, oldest file first, cut at a line boundary
|
||||
- The marker `=== LOG_END ===`
|
||||
- Every journal file, oldest first
|
||||
|
||||
The `network` object of the send-time header comes from a fresh read of the interfaces and of the route table. The resolvers come from the list that ctrld stored at its last resolver initialization, not from a fresh read. `ctrld log send --full` sends every debug file in place of the newest 10 MB. The reported size equals the number of bytes uploaded.
|
||||
|
||||
`ctrld log view` prints the same composition on the terminal and takes the same `--full` flag. ctrld accepts one upload per minute. The body is plain text without gzip.
|
||||
|
||||
In `log_path` mode the body holds the send-time header, then the `log_path` file and its backup files, oldest first. This body has no marker and no journal.
|
||||
|
||||
`ctrld log tail` follows the live debug stream and prints each new line, journal events included.
|
||||
|
||||
## Notes
|
||||
|
||||
- Internal logging is **not** a replacement for proper log file configuration in production environments
|
||||
- For production deployments, it is recommended to configure `log_path` to enable persistent file-based logging
|
||||
- Internal logs are stored in memory and will be lost if the process terminates unexpectedly
|
||||
- The internal logging system is automatically disabled when explicit logging is configured
|
||||
|
||||
- Internal logging is not a replacement for a configured `log_path` file in production.
|
||||
- The journal is small on purpose. It keeps the events that describe a problem, not the events of each query.
|
||||
- Support reads the upload without access to the host. The header line names the build, the mode, and the network, so support does not have to ask for them.
|
||||
Reference in new issue
Block a user