mirror of
https://github.com/Control-D-Inc/ctrld.git
synced 2026-10-12 01:58:46 +02:00
feat(cli): put the network state of the host in the log journal
The journal records the network state of the host and each change of that state. The support team reads one file to see the interfaces, the routes, the resolvers, and the recoveries around an incident. The same file shows the health of the query path. A change that touches only AirDrop or virtual interfaces runs no DNS work.
This commit is contained in:
Anthony Wong authored and Cuong Manh Le committed
1 parent
8934e09428
commit
8ae09ac98e
68 files changed
+10458
-442
No files matched your search
@@ -235,6 +235,10 @@ DNS intercept: pf anchor "com.controld.ctrld" active with 3 rules
|
||||
DNS intercept: pf redirect active — all outbound DNS (port 53) redirected to 127.0.0.1:53
|
||||
```
|
||||
|
||||
DNS intercept mode on macOS also runs the wake detector. That detector reports a `Host woke` journal event with the source `detector` and the length of the sleep.
|
||||
Outside DNS intercept mode the netmon delta is the only wake source, and it reports no length of its own.
|
||||
The events of the journal are listed in [Network-recovery diagnostics](network-recovery-diagnostics.md#journal-events).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Windows
|
||||
@@ -306,12 +310,14 @@ VPN apps commonly add rules like `pass out quick on ipsec0 inet all` that match
|
||||
|
||||
### 3. Dynamic Tunnel Interface Detection
|
||||
|
||||
The network change monitor (`validInterfacesMap()`) only tracks physical hardware ports (en0, bridge0, etc.) — it doesn't see tunnel interfaces (utun*, ipsec*, etc.) created by VPN software. When a VPN connects and creates a new interface (e.g., utun420 for WireGuard), ctrld detects this through a separate tunnel interface change check and rebuilds the pf anchor to include explicit intercept rules for the new interface. This runs on every network change event, even if no physical interface changed.
|
||||
The network change monitor (`validInterfacesMap()`) only tracks physical hardware ports (en0, bridge0, etc.) — it doesn't see tunnel interfaces (utun*, ipsec*, etc.) created by VPN software. When a VPN connects and creates a new interface (e.g., utun420 for WireGuard), ctrld detects this through a separate tunnel interface change check and rebuilds the pf anchor to include explicit intercept rules for the new interface. This runs on every network change event that is not noise, even if no physical interface changed. A delta that touches AirDrop or virtual adapters alone is noise, and ctrld skips the handler for it, so no tunnel discovery runs.
|
||||
|
||||
### 4. pf Watchdog + Network Change Hooks
|
||||
|
||||
A background watchdog (30s interval) plus immediate checks on network change events detect when another program replaces the entire pf ruleset (e.g., Windscribe's `pfctl -f /etc/pf.conf`). When detected, ctrld rebuilds its anchor with up-to-date tunnel interface rules and re-injects the anchor reference at the top of the ruleset. A 2-second delayed re-check catches race conditions where the other program clears rules slightly after the network event.
|
||||
|
||||
A noise delta starts no immediate check and no delayed re-check. The watchdog keeps its own 30 s cadence, so it still restores a missing anchor during a delta storm.
|
||||
|
||||
### 4a. Active Interception Probe (pf Translation State Corruption)
|
||||
|
||||
Programs like Parallels Desktop reload `/etc/pf.conf` when creating/destroying virtual network interfaces (bridge100, vmenet0). This can corrupt pf's internal translation engine — rdr rules survive in text form but stop evaluating, causing DNS interception to silently fail while the watchdog reports "intact."
|
||||
@@ -639,6 +645,8 @@ Network change detected (netmon callback)
|
||||
└─ (same as 2s — catches slower VPN teardowns)
|
||||
```
|
||||
|
||||
A noise delta skips this whole chain. ctrld runs no pf read, no tunnel discovery, no OS resolver read, and no VPN DNS refresh for it. The noise class holds the AirDrop interfaces (`awdl*`, `llw*`) and the virtual adapters. The pf watchdog keeps its own 30 s cadence, so it still tests the rules during a delta storm.
|
||||
|
||||
### VPN Connect Sequence
|
||||
|
||||
1. VPN creates tunnel interface (e.g., utun420)
|
||||
|
||||
@@ -8,10 +8,139 @@ Every `ctrld log send` upload holds the whole journal, so a reproduction does no
|
||||
Match timestamps and IDs. The last retained error does not describe the current network in every case.
|
||||
Each upload starts with a header line that names the build, the intercept mode, and the network at send time.
|
||||
|
||||
## Journal events
|
||||
|
||||
These events describe the host network. Each one carries `journal=true`, so it stays on disk after a rotation and after a restart.
|
||||
netmon reports every network change as a delta, which holds the state before the change and the state after it.
|
||||
The table gives the trigger, the level, and the fields of each event.
|
||||
|
||||
| Event | Trigger | Level | Fields |
|
||||
|---|---|---|---|
|
||||
| `Network snapshot` | Start, an accepted transition, a recovery begin, and a recovery end. Two limits hold some of them back | info | `trigger` (`start`, `transition`, `recovery_begin`, `recovery_end`), `network` |
|
||||
| `Network interface changed` | One event per changed interface of a delta that is not noise. A mixed delta gives an event for the interfaces outside the noise class only | info | `transition_id`, `interface`, `action`, `class`, `hardware_port`, `service`, `ips_before`, `ips_after`, `flags`, `mtu`, `is_default_route` |
|
||||
| `Network transition` | The end of every network callback that is not noise | info, debug for the outcomes `ignored` and `snapshot_superseded` | `transition_id`, `outcome`, `is_major_change`, `changed`, `interface`, `changed_interfaces`, `time_jumped`, `default_route_before`, `default_route_after`, `have_v4_before`, `have_v4_after`, `have_v6_before`, `have_v6_after`, `source_ipv4_before`, `source_ipv4_after`, `source_ipv6_before`, `source_ipv6_after`, `source_snapshot_available` |
|
||||
| `Network delta noise` | Each noise delta, and one journal line per 10 minutes of a storm | debug, info for the journal line | `interfaces`, `count`, `first_at`, `last_at` |
|
||||
| `Host woke` | One event per wake: a time jump in a netmon delta, or the macOS wake detector, which runs in DNS intercept mode only | info | `gap_known`, `gap_ms` (only when `gap_known` is true), `source` (`netmon`, `detector`), `default_route`, `interfaces_up` |
|
||||
| `OS resolver set changed` | A nameserver read that differs from the read before | info | `source` (`scutil`, `dhcp`, `resolv.conf`, `unknown`), `before`, `after`, `default_route`, `reason` (`start`, `recovery`, `delayed_recheck`, `wake_probe`, `vpn_settle`, `unspecified`) |
|
||||
| `Reinitialized OS resolver with nameservers: …` | ctrld read the nameservers of the system again and built the OS resolver again | info | none, the nameservers are part of the message |
|
||||
| `DNS configuration changed` | A `scutil --dns` poll that finds a changed resolver, one event per resolver (macOS) | info | `nameservers`, `if_index`, `interface`, `scoped`, `action` (`added`, `changed`, `removed`), `flags`, `search_domain_count`, `order`, `reachable` |
|
||||
| `Recovery begin` | The start of every recovery | info | `transition_id`, `recovery_generation`, `recovery_reason`, `intercept` |
|
||||
| `Recovery end` | The end of every recovery | info, warn for a canceled recovery and for a recovery with a failure | `transition_id`, `recovery_generation`, `recovery_reason`, `outcome`, `had_failure`, `duration_ms`, `recovered_upstream`, `dhcp_servers`, `dhcp_server_count`, `intercept_target_action`, `bypass_active` |
|
||||
| `Upstream … recovered …` | An upstream that was down answers again | info | `down_for_ms` |
|
||||
| `Recovery canceled with no successor …` | A recovery stops and no other recovery follows it | info | none |
|
||||
| `DNS intercept recovery: enabling DHCP bypass …` | ctrld sends the queries of a recovery to the OS resolver | info | none |
|
||||
| `DNS intercept recovery complete: disabling DHCP bypass …` | The recovery ends and the normal query path returns | info | none |
|
||||
| `DNS intercept recovery: found DHCP nameservers: …` | ctrld read the DHCP nameservers of the service | info | `dhcp_servers` |
|
||||
| `DNS intercept recovery: no DHCP nameservers found` | The service gave no DHCP nameserver | warn | none |
|
||||
| `intercept DNS target: service … set …` | ctrld wrote the loopback target into a network service | warn | `service`, `target`, `reason` |
|
||||
| `intercept DNS target: removed …` | ctrld took the loopback target out of the service | info | `service`, `target`, `reason` |
|
||||
| `intercept DNS target: … DNS changed externally …` | Another program changed the DNS of the service, so the target of ctrld is gone (macOS) | info | `service`, `target`, `reason` (`external_change`) |
|
||||
| `dns64: discovered NAT64 prefix …` | A NAT64 prefix that differs from the result before | info | none |
|
||||
| `dns64: no NAT64 prefix present …` | A no-prefix result that differs from the result before | info | none |
|
||||
| `DNS intercept: interface appeared/disappeared …` | An interface appeared or disappeared, and the probe monitor starts | info | `interface`, `class`, `action` |
|
||||
| `PF anchor list changed` | A missing anchor, a restored anchor, a stabilization start, and a stabilization end (macOS) | warn for the reason `missing`, info for the rest | `reason`, `anchors`, `anchors_known`, `pf_enabled`, `pf_since` |
|
||||
| `Tunnel interface changed` | The set of tunnel interfaces changes (macOS). A retry of the same change writes no second event | info | `added`, `removed`, `owner` |
|
||||
| `DNS intercept: VPN connecting …` | A new tunnel interface starts the stabilization mode (macOS) | info | none |
|
||||
| `DNS intercept: pf stable for …` | The pf ruleset stops changing and stabilization ends (macOS) | info | none |
|
||||
| `Query health` | A change of the health class that holds for two evaluations, and a heartbeat every 15 minutes | info | `class`, `window_s`, `queries`, `failed_queries`, `cache_hits`, `failures_by_class`, `upstreams_down`, `bypass_active`, `cache_hit_ratio` |
|
||||
|
||||
No event in this catalog holds a query name, an upstream URL, or a token.
|
||||
The journal does hold network addresses. It holds the LAN and VPN resolver addresses, the gateways, the addresses of the host, and the DHCP servers of a service.
|
||||
It holds no search suffix. `DNS configuration changed` counts the suffixes in `search_domain_count` and logs the count alone.
|
||||
The field `owner` of `Tunnel interface changed` names the VPN product that owns the tunnel. On Windows the field `hardware_port` names the VPN product of an adapter in the same way.
|
||||
The Wi-Fi network name is never logged. On macOS 14 and later, only a process with the Location Services permission can read it, and a daemon cannot request that permission.
|
||||
|
||||
### The network object
|
||||
|
||||
The `network` object describes the host network at one moment.
|
||||
The `Network snapshot` event and the header line of every log file render the same object, with these fields:
|
||||
|
||||
- Routes: `default_route_v4`, `default_route_v6`, `gateway_v4`, `gateway_v6`
|
||||
- Families: `have_v4`, `have_v6`
|
||||
- Interfaces: `interfaces`, an array of `name`, `class`, `up`, `ips`, `hardware_port`, `service`, and the count `interfaces_omitted`
|
||||
- Resolvers: `resolvers`, the host and the port of each OS nameserver
|
||||
- Source addresses: `source_ipv4`, `source_ipv6`
|
||||
- Intercept state: `intercept_target`, `dns_less_target`, `bypass_active`, `recovery_running`, `pf_stabilizing`
|
||||
- Link: `link_type`, `tethered`, `clat_present`, `nat64_prefix`
|
||||
|
||||
The field `default_route_interface` is gone. Read `default_route_v4` and `default_route_v6` in its place.
|
||||
The two routes and the two gateways come from the route table of macOS, one route per address family. A tunnel that owns the default route has no next hop, so its route names the tunnel and its gateway stays empty.
|
||||
Windows and Linux read no route table in this release. They fill `default_route_v4` from the network monitor and leave `default_route_v6`, `gateway_v4`, and `gateway_v6` empty.
|
||||
|
||||
`link_type` is one of `wifi`, `ethernet`, `usb_tether`, `tunnel`, and `unknown`.
|
||||
macOS reads it from the name of the hardware port. Linux reads `wifi`, `ethernet`, `usb_tether`, and `tunnel` from the kernel. Windows names a tunnel and leaves the other links unknown.
|
||||
The field `service` names a macOS network service. It stays empty on Windows and on Linux.
|
||||
`tethered` is true on every platform when the interface of the default route holds an address of the range 172.20.10.0/28.
|
||||
It is also true when the default gateway is in that range, and when the hardware port of the default route is `iPhone USB`.
|
||||
|
||||
The list `interfaces` holds the interfaces that are up or hold an address, and it stops at 32 entries.
|
||||
`interfaces_omitted` counts the interfaces that the list leaves out, the ones over the limit included.
|
||||
A container host holds hundreds of interfaces, and a list of all of them fills the journal.
|
||||
|
||||
ctrld does not write a snapshot for every trigger. A snapshot equal to the snapshot written last reaches no journal.
|
||||
Every other snapshot waits 60 s after the snapshot written last. A new value of `default_route_v4`, `default_route_v6`, or `resolvers` passes that wait.
|
||||
The trigger `start` always writes.
|
||||
|
||||
### Noise class
|
||||
|
||||
A noise delta changes interfaces of the noise class only, and it does not move the default route.
|
||||
The noise class holds `awdl*` and `llw*` on macOS.
|
||||
On Linux it holds the names `docker*`, `veth*`, `virbr*`, and `br-*`, plus the devices of the tun and tap types.
|
||||
On Windows it holds the virtual adapters without a connector. A Windows adapter whose description names TAP, Wintun, WireGuard, or VPN is a tunnel, not a noise interface.
|
||||
On Linux and on Windows, an interface that holds a global address on either side of the delta is never noise.
|
||||
A delta is also not noise when it flips `have_v4` or `have_v6`, or when it carries a time jump.
|
||||
A noise delta runs no reconcile work.
|
||||
ctrld starts no `scutil` process, reads no PF rule, discovers no tunnel, refreshes no VPN DNS, and schedules no delayed recheck for it.
|
||||
A delta that touches a hardware port and a noise interface in the same callback is not noise.
|
||||
The PF watchdog keeps its own 30 s timer, so it still tests the PF rules during a delta storm.
|
||||
|
||||
Each noise delta logs one debug line. The first noise delta of a storm also writes a journal line with `count` 1.
|
||||
While the storm lasts, one journal line of the same name arrives every 10 minutes, with the count of that window.
|
||||
AirDrop in the Finder is the common source of such a storm on macOS.
|
||||
|
||||
### DNS configuration poll (macOS)
|
||||
|
||||
ctrld polls `scutil --dns` on macOS only. The first poll runs at start.
|
||||
The poll runs every 15 s for 10 minutes after a network change, a wake, or the end of a recovery.
|
||||
Each of these three events starts the fast cadence at once, even while the poll waits.
|
||||
At rest the poll runs every 5 minutes. Each changed resolver of a poll gives one `DNS configuration changed` event.
|
||||
|
||||
### Change-only lines
|
||||
|
||||
These lines log at debug level on change only. Each one carries `repeats`, the number of equal results before it:
|
||||
|
||||
- `Got system nameservers`
|
||||
- `Final available nameservers`
|
||||
- `pf anchor intact`
|
||||
- `discovered active tunnel interfaces`
|
||||
- `VPN DNS refresh completed`
|
||||
|
||||
### PF anchor list (macOS)
|
||||
|
||||
A `PF anchor list changed` event reaches the journal when its reason, its anchor list, or `pf_enabled` differs from the event before it.
|
||||
The field `pf_since` is no part of that test, because the uptime of pf grows between two reads of one state.
|
||||
A failed read of the anchor names gives `anchors_known` false and no anchor list.
|
||||
|
||||
### Host woke
|
||||
|
||||
ctrld writes one `Host woke` event per wake. Two sources that report the same wake inside 30 s give one event.
|
||||
The macOS wake detector runs in DNS intercept mode only. Outside that mode, and on Windows and Linux, the netmon delta is the only source.
|
||||
The netmon source knows no gap, so its report waits 8 s for the detector. A detector report inside that hold gives the event, with `gap_known` true and the length of the sleep in `gap_ms`. Without a detector report, the netmon report goes out with `gap_known` false and no `gap_ms`. Every later report of the same wake is dropped.
|
||||
|
||||
### Query health
|
||||
|
||||
ctrld grades the query path over a window of 15 minutes and reads the window every minute.
|
||||
Each failed client query counts one failure, whatever number of upstreams that query tried.
|
||||
`failures_by_class` keeps one count per failure class. The class `internal_domain` holds the failures of an Internal Domain resolver.
|
||||
The grade leaves the class `internal_domain` out, and `upstreams_down` counts no Internal Domain resolver.
|
||||
A laptop away from the company network cannot reach these resolvers. Without this rule the grade stays degraded for the whole day.
|
||||
A change of the class reports after it holds for two reads of the window.
|
||||
|
||||
## Transitions and recovery
|
||||
|
||||
`Network transition` debug events include a `transition_id`, affected interface, default route, and stored source addresses before and after the callback.
|
||||
The outcome identifies an accepted, ignored, superseded, or unusable change.
|
||||
`Network transition` events include a `transition_id`, affected interface, default route, and stored source addresses before and after the callback.
|
||||
The outcome identifies an accepted, ignored, superseded, snapshot_superseded, or unusable change.
|
||||
A callback whose state read a newer snapshot gives the outcome `snapshot_superseded` at debug level with `transition_id` 0, and it gives no interface event.
|
||||
Newer accepted changes supersede pending recovery. Ignored changes do not cancel accepted recovery.
|
||||
Source commits validate addresses against the current network state.
|
||||
Netmon does not order minor callbacks within one major cache epoch.
|
||||
@@ -22,8 +151,9 @@ This read does not change host DNS. A failed read defers source writes, but reco
|
||||
The warning gives the reason. Every warning enters the journal, so it stays on disk after a debug rotation and after a restart.
|
||||
`Recovery begin` and `Recovery end` connect the transition to a `recovery_generation`, reason, outcome, and elapsed time.
|
||||
Only the first upstream failure in each recovery produces a bounded summary.
|
||||
If a recovery encounters a failure, its final outcome also enters the journal.
|
||||
Normal successful recovery stays at debug level.
|
||||
Every recovery outcome enters the journal.
|
||||
A canceled recovery and a recovery with a failure log at warn level.
|
||||
For the outcome `superseded`, `intercept_target_action` is `unchanged` and `bypass_active` is false. The recovery that follows owns both of them.
|
||||
|
||||
These summaries exclude upstream URLs and queried names.
|
||||
Each PF probe records the recovery generation at its start.
|
||||
|
||||
@@ -289,6 +289,13 @@ The monitor probes with exponential backoff (0, 0.5, 1, 2, 4s) to win the race
|
||||
against async pf reloads. Only one monitor runs at a time (singleton). The
|
||||
watchdog also runs the probe every 30s as a safety net.
|
||||
|
||||
A network delta that touches AirDrop or virtual adapters alone is noise. The
|
||||
handler returns at once for it, so no probe monitor starts and no pf read runs.
|
||||
A delta that moves the default route, that flips the v4 or the v6 family, or
|
||||
that carries a time jump is not noise, whatever its interfaces are.
|
||||
The watchdog keeps its own 30 s cadence, so it still tests the rules during a
|
||||
delta storm.
|
||||
|
||||
The full pf reload is VPN-safe: it reassembles from `pfctl -sr` + `pfctl -sn`
|
||||
(the current running state), preserving all existing anchors and rules.
|
||||
|
||||
|
||||
@@ -45,12 +45,25 @@ The message of the line is `Log header`. The line carries these fields:
|
||||
- Mode: `intercept_mode`, `listeners`, `upstream_count`, and `upstream_types`
|
||||
- Identity: `resolver_uid`
|
||||
- Files: `log_files`
|
||||
- Network: `network` with `default_route_interface`, `have_v4`, `have_v6`, and `interfaces`
|
||||
- Network: `network`, the network snapshot of the host
|
||||
|
||||
The header holds no provisioning token, no upstream endpoint URL, and no query name. `upstream_types` holds the type words, for example `doh` or `os`. `listeners` holds the listen addresses.
|
||||
|
||||
The `network` object holds the routes, the gateways, the interfaces, the resolvers, the link type, and the intercept state. It has no `default_route_interface` field. The runbook [Network-recovery diagnostics](network-recovery-diagnostics.md) lists every field of the object.
|
||||
|
||||
After each rotation ctrld logs one `Log rotated` event at info level, with the fields `file`, `bytes_written`, `first_event_at`, `last_event_at`, and `backups`. This event is a journal event, so it lands in the debug stream and in the journal.
|
||||
|
||||
## Network state events
|
||||
|
||||
The journal holds the events that describe the host network. `Network snapshot` renders the `network` object at start, at each accepted transition, and at each begin and end of a recovery.
|
||||
A snapshot equal to the snapshot written last reaches no journal, and one snapshot per 60 s is the limit. A new default route and a new resolver set pass that limit. `Network interface changed`, `Network transition`, `Host woke`, `OS resolver set changed`, and `DNS configuration changed` name each change of the network. `Recovery begin`, `Recovery end`, `PF anchor list changed`, and `Tunnel interface changed` name the repair work.
|
||||
|
||||
AirDrop and virtual adapters produce a storm of network callbacks. ctrld puts these interfaces in a noise class. A delta that touches noise interfaces alone skips the handler. The first delta of a storm writes one `Network delta noise` journal line, and one more line covers each 10 minutes of the storm.
|
||||
|
||||
`Query health` grades the query path over a window of 15 minutes. It logs at each change between the classes `healthy`, `degraded`, and `failing`, and once every 15 minutes as a heartbeat. A class change reports after it holds for two reads of the window.
|
||||
|
||||
The runbook [Network-recovery diagnostics](network-recovery-diagnostics.md) lists every event with its trigger, its level, and its fields.
|
||||
|
||||
## Sampling of per-query errors
|
||||
|
||||
A broken network produces one error line per query. ctrld bounds this flood for each error class and each upstream. The first 5 lines in a 60 s window stay at error level. Later lines in the same window drop to debug level, so the debug stream stays complete. A run with `-v` holds every stream at info, and the debug stream then loses these lines.
|
||||
@@ -79,7 +92,7 @@ Both keys also apply to the `log_path` file, which rotates the same way. The jou
|
||||
- The marker `=== LOG_END ===`
|
||||
- Every journal file, oldest first
|
||||
|
||||
The `network` object of the send-time header comes from a fresh read of the interfaces and the default route. `ctrld log send --full` sends every debug file in place of the newest 10 MB. The reported size equals the number of bytes uploaded.
|
||||
The `network` object of the send-time header comes from a fresh read of the interfaces and of the route table. The resolvers come from the list that ctrld stored at its last resolver initialization, not from a fresh read. `ctrld log send --full` sends every debug file in place of the newest 10 MB. The reported size equals the number of bytes uploaded.
|
||||
|
||||
`ctrld log view` prints the same composition on the terminal and takes the same `--full` flag. ctrld accepts one upload per minute. The body is plain text without gzip.
|
||||
|
||||
|
||||
Reference in new issue
Block a user