mirror of
https://github.com/Control-D-Inc/ctrld.git
synced 2026-10-12 01:58:46 +02:00
docs: port startup DNS stall documentation
This commit is contained in:
1 parent
f914105aec
commit
8d8fc54d23
2 files changed
+57
No files matched your search
@@ -45,6 +45,55 @@ This affects any VPN client that implements `block-outside-dns` via WFP, includi
|
||||
|
||||
---
|
||||
|
||||
### DNS Stalls for Up to ~100 Seconds After Start, and the Install Rolls Back
|
||||
|
||||
**Issue**: On Windows in `dns` intercept mode, a freshly started ctrld can leave every query unanswered for a minute or more. The start self-check fails in the meantime and rolls the install back (`SERVICE_SELFCHECK_FAILED`, exit 53). This happens with `ctrld start --config <file>` and also with `ctrld start --cd <uid>`.
|
||||
|
||||
**Status**: Fixed on the `v1.0` branch in `c40cc8d7`, not yet released. v1.5.7 is affected.
|
||||
|
||||
**Description**: The first DoH request sets up the upstream transport, and that setup calls `ctrld.HasIPv6()`. In the affected versions `HasIPv6()` also started a Tailscale `netmon` monitor inside its `sync.Once`. Building the monitor's first interface state calls `getPACWindows`, which asks WinHTTP for proxy auto-discovery (WPAD). WPAD discovery resolves names through the Windows DNS Client. NRPT already sends those lookups to ctrld, and ctrld cannot answer until the same `sync.Once` returns. So every DNS request waits behind proxy discovery, and proxy discovery waits on DNS, until WinHTTP's own timeouts expire. Then all pending queries complete at once.
|
||||
|
||||
It is a race between WinHTTP's proxy discovery and ctrld installing its NRPT rule:
|
||||
|
||||
- If discovery finishes before the rule is in place, nothing happens.
|
||||
- With `--cd`, `HasIPv6()` is first called during the API fetch, only about 75 to 95 ms before the NRPT rule becomes active. That window is too short wherever discovery is slow, so `--cd` installs are affected too.
|
||||
- With `--config`, nothing calls `HasIPv6()` before the rule, so the stall is near-certain.
|
||||
- A network where the proxy-discovery lookups are answered quickly outside ctrld avoids it. On a domain member, for example, the AD domain's names go to the domain controller through the auto-added AD split rule.
|
||||
- How long the stall lasts depends on how long WinHTTP's discovery takes on that network.
|
||||
|
||||
A goroutine dump from an affected build (`ecf7955e`) taken during the stall shows the DoH transport setup blocked in the syscall, with 111 DNS request goroutines parked in `sync.Once` behind it:
|
||||
|
||||
```
|
||||
UpstreamConfig.dohTransport → ensureSetupTransport (sync.Once)
|
||||
→ SetupTransport → ctrld.HasIPv6 (sync.Once)
|
||||
→ netmon.New → netmon.GetState → getPACWindows [Windows syscall]
|
||||
```
|
||||
|
||||
While the stall lasts, ctrld opens no connection to its upstream, although the network and the upstream both answer normally. Windows' certificate service lookups (`ctldl.windowsupdate.com`, Sectigo/USERTrust OCSP and CRL hosts) also queue in ctrld, but they are a consequence of the stall, not its cause. Warming Windows' certificate state before the start does not prevent it.
|
||||
|
||||
**Fix**: `c40cc8d7` ("put the network state of the host in the log journal") removed the monitor from `HasIPv6()`. The process's one network monitor now reports IPv6 state through `SetIPv6Available`, and `HasIPv6()` only runs a bounded IPv6 check. On master (2.0) the same change arrived with `c5ad04ab`.
|
||||
|
||||
**Workarounds** (affected versions):
|
||||
1. **Recommended**: Add `--skip_self_checks` so the service is not rolled back. DNS then recovers by itself once WinHTTP's proxy discovery times out, after about 100 seconds in the QA lab.
|
||||
2. **Alternative**: Retry the install. Whether it stalls depends on how quickly proxy discovery finishes, so a second attempt can succeed.
|
||||
3. Take a build with `c40cc8d7` once one is released.
|
||||
|
||||
**Reproduction** (ctrld-qa lab, workgroup endpoint on the simulated home network):
|
||||
|
||||
- `config-file-upstream-ready` starts ctrld from a local config and requires Control D resolution within 20 seconds.
|
||||
- It fails on every build that has the monitor in `HasIPv6()`: 10 of 10 runs, on the v1.5.7 release and tag, `ecf7955e`, and 2.0 commits before `c5ad04ab`.
|
||||
- It passes on every build without the monitor: 9 of 9 runs, on `c40cc8d7`, the `v1.0` tip `cc849871`, and master from `6dcdddc2`.
|
||||
- `smoke-lifecycle` is a normal `--cd` install.
|
||||
- The v1.5.7 release failed its start self-check in 4 of 4 runs, each with the same signature: `HasIPv6()` 75 to 95 ms before the NRPT rule, then no upstream reply until the rollback.
|
||||
- The `v1.0` tip `cc849871` passed.
|
||||
- On the domain-joined endpoint, `--cd` installs pass.
|
||||
|
||||
**Affected Versions**: ctrld ≤ v1.5.7 on Windows, `dns` intercept mode. Near-certain with `--config`; with `--cd`, it depends on how quickly WinHTTP proxy discovery finishes on the network.
|
||||
|
||||
**Last Updated**: 09/30/2026
|
||||
|
||||
---
|
||||
|
||||
## Contributing to Known Issues
|
||||
|
||||
If you encounter an issue not listed here, please:
|
||||
|
||||
Reference in new issue
Block a user