docs: port startup DNS stall documentation

This commit is contained in:
Dev Scribe authored and Cuong Manh Le committed 2026-10-07 14:17:16 +07:00
1 parent f914105aec
commit 8d8fc54d23
2 files changed
+57

No files matched your search

+49
View File
@@ -45,6 +45,55 @@ This affects any VPN client that implements `block-outside-dns` via WFP, includi
---
### DNS Stalls for Up to ~100 Seconds After Start, and the Install Rolls Back
**Issue**: On Windows in `dns` intercept mode, a freshly started ctrld can leave every query unanswered for a minute or more. The start self-check fails in the meantime and rolls the install back (`SERVICE_SELFCHECK_FAILED`, exit 53). This happens with `ctrld start --config <file>` and also with `ctrld start --cd <uid>`.
**Status**: Fixed on the `v1.0` branch in `c40cc8d7`, not yet released. v1.5.7 is affected.
**Description**: The first DoH request sets up the upstream transport, and that setup calls `ctrld.HasIPv6()`. In the affected versions `HasIPv6()` also started a Tailscale `netmon` monitor inside its `sync.Once`. Building the monitor's first interface state calls `getPACWindows`, which asks WinHTTP for proxy auto-discovery (WPAD). WPAD discovery resolves names through the Windows DNS Client. NRPT already sends those lookups to ctrld, and ctrld cannot answer until the same `sync.Once` returns. So every DNS request waits behind proxy discovery, and proxy discovery waits on DNS, until WinHTTP's own timeouts expire. Then all pending queries complete at once.
It is a race between WinHTTP's proxy discovery and ctrld installing its NRPT rule:
- If discovery finishes before the rule is in place, nothing happens.
- With `--cd`, `HasIPv6()` is first called during the API fetch, only about 75 to 95 ms before the NRPT rule becomes active. That window is too short wherever discovery is slow, so `--cd` installs are affected too.
- With `--config`, nothing calls `HasIPv6()` before the rule, so the stall is near-certain.
- A network where the proxy-discovery lookups are answered quickly outside ctrld avoids it. On a domain member, for example, the AD domain's names go to the domain controller through the auto-added AD split rule.
- How long the stall lasts depends on how long WinHTTP's discovery takes on that network.
A goroutine dump from an affected build (`ecf7955e`) taken during the stall shows the DoH transport setup blocked in the syscall, with 111 DNS request goroutines parked in `sync.Once` behind it:
```
UpstreamConfig.dohTransport → ensureSetupTransport (sync.Once)
→ SetupTransport → ctrld.HasIPv6 (sync.Once)
→ netmon.New → netmon.GetState → getPACWindows [Windows syscall]
```
While the stall lasts, ctrld opens no connection to its upstream, although the network and the upstream both answer normally. Windows' certificate service lookups (`ctldl.windowsupdate.com`, Sectigo/USERTrust OCSP and CRL hosts) also queue in ctrld, but they are a consequence of the stall, not its cause. Warming Windows' certificate state before the start does not prevent it.
**Fix**: `c40cc8d7` ("put the network state of the host in the log journal") removed the monitor from `HasIPv6()`. The process's one network monitor now reports IPv6 state through `SetIPv6Available`, and `HasIPv6()` only runs a bounded IPv6 check. On master (2.0) the same change arrived with `c5ad04ab`.
**Workarounds** (affected versions):
1. **Recommended**: Add `--skip_self_checks` so the service is not rolled back. DNS then recovers by itself once WinHTTP's proxy discovery times out, after about 100 seconds in the QA lab.
2. **Alternative**: Retry the install. Whether it stalls depends on how quickly proxy discovery finishes, so a second attempt can succeed.
3. Take a build with `c40cc8d7` once one is released.
**Reproduction** (ctrld-qa lab, workgroup endpoint on the simulated home network):
- `config-file-upstream-ready` starts ctrld from a local config and requires Control D resolution within 20 seconds.
- It fails on every build that has the monitor in `HasIPv6()`: 10 of 10 runs, on the v1.5.7 release and tag, `ecf7955e`, and 2.0 commits before `c5ad04ab`.
- It passes on every build without the monitor: 9 of 9 runs, on `c40cc8d7`, the `v1.0` tip `cc849871`, and master from `6dcdddc2`.
- `smoke-lifecycle` is a normal `--cd` install.
- The v1.5.7 release failed its start self-check in 4 of 4 runs, each with the same signature: `HasIPv6()` 75 to 95 ms before the NRPT rule, then no upstream reply until the rollback.
- The `v1.0` tip `cc849871` passed.
- On the domain-joined endpoint, `--cd` installs pass.
**Affected Versions**: ctrld ≤ v1.5.7 on Windows, `dns` intercept mode. Near-certain with `--config`; with `--cd`, it depends on how quickly WinHTTP proxy discovery finishes on the network.
**Last Updated**: 09/30/2026
---
## Contributing to Known Issues
If you encounter an issue not listed here, please:
+8
View File
@@ -15,6 +15,14 @@ var (
)
// HasIPv6 reports whether the current network stack has IPv6 available.
//
// Keep the first call cheap and bounded. It runs on the DNS path (upstream
// transport setup) behind a sync.Once, so every query waits for it, and under
// Windows NRPT it can run after DNS already points at ctrld. v1.5.7 started a
// netmon monitor here; its first interface state runs WinHTTP proxy discovery,
// which needs DNS through this same ctrld, and a --config start stalled for
// about 100 s (docs/known-issues.md). Network monitoring belongs to the one
// monitor that reports through SetIPv6Available.
func HasIPv6(ctx context.Context) bool {
hasIPv6Once.Do(func() {
logger := LoggerFromCtx(ctx)