diff --git a/docs/known-issues.md b/docs/known-issues.md index 4ce2f5f..7c75701 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -45,6 +45,55 @@ This affects any VPN client that implements `block-outside-dns` via WFP, includi --- +### DNS Stalls for Up to ~100 Seconds After Start, and the Install Rolls Back + +**Issue**: On Windows in `dns` intercept mode, a freshly started ctrld can leave every query unanswered for a minute or more. The start self-check fails in the meantime and rolls the install back (`SERVICE_SELFCHECK_FAILED`, exit 53). This happens with `ctrld start --config ` and also with `ctrld start --cd `. + +**Status**: Fixed on the `v1.0` branch in `c40cc8d7`, not yet released. v1.5.7 is affected. + +**Description**: The first DoH request sets up the upstream transport, and that setup calls `ctrld.HasIPv6()`. In the affected versions `HasIPv6()` also started a Tailscale `netmon` monitor inside its `sync.Once`. Building the monitor's first interface state calls `getPACWindows`, which asks WinHTTP for proxy auto-discovery (WPAD). WPAD discovery resolves names through the Windows DNS Client. NRPT already sends those lookups to ctrld, and ctrld cannot answer until the same `sync.Once` returns. So every DNS request waits behind proxy discovery, and proxy discovery waits on DNS, until WinHTTP's own timeouts expire. Then all pending queries complete at once. + +It is a race between WinHTTP's proxy discovery and ctrld installing its NRPT rule: + +- If discovery finishes before the rule is in place, nothing happens. +- With `--cd`, `HasIPv6()` is first called during the API fetch, only about 75 to 95 ms before the NRPT rule becomes active. That window is too short wherever discovery is slow, so `--cd` installs are affected too. +- With `--config`, nothing calls `HasIPv6()` before the rule, so the stall is near-certain. +- A network where the proxy-discovery lookups are answered quickly outside ctrld avoids it. On a domain member, for example, the AD domain's names go to the domain controller through the auto-added AD split rule. +- How long the stall lasts depends on how long WinHTTP's discovery takes on that network. + +A goroutine dump from an affected build (`ecf7955e`) taken during the stall shows the DoH transport setup blocked in the syscall, with 111 DNS request goroutines parked in `sync.Once` behind it: + +``` +UpstreamConfig.dohTransport → ensureSetupTransport (sync.Once) + → SetupTransport → ctrld.HasIPv6 (sync.Once) + → netmon.New → netmon.GetState → getPACWindows [Windows syscall] +``` + +While the stall lasts, ctrld opens no connection to its upstream, although the network and the upstream both answer normally. Windows' certificate service lookups (`ctldl.windowsupdate.com`, Sectigo/USERTrust OCSP and CRL hosts) also queue in ctrld, but they are a consequence of the stall, not its cause. Warming Windows' certificate state before the start does not prevent it. + +**Fix**: `c40cc8d7` ("put the network state of the host in the log journal") removed the monitor from `HasIPv6()`. The process's one network monitor now reports IPv6 state through `SetIPv6Available`, and `HasIPv6()` only runs a bounded IPv6 check. On master (2.0) the same change arrived with `c5ad04ab`. + +**Workarounds** (affected versions): +1. **Recommended**: Add `--skip_self_checks` so the service is not rolled back. DNS then recovers by itself once WinHTTP's proxy discovery times out, after about 100 seconds in the QA lab. +2. **Alternative**: Retry the install. Whether it stalls depends on how quickly proxy discovery finishes, so a second attempt can succeed. +3. Take a build with `c40cc8d7` once one is released. + +**Reproduction** (ctrld-qa lab, workgroup endpoint on the simulated home network): + +- `config-file-upstream-ready` starts ctrld from a local config and requires Control D resolution within 20 seconds. + - It fails on every build that has the monitor in `HasIPv6()`: 10 of 10 runs, on the v1.5.7 release and tag, `ecf7955e`, and 2.0 commits before `c5ad04ab`. + - It passes on every build without the monitor: 9 of 9 runs, on `c40cc8d7`, the `v1.0` tip `cc849871`, and master from `6dcdddc2`. +- `smoke-lifecycle` is a normal `--cd` install. + - The v1.5.7 release failed its start self-check in 4 of 4 runs, each with the same signature: `HasIPv6()` 75 to 95 ms before the NRPT rule, then no upstream reply until the rollback. + - The `v1.0` tip `cc849871` passed. +- On the domain-joined endpoint, `--cd` installs pass. + +**Affected Versions**: ctrld ≤ v1.5.7 on Windows, `dns` intercept mode. Near-certain with `--config`; with `--cd`, it depends on how quickly WinHTTP proxy discovery finishes on the network. + +**Last Updated**: 09/30/2026 + +--- + ## Contributing to Known Issues If you encounter an issue not listed here, please: diff --git a/net.go b/net.go index 27dd75b..b20d85a 100644 --- a/net.go +++ b/net.go @@ -15,6 +15,14 @@ var ( ) // HasIPv6 reports whether the current network stack has IPv6 available. +// +// Keep the first call cheap and bounded. It runs on the DNS path (upstream +// transport setup) behind a sync.Once, so every query waits for it, and under +// Windows NRPT it can run after DNS already points at ctrld. v1.5.7 started a +// netmon monitor here; its first interface state runs WinHTTP proxy discovery, +// which needs DNS through this same ctrld, and a --config start stalled for +// about 100 s (docs/known-issues.md). Network monitoring belongs to the one +// monitor that reports through SetIPv6Available. func HasIPv6(ctx context.Context) bool { hasIPv6Once.Do(func() { logger := LoggerFromCtx(ctx)