TestOSRecoverySkipSurvivesDebugTruncation passed time.Now() as the
argument after readLogReader, and Go evaluates arguments left to right,
so the lower bound was taken after the upload header was rendered. The
header check failed whenever a second boundary fell between the two:
header time = 11:13:44, want a time between 11:13:45.007 and now
Take the bound first, as the other splitUpload callers do.
Internal Domains now support three modes. "os" (network default) is
unchanged. "resolvers" becomes explicit resolver with network fallback,
the default explicit selection: the configured resolvers are tried
first, and a timeout, unreachable resolver, SERVFAIL, NXDOMAIN, REFUSED
or NOTIMP hands the query to the network resolvers - matching VPN DNS
servers, then domain-less VPN DNS servers, then the OS resolver's LAN
nameservers. A valid empty answer stays final. "resolvers_only" keeps
the previous strict behavior.
The OS step uses a new LanOnlyQueryCtx: the OS resolver then asks only
its LAN-classified nameservers (private, loopback, link-local, CGNAT)
and sends nothing when it has none, so the private name never reaches a
public nameserver, whether DHCP supplied it or it is ctrld's own public
fallback. LanQueryCtx only dropped the latter. LAN-only queries get
their own singleflight and hot-cache key, so they are never answered
with an ordinary query's public answer.
In fallback mode a configured resolver that the upstream monitor has
marked down is skipped instead of waited on, so an endpoint off the
organization network reaches the fallback without paying the
resolver's 2s timeout on every query. Nothing else marks a generated
resolver up again, so skipping one re-checks it in the background, at
most once every 30s; an answer marks it up. Strict mode never skips.
An absent mode with resolvers gets the fallback mode. An unrecognized
mode with resolvers fails closed to explicit resolver only, so a wire
value this build does not know never sends the domain to the Control D
upstream; without resolvers it is dropped. Switching between the
explicit modes is detected on refresh.
Generated upstreams are now identified by an UpstreamConfig.InternalDomain
marker that records their mode and that no configuration file can set,
instead of by the internal_ key prefix. An upstream a local ctrld.toml or
custom config defines as internal_foo, or as a copy of a generated one,
is an ordinary upstream again: it keeps the OS-resolver catch-all,
recovery and query-health grading.
The "resolvers_only" wire value is provisional until the API agrees on
it; the dashboard does not offer strict mode yet.
Decode networksetup entries once for interface lookup and DNS-target
selection. Preserve first-enabled lookup and fail-closed DNS uniqueness
as separate policies. Treat only the (*) index as disabled, preserving
asterisks and whitespace in enabled service names.
Add shared-parser, caller-policy and incomplete-read regressions.
captureDebugMainLog swaps the process-wide mainLog, so every
mainLog.Load().Warn()/.Error() site in the package writes into the
buffer while it is installed, including goroutines left running by
other tests. retainedProbeLines returned every retained line and the
tests asserted field values over all of them, so one unrelated warn
failed the run.
Test_checkDnsLoopKeepsACustomKeyOutOfTheJournal hit this on Windows,
where the loop probe held the buffer open for two seconds:
upstream_probe_journal_test.go: field "upstream": got , want upstream.custom
The sibling tests were exposed the same way; they index retained[0]
after a strict length check, so a stray line fails them as got 2, want 1.
Let retainedProbeLines take the messages a caller owns and return only
those lines. wantNoProbeSecret keeps scanning every retained line,
which is what that leak check is for.
checkDnsLoop resolved with context.Background() and no dns.Client
timeout, so the DNS client fell back to its own 2s default and the
configured upstream timeout was ignored. The probes run serially on a
one-minute ticker, so each unreachable local upstream stalled the loop
for a fixed 2s no matter what the config asked for.
This is most visible on Windows, where UDP to a closed loopback port
does not surface an immediate refusal the way it does on Linux, so the
probe waits out the full default.
Apply the upstream timeout the way checkUpstreamOnce already does, and
keep 2s for an upstream that configures none so the unconfigured case
is unchanged.
The probe context deliberately carries no logger. A resolver logs its
own failure at error level with the endpoint in the message, which
would place an Internal Domain resolver address in the retained
journal; logUpstreamProbeFailure remains the reporter and bounds what
the line may hold. TestInternalDomainsLoopCheckHidesResolverAddress
covers this.
Use Write rather than Stat to verify retained file handles are closed.
Windows Stat returns ERROR_INVALID_HANDLE instead of os.ErrClosed.
Retain final-cleanup content assertions and strict closed-error matching.
The genuine-failure cases fed a fabricated error straight to
logIfaceLookupFailure, so they exercised only its own branching. A
restoration failure never reaches that helper, and no test reached the
enumeration classification in netInterface at all: discarding the
enumeration error, or downgrading an actual restoration failure to
debug, both kept the suite green.
The host boundaries of the reset path are now variables, in the style
the package already uses: the interface enumeration, the interface
lookup, the DNS setters and the NetworkManager restore. The tests stub
them, so a run changes no host DNS or NetworkManager state, and drive
the real netInterface and resetDNSForRunningIface to assert that a
failed enumeration keeps its cause and is not reported as a missing
interface, and that a failure to restore the saved static config or to
reset to DHCP on an interface that does exist stays at error level.
Both surviving mutations, and three more, now fail.
A successful macOS upgrade printed "could not get interface
error=interface not found iface=en5" before reporting success, because
the DNS cleanup logged every interface lookup failure at error level.
An interface that is gone, an unplugged adapter or a torn down tether,
has no DNS settings left to restore, so the skip is now a debug
diagnostic naming the interface and the work it skipped.
Only that condition is quiet. netInterface returns a sentinel error for
a missing interface and no longer reports a failed enumeration as one,
so any other lookup failure and any restoration failure on an interface
that does exist stay at error level.
The journal records the network state of the host and each change of
that state. The support team reads one file to see the interfaces, the
routes, the resolvers, and the recoveries around an incident. The same
file shows the health of the query path. A change that touches only
AirDrop or virtual interfaces runs no DNS work.
ctrld keeps a second log file, ctrld-journal.log, next to ctrld.log. The
journal holds all warnings and errors and each event with the field
journal=true. It stays on disk after a start of the service and after a
self-upgrade. Each log file starts with a Log header line. Each log send
upload starts with the same line, made at send time, and has a limit of
about 16 MB.
A failed handback restored ctrld's rule via addNRPTCatchAllRule, which
also writes the GP-path rule whenever another GP rule exists - leaving
CtrldCatchAll beside the administrator's catch-all. Restore only the
local store while that child is still present.
Consume resolver.split_dns and generate a policy rule per domain, plus its
wildcard, so a suffix and its subdomains route to the OS resolver when no
resolvers are given, or to the configured resolvers when they are.
Explicit resolvers are exclusive: when all of them fail the query returns
SERVFAIL rather than taking the OS catch-all or starting recovery, and it
skips VPN split routing, which is auto-detected. Excludes and custom config
keep precedence; a refresh regenerates the config, so removals leave no
stale rule or upstream.
Rule text survives a suspend unchanged, so the watchdog reports the pf
anchor intact while nothing translates, and the probe monitor is only
armed by interface changes with a schedule frozen through the sleep.
Detect the resume from the wall-clock gap and probe from there, with a
bounded repair: at most two forced reloads across the retry schedule.
Refresh the OS resolver before every probe - the probe aims at the first
OS nameserver and passes when it has none, so a pre-sleep list makes it
meaningless. Stand down for stabilization, which verifies on completion.
macOS intercept mode fails totally on networks that provide no usable
IPv4 DNS (e.g. IPv6-only iPhone tethering with 464XLAT): the pf ruleset
blocks all outbound IPv6 port 53, and with no IPv4 DNS configured
mDNSResponder emits no DNS packets at all, so pf has nothing to
intercept while the Control D upstream stays provably healthy.
When network-change recovery discovers no usable IPv4 DNS on the
default-route service, set 127.0.0.1 as that service DNS so macOS can
emit queries that land directly on the listener. The entry is removed
when the network regains IPv4 DNS and on intercept shutdown; networks
that provide IPv4 DNS are never modified. Runs on the already-debounced
recovery path so interface flaps do not churn networksetup.
Also fix the canceled-recovery state leak: the cancellation early
return never reset recoveryBypass/recoveryRunning, so a flap burst
ending in a canceled recovery left the daemon in bypass forever with
the DNS watchdog disabled. Cleanup is generation-gated so a superseded
recovery never clears state owned by its successor.
tryUpdateListenerConfig treated an explicit --intercept-mode off the same
as an empty flag and fell back to the persisted config value. run() selects
the listener strategy before it clears the persisted mode, so the first
start after a revert to standard mode selected the intercept strategy from
a stale dns/hard value while setDNS kept interception off.
Extract the resolution into listenerInterceptMode and make an explicit off
final, the same contract as setDNS. Add a regression test that fails
without the fix.
The warning reported err, the resolver-config fetch error, which is nil on
every path that reaches it - so a rejected custom config was logged with no
reason attached. cfgErr holds the validation failure.
"ctrld status" reported the service manager's view and nothing else, so it
printed "Service is running" and exited 0 for a process that was alive and
registered as started but had never got past startup: no control socket, no DNS
listener, no policy applied. The one command an operator reaches for first
confirmed the service was fine while the host had no working DNS.
Probe the control server's /started endpoint before reporting success. That
endpoint only answers once the onStarted hooks have completed, which is after the
listeners are up, so a successful probe means the process is serving rather than
merely alive. A service that is registered as running but cannot confirm startup
is now reported as such, with a pointer to the log, and exits 3 - distinct from
stopped (1) and unknown (2), because it needs a different response.
A probe blocked by permissions is not evidence of a broken service: an
unprivileged caller still gets "Service is running", with a note that startup was
not verified. The probe is bounded by a short timeout so status stays fast.
Document the exit codes in the command's help, and cover the probe (ready, not
finished starting, no socket, timed out) and the classification, including that
an unreadable socket is not reported as a failure.
The not-ready verdict is only reported when the probe could have found the
daemon's socket. socketDir() is caller-relative on unix - the system directory
when writable, the caller's home otherwise - so an unprivileged "ctrld status"
looks somewhere the root-owned daemon never listened and gets ENOENT, which is
"wrong path", not "not ready". Since only darwin has an elevation PreRun and the
root-level alias has none, that is the normal invocation; reporting exit 3 there
would have told a monitoring check to restart healthy daemons. Such a caller now
gets the service manager's view with startup reported as unverified. Windows and
mobile resolve the same directory for every caller, so the verdict stays fully
available on the platform the hung start was seen on. A successful probe is still
conclusive whoever ran it.
Rollback ran os.Remove(bin) while the replacement service was still running from
that image. Windows locks a running executable, so the remove failed with
"Access is denied" - and it was fatal, so the os.Rename that restores the
previous binary never ran. The upgrade ended with the broken replacement still
installed and the working binary stranded at its _previous name.
Readiness failing is not evidence the process exited: the service manager can
report a started service whose process never became operational. So rollback now
stops the service and waits until the manager reports it stopped before touching
the executable, then cleans up DNS the way the restart path's Cleanup task does.
Restoring is now conditional on the previous binary reporting a version, since a
_previous file that exists but produces no version output would trade a service
that starts and hangs for one that cannot start at all. When it is unusable,
rollback keeps it for inspection, leaves the installed binary alone, and says so
instead of pressing on. The --version probe is bounded by a timeout so a binary
that hangs cannot hang the upgrade.
Remaining failures are reported rather than fatal, so each one says what state
the host was left in. os.Remove is retried while the path stays locked, since
Windows releases an image lock asynchronously after the process exits.
The helpers live in a new file rather than in commands.go, and the rollback is
extracted into rollbackToPreviousBinary() so it can be covered: the stop happens
while the executable is still present, an unusable previous binary is kept
without swapping or restarting, and a failed stop aborts before anything is
modified. Reversing the stop and the remove fails these tests.
The version probe is called through a variable so those tests do not have to
stage a runnable executable. Staging one is not portable: oldBin is
bin+"_previous", so a fixture named "ctrld" yields the extension-less
"ctrld_previous", which Windows refuses to execute, and a symlink to the test
binary needs a privilege Windows does not grant by default. The probe itself is
still covered against the real test binary. Production is unaffected: ctrld.exe
_previous does have an extension, and os/exec only appends PATHEXT entries when
a path has none at all - noted at binaryVersion so the suffix is not renamed
into something extension-less by accident.
Both API requests and binary downloads retry against a hard-coded IP when the
attempt via hostname fails. Both then overwrote the first error with the
fallback's, so only the last failure was reported.
That discarded the diagnosis. When the hostname attempt is denied locally -
WSAEACCES on Windows, "An attempt was made to access a socket in a way forbidden
by its access permissions", which means the host is blocking ctrld - and the
direct-ip fallback fails with an unreachable IPv6 route, what surfaces to the
operator is "dial tcp6: no route to host": a routing problem that does not
exist, while the error naming the real cause is visible only in debug logs.
Report both failures instead, keeping the error chain intact so errors.Is still
matches either one. Also switch the final wrap in doWithRetry from %v to %w,
which had been flattening the chain even when a single error was reported.
This also changes retry classification, which is worth stating explicitly because
it is not obvious from "report both errors". processCDFlags decides whether to
keep backing off with errUrlNetworkError, which uses errors.As - and errors.As
returns the *first* match in the tree. Wrapping the hostname attempt first
therefore hands the predicate that attempt's failure, where previously only the
fallback's error survived to be classified.
The effect is intended. A locally denied socket (WSAEACCES) is not a transient
network error, so preflight now fails fast and reports instead of retrying
against a firewall that is not going to clear on its own. The case that justifies
retrying forever, a network unreachable on both attempts at boot, is unchanged.
Both classifications are pinned by tests, along with the wrap order they depend on
at each composition site, so reversing it fails loudly rather than silently
restoring the old behaviour.
processCDFlags retries the resolver-config fetch indefinitely by design: a
device that has no working network at boot must eventually come up. The loop had
no cancellation, so a stop request arriving while the API is unreachable was
ignored - the process kept retrying long after the service reported itself
stopped, doing work on behalf of a service the OS considers stopped.
Thread a context through processCDFlags and derive it from p.stopCh, in both the
startup preflight and the config-reload path. The loop now returns as soon as the
context is cancelled, checked both before a retry and after backoff returns
(backoff can wake up on cancellation). A stop during preflight exits the way a
normal stop does, without Fatal, so the service manager does not treat it as a
failed start and apply its restart policy to a service the operator just asked
to stop.
Bind the two API requests themselves as well, so a stop does not have to wait out
an in-flight request. Without this the loop honours a stop only between attempts,
which leaves up to defaultTimeout (20s) of a request the service is no longer
interested in - the same "still working after Service stopped" the loop change
exists to end, one layer down.
Doing so means a context parameter on FetchResolverConfig, FetchResolverUID,
UpdateCustomLastFailed and SendLogs, since all four reach a request builder. The
callers that have no context pass context.Background(), which is what master
effectively does at those sites: its loggerCtx carries a logger, not
cancellation. doWithFallback needs no parameter, because it clones the request
with req.Context() and so inherits the binding. This also repairs
internal/controld/controld_test.go, which is behind //go:build controld and had
already been written against the context-taking signature, so it could not
compile.
Cover the cancellation paths; removing either check makes the tests hang until
timeout.
Sampling the stop state is the whole point of runAPIPreflight rather than doing
this inline. A stop and a failure need opposite handling - one exits quietly, the
other self-uninstalls a deleted device, surfaces the error to a mobile app, and
reports a failed start - so the two must not be confused. Reading it from the
context after cancelling would report "stopped" for every failure, since
CancelFunc sets ctx.Err() regardless of whether anyone asked to stop; the stop
channel is read directly instead, which also does not depend on the context
watcher goroutine having been scheduled.
The post-stabilization reconcile verifies rule text, which cannot tell a
live redirect from an anchor pf has stopped evaluating. Sleep/wake QA
caught exactly that split: references intact, anchor rules intact,
post-load verification passed, and every query through the system
resolver timing out while the direct listener answered.
Nothing else probed. The interception probe monitor stands down while
stabilization owns pf and is never re-armed afterwards, so functional
recovery waited for the periodic watchdog - 11 seconds in the captured
run, up to a full 30-second interval - on a host whose link and default
route were already back. The watchdog's probe then failed once, forced a
reload, and public and VPN split-DNS both recovered immediately.
Probe once at the end of stabilization and, if it fails, force exactly
one reload and confirm with one more probe. Not the probe monitor: that
keeps probing for ~7.5s and can force a reload per failed probe, where
this path needs a single bounded repair before handing back to the
watchdog. Skipped when a monitor already owns probing, when intercept
state is gone, or during exec backoff.
Hand ownership over deterministically rather than skipping on sight. A
probe monitor started by an ignored network change claimed
functional-probe ownership before checking whether it could work, then
stood down because stabilization still owned pf; the verifier read that
claimed flag as "somebody is probing" and skipped, so neither path
probed and recovery fell back to the watchdog anyway. The monitor now
checks eligibility before claiming, and the verifier waits out a holder
that releases, yielding only to one that keeps probing.
Extract the completion block into finishPFStabilization so the wiring is
testable, and cover the bounded repair, the healthy path that must not
reload, a prober that claims and stands down, a prober that keeps
working, and a monitor that must not claim ownership while stabilizing.
- Skip systemd-resolved initialization on Android (ChromeOS crash fix)
- Skip system DNS discovery commands on iOS mobile (sandbox restrictions)
- Skip route-based DNS discovery on Android
- Skip systemd resolver on Android
- Add mobile platform checks to prevent sandbox access violations
These changes ensure ctrld works correctly in mobile sandboxed environments
where system commands and file access are restricted.
Running with --silent still created and grew log files in cd mode.
needInternalLogging() only checked cdUID and Service.LogPath, so a
--silent flag enabled internal logging, persisted it to disk, and
reset the global log level back to debug, overriding the NoLevel that
--silent had set.
Return false from needInternalLogging() when silent is set, so ctrld
neither creates the internal log file nor writes debug logs. Add
regression tests asserting needInternalLogging() is false in silent mode
and that initInternalLogging() creates no log file.
Refs https://github.com/Control-D-Inc/ctrld/issues/320
Defense in depth against cache poisoning: a compromised or misbehaving
upstream can return an answer for a different name than was asked (e.g.
records for attacker.example in response to a query for victim.example).
Such an answer would be cached under the legitimate request key and
served to subsequent queries.
Validate that the upstream answer echoes the request's question
(case-insensitive name plus Qtype/Qclass, per RFC 1035 section 4.1.2)
before serving or caching it. A mismatch is logged at debug level and
the upstream is skipped, failing safe to the next upstream or SERVFAIL.
Refs github.com/Control-D-Inc/ctrld/issues/322
On networks using 464XLAT (common on IPv6-only cellular carriers and iPhone
hotspots), the local machine's DNS queries can reach ctrld's listener with a
source address in the RFC 7335 IPv4 Service Continuity Prefix (192.0.0.0/29,
e.g. 192.0.0.2 on the CLAT/host side). isWanClient classified 192.0.0.x as a
WAN client, so with allow_wan_clients unset (the default) the query was refused,
breaking DNS resolution entirely on the affected connection even though it
originated from the local host.
Recognize the IPv4 Service Continuity Prefix as a local range, mirroring the
existing CGNAT special case:
- Add ipv4ServiceContinuityPrefix (192.0.0.0/29) and an isServiceContinuityAddr
helper (single definition, reused by both call sites).
- isWanClient excludes the range, so 464XLAT/CLAT queries are served normally.
- isPrivatePtrLookup treats the range as private so reverse lookups are handled
consistently.
Scoped to 192.0.0.0/29 (the exact 464XLAT range); this does not weaken
allow_wan_clients since no globally routable remote client can appear from it.
On macOS DNS-intercept mode, when mDNSResponder owns *:53 ctrld falls back
to listening on 127.0.0.1:5354, and the pf rdr rules correctly redirect DNS
to the bound port at startup. However, DNS resolution later breaks with an
endless watchdog "anchor intact but probe FAILED -> force reload" loop and
`dig @127.0.0.1` timeouts.
Root cause is config reload. In CD mode, apiConfigReload refetches the
generated config (which always declares port 53) every hour and the reload
merge in runWait only inherits the running port when the new port is 0. The
generated config explicitly says 53, so `*p.cfg = *newCfg` reverts p.cfg to
port 53. The DNS listener goroutines are started only when !reload, so they
are never re-bound and stay on 5354. Every subsequent pf rebuild reads p.cfg
and targets the dead port 53.
Fix: after applying the reloaded config in DNS-intercept mode on darwin,
restore the actual bound listener IP/Port into the in-memory config via the
new preserveBoundListeners helper, logging the configured-vs-actual
divergence. A reload cannot move the running listener anyway, so this keeps
p.cfg consistent with reality; all pf rdr rules and the watchdog probe then
target the live port. The on-disk generated config is intentionally left
unchanged (still 53), so no generated-config change is required.
Windows DNS intercept mode runs an NRPT health monitor that restores the
catch-all rule and re-signals DNS Client whenever Windows stops routing
queries to the local listener. When another agent (MDM, VPN, GPO) keeps
putting NRPT back into a broken state, that loop never converges: ctrld
repeatedly calls RefreshPolicyEx, Dnscache paramchange, and flushes the
DNS cache, producing continuous flash writes and SIEM noise while never
fixing anything.
Add a recovery limiter that trips after a configurable number of
consecutive recovery flows and enters a cooldown, during which recovery
is suppressed (logged at most once every 5 minutes). Clearing the
circuit requires two consecutive stable health successes rather than
one, because a probe can pass briefly right after delete/re-add even
when the underlying NRPT state is still broken.
New [service] options gate the behavior and default to the previous
unlimited behavior:
- nrpt_recovery_max_attempts (default 0 = unlimited)
- nrpt_recovery_cooldown (default 30m)
Also collapse the repeated refresh + paramchange + flush sequence into a
single signalNRPTChange() helper, and make cleanGPPath /
cleanEmptyNRPTParent only mutate the registry and report whether cleanup
happened, so callers send exactly one DNS Client change signal instead
of several. When the GP DnsPolicyConfig parent exists but is empty,
nrptProbeAndHeal now cleans it and signals once before spending the
normal policy-refresh retry budget, since those retries cannot succeed
while DNS Client is stuck in GP mode.
Add unit tests for the limiter's cooldown, stable-reset, and unlimited
paths, and document the new options and the empty-GP repro.
When IPv6 is available locally but the selected IPv6 DoH endpoint is
unroutable (e.g. dialing [2606:1a40::22]:443 returns "no route to host"
while IPv4 stays usable), ctrld re-bootstrapped and re-dialed the endpoint
every ~2s. A weekend soak produced ~46.7k "no route to host" lines, with
the dial/health-check loop dominating the log during bad windows.
Add bounded backoff/suppression for network-unreachable endpoints at two
levels:
- ParallelDialer (internal/net): track dial addresses that fail with
ENETUNREACH/EHOSTUNREACH and skip them for an exponentially growing,
bounded window (5s -> 60s). A successful dial clears the entry
immediately, so recovery is preserved when the route returns. When every
candidate is suppressed the dial fails fast and quietly instead of
hammering known-unroutable addresses.
- Upstream recovery loop (cmd/cli): demote unreachable check failures to
debug and back off the retry cadence (2s -> 60s) for an unreachable
streak; any other failure resets to the base cadence.
The new IsUnreachable classifier lives in internal/net and is reused by
cmd/cli's errNetworkError, so the unreachable-errno matching has a single
definition. Note the explicit winsock constants (10051/10065) are required
on Windows: syscall.ENETUNREACH/EHOSTUNREACH are Go's portable "invented"
values and never equal the raw WSA codes a failing connect surfaces.
Suppression and backoff are always bounded, so IPv6 is never disabled until
restart and recovers on its own once the route is back. Split-stack
selection and the #549 macOS intercept recovery work are untouched.
Adds unit tests for the classifier, the dialer's suppression tracker, and
the recovery backoff schedule.
Cuong Manh Le