TestOSRecoverySkipSurvivesDebugTruncation passed time.Now() as the
argument after readLogReader, and Go evaluates arguments left to right,
so the lower bound was taken after the upload header was rendered. The
header check failed whenever a second boundary fell between the two:
header time = 11:13:44, want a time between 11:13:45.007 and now
Take the bound first, as the other splitUpload callers do.
Internal Domains now support three modes. "os" (network default) is
unchanged. "resolvers" becomes explicit resolver with network fallback,
the default explicit selection: the configured resolvers are tried
first, and a timeout, unreachable resolver, SERVFAIL, NXDOMAIN, REFUSED
or NOTIMP hands the query to the network resolvers - matching VPN DNS
servers, then domain-less VPN DNS servers, then the OS resolver's LAN
nameservers. A valid empty answer stays final. "resolvers_only" keeps
the previous strict behavior.
The OS step uses a new LanOnlyQueryCtx: the OS resolver then asks only
its LAN-classified nameservers (private, loopback, link-local, CGNAT)
and sends nothing when it has none, so the private name never reaches a
public nameserver, whether DHCP supplied it or it is ctrld's own public
fallback. LanQueryCtx only dropped the latter. LAN-only queries get
their own singleflight and hot-cache key, so they are never answered
with an ordinary query's public answer.
In fallback mode a configured resolver that the upstream monitor has
marked down is skipped instead of waited on, so an endpoint off the
organization network reaches the fallback without paying the
resolver's 2s timeout on every query. Nothing else marks a generated
resolver up again, so skipping one re-checks it in the background, at
most once every 30s; an answer marks it up. Strict mode never skips.
An absent mode with resolvers gets the fallback mode. An unrecognized
mode with resolvers fails closed to explicit resolver only, so a wire
value this build does not know never sends the domain to the Control D
upstream; without resolvers it is dropped. Switching between the
explicit modes is detected on refresh.
Generated upstreams are now identified by an UpstreamConfig.InternalDomain
marker that records their mode and that no configuration file can set,
instead of by the internal_ key prefix. An upstream a local ctrld.toml or
custom config defines as internal_foo, or as a copy of a generated one,
is an ordinary upstream again: it keeps the OS-resolver catch-all,
recovery and query-health grading.
The "resolvers_only" wire value is provisional until the API agrees on
it; the dashboard does not offer strict mode yet.
Decode networksetup entries once for interface lookup and DNS-target
selection. Preserve first-enabled lookup and fail-closed DNS uniqueness
as separate policies. Treat only the (*) index as disabled, preserving
asterisks and whitespace in enabled service names.
Add shared-parser, caller-policy and incomplete-read regressions.
captureDebugMainLog swaps the process-wide mainLog, so every
mainLog.Load().Warn()/.Error() site in the package writes into the
buffer while it is installed, including goroutines left running by
other tests. retainedProbeLines returned every retained line and the
tests asserted field values over all of them, so one unrelated warn
failed the run.
Test_checkDnsLoopKeepsACustomKeyOutOfTheJournal hit this on Windows,
where the loop probe held the buffer open for two seconds:
upstream_probe_journal_test.go: field "upstream": got , want upstream.custom
The sibling tests were exposed the same way; they index retained[0]
after a strict length check, so a stray line fails them as got 2, want 1.
Let retainedProbeLines take the messages a caller owns and return only
those lines. wantNoProbeSecret keeps scanning every retained line,
which is what that leak check is for.
checkDnsLoop resolved with context.Background() and no dns.Client
timeout, so the DNS client fell back to its own 2s default and the
configured upstream timeout was ignored. The probes run serially on a
one-minute ticker, so each unreachable local upstream stalled the loop
for a fixed 2s no matter what the config asked for.
This is most visible on Windows, where UDP to a closed loopback port
does not surface an immediate refusal the way it does on Linux, so the
probe waits out the full default.
Apply the upstream timeout the way checkUpstreamOnce already does, and
keep 2s for an upstream that configures none so the unconfigured case
is unchanged.
The probe context deliberately carries no logger. A resolver logs its
own failure at error level with the endpoint in the message, which
would place an Internal Domain resolver address in the retained
journal; logUpstreamProbeFailure remains the reporter and bounds what
the line may hold. TestInternalDomainsLoopCheckHidesResolverAddress
covers this.
Use Write rather than Stat to verify retained file handles are closed.
Windows Stat returns ERROR_INVALID_HANDLE instead of os.ErrClosed.
Retain final-cleanup content assertions and strict closed-error matching.
NextDNS serves alternative DoH endpoints beside dns.nextdns.io: the
ultralow and anycast variants of dns, dns1 and dns2. They are the same
service and take the same client info headers, but isNextDNS matched
the one host, so an upstream pointed at any of them was treated as a
third party and sent no client info.
Recognition now follows the parent domain through dns.IsSubDomain, as
IsControlD already does for the ControlD domains. The label by label
comparison is case insensitive and keeps lookalikes such as
notnextdns.io and nextdns.io.example.com out.
Based on the change proposed by Mike (Github username @mike406).
See: https://github.com/Control-D-Inc/ctrld/pull/335.
The genuine-failure cases fed a fabricated error straight to
logIfaceLookupFailure, so they exercised only its own branching. A
restoration failure never reaches that helper, and no test reached the
enumeration classification in netInterface at all: discarding the
enumeration error, or downgrading an actual restoration failure to
debug, both kept the suite green.
The host boundaries of the reset path are now variables, in the style
the package already uses: the interface enumeration, the interface
lookup, the DNS setters and the NetworkManager restore. The tests stub
them, so a run changes no host DNS or NetworkManager state, and drive
the real netInterface and resetDNSForRunningIface to assert that a
failed enumeration keeps its cause and is not reported as a missing
interface, and that a failure to restore the saved static config or to
reset to DHCP on an interface that does exist stays at error level.
Both surviving mutations, and three more, now fail.
A successful macOS upgrade printed "could not get interface
error=interface not found iface=en5" before reporting success, because
the DNS cleanup logged every interface lookup failure at error level.
An interface that is gone, an unplugged adapter or a torn down tether,
has no DNS settings left to restore, so the skip is now a debug
diagnostic naming the interface and the work it skipped.
Only that condition is quiet. netInterface returns a sentinel error for
a missing interface and no longer reports a failed enumeration as one,
so any other lookup failure and any restoration failure on an interface
that does exist stay at error level.
The journal records the network state of the host and each change of
that state. The support team reads one file to see the interfaces, the
routes, the resolvers, and the recoveries around an incident. The same
file shows the health of the query path. A change that touches only
AirDrop or virtual interfaces runs no DNS work.
ctrld keeps a second log file, ctrld-journal.log, next to ctrld.log. The
journal holds all warnings and errors and each event with the field
journal=true. It stays on disk after a start of the service and after a
self-upgrade. Each log file starts with a Log header line. Each log send
upload starts with the same line, made at send time, and has a limit of
about 16 MB.
A failed handback restored ctrld's rule via addNRPTCatchAllRule, which
also writes the GP-path rule whenever another GP rule exists - leaving
CtrldCatchAll beside the administrator's catch-all. Restore only the
local store while that child is still present.
Consume resolver.split_dns and generate a policy rule per domain, plus its
wildcard, so a suffix and its subdomains route to the OS resolver when no
resolvers are given, or to the configured resolvers when they are.
Explicit resolvers are exclusive: when all of them fail the query returns
SERVFAIL rather than taking the OS catch-all or starting recovery, and it
skips VPN split routing, which is auto-detected. Excludes and custom config
keep precedence; a refresh regenerates the config, so removals leave no
stale rule or upstream.
Rule text survives a suspend unchanged, so the watchdog reports the pf
anchor intact while nothing translates, and the probe monitor is only
armed by interface changes with a schedule frozen through the sleep.
Detect the resume from the wall-clock gap and probe from there, with a
bounded repair: at most two forced reloads across the retry schedule.
Refresh the OS resolver before every probe - the probe aims at the first
OS nameserver and passes when it has none, so a pre-sleep list makes it
meaningless. Stand down for stabilization, which verifies on completion.
macOS intercept mode fails totally on networks that provide no usable
IPv4 DNS (e.g. IPv6-only iPhone tethering with 464XLAT): the pf ruleset
blocks all outbound IPv6 port 53, and with no IPv4 DNS configured
mDNSResponder emits no DNS packets at all, so pf has nothing to
intercept while the Control D upstream stays provably healthy.
When network-change recovery discovers no usable IPv4 DNS on the
default-route service, set 127.0.0.1 as that service DNS so macOS can
emit queries that land directly on the listener. The entry is removed
when the network regains IPv4 DNS and on intercept shutdown; networks
that provide IPv4 DNS are never modified. Runs on the already-debounced
recovery path so interface flaps do not churn networksetup.
Also fix the canceled-recovery state leak: the cancellation early
return never reset recoveryBypass/recoveryRunning, so a flap burst
ending in a canceled recovery left the daemon in bypass forever with
the DNS watchdog disabled. Cleanup is generation-gated so a superseded
recovery never clears state owned by its successor.
tryUpdateListenerConfig treated an explicit --intercept-mode off the same
as an empty flag and fell back to the persisted config value. run() selects
the listener strategy before it clears the persisted mode, so the first
start after a revert to standard mode selected the intercept strategy from
a stale dns/hard value while setDNS kept interception off.
Extract the resolution into listenerInterceptMode and make an explicit off
final, the same contract as setDNS. Add a regression test that fails
without the fix.
The warning reported err, the resolver-config fetch error, which is nil on
every path that reaches it - so a rejected custom config was logged with no
reason attached. cfgErr holds the validation failure.
"ctrld status" reported the service manager's view and nothing else, so it
printed "Service is running" and exited 0 for a process that was alive and
registered as started but had never got past startup: no control socket, no DNS
listener, no policy applied. The one command an operator reaches for first
confirmed the service was fine while the host had no working DNS.
Probe the control server's /started endpoint before reporting success. That
endpoint only answers once the onStarted hooks have completed, which is after the
listeners are up, so a successful probe means the process is serving rather than
merely alive. A service that is registered as running but cannot confirm startup
is now reported as such, with a pointer to the log, and exits 3 - distinct from
stopped (1) and unknown (2), because it needs a different response.
A probe blocked by permissions is not evidence of a broken service: an
unprivileged caller still gets "Service is running", with a note that startup was
not verified. The probe is bounded by a short timeout so status stays fast.
Document the exit codes in the command's help, and cover the probe (ready, not
finished starting, no socket, timed out) and the classification, including that
an unreadable socket is not reported as a failure.
The not-ready verdict is only reported when the probe could have found the
daemon's socket. socketDir() is caller-relative on unix - the system directory
when writable, the caller's home otherwise - so an unprivileged "ctrld status"
looks somewhere the root-owned daemon never listened and gets ENOENT, which is
"wrong path", not "not ready". Since only darwin has an elevation PreRun and the
root-level alias has none, that is the normal invocation; reporting exit 3 there
would have told a monitoring check to restart healthy daemons. Such a caller now
gets the service manager's view with startup reported as unverified. Windows and
mobile resolve the same directory for every caller, so the verdict stays fully
available on the platform the hung start was seen on. A successful probe is still
conclusive whoever ran it.
Rollback ran os.Remove(bin) while the replacement service was still running from
that image. Windows locks a running executable, so the remove failed with
"Access is denied" - and it was fatal, so the os.Rename that restores the
previous binary never ran. The upgrade ended with the broken replacement still
installed and the working binary stranded at its _previous name.
Readiness failing is not evidence the process exited: the service manager can
report a started service whose process never became operational. So rollback now
stops the service and waits until the manager reports it stopped before touching
the executable, then cleans up DNS the way the restart path's Cleanup task does.
Restoring is now conditional on the previous binary reporting a version, since a
_previous file that exists but produces no version output would trade a service
that starts and hangs for one that cannot start at all. When it is unusable,
rollback keeps it for inspection, leaves the installed binary alone, and says so
instead of pressing on. The --version probe is bounded by a timeout so a binary
that hangs cannot hang the upgrade.
Remaining failures are reported rather than fatal, so each one says what state
the host was left in. os.Remove is retried while the path stays locked, since
Windows releases an image lock asynchronously after the process exits.
The helpers live in a new file rather than in commands.go, and the rollback is
extracted into rollbackToPreviousBinary() so it can be covered: the stop happens
while the executable is still present, an unusable previous binary is kept
without swapping or restarting, and a failed stop aborts before anything is
modified. Reversing the stop and the remove fails these tests.
The version probe is called through a variable so those tests do not have to
stage a runnable executable. Staging one is not portable: oldBin is
bin+"_previous", so a fixture named "ctrld" yields the extension-less
"ctrld_previous", which Windows refuses to execute, and a symlink to the test
binary needs a privilege Windows does not grant by default. The probe itself is
still covered against the real test binary. Production is unaffected: ctrld.exe
_previous does have an extension, and os/exec only appends PATHEXT entries when
a path has none at all - noted at binaryVersion so the suffix is not renamed
into something extension-less by accident.
Both API requests and binary downloads retry against a hard-coded IP when the
attempt via hostname fails. Both then overwrote the first error with the
fallback's, so only the last failure was reported.
That discarded the diagnosis. When the hostname attempt is denied locally -
WSAEACCES on Windows, "An attempt was made to access a socket in a way forbidden
by its access permissions", which means the host is blocking ctrld - and the
direct-ip fallback fails with an unreachable IPv6 route, what surfaces to the
operator is "dial tcp6: no route to host": a routing problem that does not
exist, while the error naming the real cause is visible only in debug logs.
Report both failures instead, keeping the error chain intact so errors.Is still
matches either one. Also switch the final wrap in doWithRetry from %v to %w,
which had been flattening the chain even when a single error was reported.
This also changes retry classification, which is worth stating explicitly because
it is not obvious from "report both errors". processCDFlags decides whether to
keep backing off with errUrlNetworkError, which uses errors.As - and errors.As
returns the *first* match in the tree. Wrapping the hostname attempt first
therefore hands the predicate that attempt's failure, where previously only the
fallback's error survived to be classified.
The effect is intended. A locally denied socket (WSAEACCES) is not a transient
network error, so preflight now fails fast and reports instead of retrying
against a firewall that is not going to clear on its own. The case that justifies
retrying forever, a network unreachable on both attempts at boot, is unchanged.
Both classifications are pinned by tests, along with the wrap order they depend on
at each composition site, so reversing it fails loudly rather than silently
restoring the old behaviour.
processCDFlags retries the resolver-config fetch indefinitely by design: a
device that has no working network at boot must eventually come up. The loop had
no cancellation, so a stop request arriving while the API is unreachable was
ignored - the process kept retrying long after the service reported itself
stopped, doing work on behalf of a service the OS considers stopped.
Thread a context through processCDFlags and derive it from p.stopCh, in both the
startup preflight and the config-reload path. The loop now returns as soon as the
context is cancelled, checked both before a retry and after backoff returns
(backoff can wake up on cancellation). A stop during preflight exits the way a
normal stop does, without Fatal, so the service manager does not treat it as a
failed start and apply its restart policy to a service the operator just asked
to stop.
Bind the two API requests themselves as well, so a stop does not have to wait out
an in-flight request. Without this the loop honours a stop only between attempts,
which leaves up to defaultTimeout (20s) of a request the service is no longer
interested in - the same "still working after Service stopped" the loop change
exists to end, one layer down.
Doing so means a context parameter on FetchResolverConfig, FetchResolverUID,
UpdateCustomLastFailed and SendLogs, since all four reach a request builder. The
callers that have no context pass context.Background(), which is what master
effectively does at those sites: its loggerCtx carries a logger, not
cancellation. doWithFallback needs no parameter, because it clones the request
with req.Context() and so inherits the binding. This also repairs
internal/controld/controld_test.go, which is behind //go:build controld and had
already been written against the context-taking signature, so it could not
compile.
Cover the cancellation paths; removing either check makes the tests hang until
timeout.
Sampling the stop state is the whole point of runAPIPreflight rather than doing
this inline. A stop and a failure need opposite handling - one exits quietly, the
other self-uninstalls a deleted device, surfaces the error to a mobile app, and
reports a failed start - so the two must not be confused. Reading it from the
context after cancelling would report "stopped" for every failure, since
CancelFunc sets ctx.Err() regardless of whether anyone asked to stop; the stop
channel is read directly instead, which also does not depend on the context
watcher goroutine having been scheduled.
The post-stabilization reconcile verifies rule text, which cannot tell a
live redirect from an anchor pf has stopped evaluating. Sleep/wake QA
caught exactly that split: references intact, anchor rules intact,
post-load verification passed, and every query through the system
resolver timing out while the direct listener answered.
Nothing else probed. The interception probe monitor stands down while
stabilization owns pf and is never re-armed afterwards, so functional
recovery waited for the periodic watchdog - 11 seconds in the captured
run, up to a full 30-second interval - on a host whose link and default
route were already back. The watchdog's probe then failed once, forced a
reload, and public and VPN split-DNS both recovered immediately.
Probe once at the end of stabilization and, if it fails, force exactly
one reload and confirm with one more probe. Not the probe monitor: that
keeps probing for ~7.5s and can force a reload per failed probe, where
this path needs a single bounded repair before handing back to the
watchdog. Skipped when a monitor already owns probing, when intercept
state is gone, or during exec backoff.
Hand ownership over deterministically rather than skipping on sight. A
probe monitor started by an ignored network change claimed
functional-probe ownership before checking whether it could work, then
stood down because stabilization still owned pf; the verifier read that
claimed flag as "somebody is probing" and skipped, so neither path
probed and recovery fell back to the watchdog anyway. The monitor now
checks eligibility before claiming, and the verifier waits out a holder
that releases, yielding only to one that keeps probing.
Extract the completion block into finishPFStabilization so the wiring is
testable, and cover the bounded repair, the healthy path that must not
reload, a prober that claims and stands down, a prober that keeps
working, and a monitor that must not claim ownership while stabilizing.
Cuong Manh Le