Internal Domains now support three modes. "os" is unchanged.
"resolvers" becomes explicit resolver with network fallback: the
configured resolvers are tried first, and a timeout, unreachable
resolver, SERVFAIL or NXDOMAIN hands the query to matching VPN DNS
servers, then domain-less VPN DNS servers, then the OS resolver's LAN
nameservers through the new LanOnlyQueryCtx. "resolvers_only" keeps
the previous strict behavior. Down fallback-mode resolvers are skipped
and re-checked in the background at most every 30s. Generated upstreams
are identified by the UpstreamConfig.InternalDomain marker instead of
the internal_ key prefix.
Master adaptations: the fallback state lives on proxyRequest and is
applied in tryUpstreams, shouldContinueWithNextUpstream and
handleAllUpstreamsFailure instead of v1.0's monolithic proxy loop.
Resolver construction uses master's context-aware NewResolver, and the
background re-check marks its context private so transport errors that
name the resolver stay at debug. Tests use master's zap JSON capture
and VPN manager logger.
A maintenance fallback start holds no resolver config, and Allowed
Destinations (a master-only Firewall Mode feature) are not persisted with
the configuration on disk. Under Firewall Mode the exception set is
therefore empty until the API answers, so raw-IP destinations the
organization allowed stay blocked for the window, and a persisted pf
table from the previous run is replaced by the empty set.
Keep that fail-closed on purpose: the list is the organization's network
topology and is kept out of persisted state. Journal a Warn when a
fallback start applies the empty set under Firewall Mode so the blocked
destinations are explained, and pin that the first answer after recovery
re-applies the list.
Also capitalize the two managed-config marker log messages to match the
rest of master.
A hard maintenance window took DNS down on endpoints that already had a
working configuration: the bootstrap preflight read the maintenance answer
as a failed fetch and exited, turning a temporary API outage into an
endpoint outage.
Startup now falls back to the configuration on disk when a content- and
identity-bound marker records that the API produced it. Without one, ctrld
stops with API_MAINTENANCE (exit 37). Maintenance never self-uninstalls or
counts as a permanent rejection, a fallback run retries every 5-7 minutes
until a recovered configuration is applied, and deactivation is refused
with 503 while the PIN cannot be verified.
Master adaptation: pending-recovery and PIN-known handling live in
applyFetchedResolverConfig; logging uses *ctrld.Logger; the reload fetch
seam is fetchResolverConfigFn; the control-server and self-uninstall
probes use the existing fetchResolverConfig seam with loggerCtx.
During the reported incident the API answered "Maintenance is in
progress. Please try again later." after a successful DNS resolution
and TCP connection, and ctrld reported it as API_UNREACHABLE. Nothing
in the client could tell that answer apart from the API refusing the
request, so callers had to guess from an HTTP status that does not
carry the distinction.
Add IsMaintenance. The API answers every public request in hard
maintenance mode with HTTP 503 and error.code 50302
(Application::ERROR_HARD_MAINTENANCE), exported as MaintenanceCode, and
IsMaintenance classifies on that code. It falls back to the message for
deployments that answer without the code, which a test covers so that
path cannot silently drop. The answer's "body" is an empty JSON array;
a test pins that the error decode does not depend on it.
50301 (service unavailable) and 50003 (read-only mode) are deliberately
not maintenance: the first is a failure, and read-only only gates
authenticated controllers, so it never answers /utility or /logs.
Maintenance is temporary and says nothing about the configuration that
was asked for. The HTTP status cannot classify it on its own, because a
maintenance answer can arrive with a client-error status that otherwise
means the same request will be refused again, so callers must ask this
question before they act on a status.
The previous change marked a rule-matched AD query as a LAN query, but
two paths still reach the OS resolver without that mark, so osResolver
raced the domain controller against 76.76.2.0 in plaintext:
- the DNS-intercept recovery bypass, which forwards queries before
rule handling runs
- the OS-resolver retry catch-all after an explicit upstream fails
with leak_on_upstream_failure enabled
Mark a query under the detected AD domain on both paths. Other names
keep the public fallback, and an explicit upstream is still tried first.
Build resolvers in queryUpstream() through newResolverFn, so tests drive proxy()
and assert the LAN mark that reaches each resolver, instead of testing
the predicate alone.
The auto-added AD split rules route corp names to upstream.os, but a rule
match returned before the LAN-query marking, so osResolver raced the domain
controller against 76.76.2.0 and sent every AD lookup (DC hostnames,
DC-locator SRV names carrying the machine name, WPAD) there in plaintext.
Mark a matched query under the machine's AD domain whose upstreams are the
OS resolver alone as a LAN query, which drops 76.76.2.0 for it.
Decode networksetup entries once for interface lookup and DNS-target
selection. Preserve first-enabled lookup and fail-closed DNS uniqueness
as separate policies. Treat only the (*) index as disabled, preserving
asterisks and whitespace in enabled service names.
captureDebugMainLog swaps the process-wide mainLog, so every
mainLog.Load().Warn()/.Error() site in the package writes into the
buffer while it is installed, including goroutines left running by
other tests. retainedProbeLines returned every retained line and the
tests asserted field values over all of them, so one unrelated warn
failed the run.
Test_checkDnsLoopKeepsACustomKeyOutOfTheJournal hit this on Windows,
where the loop probe held the buffer open for two seconds:
upstream_probe_journal_test.go: field "upstream": got , want upstream.custom
The sibling tests were exposed the same way; they index retained[0]
after a strict length check, so a stray line fails them as got 2, want 1.
Let retainedProbeLines take the messages a caller owns and return only
those lines. wantNoProbeSecret keeps scanning every retained line,
which is what that leak check is for.
checkDnsLoop resolved with context.Background() and no dns.Client
timeout, so the DNS client fell back to its own 2s default and the
configured upstream timeout was ignored. The probes run serially on a
one-minute ticker, so each unreachable local upstream stalled the loop
for a fixed 2s no matter what the config asked for.
This is most visible on Windows, where UDP to a closed loopback port
does not surface an immediate refusal the way it does on Linux, so the
probe waits out the full default.
Apply the upstream timeout the way checkUpstreamOnce already does, and
keep 2s for an upstream that configures none so the unconfigured case
is unchanged.
The probe context deliberately carries no logger. A resolver logs its
own failure at error level with the endpoint in the message, which
would place an Internal Domain resolver address in the retained
journal; logUpstreamProbeFailure remains the reporter and bounds what
the line may hold. TestInternalDomainsLoopCheckHidesResolverAddress
covers this.
Remove empty and root domain (".") entries from search domains list
to prevent systemd-resolved errors. This addresses the issue where
systemd doesn't allow root domain in search domains configuration.
The filtering ensures only valid search domains are passed to
systemd-resolved, preventing DNS operation failures.
NextDNS serves alternative DoH endpoints beside dns.nextdns.io: the
ultralow and anycast variants of dns, dns1 and dns2. They are the same
service and take the same client info headers, but isNextDNS matched
the one host, so an upstream pointed at any of them was treated as a
third party and sent no client info.
Recognition now follows the parent domain through dns.IsSubDomain, as
IsControlD already does for the ControlD domains. The label by label
comparison is case insensitive and keeps lookalikes such as
notnextdns.io and nextdns.io.example.com out.
Based on the change proposed by Mike (Github username @mike406).
See: https://github.com/Control-D-Inc/ctrld/pull/335.
Keep missing interfaces as debug skips while preserving enumeration
causes and error-level failures restoring DNS on existing interfaces.
Adapt the logger and NetworkManager method to master. Stub saved DNS
lookup in tests because master stores it through ctrld rather than the
CLI homedir; tests must not touch the service's saved DNS files.
Running with --silent still created and grew log files in cd mode.
needInternalLogging() only checked cdUID and Service.LogPath, so a
--silent flag enabled internal logging, persisted it to disk, and
reset the global log level back to debug, overriding the NoLevel that
--silent had set.
Return false from needInternalLogging() when silent is set, so ctrld
neither creates the internal log file nor writes debug logs. Add
regression tests asserting needInternalLogging() is false in silent mode
and that initInternalLogging() creates no log file.
A failed handback restored ctrld's rule via addNRPTCatchAllRule, which
also writes the GP-path rule whenever another GP rule exists - leaving
CtrldCatchAll beside the administrator's catch-all. Restore only the
local store while that child is still present.
Port the final merged change from !1009 to master, retaining master's
context-aware logger/resolver APIs, proxy decomposition and firewall
refresh handling. Adapt source regression tests and cover master's
refresh handler and transport-level private-address diagnostics.
Rule text survives a suspend unchanged, so the watchdog reports the pf
anchor intact while nothing translates, and the probe monitor is only
armed by interface changes with a schedule frozen through the sleep.
Detect the resume from the wall-clock gap and probe from there, with a
bounded repair: at most two forced reloads across the retry schedule.
Refresh the OS resolver before every probe - the probe aims at the first
OS nameserver and passes when it has none, so a pre-sleep list makes it
meaningless. Stand down for stabilization, which verifies on completion.
The post-stabilization reconcile verifies rule text, which cannot tell a
live redirect from an anchor pf has stopped evaluating. Sleep/wake QA
caught exactly that split: references intact, anchor rules intact,
post-load verification passed, and every query through the system
resolver timing out while the direct listener answered.
Nothing else probed. The interception probe monitor stands down while
stabilization owns pf and is never re-armed afterwards, so functional
recovery waited for the periodic watchdog - 11 seconds in the captured
run, up to a full 30-second interval - on a host whose link and default
route were already back. The watchdog's probe then failed once, forced a
reload, and public and VPN split-DNS both recovered immediately.
Probe once at the end of stabilization and, if it fails, force exactly
one reload and confirm with one more probe. Not the probe monitor: that
keeps probing for ~7.5s and can force a reload per failed probe, where
this path needs a single bounded repair before handing back to the
watchdog. Skipped when a monitor already owns probing, when intercept
state is gone, or during exec backoff.
Hand ownership over deterministically rather than skipping on sight. A
probe monitor started by an ignored network change claimed
functional-probe ownership before checking whether it could work, then
stood down because stabilization still owned pf; the verifier read that
claimed flag as "somebody is probing" and skipped, so neither path
probed and recovery fell back to the watchdog anyway. The monitor now
checks eligibility before claiming, and the verifier waits out a holder
that releases, yielding only to one that keeps probing.
Extract the completion block into finishPFStabilization so the wiring is
testable, and cover the bounded repair, the healthy path that must not
reload, a prober that claims and stands down, a prober that keeps
working, and a monitor that must not claim ownership while stabilizing.
Port !997 from v1.0 onto the context-aware master recovery lifecycle. Preserve master logging and resolver APIs while adding macOS default-route DHCP detection, temporary DNS-target cleanup, and atomic recovery ownership.
Includes parser, lifecycle, failure, and concurrency regressions plus the corrected macOS QA helper. Relates to #533 and #597.
Firewall Mode permits only what ctrld resolved through its own listener.
The API transport resolves api.controld.com through the OS nameservers and
falls back to hardcoded addresses, so nothing ever teaches the allowlist
about it and ctrld's own block-all filters deny its control-plane socket.
The upgrade download server has the same shape: performUpgrade spawns a
detached child process, and the WFP filters carry no process condition, so
the service blocks its own upgrade.
Permit both permanently, at startup and on reload. For the API that means
the resolved addresses and the transport's direct fallbacks - the fallbacks
are what it dials when DNS is unusable, which is the state a blocked ctrld
is in. For the download server only the fallback IP is needed, since its
hostname lookup does go through the listener and is learned.
APIDomain/APIEndpointIPs are exported so the permitted set and the dialed
set cannot drift apart.
Call initPlatformFirewall on reload even when enforcement is already up.
AddPermanent fires no change callback, so an address permitted by a reload
reached memory only while the platform never heard about it. Each
platform's re-entry is a refresh: Windows reinstalls the permanent filters
it is missing, macOS returns early.
Also keep every dial attempt in the transport's error. It returned only the
last stage, an unroutable IPv6 address reporting "no route to host", hiding
the IPv4 WSAEACCES that named the real cause. The direct IPs are still
always dialed, so the API stays reachable without DNS; only a duplicate
dial of an address the resolver already returned is dropped.
Firewall Mode only permits what ctrld resolved, so an approved service addressed
by literal IP - with no DNS lookup to observe - is unreachable, and the only
workaround was turning the mode off. The API now sends the effective per-org list
in destination_ips of every resolver-config response.
Apply it as a set rather than as additions: each refresh replaces the previous
snapshot, so an entry added upstream takes effect and one removed upstream stops
bypassing enforcement. This happens inside the refresh handler before its early
returns, so scheduled and forced refreshes both carry it, and without a ctrld
reload. Entries carry no TTL and survive the allowlist flushes that follow a
profile or network change.
Track what the API asked for separately from what pf/WFP accepted, because
mirroring can fail and the next refresh - carrying an identical list - would
compute no delta to retry. The applied snapshot advances only on success, and the
difference is retried by the next refresh and by a reconcile every 5 minutes,
reported meanwhile as allowed_destinations_pending. Enforcement coming up
replaces the whole set rather than adding to it: the macOS table is a persist
table that can still hold what a previous run put there. Enforcement is versioned
by a generation advanced under the same lock the mirror is called with, so a
maintenance worker outliving its run cannot reinstall permits into enforcement
that is gone.
macOS keeps the set in a second pf table, <ctrld_allowed_dst>; Windows in
per-entry WFP permit filters in their own map - apart from the DNS-resolved
entries so a flush of those leaves them installed. Linux is unchanged, the mode
already fails open there, and devices with Firewall Mode off are unaffected.
Lookups binary search sorted per-family address ranges, so the per-connection hot
path stays flat at ~40ns rather than growing with the list. Addresses are logged
at debug level only: the list is organization network topology, and Info-level
logs are persisted and travel in support bundles.
Indirect the refresh's fetch and split its handler out of the fetch loop so both
refresh paths are driven end to end in tests without an API server.
tryUpdateListenerConfig treated an explicit --intercept-mode off the same
as an empty flag and fell back to the persisted config value. run() selects
the listener strategy before it clears the persisted mode, so the first
start after a revert to standard mode selected the intercept strategy from
a stale dns/hard value while setDNS kept interception off.
Extract the resolution into listenerInterceptMode and make an explicit off
final, the same contract as setDNS. Add a regression test that fails
without the fix.
The warning reported err, the resolver-config fetch error, which is nil on
every path that reaches it - so a rejected custom config was logged with no
reason attached. cfgErr holds the validation failure.
"ctrld status" reported the service manager's view and nothing else, so it
printed "Service is running" and exited 0 for the incident's process: alive,
registered as started, stuck in API preflight, with no control socket, no DNS
listener and no policy applied. The one command an operator reaches for first
confirmed the service was fine while the host had no working DNS.
Probe the control server's /started endpoint before reporting success. That
endpoint only answers once the onStarted hooks have completed, which is after the
listeners are up, so a successful probe means the process is serving rather than
merely alive. A service that is registered as running but cannot confirm startup
is now reported as such, with a pointer to the log, and exits 3 - distinct from
stopped (1) and unknown (2), because it needs a different response.
A probe blocked by permissions is not evidence of a broken service: an
unprivileged caller still gets "Service is running", with a note that startup
was not verified. The probe is bounded by a short timeout so status stays fast.
Document the exit codes in the command's help, and cover the probe (ready, not
finished starting, no socket, timed out) and the classification, including that
an unreadable socket is not reported as a failure.
The not-ready verdict is only reported when the probe could have found the
daemon's socket. socketDir() is caller-relative on unix - the system directory
when writable, the caller's home otherwise - so an unprivileged "ctrld status"
looks somewhere the root-owned daemon never listened and gets ENOENT, which is
"wrong path", not "not ready". Since only darwin has an elevation PreRun and the
root-level alias has none, that is the normal invocation; reporting exit 3 there
would have told a monitoring check to restart healthy daemons. Such a caller now
gets the service manager's view with startup reported as unverified. Windows and
mobile resolve the same directory for every caller, so the verdict stays fully
available on the platform the hung start was seen on. A successful probe is still
conclusive whoever ran it.
Rollback ran os.Remove(bin) while the replacement service was still running from
that image. Windows locks a running executable, so the remove failed with
"Access is denied" - and it was fatal, so the os.Rename that restores the
previous binary never ran. The upgrade ended with the broken replacement still
installed and the working binary stranded at its _previous name, which is what
the Firewall Mode incident produced.
Readiness failing is not evidence the process exited: the service manager can
report a started service whose process never became operational. So rollback now
stops the service and waits until the manager reports it stopped before touching
the executable, then resets DNS the way the restart path's Cleanup task does.
That also leaves the host in a known state - a stopped ctrld holds no WFP or
NRPT enforcement, so a replacement that was blocking traffic stops blocking it
at rollback instead of at the next reboot.
Restoring is now conditional on the previous binary reporting a version. The
incident's _previous file existed but produced no version output; renaming that
over the current binary would have traded a service that starts and hangs for
one that cannot start at all. When it is unusable, rollback keeps it for
inspection, leaves the installed binary alone, and says so instead of pressing
on. The --version probe is bounded by a timeout so a binary that hangs cannot
hang the upgrade.
Remaining failures return errors rather than calling Fatal, so each one reports
what state the host was left in. os.Remove is retried while the path stays
locked, since Windows releases an image lock asynchronously after the process
exits.
Extract the rollback into rollbackToPreviousBinary() and cover it: the stop
happens while the executable is still present, an unusable previous binary is
kept without swapping or restarting, and a failed stop aborts before anything is
modified. Reversing the stop and the remove fails these tests.
The version probe is called through a variable so those tests do not have to
stage a runnable executable. Staging one is not portable: oldBin is
bin+"_previous", so a fixture named "ctrld" yields the extension-less
"ctrld_previous", which Windows refuses to execute, and a symlink to the test
binary needs a privilege Windows does not grant by default. The probe itself is
still covered against the real test binary. Production is unaffected: ctrld.exe
_previous does have an extension, and os/exec only appends PATHEXT entries when
a path has none at all - noted at binaryVersion so the suffix is not renamed
into something extension-less by accident.
Dev Scribe