A hard maintenance window took DNS down on endpoints that already had a
working configuration: the bootstrap preflight read the maintenance answer
as a failed fetch and exited, turning a temporary API outage into an
endpoint outage.
Startup now falls back to the configuration on disk when a content- and
identity-bound marker records that the API produced it. Without one, ctrld
stops with API_MAINTENANCE (exit 37). Maintenance never self-uninstalls or
counts as a permanent rejection, a fallback run retries every 5-7 minutes
until a recovered configuration is applied, and deactivation is refused
with 503 while the PIN cannot be verified.
Master adaptation: pending-recovery and PIN-known handling live in
applyFetchedResolverConfig; logging uses *ctrld.Logger; the reload fetch
seam is fetchResolverConfigFn; the control-server and self-uninstall
probes use the existing fetchResolverConfig seam with loggerCtx.
The previous change marked a rule-matched AD query as a LAN query, but
two paths still reach the OS resolver without that mark, so osResolver
raced the domain controller against 76.76.2.0 in plaintext:
- the DNS-intercept recovery bypass, which forwards queries before
rule handling runs
- the OS-resolver retry catch-all after an explicit upstream fails
with leak_on_upstream_failure enabled
Mark a query under the detected AD domain on both paths. Other names
keep the public fallback, and an explicit upstream is still tried first.
Build resolvers in queryUpstream() through newResolverFn, so tests drive proxy()
and assert the LAN mark that reaches each resolver, instead of testing
the predicate alone.
Keep missing interfaces as debug skips while preserving enumeration
causes and error-level failures restoring DNS on existing interfaces.
Adapt the logger and NetworkManager method to master. Stub saved DNS
lookup in tests because master stores it through ctrld rather than the
CLI homedir; tests must not touch the service's saved DNS files.
Port the final merged change from !1009 to master, retaining master's
context-aware logger/resolver APIs, proxy decomposition and firewall
refresh handling. Adapt source regression tests and cover master's
refresh handler and transport-level private-address diagnostics.
Rule text survives a suspend unchanged, so the watchdog reports the pf
anchor intact while nothing translates, and the probe monitor is only
armed by interface changes with a schedule frozen through the sleep.
Detect the resume from the wall-clock gap and probe from there, with a
bounded repair: at most two forced reloads across the retry schedule.
Refresh the OS resolver before every probe - the probe aims at the first
OS nameserver and passes when it has none, so a pre-sleep list makes it
meaningless. Stand down for stabilization, which verifies on completion.
Port !997 from v1.0 onto the context-aware master recovery lifecycle. Preserve master logging and resolver APIs while adding macOS default-route DHCP detection, temporary DNS-target cleanup, and atomic recovery ownership.
Includes parser, lifecycle, failure, and concurrency regressions plus the corrected macOS QA helper. Relates to #533 and #597.
Firewall Mode only permits what ctrld resolved, so an approved service addressed
by literal IP - with no DNS lookup to observe - is unreachable, and the only
workaround was turning the mode off. The API now sends the effective per-org list
in destination_ips of every resolver-config response.
Apply it as a set rather than as additions: each refresh replaces the previous
snapshot, so an entry added upstream takes effect and one removed upstream stops
bypassing enforcement. This happens inside the refresh handler before its early
returns, so scheduled and forced refreshes both carry it, and without a ctrld
reload. Entries carry no TTL and survive the allowlist flushes that follow a
profile or network change.
Track what the API asked for separately from what pf/WFP accepted, because
mirroring can fail and the next refresh - carrying an identical list - would
compute no delta to retry. The applied snapshot advances only on success, and the
difference is retried by the next refresh and by a reconcile every 5 minutes,
reported meanwhile as allowed_destinations_pending. Enforcement coming up
replaces the whole set rather than adding to it: the macOS table is a persist
table that can still hold what a previous run put there. Enforcement is versioned
by a generation advanced under the same lock the mirror is called with, so a
maintenance worker outliving its run cannot reinstall permits into enforcement
that is gone.
macOS keeps the set in a second pf table, <ctrld_allowed_dst>; Windows in
per-entry WFP permit filters in their own map - apart from the DNS-resolved
entries so a flush of those leaves them installed. Linux is unchanged, the mode
already fails open there, and devices with Firewall Mode off are unaffected.
Lookups binary search sorted per-family address ranges, so the per-connection hot
path stays flat at ~40ns rather than growing with the list. Addresses are logged
at debug level only: the list is organization network topology, and Info-level
logs are persisted and travel in support bundles.
Indirect the refresh's fetch and split its handler out of the fetch loop so both
refresh paths are driven end to end in tests without an API server.
Both API requests and binary downloads retry against a hard-coded IP when the
attempt via hostname fails. Both then overwrote the first error with the
fallback's, so only the last failure was reported.
That discarded the diagnosis. During the Firewall Mode incident the hostname
attempt was denied locally - WSAEACCES, "An attempt was made to access a socket
in a way forbidden by its access permissions", which means the host is blocking
ctrld - while the direct-ip fallback failed with an unreachable IPv6 route. What
surfaced to the operator was "dial tcp6: no route to host", pointing at a
routing problem that did not exist, while the WSAEACCES that named the real
cause was visible only in debug logs.
Report both failures instead, keeping the error chain intact so errors.Is still
matches either one. Also switch the final wrap in doWithRetry from %v to %w,
which had been flattening the chain even when a single error was reported.
Add coverage for both paths, including that the fallback is still attempted and
that a successful fallback returns no error.
This also changes retry classification, which is worth stating explicitly because
it is not obvious from "report both errors". processCDFlags decides whether to
keep backing off with errUrlNetworkError, which uses errors.As - and errors.As
returns the first match in the tree. Wrapping the hostname attempt first
therefore hands the predicate that attempt's failure, where previously only the
fallback's error survived to be classified.
The effect is intended. A locally denied socket (WSAEACCES) is not a transient
network error, so preflight now fails fast and reports instead of retrying
against a firewall that is not going to clear on its own - the incident logged
256 retry cycles doing exactly that. The case that justifies retrying forever, a
network unreachable on both attempts at boot, is unchanged. Both classifications
are pinned by tests, along with the wrap order they depend on at each composition
site, so reversing it fails loudly rather than silently restoring the old
behaviour.
processCDFlags retries the resolver-config fetch indefinitely by design: a
device that has no working network at boot must eventually come up. The loop
had no cancellation, so a stop request arriving while the API is unreachable
was ignored - the process kept retrying long after "Service stopped" was
logged, doing work on behalf of a service the OS considers stopped. The
Windows Firewall Mode incident showed this concretely: API retries continued
15 seconds after the stop completed, so stopping the service could not release
what the process was still holding.
Thread a context through processCDFlags and derive it from p.stopCh, in both
the startup preflight and the config-reload path. The loop now returns as soon
as the context is cancelled, checked both before a retry and after backoff
returns (backoff can wake up on cancellation).
A stop during preflight now exits the way a normal stop does, without Fatal,
so the service manager does not treat it as a failed start and apply its
restart policy to a service the operator just asked to stop.
Also bind the two API requests in internal/controld to the caller's context.
They were built with http.NewRequest, so an in-flight request ignored
cancellation and waited out its own timeout instead.
Covered by tests that assert what the incident needed: a stop request during
preflight ends the retry/backoff loop and reports cancellation rather than
continuing to retry after the service reports itself stopped, an
already-cancelled context makes at most one attempt, and contextFromStopCh
handles its three cases (cancelled by stopCh, released by cancel, usable with
no stop channel). Removing either cancellation check makes these tests hang
until the test timeout. The tests are also what exercise fetchResolverConfig,
the seam this commit introduces.
Sampling the stop state is the whole point of runAPIPreflight rather than doing
this inline. A stop and a failure need opposite handling - one exits quietly, the
other self-uninstalls a deleted device, surfaces the error to a mobile app, and
reports a failed start - so the two must not be confused. Reading it from the
context after cancelling would report stopped for every failure, since
CancelFunc sets ctx.Err() regardless of whether anyone asked to stop; the stop
channel is read directly instead, which also does not depend on the context
watcher goroutine having been scheduled.
During reload operations, log and cache flags were not being processed,
which prevented runtime internal logs from working correctly. To fix this,
processLogAndCacheFlags was refactored to accept explicit viper and config
parameters instead of relying on global state, enabling it to be called
during reload with the new configuration. This ensures that log and cache
settings are properly applied when the service reloads its configuration.
Restore the previously merged macOS VPN DNS post-settle refresh,
unchanged-exemption suppression, forced pf state flush, and self-upgrade
test after they were removed by the firewall-mode merge.
Also keep Windows NRPT pointed at loopback when the listener is configured
on a wildcard address.
On macOS DNS-intercept mode, when mDNSResponder owns *:53 ctrld falls back
to listening on 127.0.0.1:5354, and the pf rdr rules correctly redirect DNS
to the bound port at startup. However, DNS resolution later breaks with an
endless watchdog "anchor intact but probe FAILED -> force reload" loop and
`dig @127.0.0.1` timeouts.
Root cause is config reload. In CD mode, apiConfigReload refetches the
generated config (which always declares port 53) every hour and the reload
merge in runWait only inherits the running port when the new port is 0. The
generated config explicitly says 53, so `*p.cfg = *newCfg` reverts p.cfg to
port 53. The DNS listener goroutines are started only when !reload, so they
are never re-bound and stay on 5354. Every subsequent pf rebuild reads p.cfg
and targets the dead port 53.
Fix: after applying the reloaded config in DNS-intercept mode on darwin,
restore the actual bound listener IP/Port into the in-memory config via the
new preserveBoundListeners helper, logging the configured-vs-actual
divergence. A reload cannot move the running listener anyway, so this keeps
p.cfg consistent with reality; all pf rdr rules and the watchdog probe then
target the live port. The on-disk generated config is intentionally left
unchanged (still 53), so no generated-config change is required.
When IPv6 is available locally but the selected IPv6 DoH endpoint is
unroutable (e.g. dialing [2606:1a40::22]:443 returns "no route to host"
while IPv4 stays usable), ctrld re-bootstrapped and re-dialed the endpoint
every ~2s. A weekend soak produced ~46.7k "no route to host" lines, with
the dial/health-check loop dominating the log during bad windows.
Add bounded backoff/suppression for network-unreachable endpoints at two
levels:
- ParallelDialer (internal/net): track dial addresses that fail with
ENETUNREACH/EHOSTUNREACH and skip them for an exponentially growing,
bounded window (5s -> 60s). A successful dial clears the entry
immediately, so recovery is preserved when the route returns. When every
candidate is suppressed the dial fails fast and quietly instead of
hammering known-unroutable addresses.
- Upstream recovery loop (cmd/cli): demote unreachable check failures to
debug and back off the retry cadence (2s -> 60s) for an unreachable
streak; any other failure resets to the base cadence.
The new IsUnreachable classifier lives in internal/net and is reused by
cmd/cli's errNetworkError, so the unreachable-errno matching has a single
definition. Note the explicit winsock constants (10051/10065) are required
on Windows: syscall.ENETUNREACH/EHOSTUNREACH are Go's portable "invented"
values and never equal the raw WSA codes a failing connect surfaces.
Suppression and backoff are always bounded, so IPv6 is never disabled until
restart and recovers on its own once the route is back. Split-stack
selection and the #549 macOS intercept recovery work are untouched.
Adds unit tests for the classifier, the dialer's suppression tracker, and
the recovery backoff schedule.
The test:windows CI job intermittently failed to clean up .testbin with
"Access to the path '...cmd_cli.test.exe' is denied". This was previously
attributed to Windows Defender scanning the large unsigned test binaries,
and mitigated with Defender exclusions and cleanup retries. That was
treating a symptom.
Root cause: performUpgrade() self-upgrades by running
exec.Command(os.Executable(), "upgrade", "prod", "-vv") as a detached,
windowless child. In the real ctrld binary this re-execs ctrld and is
correct. Under `go test`, os.Executable() is the test binary itself, and
`go test` stops flag parsing at the first positional arg ("upgrade") and
ignores the rest -- so the child silently re-runs the entire test suite.
That child hits the upgrade tests again and spawns more detached children,
recursively: a fork bomb of hidden processes that pins the runner's
CPU/memory and keeps the test binary's image file locked. Windows refuses
to delete the image of a running process, hence the "Access is denied"
during after_script. Whether any children are still alive when cleanup
runs is a timing race, which is why the failure was flaky.
Two tests reached this path: Test_performUpgrade (directly) and
Test_selfUpgradeCheck (via selfUpgradeCheck -> performUpgrade on the
"upgrade allowed" case).
Fix:
- prog.go: extract the command construction into a package-level
newUpgradeCmd var. Production behavior is unchanged.
- main_test.go: stub newUpgradeCmd once in TestMain so the whole test
binary self-execs with `-test.run=^$` (matches no tests, exits
immediately) instead of re-running the suite. This covers every test
that reaches performUpgrade, present and future, while still exercising
the cmd.Start() success path.
When multiple network changes fire in quick succession (e.g., VPN
disconnect + interface swap), the second handleRecovery() call cancels
the first but inherits stale DoH transports, causing DNS blackouts
of up to 30 seconds.
Three changes to reduce worst-case recovery from ~30s to <3s:
1. ForceReBootstrap() on recovery entry — closes dead connections and
creates fresh transports synchronously before probing, replacing the
lazy ReBootstrap() flag that left stale connections for probes to hit.
2. Debounce handleRecovery() for network changes (500ms window) — only
the recovery flow is debounced; all other state updates (IP, pf
anchor, VPN DNS, tunnel checks) still run immediately on every event.
This eliminates the cancel-and-restart race without missing state.
3. Combined effect: ForceReBootstrap closes old in-flight connections
(closeTransports) and builds new ones (SetupTransport) atomically,
so recovery probes never inherit dead connections from a prior
recovery attempt.
Treat "socket missing" (ENOENT) and connection refused as expected when
probing the log server, and only log when the error indicates something
unexpected. This prevents noisy warnings when the log server has not
started yet.
Discover while doing captive portal tests.
Remove separate watchLinkState function and integrate link state change
handling directly into monitorNetworkChanges. This consolidates network
monitoring logic into a single place and simplifies the codebase.
Update netlink dependency from v1.2.1-beta.2 to v1.3.1 and netns from
v0.0.4 to v0.0.5 to use stable versions.
Move platform-specific network interface detection from cmd/cli/ to root package
as ValidInterfaces function. This eliminates code duplication and provides a
consistent interface for determining valid physical network interfaces across
all platforms.
- Remove duplicate validInterfacesMap functions from platform-specific files
- Add context parameter to virtualInterfaces for proper logging
- Update all callers to use ctrld.ValidInterfaces instead of local functions
- Improve error handling in virtual interface detection on Linux
Replace the legacy Unix socket log communication between `ctrld start` and
`ctrld run` with a modern HTTP-based system for better reliability and
maintainability.
Benefits:
- More reliable communication protocol using standard HTTP
- Better error handling and connection management
- Cleaner separation of concerns with dedicated endpoints
- Easier to test and debug with HTTP-based communication
- More maintainable code with proper abstraction layers
This change maintains backward compatibility while providing a more robust
foundation for inter-process communication between ctrld commands.
Capitalize the first letter of all log messages throughout the codebase
to improve readability and consistency in logging output.
Key improvements:
- All log messages now start with capital letters
- Consistent formatting across all logging statements
- Improved readability for debugging and monitoring
- Enhanced user experience with better formatted messages
Files updated:
- CLI commands and service management
- Internal client information discovery
- Network operations and configuration
- DNS resolver and proxy operations
- Platform-specific implementations
This completes the final phase of the logging improvement project,
ensuring all log messages follow consistent capitalization standards
for better readability and professional appearance.
Change DNS listener context from parent context to background context
so that listeners continue running during configuration reloads.
Listener configuration changes require a service restart, not reload,
so listeners must persist across reload operations.
This prevents DNS listeners from being terminated when the parent
context is cancelled during reload operations.
- Add entry/exit logging to all ServiceCommand methods (start, stop, status, reload, restart, uninstall)
- Replace mainLog.Load() calls with consistent logger variable usage throughout
- Capitalize all logging messages for better readability
- Add error context logging for service manager initialization failures
- Add debug logging for key operations (restart sequence, cleanup, validation)
- Improve error handling with proper error context in all service commands
- Add completion logging to track command execution flow
This improves debugging capabilities and provides better operational visibility
for service management operations while maintaining clean user-facing messages.
Move the network monitoring goroutine initialization outside the listener
loop to prevent it from being started multiple times. Previously, the
network monitoring was started once per listener during first run, which
was unnecessary and could lead to multiple monitoring instances.
The change ensures network monitoring is started only once per program
execution cycle, improving efficiency and preventing potential resource
waste from duplicate monitoring goroutines.
- Extract network monitoring goroutine from listener loop
- Start network monitoring once per run cycle instead of per listener
- Maintain same functionality while improving resource usage
- Add appVersion variable to store curVersion() result during init
- Change initCLI() to return *cobra.Command
- Move rootCmd creation inside initCLI() as local variable
- Replace all rootCmd.Version usage with appVersion variable
- Update Main() function to capture returned rootCmd from initCLI()
- Remove sync.Once guard from tests and use initCLI() directly
- Remove sync import from test file as it's no longer needed
This refactoring improves encapsulation by eliminating global state,
reduces version computation overhead, and simplifies test setup by
removing the need for sync.Once guards. All tests pass and the
application builds successfully.
- Move network monitoring initialization out of serveDNS() function
- Start network monitoring in a separate goroutine during program startup
- Remove context parameter from monitorNetworkChanges() as it's not used
- Simplify serveDNS() function signature by removing unused context parameter
- Ensure network monitoring starts only once during initial run, not on reload
This change improves separation of concerns by isolating network monitoring
from DNS serving logic, and prevents potential issues with multiple
monitoring goroutines if starting multiple listeners.
- Replace all direct newService() calls with ServiceCommand initialization
- Update command constructors to use ServiceCommand instead of ServiceManager
- Simplify LogCommand and UpgradeCommand structs by removing serviceManager field
- Remove unused global svcConfig variable from prog.go
- Improve consistency and centralize service creation logic
This change establishes a consistent pattern for service operations across
the codebase, making it easier to maintain and extend service-related
functionality.
Replace github.com/rs/zerolog with go.uber.org/zap throughout the codebase
to improve performance and provide better structured logging capabilities.
Key changes:
- Replace zerolog imports with zap and zapcore
- Implement custom Logger wrapper in log.go to maintain zerolog-like API
- Add LogEvent struct with chained methods (Str, Int, Err, Bool, etc.)
- Update all logging calls to use the new zap-based wrapper
- Replace JSON encoders with Console encoders for better readability
Benefits:
- Better performance with zap's optimized logging
- Consistent structured logging across all components
- Maintained zerolog-like API for easy migration
- Proper field context preservation for debugging
- Multi-core logging architecture for better output control
All tests pass and build succeeds.
- Add explicit foundDefaultRoute boolean variable to track default route discovery
- Initialize foundDefaultRoute to false and set to true only in success case
- Replace tautological condition `err == nil` with meaningful `foundDefaultRoute` check
- Fixes "tautological condition: nil == nil" linter error
The error occurred because err was being reused from net.Interfaces() call,
making the condition always true. Now we explicitly track whether a default
route was successfully found.
- Move version checking logic to shouldUpgrade for testability
- Move upgrade command execution to performUpgrade
- selfUpgradeCheck now composes these two for clarity
- Update and expand tests: focus on logic, not side effects
- Improves maintainability, testability, and separation of concerns
Dev Scribe