Port !997 from v1.0 onto the context-aware master recovery lifecycle. Preserve master logging and resolver APIs while adding macOS default-route DHCP detection, temporary DNS-target cleanup, and atomic recovery ownership.
Includes parser, lifecycle, failure, and concurrency regressions plus the corrected macOS QA helper. Relates to #533 and #597.
Firewall Mode only permits what ctrld resolved, so an approved service addressed
by literal IP - with no DNS lookup to observe - is unreachable, and the only
workaround was turning the mode off. The API now sends the effective per-org list
in destination_ips of every resolver-config response.
Apply it as a set rather than as additions: each refresh replaces the previous
snapshot, so an entry added upstream takes effect and one removed upstream stops
bypassing enforcement. This happens inside the refresh handler before its early
returns, so scheduled and forced refreshes both carry it, and without a ctrld
reload. Entries carry no TTL and survive the allowlist flushes that follow a
profile or network change.
Track what the API asked for separately from what pf/WFP accepted, because
mirroring can fail and the next refresh - carrying an identical list - would
compute no delta to retry. The applied snapshot advances only on success, and the
difference is retried by the next refresh and by a reconcile every 5 minutes,
reported meanwhile as allowed_destinations_pending. Enforcement coming up
replaces the whole set rather than adding to it: the macOS table is a persist
table that can still hold what a previous run put there. Enforcement is versioned
by a generation advanced under the same lock the mirror is called with, so a
maintenance worker outliving its run cannot reinstall permits into enforcement
that is gone.
macOS keeps the set in a second pf table, <ctrld_allowed_dst>; Windows in
per-entry WFP permit filters in their own map - apart from the DNS-resolved
entries so a flush of those leaves them installed. Linux is unchanged, the mode
already fails open there, and devices with Firewall Mode off are unaffected.
Lookups binary search sorted per-family address ranges, so the per-connection hot
path stays flat at ~40ns rather than growing with the list. Addresses are logged
at debug level only: the list is organization network topology, and Info-level
logs are persisted and travel in support bundles.
Indirect the refresh's fetch and split its handler out of the fetch loop so both
refresh paths are driven end to end in tests without an API server.
Both API requests and binary downloads retry against a hard-coded IP when the
attempt via hostname fails. Both then overwrote the first error with the
fallback's, so only the last failure was reported.
That discarded the diagnosis. During the Firewall Mode incident the hostname
attempt was denied locally - WSAEACCES, "An attempt was made to access a socket
in a way forbidden by its access permissions", which means the host is blocking
ctrld - while the direct-ip fallback failed with an unreachable IPv6 route. What
surfaced to the operator was "dial tcp6: no route to host", pointing at a
routing problem that did not exist, while the WSAEACCES that named the real
cause was visible only in debug logs.
Report both failures instead, keeping the error chain intact so errors.Is still
matches either one. Also switch the final wrap in doWithRetry from %v to %w,
which had been flattening the chain even when a single error was reported.
Add coverage for both paths, including that the fallback is still attempted and
that a successful fallback returns no error.
This also changes retry classification, which is worth stating explicitly because
it is not obvious from "report both errors". processCDFlags decides whether to
keep backing off with errUrlNetworkError, which uses errors.As - and errors.As
returns the first match in the tree. Wrapping the hostname attempt first
therefore hands the predicate that attempt's failure, where previously only the
fallback's error survived to be classified.
The effect is intended. A locally denied socket (WSAEACCES) is not a transient
network error, so preflight now fails fast and reports instead of retrying
against a firewall that is not going to clear on its own - the incident logged
256 retry cycles doing exactly that. The case that justifies retrying forever, a
network unreachable on both attempts at boot, is unchanged. Both classifications
are pinned by tests, along with the wrap order they depend on at each composition
site, so reversing it fails loudly rather than silently restoring the old
behaviour.
processCDFlags retries the resolver-config fetch indefinitely by design: a
device that has no working network at boot must eventually come up. The loop
had no cancellation, so a stop request arriving while the API is unreachable
was ignored - the process kept retrying long after "Service stopped" was
logged, doing work on behalf of a service the OS considers stopped. The
Windows Firewall Mode incident showed this concretely: API retries continued
15 seconds after the stop completed, so stopping the service could not release
what the process was still holding.
Thread a context through processCDFlags and derive it from p.stopCh, in both
the startup preflight and the config-reload path. The loop now returns as soon
as the context is cancelled, checked both before a retry and after backoff
returns (backoff can wake up on cancellation).
A stop during preflight now exits the way a normal stop does, without Fatal,
so the service manager does not treat it as a failed start and apply its
restart policy to a service the operator just asked to stop.
Also bind the two API requests in internal/controld to the caller's context.
They were built with http.NewRequest, so an in-flight request ignored
cancellation and waited out its own timeout instead.
Covered by tests that assert what the incident needed: a stop request during
preflight ends the retry/backoff loop and reports cancellation rather than
continuing to retry after the service reports itself stopped, an
already-cancelled context makes at most one attempt, and contextFromStopCh
handles its three cases (cancelled by stopCh, released by cancel, usable with
no stop channel). Removing either cancellation check makes these tests hang
until the test timeout. The tests are also what exercise fetchResolverConfig,
the seam this commit introduces.
Sampling the stop state is the whole point of runAPIPreflight rather than doing
this inline. A stop and a failure need opposite handling - one exits quietly, the
other self-uninstalls a deleted device, surfaces the error to a mobile app, and
reports a failed start - so the two must not be confused. Reading it from the
context after cancelling would report stopped for every failure, since
CancelFunc sets ctx.Err() regardless of whether anyone asked to stop; the stop
channel is read directly instead, which also does not depend on the context
watcher goroutine having been scheduled.
During reload operations, log and cache flags were not being processed,
which prevented runtime internal logs from working correctly. To fix this,
processLogAndCacheFlags was refactored to accept explicit viper and config
parameters instead of relying on global state, enabling it to be called
during reload with the new configuration. This ensures that log and cache
settings are properly applied when the service reloads its configuration.
Restore the previously merged macOS VPN DNS post-settle refresh,
unchanged-exemption suppression, forced pf state flush, and self-upgrade
test after they were removed by the firewall-mode merge.
Also keep Windows NRPT pointed at loopback when the listener is configured
on a wildcard address.
On macOS DNS-intercept mode, when mDNSResponder owns *:53 ctrld falls back
to listening on 127.0.0.1:5354, and the pf rdr rules correctly redirect DNS
to the bound port at startup. However, DNS resolution later breaks with an
endless watchdog "anchor intact but probe FAILED -> force reload" loop and
`dig @127.0.0.1` timeouts.
Root cause is config reload. In CD mode, apiConfigReload refetches the
generated config (which always declares port 53) every hour and the reload
merge in runWait only inherits the running port when the new port is 0. The
generated config explicitly says 53, so `*p.cfg = *newCfg` reverts p.cfg to
port 53. The DNS listener goroutines are started only when !reload, so they
are never re-bound and stay on 5354. Every subsequent pf rebuild reads p.cfg
and targets the dead port 53.
Fix: after applying the reloaded config in DNS-intercept mode on darwin,
restore the actual bound listener IP/Port into the in-memory config via the
new preserveBoundListeners helper, logging the configured-vs-actual
divergence. A reload cannot move the running listener anyway, so this keeps
p.cfg consistent with reality; all pf rdr rules and the watchdog probe then
target the live port. The on-disk generated config is intentionally left
unchanged (still 53), so no generated-config change is required.
When IPv6 is available locally but the selected IPv6 DoH endpoint is
unroutable (e.g. dialing [2606:1a40::22]:443 returns "no route to host"
while IPv4 stays usable), ctrld re-bootstrapped and re-dialed the endpoint
every ~2s. A weekend soak produced ~46.7k "no route to host" lines, with
the dial/health-check loop dominating the log during bad windows.
Add bounded backoff/suppression for network-unreachable endpoints at two
levels:
- ParallelDialer (internal/net): track dial addresses that fail with
ENETUNREACH/EHOSTUNREACH and skip them for an exponentially growing,
bounded window (5s -> 60s). A successful dial clears the entry
immediately, so recovery is preserved when the route returns. When every
candidate is suppressed the dial fails fast and quietly instead of
hammering known-unroutable addresses.
- Upstream recovery loop (cmd/cli): demote unreachable check failures to
debug and back off the retry cadence (2s -> 60s) for an unreachable
streak; any other failure resets to the base cadence.
The new IsUnreachable classifier lives in internal/net and is reused by
cmd/cli's errNetworkError, so the unreachable-errno matching has a single
definition. Note the explicit winsock constants (10051/10065) are required
on Windows: syscall.ENETUNREACH/EHOSTUNREACH are Go's portable "invented"
values and never equal the raw WSA codes a failing connect surfaces.
Suppression and backoff are always bounded, so IPv6 is never disabled until
restart and recovers on its own once the route is back. Split-stack
selection and the #549 macOS intercept recovery work are untouched.
Adds unit tests for the classifier, the dialer's suppression tracker, and
the recovery backoff schedule.
The test:windows CI job intermittently failed to clean up .testbin with
"Access to the path '...cmd_cli.test.exe' is denied". This was previously
attributed to Windows Defender scanning the large unsigned test binaries,
and mitigated with Defender exclusions and cleanup retries. That was
treating a symptom.
Root cause: performUpgrade() self-upgrades by running
exec.Command(os.Executable(), "upgrade", "prod", "-vv") as a detached,
windowless child. In the real ctrld binary this re-execs ctrld and is
correct. Under `go test`, os.Executable() is the test binary itself, and
`go test` stops flag parsing at the first positional arg ("upgrade") and
ignores the rest -- so the child silently re-runs the entire test suite.
That child hits the upgrade tests again and spawns more detached children,
recursively: a fork bomb of hidden processes that pins the runner's
CPU/memory and keeps the test binary's image file locked. Windows refuses
to delete the image of a running process, hence the "Access is denied"
during after_script. Whether any children are still alive when cleanup
runs is a timing race, which is why the failure was flaky.
Two tests reached this path: Test_performUpgrade (directly) and
Test_selfUpgradeCheck (via selfUpgradeCheck -> performUpgrade on the
"upgrade allowed" case).
Fix:
- prog.go: extract the command construction into a package-level
newUpgradeCmd var. Production behavior is unchanged.
- main_test.go: stub newUpgradeCmd once in TestMain so the whole test
binary self-execs with `-test.run=^$` (matches no tests, exits
immediately) instead of re-running the suite. This covers every test
that reaches performUpgrade, present and future, while still exercising
the cmd.Start() success path.
When multiple network changes fire in quick succession (e.g., VPN
disconnect + interface swap), the second handleRecovery() call cancels
the first but inherits stale DoH transports, causing DNS blackouts
of up to 30 seconds.
Three changes to reduce worst-case recovery from ~30s to <3s:
1. ForceReBootstrap() on recovery entry — closes dead connections and
creates fresh transports synchronously before probing, replacing the
lazy ReBootstrap() flag that left stale connections for probes to hit.
2. Debounce handleRecovery() for network changes (500ms window) — only
the recovery flow is debounced; all other state updates (IP, pf
anchor, VPN DNS, tunnel checks) still run immediately on every event.
This eliminates the cancel-and-restart race without missing state.
3. Combined effect: ForceReBootstrap closes old in-flight connections
(closeTransports) and builds new ones (SetupTransport) atomically,
so recovery probes never inherit dead connections from a prior
recovery attempt.
Treat "socket missing" (ENOENT) and connection refused as expected when
probing the log server, and only log when the error indicates something
unexpected. This prevents noisy warnings when the log server has not
started yet.
Discover while doing captive portal tests.
Remove separate watchLinkState function and integrate link state change
handling directly into monitorNetworkChanges. This consolidates network
monitoring logic into a single place and simplifies the codebase.
Update netlink dependency from v1.2.1-beta.2 to v1.3.1 and netns from
v0.0.4 to v0.0.5 to use stable versions.
Move platform-specific network interface detection from cmd/cli/ to root package
as ValidInterfaces function. This eliminates code duplication and provides a
consistent interface for determining valid physical network interfaces across
all platforms.
- Remove duplicate validInterfacesMap functions from platform-specific files
- Add context parameter to virtualInterfaces for proper logging
- Update all callers to use ctrld.ValidInterfaces instead of local functions
- Improve error handling in virtual interface detection on Linux
Replace the legacy Unix socket log communication between `ctrld start` and
`ctrld run` with a modern HTTP-based system for better reliability and
maintainability.
Benefits:
- More reliable communication protocol using standard HTTP
- Better error handling and connection management
- Cleaner separation of concerns with dedicated endpoints
- Easier to test and debug with HTTP-based communication
- More maintainable code with proper abstraction layers
This change maintains backward compatibility while providing a more robust
foundation for inter-process communication between ctrld commands.
Capitalize the first letter of all log messages throughout the codebase
to improve readability and consistency in logging output.
Key improvements:
- All log messages now start with capital letters
- Consistent formatting across all logging statements
- Improved readability for debugging and monitoring
- Enhanced user experience with better formatted messages
Files updated:
- CLI commands and service management
- Internal client information discovery
- Network operations and configuration
- DNS resolver and proxy operations
- Platform-specific implementations
This completes the final phase of the logging improvement project,
ensuring all log messages follow consistent capitalization standards
for better readability and professional appearance.
Change DNS listener context from parent context to background context
so that listeners continue running during configuration reloads.
Listener configuration changes require a service restart, not reload,
so listeners must persist across reload operations.
This prevents DNS listeners from being terminated when the parent
context is cancelled during reload operations.
- Add entry/exit logging to all ServiceCommand methods (start, stop, status, reload, restart, uninstall)
- Replace mainLog.Load() calls with consistent logger variable usage throughout
- Capitalize all logging messages for better readability
- Add error context logging for service manager initialization failures
- Add debug logging for key operations (restart sequence, cleanup, validation)
- Improve error handling with proper error context in all service commands
- Add completion logging to track command execution flow
This improves debugging capabilities and provides better operational visibility
for service management operations while maintaining clean user-facing messages.
Move the network monitoring goroutine initialization outside the listener
loop to prevent it from being started multiple times. Previously, the
network monitoring was started once per listener during first run, which
was unnecessary and could lead to multiple monitoring instances.
The change ensures network monitoring is started only once per program
execution cycle, improving efficiency and preventing potential resource
waste from duplicate monitoring goroutines.
- Extract network monitoring goroutine from listener loop
- Start network monitoring once per run cycle instead of per listener
- Maintain same functionality while improving resource usage
- Add appVersion variable to store curVersion() result during init
- Change initCLI() to return *cobra.Command
- Move rootCmd creation inside initCLI() as local variable
- Replace all rootCmd.Version usage with appVersion variable
- Update Main() function to capture returned rootCmd from initCLI()
- Remove sync.Once guard from tests and use initCLI() directly
- Remove sync import from test file as it's no longer needed
This refactoring improves encapsulation by eliminating global state,
reduces version computation overhead, and simplifies test setup by
removing the need for sync.Once guards. All tests pass and the
application builds successfully.
- Move network monitoring initialization out of serveDNS() function
- Start network monitoring in a separate goroutine during program startup
- Remove context parameter from monitorNetworkChanges() as it's not used
- Simplify serveDNS() function signature by removing unused context parameter
- Ensure network monitoring starts only once during initial run, not on reload
This change improves separation of concerns by isolating network monitoring
from DNS serving logic, and prevents potential issues with multiple
monitoring goroutines if starting multiple listeners.
- Replace all direct newService() calls with ServiceCommand initialization
- Update command constructors to use ServiceCommand instead of ServiceManager
- Simplify LogCommand and UpgradeCommand structs by removing serviceManager field
- Remove unused global svcConfig variable from prog.go
- Improve consistency and centralize service creation logic
This change establishes a consistent pattern for service operations across
the codebase, making it easier to maintain and extend service-related
functionality.
Replace github.com/rs/zerolog with go.uber.org/zap throughout the codebase
to improve performance and provide better structured logging capabilities.
Key changes:
- Replace zerolog imports with zap and zapcore
- Implement custom Logger wrapper in log.go to maintain zerolog-like API
- Add LogEvent struct with chained methods (Str, Int, Err, Bool, etc.)
- Update all logging calls to use the new zap-based wrapper
- Replace JSON encoders with Console encoders for better readability
Benefits:
- Better performance with zap's optimized logging
- Consistent structured logging across all components
- Maintained zerolog-like API for easy migration
- Proper field context preservation for debugging
- Multi-core logging architecture for better output control
All tests pass and build succeeds.
- Add explicit foundDefaultRoute boolean variable to track default route discovery
- Initialize foundDefaultRoute to false and set to true only in success case
- Replace tautological condition `err == nil` with meaningful `foundDefaultRoute` check
- Fixes "tautological condition: nil == nil" linter error
The error occurred because err was being reused from net.Interfaces() call,
making the condition always true. Now we explicitly track whether a default
route was successfully found.
- Move version checking logic to shouldUpgrade for testability
- Move upgrade command execution to performUpgrade
- selfUpgradeCheck now composes these two for clarity
- Update and expand tests: focus on logic, not side effects
- Improves maintainability, testability, and separation of concerns
Add context parameter to validInterfacesMap for better error handling and
logging. Move Windows-specific network adapter validation logic to the
ctrld package. Key changes include:
- Add context parameter to validInterfacesMap across all platforms
- Move Windows validInterfaces to ctrld.ValidInterfaces
- Improve error handling for virtual interface detection on Linux
- Update all callers to pass appropriate context
This change improves error reporting and makes the interface validation
code more maintainable across different platforms.
By adding a logger field to "prog" struct, and use this field inside its
method instead of always accessing global mainLog variable. This at
least ensure more consistent usage of the logger during ctrld prog
runtime, and also help refactoring the code more easily in the future
(like replacing the logger library).
So setting up logging for ctrld binary and ctrld packages could be done
more easily, decouple the required setup for interactive vs daemon
running.
This is the first step toward replacing rs/zerolog libary with a
different logging library.
This commit reverts changes from v1.4.5 to v1.4.7, to prepare for v2.0.0
branch codes.
Changes includes in these releases have been included in v2.0.0 branch
already.
Details:
Revert "feat: add --rfc1918 flag for explicit LAN client support"
This reverts commit 0e3f764299.
Revert "Upgrade quic-go to v0.54.0"
This reverts commit e52402eb0c.
Revert "docs: add known issues documentation for Darwin 15.5 upgrade issue"
This reverts commit 2133f31854.
Revert "start mobile library with provision id and custom hostname."
This reverts commit a198a5cd65.
Revert "Add OPNsense new lease file"
This reverts commit 7af29cfbc0.
Revert ".github/workflows: bump go version to 1.24.x"
This reverts commit ce1a165348.
Revert "fix: ensure upstream health checks can handle large DNS responses"
This reverts commit fd48e6d795.
Revert "refactor(prog): move network monitoring outside listener loop"
This reverts commit d71d1341b6.
Revert "fix: correct Windows API constants to fix domain join detection"
This reverts commit 21855df4af.
Revert "refactor: move network monitoring to separate goroutine"
This reverts commit 66e2d3a40a.
Revert "refactor: extract empty string filtering to reusable function"
This reverts commit 36a7423634.
Revert "cmd/cli: ignore empty positional argument for start command"
This reverts commit e616091249.
Revert "Avoiding Windows runners file locking issue"
This reverts commit 0948161529.
Revert "refactor: split selfUpgradeCheck into version check and upgrade execution"
This reverts commit ce29b5d217.
Revert "internal/router: support Ubios 4.3+"
This reverts commit de24fa293e.
Revert "internal/router: support Merlin Guest Network Pro VLAN"
This reverts commit 6663925c4d.
Move the network monitoring goroutine initialization outside the listener
loop to prevent it from being started multiple times. Previously, the
network monitoring was started once per listener during first run, which
was unnecessary and could lead to multiple monitoring instances.
The change ensures network monitoring is started only once per program
execution cycle, improving efficiency and preventing potential resource
waste from duplicate monitoring goroutines.
- Extract network monitoring goroutine from listener loop
- Start network monitoring once per run cycle instead of per listener
- Maintain same functionality while improving resource usage
- Move network monitoring initialization out of serveDNS() function
- Start network monitoring in a separate goroutine during program startup
- Remove context parameter from monitorNetworkChanges() as it's not used
- Simplify serveDNS() function signature by removing unused context parameter
- Ensure network monitoring starts only once during initial run, not on reload
This change improves separation of concerns by isolating network monitoring
from DNS serving logic, and prevents potential issues with multiple
monitoring goroutines if starting multiple listeners.
- Move version checking logic to shouldUpgrade for testability
- Move upgrade command execution to performUpgrade
- selfUpgradeCheck now composes these two for clarity
- Update and expand tests: focus on logic, not side effects
- Improves maintainability, testability, and separation of concerns
Fixes search domains not being preserved when the resolv.conf file is
reverted to its previous state. This ensures that important domain
search configuration is maintained during DNS configuration changes.
The search domains handling was missing in setResolvConf function,
which is responsible for restoring DNS settings.
So using "ctrld stop" or service manager to stop ctrld will end up with
the same result, stopped ctrld with a working DNS, and deactivation pin
code will always have effects if set.
Anthony Wong