`ctrld start --cd-org <code>` logged its arguments as-is in the debug line
`intercept upgrade check: args=[...]`. redactedArgs formats them through
redactSecrets with provisionSecrets, covering --cd-org and both parts of
--cd, in either flag form.
- Stop closes the control server right after the deactivation-pin check,
so a stopping process no longer answers /started for its successor.
releaseResources still closes it for stop paths that skip Stop; a
sync.Once makes the double close safe.
- upgrade and restart wait for a 200 from /started via
newReadySocketControlClient; a 408 (start never finished) now rolls the
upgrade back and makes restart warn instead of reporting success.
- The plain socket client still accepts any answer, for the
deactivation-pin path.
Internal Domains now support three modes. "os" is unchanged.
"resolvers" becomes explicit resolver with network fallback: the
configured resolvers are tried first, and a timeout, unreachable
resolver, SERVFAIL or NXDOMAIN hands the query to matching VPN DNS
servers, then domain-less VPN DNS servers, then the OS resolver's LAN
nameservers through the new LanOnlyQueryCtx. "resolvers_only" keeps
the previous strict behavior. Down fallback-mode resolvers are skipped
and re-checked in the background at most every 30s. Generated upstreams
are identified by the UpstreamConfig.InternalDomain marker instead of
the internal_ key prefix.
Master adaptations: the fallback state lives on proxyRequest and is
applied in tryUpstreams, shouldContinueWithNextUpstream and
handleAllUpstreamsFailure instead of v1.0's monolithic proxy loop.
Resolver construction uses master's context-aware NewResolver, and the
background re-check marks its context private so transport errors that
name the resolver stay at debug. Tests use master's zap JSON capture
and VPN manager logger.
A hard maintenance window took DNS down on endpoints that already had a
working configuration: the bootstrap preflight read the maintenance answer
as a failed fetch and exited, turning a temporary API outage into an
endpoint outage.
Startup now falls back to the configuration on disk when a content- and
identity-bound marker records that the API produced it. Without one, ctrld
stops with API_MAINTENANCE (exit 37). Maintenance never self-uninstalls or
counts as a permanent rejection, a fallback run retries every 5-7 minutes
until a recovered configuration is applied, and deactivation is refused
with 503 while the PIN cannot be verified.
Master adaptation: pending-recovery and PIN-known handling live in
applyFetchedResolverConfig; logging uses *ctrld.Logger; the reload fetch
seam is fetchResolverConfigFn; the control-server and self-uninstall
probes use the existing fetchResolverConfig seam with loggerCtx.
Keep missing interfaces as debug skips while preserving enumeration
causes and error-level failures restoring DNS on existing interfaces.
Adapt the logger and NetworkManager method to master. Stub saved DNS
lookup in tests because master stores it through ctrld rather than the
CLI homedir; tests must not touch the service's saved DNS files.
Port the final merged change from !1009 to master, retaining master's
context-aware logger/resolver APIs, proxy decomposition and firewall
refresh handling. Adapt source regression tests and cover master's
refresh handler and transport-level private-address diagnostics.
tryUpdateListenerConfig treated an explicit --intercept-mode off the same
as an empty flag and fell back to the persisted config value. run() selects
the listener strategy before it clears the persisted mode, so the first
start after a revert to standard mode selected the intercept strategy from
a stale dns/hard value while setDNS kept interception off.
Extract the resolution into listenerInterceptMode and make an explicit off
final, the same contract as setDNS. Add a regression test that fails
without the fix.
The warning reported err, the resolver-config fetch error, which is nil on
every path that reaches it - so a rejected custom config was logged with no
reason attached. cfgErr holds the validation failure.
A host carrying orphaned WFP filters from a ctrld build that predates
session-scoped ownership cannot recover on its own. In Firewall Mode those
filters block all non-allowlisted outbound traffic machine-wide, which denies
the replacement ctrld's own API bootstrap. API-managed startup then sits in the
resolver-config retry loop forever and never reaches startWFPFilters, where the
only stale-sublayer cleanup lived. Cleanup needs startup, startup needs the
network, the network needs cleanup - the host stays locked out until a reboot.
Add cleanupStaleDNSInterceptState() and call it early in run(), before the
network-up wait and before the API preflight. On Windows it deletes ctrld's WFP
sublayer, which takes its child filters with it, so a previous process's
enforcement is gone before this process makes its first connection. Objects
owned by a live session cannot be deleted, so a running ctrld is unaffected and
"nothing to clean up" stays at debug level; an actual removal logs a warning,
since it means a previous ctrld left machine-wide enforcement installed.
macOS and the other platforms get no-op implementations: pf enforcement does not
outlive the process, and startDNSIntercept already flushes the anchor and
removes a stale anchor file before loading rules.
The cleanup inside startWFPFilters stays as a second line of defense.
The cleanup session is deliberately NOT dynamic. FwpmSubLayerDeleteByKey0 is
documented to fail with FWP_E_DYNAMIC_SESSION_IN_PROGRESS when called from a
dynamic session for an object that was not added in one, and the only orphans
that can exist are exactly those: a ctrld predating session-scoped ownership
added its sublayer statically. Anything a newer ctrld leaves behind is removed by
the OS when its session ends. Opening this session dynamically would have made
the cleanup a no-op in the one case it exists for, while still logging "no stale
WFP state".
A concurrently running ctrld is protected by documented ownership rather than by
the child-filter question below: a session-scoped ctrld's sublayer belongs to a
different dynamic session, so the delete fails with FWP_E_WRONG_SESSION. That
matters because this call is made unconditionally at startup, in every intercept
mode, so an interactive "ctrld run" alongside a healthy service reaches it.
A ctrld predating session scoping has no such protection: it holds a non-dynamic
sublayer, which is exactly what this targets and is indistinguishable from an
orphan. Two guards cover that instead. Elevation, because opening a WFP engine
and deleting ctrld's sublayer must not be reachable from an unprivileged local
process - FwpmEngineOpen0 is expected to fail without elevation, but that is a
property of the API rather than something this code checked, and it was the only
barrier. And interactive invocation, since a service start is not interactive: the
deadlock case still gets cleaned, while a hand-run "ctrld run" beside a live
service does not strip its enforcement. That second guard requires positive
evidence of absence - ctrldServiceLiveness answers unknown for an unreachable SCM
or a service mid-stop, and unknown skips the cleanup exactly as running does,
because "could not be observed" is not "not there".
Both are asserted against the deletion itself, not only against the predicate: a
caller-level test substitutes the WFP delete and the guard inputs, so a future
change that stops consulting the guard, or consults it and deletes anyway, fails
rather than staying green.
Delete failures are no longer collapsed into "nothing to clean up". Only
FWP_E_SUBLAYER_NOT_FOUND means that; any other code means state exists under our
GUID that we could not remove, which is the lockout condition itself, so it is
logged as a warning naming the code.
Whether deleting the sublayer is sufficient is left explicitly UNRESOLVED rather
than asserted. It is sufficient only if the delete also removes the filters
inside it. FwpmSubLayerDeleteByKey0's Remarks say nothing about child filters
either way, while object management states that an object cannot be deleted until
everything referencing it has been - and FWP_E_IN_USE exists for that. Whether a
filter's subLayerKey counts as such a reference is not documented, and this code
cannot be exercised off-Windows, so the earlier claim that the delete "takes its
child filters with it" is removed from the comment here and from
docs/wfp-dns-intercept.md, which carried it from before this branch.
The behaviour is safe under both readings: the delete is attempted, and
FWP_E_IN_USE is reported rather than counted as success, so a support log
distinguishes "cleared it" from "could not clear it". If a live Windows check
shows FWP_E_IN_USE against orphaned filters, the cleanup must enumerate and
delete those filters first. That is deliberately not written blind:
FWPM_FILTER_ENUM_TEMPLATE0 has no sublayer field, so selecting ctrld's own
filters means reading subLayerKey at a computed offset in FWPM_FILTER0, and
getting that offset wrong would delete other software's filters - a worse failure
than not cleaning up.
Refs: https://learn.microsoft.com/en-us/windows/win32/api/fwpmu/nf-fwpmu-fwpmsublayerdeletebykey0
Refs: https://learn.microsoft.com/en-us/windows/win32/fwp/object-management
processCDFlags retries the resolver-config fetch indefinitely by design: a
device that has no working network at boot must eventually come up. The loop
had no cancellation, so a stop request arriving while the API is unreachable
was ignored - the process kept retrying long after "Service stopped" was
logged, doing work on behalf of a service the OS considers stopped. The
Windows Firewall Mode incident showed this concretely: API retries continued
15 seconds after the stop completed, so stopping the service could not release
what the process was still holding.
Thread a context through processCDFlags and derive it from p.stopCh, in both
the startup preflight and the config-reload path. The loop now returns as soon
as the context is cancelled, checked both before a retry and after backoff
returns (backoff can wake up on cancellation).
A stop during preflight now exits the way a normal stop does, without Fatal,
so the service manager does not treat it as a failed start and apply its
restart policy to a service the operator just asked to stop.
Also bind the two API requests in internal/controld to the caller's context.
They were built with http.NewRequest, so an in-flight request ignored
cancellation and waited out its own timeout instead.
Covered by tests that assert what the incident needed: a stop request during
preflight ends the retry/backoff loop and reports cancellation rather than
continuing to retry after the service reports itself stopped, an
already-cancelled context makes at most one attempt, and contextFromStopCh
handles its three cases (cancelled by stopCh, released by cancel, usable with
no stop channel). Removing either cancellation check makes these tests hang
until the test timeout. The tests are also what exercise fetchResolverConfig,
the seam this commit introduces.
Sampling the stop state is the whole point of runAPIPreflight rather than doing
this inline. A stop and a failure need opposite handling - one exits quietly, the
other self-uninstalls a deleted device, surfaces the error to a mobile app, and
reports a failed start - so the two must not be confused. Reading it from the
context after cancelling would report stopped for every failure, since
CancelFunc sets ctx.Err() regardless of whether anyone asked to stop; the stop
channel is read directly instead, which also does not depend on the context
watcher goroutine having been scheduled.
During reload operations, log and cache flags were not being processed,
which prevented runtime internal logs from working correctly. To fix this,
processLogAndCacheFlags was refactored to accept explicit viper and config
parameters instead of relying on global state, enabling it to be called
during reload with the new configuration. This ensures that log and cache
settings are properly applied when the service reloads its configuration.
On macOS DNS-intercept mode, when mDNSResponder owns *:53 ctrld falls back
to listening on 127.0.0.1:5354, and the pf rdr rules correctly redirect DNS
to the bound port at startup. However, DNS resolution later breaks with an
endless watchdog "anchor intact but probe FAILED -> force reload" loop and
`dig @127.0.0.1` timeouts.
Root cause is config reload. In CD mode, apiConfigReload refetches the
generated config (which always declares port 53) every hour and the reload
merge in runWait only inherits the running port when the new port is 0. The
generated config explicitly says 53, so `*p.cfg = *newCfg` reverts p.cfg to
port 53. The DNS listener goroutines are started only when !reload, so they
are never re-bound and stay on 5354. Every subsequent pf rebuild reads p.cfg
and targets the dead port 53.
Fix: after applying the reloaded config in DNS-intercept mode on darwin,
restore the actual bound listener IP/Port into the in-memory config via the
new preserveBoundListeners helper, logging the configured-vs-actual
divergence. A reload cannot move the running listener anyway, so this keeps
p.cfg consistent with reality; all pf rdr rules and the watchdog probe then
target the live port. The on-disk generated config is intentionally left
unchanged (still 53), so no generated-config change is required.
Currently there is no limit on PIN attempts, allowing unlimited
brute force if an attacker gains socket access. While the socket is
root-only by default, rate limiting is cheap defense-in-depth.
Current code writes to a predictable path, which on systems without
`fs.protected_symlinks` (e.g. embedded routers) could allow a local
attacker with API compromise to perform symlink attacks.
When port 53 is taken (e.g. by mDNSResponder), ctrld failed with
'could not find available listen ip and port' instead of falling back
to port 5354. Root cause: tryUpdateListenerConfig() checked the
dnsIntercept bool, which is derived in prog.run() AFTER listener
config is resolved.
Fix: check interceptMode string directly (CLI flag + config fallback)
in a new tryUpdateListenerConfigIntercept() that tries 127.0.0.1:53
then 127.0.0.1:5354.
Also updates buildPFAnchorRules() to use the actual listener IP/port
from config instead of hardcoded 127.0.0.1:53, so pf rules redirect
to wherever ctrld is actually listening.
Treat "socket missing" (ENOENT) and connection refused as expected when
probing the log server, and only log when the error indicates something
unexpected. This prevents noisy warnings when the log server has not
started yet.
Discover while doing captive portal tests.
Replace the legacy Unix socket log communication between `ctrld start` and
`ctrld run` with a modern HTTP-based system for better reliability and
maintainability.
Benefits:
- More reliable communication protocol using standard HTTP
- Better error handling and connection management
- Cleaner separation of concerns with dedicated endpoints
- Easier to test and debug with HTTP-based communication
- More maintainable code with proper abstraction layers
This change maintains backward compatibility while providing a more robust
foundation for inter-process communication between ctrld commands.
Capitalize the first letter of all log messages throughout the codebase
to improve readability and consistency in logging output.
Key improvements:
- All log messages now start with capital letters
- Consistent formatting across all logging statements
- Improved readability for debugging and monitoring
- Enhanced user experience with better formatted messages
Files updated:
- CLI commands and service management
- Internal client information discovery
- Network operations and configuration
- DNS resolver and proxy operations
- Platform-specific implementations
This completes the final phase of the logging improvement project,
ensuring all log messages follow consistent capitalization standards
for better readability and professional appearance.
Add comprehensive logging to configuration management and network operations
across all supported platforms to improve visibility into system setup and
network configuration processes.
Key improvements:
- Configuration initialization and validation logging
- CLI flag processing visibility (listen, log, cache flags)
- IP allocation/deallocation tracking across platforms
- DNS configuration operations logging (Linux, macOS, FreeBSD)
- Upstream bootstrap and fallback operation tracking
- Listener configuration initialization logging
This provides complete visibility into configuration management and network
setup operations, helping identify configuration issues and network setup
problems across different platforms.
This commit extends the documentation effort by adding detailed explanatory
comments to key CLI components and core functionality throughout the cmd/
directory. The changes focus on explaining WHY certain logic is needed,
not just WHAT the code does, improving code maintainability and helping
developers understand complex business decisions.
Key improvements:
- Main entry points: Document CLI initialization, logging setup, and cache
configuration with reasoning for design decisions
- DNS proxy core: Explain DNS proxy constants, data structures, and core
processing pipeline for handling DNS queries
- Service management: Document service command structure, configuration
patterns, and platform-specific service handling
- Logging infrastructure: Explain log buffer management, level encoders,
and log formatting decisions for different use cases
- Metrics and monitoring: Document Prometheus metrics structure, HTTP
endpoints, and conditional metric collection for performance
- Network handling: Explain Linux-specific network interface filtering,
virtual interface detection, and DNS configuration management
- Hostname validation: Document RFC1123 compliance and DNS naming
standards for system compatibility
- Mobile integration: Explain HTTP retry logic, fallback mechanisms, and
mobile platform integration patterns
- Connection management: Document connection wrapper design to prevent
log pollution during process lifecycle
Technical details:
- Added explanatory comments to 11 additional files in cmd/cli/
- Maintained consistent documentation style and format
- Preserved all existing functionality while improving code clarity
- Enhanced understanding of complex business logic and platform-specific
behavior
These comments help future developers understand the reasoning behind
complex decisions, making the codebase more maintainable and reducing
the risk of incorrect modifications during maintenance.
This commit adds detailed explanatory comments throughout the codebase to explain
WHY certain logic is needed, not just WHAT the code does. This improves code
maintainability and helps developers understand the reasoning behind complex
decisions.
Key improvements:
- Version string processing: Explain why "v" prefix is added for semantic versioning
- Control-D configuration: Explain why config is reset to prevent mixing of settings
- DNS server categorization: Explain LAN vs public server handling for performance
- Listener configuration: Document complex fallback logic for port/IP selection
- MAC address normalization: Explain cross-platform compatibility needs
- IPv6 address processing: Document Unix-specific interface suffix handling
- Log content truncation: Explain why large content is limited to prevent flooding
- IP address categorization: Document RFC1918 prioritization logic
- IPv4/IPv6 separation: Explain network stack compatibility needs
- DNS priority logic: Document different priority levels for different scenarios
- Domain controller processing: Explain Windows API prefix handling
- Reverse mapping creation: Document API encoding/decoding needs
- Default value fallbacks: Explain why defaults prevent system failures
- IP stack configuration: Document different defaults for different upstream types
These comments help future developers understand the reasoning behind complex
business logic, making the codebase more maintainable and reducing the risk of
incorrect modifications during maintenance.
- Add appVersion variable to store curVersion() result during init
- Change initCLI() to return *cobra.Command
- Move rootCmd creation inside initCLI() as local variable
- Replace all rootCmd.Version usage with appVersion variable
- Update Main() function to capture returned rootCmd from initCLI()
- Remove sync.Once guard from tests and use initCLI() directly
- Remove sync import from test file as it's no longer needed
This refactoring improves encapsulation by eliminating global state,
reduces version computation overhead, and simplifies test setup by
removing the need for sync.Once guards. All tests pass and the
application builds successfully.
- Update all Init*Cmd function signatures to accept rootCmd parameter:
* InitServiceCmd(rootCmd *cobra.Command)
* InitClientsCmd(rootCmd *cobra.Command)
* InitLogCmd(rootCmd *cobra.Command)
* InitUpgradeCmd(rootCmd *cobra.Command)
* InitRunCmd(rootCmd *cobra.Command)
* InitInterfacesCmd(rootCmd *cobra.Command)
- Update function calls in cli.go to pass rootCmd parameter
- Update InitInterfacesCmd call in commands_service.go
Benefits:
- Eliminates global state dependency on rootCmd variable
- Makes dependencies explicit in function signatures
- Improves testability by allowing different root commands
- Better encapsulation and modularity
- Replace all direct newService() calls with ServiceCommand initialization
- Update command constructors to use ServiceCommand instead of ServiceManager
- Simplify LogCommand and UpgradeCommand structs by removing serviceManager field
- Remove unused global svcConfig variable from prog.go
- Improve consistency and centralize service creation logic
This change establishes a consistent pattern for service operations across
the codebase, making it easier to maintain and extend service-related
functionality.
Create commands_run.go following the same modular pattern as other
command files. Move initRunCmd logic to InitRunCmd function with
consistent naming and complete functionality preservation.
Update cli.go to use InitRunCmd() instead of initRunCmd() and clean
up commands.go by removing the old function and unused imports.
This completes the modular refactoring pattern where each command type
has its own dedicated file with focused responsibility.
Remove all unused service command functions (initStartCmd, initStopCmd,
initRestartCmd, initReloadCmd, initStatusCmd, initUninstallCmd,
initInterfacesCmd, initClientsCmd, initUpgradeCmd, initServicesCmd)
from commands.go since they have been replaced by modular implementations
in dedicated files. Keep only essential functions: CommandRunner interface,
ServiceManager struct, NewServiceManager function, Status method,
initRunCmd function, and filterEmptyStrings function.
Update cli.go to use InitClientsCmd() and InitUpgradeCmd() instead of
the old init functions. Clean up unused imports and simplify
filterEmptyStrings implementation.
This reduces commands.go from 1202 lines to 103 lines (91% reduction)
and eliminates code duplication while improving maintainability.
Replace individual service command initialization with unified InitServiceCmd()
that creates a complete service command hierarchy. Port all original logic
from initStartCmd, initStopCmd, initRestartCmd, initReloadCmd, initStatusCmd,
and initUninstallCmd into ServiceCommand methods with proper dependency injection.
Key changes:
- Port complete Start logic including config validation, service installation,
DNS management, and self-check functionality
- Port complete Stop logic with deactivation pin validation and DNS cleanup
- Port complete Restart logic with config validation and DNS restoration
- Port complete Reload logic with HTTP status handling and restart fallback
- Port complete Status logic with proper exit codes
- Port complete Uninstall logic with cleanup file removal
- Add all necessary flags to service commands (iface, pin, etc.)
- Use InitInterfacesCmd() for interfaces subcommand
- Simplify cli.go by replacing multiple init calls with single InitServiceCmd()
This refactoring eliminates code duplication, improves maintainability, and
ensures all service commands have their complete original functionality.
Remove the old initLogCmd function from commands.go and update cli.go
to use the new InitLogCmd function from commands_log.go. Complete
the log command refactoring by adding the missing InitLogCmd function
with proper command structure and error handling.
- Add condition to skip port 53 attempts when using zero IP address
- Improve error logging by using structured error field instead of string formatting
- Remove redundant error information from log message format
The changes prevent unnecessary port 53 binding attempts when using zero IP
addresses and improve log readability by using zap's structured error fields.
Replace github.com/rs/zerolog with go.uber.org/zap throughout the codebase
to improve performance and provide better structured logging capabilities.
Key changes:
- Replace zerolog imports with zap and zapcore
- Implement custom Logger wrapper in log.go to maintain zerolog-like API
- Add LogEvent struct with chained methods (Str, Int, Err, Bool, etc.)
- Update all logging calls to use the new zap-based wrapper
- Replace JSON encoders with Console encoders for better readability
Benefits:
- Better performance with zap's optimized logging
- Consistent structured logging across all components
- Maintained zerolog-like API for easy migration
- Proper field context preservation for debugging
- Multi-core logging architecture for better output control
All tests pass and build succeeds.
Dev Scribe