mirror of
https://github.com/Control-D-Inc/ctrld.git
synced 2026-09-04 13:36:35 +02:00
fix(firewall): support VM/container egress under macOS Firewall Mode
macOS Firewall Mode blocked forwarded/NATed VM/container egress: a guest
resolves DNS through a path host ctrld does not observe, so the guest-resolved
public IP never enters <ctrld_allowed> and the blanket outbound block drops the
guest's TCP/443.
Make VM/container guests first-class Firewall Mode clients by forcing their DNS
through ctrld. The trusted source subnets are the UNION of:
- Auto-detected VM/NAT networks (default, no config): interfaces that are up,
carry an RFC1918 IPv4 address, and whose VM ownership can be proven either
by a vendor-specific name (vnic/vboxnet/vmnet) or by being a bridge* whose
member list contains a vendor VM interface (typically vmenet*). Each keeps
its ingress interface, and its pf rules are scoped "on <iface>" so an
unrelated interface on the same private range is never affected.
- service.firewall_forwarded_sources (opt-in): explicit IPv4 CIDRs, matched on
the source CIDR alone, for stacks whose ownership cannot be proven. Config
adds only; an invalid or non-IPv4 entry is dropped with a warning.
Checking bridge membership is what makes the common case work without config.
Every vmnet.framework stack - UTM and other Virtualization.framework guests,
Docker Desktop, Multipass, Fusion 12.1+ NAT - puts the RFC1918 gateway address
on a bridge10x interface and attaches the vendor-named vmenet* interface as an
address-less member, so matching on interface name alone never sees them.
Membership is the ownership proof a bridge name lacks: macOS shares that
namespace with Thunderbolt/aggregated links (ctrld's own tunnel-change code
treats bridge0 as physical), and such a bridge has en* members, so it stays
untrusted however private its address. Members are read with ifconfig, and only
for a bridge that already carries an RFC1918 IPv4 address, so a host with no VM
running executes no subprocess.
Per source subnet, plaintext DNS (53) is force-routed through ctrld (route-to
lo0 -> existing rdr-on-lo0) so guest resolutions are policy-enforced and
populate the allowlist; guest egress to allowed IPs is then permitted by the
existing <ctrld_allowed> rule. DoT (853) is blocked so guests cannot swap in an
alternate resolver. DoH/443 is a documented limitation.
Rules are emitted strictly per address family. Sources are IPv4 (interception
targets ctrld's IPv4 listener), so only inet rules are generated: pf rejects an
entire anchor over a single "inet6 ... from 192.168.x.0/24" mismatch, which
would take DNS interception down with it. Guest IPv6 DoT is covered by the
blanket IPv6 block instead, since such a resolver never enters <ctrld_allowed>.
firewall_forwarded_sources is deliberately not validated with `cidr`. Entries
are checked where they are parsed and a bad one is dropped while the rest of the
set still applies. A hard validator would make one typo in an MDM-pushed subnet
fatal at startup - validateConfig exits the process - taking DNS service down
for the whole host over a line that only ever widened a firewall allowance. The
warning fires when the set of unusable entries changes rather than on every
parse, since config is re-read on every anchor build and every watchdog tick.
Reconcile the trust set at runtime, since VM interfaces appear and disappear
while ctrld runs and no existing path rebuilds an intact anchor for that
(ensurePFAnchorActive returns early, checkTunnelInterfaceChanges tracks only
tunnels, pfInterceptMonitor rebuilds only after a failed host probe). On a
change, rebuild the anchor and drop the pf states of the affected subnets
(targeted pfctl -k, not a global state flush), because rules govern only new
states: a stopped guest would otherwise keep using states created while it was
trusted, and a newly trusted one would keep bypassing interception until its
states expired. Reconciliation runs on interface appear/disappear, on network
changes, on the delayed post-change re-checks (a VM network often gets its
address after its interface appears), and on the pf watchdog tick, which bounds
guest start/stop convergence to one interval even with no network event.
Convergence is not latched on failure: the applied set advances, and states are
killed, only after pf has accepted the new anchor. reloadForwardedSourceAnchor
reports write/pfctl failures to the caller, which then keeps the previous set
recorded and logs a warning, so the next reconcile retries the same transition
instead of going quiet with the old anchor still installed. The whole
compare-reload-record sequence is serialized so a watchdog tick and a network
change cannot both rebuild or interleave snapshots.
Every anchor rebuild records the forwarded-source set it installed, and takes
that set as a parameter rather than re-detecting internally. Otherwise a rebuild
triggered by something else (tunnel change, watchdog restore, VPN DNS
exemptions, forced reload, startup) leaves the snapshot at the older set and the
next reconcile "discovers" the same change again: another rebuild, another round
of killed guest states, and a transition logged for something already in effect.
Passing the set in also means what pf loaded is exactly what gets recorded, so
an interface appearing mid-reconcile cannot leave the snapshot describing a set
that was never installed.
The anchor file is replaced atomically (temp file plus rename). Seven paths
rebuild it from timers and network-change callbacks in their own goroutines with
no lock between them, and os.WriteFile truncates before writing, so a pfctl -f
racing that window could read a partial ruleset and reject the anchor - taking
DNS interception down until the next watchdog restore.
Report the effective trust set at startup and on every change, naming each
subnet's origin ("192.168.64.0/24 (configured)" vs "(auto-detected on
bridge100)"), and say so explicitly when the set is empty, including what
auto-detection requires. Configured entries previously produced no log output at
all, so an admin who set firewall_forwarded_sources could not confirm it had
taken effect without reading pf rules. Per-source detection logging is at debug,
since detection re-runs on every anchor build.
Explicit, per-subnet trust boundary (RFC1918-only auto-detect, proven VM
ownership, interface-scoped rules, config adds only), not an interface-wide
permit: direct public IPs the guest never resolved through ctrld stay blocked.
Tests cover rule generation and its interface scoping, the no-blanket-permit
boundary, single-family emission (no inet6 rule for an IPv4 source, IPv6 sources
skipped), a real pfctl -n -f parse of both the forwarded-source rules alone and
the full anchor (group-scoped rules stripped, since _ctrld exists only where the
service is installed), invalid/duplicate/non-IPv4 config entries, that a bad
entry does not fail config validation, the union set and its signature, a
deterministic guest start/stop lifecycle asserting both the rebuild points and
which subnets' states must be dropped, anchor-reload failure followed by a
successful retry, that reconcile is inert outside firewall mode, ifconfig member
parsing against real bridge output, and the trust decision per interface -
including everything that must NOT qualify: a Thunderbolt bridge, a
public-range VM bridge, an IPv6-only bridge, an address-less vendor interface, a
physical uplink and a VPN tunnel. docs/firewall-mode.md documents the boundary,
what auto-detection can and cannot prove, the address-family constraint, the
lifecycle/retry behavior, how to confirm the trust set from the log, and that a
resolver running inside the guest is not supported (its encrypted upstream
leaves the host nothing to allowlist).
This commit is contained in:
@@ -12,6 +12,7 @@ import (
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
@@ -83,6 +84,51 @@ const (
|
||||
pfAnchorFile = "/etc/pf.anchors/com.controld.ctrld"
|
||||
)
|
||||
|
||||
// pfAnchorWriteMu serializes writers of the anchor file. Seven code paths rebuild it
|
||||
// (startup, tunnel change, watchdog restore, VPN DNS exemptions, forced reload,
|
||||
// forwarded-source reconcile, firewall shutdown) from timers and network-change
|
||||
// callbacks in their own goroutines, with no lock between them.
|
||||
var pfAnchorWriteMu sync.Mutex
|
||||
|
||||
// writePFAnchorFile replaces the anchor file atomically: write a temp file in the
|
||||
// same directory, then rename over the target.
|
||||
//
|
||||
// os.WriteFile truncates and then writes, so a pfctl -f racing that window can read a
|
||||
// partial ruleset and reject the anchor - taking DNS interception down until the next
|
||||
// watchdog restore. A rename is atomic, so a concurrent reader sees either the whole
|
||||
// previous ruleset or the whole new one.
|
||||
//
|
||||
// Two writers can still be followed by two loads in either order, but every load now
|
||||
// sees a complete ruleset, and each writer's load re-applies a full anchor, so the
|
||||
// worst case is redundant work rather than a broken anchor.
|
||||
func writePFAnchorFile(rules string) error {
|
||||
pfAnchorWriteMu.Lock()
|
||||
defer pfAnchorWriteMu.Unlock()
|
||||
|
||||
tmp, err := os.CreateTemp(filepath.Dir(pfAnchorFile), ".ctrld-anchor-*")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
defer func() {
|
||||
// No-op once the rename succeeded; removes the temp file on any failure path.
|
||||
_ = os.Remove(tmpName)
|
||||
}()
|
||||
|
||||
if _, err := tmp.WriteString(rules); err != nil {
|
||||
tmp.Close()
|
||||
return err
|
||||
}
|
||||
if err := tmp.Chmod(0644); err != nil {
|
||||
tmp.Close()
|
||||
return err
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmpName, pfAnchorFile)
|
||||
}
|
||||
|
||||
// pfState holds the state of the pf DNS interception on macOS.
|
||||
type pfState struct {
|
||||
anchorFile string
|
||||
@@ -240,12 +286,13 @@ func (p *prog) startDNSIntercept() error {
|
||||
}
|
||||
}
|
||||
|
||||
rules := p.buildPFAnchorRules(initialExemptions)
|
||||
forwarded := p.currentForwardedSources()
|
||||
rules := p.buildPFAnchorRulesWith(initialExemptions, forwarded)
|
||||
|
||||
if err := os.MkdirAll(pfAnchorDir, 0755); err != nil {
|
||||
return fmt.Errorf("dns intercept: failed to create pf anchor directory %s: %w", pfAnchorDir, err)
|
||||
}
|
||||
if err := os.WriteFile(pfAnchorFile, []byte(rules), 0644); err != nil {
|
||||
if err := writePFAnchorFile(rules); err != nil {
|
||||
return fmt.Errorf("dns intercept: failed to write pf anchor file %s: %w", pfAnchorFile, err)
|
||||
}
|
||||
mainLog.Load().Debug().Msgf("DNS intercept: wrote pf anchor file: %s", pfAnchorFile)
|
||||
@@ -255,6 +302,9 @@ func (p *prog) startDNSIntercept() error {
|
||||
os.Remove(pfAnchorFile)
|
||||
return fmt.Errorf("dns intercept: failed to load pf anchor: %w (output: %s)", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
// Baseline the reconcile snapshot: the anchor just loaded already contains this
|
||||
// set, so the first watchdog tick must not treat it as a change.
|
||||
p.recordAppliedForwardedSources(forwarded)
|
||||
mainLog.Load().Debug().Msgf("DNS intercept: loaded pf anchor %q from %s", pfAnchorName, pfAnchorFile)
|
||||
|
||||
if err := p.ensurePFAnchorReference(); err != nil {
|
||||
@@ -747,7 +797,22 @@ func (p *prog) validateDNSIntercept() error {
|
||||
// from the redirect to prevent ctrld from querying itself in a loop.
|
||||
//
|
||||
// pf requires strict rule ordering: translation (rdr) BEFORE filtering (pass).
|
||||
// buildPFAnchorRules detects the current forwarded-source set itself. Callers that
|
||||
// need to know which set the anchor was built from - so they can record it as applied
|
||||
// - should use buildPFAnchorRulesWith instead.
|
||||
func (p *prog) buildPFAnchorRules(vpnExemptions []vpnDNSExemption) string {
|
||||
return p.buildPFAnchorRulesWith(vpnExemptions, p.currentForwardedSources())
|
||||
}
|
||||
|
||||
// buildPFAnchorRulesWith is buildPFAnchorRules over an explicit forwarded-source set.
|
||||
//
|
||||
// Every rebuild path used to re-detect internally, which meant the caller could not
|
||||
// tell what it had just installed. The reconcile snapshot therefore stayed at the
|
||||
// older set, and the next reconcile "discovered" the same change again: another
|
||||
// rebuild, another round of killed states, and a transition logged for something that
|
||||
// had already taken effect. Passing the set in lets each path record exactly what pf
|
||||
// accepted.
|
||||
func (p *prog) buildPFAnchorRulesWith(vpnExemptions []vpnDNSExemption, forwardedSources []forwardedSource) string {
|
||||
// Read the actual listener address from config. In intercept mode, ctrld may
|
||||
// be on a non-standard port (e.g., 127.0.0.1:5354) if mDNSResponder holds *:53.
|
||||
// The pf rdr rules must redirect to wherever ctrld is actually listening.
|
||||
@@ -969,6 +1034,9 @@ func (p *prog) buildPFAnchorRules(vpnExemptions []vpnDNSExemption) string {
|
||||
// DNS intercept rules must evaluate first so that DNS queries work (they're how
|
||||
// IPs get into the allowlist in the first place).
|
||||
if p.firewallModeEnabled() {
|
||||
// Make declared VM/container source subnets first-class firewall
|
||||
// clients (DNS forced through ctrld) before the blanket allowlist block.
|
||||
rules.WriteString(buildPFForwardedSourceRulesFor(forwardedSources, listenerIP))
|
||||
rules.WriteString(buildPFFirewallRules())
|
||||
}
|
||||
|
||||
@@ -1128,8 +1196,9 @@ func (p *prog) checkTunnelInterfaceChanges() bool {
|
||||
if p.vpnDNS != nil {
|
||||
vpnExemptions = p.vpnDNS.CurrentExemptions()
|
||||
}
|
||||
rulesStr := p.buildPFAnchorRules(vpnExemptions)
|
||||
if err := os.WriteFile(pfAnchorFile, []byte(rulesStr), 0644); err != nil {
|
||||
forwarded := p.currentForwardedSources()
|
||||
rulesStr := p.buildPFAnchorRulesWith(vpnExemptions, forwarded)
|
||||
if err := writePFAnchorFile(rulesStr); err != nil {
|
||||
mainLog.Load().Error().Err(err).Msg("DNS intercept: failed to write rebuilt anchor file")
|
||||
return true
|
||||
}
|
||||
@@ -1138,6 +1207,7 @@ func (p *prog) checkTunnelInterfaceChanges() bool {
|
||||
mainLog.Load().Error().Err(err).Msgf("DNS intercept: failed to reload rebuilt anchor (output: %s)", strings.TrimSpace(string(out)))
|
||||
return true
|
||||
}
|
||||
p.recordAppliedForwardedSources(forwarded)
|
||||
|
||||
flushPFStates()
|
||||
mainLog.Load().Info().Msgf("DNS intercept: rebuilt pf anchor with %d tunnel interfaces", len(current))
|
||||
@@ -1388,13 +1458,15 @@ func (p *prog) ensurePFAnchorActive() bool {
|
||||
if p.vpnDNS != nil {
|
||||
vpnExemptions = p.vpnDNS.CurrentExemptions()
|
||||
}
|
||||
rulesStr := p.buildPFAnchorRules(vpnExemptions)
|
||||
if err := os.WriteFile(pfAnchorFile, []byte(rulesStr), 0644); err != nil {
|
||||
forwarded := p.currentForwardedSources()
|
||||
rulesStr := p.buildPFAnchorRulesWith(vpnExemptions, forwarded)
|
||||
if err := writePFAnchorFile(rulesStr); err != nil {
|
||||
mainLog.Load().Error().Err(err).Msg("DNS intercept watchdog: failed to write anchor file")
|
||||
} else if out, err := exec.Command("pfctl", "-a", pfAnchorName, "-f", pfAnchorFile).CombinedOutput(); err != nil {
|
||||
p.pfBackoffResourceExhaustion(err, out, "load rebuilt anchor")
|
||||
mainLog.Load().Error().Err(err).Msgf("DNS intercept watchdog: failed to load rebuilt anchor (output: %s)", strings.TrimSpace(string(out)))
|
||||
} else {
|
||||
p.recordAppliedForwardedSources(forwarded)
|
||||
flushPFStates()
|
||||
mainLog.Load().Info().Msg("DNS intercept watchdog: rebuilt and loaded anchor rules")
|
||||
}
|
||||
@@ -1510,6 +1582,10 @@ func (p *prog) scheduleDelayedRechecks() {
|
||||
if p.vpnDNS != nil {
|
||||
p.vpnDNS.Refresh(ctx, true)
|
||||
}
|
||||
// A VM/container network often gets its address slightly after the
|
||||
// interface appears, so the immediate handler can see it without a subnet.
|
||||
// Re-check here (no-op when the forwarded-source set is unchanged).
|
||||
p.reconcileForwardedSources()
|
||||
})
|
||||
p.pfDelayedRecheckTimers = append(p.pfDelayedRecheckTimers, timer)
|
||||
}
|
||||
@@ -1567,6 +1643,13 @@ func (p *prog) pfWatchdog() {
|
||||
mainLog.Load().Info().Msgf("DNS intercept watchdog: pf anchor stable again after %d consecutive restores", old)
|
||||
}
|
||||
}
|
||||
|
||||
// VM/container networks appear and disappear without always
|
||||
// producing a network-change event, and an intact anchor passes every
|
||||
// check above. Reconciling here bounds how long a started guest can go
|
||||
// untrusted (or a stopped one stay trusted) to one watchdog interval.
|
||||
// No-op when the effective forwarded-source set is unchanged.
|
||||
p.reconcileForwardedSources()
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1581,9 +1664,10 @@ func (p *prog) exemptVPNDNSServers(exemptions []vpnDNSExemption) error {
|
||||
return fmt.Errorf("pf state not available")
|
||||
}
|
||||
|
||||
rulesStr := p.buildPFAnchorRules(exemptions)
|
||||
forwarded := p.currentForwardedSources()
|
||||
rulesStr := p.buildPFAnchorRulesWith(exemptions, forwarded)
|
||||
|
||||
if err := os.WriteFile(pfAnchorFile, []byte(rulesStr), 0644); err != nil {
|
||||
if err := writePFAnchorFile(rulesStr); err != nil {
|
||||
return fmt.Errorf("dns intercept: failed to rewrite pf anchor: %w", err)
|
||||
}
|
||||
|
||||
@@ -1591,6 +1675,7 @@ func (p *prog) exemptVPNDNSServers(exemptions []vpnDNSExemption) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("dns intercept: failed to reload pf anchor: %w (output: %s)", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
p.recordAppliedForwardedSources(forwarded)
|
||||
|
||||
// Flush stale pf states so packets are re-evaluated against new rules.
|
||||
flushPFStates()
|
||||
@@ -1832,11 +1917,14 @@ func (p *prog) forceReloadPFMainRuleset() {
|
||||
if p.vpnDNS != nil {
|
||||
vpnExemptions = p.vpnDNS.CurrentExemptions()
|
||||
}
|
||||
rulesStr := p.buildPFAnchorRules(vpnExemptions)
|
||||
if err := os.WriteFile(pfAnchorFile, []byte(rulesStr), 0644); err != nil {
|
||||
forwarded := p.currentForwardedSources()
|
||||
rulesStr := p.buildPFAnchorRulesWith(vpnExemptions, forwarded)
|
||||
if err := writePFAnchorFile(rulesStr); err != nil {
|
||||
mainLog.Load().Error().Err(err).Msg("DNS intercept: force reload — failed to write anchor file")
|
||||
} else if out, err := exec.Command("pfctl", "-a", pfAnchorName, "-f", pfAnchorFile).CombinedOutput(); err != nil {
|
||||
mainLog.Load().Error().Err(err).Msgf("DNS intercept: force reload — failed to load anchor (output: %s)", strings.TrimSpace(string(out)))
|
||||
} else {
|
||||
p.recordAppliedForwardedSources(forwarded)
|
||||
}
|
||||
|
||||
// Flush stale rdr/reply states after the forced ruleset + anchor reload.
|
||||
|
||||
Reference in New Issue
Block a user