Files
ctrld/cmd/cli/firewall_darwin_bridge_test.go
T
Cuong Manh Le 171dd0a7e6 fix(firewall): support VM/container egress under macOS Firewall Mode
macOS Firewall Mode blocked forwarded/NATed VM/container egress: a guest
resolves DNS through a path host ctrld does not observe, so the guest-resolved
public IP never enters <ctrld_allowed> and the blanket outbound block drops the
guest's TCP/443.

Make VM/container guests first-class Firewall Mode clients by forcing their DNS
through ctrld. The trusted source subnets are the UNION of:

  - Auto-detected VM/NAT networks (default, no config): interfaces that are up,
    carry an RFC1918 IPv4 address, and whose VM ownership can be proven either
    by a vendor-specific name (vnic/vboxnet/vmnet) or by being a bridge* whose
    member list contains a vendor VM interface (typically vmenet*). Each keeps
    its ingress interface, and its pf rules are scoped "on <iface>" so an
    unrelated interface on the same private range is never affected.
  - service.firewall_forwarded_sources (opt-in): explicit IPv4 CIDRs, matched on
    the source CIDR alone, for stacks whose ownership cannot be proven. Config
    adds only; an invalid or non-IPv4 entry is dropped with a warning.

Checking bridge membership is what makes the common case work without config.
Every vmnet.framework stack - UTM and other Virtualization.framework guests,
Docker Desktop, Multipass, Fusion 12.1+ NAT - puts the RFC1918 gateway address
on a bridge10x interface and attaches the vendor-named vmenet* interface as an
address-less member, so matching on interface name alone never sees them.
Membership is the ownership proof a bridge name lacks: macOS shares that
namespace with Thunderbolt/aggregated links (ctrld's own tunnel-change code
treats bridge0 as physical), and such a bridge has en* members, so it stays
untrusted however private its address. Members are read with ifconfig, and only
for a bridge that already carries an RFC1918 IPv4 address, so a host with no VM
running executes no subprocess.

Per source subnet, plaintext DNS (53) is force-routed through ctrld (route-to
lo0 -> existing rdr-on-lo0) so guest resolutions are policy-enforced and
populate the allowlist; guest egress to allowed IPs is then permitted by the
existing <ctrld_allowed> rule. DoT (853) is blocked so guests cannot swap in an
alternate resolver. DoH/443 is a documented limitation.

Rules are emitted strictly per address family. Sources are IPv4 (interception
targets ctrld's IPv4 listener), so only inet rules are generated: pf rejects an
entire anchor over a single "inet6 ... from 192.168.x.0/24" mismatch, which
would take DNS interception down with it. Guest IPv6 DoT is covered by the
blanket IPv6 block instead, since such a resolver never enters <ctrld_allowed>.

firewall_forwarded_sources is deliberately not validated with `cidr`. Entries
are checked where they are parsed and a bad one is dropped while the rest of the
set still applies. A hard validator would make one typo in an MDM-pushed subnet
fatal at startup - validateConfig exits the process - taking DNS service down
for the whole host over a line that only ever widened a firewall allowance. The
warning fires when the set of unusable entries changes rather than on every
parse, since config is re-read on every anchor build and every watchdog tick.

Reconcile the trust set at runtime, since VM interfaces appear and disappear
while ctrld runs and no existing path rebuilds an intact anchor for that
(ensurePFAnchorActive returns early, checkTunnelInterfaceChanges tracks only
tunnels, pfInterceptMonitor rebuilds only after a failed host probe). On a
change, rebuild the anchor and drop the pf states of the affected subnets
(targeted pfctl -k, not a global state flush), because rules govern only new
states: a stopped guest would otherwise keep using states created while it was
trusted, and a newly trusted one would keep bypassing interception until its
states expired. Reconciliation runs on interface appear/disappear, on network
changes, on the delayed post-change re-checks (a VM network often gets its
address after its interface appears), and on the pf watchdog tick, which bounds
guest start/stop convergence to one interval even with no network event.

Convergence is not latched on failure: the applied set advances, and states are
killed, only after pf has accepted the new anchor. reloadForwardedSourceAnchor
reports write/pfctl failures to the caller, which then keeps the previous set
recorded and logs a warning, so the next reconcile retries the same transition
instead of going quiet with the old anchor still installed. The whole
compare-reload-record sequence is serialized so a watchdog tick and a network
change cannot both rebuild or interleave snapshots.

Every anchor rebuild records the forwarded-source set it installed, and takes
that set as a parameter rather than re-detecting internally. Otherwise a rebuild
triggered by something else (tunnel change, watchdog restore, VPN DNS
exemptions, forced reload, startup) leaves the snapshot at the older set and the
next reconcile "discovers" the same change again: another rebuild, another round
of killed guest states, and a transition logged for something already in effect.
Passing the set in also means what pf loaded is exactly what gets recorded, so
an interface appearing mid-reconcile cannot leave the snapshot describing a set
that was never installed.

The anchor file is replaced atomically (temp file plus rename). Seven paths
rebuild it from timers and network-change callbacks in their own goroutines with
no lock between them, and os.WriteFile truncates before writing, so a pfctl -f
racing that window could read a partial ruleset and reject the anchor - taking
DNS interception down until the next watchdog restore.

Report the effective trust set at startup and on every change, naming each
subnet's origin ("192.168.64.0/24 (configured)" vs "(auto-detected on
bridge100)"), and say so explicitly when the set is empty, including what
auto-detection requires. Configured entries previously produced no log output at
all, so an admin who set firewall_forwarded_sources could not confirm it had
taken effect without reading pf rules. Per-source detection logging is at debug,
since detection re-runs on every anchor build.

Explicit, per-subnet trust boundary (RFC1918-only auto-detect, proven VM
ownership, interface-scoped rules, config adds only), not an interface-wide
permit: direct public IPs the guest never resolved through ctrld stay blocked.

Tests cover rule generation and its interface scoping, the no-blanket-permit
boundary, single-family emission (no inet6 rule for an IPv4 source, IPv6 sources
skipped), a real pfctl -n -f parse of both the forwarded-source rules alone and
the full anchor (group-scoped rules stripped, since _ctrld exists only where the
service is installed), invalid/duplicate/non-IPv4 config entries, that a bad
entry does not fail config validation, the union set and its signature, a
deterministic guest start/stop lifecycle asserting both the rebuild points and
which subnets' states must be dropped, anchor-reload failure followed by a
successful retry, that reconcile is inert outside firewall mode, ifconfig member
parsing against real bridge output, and the trust decision per interface -
including everything that must NOT qualify: a Thunderbolt bridge, a
public-range VM bridge, an IPv6-only bridge, an address-less vendor interface, a
physical uplink and a VPN tunnel. docs/firewall-mode.md documents the boundary,
what auto-detection can and cannot prove, the address-family constraint, the
lifecycle/retry behavior, how to confirm the trust set from the log, and that a
resolver running inside the guest is not supported (its encrypted upstream
leaves the host nothing to allowlist).
2026-08-04 14:12:15 +07:00

417 lines
15 KiB
Go

//go:build darwin
package cli
import (
"net"
"net/netip"
"testing"
"github.com/Control-D-Inc/ctrld/internal/firewall"
)
// vmnetBridgeIfconfig is ifconfig output for a vmnet.framework bridge: the RFC1918
// gateway lives here, and the vendor-named vmenet0 is an address-less member. This is
// the shape that name-only detection could never see.
const vmnetBridgeIfconfig = `bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
ether 5e:cf:7f:9a:1b:64
inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x0
member: vmenet0 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 22 priority 0 path cost 0
nd6 options=201<PERFORMNUD,DAD>
media: <unknown type>
status: active
`
// thunderboltBridgeIfconfig is ifconfig output for the Thunderbolt bridge macOS
// creates by default. It can carry an RFC1918 address, and its members are physical
// interfaces - trusting it would force-route unrelated same-subnet traffic.
const thunderboltBridgeIfconfig = `bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether 36:12:8a:1f:2b:00
inet 192.168.10.5 netmask 0xffffff00 broadcast 192.168.10.255
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
member: en1 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 9 priority 0 path cost 0
member: en2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 10 priority 0 path cost 0
nd6 options=201<PERFORMNUD,DAD>
media: <unknown type>
status: inactive
`
func TestParseBridgeMembers(t *testing.T) {
tests := []struct {
name string
out string
want []string
}{
{"vmnet.framework bridge", vmnetBridgeIfconfig, []string{"vmenet0"}},
{"thunderbolt bridge", thunderboltBridgeIfconfig, []string{"en1", "en2"}},
{"no members", "bridge2: flags=8822<BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500\n\tether 1a:2b:3c\n", nil},
{"empty output", "", nil},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got := parseBridgeMembers(tc.out)
if len(got) != len(tc.want) {
t.Fatalf("parseBridgeMembers() = %v, want %v", got, tc.want)
}
for i := range got {
if got[i] != tc.want[i] {
t.Errorf("member[%d] = %q, want %q", i, got[i], tc.want[i])
}
}
})
}
}
func TestBridgeHasVMMember(t *testing.T) {
tests := []struct {
name string
members []string
want bool
}{
{"vmnet.framework member", []string{"vmenet0"}, true},
{"parallels member", []string{"vnic0"}, true},
{"mixed with vm member", []string{"en1", "vmenet2"}, true},
{"physical members only", []string{"en1", "en2"}, false},
{"no members", nil, false},
// A name that merely looks bridge-ish proves nothing.
{"bridge member", []string{"bridge1"}, false},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := bridgeHasVMMember(tc.members); got != tc.want {
t.Errorf("bridgeHasVMMember(%v) = %v, want %v", tc.members, got, tc.want)
}
})
}
}
func mustAddrs(t *testing.T, cidrs ...string) []net.Addr {
t.Helper()
var out []net.Addr
for _, c := range cidrs {
ip, ipnet, err := net.ParseCIDR(c)
if err != nil {
t.Fatalf("bad test CIDR %q: %v", c, err)
}
out = append(out, &net.IPNet{IP: ip, Mask: ipnet.Mask})
}
return out
}
// TestForwardedSourcesForIface covers the trust decision, including every case that
// must NOT be auto-trusted. This is a security boundary: anything that qualifies here
// gets its guest traffic passed to allowed public destinations.
func TestForwardedSourcesForIface(t *testing.T) {
tests := []struct {
name string
iface string
addrs []string
members []string
wantCIDR []string
}{
{
// The case name-only detection missed: address on the bridge, vendor
// interface bridged into it. This is UTM/Docker/Multipass/Fusion 12.1+.
name: "bridge with vmenet member is trusted",
iface: "bridge100",
addrs: []string{"192.168.64.1/24"},
members: []string{"vmenet0"},
wantCIDR: []string{"192.168.64.0/24"},
},
{
// The reason membership is required rather than the bridge name.
name: "thunderbolt bridge is not trusted",
iface: "bridge0",
addrs: []string{"192.168.10.5/24"},
members: []string{"en1", "en2"},
},
{
name: "vendor-named interface with its own address is trusted",
iface: "vnic0",
addrs: []string{"10.211.55.2/24"},
wantCIDR: []string{"10.211.55.0/24"},
},
{
// vmenet* under vmnet.framework: up, but no address of its own.
name: "address-less vendor interface yields nothing",
iface: "vmenet0",
addrs: nil,
},
{
// The RFC1918 boundary: a VM network on a public range is never trusted.
name: "public range on a VM bridge is not trusted",
iface: "bridge100",
addrs: []string{"93.184.216.34/24"},
members: []string{"vmenet0"},
},
{
// Interception is IPv4-only; an IPv6-only VM bridge must not qualify.
name: "ipv6 only is not trusted",
iface: "bridge100",
addrs: []string{"fd00::1/64"},
members: []string{"vmenet0"},
},
{
name: "physical uplink is not trusted",
iface: "en0",
addrs: []string{"192.168.1.20/24"},
},
{
name: "vpn tunnel is not trusted",
iface: "utun4",
addrs: []string{"10.2.0.2/24"},
},
{
name: "mixed addresses keep only the private ipv4 one",
iface: "bridge101",
addrs: []string{"fd00::1/64", "192.168.105.1/24"},
members: []string{"vmenet1"},
wantCIDR: []string{"192.168.105.0/24"},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
members := func(string) []string { return tc.members }
got, reason := forwardedSourcesForIface(tc.iface, mustAddrs(t, tc.addrs...), members)
if len(got) != len(tc.wantCIDR) {
t.Fatalf("got %d sources %v, want %d %v", len(got), got, len(tc.wantCIDR), tc.wantCIDR)
}
for i, want := range tc.wantCIDR {
if got[i].prefix != netip.MustParsePrefix(want) {
t.Errorf("prefix[%d] = %s, want %s", i, got[i].prefix, want)
}
// Rules must be scoped to the interface traffic actually arrives on.
if got[i].iface != tc.iface {
t.Errorf("source %s scoped to %q, want %q", got[i].prefix, got[i].iface, tc.iface)
}
}
if len(got) > 0 && reason == "" {
t.Error("a trusted source must report why it qualified")
}
})
}
}
// TestForwardedSourcesForIface_NoMemberLookupWithoutAddress verifies the ifconfig call
// is skipped for a bridge that cannot qualify anyway. Detection runs on every anchor
// build and every watchdog tick, so this keeps a typical host at zero subprocesses.
func TestForwardedSourcesForIface_NoMemberLookupWithoutAddress(t *testing.T) {
called := false
members := func(string) []string {
called = true
return []string{"vmenet0"}
}
if got, _ := forwardedSourcesForIface("bridge100", nil, members); got != nil {
t.Errorf("address-less bridge must yield nothing, got %v", got)
}
if called {
t.Error("member list must not be queried for a bridge with no RFC1918 address")
}
// A public-range bridge is equally hopeless, and equally must not exec.
called = false
if got, _ := forwardedSourcesForIface("bridge100", mustAddrs(t, "93.184.216.34/24"), members); got != nil {
t.Errorf("public-range bridge must yield nothing, got %v", got)
}
if called {
t.Error("member list must not be queried for a bridge with no RFC1918 address")
}
}
// TestParseForwardedSourceConfig covers the parse/reject split: usable entries survive
// alongside bad ones, and each rejection carries a reason to report.
func TestParseForwardedSourceConfig(t *testing.T) {
sources, rejected := parseForwardedSourceConfig([]string{
"192.168.64.7/24", // host bits get normalized
"not-a-cidr", // malformed
"fd00::/64", // not IPv4
" 10.0.0.0/8 ", // surrounding space tolerated
})
wantSources := map[string]bool{"192.168.64.0/24": true, "10.0.0.0/8": true}
if len(sources) != len(wantSources) {
t.Fatalf("got %d usable sources %v, want %d", len(sources), sources, len(wantSources))
}
for _, src := range sources {
if !wantSources[src.prefix.String()] {
t.Errorf("unexpected usable prefix %s", src.prefix)
}
if src.iface != "" {
t.Errorf("configured source %s must have no interface scope, got %q", src.prefix, src.iface)
}
}
if len(rejected) != 2 {
t.Fatalf("got %d rejections %v, want 2", len(rejected), rejected)
}
for _, r := range rejected {
if r.value != "not-a-cidr" && r.value != "fd00::/64" {
t.Errorf("unexpected rejected value %q", r.value)
}
if r.reason == "" {
t.Errorf("rejection of %q carries no reason", r.value)
}
}
}
func TestParseForwardedSourceConfig_Empty(t *testing.T) {
sources, rejected := parseForwardedSourceConfig(nil)
if sources != nil || rejected != nil {
t.Errorf("empty config must yield nothing, got %v / %v", sources, rejected)
}
}
// TestRejectedForwardedSourcesKey verifies the signature ignores order, so re-parsing
// an unchanged config is recognised as nothing new, while a changed set is not.
func TestRejectedForwardedSourcesKey(t *testing.T) {
a := []rejectedForwardedSource{{value: "x", reason: "r1"}, {value: "y", reason: "r2"}}
b := []rejectedForwardedSource{{value: "y", reason: "r2"}, {value: "x", reason: "r1"}}
if rejectedForwardedSourcesKey(a) != rejectedForwardedSourcesKey(b) {
t.Error("key must be order-independent")
}
if rejectedForwardedSourcesKey(nil) != "" {
t.Error("no rejections must produce an empty key")
}
c := []rejectedForwardedSource{{value: "x", reason: "r1"}}
if rejectedForwardedSourcesKey(a) == rejectedForwardedSourcesKey(c) {
t.Error("different rejection sets must produce different keys")
}
}
// resetForwardedSourceWarnTracker clears the process-wide warning dedupe so each test
// starts from "nothing reported yet".
func resetForwardedSourceWarnTracker(t *testing.T) {
t.Helper()
forwardedSourceWarnTracker.mu.Lock()
forwardedSourceWarnTracker.key = ""
forwardedSourceWarnTracker.mu.Unlock()
}
func trackedRejectionKey() string {
forwardedSourceWarnTracker.mu.Lock()
defer forwardedSourceWarnTracker.mu.Unlock()
return forwardedSourceWarnTracker.key
}
// TestWarnRejectedForwardedSources_OnlyOnChange verifies a standing bad entry is
// reported once rather than on every watchdog tick, and that a newly-introduced one is
// still reported after a config reload.
//
// The dedupe must not depend on pfFirewallState: no state is installed here, matching
// the window where Firewall Mode is on but pf enforcement is still deferred until
// intercept mode starts.
func TestWarnRejectedForwardedSources_OnlyOnChange(t *testing.T) {
resetForwardedSourceWarnTracker(t)
p := progWithForwardedSources("not-a-cidr", "192.168.64.0/24")
// First parse reports; the signature is now recorded.
p.firewallForwardedSources()
first := trackedRejectionKey()
if first == "" {
t.Fatal("a rejected entry must be recorded as reported")
}
// Re-parsing the same config (every anchor build, every 30s tick) must not change
// what is recorded - that is what stops the repeated warning.
for i := 0; i < 5; i++ {
p.firewallForwardedSources()
}
if got := trackedRejectionKey(); got != first {
t.Errorf("recorded rejection key changed on re-parse: %q -> %q", first, got)
}
// A config reload that introduces a different bad entry must be reported.
p.cfg.Service.FirewallForwardedSources = []string{"also-not-a-cidr"}
p.firewallForwardedSources()
if trackedRejectionKey() == first {
t.Error("a newly-introduced bad entry must be reported, not suppressed")
}
// Fixing the config clears the recorded set, so a later regression reports again.
p.cfg.Service.FirewallForwardedSources = []string{"192.168.64.0/24"}
p.firewallForwardedSources()
if got := trackedRejectionKey(); got != "" {
t.Errorf("a clean config must clear the recorded rejections, got %q", got)
}
}
// TestCurrentForwardedSources_GatedOnFirewallMode verifies no detection or config
// parsing happens with Firewall Mode off. Anchor rebuilds run on tunnel changes,
// watchdog restores and VPN DNS updates regardless of firewall mode, so an ungated
// call would enumerate interfaces, exec ifconfig and re-report bad config entries on
// every one of them.
func TestCurrentForwardedSources_GatedOnFirewallMode(t *testing.T) {
resetForwardedSourceWarnTracker(t)
p := progWithForwardedSources("not-a-cidr", "192.168.64.0/24")
// Firewall mode off: nothing detected, and the bad entry is not even looked at.
if got := p.currentForwardedSources(); got != nil {
t.Errorf("firewall mode off must yield no sources, got %v", got)
}
if got := trackedRejectionKey(); got != "" {
t.Errorf("config must not be parsed with firewall mode off, but a rejection was recorded: %q", got)
}
// With firewall mode on (allowList present), the configured entry is honoured and
// the bad one reported.
p.allowList = firewall.New()
got := p.currentForwardedSources()
if len(got) != 1 || got[0].prefix != netip.MustParsePrefix("192.168.64.0/24") {
t.Errorf("firewall mode on must yield the configured source, got %v", got)
}
if trackedRejectionKey() == "" {
t.Error("the unusable entry must be reported once firewall mode is on")
}
}
// TestRecordAppliedForwardedSources verifies a full-anchor rebuild can baseline the
// reconcile snapshot, so the next reconcile does not redo the same change.
func TestRecordAppliedForwardedSources(t *testing.T) {
p := progWithForwardedSources()
state := &pfFirewallState{}
p.platformFirewallState = state
sources := []forwardedSource{{prefix: netip.MustParsePrefix("192.168.64.0/24"), iface: "bridge100"}}
p.recordAppliedForwardedSources(sources)
if state.lastForwardedKey != forwardedSourceSetKey(sources) {
t.Errorf("snapshot key = %q, want %q", state.lastForwardedKey, forwardedSourceSetKey(sources))
}
if len(state.lastForwardedSources) != 1 || state.lastForwardedSources[0] != sources[0] {
t.Errorf("snapshot sources = %v, want %v", state.lastForwardedSources, sources)
}
// A reconcile against the same set must now find nothing to do: no second rebuild,
// no killed states, no transition logged for something already in effect.
reloads := 0
_, _, changed, err := state.applyForwardedSourceChange(sources, func() error { reloads++; return nil })
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if changed {
t.Error("reconcile treated an already-applied set as a change")
}
if reloads != 0 {
t.Errorf("anchor was rebuilt %d times for an unchanged set, want 0", reloads)
}
}
// TestRecordAppliedForwardedSources_NoState verifies recording is a no-op when
// firewall mode is off, since the rebuild paths call it unconditionally.
func TestRecordAppliedForwardedSources_NoState(t *testing.T) {
p := progWithForwardedSources()
p.recordAppliedForwardedSources([]forwardedSource{{prefix: netip.MustParsePrefix("10.0.0.0/8")}})
}