CalvinBackup/ctrld
1
0
Fork 0
mirror of https://github.com/Control-D-Inc/ctrld.git synced 2026-09-04 13:36:35 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(firewall): support VM/container egress under macOS Firewall Mode

Browse Source 
macOS Firewall Mode blocked forwarded/NATed VM/container egress: a guest
resolves DNS through a path host ctrld does not observe, so the guest-resolved
public IP never enters <ctrld_allowed> and the blanket outbound block drops the
guest's TCP/443.

Make VM/container guests first-class Firewall Mode clients by forcing their DNS
through ctrld. The trusted source subnets are the UNION of:

  - Auto-detected VM/NAT networks (default, no config): interfaces that are up,
    carry an RFC1918 IPv4 address, and whose VM ownership can be proven either
    by a vendor-specific name (vnic/vboxnet/vmnet) or by being a bridge* whose
    member list contains a vendor VM interface (typically vmenet*). Each keeps
    its ingress interface, and its pf rules are scoped "on <iface>" so an
    unrelated interface on the same private range is never affected.
  - service.firewall_forwarded_sources (opt-in): explicit IPv4 CIDRs, matched on
    the source CIDR alone, for stacks whose ownership cannot be proven. Config
    adds only; an invalid or non-IPv4 entry is dropped with a warning.

Checking bridge membership is what makes the common case work without config.
Every vmnet.framework stack - UTM and other Virtualization.framework guests,
Docker Desktop, Multipass, Fusion 12.1+ NAT - puts the RFC1918 gateway address
on a bridge10x interface and attaches the vendor-named vmenet* interface as an
address-less member, so matching on interface name alone never sees them.
Membership is the ownership proof a bridge name lacks: macOS shares that
namespace with Thunderbolt/aggregated links (ctrld's own tunnel-change code
treats bridge0 as physical), and such a bridge has en* members, so it stays
untrusted however private its address. Members are read with ifconfig, and only
for a bridge that already carries an RFC1918 IPv4 address, so a host with no VM
running executes no subprocess.

Per source subnet, plaintext DNS (53) is force-routed through ctrld (route-to
lo0 -> existing rdr-on-lo0) so guest resolutions are policy-enforced and
populate the allowlist; guest egress to allowed IPs is then permitted by the
existing <ctrld_allowed> rule. DoT (853) is blocked so guests cannot swap in an
alternate resolver. DoH/443 is a documented limitation.

Rules are emitted strictly per address family. Sources are IPv4 (interception
targets ctrld's IPv4 listener), so only inet rules are generated: pf rejects an
entire anchor over a single "inet6 ... from 192.168.x.0/24" mismatch, which
would take DNS interception down with it. Guest IPv6 DoT is covered by the
blanket IPv6 block instead, since such a resolver never enters <ctrld_allowed>.

firewall_forwarded_sources is deliberately not validated with `cidr`. Entries
are checked where they are parsed and a bad one is dropped while the rest of the
set still applies. A hard validator would make one typo in an MDM-pushed subnet
fatal at startup - validateConfig exits the process - taking DNS service down
for the whole host over a line that only ever widened a firewall allowance. The
warning fires when the set of unusable entries changes rather than on every
parse, since config is re-read on every anchor build and every watchdog tick.

Reconcile the trust set at runtime, since VM interfaces appear and disappear
while ctrld runs and no existing path rebuilds an intact anchor for that
(ensurePFAnchorActive returns early, checkTunnelInterfaceChanges tracks only
tunnels, pfInterceptMonitor rebuilds only after a failed host probe). On a
change, rebuild the anchor and drop the pf states of the affected subnets
(targeted pfctl -k, not a global state flush), because rules govern only new
states: a stopped guest would otherwise keep using states created while it was
trusted, and a newly trusted one would keep bypassing interception until its
states expired. Reconciliation runs on interface appear/disappear, on network
changes, on the delayed post-change re-checks (a VM network often gets its
address after its interface appears), and on the pf watchdog tick, which bounds
guest start/stop convergence to one interval even with no network event.

Convergence is not latched on failure: the applied set advances, and states are
killed, only after pf has accepted the new anchor. reloadForwardedSourceAnchor
reports write/pfctl failures to the caller, which then keeps the previous set
recorded and logs a warning, so the next reconcile retries the same transition
instead of going quiet with the old anchor still installed. The whole
compare-reload-record sequence is serialized so a watchdog tick and a network
change cannot both rebuild or interleave snapshots.

Every anchor rebuild records the forwarded-source set it installed, and takes
that set as a parameter rather than re-detecting internally. Otherwise a rebuild
triggered by something else (tunnel change, watchdog restore, VPN DNS
exemptions, forced reload, startup) leaves the snapshot at the older set and the
next reconcile "discovers" the same change again: another rebuild, another round
of killed guest states, and a transition logged for something already in effect.
Passing the set in also means what pf loaded is exactly what gets recorded, so
an interface appearing mid-reconcile cannot leave the snapshot describing a set
that was never installed.

The anchor file is replaced atomically (temp file plus rename). Seven paths
rebuild it from timers and network-change callbacks in their own goroutines with
no lock between them, and os.WriteFile truncates before writing, so a pfctl -f
racing that window could read a partial ruleset and reject the anchor - taking
DNS interception down until the next watchdog restore.

Report the effective trust set at startup and on every change, naming each
subnet's origin ("192.168.64.0/24 (configured)" vs "(auto-detected on
bridge100)"), and say so explicitly when the set is empty, including what
auto-detection requires. Configured entries previously produced no log output at
all, so an admin who set firewall_forwarded_sources could not confirm it had
taken effect without reading pf rules. Per-source detection logging is at debug,
since detection re-runs on every anchor build.

Explicit, per-subnet trust boundary (RFC1918-only auto-detect, proven VM
ownership, interface-scoped rules, config adds only), not an interface-wide
permit: direct public IPs the guest never resolved through ctrld stay blocked.

Tests cover rule generation and its interface scoping, the no-blanket-permit
boundary, single-family emission (no inet6 rule for an IPv4 source, IPv6 sources
skipped), a real pfctl -n -f parse of both the forwarded-source rules alone and
the full anchor (group-scoped rules stripped, since _ctrld exists only where the
service is installed), invalid/duplicate/non-IPv4 config entries, that a bad
entry does not fail config validation, the union set and its signature, a
deterministic guest start/stop lifecycle asserting both the rebuild points and
which subnets' states must be dropped, anchor-reload failure followed by a
successful retry, that reconcile is inert outside firewall mode, ifconfig member
parsing against real bridge output, and the trust decision per interface -
including everything that must NOT qualify: a Thunderbolt bridge, a
public-range VM bridge, an IPv6-only bridge, an address-less vendor interface, a
physical uplink and a VPN tunnel. docs/firewall-mode.md documents the boundary,
what auto-detection can and cannot prove, the address-family constraint, the
lifecycle/retry behavior, how to confirm the trust set from the log, and that a
resolver running inside the guest is not supported (its encrypted upstream
leaves the host nothing to allowlist).
This commit is contained in:
Cuong Manh Le
2026-08-04 14:12:15 +07:00
parent cbef8b911c
commit 171dd0a7e6
10 changed files with 2027 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/firewall-mode.md
+167 -11
View File
@@ -1,7 +1,7 @@
# Firewall Mode
Firewall mode makes DNS policy unbypassable by blocking outbound connections to any
IP that wasn't resolved by ctrld. This closes the "DNS gap" — where apps use hardcoded
IP that wasn't resolved by ctrld. This closes the "DNS gap" - where apps use hardcoded
IPs, direct-IP fallbacks, or alternative DNS resolvers to bypass DNS-based filtering.
## How It Works
@@ -73,14 +73,14 @@ These IPs are always allowed regardless of DNS resolution:
| Range | Reason |
|-------|--------|
| `127.0.0.0/8`, `::1` | Loopback — local services |
| `10.0.0.0/8` | RFC1918 — LAN, printers, NAS |
| `172.16.0.0/12` | RFC1918 — LAN |
| `192.168.0.0/16` | RFC1918 — LAN |
| `169.254.0.0/16`, `fe80::/10` | Link-local — DHCP, mDNS |
| `100.64.0.0/10` | CGNAT — Tailscale, carrier NAT |
| `224.0.0.0/4`, `ff00::/8` | Multicast — mDNS, SSDP |
| ctrld listener IPs | Self — DNS proxy must be reachable |
| `127.0.0.0/8`, `::1` | Loopback - local services |
| `10.0.0.0/8` | RFC1918 - LAN, printers, NAS |
| `172.16.0.0/12` | RFC1918 - LAN |
| `192.168.0.0/16` | RFC1918 - LAN |
| `169.254.0.0/16`, `fe80::/10` | Link-local - DHCP, mDNS |
| `100.64.0.0/10` | CGNAT - Tailscale, carrier NAT |
| `224.0.0.0/4`, `ff00::/8` | Multicast - mDNS, SSDP |
| ctrld listener IPs | Self - DNS proxy must be reachable |
| Upstream resolver IPs | DoH/DoT/DoQ endpoints |
## Live Profile Updates
@@ -92,7 +92,7 @@ When a ControlD profile changes (domain goes from allowed → blocked or vice ve
3. Subsequent DNS queries repopulate the allowlist under the new policy
4. Brief connectivity interruption (~seconds) while DNS cache repopulates
This is the "flush and repopulate" strategy — simple and correct, with a small
This is the "flush and repopulate" strategy - simple and correct, with a small
tradeoff of a brief connectivity blip on config changes.
## Network State Changes
@@ -134,6 +134,162 @@ remain blocked until the app performs DNS resolution again. This is an accepted
v1 tradeoff and should be called out in release notes and compatibility testing
for common apps.
## VM / Container Workloads (macOS)
By default a VM or container resolves DNS through a path the host ctrld does not
observe (the hypervisor's own resolver on the guest bridge, or a resolver the
guest is configured to use). The guest-resolved public IP therefore never enters
`<ctrld_allowed>`, and the guest's forwarded/NATed egress to that IP is dropped by
the blanket block - DNS "works" inside the guest but TCP/443 fails. (Tracked as
issue #569.)
Exempting the whole bridge interface would turn the guest into a policy bypass,
so it is intentionally **not** done. Instead ctrld makes those guests first-class
Firewall Mode clients by forcing their DNS through itself. The trusted source
subnets are the **union** of:
1. **Auto-detected VM networks (default, no config).** At pf-anchor build time an
interface is trusted only when it is up, carries an **RFC1918 IPv4** network, and
its VM ownership can be proven one of two ways:
- **its own name is vendor-specific** - `vnic` (Parallels), `vboxnet`
(VirtualBox host-only), `vmnet` (legacy kext-based VMware Fusion on Intel);
- **it is a `bridge*` whose member list contains a vendor VM interface**
(typically `vmenet*`). This is the case for every `vmnet.framework` stack -
UTM and other Virtualization.framework guests, Docker Desktop, Multipass, and
Fusion 12.1+ NAT - where the RFC1918 gateway address sits on `bridge10x` and
the vendor-named `vmenet*` interface is an address-less member of it. Matching
on interface name alone never sees those stacks.
Physical uplinks (`en*`), loopback, VPN tunnels (`utun*`), public ranges, and
IPv6 never qualify. Each auto-trusted subnet is logged at debug level
(`Firewall: auto-detected VM/container network for forwarded DNS`, with the
`reason` field naming the proof), and its pf rules are scoped to the interface it
was detected on (`on <iface>`), so an unrelated interface carrying the same
private range is never affected.
A `bridge*` **name** is still not proof of anything: macOS uses that namespace
for Thunderbolt and aggregated links too (ctrld's own tunnel-change code treats
`bridge0` as physical). Membership is what distinguishes them - a Thunderbolt
bridge has `en*` members and is never trusted, however private its address.
2. **Configured subnets (opt-in).** Needed for any stack whose ownership
auto-detection cannot prove - a VM network on a plain interface with no vendor
name, a bridge with no vendor member, or a deliberately non-RFC1918 range:
```toml
[service]
firewall_mode = "on"
intercept_mode = "hard"
# Only needed when auto-detection cannot prove the network is a VM network.
firewall_forwarded_sources = ["192.168.64.0/24"]
```
Find the subnet with `ifconfig` on the host - for a `vmnet.framework` stack it is
the `bridge1xx` interface serving the VM. Use the network address in CIDR form;
host bits are normalized away. A configured entry matches on the source CIDR
alone - it is an admin opt-in, so it is not tied to one interface. Entries must be
**IPv4**; interception targets ctrld's IPv4 listener, so an IPv6 entry is ignored
with a warning. Config only **adds** to auto-detection; it never disables it.
A malformed or non-IPv4 entry is dropped with a warning and the rest of the set
still applies - this field is deliberately **not** hard-validated at startup, so a
typo in an MDM-pushed subnet cannot stop ctrld from serving DNS. The warning is
logged when the set of bad entries changes, not on every internal rebuild, so a
standing typo does not fill the log.
### Guest start/stop and network changes
VM/container interfaces come and go while ctrld runs, and the pf watchdog does not
rebuild an anchor whose rules are still intact. ctrld therefore tracks the effective
forwarded-source set (auto-detected ∪ configured) and, whenever it changes,
rebuilds the anchor and drops the pf states of the affected subnets (targeted
`pfctl -k <subnet>`, not a global state flush) so the new policy applies
immediately instead of when old states expire. A guest's in-flight connections are
re-established under the new rules.
Reconciliation runs on interface appear/disappear, on network changes, on the
delayed post-change re-checks (a new VM network often gets its address slightly
after its interface appears), and on the pf watchdog tick - which bounds how long a
started guest can go untrusted, or a stopped guest stay trusted, to one watchdog
interval even if no network event fires. Each transition is logged with the subnets
that gained and lost trust.
The set ctrld considers applied only advances once pf has actually accepted the new
anchor. If the write or `pfctl -f` fails, the previous set stays recorded, nothing is
flushed, a warning is logged, and the next reconciliation (at the latest the next
watchdog tick) retries the same transition - so a transient failure cannot leave the
old anchor installed while ctrld believes the change is done.
### Supported behavior and trust boundary
For each source subnet (auto-detected or configured), when Firewall Mode +
intercept are active, ctrld:
- **Forces guest plaintext DNS (port 53) through ctrld** (pf `route-to lo0` onto
the existing loopback redirect). Every guest resolution is policy-enforced and
populates `<ctrld_allowed>`, so the guest's egress to allowed destinations is
then permitted by the same allowlist rule as the host.
- **Blocks guest IPv4 DoT (port 853)** so a guest cannot swap in an alternate
encrypted resolver to escape policy. Rules are emitted for the source's own
address family only (all sources are IPv4) - pf refuses to load an entire anchor
containing an `inet6` rule with an IPv4 source, which would take DNS interception
down with it. Guest traffic to an IPv6 DoT resolver is instead covered by the
blanket IPv6 outbound block, since such a resolver never enters `<ctrld_allowed>`.
The boundary is explicit and per-subnet - a guest still **cannot** bypass Control
D policy via a direct public IP (never resolved through ctrld ⇒ never allowlisted)
or an alternate plaintext/DoT resolver. It is not an interface-wide permit.
### Confirming what is trusted
At startup (and on every change) ctrld logs the effective set, naming each subnet's
origin, so `firewall_forwarded_sources` can be verified without reading pf rules:
```
Firewall: forwarded-workload (VM/container) DNS interception active for these source subnets count=2 sources=["192.168.64.0/24 (auto-detected on bridge100)","192.168.252.0/24 (configured)"]
```
The interface named is where the address lives, which for a `vmnet.framework` stack
is the bridge (`bridge100`), not its `vmenet*` member.
When the set is empty the log says so explicitly, rather than staying silent:
```
Firewall: no forwarded-workload (VM/container) sources — guest DNS is not intercepted. Auto-detection needs an up interface with an RFC1918 IPv4 address that is either vendor-named (vnic*, vboxnet*, vmnet*) or a bridge with a VM member (vmenet*); anything else must be listed in service.firewall_forwarded_sources
```
Note that `firewall_forwarded_sources` is a **local** config setting. If it is not in
`/etc/controld/ctrld.toml` on the device, ctrld has nothing to act on - check the file
itself, not only the dashboard.
### Limitations
- **A resolver running inside the guest is not supported** while Firewall Mode is on.
The design depends on the guest sending plaintext DNS (port 53) that host ctrld can
observe. A guest-side resolver (ctrld, systemd-resolved with DoT, dnscrypt, ...)
sends its upstream queries encrypted instead, so the host learns no addresses and
the guest's egress is blocked - and its DoT is blocked outright by the port-853 rule.
That is the trust boundary working as intended, not a regression: a guest that
resolves privately could otherwise reach any destination it liked. Point the guest
at the host bridge address (its default DHCP resolver) and let host ctrld enforce
policy for it.
- **DoH over 443** inside the guest is indistinguishable from ordinary HTTPS and
is not intercepted. To keep enforcement strict, disable DoH in the guest OS/
browser, or restrict the guest to the host resolver.
- **IPv6 guest DNS** is not redirected (ctrld's intercept listener is IPv4); the
anchor's existing IPv6 DNS block forces guests to fall back to interceptable
IPv4 DNS. IPv6 forwarded sources are therefore unsupported: an IPv6
`firewall_forwarded_sources` entry is ignored with a warning rather than emitted
as a rule.
- Auto-detection needs ownership proof: a vendor interface name, or a bridge with a
vendor VM member. A VM network on a plain unrecognized interface, or a bridge whose
hypervisor attaches no vendor-named member, needs an explicit
`firewall_forwarded_sources` entry. Bridge membership is read with `ifconfig`, and
only for a bridge that already carries an RFC1918 IPv4 address.
- macOS only. Windows (WFP) Firewall Mode VM behavior is tracked separately
(#568).
## Metrics
Allowlist stats are logged every 5 minutes:
@@ -151,7 +307,7 @@ Firewall allowlist stats allowed_ips=142 permanent_ips=18 tracked_domains=89 tot
- Check allowlist stats for hit/miss ratio
### Certain apps don't work
- The app may be using hardcoded IPs (this is the intended behavior — those IPs aren't DNS-resolved)
- The app may be using hardcoded IPs (this is the intended behavior - those IPs aren't DNS-resolved)
- Check if the app uses a custom DNS resolver that bypasses ctrld
- RFC1918 traffic is always allowed, so LAN-only apps should work