cli: add network fallback and explicit-only Internal Domain modes

Internal Domains now support three modes. "os" (network default) is
unchanged. "resolvers" becomes explicit resolver with network fallback,
the default explicit selection: the configured resolvers are tried
first, and a timeout, unreachable resolver, SERVFAIL, NXDOMAIN, REFUSED
or NOTIMP hands the query to the network resolvers - matching VPN DNS
servers, then domain-less VPN DNS servers, then the OS resolver's LAN
nameservers. A valid empty answer stays final. "resolvers_only" keeps
the previous strict behavior.

The OS step uses a new LanOnlyQueryCtx: the OS resolver then asks only
its LAN-classified nameservers (private, loopback, link-local, CGNAT)
and sends nothing when it has none, so the private name never reaches a
public nameserver, whether DHCP supplied it or it is ctrld's own public
fallback. LanQueryCtx only dropped the latter. LAN-only queries get
their own singleflight and hot-cache key, so they are never answered
with an ordinary query's public answer.

In fallback mode a configured resolver that the upstream monitor has
marked down is skipped instead of waited on, so an endpoint off the
organization network reaches the fallback without paying the
resolver's 2s timeout on every query. Nothing else marks a generated
resolver up again, so skipping one re-checks it in the background, at
most once every 30s; an answer marks it up. Strict mode never skips.

An absent mode with resolvers gets the fallback mode. An unrecognized
mode with resolvers fails closed to explicit resolver only, so a wire
value this build does not know never sends the domain to the Control D
upstream; without resolvers it is dropped. Switching between the
explicit modes is detected on refresh.

Generated upstreams are now identified by an UpstreamConfig.InternalDomain
marker that records their mode and that no configuration file can set,
instead of by the internal_ key prefix. An upstream a local ctrld.toml or
custom config defines as internal_foo, or as a copy of a generated one,
is an ordinary upstream again: it keeps the OS-resolver catch-all,
recovery and query-health grading.

The "resolvers_only" wire value is provisional until the API agrees on
it; the dashboard does not offer strict mode yet.

cmd/cli/dns_proxy.go

@@ -577,11 +577,13 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 	//
 	// An Internal Domain with explicit resolvers does not take part. The bypass
 	// exists because general DNS is broken, which says nothing about the
-	// administrator's own resolvers: they may well be reachable, and sending a
-	// private name to the network's DNS is the disclosure the explicit
-	// selection exists to prevent. Such a query continues to the normal flow,
-	// where it reaches only its configured resolvers and fails if none answer.
-	if dnsIntercept && p.recoveryBypass.Load() && !internalDomainExplicitUpstreams(req.ufr.upstreams) {
+	// administrator's own resolvers: they may well be reachable, and they are
+	// to be tried first. Such a query continues to the normal flow, where it
+	// reaches its configured resolvers first. In "explicit resolver only" mode
+	// it fails if none answer; otherwise it falls back to the VPN resolvers and
+	// the network's LAN nameservers, never to the public nameservers the bypass
+	// may use.
+	if dnsIntercept && p.recoveryBypass.Load() && !p.internalDomainExplicitUpstreams(req.ufr.upstreams) {
 		ctrld.Log(ctx, mainLog.Load().Debug(), "Recovery bypass active: forwarding to OS resolver")
 		resolver, err := ctrld.NewResolver(osUpstreamConfig)
 		if err == nil {
@@ -713,13 +715,15 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 	//
 	// An Internal Domain with explicit resolvers is skipped here: the
 	// administrator named the resolvers for that suffix, and VPN suffixes
-	// are auto-detected, so handing the query to a VPN DNS server would
-	// override an explicit selection with a discovered one.
+	// are auto-detected, so handing the query to a VPN DNS server first would
+	// override an explicit selection with a discovered one. In fallback mode
+	// the VPN DNS servers are asked later, once the configured resolvers have
+	// not resolved the name.
 	//
 	// Internal Domains in OS-resolver mode are not skipped, they ask for the
 	// endpoint's default resolution, which under intercept mode includes
 	// the VPN's own resolver.
-	if dnsIntercept && p.vpnDNS != nil && len(req.msg.Question) > 0 && !internalDomainExplicitUpstreams(upstreams) {
+	if dnsIntercept && p.vpnDNS != nil && len(req.msg.Question) > 0 && !p.internalDomainExplicitUpstreams(upstreams) {
 		domain := req.msg.Question[0].Name
 		if vpnServers := p.vpnDNS.UpstreamForDomain(domain); len(vpnServers) > 0 {
 			ctrld.Log(ctx, mainLog.Load().Debug(), "VPN DNS route matched for domain %s, using servers: %v", domain, vpnServers)
@@ -871,7 +875,7 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 		// generated Internal Domain upstream is the organization's private
 		// resolver address. Report the classification through the sampler and
 		// keep the address-bearing error at debug.
-		if isInternalDomainUpstream(upstream) {
+		if isGeneratedInternalDomainUpstream(upstreamConfig) {
 			ctrld.Log(ctx, mainLog.Load().Debug().Err(err), "failed to resolve query")
 			ctrld.Log(ctx, p.querySampler.event(sampleClassInternalDomain, upstream).Str("failure", internalDomainFailureReason(err)),
 				"failed to resolve query using an Internal Domain resolver")
@@ -897,6 +901,12 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 
 		return nil
 	}
+	// An Internal Domain in explicit-with-network-fallback mode moves on from a
+	// configured resolver that answers SERVFAIL or NXDOMAIN, as it does from
+	// one that does not answer. internalNegative keeps the best of those
+	// answers, so a name nobody resolves gets a real negative answer.
+	internalFallback := p.internalDomainFallbackUpstreams(upstreams)
+	var internalNegative *dns.Msg
 	for n, upstreamConfig := range upstreamConfigs {
 		if upstreamConfig == nil {
 			continue
@@ -911,6 +921,16 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 			ctrld.Log(ctx, logger, "DNS loop detected")
 			continue
 		}
+		// A fallback-mode Internal Domain resolver that the monitor reports as
+		// down is not waited on: off the organization network every query
+		// would otherwise pay its timeout before the network fallback runs.
+		// Nothing else marks a generated resolver up again, so skipping it
+		// starts a background re-check that does.
+		if internalFallback && p.um.isDown(upstreams[n]) {
+			ctrld.Log(ctx, mainLog.Load().Debug(), "internal domain resolver %s is down, skipping it", upstreams[n])
+			p.internalDomainProbes.probe(p.um, upstreams[n], upstreamConfig, req.msg)
+			continue
+		}
 		answer := resolve(upstreams[n], upstreamConfig, req.msg)
 		if answer == nil {
 			if serveStaleCache && staleAnswer != nil {
@@ -945,6 +965,12 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 			ctrld.Log(ctx, mainLog.Load().Debug(), "failover rcode matched, process to next upstream")
 			continue
 		}
+		if internalFallback && internalDomainFallbackRcode(answer.Rcode) {
+			ctrld.Log(ctx, mainLog.Load().Debug(), "internal domain resolver %s answered %s, trying next resolver",
+				upstreams[n], dns.RcodeToString[answer.Rcode])
+			internalNegative = betterInternalDomainNegative(internalNegative, answer)
+			continue
+		}
 
 		// set compression, as it is not set by default when unpacking
 		answer.Compress = true
@@ -989,15 +1015,40 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 		res.upstream = upstreamConfig.Endpoint
 		return res
 	}
+	if internalFallback {
+		ctrld.Log(ctx, mainLog.Load().Debug(), "internal domain resolvers did not resolve the query; trying network resolvers")
+		answer, negative := p.resolveInternalDomainOnNetwork(ctx, req.msg)
+		if answer != nil {
+			answer.Compress = true
+			if cacheable {
+				p.addCachedResponse(dnscache.NewKey(req.msg, upstreams[0]), answer)
+				ctrld.Log(ctx, mainLog.Load().Debug(), "add cached response")
+			}
+			hostname := ""
+			if req.ci != nil {
+				hostname = req.ci.Hostname
+			}
+			ctrld.Log(ctx, mainLog.Load().Info(), "REPLY: internal domain network fallback -> %s (%s): %s", req.ufr.srcAddr, hostname, dns.RcodeToString[answer.Rcode])
+			res.answer = answer
+			return res
+		}
+		if negative = betterInternalDomainNegative(internalNegative, negative); negative != nil {
+			ctrld.Log(ctx, mainLog.Load().Debug(), "no resolver resolved the internal domain; returning %s", dns.RcodeToString[negative.Rcode])
+			negative.Compress = true
+			res.answer = negative
+			return res
+		}
+	}
 	ctrld.Log(ctx, p.querySampler.event(sampleClassAllEndpointsFailed, ""), "all %v endpoints failed", journalUpstreamNames(upstreams))
 
-	// An Internal Domain with explicit resolvers is served only by the
-	// configured resolvers. When all of them are unreachable the query fails
-	// here: the OS-resolver catch-all below would send a private name to a resolver
-	// that was not selected for it, and the recovery flow exists for the loss of
-	// general DNS, not for one unavailable internal server.
-	if internalDomainExplicitUpstreams(upstreams) {
-		ctrld.Log(ctx, mainLog.Load().Debug(), "internal domain resolvers unreachable; not falling back")
+	// An Internal Domain with explicit resolvers never takes the OS-resolver
+	// catch-all below: it can reach public nameservers, ctrld's own fallback
+	// included, which no mode selects for a private name. Fallback mode has
+	// already tried the VPN resolvers and the network's LAN nameservers above.
+	// The recovery flow exists for the loss of general DNS, not for one
+	// unavailable internal server, so it is not started either.
+	if p.internalDomainExplicitUpstreams(upstreams) {
+		ctrld.Log(ctx, mainLog.Load().Debug(), "internal domain resolvers unreachable; not using the OS resolver catch all")
 	} else if p.leakOnUpstreamFailure() {
 		if p.um.countHealthy(upstreams) == 0 {
 			p.recoveryCancelMu.Lock()
@@ -1046,7 +1097,7 @@ func (p *prog) proxy(ctx context.Context, req *proxyRequest) *proxyResponse {
 // endpoint away from the organization network never reaches them, and that is
 // not an outage of the query path.
 func (p *prog) countFailedClientQuery(upstreams []string) {
-	if internalDomainExplicitUpstreams(upstreams) {
+	if p.internalDomainExplicitUpstreams(upstreams) {
 		return
 	}
 	p.health.countFailedQuery()
@@ -2705,7 +2756,7 @@ func (p *prog) buildRecoveryUpstreams(reason RecoveryReason) map[string]*ctrld.U
 	case RecoveryReasonOSFailure:
 		for k, uc := range p.cfg.Upstream {
 			name := upstreamPrefix + k
-			if uc != nil && uc.Type != ctrld.ResolverTypeOS && !isInternalDomainUpstream(name) {
+			if uc != nil && uc.Type != ctrld.ResolverTypeOS && !isGeneratedInternalDomainUpstream(uc) {
 				upstreams[name] = uc
 			}
 		}