all: keep the first-attempt error when the direct-ip fallback also fails

Both API requests and binary downloads retry against a hard-coded IP when the
attempt via hostname fails. Both then overwrote the first error with the
fallback's, so only the last failure was reported.

That discarded the diagnosis. During the Firewall Mode incident the hostname
attempt was denied locally - WSAEACCES, "An attempt was made to access a socket
in a way forbidden by its access permissions", which means the host is blocking
ctrld - while the direct-ip fallback failed with an unreachable IPv6 route. What
surfaced to the operator was "dial tcp6: no route to host", pointing at a
routing problem that did not exist, while the WSAEACCES that named the real
cause was visible only in debug logs.

Report both failures instead, keeping the error chain intact so errors.Is still
matches either one. Also switch the final wrap in doWithRetry from %v to %w,
which had been flattening the chain even when a single error was reported.

Add coverage for both paths, including that the fallback is still attempted and
that a successful fallback returns no error.

This also changes retry classification, which is worth stating explicitly because
it is not obvious from "report both errors". processCDFlags decides whether to
keep backing off with errUrlNetworkError, which uses errors.As - and errors.As
returns the first match in the tree. Wrapping the hostname attempt first
therefore hands the predicate that attempt's failure, where previously only the
fallback's error survived to be classified.

The effect is intended. A locally denied socket (WSAEACCES) is not a transient
network error, so preflight now fails fast and reports instead of retrying
against a firewall that is not going to clear on its own - the incident logged
256 retry cycles doing exactly that. The case that justifies retrying forever, a
network unreachable on both attempts at boot, is unchanged. Both classifications
are pinned by tests, along with the wrap order they depend on at each composition
site, so reversing it fails loudly rather than silently restoring the old
behaviour.
This commit is contained in:
Cuong Manh Le
2026-08-14 15:29:32 +07:00
parent 891d5d9821
commit 052b057756
5 changed files with 489 additions and 17 deletions
+45 -3
View File
@@ -1500,14 +1500,56 @@ var (
windowsEADDRINUSE = syscall.Errno(10048)
)
// errUrlNetworkError reports whether a failed HTTP attempt is worth retrying.
//
// The two-attempt paths compose one *url.Error per attempt - hostname first, then the
// direct-IP fallback - so this walks them in order rather than classifying only the first
// one errors.As happens to find. Each attempt can say one of three things:
//
// - retryable (unreachable, refused, temporary): retry, whichever attempt said it;
// - a name-resolution failure: no verdict. Only the hostname attempt resolves DNS, and
// at boot behind a captive portal or before the router's forwarder is up it fails
// this way while the network is merely not ready yet. Consult the next attempt;
// - anything else, notably a locally denied socket (WSAEACCES from a firewall blocking
// ctrld): definitive. Stop, because retrying cannot clear it - the Firewall Mode
// incident spent 256 retry cycles against filters that were never going to clear.
func errUrlNetworkError(err error) bool {
var urlErr *url.Error
if errors.As(err, &urlErr) {
return errNetworkError(urlErr.Err)
for _, attempt := range attemptErrors(err) {
var urlErr *url.Error
if !errors.As(attempt, &urlErr) {
continue
}
switch {
case errNetworkError(urlErr.Err):
return true
case errDNSResolutionFailure(urlErr.Err):
// Neutral; let a later attempt decide.
default:
return false
}
}
return false
}
// attemptErrors returns the per-attempt errors recorded in err, in the order they were
// tried. A composed fallback error wraps one per attempt; anything else is a single
// attempt.
func attemptErrors(err error) []error {
if multi, ok := err.(interface{ Unwrap() []error }); ok {
return multi.Unwrap()
}
return []error{err}
}
// errDNSResolutionFailure reports whether err is a name-resolution failure. Go marks a
// *net.DNSError as temporary only for socket failures that reached the server, so a
// SERVFAIL or "no such host" answer is not temporary - but it is also not evidence that
// retrying is pointless, which is why callers treat it as no verdict.
func errDNSResolutionFailure(err error) bool {
var dnsErr *net.DNSError
return errors.As(err, &dnsErr)
}
func errNetworkError(err error) bool {
var opErr *net.OpError
if errors.As(err, &opErr) {