mirror of
https://github.com/Control-D-Inc/ctrld.git
synced 2026-09-04 13:36:35 +02:00
all: keep the first-attempt error when the direct-ip fallback also fails
Both API requests and binary downloads retry against a hard-coded IP when the attempt via hostname fails. Both then overwrote the first error with the fallback's, so only the last failure was reported. That discarded the diagnosis. During the Firewall Mode incident the hostname attempt was denied locally - WSAEACCES, "An attempt was made to access a socket in a way forbidden by its access permissions", which means the host is blocking ctrld - while the direct-ip fallback failed with an unreachable IPv6 route. What surfaced to the operator was "dial tcp6: no route to host", pointing at a routing problem that did not exist, while the WSAEACCES that named the real cause was visible only in debug logs. Report both failures instead, keeping the error chain intact so errors.Is still matches either one. Also switch the final wrap in doWithRetry from %v to %w, which had been flattening the chain even when a single error was reported. Add coverage for both paths, including that the fallback is still attempted and that a successful fallback returns no error. This also changes retry classification, which is worth stating explicitly because it is not obvious from "report both errors". processCDFlags decides whether to keep backing off with errUrlNetworkError, which uses errors.As - and errors.As returns the first match in the tree. Wrapping the hostname attempt first therefore hands the predicate that attempt's failure, where previously only the fallback's error survived to be classified. The effect is intended. A locally denied socket (WSAEACCES) is not a transient network error, so preflight now fails fast and reports instead of retrying against a firewall that is not going to clear on its own - the incident logged 256 retry cycles doing exactly that. The case that justifies retrying forever, a network unreachable on both attempts at boot, is unchanged. Both classifications are pinned by tests, along with the wrap order they depend on at each composition site, so reversing it fails loudly rather than silently restoring the old behaviour.
This commit is contained in:
@@ -390,17 +390,31 @@ func addrsFromPort(ips []string, port string) []string {
|
||||
return addrs
|
||||
}
|
||||
|
||||
// doWithFallback sends req, retrying against apiIp directly if the first attempt
|
||||
// fails (typically because DNS is not usable yet).
|
||||
//
|
||||
// Both failures are reported. The first attempt carries the diagnosis - on Windows
|
||||
// a local firewall denying the socket surfaces there as WSAEACCES ("An attempt was
|
||||
// made to access a socket in a way forbidden by its access permissions"), which
|
||||
// says the host is blocking ctrld rather than that the network is down. Returning
|
||||
// only the fallback error hid that behind a bare "no route to host" from the IPv6
|
||||
// attempt and sent the Firewall Mode incident investigation after a routing
|
||||
// problem that did not exist.
|
||||
func doWithFallback(ctx context.Context, client *http.Client, req *http.Request, apiIp string) (*http.Response, error) {
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
logger := ctrld.LoggerFromCtx(ctx)
|
||||
logger.Warn().Err(err).Msgf("Failed to send request, fallback to direct ip: %s", apiIp)
|
||||
ipReq := req.Clone(req.Context())
|
||||
ipReq.Host = apiIp
|
||||
ipReq.URL.Host = apiIp
|
||||
resp, err = client.Do(ipReq)
|
||||
if err == nil {
|
||||
return resp, nil
|
||||
}
|
||||
return resp, err
|
||||
logger := ctrld.LoggerFromCtx(ctx)
|
||||
logger.Warn().Err(err).Msgf("Failed to send request, fallback to direct ip: %s", apiIp)
|
||||
ipReq := req.Clone(req.Context())
|
||||
ipReq.Host = apiIp
|
||||
ipReq.URL.Host = apiIp
|
||||
resp, fallbackErr := client.Do(ipReq)
|
||||
if fallbackErr != nil {
|
||||
return nil, fmt.Errorf("request failed: %w; fallback to direct ip %s failed: %w", err, apiIp, fallbackErr)
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// apiServerIP returns the direct IP to connect to API server.
|
||||
@@ -410,3 +424,10 @@ func apiServerIP(cdDev bool) string {
|
||||
}
|
||||
return apiDomainComIPv4
|
||||
}
|
||||
|
||||
// DoWithFallbackForTest exposes doWithFallback so tests outside this package can drive
|
||||
// the real two-attempt composition through the real retry predicate, rather than
|
||||
// asserting a copy of this error shape against another copy of it.
|
||||
func DoWithFallbackForTest(ctx context.Context, client *http.Client, req *http.Request, apiIp string) (*http.Response, error) {
|
||||
return doWithFallback(ctx, client, req, apiIp)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user