all: keep the first-attempt error when the direct-ip fallback also fails

Both API requests and binary downloads retry against a hard-coded IP when the
attempt via hostname fails. Both then overwrote the first error with the
fallback's, so only the last failure was reported.

That discarded the diagnosis. During the Firewall Mode incident the hostname
attempt was denied locally - WSAEACCES, "An attempt was made to access a socket
in a way forbidden by its access permissions", which means the host is blocking
ctrld - while the direct-ip fallback failed with an unreachable IPv6 route. What
surfaced to the operator was "dial tcp6: no route to host", pointing at a
routing problem that did not exist, while the WSAEACCES that named the real
cause was visible only in debug logs.

Report both failures instead, keeping the error chain intact so errors.Is still
matches either one. Also switch the final wrap in doWithRetry from %v to %w,
which had been flattening the chain even when a single error was reported.

Add coverage for both paths, including that the fallback is still attempted and
that a successful fallback returns no error.

This also changes retry classification, which is worth stating explicitly because
it is not obvious from "report both errors". processCDFlags decides whether to
keep backing off with errUrlNetworkError, which uses errors.As - and errors.As
returns the first match in the tree. Wrapping the hostname attempt first
therefore hands the predicate that attempt's failure, where previously only the
fallback's error survived to be classified.

The effect is intended. A locally denied socket (WSAEACCES) is not a transient
network error, so preflight now fails fast and reports instead of retrying
against a firewall that is not going to clear on its own - the incident logged
256 retry cycles doing exactly that. The case that justifies retrying forever, a
network unreachable on both attempts at boot, is unchanged. Both classifications
are pinned by tests, along with the wrap order they depend on at each composition
site, so reversing it fails loudly rather than silently restoring the old
behaviour.
This commit is contained in:
Cuong Manh Le
2026-08-14 15:29:32 +07:00
parent 891d5d9821
commit 052b057756
5 changed files with 489 additions and 17 deletions
+17 -6
View File
@@ -64,8 +64,13 @@ func httpClientWithFallback(timeout time.Duration) *http.Client {
// doWithRetry performs an HTTP request with retries
// This improves reliability by automatically retrying failed requests with exponential backoff
func doWithRetry(req *http.Request, maxRetries int, ip string) (*http.Response, error) {
return doWithRetryClient(httpClientWithFallback(defaultHTTPTimeout), req, maxRetries, ip)
}
// doWithRetryClient is doWithRetry with an injectable client, so the retry and
// error-composition behaviour can be tested without real network access.
func doWithRetryClient(client *http.Client, req *http.Request, maxRetries int, ip string) (*http.Response, error) {
var lastErr error
client := httpClientWithFallback(defaultHTTPTimeout)
var ipReq *http.Request
if ip != "" {
ipReq = req.Clone(req.Context())
@@ -82,22 +87,28 @@ func doWithRetry(req *http.Request, maxRetries int, ip string) (*http.Response,
if err == nil {
return resp, nil
}
// Keep the hostname attempt's error: it carries the diagnosis (on Windows,
// a local firewall denying the socket shows up here as WSAEACCES), while the
// direct-ip fallback often fails for an unrelated reason such as an
// unreachable IPv6 route.
attemptErr := err
if ipReq != nil {
mainLog.Load().Warn().Err(err).Msgf("Dial to %q failed", req.Host)
mainLog.Load().Warn().Msgf("Fallback to direct ip to download prod version: %q", ip)
resp, err = client.Do(ipReq)
if err == nil {
resp, fallbackErr := client.Do(ipReq)
if fallbackErr == nil {
return resp, nil
}
attemptErr = fmt.Errorf("%w; fallback to direct ip %s failed: %w", attemptErr, ip, fallbackErr)
}
lastErr = err
mainLog.Load().Debug().Err(err).
lastErr = attemptErr
mainLog.Load().Debug().Err(attemptErr).
Str("method", req.Method).
Str("url", req.URL.String()).
Msgf("HTTP request attempt %d/%d failed", attempt+1, maxRetries)
}
return nil, fmt.Errorf("failed after %d attempts to %s %s: %v", maxRetries, req.Method, req.URL, lastErr)
return nil, fmt.Errorf("failed after %d attempts to %s %s: %w", maxRetries, req.Method, req.URL, lastErr)
}
// Helper for making GET requests with retries