cmd/cli: stop the replacement before rolling its binary back

Rollback ran os.Remove(bin) while the replacement service was still running from
that image. Windows locks a running executable, so the remove failed with
"Access is denied" - and it was fatal, so the os.Rename that restores the
previous binary never ran. The upgrade ended with the broken replacement still
installed and the working binary stranded at its _previous name.

Readiness failing is not evidence the process exited: the service manager can
report a started service whose process never became operational. So rollback now
stops the service and waits until the manager reports it stopped before touching
the executable, then cleans up DNS the way the restart path's Cleanup task does.

Restoring is now conditional on the previous binary reporting a version, since a
_previous file that exists but produces no version output would trade a service
that starts and hangs for one that cannot start at all. When it is unusable,
rollback keeps it for inspection, leaves the installed binary alone, and says so
instead of pressing on. The --version probe is bounded by a timeout so a binary
that hangs cannot hang the upgrade.

Remaining failures are reported rather than fatal, so each one says what state
the host was left in. os.Remove is retried while the path stays locked, since
Windows releases an image lock asynchronously after the process exits.

The helpers live in a new file rather than in commands.go, and the rollback is
extracted into rollbackToPreviousBinary() so it can be covered: the stop happens
while the executable is still present, an unusable previous binary is kept
without swapping or restarting, and a failed stop aborts before anything is
modified. Reversing the stop and the remove fails these tests.

The version probe is called through a variable so those tests do not have to
stage a runnable executable. Staging one is not portable: oldBin is
bin+"_previous", so a fixture named "ctrld" yields the extension-less
"ctrld_previous", which Windows refuses to execute, and a symlink to the test
binary needs a privilege Windows does not grant by default. The probe itself is
still covered against the real test binary. Production is unaffected: ctrld.exe
_previous does have an extension, and os/exec only appends PATHEXT entries when
a path has none at all - noted at binaryVersion so the suffix is not renamed
into something extension-less by accident.
This commit is contained in:
Cuong Manh Le
2026-08-21 14:50:11 +07:00
parent 2400f27962
commit 5c9d3dec4e
4 changed files with 521 additions and 14 deletions
+17 -14
View File
@@ -1501,28 +1501,31 @@ func initUpgradeCmd() *cobra.Command {
if doRestart() {
_ = os.Remove(oldBin)
_ = os.Chmod(bin, 0755)
ver := "unknown version"
out, err := exec.Command(bin, "--version").CombinedOutput()
ver, err := binaryVersion(bin)
if err != nil {
mainLog.Load().Warn().Err(err).Msg("Failed to get new binary version")
}
if after, found := strings.CutPrefix(string(out), "ctrld version "); found {
ver = after
ver = "unknown version"
}
mainLog.Load().Notice().Msgf("Upgrade successful - %s", ver)
return
}
mainLog.Load().Warn().Msgf("Upgrade failed, restoring previous binary: %s", oldBin)
if err := os.Remove(bin); err != nil {
mainLog.Load().Fatal().Err(err).Msg("failed to remove new binary")
mainLog.Load().Warn().Msg("Upgrade failed: the new binary did not become ready")
stop := func() error {
if !svcInstalled {
return nil
}
if err := stopServiceAndWait(s, upgradeStopTimeout); err != nil {
return err
}
// Mirror the Cleanup task in doRestart: leave DNS settings as the OS
// had them, not as a half-started ctrld left them.
p.router.Cleanup()
p.resetDNS(false, true)
return nil
}
if err := os.Rename(oldBin, bin); err != nil {
mainLog.Load().Fatal().Err(err).Msg("failed to restore old binary")
}
if doRestart() {
mainLog.Load().Notice().Msg("Restored previous binary successfully")
return
if err := rollbackToPreviousBinary(bin, oldBin, stop, doRestart); err != nil {
mainLog.Load().Error().Err(err).Msg("Rollback did not complete")
}
},
}