Rollback ran os.Remove(bin) while the replacement service was still running from
that image. Windows locks a running executable, so the remove failed with
"Access is denied" - and it was fatal, so the os.Rename that restores the
previous binary never ran. The upgrade ended with the broken replacement still
installed and the working binary stranded at its _previous name, which is what
the Firewall Mode incident produced.
Readiness failing is not evidence the process exited: the service manager can
report a started service whose process never became operational. So rollback now
stops the service and waits until the manager reports it stopped before touching
the executable, then resets DNS the way the restart path's Cleanup task does.
That also leaves the host in a known state - a stopped ctrld holds no WFP or
NRPT enforcement, so a replacement that was blocking traffic stops blocking it
at rollback instead of at the next reboot.
Restoring is now conditional on the previous binary reporting a version. The
incident's _previous file existed but produced no version output; renaming that
over the current binary would have traded a service that starts and hangs for
one that cannot start at all. When it is unusable, rollback keeps it for
inspection, leaves the installed binary alone, and says so instead of pressing
on. The --version probe is bounded by a timeout so a binary that hangs cannot
hang the upgrade.
Remaining failures return errors rather than calling Fatal, so each one reports
what state the host was left in. os.Remove is retried while the path stays
locked, since Windows releases an image lock asynchronously after the process
exits.
Extract the rollback into rollbackToPreviousBinary() and cover it: the stop
happens while the executable is still present, an unusable previous binary is
kept without swapping or restarting, and a failed stop aborts before anything is
modified. Reversing the stop and the remove fails these tests.
The version probe is called through a variable so those tests do not have to
stage a runnable executable. Staging one is not portable: oldBin is
bin+"_previous", so a fixture named "ctrld" yields the extension-less
"ctrld_previous", which Windows refuses to execute, and a symlink to the test
binary needs a privilege Windows does not grant by default. The probe itself is
still covered against the real test binary. Production is unaffected: ctrld.exe
_previous does have an extension, and os/exec only appends PATHEXT entries when
a path has none at all - noted at binaryVersion so the suffix is not renamed
into something extension-less by accident.
Cuong Manh Le
88e076abe7