mirror of
https://github.com/Control-D-Inc/ctrld.git
synced 2026-09-04 13:36:35 +02:00
Cuong Manh Le
88e076abe7
Rollback ran os.Remove(bin) while the replacement service was still running from that image. Windows locks a running executable, so the remove failed with "Access is denied" - and it was fatal, so the os.Rename that restores the previous binary never ran. The upgrade ended with the broken replacement still installed and the working binary stranded at its _previous name, which is what the Firewall Mode incident produced. Readiness failing is not evidence the process exited: the service manager can report a started service whose process never became operational. So rollback now stops the service and waits until the manager reports it stopped before touching the executable, then resets DNS the way the restart path's Cleanup task does. That also leaves the host in a known state - a stopped ctrld holds no WFP or NRPT enforcement, so a replacement that was blocking traffic stops blocking it at rollback instead of at the next reboot. Restoring is now conditional on the previous binary reporting a version. The incident's _previous file existed but produced no version output; renaming that over the current binary would have traded a service that starts and hangs for one that cannot start at all. When it is unusable, rollback keeps it for inspection, leaves the installed binary alone, and says so instead of pressing on. The --version probe is bounded by a timeout so a binary that hangs cannot hang the upgrade. Remaining failures return errors rather than calling Fatal, so each one reports what state the host was left in. os.Remove is retried while the path stays locked, since Windows releases an image lock asynchronously after the process exits. Extract the rollback into rollbackToPreviousBinary() and cover it: the stop happens while the executable is still present, an unusable previous binary is kept without swapping or restarting, and a failed stop aborts before anything is modified. Reversing the stop and the remove fails these tests. The version probe is called through a variable so those tests do not have to stage a runnable executable. Staging one is not portable: oldBin is bin+"_previous", so a fixture named "ctrld" yields the extension-less "ctrld_previous", which Windows refuses to execute, and a symlink to the test binary needs a privilege Windows does not grant by default. The probe itself is still covered against the real test binary. Production is unaffected: ctrld.exe _previous does have an extension, and os/exec only appends PATHEXT entries when a path has none at all - noted at binaryVersion so the suffix is not renamed into something extension-less by accident.
70 lines
2.3 KiB
Go
70 lines
2.3 KiB
Go
package cli
|
|
|
|
import (
|
|
"fmt"
|
|
"os"
|
|
"os/exec"
|
|
"strings"
|
|
"testing"
|
|
|
|
"go.uber.org/zap"
|
|
"go.uber.org/zap/zapcore"
|
|
|
|
"github.com/Control-D-Inc/ctrld"
|
|
)
|
|
|
|
var logOutput strings.Builder
|
|
|
|
// envFakeVersionOutput makes this test binary impersonate a ctrld executable: when
|
|
// set, the process writes the value to stdout and exits without running any test, so
|
|
// binaryVersion() can be exercised on every platform without building or shipping a
|
|
// fixture binary. The value envFakeVersionSilent produces no output at all, which
|
|
// reproduces the unusable ctrld.exe_previous seen in the Firewall Mode incident.
|
|
//
|
|
// This must be handled before m.Run(), which is what parses the test flags: the child
|
|
// is invoked as "<binary> --version" and would otherwise die on an unknown flag.
|
|
const (
|
|
envFakeVersionOutput = "CTRLD_TEST_FAKE_VERSION_OUTPUT"
|
|
envFakeVersionSilent = "<silent>"
|
|
)
|
|
|
|
func TestMain(m *testing.M) {
|
|
if out := os.Getenv(envFakeVersionOutput); out != "" {
|
|
if out != envFakeVersionSilent {
|
|
fmt.Println(out)
|
|
}
|
|
os.Exit(0)
|
|
}
|
|
|
|
// Create a custom writer that writes to logOutput
|
|
writer := zapcore.AddSync(&logOutput)
|
|
|
|
// Create zap encoder
|
|
encoderConfig := zap.NewDevelopmentEncoderConfig()
|
|
encoder := zapcore.NewConsoleEncoder(encoderConfig)
|
|
|
|
// Create core that writes to our string builder
|
|
core := zapcore.NewCore(encoder, writer, zap.DebugLevel)
|
|
|
|
// Create logger
|
|
l := zap.New(core)
|
|
|
|
mainLog.Store(&ctrld.Logger{Logger: l})
|
|
|
|
// Stub the self-upgrade command builder for the whole test binary. The real
|
|
// builder execs os.Executable() — which under `go test` IS this test binary
|
|
// — with positional args ("upgrade", ...). `go test` stops flag parsing at
|
|
// the first positional arg and ignores the rest, so the child just re-runs
|
|
// the entire suite, hits the upgrade tests again, and spawns more children:
|
|
// a fork bomb of detached processes that stalls the host and (on Windows)
|
|
// holds the test binary's image locked, breaking CI artifact cleanup.
|
|
// Point it at the test binary with a no-match -test.run so any test that
|
|
// reaches performUpgrade still exercises the cmd.Start() success path while
|
|
// the child exits immediately without recursing.
|
|
newUpgradeCmd = func(exe string) *exec.Cmd {
|
|
return exec.Command(exe, "-test.run=^$")
|
|
}
|
|
|
|
os.Exit(m.Run())
|
|
}
|