From c48220a404c9c04e37c355be98e3e8b3e40bb5eb Mon Sep 17 00:00:00 2001 From: Anthony Wong Date: Thu, 3 Sep 2026 22:24:18 +0000 Subject: [PATCH] docs: complete the provisioning failure runbook and canary matrix --- docs/provisioning-failure-codes.md | 152 ++++++++++++++++++++++------- 1 file changed, 117 insertions(+), 35 deletions(-) diff --git a/docs/provisioning-failure-codes.md b/docs/provisioning-failure-codes.md index 5185e9f..e1f3ea6 100644 --- a/docs/provisioning-failure-codes.md +++ b/docs/provisioning-failure-codes.md @@ -15,8 +15,9 @@ stable code on three surfaces: `stage=[a-z]* code=[A-Z_]* (exit [0-9]*)`) into their own logs without risking token leakage from other output. - **Exit code** — stage-scoped: input 20–29, bootstrap 30–39, - listener 40–49, service 50–59. Unrelated existing contracts are - unchanged (`ctrld-client status` exits 0–3; invalid deactivation pin exits 126). + listener 40–49, service 50–59. This does not change other existing + exit-code contracts: `ctrld-client status` still exits 0–3, and an invalid + deactivation pin still exits 126. A customer or administrator only needs to report the code (or the whole output line). The table below is the maintained support mapping; it must @@ -24,39 +25,109 @@ stay in sync with `cmd/cli/provision_result.go` and changes in the same MR. ## Codes +For every code below, first run `sudo ctrld-client diag` (`sudo ctrld-client diag --json` +gives a paste-safe, machine-readable copy). Get that output before you do +anything else. The command needs no running service. Without root, the +last-provisioning-result and service-state sections report permission +denied, so run it with `sudo`. It repeats the same stage, code, and message +the customer already reported. +The **Next action / evidence** column lists what to look at next. + | Code | Stage | Exit | Failure scenario | Next action / evidence | |---|---|---|---|---| -| `PROVISION_TOKEN_MALFORMED` | input | 21 | The `--cd-org` value is clearly not a provisioning code: explicitly empty, too short, too long, or containing whitespace or control characters. Checked before any network call. A missing `org-v1-` prefix is not this code — that only logs a warning, since legacy codes may lack it. | Check the provisioning code was copied in full, with no extra whitespace. The result file never echoes the value. | -| `CUSTOM_HOSTNAME_INVALID` | input | 22 | The `--custom-hostname` value fails ctrld's own hostname rule (`validHostname`): not 3–64 characters, or not RFC1123 hostname format. Checked before any network call. The API itself does not reject a bad hostname during provisioning — ControlD folds/strips characters like space, `+`, and `.` when it registers the device name, so a value ctrld accepts may still register under an adjusted name (logged as a notice, not a failure). | Fix `--custom-hostname` per the message: it names the offending character(s) and the allowed format. | -| `INTERCEPT_MODE_INVALID` | input | 23 | The `--intercept-mode` value is not one of `off`, `dns`, or `hard`. Checked before installing the service. | Re-run with a valid `--intercept-mode` value. | -| `INVALID_FLAG_COMBINATION` | input | 24 | `--cd`/`--cd-org` used together with `--nextdns`, or `--proto` set to anything other than `doh`/`doh3` once `--cd` is in play. Checked before any network call or service install. | Re-run without the conflicting flag, or fix the invalid value. The message names the exact flags involved. | -| `API_UNREACHABLE` | bootstrap | 30 | The Control D API could not be reached or answered with a retryable error (network failure, proxy interference, 5xx, timeout) and retries ran out. The service manager may retry the service later. | Check the device's network path to `api.controld.com` (DNS, proxy, firewall, captive portal). Ask for the result file's `message` and whether other TLS traffic works. | -| `API_REJECTED` | bootstrap | 31 | The API answered and permanently rejected the configuration (4xx other than 408/429): bad or revoked token, malformed request. ctrld exits without burning service-manager restarts because retrying cannot change the answer. | Verify the provision token / org configuration in the Control D dashboard. Re-push after fixing credentials. Evidence: HTTP status in the result file `message`. | -| `API_DEVICE_INVALID` | bootstrap | 32 | The API reports the device/resolver no longer exists (error code 40402). When the daemon's own bootstrap preflight discovers this, it self-uninstalls because the identity is gone server-side. The direct `--cd ` install path (`ctrld-client start --cd`) can also hit this code, before the service exists; there is nothing to uninstall yet on that path. | Confirm the device was deleted or re-provisioned in the dashboard; re-provision with a current token. No local evidence needed beyond the code. | -| `TOKEN_INVALID` | bootstrap | 33 | The API rejected the `--cd-org` provisioning code with reason `token_invalid`: the code is not recognized. | Check the code and re-enter it exactly as given. | -| `TOKEN_EXPIRED` | bootstrap | 34 | The API rejected the `--cd-org` provisioning code with reason `token_expired`. | Get a new provisioning code from your administrator. | -| `TOKEN_LIMIT_REACHED` | bootstrap | 35 | The API rejected the `--cd-org` provisioning code with reason `token_limit_reached`: it has reached its device limit. | Free up a device slot or use a different provisioning code. | -| `TOKEN_DISABLED` | bootstrap | 36 | The API rejected the `--cd-org` provisioning code with reason `token_disabled`: the code was invalidated. | Download a profile from an active provisioning code. | +| `PROVISION_TOKEN_MALFORMED` | input | 21 | The `--cd-org` value is not a provisioning code: it is empty, too short, too long, or it contains whitespace or control characters. ctrld checks this before any network call. A missing `org-v1-` prefix is not this code. That case only logs a warning, because legacy codes can lack the prefix. | Make sure that the provisioning code was copied in full, with no extra whitespace. The result file never echoes the value. | +| `CUSTOM_HOSTNAME_INVALID` | input | 22 | The `--custom-hostname` value fails ctrld's own hostname rule (`validHostname`): it is not 3 to 64 characters, or it is not in RFC1123 hostname format. ctrld checks this before any network call. The API itself does not reject a bad hostname during provisioning. When ControlD registers the device name, it folds or strips characters like space, `+`, and `.`. Because of this, a value ctrld accepts can still register under an adjusted name. That case only logs a notice, not a failure. | Fix `--custom-hostname` following the message: it names the offending characters and the allowed format. | +| `INTERCEPT_MODE_INVALID` | input | 23 | The `--intercept-mode` value is not one of `off`, `dns`, or `hard`. ctrld checks this before it installs the service. | Re-run with a valid `--intercept-mode` value. | +| `INVALID_FLAG_COMBINATION` | input | 24 | `--cd` or `--cd-org` is used together with `--nextdns`, or `--proto` is set to anything other than `doh`/`doh3` while `--cd` is in play. ctrld checks this before any network call or service install. | Re-run without the conflicting flag, or fix the invalid value. The message names the exact flags involved. | +| `API_UNREACHABLE` | bootstrap | 30 | The Control D API could not be reached or answered with a retryable error (network failure, proxy interference, 5xx, timeout) and retries ran out. The service manager may retry the service later. | Make sure that the device has network access to `api.controld.com` (DNS, proxy, firewall, captive portal). `ctrld-client diag`'s `api_reachability` block already ran this same probe. Ask whether other TLS traffic works. | +| `API_REJECTED` | bootstrap | 31 | The API answered and permanently rejected the configuration (4xx other than 408/429): bad or revoked token, malformed request. ctrld exits without burning service-manager restarts because retrying cannot change the answer. This is also the fallback for the four `TOKEN_*` codes below. Those codes need the API response to carry `error.metadata.reason`. An API deployment that does not yet populate that field reports every provisioning-code rejection as `API_REJECTED` instead. | Verify the provision token / org configuration in the Control D dashboard. Re-push after fixing credentials. Evidence: HTTP status in the result file `message`. | +| `API_DEVICE_INVALID` | bootstrap | 32 | The API reports the device/resolver no longer exists (error code 40402). When the daemon's own bootstrap preflight discovers this, it self-uninstalls, because the identity is gone server-side. The direct `--cd ` install path (`ctrld-client start --cd`) can also hit this code before the service exists. On that path there is nothing to uninstall yet. | Confirm the device was deleted or re-provisioned in the dashboard; re-provision with a current token. No local evidence needed beyond the code. | +| `TOKEN_INVALID` | bootstrap | 33 | The API rejected the `--cd-org` provisioning code with reason `token_invalid`: the code is not recognized. This code needs the deployed API to send `error.metadata.reason: "token_invalid"`. See `API_REJECTED` above for the fallback when it does not. | Make sure that the code is correct. Re-enter it exactly as given. | +| `TOKEN_EXPIRED` | bootstrap | 34 | The API rejected the `--cd-org` provisioning code with reason `token_expired`. This has the same API-reason dependency as `TOKEN_INVALID`. | Get a new provisioning code from your administrator. | +| `TOKEN_LIMIT_REACHED` | bootstrap | 35 | The API rejected the `--cd-org` provisioning code with reason `token_limit_reached`: it reached its device limit. This has the same API-reason dependency as `TOKEN_INVALID`. | Free up a device slot or use a different provisioning code. | +| `TOKEN_DISABLED` | bootstrap | 36 | The API rejected the `--cd-org` provisioning code with reason `token_disabled`: the code was invalidated. This has the same API-reason dependency as `TOKEN_INVALID`. | Download a profile from an active provisioning code. | | `LISTENER_BIND_FAILED` | listener | 41 | No listen address could be bound after all fallbacks (configured address, 0.0.0.0:53, localhost:53, random) were exhausted. `detail.attempts` records each tried address with the UDP/TCP OS error, e.g. `address already in use` (another DNS service owns the port) or `can't assign requested address` (address not on any interface). | Read `detail.attempts`: `address already in use` → find the process owning the port (`sudo lsof -i :53 -nP`); `can't assign requested address` → the configured IP is not present on the device. Then fix the conflict or the listener config. | | `LISTENER_CONFIGURED_ADDR_UNAVAILABLE` | listener | 42 | An explicitly configured listener address could not be bound and configuration checks forbid falling back to another address, or (macOS intercept mode) the required explicit address is unavailable. | The configured `ip:port` in the listener config is wrong for this device or occupied. Verify the address exists on an interface and nothing else binds it; correct the config rather than expecting fallback. | | `SERVICE_INSTALL_FAILED` | service | 51 | The OS service manager refused to install the service (launchd/systemd/SCM registration failed). | Check OS-level constraints: permissions/elevation, MDM policy blocking daemon installation, corrupted previous install. Evidence: result file `message` (service manager error), plus `launchctl print system/ctrld-client` / `systemctl status ctrld-client` / SCM state. | | `SERVICE_START_FAILED` | service | 52 | The service installed but the service manager could not start it. | Check the service manager's own log for the start error, then the ctrld home dir `ctrld.log`. Often permissions or a binary quarantined by security tooling. | | `SERVICE_SELFCHECK_FAILED` | service | 53 | The service started but never became healthy: no fresher failure was reported by the daemon, and the post-install DNS self-check failed. On a fresh install or an upgrade, the just-installed service is rolled back (uninstalled). A restart of an already installed service keeps that service installed. If the daemon itself recorded a more specific failure (e.g. a listener code), that code is reported instead of this one. | Ask for the drained service log printed by `ctrld-client start` and the result file. If the service was running but unreachable, check host firewall rules intercepting DNS to the listener. | -| `UNCLASSIFIED` | service | 59 | A terminal failure on the provisioning boundary that predates a dedicated code: a config unmarshal, a file-system or environment failure (writing the config file, reading `socketDir`, respawning as a daemon), a service-argument update failing mid-upgrade, or a DNS-intercept failure the interface-DNS fallback cannot safely take over from. Every one of these used to be a bare crash with nothing to read; now they all persist a result file. | Read the result file `message`: it names the specific operation that failed and the underlying OS error. Treat this the same as any other stage-scoped failure when reporting it. | +| `UNCLASSIFIED` | service | 59 | This code covers terminal failures on the provisioning boundary that predate a dedicated code. Examples include a config unmarshal, and a file-system or environment failure such as writing the config file, reading `socketDir`, or respawning as a daemon. Other examples are a service-argument update failing mid-upgrade, and a DNS-intercept failure the interface-DNS fallback cannot safely take over from. Every one of these used to be a bare crash with nothing to read. Now they all persist a result file. | Read the result file `message`: it names the specific operation that failed and the underlying OS error. When you report it, treat this the same as any other stage-scoped failure. | -The package rows below are emitted by the macOS installer script -(`scripts/pkg/postinstall`), not by ctrld itself, before ctrld ever runs. -They only ever exit 1 (the script's own exit code) and never appear in -`provision_result.json`. The same script also reuses `PROVISION_TOKEN_MALFORMED` -and `CUSTOM_HOSTNAME_INVALID` from the table above for its own pre-flight checks -on the managed-prefs `ProvisionToken`/`CustomHostname` values (same rule, -also logged as `stage=package (exit 1)`), so those two codes can appear with -either stage depending on which side rejected the value. +The macOS installer script (`scripts/pkg/postinstall`) emits the package rows +below, not ctrld itself, before ctrld ever runs. They only ever exit 1 (the +script's own exit code), and they never appear in `provision_result.json`. +The same script also reuses `PROVISION_TOKEN_MALFORMED` and +`CUSTOM_HOSTNAME_INVALID` from the table above, for its own pre-flight +checks on the managed-prefs `ProvisionToken`/`CustomHostname` values. It +uses the same rule, and also logs it as `stage=package (exit 1)`. So those +two codes can appear with either stage, depending on which side rejected +the value. -| `PROFILE_PREFS_MISSING` | package | 1 | The MDM configuration profile (managed prefs domain `com.controld.ctrld`) has no `ProvisionToken`, or an empty one, after the postinstall script's wait loop runs out. | Scope the `com.controld.ctrld` configuration profile to the device, then reinstall the package or run `sudo ctrld-client start --cd-org=` by hand. | -| `INTERCEPT_MODE_INVALID` | package | 1 | On a fresh install, the profile's `InterceptMode` is set but is not one of the managed-prefs values (empty, `standard`, `intercept-dns`). A fresh install fails visibly rather than silently falling back, since a managed deployment should not provision into an unintended mode. On an upgrade the same bad value only logs a warning and keeps the existing service mode, so a bad profile edit cannot brick an already-working fleet. | Fix `InterceptMode` in the profile to one of the allowed values, then reinstall the package. | -| `SERVICE_RELOAD_FAILED` | package | 1 | An upgrade reused the existing service mode (no managed `InterceptMode` override), and `launchctl load` on the installed plist failed. | Run `sudo launchctl load /Library/LaunchDaemons/ctrld-client.plist` by hand and check its error. | -| `TEMP_FILE_UNAVAILABLE` | package | 1 | The script could not create the private temp file it uses to capture ctrld's output (`/tmp` or `TMPDIR` full or unwritable). ctrld is never invoked, so no diagnostics can be appended. | Free space in `/tmp` or `TMPDIR` (or fix its permissions), then reinstall the package. | +| `PROFILE_PREFS_MISSING` | package | 1 | The MDM configuration profile (managed prefs domain `com.controld.ctrld`) has no `ProvisionToken`, or the value is empty. This shows up after the postinstall script's wait loop ends: 12 attempts, 10 seconds apart, about 2 minutes. | Scope the `com.controld.ctrld` configuration profile to the device. If the profile is already on the device, set its `ProvisionToken` value. Then reinstall the package, or run `sudo ctrld-client start --cd-org=` by hand. | +| `INTERCEPT_MODE_INVALID` | package | 1 | On a fresh install, the profile's `InterceptMode` is set, but it is not one of the managed-prefs values (empty, `standard`, `intercept-dns`). A fresh install fails visibly, instead of silently falling back, because a managed deployment must not provision into an unintended mode. On an upgrade, the same bad value only logs a warning and keeps the existing service mode. So a bad profile edit cannot break an already-working fleet. | Fix `InterceptMode` in the profile to one of the allowed values. Then reinstall the package. | +| `SERVICE_RELOAD_FAILED` | package | 1 | An upgrade reused the existing service mode (no managed `InterceptMode` override), and `launchctl load` on the installed plist failed. | Run `sudo launchctl load /Library/LaunchDaemons/ctrld-client.plist` by hand, and read the error it gives. | +| `TEMP_FILE_UNAVAILABLE` | package | 1 | The script failed to create the private temp file it uses to capture ctrld's output (`/tmp` or `TMPDIR` is full or unwritable). Because ctrld never runs, the script has no diagnostics to add. | Free space in `/tmp` or `TMPDIR`, or fix its permissions. Then reinstall the package. | + +## Customer-visible message shape + +Every failure shows the same two-part shape: the fixed identifier line, then +one free-text line specific to the code. Neither part ever contains the +provision token, the resolver/device ID, or config contents. + +- **Input stage** — names the offending flag or value and the rule it + broke. For example: `--cd-org provisioning code is malformed: must be + 6-64 characters with no whitespace or control characters`. For + `CUSTOM_HOSTNAME_INVALID`, the message names the specific offending + characters and the allowed format. +- **Bootstrap, `TOKEN_*`** — one fixed sentence per code (see the table + above). For example: `the provisioning code has expired; get a new + provisioning code from your administrator`. +- **Bootstrap, `API_UNREACHABLE`/`API_REJECTED`/`API_DEVICE_INVALID`** — + When the failure surfaces during a running install's periodic resolver + refresh, the message is `ControlD API rejected this configuration (HTTP + status )`. When it surfaces during the initial provisioning call, the + message is `provision token exchange failed: ` or `failed to fetch + resolver config: `. Neither ever includes the request or response + body. +- **Listener** — names the address or addresses tried. `detail.attempts` in + the result file carries the OS error for each address and protocol. +- **Service** — For `SERVICE_INSTALL_FAILED` or `SERVICE_START_FAILED`, the + message is ` failed: ` (`` is `Install` or `Start`). + When the daemon itself reported no fresher failure, `SERVICE_SELFCHECK_FAILED` + shows one of three messages, depending on what the self-check saw. When + there is no error and the service is not running, the message is `ctrld + service did not pass its post-start self-check`. When the test query + itself errored, the message is `An error occurred while performing test + query: `. When the service was running but the query silently + failed, the message is `ctrld service was running, but a DNS query could + not be sent to its listener; check firewall rules + blocking/intercepting/redirecting DNS queries`. +- **Package** — the plain-language second line the installer script logs + right after the identifier line. Each package row's *Next action* cell + above quotes this line verbatim, because the script has no separate + result file to carry it. + +## Where each code surfaces + +- **Every ctrld-native code** (every row above except `package`): the CLI + output / persisted `ctrld.log`, `provision_result.json`, and `ctrld-client diag` + (which reads that same file). +- **Package-stage codes** (`PROFILE_PREFS_MISSING`, `SERVICE_RELOAD_FAILED`, + `TEMP_FILE_UNAVAILABLE`, the package's own `INTERCEPT_MODE_INVALID` check + on the managed-prefs value, and the package pre-flight's reused + `PROVISION_TOKEN_MALFORMED` and `CUSTOM_HOSTNAME_INVALID`): only + `/var/log/install.log`. ctrld never ran, so no result file exists yet. + `sudo ctrld-client diag` run right after this reports "none recorded" for the last + provisioning result. That is still useful: it shows that the package + script's own gate rejected the profile before ctrld had a chance to run. +- **Any ctrld-native code can also reach `/var/log/install.log`.** The + postinstall script invokes ctrld in two cases: to provision on a fresh + install, or to reapply a managed intercept mode on an upgrade. Either + time, it greps ctrld's own identifier line out of the captured output. + It re-logs that line, and appends `(ctrld exit )`. So a bootstrap or + listener code, not only the three package-only codes, can appear in the + MDM install log. Right after any installer-script failure, it also + appends a fresh `sudo ctrld-client diag` run, each line prefixed `diag:`. ## Reading the result file @@ -77,20 +148,31 @@ file is absent. ## Support: `ctrld-client diag` Run `sudo ctrld-client diag` (or `sudo ctrld-client diag --json`) to collect the facts -support needs for a provisioning ticket in one copy-paste-safe command: -client version, MDM-managed preferences (macOS only, token reported as -present/absent only, an empty value counts as absent), the last -provisioning result, service state, and whether the Control D API is -reachable. It needs no config and always exits 0. A failure it finds is -part of the report, not a command failure. It also runs without root, but -then the last-provisioning-result and the service-state sections report -permission denied. Never asks for or prints the provisioning -token itself. +support needs for a provisioning ticket, in one copy-paste-safe command. It +collects the client version and the MDM-managed preferences (macOS only, +the token reported only as present or absent, an empty value counts as +absent). It also collects the last provisioning result, the service state, +and whether the Control D API is reachable. It needs no config. It also +runs without root, but then the last-provisioning-result and the +service-state sections report permission denied, and the text report names +the step that gets the rights it needs. Even when it finds a failure, it +still exits 0: that failure is part of the report, not a command failure. +It never asks for or prints the provisioning token itself. + +The `provision_result.status` field in the JSON report is one of `none` +(no result file), `recorded`, `untrusted` (a file that fails the contract +check), `unreadable` (permission denied), or `corrupt` (a file that could +not be parsed). ## Rules for maintainers - Codes are append-only once released. Never rename, renumber, or reuse a code or exit number; add a new one and note the deprecation here. +- API rejection reasons are append-only too: `tokenFailureCodeForReason` + maps a fixed set of `error.metadata.reason` strings to their `TOKEN_*` + code. An unrecognized or absent reason falls back to `API_REJECTED` (see + `apiFailureCode`). So a new reason string does nothing until both a + matching code and a row here exist. - Every code added in `cmd/cli/provision_result.go` needs a row here in the same MR. Tests enforce the code/stage/exit maps and that this table has exactly one `stage != package` row per ctrld code. The `package` rows