docs: complete the provisioning failure runbook and canary matrix

docs/provisioning-failure-codes.md

@@ -15,8 +15,9 @@ stable code on three surfaces:
   `stage=[a-z]* code=[A-Z_]* (exit [0-9]*)`) into their own logs without
   risking token leakage from other output.
 - **Exit code** — stage-scoped: input 20–29, bootstrap 30–39,
-  listener 40–49, service 50–59. Unrelated existing contracts are
-  unchanged (`ctrld-client status` exits 0–3; invalid deactivation pin exits 126).
+  listener 40–49, service 50–59. This does not change other existing
+  exit-code contracts: `ctrld-client status` still exits 0–3, and an invalid
+  deactivation pin still exits 126.
 
 A customer or administrator only needs to report the code (or the whole
 output line). The table below is the maintained support mapping; it must
@@ -24,39 +25,109 @@ stay in sync with `cmd/cli/provision_result.go` and changes in the same MR.
 
 ## Codes
 
+For every code below, first run `sudo ctrld-client diag` (`sudo ctrld-client diag --json`
+gives a paste-safe, machine-readable copy). Get that output before you do
+anything else. The command needs no running service. Without root, the
+last-provisioning-result and service-state sections report permission
+denied, so run it with `sudo`. It repeats the same stage, code, and message
+the customer already reported.
+The **Next action / evidence** column lists what to look at next.
+
 | Code | Stage | Exit | Failure scenario | Next action / evidence |
 |---|---|---|---|---|
-| `PROVISION_TOKEN_MALFORMED` | input | 21 | The `--cd-org` value is clearly not a provisioning code: explicitly empty, too short, too long, or containing whitespace or control characters. Checked before any network call. A missing `org-v1-` prefix is not this code — that only logs a warning, since legacy codes may lack it. | Check the provisioning code was copied in full, with no extra whitespace. The result file never echoes the value. |
-| `CUSTOM_HOSTNAME_INVALID` | input | 22 | The `--custom-hostname` value fails ctrld's own hostname rule (`validHostname`): not 3–64 characters, or not RFC1123 hostname format. Checked before any network call. The API itself does not reject a bad hostname during provisioning — ControlD folds/strips characters like space, `+`, and `.` when it registers the device name, so a value ctrld accepts may still register under an adjusted name (logged as a notice, not a failure). | Fix `--custom-hostname` per the message: it names the offending character(s) and the allowed format. |
-| `INTERCEPT_MODE_INVALID` | input | 23 | The `--intercept-mode` value is not one of `off`, `dns`, or `hard`. Checked before installing the service. | Re-run with a valid `--intercept-mode` value. |
-| `INVALID_FLAG_COMBINATION` | input | 24 | `--cd`/`--cd-org` used together with `--nextdns`, or `--proto` set to anything other than `doh`/`doh3` once `--cd` is in play. Checked before any network call or service install. | Re-run without the conflicting flag, or fix the invalid value. The message names the exact flags involved. |
-| `API_UNREACHABLE` | bootstrap | 30 | The Control D API could not be reached or answered with a retryable error (network failure, proxy interference, 5xx, timeout) and retries ran out. The service manager may retry the service later. | Check the device's network path to `api.controld.com` (DNS, proxy, firewall, captive portal). Ask for the result file's `message` and whether other TLS traffic works. |
-| `API_REJECTED` | bootstrap | 31 | The API answered and permanently rejected the configuration (4xx other than 408/429): bad or revoked token, malformed request. ctrld exits without burning service-manager restarts because retrying cannot change the answer. | Verify the provision token / org configuration in the Control D dashboard. Re-push after fixing credentials. Evidence: HTTP status in the result file `message`. |
-| `API_DEVICE_INVALID` | bootstrap | 32 | The API reports the device/resolver no longer exists (error code 40402). When the daemon's own bootstrap preflight discovers this, it self-uninstalls because the identity is gone server-side. The direct `--cd <uid>` install path (`ctrld-client start --cd`) can also hit this code, before the service exists; there is nothing to uninstall yet on that path. | Confirm the device was deleted or re-provisioned in the dashboard; re-provision with a current token. No local evidence needed beyond the code. |
-| `TOKEN_INVALID` | bootstrap | 33 | The API rejected the `--cd-org` provisioning code with reason `token_invalid`: the code is not recognized. | Check the code and re-enter it exactly as given. |
-| `TOKEN_EXPIRED` | bootstrap | 34 | The API rejected the `--cd-org` provisioning code with reason `token_expired`. | Get a new provisioning code from your administrator. |
-| `TOKEN_LIMIT_REACHED` | bootstrap | 35 | The API rejected the `--cd-org` provisioning code with reason `token_limit_reached`: it has reached its device limit. | Free up a device slot or use a different provisioning code. |
-| `TOKEN_DISABLED` | bootstrap | 36 | The API rejected the `--cd-org` provisioning code with reason `token_disabled`: the code was invalidated. | Download a profile from an active provisioning code. |
+| `PROVISION_TOKEN_MALFORMED` | input | 21 | The `--cd-org` value is not a provisioning code: it is empty, too short, too long, or it contains whitespace or control characters. ctrld checks this before any network call. A missing `org-v1-` prefix is not this code. That case only logs a warning, because legacy codes can lack the prefix. | Make sure that the provisioning code was copied in full, with no extra whitespace. The result file never echoes the value. |
+| `CUSTOM_HOSTNAME_INVALID` | input | 22 | The `--custom-hostname` value fails ctrld's own hostname rule (`validHostname`): it is not 3 to 64 characters, or it is not in RFC1123 hostname format. ctrld checks this before any network call. The API itself does not reject a bad hostname during provisioning. When ControlD registers the device name, it folds or strips characters like space, `+`, and `.`. Because of this, a value ctrld accepts can still register under an adjusted name. That case only logs a notice, not a failure. | Fix `--custom-hostname` following the message: it names the offending characters and the allowed format. |
+| `INTERCEPT_MODE_INVALID` | input | 23 | The `--intercept-mode` value is not one of `off`, `dns`, or `hard`. ctrld checks this before it installs the service. | Re-run with a valid `--intercept-mode` value. |
+| `INVALID_FLAG_COMBINATION` | input | 24 | `--cd` or `--cd-org` is used together with `--nextdns`, or `--proto` is set to anything other than `doh`/`doh3` while `--cd` is in play. ctrld checks this before any network call or service install. | Re-run without the conflicting flag, or fix the invalid value. The message names the exact flags involved. |
+| `API_UNREACHABLE` | bootstrap | 30 | The Control D API could not be reached or answered with a retryable error (network failure, proxy interference, 5xx, timeout) and retries ran out. The service manager may retry the service later. | Make sure that the device has network access to `api.controld.com` (DNS, proxy, firewall, captive portal). `ctrld-client diag`'s `api_reachability` block already ran this same probe. Ask whether other TLS traffic works. |
+| `API_REJECTED` | bootstrap | 31 | The API answered and permanently rejected the configuration (4xx other than 408/429): bad or revoked token, malformed request. ctrld exits without burning service-manager restarts because retrying cannot change the answer. This is also the fallback for the four `TOKEN_*` codes below. Those codes need the API response to carry `error.metadata.reason`. An API deployment that does not yet populate that field reports every provisioning-code rejection as `API_REJECTED` instead. | Verify the provision token / org configuration in the Control D dashboard. Re-push after fixing credentials. Evidence: HTTP status in the result file `message`. |
+| `API_DEVICE_INVALID` | bootstrap | 32 | The API reports the device/resolver no longer exists (error code 40402). When the daemon's own bootstrap preflight discovers this, it self-uninstalls, because the identity is gone server-side. The direct `--cd <uid>` install path (`ctrld-client start --cd`) can also hit this code before the service exists. On that path there is nothing to uninstall yet. | Confirm the device was deleted or re-provisioned in the dashboard; re-provision with a current token. No local evidence needed beyond the code. |
+| `TOKEN_INVALID` | bootstrap | 33 | The API rejected the `--cd-org` provisioning code with reason `token_invalid`: the code is not recognized. This code needs the deployed API to send `error.metadata.reason: "token_invalid"`. See `API_REJECTED` above for the fallback when it does not. | Make sure that the code is correct. Re-enter it exactly as given. |
+| `TOKEN_EXPIRED` | bootstrap | 34 | The API rejected the `--cd-org` provisioning code with reason `token_expired`. This has the same API-reason dependency as `TOKEN_INVALID`. | Get a new provisioning code from your administrator. |
+| `TOKEN_LIMIT_REACHED` | bootstrap | 35 | The API rejected the `--cd-org` provisioning code with reason `token_limit_reached`: it reached its device limit. This has the same API-reason dependency as `TOKEN_INVALID`. | Free up a device slot or use a different provisioning code. |
+| `TOKEN_DISABLED` | bootstrap | 36 | The API rejected the `--cd-org` provisioning code with reason `token_disabled`: the code was invalidated. This has the same API-reason dependency as `TOKEN_INVALID`. | Download a profile from an active provisioning code. |
 | `LISTENER_BIND_FAILED` | listener | 41 | No listen address could be bound after all fallbacks (configured address, 0.0.0.0:53, localhost:53, random) were exhausted. `detail.attempts` records each tried address with the UDP/TCP OS error, e.g. `address already in use` (another DNS service owns the port) or `can't assign requested address` (address not on any interface). | Read `detail.attempts`: `address already in use` → find the process owning the port (`sudo lsof -i :53 -nP`); `can't assign requested address` → the configured IP is not present on the device. Then fix the conflict or the listener config. |
 | `LISTENER_CONFIGURED_ADDR_UNAVAILABLE` | listener | 42 | An explicitly configured listener address could not be bound and configuration checks forbid falling back to another address, or (macOS intercept mode) the required explicit address is unavailable. | The configured `ip:port` in the listener config is wrong for this device or occupied. Verify the address exists on an interface and nothing else binds it; correct the config rather than expecting fallback. |
 | `SERVICE_INSTALL_FAILED` | service | 51 | The OS service manager refused to install the service (launchd/systemd/SCM registration failed). | Check OS-level constraints: permissions/elevation, MDM policy blocking daemon installation, corrupted previous install. Evidence: result file `message` (service manager error), plus `launchctl print system/ctrld-client` / `systemctl status ctrld-client` / SCM state. |
 | `SERVICE_START_FAILED` | service | 52 | The service installed but the service manager could not start it. | Check the service manager's own log for the start error, then the ctrld home dir `ctrld.log`. Often permissions or a binary quarantined by security tooling. |
 | `SERVICE_SELFCHECK_FAILED` | service | 53 | The service started but never became healthy: no fresher failure was reported by the daemon, and the post-install DNS self-check failed. On a fresh install or an upgrade, the just-installed service is rolled back (uninstalled). A restart of an already installed service keeps that service installed. If the daemon itself recorded a more specific failure (e.g. a listener code), that code is reported instead of this one. | Ask for the drained service log printed by `ctrld-client start` and the result file. If the service was running but unreachable, check host firewall rules intercepting DNS to the listener. |
-| `UNCLASSIFIED` | service | 59 | A terminal failure on the provisioning boundary that predates a dedicated code: a config unmarshal, a file-system or environment failure (writing the config file, reading `socketDir`, respawning as a daemon), a service-argument update failing mid-upgrade, or a DNS-intercept failure the interface-DNS fallback cannot safely take over from. Every one of these used to be a bare crash with nothing to read; now they all persist a result file. | Read the result file `message`: it names the specific operation that failed and the underlying OS error. Treat this the same as any other stage-scoped failure when reporting it. |
+| `UNCLASSIFIED` | service | 59 | This code covers terminal failures on the provisioning boundary that predate a dedicated code. Examples include a config unmarshal, and a file-system or environment failure such as writing the config file, reading `socketDir`, or respawning as a daemon. Other examples are a service-argument update failing mid-upgrade, and a DNS-intercept failure the interface-DNS fallback cannot safely take over from. Every one of these used to be a bare crash with nothing to read. Now they all persist a result file. | Read the result file `message`: it names the specific operation that failed and the underlying OS error. When you report it, treat this the same as any other stage-scoped failure. |
 
-The package rows below are emitted by the macOS installer script
-(`scripts/pkg/postinstall`), not by ctrld itself, before ctrld ever runs.
-They only ever exit 1 (the script's own exit code) and never appear in
-`provision_result.json`. The same script also reuses `PROVISION_TOKEN_MALFORMED`
-and `CUSTOM_HOSTNAME_INVALID` from the table above for its own pre-flight checks
-on the managed-prefs `ProvisionToken`/`CustomHostname` values (same rule,
-also logged as `stage=package (exit 1)`), so those two codes can appear with
-either stage depending on which side rejected the value.
+The macOS installer script (`scripts/pkg/postinstall`) emits the package rows
+below, not ctrld itself, before ctrld ever runs. They only ever exit 1 (the
+script's own exit code), and they never appear in `provision_result.json`.
+The same script also reuses `PROVISION_TOKEN_MALFORMED` and
+`CUSTOM_HOSTNAME_INVALID` from the table above, for its own pre-flight
+checks on the managed-prefs `ProvisionToken`/`CustomHostname` values. It
+uses the same rule, and also logs it as `stage=package (exit 1)`. So those
+two codes can appear with either stage, depending on which side rejected
+the value.
 
-| `PROFILE_PREFS_MISSING` | package | 1 | The MDM configuration profile (managed prefs domain `com.controld.ctrld`) has no `ProvisionToken`, or an empty one, after the postinstall script's wait loop runs out. | Scope the `com.controld.ctrld` configuration profile to the device, then reinstall the package or run `sudo ctrld-client start --cd-org=<token>` by hand. |
-| `INTERCEPT_MODE_INVALID` | package | 1 | On a fresh install, the profile's `InterceptMode` is set but is not one of the managed-prefs values (empty, `standard`, `intercept-dns`). A fresh install fails visibly rather than silently falling back, since a managed deployment should not provision into an unintended mode. On an upgrade the same bad value only logs a warning and keeps the existing service mode, so a bad profile edit cannot brick an already-working fleet. | Fix `InterceptMode` in the profile to one of the allowed values, then reinstall the package. |
-| `SERVICE_RELOAD_FAILED` | package | 1 | An upgrade reused the existing service mode (no managed `InterceptMode` override), and `launchctl load` on the installed plist failed. | Run `sudo launchctl load /Library/LaunchDaemons/ctrld-client.plist` by hand and check its error. |
-| `TEMP_FILE_UNAVAILABLE` | package | 1 | The script could not create the private temp file it uses to capture ctrld's output (`/tmp` or `TMPDIR` full or unwritable). ctrld is never invoked, so no diagnostics can be appended. | Free space in `/tmp` or `TMPDIR` (or fix its permissions), then reinstall the package. |
+| `PROFILE_PREFS_MISSING` | package | 1 | The MDM configuration profile (managed prefs domain `com.controld.ctrld`) has no `ProvisionToken`, or the value is empty. This shows up after the postinstall script's wait loop ends: 12 attempts, 10 seconds apart, about 2 minutes. | Scope the `com.controld.ctrld` configuration profile to the device. If the profile is already on the device, set its `ProvisionToken` value. Then reinstall the package, or run `sudo ctrld-client start --cd-org=<token>` by hand. |
+| `INTERCEPT_MODE_INVALID` | package | 1 | On a fresh install, the profile's `InterceptMode` is set, but it is not one of the managed-prefs values (empty, `standard`, `intercept-dns`). A fresh install fails visibly, instead of silently falling back, because a managed deployment must not provision into an unintended mode. On an upgrade, the same bad value only logs a warning and keeps the existing service mode. So a bad profile edit cannot break an already-working fleet. | Fix `InterceptMode` in the profile to one of the allowed values. Then reinstall the package. |
+| `SERVICE_RELOAD_FAILED` | package | 1 | An upgrade reused the existing service mode (no managed `InterceptMode` override), and `launchctl load` on the installed plist failed. | Run `sudo launchctl load /Library/LaunchDaemons/ctrld-client.plist` by hand, and read the error it gives. |
+| `TEMP_FILE_UNAVAILABLE` | package | 1 | The script failed to create the private temp file it uses to capture ctrld's output (`/tmp` or `TMPDIR` is full or unwritable). Because ctrld never runs, the script has no diagnostics to add. | Free space in `/tmp` or `TMPDIR`, or fix its permissions. Then reinstall the package. |
+
+## Customer-visible message shape
+
+Every failure shows the same two-part shape: the fixed identifier line, then
+one free-text line specific to the code. Neither part ever contains the
+provision token, the resolver/device ID, or config contents.
+
+- **Input stage** — names the offending flag or value and the rule it
+  broke. For example: `--cd-org provisioning code is malformed: must be
+  6-64 characters with no whitespace or control characters`. For
+  `CUSTOM_HOSTNAME_INVALID`, the message names the specific offending
+  characters and the allowed format.
+- **Bootstrap, `TOKEN_*`** — one fixed sentence per code (see the table
+  above). For example: `the provisioning code has expired; get a new
+  provisioning code from your administrator`.
+- **Bootstrap, `API_UNREACHABLE`/`API_REJECTED`/`API_DEVICE_INVALID`** —
+  When the failure surfaces during a running install's periodic resolver
+  refresh, the message is `ControlD API rejected this configuration (HTTP
+  status <n>)`. When it surfaces during the initial provisioning call, the
+  message is `provision token exchange failed: <err>` or `failed to fetch
+  resolver config: <err>`. Neither ever includes the request or response
+  body.
+- **Listener** — names the address or addresses tried. `detail.attempts` in
+  the result file carries the OS error for each address and protocol.
+- **Service** — For `SERVICE_INSTALL_FAILED` or `SERVICE_START_FAILED`, the
+  message is `<task> failed: <err>` (`<task>` is `Install` or `Start`).
+  When the daemon itself reported no fresher failure, `SERVICE_SELFCHECK_FAILED`
+  shows one of three messages, depending on what the self-check saw. When
+  there is no error and the service is not running, the message is `ctrld
+  service did not pass its post-start self-check`. When the test query
+  itself errored, the message is `An error occurred while performing test
+  query: <err>`. When the service was running but the query silently
+  failed, the message is `ctrld service was running, but a DNS query could
+  not be sent to its listener; check firewall rules
+  blocking/intercepting/redirecting DNS queries`.
+- **Package** — the plain-language second line the installer script logs
+  right after the identifier line. Each package row's *Next action* cell
+  above quotes this line verbatim, because the script has no separate
+  result file to carry it.
+
+## Where each code surfaces
+
+- **Every ctrld-native code** (every row above except `package`): the CLI
+  output / persisted `ctrld.log`, `provision_result.json`, and `ctrld-client diag`
+  (which reads that same file).
+- **Package-stage codes** (`PROFILE_PREFS_MISSING`, `SERVICE_RELOAD_FAILED`,
+  `TEMP_FILE_UNAVAILABLE`, the package's own `INTERCEPT_MODE_INVALID` check
+  on the managed-prefs value, and the package pre-flight's reused
+  `PROVISION_TOKEN_MALFORMED` and `CUSTOM_HOSTNAME_INVALID`): only
+  `/var/log/install.log`. ctrld never ran, so no result file exists yet.
+  `sudo ctrld-client diag` run right after this reports "none recorded" for the last
+  provisioning result. That is still useful: it shows that the package
+  script's own gate rejected the profile before ctrld had a chance to run.
+- **Any ctrld-native code can also reach `/var/log/install.log`.** The
+  postinstall script invokes ctrld in two cases: to provision on a fresh
+  install, or to reapply a managed intercept mode on an upgrade. Either
+  time, it greps ctrld's own identifier line out of the captured output.
+  It re-logs that line, and appends `(ctrld exit <n>)`. So a bootstrap or
+  listener code, not only the three package-only codes, can appear in the
+  MDM install log. Right after any installer-script failure, it also
+  appends a fresh `sudo ctrld-client diag` run, each line prefixed `diag:`.
 
 ## Reading the result file
 
@@ -77,20 +148,31 @@ file is absent.
 ## Support: `ctrld-client diag`
 
 Run `sudo ctrld-client diag` (or `sudo ctrld-client diag --json`) to collect the facts
-support needs for a provisioning ticket in one copy-paste-safe command:
-client version, MDM-managed preferences (macOS only, token reported as
-present/absent only, an empty value counts as absent), the last
-provisioning result, service state, and whether the Control D API is
-reachable. It needs no config and always exits 0. A failure it finds is
-part of the report, not a command failure. It also runs without root, but
-then the last-provisioning-result and the service-state sections report
-permission denied. Never asks for or prints the provisioning
-token itself.
+support needs for a provisioning ticket, in one copy-paste-safe command. It
+collects the client version and the MDM-managed preferences (macOS only,
+the token reported only as present or absent, an empty value counts as
+absent). It also collects the last provisioning result, the service state,
+and whether the Control D API is reachable. It needs no config. It also
+runs without root, but then the last-provisioning-result and the
+service-state sections report permission denied, and the text report names
+the step that gets the rights it needs. Even when it finds a failure, it
+still exits 0: that failure is part of the report, not a command failure.
+It never asks for or prints the provisioning token itself.
+
+The `provision_result.status` field in the JSON report is one of `none`
+(no result file), `recorded`, `untrusted` (a file that fails the contract
+check), `unreadable` (permission denied), or `corrupt` (a file that could
+not be parsed).
 
 ## Rules for maintainers
 
 - Codes are append-only once released. Never rename, renumber, or reuse a
   code or exit number; add a new one and note the deprecation here.
+- API rejection reasons are append-only too: `tokenFailureCodeForReason`
+  maps a fixed set of `error.metadata.reason` strings to their `TOKEN_*`
+  code. An unrecognized or absent reason falls back to `API_REJECTED` (see
+  `apiFailureCode`). So a new reason string does nothing until both a
+  matching code and a row here exist.
 - Every code added in `cmd/cli/provision_result.go` needs a row here in the
   same MR. Tests enforce the code/stage/exit maps and that this table has
   exactly one `stage != package` row per ctrld code. The `package` rows