ctrld/SPEC.md

# SPEC: Stable customer-visible provisioning failure codes

Issue: [#586](https://gitlab.int.windscribe.com/controld/clients/ctrld/-/issues/586)
Requested by: Catt Garrod (@catt). Scope expanded by: Anthony Wong (@anthony).

## 1. Objective

Terminal provisioning failures in ctrld — bootstrap/API setup, listener
binding, and service installation/startup — must produce a stable,
support-facing failure identifier that survives process exit and reaches
both manual CLI users and MDM-driven installs. A customer or admin reports
one code; Support maps it to a scenario and a next action without asking
for reruns or verbose logs.

Motivating incident (v1.5.5, macOS): provisioning reached the Control D
API, then died with only `FTL listener.0 could not find available listen
ip and port`. The per-address UDP/TCP bind errors existed only at Info
level in an in-memory logger and vanished on exit. The macOS pkg
`postinstall` discards ctrld's stdout/stderr entirely and judges success
by plist existence, so nothing useful reached the MDM log.

**Users:** end customers and IT admins reporting failures; Support agents
triaging them; MDM/RMM operators reading installer logs.

### Failure contract (agreed design)

Three surfaces, all carrying the same identifier:

1. **Result file** — on terminal provisioning failure, ctrld writes a
   small redacted JSON file (atomic write: temp + rename) in the ctrld
   home directory (same base dir as the internal `ctrld.log`,
   via `absHomeDir`). Removed/overwritten on later successful
   provisioning so stale failures don't mislead. Schema:

   ```json
   {
     "version": 1,
     "timestamp": "2026-08-18T12:00:00Z",
     "stage": "listener",
     "code": "LISTENER_BIND_FAILED",
     "exit_code": 41,
     "message": "could not find available listen ip and port",
     "detail": {
       "attempts": [
         {"addr": "127.0.0.1:53", "proto": "udp", "os_error": "address already in use"}
       ]
     }
   }
   ```

   `detail` is bounded (cap recorded bind attempts; cap string lengths)
   and redacted by construction: no provisioning tokens, resolver IDs,
   config contents, or unrelated host data.

2. **Exit code + final stderr line** — the installer-facing command
   (`ctrld start`, and `ctrld run` when run manually in the foreground)
   exits with a stage-scoped code and prints one final line containing
   the string code and stage, e.g.
   `provisioning failed: stage=listener code=LISTENER_BIND_FAILED (exit 41)`.

3. **Installer log (MDM path)** — `scripts/pkg/postinstall` stops
   discarding the signal: it captures `ctrld start`'s output to a
   private temp file, extracts only the fixed-charset identifier line
   (`stage=[a-z]* code=[A-Z_]* (exit [0-9]*)` — structurally unable to
   carry the token), and echoes it with the exit code into the
   installer log. The result file's `message`/`detail` fields are
   deliberately never surfaced there. The plist-existence check remains
   the final success gate.

### Identifier format

- **Primary identifier: stable string codes.** Initial set —
  bootstrap: `API_UNREACHABLE`, `API_REJECTED`, `API_DEVICE_INVALID`;
  listener: `LISTENER_BIND_FAILED`, `LISTENER_CONFIGURED_ADDR_UNAVAILABLE`;
  service: `SERVICE_INSTALL_FAILED`, `SERVICE_START_FAILED`,
  `SERVICE_SELFCHECK_FAILED`. Codes are append-only; renames are new
  codes plus a deprecation note in the mapping doc.
- **Secondary: stage-scoped process exit codes** as a coarse machine
  signal: bootstrap 30–39, listener 40–49, service install/start 50–59.
  Each string code owns one exit code. Existing contracts are untouched:
  `ctrld status` 0–3, deactivation-pin 126, success 0.
- One underlying failure maps to one code on every path (manual CLI and
  MDM), on both branches.

### Propagation (daemon → installer)

The listener/bootstrap fatals fire inside the daemon process
(`ctrld run` under launchd/systemd/SCM), not in `ctrld start`. The
daemon writes the result file before exiting; the existing log-socket
exit notification (`notifyExitToLogServer`) already unblocks `ctrld
start`'s self-check. `ctrld start` then reads the result file, prints
the identifier, and exits with the mapped stage exit code. The daemon's
own exit-status semantics toward service managers are preserved —
in particular the deliberate exit-0 on permanent API rejection that
protects the restart-policy budget; the result file carries the failure
identity in that case.

### Support mapping

`docs/provisioning-failure-codes.md` in this repo: one row per code —
code, stage, exit code, failure scenario, next safe troubleshooting
action or evidence request. Updated in the same MR whenever a code is
added or changed.

### Branch scope

Full implementation on **both** `v1.0` (release line for v1.5.5) and
`master`. The branches diverge heavily (`v1.0`: zerolog fork,
`commands.go`, `service_status.go`, macOS pkg scripts; `master`: zap,
inline commands, no pkg scripts), so this is one shared contract
(codes, exit-code ranges, file schema, doc) implemented twice, as two
MRs referencing #586.

## 2. Commands

- Build: `go build ./...`
- Test: `go test ./cmd/cli/...` (full: `go test ./...`)
- Vet: `go vet ./...`
- Branch workflow: feature branch off `v1.0` for the v1.0 MR; separate
  feature branch off `master` for the port MR. Rebase, never merge the
  base branch in.

## 3. Project structure

New and touched files on `v1.0` (master port mirrors the same contract
at its equivalent emission points in its `cli.go`):

- `cmd/cli/provision_result.go` (new) — stage + code enums, exit-code
  mapping, result-file schema, atomic write/read/clear helpers,
  bounded/redacted detail builders. Pattern follows `service_status.go`
  (small file: named constants + classifier + dedicated tests).
- `cmd/cli/provision_result_test.go` (new).
- `cmd/cli/cli.go` — emission points: `run()` bootstrap failure branches
  (permanent rejection, invalid-device, fatal fetch), and
  `tryUpdateListenerConfig` / `tryUpdateListenerConfigIntercept` fatals,
  which now record per-attempt `{addr, proto, os_error}` bind detail.
- `cmd/cli/commands.go` — `initStartCmd`: doTasks install/start failures
  and the self-check failure branch read the result file, print the
  identifier, and exit with the stage code (replacing bare `os.Exit(1)`
  on those paths).
- `scripts/pkg/postinstall` — propagate exit code + result-file contents
  into the installer log (v1.0 only; master has no pkg scripts).
- `docs/provisioning-failure-codes.md` (new) — support mapping.

## 4. Code style

- Per repo conventions and global rules: guard clauses, small functions,
  descriptive names, explicit error handling — never weaken existing
  handling (e.g. keep the permanent-rejection exit-0 rationale intact).
- Comments only for non-obvious constraints (e.g. why the daemon must
  still exit 0 on permanent rejection), simple-english, self-contained —
  no issue/MR references in code.
- Match each branch's logging idiom: zerolog fork on `v1.0`, zap on
  `master`. No new dependencies.
- Conventional Commits; MR titles in simple-english; both MRs reference
  #586 (release-line MR carries `Closes #586`).

## 5. Testing strategy

Test-first where the harness allows. Coverage required by the issue:

- **Code/mapping unit tests** — every string code maps to exactly one
  stage and one in-range exit code; ranges don't collide with existing
  contracts (0–3 status, 126 pin).
- **Result file round-trip** — write/read/clear; atomic write; stale
  file removed on success.
- **Redaction** — serialize a result built from inputs containing a
  provision token, resolver ID, and config content; assert none appear.
- **Listener bind failure (regression test for the incident)** — occupy
  a port, drive the listener-config path to exhaustion, assert the
  result records `LISTENER_BIND_FAILED` with attempted address, UDP/TCP
  operation, and OS error (`address already in use`-class).
- **Bootstrap failures** — mock API: permanent 4xx → `API_REJECTED`;
  invalid-device 40402 → `API_DEVICE_INVALID`; unreachable →
  `API_UNREACHABLE`.
- **Service install/start/self-check failures** — injected task
  failures assert code selection and `ctrld start` exit code.
- **MDM surface** — shell-level check of `postinstall` failure branch
  (result file present → correct log line and exit), aligned with the
  existing `test-scripts/` approach; manual pkg verification steps
  documented in the MR.
- Both branches: the shared contract tests exist on both; branch-specific
  emission tests match each branch's structure.

## 6. Boundaries

**Always:**
- Redact tokens, resolver IDs, config contents, host data from every
  customer-visible surface (result file, stderr line, installer log).
- Preserve existing exit-code contracts (`ctrld status` 0–3, pin 126)
  and the daemon's service-manager-facing exit semantics.
- Bound all recorded detail (attempt counts, string lengths).
- Keep codes append-only once merged.

**Ask first:**
- Changing the daemon's (`ctrld run` under a service manager) exit codes
  or restart-relevant behavior beyond writing the result file.
- Adding any persisted file outside the ctrld home directory.
- Expanding scope to runtime (post-provisioning) failures — this ticket
  owns terminal provisioning failures only.

**Never:**
- Print or persist the provisioning token (the reason postinstall
  discards output today — the replacement surface must stay token-free).
- Auto-detect or kill conflicting processes (explicitly out of scope).
- Break `ctrld status`'s documented exit-code contract.