Files
gstack/docs/cso-release-qualification.md
Garry TanandOpenAI Codex 4a3c6a8a3c v1.87.0.0 feat: add verified CSO audits and replayable repair bundles (#2852)
* feat(cso): add verified audits and replayable repair bundles

* fix(cso): harden qualification and setup boundaries

* fix(cso): assemble security canaries at runtime

* fix(cso): bound release proof and maintenance work

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* fix(cso): require complete evaluation reports

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* fix(cso): replay expired snapshots from supplied source

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* test(cso): synchronize DNS cancellation assertion

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* chore(ship): exempt repository owner from liveness proof

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* test(cso): make recheck retention overlap deterministic

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* chore: bump version and changelog (v1.85.0.0)

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* fix(cso): pass native release gates

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* chore: move release to v1.86.0.0

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* fix(cso): resolve rechecks by finding

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* chore: move release to v1.87.0.0

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* fix(cso): pass macOS and Windows release gates

Normalize BSD wc output, compare Windows paths by filesystem identity, preserve portable snapshot race coverage, and narrow POSIX-only Windows fixtures.

Co-Authored-By: OpenAI Codex <noreply@openai.com>

* fix(cso): harden native verification gates

* fix(cso): refine Windows native diagnostics

* test(cso): isolate Windows Git startup failure

* test(cso): stabilize Windows native diagnostics

* fix(cso): support hardened Git on Windows

* fix(cso): close final verification gaps

* test(cso): bound cold Docker fixture setup

* fix(cso): restore cross-platform free-suite gates

---------

Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-14 15:14:58 -07:00

13 KiB
Raw Permalink Blame History

Building and qualifying CSO runtimes

bun run build:cso compiles the trusted launcher, helper, and independent watchdog. Bun must support all four --no-compile-autoload-* switches. macOS and Linux also require a C compiler (cc, Clang, or GCC). Linux links the launcher statically; macOS signs it with the hardened runtime. bun run build includes this step and remains a strict release/developer build. Setup feature-probes the four Bun switches and the host native toolchain before building. If either is unavailable, it builds the rest of gstack, removes any stale CSO helpers, and reports /cso unavailable; the skill then reports not assessed and never substitutes repository tooling. Re-run setup after installing the prerequisite. When the probe succeeds, setup keeps the strict failure behavior and checks the artifacts for freshness. The generated bin/gstack-cso-launcher is the setup-built direct command; the compiled core is not the entrypoint. The core's direct-invocation marker catches accidental misuse; it does not authenticate a process against the trusted same-user host. The native launcher is the pre-runtime boundary that strips Bun, Node, and loader injection variables. CSO is not exposed as an npm package bin: its native launcher, compiled core, watchdog, and hidden .gstack-cso-generation manifest must be built and installed together for the current host. Windows also requires the empty .gstack-cso-generation.lock file from that same publication. Windows builds the native launcher with Visual Studio 2022 Build Tools (Desktop development with C++), discovered from Git Bash through the installed Developer PowerShell script; only the core is compiled with Bun. The dedicated Windows CI job checks startup, argument forwarding, and runtime injection separately from static audit and private-state support. POSIX watchdog execution remains unavailable there.

Every successful fresh install or upgrade asks the trusted launcher to preload the current host platform's qualified runtime and scanner images. This is the only automatic image acquisition path. It first validates the committed catalogs, pins a local Unix-socket Docker endpoint, rejects remote TCP/SSH contexts, and uses a new empty Docker configuration so pulls are anonymous. Only fully qualified registry/repository@sha256:... catalog entries are requested. Missing Docker or public registry access is a visible, nonfatal prerequisite: static audits remain available and a later ./setup retries. The preload stage reserves 30 seconds for endpoint admission and gives each declared image a separate 30-second inspection/download window. Set GSTACK_CSO_IMAGE_PULL_TIMEOUT_SECONDS=120 when registry speed requires a longer per-image window (accepted range: 5300 seconds). The helper derives an aggregate deadline from the catalog size and caps the complete stage at one hour. A per-image timeout leaves that digest unavailable and continues with the next entry; reaching the hard aggregate deadline stops all remaining work. When a registry prerequisite stops further downloads, setup still checks later catalog entries locally and reports every already-present digest. A later setup resumes from exact local digests. Application, verifier, and scanner execution still use --pull=never with no egress. gstack-cso doctor --repo PATH never downloads; it reports a qualified profile unavailable until that exact digest and platform are already present in the local daemon.

The ordinary free suite exercises contracts without requiring Docker. The cso-docker-integration CI job sets GSTACK_CSO_DOCKER_TESTS=1; unavailable Docker or missing containment prerequisites fail that job. Run it locally with a working local daemon:

bun run build:cso
GSTACK_CSO_DOCKER_TESTS=1 bun run test:cso:docker

When the staged-image workflow supplies GSTACK_CSO_TEST_IMAGE, platform, stack, and exact tool versions, the same command also runs stack-specific qualification journeys. A second native qualification job waits for every same-architecture staging job, so Rails receives both its application digest and the staged PostgreSQL digest; a missing matching artifact fails the gate. The Node journey starts with an empty public archive cache, acquires an integrity-pinned dependency through the registry-restricted broker, installs and executes it offline, reproduces a producer-visible fixture defect, and verifies a patch bound to its recorded closure and preparation proofs. The helper issues a runtime_tested RepairBundle only when its out-of-process witness authenticates the external assertions. Canonical project tests run in a target-controlled process, so their completion remains self_reported: target code can forge reporter output or terminate the runner. Recorded command, count, exit, and output hashes are diagnostic evidence; they do not establish target-independent test completion. The tested state remains reserved for a future target-independent witness and is not emitted today. Until the runtime-tested bundle, replay, and current-source recheck journey passes, the held-out repair gate remains pending and catalog promotion is prohibited. The candidate journey also asserts that no run-owned containers, preparation call directories, archive staging directories, or verification work directories remain. A skipped staged-image journey is not a passing cold-start or private held-out-repair result. Bun and Python have cold acquisition, offline install, boot, control, and project-test journeys. Rails runs those checks twice against the same cold public-gem inputs: once with two disposable SQLite connections and once with two databases in a fresh staged PostgreSQL sidecar. The Rails fixture builds the source-platform sqlite3 and pg native gems only in the offline execution phase.

For hashed Python requirements resolved by pip, the broker records the requested index URL, allowed contacted hosts, and verified wheel hash. Pip's internal files.pythonhosted.org response URL is opaque to that TLS boundary, so resolvedUrl remains null; registryResponseSha256 identifies the acquired archive bytes, not a verified response URL.

Image publication starts with the committed build-inputs.json. Its reviewed revision pins distinct native manifests for every stack and the SBOM generator. Each profile records the inspected source tag and immutable index for provenance, then baseImages keyed by linux/amd64 and linux/arm64; Python additionally records native uv manifests. CI fails if a source tag no longer resolves to the reviewed index, a platform manifest is absent from that index, its image configuration names another architecture, or an executable reports a different runtime or package-manager version. Bun does not declare a Node version because its container's node path is a Bun-backed fallback rather than a Node release. A Bun project with a Node engine constraint therefore receives an explicit prerequisite instead of a fabricated compatibility result.

Configure the repository's cso-runtime-release GitHub environment with required maintainer review and deployments restricted to protected main. The cso-runtime-images.yml workflow runs a build-only native matrix on pull requests. Its manual publication jobs only accept main. They build on native amd64 and arm64 runners, publish to the cso-staging namespace, signs provenance and SBOM attestations, and verifies the workflow identity and source commit. It then pulls that exact digest into the local daemon for tests that prohibit implicit pulls. The per-image smoke test executes the embedded verifier against a loopback service with both a passing control and an intentionally failing control, alongside the containment suite. After all staging rows finish, native qualification jobs run the matching cold-start journey described above. The Node row additionally runs the complete lifecycle journey. Registry publication is staging, not runtime qualification.

GitHub Container Registry creates new packages private. Before a staging digest can enter native or private qualification, a package administrator must make its package public in GitHub's package settings. This is an explicit bootstrap step because GitHub documents the visibility change as irreversible and does not provide a supported workflow API for it. The release jobs use their scoped workflow token to require visibility: public from GitHub's package REST endpoint, then pull the exact platform digest with docker --config pointing at a new directory whose config.json contains only an empty auths object. The GitHub token, Docker credentials, contexts, certificate paths, and credential helpers are absent from that Docker client. A private package, metadata mismatch, failed anonymous pull, or different returned digest stops staging, qualification, and promotion. Both runtime and scanner promotion repeat this anonymous pull check on fresh hosted runners before opening a catalog PR.

Review the retained staging artifacts together with all checks in qualification.json. Cold-start fixtures, held-out repair evaluations, precision/recall thresholds, crash cleanup, and all secret-canary checks must pass before a reviewed catalog change can make a digest executable. A successful protected-main qualification run packages one qualified-runtime.json statement for each stack and architecture under the cso-qualified-runtime-statements artifact. All ten statements bind the same source commit and workflow run. The protected cso-runtime-promote.yml workflow authenticates that run through the Actions API, checks every statement against the committed build profiles, and emits an attested runtime-catalog.candidate.json. The protected job immediately verifies the candidate attestation against its workflow identity, protected-main source ref, source commit, and exact file digest. It then compares previousRevision with the source catalog, commits those same bytes to a fresh branch, and opens a normal review PR. A concurrent catalog promotion fails the compare-and-swap or produces a merge conflict instead of silently replacing the newer matrix. The catalog stores a separately recomputable digest of the retained runtime matrix and the digest of the complete external qualification statements. The helper rejects a post-generation image or qualification mutation even if the statement digest field was left unchanged. Rollback retains the prior compatible catalog revision.

The public qualification ingress is cso-runtime-qualification.yml. Set the protected environment variable CSO_QUALIFICATION_ACTOR to the GitHub service account used by the private evaluator. After its held-out assertions and accuracy run pass, that account sends a repository_dispatch event of type cso-runtime-qualified. The payload has one key, statements, containing ten objects with schemaVersion, helperAbi, state, buildRevision, runtimeId, stack, platform, immutable image, exact versions, sourceCommit, and the true-valued checks accepted by cso-runtime-promotion.ts. The ingress checks the actor and matrix size, re-verifies both OCI attestations against the protected staging workflow and source commit, replaces the workflow, time, SBOM digest, and provenance digest with values it observed, then runs the catalog generator as its final schema gate. It uploads the statements only after that gate passes. The payload carries no held-out assertion, application source, finding, or repair bundle.

The current reviewed profiles and unpromoted catalog establish this release process; they do not claim published gstack images, passed containment, or measured security accuracy. Private held-out find-to-repair evaluations remain mandatory release gates for Node, Bun, Python, and Rails. The producer-visible Node lifecycle fixture and the other cold-start journeys do not satisfy those gates. Qualification artifacts therefore record private checks as "pending" and qualified: false until all four private evaluations and the accuracy thresholds pass. The remaining external prerequisite is one successful protected-main run that uploads all ten authenticated statements. This sandbox cannot create that evidence because it has neither a native arm64 runner nor the private held-out assertions.

Attestation behavior follows Docker's SBOM documentation and GitHub CLI's verification policy. Public-package behavior follows GitHub's package permissions documentation, GitHub's package REST endpoint, and Docker's client configuration documentation, inspected on September 11, 2026.