gstack/docs/cso-release-qualification.md

Building and qualifying CSO runtimes

bun run build:cso compiles the trusted launcher, helper, and independent watchdog. Bun must support all four --no-compile-autoload-* switches. macOS and Linux also require a C compiler (cc, Clang, or GCC). Linux links the launcher statically; macOS signs it with the hardened runtime. bun run build includes this step and remains a strict release/developer build. Setup feature-probes the four Bun switches and the host native toolchain before building. If either is unavailable, it builds the rest of gstack, removes any stale CSO helpers, and reports /cso unavailable; the skill then reports not assessed and never substitutes repository tooling. Re-run setup after installing the prerequisite. When the probe succeeds, setup keeps the strict failure behavior and checks the artifacts for freshness. The generated bin/gstack-cso-launcher is the setup-built direct command; the compiled core is not the entrypoint. The core's direct-invocation marker catches accidental misuse; it does not authenticate a process against the trusted same-user host. The native launcher is the pre-runtime boundary that strips Bun, Node, and loader injection variables. CSO is not exposed as an npm package bin: its native launcher, compiled core, watchdog, and hidden .gstack-cso-generation manifest must be built and installed together for the current host. Windows also requires the empty .gstack-cso-generation.lock file from that same publication. Windows builds the native launcher with Visual Studio 2022 Build Tools (Desktop development with C++), discovered from Git Bash through the installed Developer PowerShell script; only the core is compiled with Bun. The dedicated Windows CI job checks startup, argument forwarding, and runtime injection separately from static audit and private-state support. POSIX watchdog execution remains unavailable there.

No qualified runtime or scanner image exists yet, so contained runtime and scanner execution is unavailable and setup preloads nothing: the scanner catalog is empty while its build inputs await review, and the runtime catalog's profiles stay unqualified until a private evaluator, still being built, can qualify them.

Every successful fresh install or upgrade asks the trusted launcher to preload the current host platform's qualified runtime and scanner images. This is the only automatic image acquisition path. It first validates the committed catalogs, pins a local Unix-socket Docker endpoint, rejects remote TCP/SSH contexts, and uses a new empty Docker configuration so pulls are anonymous. Only fully qualified registry/repository@sha256:... catalog entries are requested. Missing Docker or public registry access is a visible, nonfatal prerequisite: static audits remain available and a later ./setup retries. The preload stage reserves 30 seconds for endpoint admission and gives each declared image a separate 30-second inspection/download window. Set GSTACK_CSO_IMAGE_PULL_TIMEOUT_SECONDS=120 when registry speed requires a longer per-image window (accepted range: 5–300 seconds). The helper derives an aggregate deadline from the catalog size and caps the complete stage at one hour. A per-image timeout leaves that digest unavailable and continues with the next entry; reaching the hard aggregate deadline stops all remaining work. When a registry prerequisite stops further downloads, setup still checks later catalog entries locally and reports every already-present digest. A later setup resumes from exact local digests. Application, verifier, and scanner execution still use --pull=never with no egress. gstack-cso doctor --repo PATH never downloads; it reports a qualified profile unavailable until that exact digest and platform are already present in the local daemon.

The ordinary free suite exercises contracts without requiring Docker. The free lane's cso-docker-integration job sets GSTACK_CSO_DOCKER_TESTS=1 and runs test/cso-docker-integration.test.ts; unavailable Docker or missing containment prerequisites fail that job. The node-lifecycle and stack-cold files need a staged stack image, so they run in the CSO runtime workflows through test:cso:docker. Run all three locally with a working local daemon:

bun run build:cso
GSTACK_CSO_DOCKER_TESTS=1 bun run test:cso:docker

When the staged-image workflow supplies GSTACK_CSO_TEST_IMAGE, platform, stack, and exact tool versions, the same command also runs stack-specific qualification journeys. A second native qualification job waits for every same-architecture staging job, so Rails receives both its application digest and the staged PostgreSQL digest; a missing matching artifact fails the gate. The Node journey starts with an empty public archive cache, acquires an integrity-pinned dependency through the registry-restricted broker, installs and executes it offline, reproduces a producer-visible fixture defect, and verifies a patch bound to its recorded closure and preparation proofs. The helper issues a runtime_tested RepairBundle only when its out-of-process witness authenticates the external assertions. Canonical project tests run in a target-controlled process, so their completion remains self_reported: target code can forge reporter output or terminate the runner. Recorded command, count, exit, and output hashes are diagnostic evidence; they do not establish target-independent test completion. The tested state remains reserved for a future target-independent witness and is not emitted today. Until the runtime-tested bundle, replay, and current-source recheck journey passes, the held-out repair gate remains pending and catalog promotion is prohibited. The candidate journey also asserts that no run-owned containers, preparation call directories, archive staging directories, or verification work directories remain. A skipped staged-image journey is not a passing cold-start or private held-out-repair result. Bun and Python have cold acquisition, offline install, boot, control, and project-test journeys. Rails runs those checks twice against the same cold public-gem inputs: once with two disposable SQLite connections and once with two databases in a fresh staged PostgreSQL sidecar. The Rails fixture builds the source-platform sqlite3 and pg native gems only in the offline execution phase.

Preparation output leaves its containers through docker cp, which cannot read a tmpfs mount. The acquisition /archives directory and the offline /work directory are therefore Docker local volumes backed by bounded tmpfs, created per container with private mode, the runtime uid/gid, nosuid, and nodev (/archives is also noexec; /work must stay executable for virtualenv and binstub launchers). The runner inspects each container and refuses to start it if Docker did not preserve those options. npm's user and global configs point at two different empty files, because npm refuses to load one file in both roles. Rails application and test containers hold on sleep after run-app copies the source; db:prepare starts only once that copy has finished. Changes to /opt/cso/preparation or an image recipe need a new staging run before native qualification can pass.

Each supervised command has a 300-second ceiling (COMMAND_TIMEOUT_MS in lib/cso/process.ts). Only the dependency fetch and install commands of the acquisition and offline preparation phases get PREPARATION_COMMAND_TIMEOUT_MS, 900 seconds. A Rails lockfile that lists only the ruby platform compiles nokogiri, sqlite3 and pg offline, which takes about 385 seconds at the app role's 0.85 CPU share. Both ceilings are cut to the time left before the journey's aggregate deadline, so the longer ceiling never extends a journey. Helper probes, exports, application starts, db:prepare and project tests keep the 300-second ceiling. A free test in test/cso-docker-mounts.test.ts pins which calls may request the longer ceiling. The CPU share is unchanged, so ISOLATION_POLICY_HASH and the profiles bound to it are unaffected.

When the in-image helper refuses to export a prepared tree, preparation fails with error code PREPARED_EXPORT_REJECTED instead of the generic TOOL_FAILED. The message carries the helper's exit status and its fixed reason, such as prepared tree contains a hard-linked file, or the filesystem errno and syscall. It never includes target paths. Nothing retries. Treat a recurrence as evidence for a defect, not as a flake to rerun.

The Rails offline install runs bundle install --jobs 1. Ruby's File.umask getter sets the process umask to 0 for an instant, and RubyGems calls it for every file it extracts. With two Bundler install threads, a directory created by one thread during that instant comes out 0777, and the export refuses the tree with prepared tree contains a publicly writable directory. That refusal is correct, so the install runs on one thread rather than the gate being relaxed. At the app role's 0.85 CPU share the second thread saved no measurable time.

For hashed Python requirements resolved by pip, the broker records the requested index URL, allowed contacted hosts, and verified wheel hash. Pip's internal files.pythonhosted.org response URL is opaque to that TLS boundary, so resolvedUrl remains null; registryResponseSha256 identifies the acquired archive bytes, not a verified response URL.

Image publication starts with the committed build-inputs.json. Its reviewed revision pins distinct native manifests for every stack and the SBOM generator. Each profile records the inspected source tag and immutable index for provenance, then baseImages keyed by linux/amd64 and linux/arm64; Python additionally records native uv manifests. CI fails if a source tag no longer resolves to the reviewed index, a platform manifest is absent from that index, its image configuration names another architecture, or an executable reports a different runtime or package-manager version. Bun does not declare a Node version because its container's node path is a Bun-backed fallback rather than a Node release. A Bun project with a Node engine constraint therefore receives an explicit prerequisite instead of a fabricated compatibility result.

Configure the repository's cso-runtime-release GitHub environment with required maintainer review and deployments restricted to protected main. The cso-runtime-images.yml workflow runs a build-only native matrix on pull requests. Its manual publication jobs only accept main. They build on native amd64 and arm64 runners, publish to the cso-staging namespace, signs provenance and SBOM attestations, and verifies the workflow identity and source commit. It then pulls that exact digest into the local daemon for tests that prohibit implicit pulls. The per-image smoke test executes the embedded verifier against a loopback service with both a passing control and an intentionally failing control, alongside the containment suite. After all staging rows finish, native qualification jobs run the matching cold-start journey described above. The Node row additionally runs the complete lifecycle journey. Each native job is named qualify-native <runtimeId> and runs the Docker suites with Bun's JUnit reporter (--reporter=junit --reporter-outfile=native-tests.junit.xml). cso-native-evidence.ts maps every native check key to the named tests that prove it for that stack and records the key true only when the report has no failed testcase and each named test appears once, passed, and made at least one assertion. A test that the environment skips, such as the staged-runtime smoke test without GSTACK_CSO_TEST_IMAGE, reports as skipped and leaves its checks false; any false native check fails the job. The pending private checks are the stack's required release gates from cso-runtime-promotion.ts minus its native keys: secretCanaryPassed and watchdogCleanupPassed for PostgreSQL, plus heldOutRepairPassed and accuracyGatesPassed for application stacks. The job uploads the JUnit report beside qualification-evidence.json, which records its digest. Registry publication is staging, not runtime qualification.

GitHub Container Registry creates new packages private. Before a staging digest can enter native or private qualification, a package administrator must make its package public in GitHub's package settings. This is an explicit bootstrap step because GitHub documents the visibility change as irreversible and does not provide a supported workflow API for it. The release jobs use their scoped workflow token to require visibility: public from GitHub's package REST endpoint, then pull the exact platform digest with docker --config pointing at a new directory whose config.json contains only an empty auths object. The GitHub token, Docker credentials, contexts, certificate paths, and credential helpers are absent from that Docker client. A private package, metadata mismatch, failed anonymous pull, or different returned digest stops staging, qualification, and promotion. Both runtime and scanner promotion repeat this anonymous pull check on fresh hosted runners before opening a catalog PR.

The first protected-main staging run therefore stops at the public-package check by design: its push creates the ten cso-staging packages as private. A package administrator makes those ten packages public, then dispatches the staging workflow again. Later runs publish new digests into the same public packages.

Review the retained staging artifacts together with all checks in qualification.json. Cold-start fixtures, held-out repair evaluations, precision/recall thresholds, crash cleanup, and all secret-canary checks must pass before a reviewed catalog change can make a digest executable. A successful protected-main qualification run packages one qualified-runtime.json statement for each stack and architecture under the cso-qualified-runtime-statements artifact. All ten statements bind the same source commit and workflow run. The protected cso-runtime-promote.yml workflow authenticates that run through the Actions API, checks every statement against the committed build profiles, and emits an attested runtime-catalog.candidate.json. The protected job immediately verifies the candidate attestation against its workflow identity, protected-main source ref, source commit, and exact file digest. It then compares previousRevision with the source catalog, commits those same bytes to a fresh branch, and opens a normal review PR. Pushes and pull requests made with the workflow's GITHUB_TOKEN start no workflows, so after pushing the branch the job dispatches free-tests.yml on it with gh workflow run free-tests.yml --ref <branch> (its actions: write grant). That run reports the required free-tests check on the branch head the PR proposes. Opening the PR also requires the repository setting Allow GitHub Actions to create and approve pull requests (Settings → Actions → General → Workflow permissions). Without it gh pr create fails after the push and the dispatch, and a maintainer opens the PR from the pushed branch. A concurrent catalog promotion fails the compare-and-swap or produces a merge conflict instead of silently replacing the newer matrix. The catalog stores a separately recomputable digest of the retained runtime matrix and the digest of the complete external qualification statements. The helper rejects a post-generation image or qualification mutation even if the statement digest field was left unchanged. Rollback retains the prior compatible catalog revision.

The public qualification ingress is cso-runtime-qualification.yml. Set the protected environment variable CSO_QUALIFICATION_ACTOR to the GitHub service account used by the private evaluator. After its held-out assertions and accuracy run pass, that account sends a repository_dispatch event of type cso-runtime-qualified. The payload has the key statements and, optionally, evaluationRef; no other key is accepted. statements contains ten objects with schemaVersion, helperAbi, state, buildRevision, runtimeId, stack, platform, immutable image, exact versions, sourceCommit, and the true-valued checks accepted by cso-runtime-promotion.ts. All ten name one sourceCommit. evaluationRef is an opaque sha256:<64 hex> reference that lets the approver match the dispatch to the private evaluator's verdict. An unprotected dispatch-summary job with no token permissions validates the payload shape and writes evaluationRef, the source commit, and each runtimeId and image pair to the run's job summary. The reviewer reads it there before approving the protected qualify job. evaluationRef never enters the normalized statements, the uploaded evidence, or the catalog.

The protected job checks the actor, re-verifies both OCI attestations against the protected staging workflow and source commit, and replaces the workflow, time, SBOM digest, and provenance digest with values it observed. Attestations are signed in the staging job, before native qualification runs, so a valid attestation alone does not show that native gates passed. cso-staging-run.ts therefore resolves the staging run from each verified SLSA provenance statement: the predicate's runDetails.metadata.invocationId must equal the signing certificate's runInvocationURI, and all ten images must resolve to the same run of this repository. Through the Actions API (actions: read) the ingress then requires that run to be .github/workflows/cso-runtime-images.yml, started by workflow_dispatch on main in this repository, completed with conclusion success, and built from the statements' sourceCommit. The latest attempt's qualify-native <runtimeId> jobs, read across every page of the jobs API, must be exactly one per committed matrix row (ten) and all success. The run and job records are retained as staging-run-evidence. The catalog generator runs last as the final schema gate, and the ingress uploads the statements only after that gate passes. The payload carries no held-out assertion, application source, finding, or repair bundle.

A qualification stays valid only while the inputs it was measured under are unchanged. cso-requalification.ts defines four requalification triggers: the helper ABI (CSO_HELPER_ABI), the isolation policy (ISOLATION_POLICY_HASH, which covers the cso-isolation-v1 policy version and every limit), preparationSha256 (a digest of every source byte compiled or copied into a runtime image: the import closure of the image's verifier and preparation helpers plus the image build context), and the build-inputs.json revision. The staging run's reviewed-inputs job records these values for its source commit as the cso-requalification-triggers artifact. The ingress downloads that artifact from the verified staging run into the evidence, and the promotion generator writes it into promotion.requalification. Generation refuses triggers that differ from the promoting checkout, so a trigger that changed between staging and promotion requires a new staging run. The free suite's cso-requalification.test.ts compares the committed catalog's recorded triggers with the current source and fails when any differs. A change to a trigger therefore either lands with a requalified catalog or withdraws the catalog's runtimes in the same pull request. Independently, the helper refuses to select a runtime recorded under another isolation policy.

The daily precision gate (dailyPrecision95 in cso-eval.ts) divides true positives by reported findings. Hypotheses count in neither term. A finding the evaluator marks instrumentation also counts in neither term: it targets the evaluator's declared fixture instrumentation, such as the canary overlay files or the egress probe route. The seeded issue adds one true positive per cell, however many times it is restated. A finding the evaluator marks incidental is a different issue that a unanimous judge panel confirmed is real in that source. It adds one true positive per distinct incidentalKey in its cell, never counts toward recall, and needs no seeded match. Recall still measures only the seeded issue, and the thresholds are unchanged.

The judge observes held-out attacks in one boot: a VerificationRequest's security field is either one assertion or an array of up to eight assertions with distinct names, judged in a single before/after pair. The before phase is intended_failure only when every assertion shows its vulnerable expectation, and the after phase passes only when every assertion passes its fixed expectation. A single assertion keeps its original hashes. The verifier is compiled into the runtime images, so array requests need images staged from a commit that includes it.

Evaluation-only helper builds

The private evaluator drives the real helper against staged digests before any gate is measured. It never edits the committed catalog. Instead, bun scripts/cso-runtime-promotion.ts evaluation-candidate --staged-root <dir> --output <new-file> converts one staging run's ten staged-image.json records into an evaluation-only catalog. Its revision is cso-eval-<runId>-<sha12>, it carries an evaluation record instead of promotion, and every runtime is an evaluation_candidate whose qualification has kind: "evaluation" and no release-gate field. Each runtime's qualifiedAt is the current time unless --created-at <ISO time> names it. An evaluation that builds one unit per platform must pass the same --created-at to every build: the scorer requires both platforms' producers to embed identical catalog bytes. Then

bun run build:cso -- --evaluation-candidate <catalog.json> --output <new-directory>

compiles the helper core and cso-eval-producer from a private copy of the source with that catalog embedded, adds the native launcher and watchdog, and writes the five-artifact producer unit to a new directory outside the checkout. Beside the unchanged .gstack-cso-generation manifest, it writes .gstack-cso-evaluation, which marks the unit evaluation-only and records the catalog revision, catalog SHA-256, and core SHA-256. The helper reports "evaluationOnly": true from --version, labels its catalog evaluation-only in runtime readiness, and refuses provision-images --setup-summary. Distribution refuses it at every step. A normal build:cso stops when lib/cso/runtime-catalog.json carries a cso-eval- revision. Setup fails when bin/ holds an .gstack-cso-evaluation manifest or a launcher that reports evaluationOnly. Catalog promotion and transitions reject evaluation catalogs, and the runtime catalog validator rejects an evaluation catalog that carries promotion, release-gate fields, or a revision that does not name its staging run and source commit.

The current reviewed profiles and unpromoted catalog establish this release process; they do not claim published gstack images, passed containment, or measured security accuracy. Private held-out find-to-repair evaluations remain mandatory release gates for Node, Bun, Python, and Rails. The producer-visible Node lifecycle fixture and the other cold-start journeys do not satisfy those gates. Qualification artifacts therefore record private checks as "pending" and qualified: false until all four private evaluations and the accuracy thresholds pass. The remaining external prerequisite is one successful protected-main run that uploads all ten authenticated statements. This sandbox cannot create that evidence because it has neither a native arm64 runner nor the private held-out assertions.

Attestation behavior follows Docker's SBOM documentation and GitHub CLI's verification policy. Public-package behavior follows GitHub's package permissions documentation, GitHub's package REST endpoint, and Docker's client configuration documentation, inspected on September 11, 2026.